Engine core — registry, context, interfaces, tracer, schema

Goal

• Build the off-path harness over the foundational RootClosure/InvocationFrame store.
• For plan/bug workflows, keep the plan/tracker API authoritative for workflow/domain state; frames hold invocation protocol plus a derived in-flight-effect resume cursor, repaired or rejected on disagreement.
• Let the state machine select every action; active capability fixes exact frame scope, ContextSnapshot drives script generation of context/model/sandbox/evaluation selection, and ResultSubmission is the only task-completion input.

ComponentMetadata is the self-description SSOT for every engine component (the registry-as-SSOT discipline of missions.md §ori_registry, applied to the engine’s own primitives): it carries intent (one-line “what this component is FOR” — the per-component mission), spec_ref (pointer to the behavioral contract the component must honor), and changelog (append-only per-component change history) ALONGSIDE the structural name/side_effect_class/inputs/outputs.

• Replaces the detached, prose, script-file-keyed improve-tooling/*-design.md model: intent + change history live WITH the component, are registry-queryable, and are surfaced by --docs (sec rails-components) so /improve-tooling reads a component’s purpose + spec + history BEFORE touching it.
• Deep rationale (rejected alternatives, dead-ends) stays in the design-log §4 Lessons prose surface; the metadata owns intent/spec/mechanical-changelog, not narrative (SRP split — no third changelog home).

Self-hosting constraint

Pure greenfield — scripts/workflow_engine/ imports nothing from the live serve path and is imported by nothing on it. No risk to the orchestrator executing this plan.

Implementation sketch

• interfaces.py: LogicGate/Action/Banner contracts plus six-part ContextAssembly, ResultSubmission, ModelRoute, SandboxPolicy, EvaluationDefinition, ContinueRuntimeView, and ClientTransportDirective contracts; import the root-closure-owned TerminalReceipt type without redefining it.
  • intent: required non-empty SINGLE-LINE str (one-line component mission) — registry load rejects an empty or newline-containing intent (the one-line contract is shape-enforced, not advisory, since --docs renders it as a one-liner).
  • spec_ref: required {doc, anchor, pin} — doc = repo-relative path to the governing spec/rule (plans/workflow-engine-rework/spec/workflow-state-machines.md or a .claude/rules/*.md), anchor = a stable heading/section anchor within doc, pin = SHA-256 of the anchored section’s text at authoring time. All three sub-fields are required non-empty; registry load rejects a spec_ref missing/empty doc, anchor, or pin. Staleness is DETECTABLE (not silent dead-ref drift): a scripts/workflow_engine/tests/ probe recomputes the anchored-section hash and FAILS on pin mismatch (the spec moved → re-review + re-pin), mirroring the envelope-version pin discipline (intel_repo/.claude_rules_copy/plan-corpus-version.json mismatch → exit 1). The pin is a CI-time staleness gate, not a production read path.
  • changelog: an INLINE append-only list of {date, change, why} entries (newest-last) co-located with the component (each metadata declaration owns its history — survives-with / dies-with the component). date = ISO-8601 YYYY-MM-DD; change + why = non-empty str. NO changelog_ref, NO pruning, NO deletion — append-only is absolute (entries only grow).
  • Registry load-time validates SHAPE only: rejects a component whose metadata omits intent/spec_ref, carries an empty/multi-line intent, carries a spec_ref missing/empty doc/anchor/pin, OR carries a malformed changelog entry (missing field / non-ISO date) — load-time has no prior snapshot, so it CANNOT enforce append-only history; that is the committed-state test’s job (test strategy below).
• context.py: WorkflowContext wraps scripts/plan_corpus/read.py (load_plan, query_next_unblocked, query_success_criteria, …) into the sole scoped plan-state surface; snapshots external reads once per reconcile (git/fs/test-all/graph/client lifecycle) so every gate sees the SAME snapshot. It never reconstructs plan state from frame, transcript, goal, loop, or scratch data.
• Consume the root-closure section’s execution-store and invocation contracts; do not redefine them.
• selectors.py: one fixed-schema selector engine for instructions, knowledge, memory, examples, tools, guardrails, and reminders; paths are one optional fact.
• assembly.py: compose six context classes plus one authoritative GuidanceEnvelope, provenance, budgets, root/frame digests, task token, sandbox/model route, and evaluation contract once per handoff. Generate every LLM-visible prompt/directive from registered script-owned catalogs plus the immutable ContextSnapshot, and hash the exact rendered bytes.
• continue_runtime.py: expose the sole re-entry read surface. ContinueRuntimeAPI.read accepts only script-issued identity-bound args, drives reconciliation to the next delivery boundary, and returns active {domain_target_ref, closure_id, generation, authoritative_state_digest, current_task=guidance.now, next_upcoming_task=guidance.next}, completed {completed_report=TerminalReceipt}, or typed retry_later {failure_receipt, keep_transport_active=true} after its bounded internal retry policy exhausts. A goal/loop makes no second call in that trigger; retry_later emits no LLM task or lifecycle mutation, and the next trigger retries. The API never accepts caller-authored task text, and the next preview grants no execution authority.
• reminders.py: activation/lifetime/repeat/evidence/violation/recurrence resolver over the shared ContextSnapshot.
• engine.py: select action, assemble context, authorize dispatch, and own ExecutionPump drive-until-quiescent; adapters remain transport-only.
• Make reconcile the fixed entry for every plan/bug workflow. Derive the next semantic node from the current plan/tracker API projection; permit a durable frame node_cursor only when bound to one unresolved journaled effect/wait/finalizer, then clear it on typed settlement and reconcile again.
• Define closed EngineTurn, ContinueRuntimeView, ClientTransportDirective, FailureDisposition, and NextActionContract schemas; consume the root-closure-owned TerminalReceipt schema. Drive dispatch/invoke/progress turns until typed quiescence; require exactly one authorized action/target/return contract for every runnable state.
• Define GateCall/ActionCall unions; require structured args for every parameterized component and reject colon-suffixed component names.
• Validate every canonical source-spec JSON example against workflow.schema.json. This section shipped the initial domain_artifact | virtual_root closure; the demonstrated direct-service non-composability in spec sec 10.3.1 reopens it, and the later engine-spec/extension/migration sections own the paired addition of protocol_root across schema, validation, vocabulary, coverage, and tests. Keep fixed reconcile for applicable workflows and reject undeclared node keys and parallel mini-grammars.
• Require explicit result_path/result_mode bindings for every non-unit Action or invoke result; validate before replace/merge/append and permit pathless discard only.
• Define retry-bounded FinalizerCall and mandatory on_finalizer_error routing; terminal commit remains provisional until every finalizer succeeds.
• Restrict action XOR invoke to task nodes; choice nodes use always, and terminal effects use finalizers/on_commit rather than bare action keys.
• Implement reconcile-time after transitions from durable per-frame entered_at; restart uses the persisted timestamp, fires once, and never delegates workflow transitions to ReminderScheduler.
• Represent settled cancellation and abandonment as explicit EngineTurn receipts; define one script-owned successor for every FailureDisposition, including transient retry/external wait and authenticated human classification.
• Build NextActionContract and mandatory state/event Reminder instances before dispatch; fail closed on missing/ambiguous mappings and expose only action-scoped tools/result schema.
• Define GuidanceEnvelope {now: NextActionContract,next: SuccessorPreview,banned} as ContextAssembly.guidance; do not duplicate its authority in other fields.
• scheduler.py: own active ReminderScheduler only; schedule durable ReminderLease refresh independently, deduplicate/acknowledge/fence, and recover leases without granting transition authority. engine.py remains the sole ExecutionPump owner.
• failure_evidence.py: normalize volatile failure fields into versioned FailureEvidenceFingerprint while preserving raw evidence.
• authorization.py + result_ingestion.py: enforce exact script prompt provenance plus tool/resume/result capabilities and accept typed LLM/reminder/tool events before any successor transition. Reject model/client-authored or paraphrased control prompts and changed ClientTransportDirective arguments.
• model_routing.py + sandbox.py: select root-pinned model/provider and task-token tool/filesystem/network/process authority from script policy.
• evaluations.py: run versioned deterministic output and trajectory/rubric evaluations; return evidence to the state machine without route authority.
• tracer.py: record context disclosures, model route, tool/sandbox effects, retries, result, cost, latency, evaluation verdicts, and transitions.
• registry.py: scan actions//logic_gates//banners/; resolve name->class; an undeclared name is a load-time error; a component whose ComponentMetadata omits intent or spec_ref is ALSO a load-time error (every registered component is self-describing). Mirror REGISTERED_HALT_STATES’s name->module registry shape. The registry is the query surface --docs (sec rails-components) renders intent/spec/changelog from.

Spec references

• plans/workflow-engine-rework/spec/workflow-state-machines.md sec 10/10.2 (two-layer engine, interfaces, node/transition model); sec 11.1 (concepts). Precedents: scripts/commit_push/state_machine.py + scripts/commit_push/rules/; scripts/impl_hygiene_review_runtime/rails.py compose_next_action/advance.

Test strategy / acceptance

• Unit tests per module.
• Matrices: node/transition; selector/fact; static/dynamic context; model/risk/budget; sandbox/tool scope; output/trajectory eval; client/assembly; root/frame/target/result/return; client lifecycle/directive/terminal-receipt match.
• Pins: action XOR invoke; no path-only selection; Action args cannot carry transition keys or successor-node references; authorized child succeeds; unauthorized target rejects; return folds once; plan/tracker workflow state wins every frame-cursor disagreement; an unbound/stale cursor schema-rejects; settled effects clear the cursor and re-enter reconcile; an artifact workflow whose initial node bypasses reconcile, whose settled-effect target bypasses it, or whose reconcile node is unreachable schema-rejects; a semantic result path without an owning sanctioned plan/tracker writer schema-rejects; an opaque protocol result reference may select only its registered consume/fold/persist Action or fail-closed engine_error; no execution projection mutates domain state except through a sanctioned state-machine Action; every declared Action error has exactly one retry/catch/terminal disposition, including InvalidInput from classify_failure routing fail-closed to engine_error.
• Liveness/direction pins: progress cannot stop; results auto-select successors; missing or non-script-generated guidance rejects; timer refresh occurs unprompted; stale/duplicate timers cannot alter authority; repeated evidence cannot spin; every goal/loop trigger can only call ContinueRuntimeAPI.read, relaying active current/next or completed report unchanged and treating retry_later as no-task/no-second-call/no-lifecycle-mutation; only an engine-sealed matching TerminalReceipt can authorize a client transport to stop.
• Metadata pins:
• (1) registry SHAPE-reject — negative pin (one case per shape rule): a component is a load-time error when its ComponentMetadata omits intent/spec_ref, carries an empty or multi-line intent, carries a spec_ref missing/empty doc/anchor/pin, OR carries a malformed changelog entry (missing field / non-ISO-8601 date).
• (2) changelog append-only — negative pin via a COMMITTED-STATE diff probe (NOT load-time, which has no prior snapshot): a scripts/workflow_engine/tests/ test diffs each component’s current changelog against its git show HEAD:<file> prior version and FAILS if any prior entry was mutated or removed (only appends pass), mirroring the ledger §5 append-only discipline.
• (3) spec_ref staleness — negative pin: a component whose spec_ref anchored-section hash no longer matches its pin raises a detectable mismatch (re-review + re-pin required).
• (4) round-trip — positive pin: intent/spec_ref/changelog round-trip from a registered component through the registry query surface (the data --docs consumes).
• (5) vocabulary use-site parity — every spec Used by section resolves to an actual target wiring/reference, and every workflow/reminder component reference resolves back to one registered vocabulary row. Gates: sc-c0de5e1f proves metadata pins (1)-(4) and sc-0003 proves the bidirectional vocabulary/use-site parity in pin (5).

Runtime gate: sc-c071a002 proves the sole ContinueRuntimeAPI implementation and its active/completed/retry-later response boundary.

Work Items

• interfaces.py: LogicGate/Action/Banner ABCs + ComponentMetadata (side_effect_class + intent + spec_ref + append-only changelog as the per-component self-description SSOT)
• context.py: WorkflowContext wrapping read.py as the scoped rails surface (snapshot-once), including immutable plan-derived WorkItemReconcileProjection plus protocol-receipt references that cannot supply semantic state
• registry.py: name->class resolver over the component folders + undeclared-name load error + source-spec audit proving every referenced atomic/composite gate is registered exactly once and every vocabulary Used by claim has an actual target use site
• selectors.py: implement the one fixed-schema selector engine for instructions, knowledge, memory, examples, tools, guardrails, and reminders (spec sec 10.2 component-model table SelectorEngine: ‘select prompt/rule/reminder definitions with one schema’), where a path is one optional fact among the typed selection facts and never the sole selection criterion (Test strategy pin: ‘no path-only selection’). This module has its own Implementation-sketch bullet and its own ‘selector/fact’ test matrix row distinct from ContextSnapshot/ContextAssembly, but currently has no corresponding work item — risking its selection logic being reimplemented ad hoc inside w-2314f6bf/w-a4a2386c/w-ef86e134 instead of centralized in one shared, registry-visible component.
• engine.py: SM driver with fixed plan/tracker-derived reconcile entry, exactly-one derived next-node route, receipt-bound in-flight node cursors, all settled effects returning through reconcile, always/action/invoke/on_done/on_commit/finalizers, reconcile-time after/entered_at semantics, total declared-error routing (including classify_failure InvalidInput -> engine_error), and next_action emit
• tracer.py append-only trajectory log with context/model/tool/sandbox/retry/result/cost/latency/eval events + schema/workflow.schema.json validation of closed state_authority, initial/reachable domain reconcile, settled-effect return targets, plan-writer-backed semantic result paths, consume-only opaque protocol result references, and virtual-root constraints
• Define ReminderDefinition, ReminderInstance, and ReminderContext plus activation classes (always, while-state, when-condition), audience, delivery, enforcement, persistence, evidence, and violation fields in a schema-validated mapping catalog
• Implement the ReminderResolver as the single typed-facts-in/reminders-out pipeline with deterministic ordering, deduplication, context budgets, canonical source references, and trace events
• scheduler.py: own the active ReminderScheduler exclusively (spec sec 10.2 folder layout: ‘active ReminderScheduler only; emits refresh control events’; sec 11.1: ‘active ReminderLease refresh is separately owned by engine-core ReminderScheduler and has no workflow-transition authority’) — schedule durable ReminderLease refresh independently of the SM driver, deduplicate/acknowledge/fence lease events, and recover leases WITHOUT granting workflow-transition authority. engine.py remains the sole ExecutionPump/transition-authority owner. This module has its own Implementation-sketch bullet (distinct from engine.py’s) but currently has no corresponding work item; w-054abfb7’s rewritten text removes the misattributed ReminderScheduler ownership this item now carries.
• Build initial six-part ContextSnapshot selection with static-minimal catalog metadata and bounded first-handoff content; exclude on-demand deeper disclosure
• Define six-part ContextAssembly with one GuidanceEnvelope authority, sandbox/model/evaluation contracts, closed three-branch ContinueRuntimeView/ClientTransportDirective schemas, imported root-closure TerminalReceipt, exact script prompt provenance, and a transport-only client contract whose goal/loop path relays active current/next or completed report unchanged and treats retry_later as no task/second call/lifecycle mutation
• Implement continue_runtime.py / ContinueRuntimeAPI.read as the sole re-entry read surface: accept only identity-bound exact args; reconcile authoritative plan/root state; return a full-identity active current_task/next_upcoming_task view, completed root-owned TerminalReceipt, or fail-closed retry_later after bounded internal retry exhaustion; make no second transport call/task/lifecycle mutation on failure; reject caller-authored task text, transport-supplied state, or preview execution authority
• Define TaskDispatch/task-token, ResultContract, ResultSubmission, and HumanDecisionReceipt boundary contracts over the root-section EntryEnvelope and identity model
• Implement one authorization/result-ingestion gateway that rejects unauthorized targets, ambient context widening, model/client-authored or paraphrased control prompts, changed transport tool arguments, stale capabilities, duplicate work, and stale results/returns
• Implement authorized on-demand knowledge/example/memory disclosure plus script-owned model routing, task-token sandbox/tool capabilities, and output/trajectory evaluation evidence
• Classify deterministic, rubric-evaluator, and human-required decisions; require trusted-broker principal authentication, signed one-use HumanDecisionReceipt, and replay protection for governance transitions
• Implement closed EngineTurn/FailureDisposition, total guidance, engine-owned ExecutionPump and active ReminderScheduler, stable failure fingerprints, drive-until-quiescent, and fail-closed authorization
• Implement closed EngineTurn/FailureDisposition/NextActionContract schemas and total guidance; implement engine.py’s ExecutionPump drive-until-quiescent loop as the SOLE owner of transition authority (adapters remain transport-only); wire failure_evidence.py’s versioned FailureEvidenceFingerprint over raw evidence for stable failure fingerprints; implement fail-closed authorization on every runnable state. ReminderScheduler ownership belongs to scheduler.py, NOT engine.py — see the sibling scheduler.py work item.

Intel dossier — pointer and tier disposition

Dossier: engine-core--s-e585be74.intel.md (this section’s content/intel/ sidecar)

Read that dossier from line 1 through EOF before acting on this section. This block is a pointer and an audit record, never a substitute: not the BLUF, not a summary, not selected tiers.