85%

Scenario harvest — the behavioral spec + parity oracle

Goal

  • Inventory every workflow/control scenario and authority surface encoded by production entrypoints, commands, adapters, hooks, continuation composers, loops, tests, rules, and skills.
  • Classify path-based misses/over-hits and wrong-target detours.
  • Inventory every place a client, prompt, transcript, or LLM currently infers workflow state, authors control text, chooses continuation, or declares completion; the target disposition is script-owned state/prompt/transition authority.
  • Use preserved scenarios for parity and folded-root cases as target semantics.

Self-hosting constraint

Run the harvest before primitive design. Re-sweep current tests/contracts before shadow parity.

Implementation sketch

  • Enumerate the existing test surface: scripts/plan_orchestrator/tests/**, scripts/plan_corpus/tests/**, scripts/commit_push/tests/**, the rails/runtime tests.
  • Enumerate the rule/skill behavioral contracts that encode un-unit-tested scenarios (routing.md, learning-ledger.md, skill-control-contract.md, state/plan-read/context discipline).
  • Enumerate every production command/entrypoint, adapter, hook, dispatcher, continuation composer, selector, retry loop, resume owner, and terminal finalizer; missing tests do not remove an authority surface from inventory.
  • Classify each control surface as Prompt, RuleUnit, Reminder, directive, human banner, or hard gate.
  • Record each selector’s required facts: workflow/state/action/role/audience/plan kind/artifact/event/path/history/capability.
  • Harvest wrong-plan cases: ambiguous correction, dirty-worktree inference, override misuse, abandoned run, duplicate task, and mismatched tool target.
  • Harvest arbitrary-yield cases: progress summary after serve/result, assistant turn end while runnable, disconnect, status/token/cost rendering, and every typed yield state.
  • Harvest reminder-timer cases: expiry during model/tool activity, injection-capable/incapable adapter, missing acknowledgement, crash before/after delivery, overdue restart, duplicate/stale timer, and state change before refresh.
  • Harvest client re-anchor cases: goal/loop absent, owned-current, owned-stale, foreign, or unavailable; exactly-one ContinueRuntimeAPI.read per turn/tick; byte-identical current_task/next_upcoming_task relay; direct plan/engine-read and early-preview-execution rejection; drifted/restored sessions; exact script-generated prompt/tool-call provenance; plan-versus-root-versus-client state ownership; matching/stale/foreign receipts; unsealed receipt-shaped copies; and idempotent replay of the real sealed TerminalReceipt.
  • Add state_authority, prompt_producer, transition_owner, and terminal_evidence to each scenario row. state_authority is closed to domain_artifact or the explicit no-artifact virtual_root; reject any row that gives the LLM, adapter, goal, loop, transcript, or scratch sentinel domain-state or completion authority.
  • Harvest fix-interference cases: active regression, tracked/untracked blocker, duplicate/cyclic blocker, shelve crash, blocker return, stale origin proof, and exact-origin resume.
  • Emit machine-readable content/scenario-inventory.json: one row per scenario with source, disposition, control class, selector facts, expected assembly, hard-control verdict, and asserted behavior.
  • Record every DROP as a deep-scrutiny decisions/ entry; a silent drop is BANNED.
  • Implement verify_scenario_inventory to reject missing fields, duplicate IDs, stale sources, uncovered production or contract surfaces, and drops without a registered decision reference.

Spec references

  • plans/workflow-engine-rework/spec/workflow-state-machines.md sec 0.1 (behavioral-preservation contract — harvest first, preserve by default).

Test strategy / acceptance

Acceptance: the verifier proves every harvested surface has one disposition and all required fields; include path, wrong-plan, arbitrary-yield, fix-interference, and client-transport matrices. Negative cases prove model/client-authored control prompts and non-engine completion claims cannot advance or terminate a workflow.

Work Items

  • Enumerate production entrypoints, commands, adapters, hooks, dispatchers, continuation composers, selectors, loops, resume owners, and finalizers independent of test coverage
  • Enumerate the existing TEST scenarios across plan_orchestrator/plan_corpus/commit_push/runtimes
  • Enumerate the RULE/SKILL behavioral-contract scenarios (un-unit-tested behaviors)
  • Classify every harvested control message using the section’s canonical taxonomy — Prompt, RuleUnit, Reminder, directive, human banner, or hard gate — and map its canonical source, audience, delivery seam, and trigger facts
  • Harvest path-based Claude rule injection misses and over-hits; convert each into a contextual selector scenario over workflow state, action, role, event, artifact, plan kind, target path, and history facts
  • Harvest plan-identity drift scenarios: ambiguous user correction, unrelated dirty-worktree state, explicit override misuse, abandoned-run resurrection, duplicate in-flight task creation, and wrong-plan tool invocation
  • Harvest queue-batch scenarios for empty, one, and many members; dynamic admissions, filters/orders/skips/blockers, interactive one-item mode, virtual-root versus nested-batch entry, and closeout-once semantics
  • Harvest proactive-guidance failures, timer lifecycle, arbitrary yields, client substitution, classification spin, blocker shelve/restore/resume, current-vs-target claim drift, and goal/loop continue-runtime-read/unchanged-current-next observer cases; record exact state authority, script prompt provenance, and matching TerminalReceipt evidence for each transport scenario
  • Record each DROP as a deep-scrutiny decisions/ justification (no silent drops)
  • Emit machine-readable content/scenario-inventory.json with stable IDs, source, disposition, control class, selector facts, expected assembly, hard-control verdict, asserted behavior, and per-row state_authority (closed to domain_artifact or the explicit no-artifact virtual_root; reject any row granting the LLM, adapter, goal, loop, transcript, or scratch sentinel domain-state or completion authority), prompt_producer, transition_owner, and terminal_evidence
  • Implement verify_scenario_inventory with source, coverage, uniqueness, required-field, and drop-decision validation

Fresh intel (regenerated)

This section is a pure read-only INVENTORY of a live, load-bearing substrate it must NOT re-derive, break, or fork — the census that all later engine-building sections (engine-core, rails-components, walking-skeleton, shadow-parity) consume as their parity oracle. Its deliverables (content/scenario-inventory.json, verify_scenario_inventory) are GREENFIELD (zero graph symbols). Its terrain is the ENTIRE shipped orchestration surface, and the graph quantifies exactly the fragmentation the harvest exists to make legible:

  • 62 files emit a next_action-shaped envelope, 20 separate next_action.schema.json files (one per runtime), 19 compose_* prompt-producers in plan_orchestrator alone, and >=4 independent drive-until-quiescent state machines (commit_push/state_machine.py, plan_orchestrator/state_machine.py, impl_hygiene_review_runtime/rails_compose.py+rails_advance.py, tpr_review_runtime/next_action_emit.py). This is the prompt_producer + transition_owner scatter the harvest catalogs.
  • 1,317 script test files (418 plan_orchestrator/tests, 375 plan_corpus/tests, 55 commit_push/tests, 415 across *_runtime) are the w-bc955685 test-scenario input.
  • The behavioral contracts the harvest names live as graph nodes: all 8 named rules (routing, learning-ledger, skill-control-contract, state-discipline, plan-read-discipline, context-discipline, workflow-ownership, script-first) as Rule nodes, 304 RuleSection, 41 Skill, 17 Hook, 8 Command — the w-c3ac8295 un-unit-tested-contract inventory.

The single most important [JOIN]: the graph CANNOT do the cross-runtime dedup the harvest’s “classify path-based misses / wrong-target detours” objective needs. parallel-drift (sibling-divergence + insights STALE) returns exactly ONE cross-file missing-abstraction row across the six canonical state-machine/emitter files, and it is intra-package (StateMachine ~ QueueState, 0.582); similar clusters strictly WITHIN each runtime package (the commit_push.StateMachine twin of plan_orchestrator.StateMachine is NOT an edge). The 20-schema / 62-emitter fragmentation is invisible to the derived edges — it is only visible by DIRECT enumeration of disk, which is precisely what the section’s implementation sketch prescribes (“Enumerate the existing test surface… Enumerate every production command/entrypoint, adapter, hook, dispatcher…”). VERDICT: the harvest is an enumerate-from-disk deliverable that the graph SEEDS and CROSS-CHECKS, never one the graph can auto-generate.

Decisions this package forces before any code is written:

  1. Seed the scenario inventory from the four hard node-classes the graph DOES hold (Rule/RuleSection, Skill, Hook, Command) + the enumerated scripts/** emitter/state-machine/finalizer surface — but treat every parallel-drift/test-posture/co-change result as STALE (a nominator, never a completeness proof).
  2. Do not fork the single continuation-ownership authority. autopilot_guards/run_intent.py:check_stop_against_run_intent is the ONE shipped Stop-hook continuation owner; the harvest inventories its “arbitrary-yield / assistant-turn-end-while-runnable” scenarios as parity semantics and must record them faithfully, NOT re-implement them (LEAK:algorithmic-duplication / STRUCTURE:single-consumer-runtime risk carried forward to engine-core).
  3. verify_scenario_inventory is a validator terrain — it MUST fail-closed (reject missing fields, duplicate IDs, stale sources, uncovered production/contract surfaces, drops-without-a-decision-ref). A permissive/early-returning verifier is the INVERTED-TDD failure mode for this exact deliverable.
  4. Every DROP needs a decisions/ entry — a silent drop is BANNED by the section body; the verifier enforces the drop-decision cross-reference.
["check_stop_against_run_intent", "detect_autopilot_context", "session_run_intent_path", "read_autopilot_intent", "record_user_interrupt", "halt", "dispatch", "exit_action", "serve_next_v7"]

DIG DEEPER

python3 -m scripts.plan_corpus.read --summary plans/workflow-engine-rework
scripts/intel-query.sh status --json | python3 -c "import sys,json;print([r['facet_generations'] for r in json.load(sys.stdin)['data']['repos'] if r['name']=='ori-platform'])"

(full dossier: scenario-harvest—s-27dad67f.intel.md)