S4 - Fail-closed streaming audit

Goal

S4 - Fail-closed streaming audit.

Implementation Sketch

Mission anchor: mission.md north-star gate. The tp-discuss design file mission.md cites (scripts/commit_push/design/2026-07-24-durability-redesign-tp-discuss.md) does NOT exist at HEAD — do not wait on it or treat it as readable spec; the work items plus the contracts below are the executable design.

Verified live starting state — re-verify against source before editing, never from memory:

• DEFECT — cli._run_queue_audit returns EXIT_OK on every path (cli.py:765 JSON, cli.py:768 empty-queue, cli.py:781 terminal) regardless of verdict mix. Fail-closed is NOT implemented at the CLI boundary today; the only nonzero exit available is an uncaught Python exception.
• DEFECT — queue_audit.audit_refs fully BUFFERS: builds out: list[RefAudit] (queue_audit.py:143), appends per ref (:161), returns only after the loop closes (:170). Nothing prints until every ref is classified, so an interrupted audit yields NOTHING — not a partial fail-closed report. Falsifier (d) cannot pass against this shape.
• COST — per ref: 2 + 3F subprocess spawns (F = files the ref touches): _rev_parse + entry_touched_paths + _subject, then _classify_file (queue_audit.py:105) x F, each running 3 _blob_oid (:97). At the live depth (264 parked refs in the wrapper, 95 in compiler_repo at recon time) that is 1000+ spawns before the first line of output. Streaming fixes visibility and interrupt-safety; batching _blob_oid (git cat-file --batch-check-style) is a SEPARATE wall-clock lever — out of scope unless streaming alone leaves the run unusable.
• ALREADY CORRECT — do NOT re-implement, widen, or weaken: AUTO_DROPPABLE_VERDICTS == frozenset({landed, empty}) (queue_audit.py:55); superseded-but-not-missing already classifies VERDICT_DIVERGED (:152-158); drop_ref’s gate refuses any non-auto-droppable verdict without confirm=True (:260); test_superseded_change_classified is already aligned to that sound behavior. S4 adds streaming, an unclassifiable branch, and the CLI exit code — it does NOT author a new classifier.
• DESIGN NOTE — drop_ref re-audits the FULL ref set to drop ONE ref (queue_audit.py:256). After the streaming rewrite that pays the whole streaming cost for a single ref: either narrow the fresh re-audit to the target ref or record in-code why the full re-audit is the fail-closed choice.

Verdict model

Rename the four verdict constants (queue_audit.py:45-48) to the plan vocabulary. Leave the per-file vocabulary (FILE_LANDED_EXACT / FILE_SUPERSEDED / FILE_MISSING, queue_audit.py:57-59) and _classify_file’s three-way blob logic unchanged.

• Rename the members inside AUTO_DROPPABLE_VERDICTS without widening it — membership stays exactly two.
• Re-point the existing negative pin (test_superseded_change_classified) at the renamed constant; the rename must not silently un-pin the diverged-never-auto-drops guarantee.
• Add unclassifiable as a NAMED branch — a git-command failure inside _classify_file / _subject / _rev_parse appends an explicit unclassifiable record. A bare traceback is not the unclassifiable path.
• Per diagnostic-ux.md, an unclassifiable line names the cause (which ref; unresolvable side-ref vs failed git command) AND the next step (queue-extract first, then re-run queue-audit). A bare unclassifiable token is not acceptable output.

Exit-code contract

_run_queue_audit (cli.py:757) derives its exit from the verdict mix, never a constant. Codes come from the existing enum (cli.py:63-71) or extend it.

• --json carries the SAME exit code as the human path; the JSON body stays RefAudit.to_dict()-compatible or is deliberately versioned.
• A ref never reached is never reported empty or clean — unreached is fail-closed, not absent.

Ownership boundaries

• S4 owns scripts/commit_push/queue_audit.py, the queue-audit surface in scripts/commit_push/cli.py, and scripts/commit_push/tests/test_queue_audit.py.
• queue_drain._exact_in_head / _retire_exact (queue_drain.py:132-139) is a SECOND, independent auto-retirement oracle: one all-paths git diff --quiet <ref> HEAD -- <paths> boolean, NO confirm gate, ZERO code sharing with queue_audit (queue_drain.py imports only .queue_git), sitting on the path /commit-push runs on EVERY invocation via drain_on_entry -> drain_queue.
• S4 does NOT modify queue_drain.py. Renaming queue_audit’s verdicts changes NOTHING about that surface’s behavior — assuming otherwise is this section’s primary false-safety risk.
• S4’s only action on it is w-93aa788f: an in-code cross-reference comment in queue_audit.py, so the two-oracle situation is visible where S4 works.
• Reconciliation (a shared exactness predicate both drop_ref and queue_drain._drain_one_locked consume, or a documented reason to diverge) belongs to S5 safe-drain (s-6c240bce) work item w-e206f51a — plan boundaries are implementation boundaries.
• Preserve-constraints, do not disturb as a side effect: drain_on_entry and the parked-queue machinery S5 depends on; AUTO_DROPPABLE_VERDICTS membership; _classify_file’s classification logic.

Verification and layer coverage

scripts/commit_push/** is TOOLING scope per CLAUDE.md §HYGIENE / CODING RULES SCOPE — compiler layers L1-L11 are N/A: wrapper Python tooling, with no compiler pipeline, executor, or .ori corpus surface. The binding requirements:

• L12 production entry point — MANDATORY. No test today drives cli._run_queue_audit or the queue-audit subprocess; every existing pin calls qa.audit_refs() / qa.drop_ref() directly, so the always-EXIT_OK defect is untestable by the current suite even in principle. A pin that drives _run_queue_audit (or the subprocess) and asserts the exit code is required; a green suite resting only on audit_refs() is LAYER_GAP:production-entry-point.
• Falsifier (c) pin — a diverged ref (HEAD evolved past the parked change, one file superseded) classifies diverged-needs-review AND drop_ref(confirm=False) refuses it. Port test_superseded_change_classified’s fixture; re-point the assertion at the renamed constant.
• Falsifier (d) pin — a runner stub raising after N refs asserts: (a) the already-classified refs’ lines are present in stdout, (b) the exit code is nonzero, (c) no unreached ref is reported empty or clean.
• loop-safety.md — NOT triggered: the iteration source stays the bounded list_side_refs() result; a generator/streaming rewrite introduces no unbounded or non-advancing loop.
• SRP/SSOT still bind (LEAK:algorithmic-duplication across the two exactness oracles) — discharged in S4 by w-93aa788f’s cross-reference, closed by S5’s w-e206f51a.

Work Items

• Rewrite queue-audit to stream one record per ref with progress
• Split verdicts: exact-in-head | diverged-needs-review | missing | empty
• Only exact-in-head|empty auto-retire; drop_ref(confirm=False) permits only those two; superseded/diverged NEVER auto-drop
• Document, in a code comment in queue_audit.py near AUTO_DROPPABLE_VERDICTS/drop_ref, the relationship between this section’s exact-in-head/diverged-needs-review verdict model and queue_drain._exact_in_head’s separate, ungated auto-retirement predicate (queue_drain.py:132-139, owned by S5 safe-drain, s-6c240bce) — so the two-oracle situation is visible in the code S4 owns and S5 does not have to rediscover it independently when it starts. Do NOT modify queue_drain.py in this section; that reconciliation is S5’s (queue_drain.py owner) decision per plan-boundaries=impl-boundaries.
• Exit nonzero on any unclassifiable ref (fail-closed)
• Wire _run_queue_audit’s (cli.py:757) exit code to the audit’s actual verdict mix, not a constant EXIT_OK: return EXIT_OK only when every audited ref’s verdict is in AUTO_DROPPABLE_VERDICTS (exact-in-head or empty); return a distinct nonzero code when any ref classifies diverged-needs-review or missing (review needed, not an error, but never a silent-success exit); return a further distinct nonzero code when any ref is unclassifiable — add a named unclassifiable branch (a git-command failure inside _classify_file/_subject/_rev_parse) that appends an explicit unclassifiable record instead of the only path today, an uncaught exception. Pin the fix with a test that drives _run_queue_audit itself (or the queue-audit subprocess), not just audit_refs() — today’s suite (12 tests) exercises only the library function and would pass unchanged even with the exit-code bug present, per layer-coverage.md L12 production-entry-point.
• Falsifier (c): diverged-file-omitting-parked -> diverged-needs-review never auto-dropped
• Falsifier (d): interrupt-audit -> streamed fail-closed nonzero, never empty-looks-clean

Recon probes and known coverage gaps

• Run BEFORE touching code: python -m scripts.commit_push queue-audit --repo . — feel the buffering and the always-0 exit directly. Then python -m scripts.commit_push queue-audit --repo . --json — the RefAudit.to_dict() shape the rewrite must preserve or deliberately version.
• Queue depth is the plan’s north-star metric: git for-each-ref refs/commit-push-queue/ | wc -l (264 wrapper, 95 compiler_repo at recon time).
• compiler_repo/diagnostics/ and the ORI_* env-var catalogue are NOT relevant here — this section has no compiler-runtime surface. The queue probes above ARE the diagnostic surface.
• Intel-graph facets are STALE for this surface: insights, sibling_divergence (MISSING_ABSTRACTION), co_change, test_topology. test-posture reports zero covering tests for audit_refs — a proven false negative; use raw CALLS-inbound-from-test edges as ground truth for this file.
• The two-oracle finding is a graph BLIND SPOT, not a recoverable stale-facet miss: queue_drain._exact_in_head has no embedding, no CALLS edge to or from queue_audit, and the cross-file MISSING_ABSTRACTION query returns zero rows. Direct source read is the only route to it — do not expect an insights refresh to resurface it, and do not treat graph silence as absence.
• No bug-tracker entry is filed against this plan (proactive hardening). The identical silent-loss-reported-as-success shape is currently OPEN in tooling-cli — BUG-07-532, BUG-07-543, BUG-07-522, BUG-07-442, BUG-07-239 — corroboration that the fail-closed requirement is not hypothetical, not work items for this section.
• Prior art and sentiment are verified-absent, not unexamined: the reference-language corpora are compiler trees with no analogue for a parked-git-ref audit (top cross-repo similar hit is semantic noise), and ori/ori-platform carry zero indexed issues. Do not re-spend recon there.
• No other plan claims this code (symbol-plans returns zero hits for audit_refs / _classify_file / _exact_in_head). The only ownership coordination is the intra-plan S4 -> S5 handoff above.

Intel dossier — pointer and tier disposition

Dossier: fail-closed-audit--s-f5aef78c.intel.md (this section’s content/intel/ sidecar)

Read that dossier from line 1 through EOF before acting on this section. This block is a pointer and an audit record, never a substitute: not the BLUF, not a summary, not selected tiers.