100%

metrics-ledger

Goal

Downscoped from the superseded plan: append-only per-run metrics history with moving deltas, consuming the runner/harness JSON. Tracks every critical metric (pass/fail/skip counts, leaks, durations, build number, sha) and its raw current-minus-previous delta across runs (rotated).

Implementation Sketch

  • A python helper (scripts/) ingests build/test-all-summary.json and appends a normalized per-run record to build/test-all-ledger.json with raw current-minus-previous deltas vs the prior run, rotated to a bounded history.
  • Surface the latest delta via state.sh + status-report.
  • pytest covers the delta computation + rotation.

Spec References

compiler_repo/build/test-all-summary.json (from section 02); compiler_repo/diagnostics/state.sh; the superseded plans/test-all-metrics-ledger sections (ledger-runtime, state-and-status-integration).

Success-Criterion Probe Requirement

  • The ledger success criterion (the ledger runtime pytest suite passes and computes delta as raw current-minus-previous against the prior run) currently carries a non-falsifiable echo ok probe_ref in plan.json (rendered in status.md).
  • An echo ok probe can never fail and never runs the pytest it claims to verify, so it satisfies the L12-style production-verification gate without exercising the production entry point — a LAYER_GAP:production-entry-point per .claude/rules/layer-coverage.md §1 L12.
  • Before this section closes, the probe MUST be rewritten (via the script-only plan.json success_criteria surface) to invoke the runtime pytest and assert the delta computation: a probe that runs python -m pytest <ledger-runtime>/tests AND asserts delta == current - previous on a known two-run fixture — driving the real ledger entry point on a real test-all-summary.json input and asserting the appended ledger record’s delta field.
  • The criterion is correctly pending/no-evidence while the section is unimplemented; the requirement is that the probe become a real falsifiable check, not that the criterion flip green prematurely.

Work Items

  • Build an append-only ledger runtime that ingests the harness JSON, appends a normalized per-run record with raw current-previous deltas, and rotates to a bounded history; pytest covers delta + rotation.
  • Surface the latest ledger delta through state.sh + status-report.
  • Re-ground the Rust-suite parse path away from the brittle text scrape onto a structured machine-format source, reusing the shared compiler_repo/diagnostics/parse_test_json.py helper (reuse, not a second parser) — closes the control-char parse-error vulnerability for the Rust suite, the follow-up deferred from s-bec662fc. Scope notes that must be settled before implementation (all surfaced by the s-760e05ed nextest section landing AFTER this item was first written):
    • Function locus: the parser under change is parse_rust_results, defined at compiler_repo/test-all.sh:334 with its body spanning lines 334-368 (verified at HEAD). The original “test-all.sh:331-342” citation was stale (lines 331-342 cover only the leading comment block + first branch); use the 334-368 def-and-body range.
    • Active runner is nextest, not cargo test: with s-760e05ed complete, parse_rust_results now primarily scrapes the cargo-nextest Summary [ aggregate line (compiler_repo/test-all.sh:339-350). The original target “cargo test —message-format json” emits libtest JSON — a DIFFERENT schema than cargo nextest run --message-format libtest-json (nextest’s experimental libtest-json passthrough) — so the migration target named for cargo-test is for the runner nextest displaced. Implementation MUST target the structured output of the nextest path actually in use (nextest’s machine-readable summary / libtest-json passthrough), not the cargo-test libtest JSON named pre-nextest.
    • Helper-schema fit: confirm parse_test_json.py:_load (and its emit_json → test-all-summary.json schema) accepts the nextest structured shape, or extend the shared helper rather than forking a second parser; a schema mismatch between the nextest JSON and the helper’s expected shape is itself in-scope for this item.

This item carries its own deliverable (a structured-format Rust-suite parser), its own success criterion (control-char parse-error vulnerability closed for the Rust suite, parsed via the shared helper), and is independently workable from the ledger runtime (w-144b0280) and state.sh surfacing (w-78826a91). It meets the routing.md §4 promotion independence test and is a candidate for promotion to a letter-suffix sibling section; until promoted via scripts/plan_corpus/promote_subsection.py, it lives here under the single ## Work Items heading (the prior split ## Items heading is consolidated — all three are plan.json work_items owned by this section, and status.md renders the section as 0/3 against one roster).