100%

test-all-json-consumption

Goal

Make test-all.sh consume the runner’s JSON instead of scraping "N passed, M failed" text, and retire the hand-rolled bash JSON escaper - which fixes the live test_suite.status=parse-error bug (control chars in failure messages produced invalid JSON).

Implementation Sketch

Replace parse_ori_results’ text-grep + the bash JSON emission with: invoke ori test —format json, parse with a python3 helper, emit the harness summary JSON via json.dumps (never bash string-building). Adversarial-message fixture (tabs/newlines/quotes/unicode) passes jq empty.

Migration Boundary — what JSON replaces vs what stays text

parse_ori_results (test-all.sh:344-371) currently text-greps the runner’s human summary line (grep -E "[0-9]+ passed, [0-9]+ failed" at line 359) to populate ${prefix}_{PASSED,FAILED,SKIPPED,LCFAIL,CRASHED}. Those parsed counts feed TWO consumers: the per-suite count display table AND emit_json’s test-all-summary.json payload. This section migrates BOTH consumers off the text scrape onto the runner’s --format json payload — the text grep is the SSOT that produced the parse-error bug, so leaving either consumer on it would leave the bug live.

In scope (migrates to JSON consumption):

  • parse_ori_results’ count extraction — replaced by the diagnostics/parse_test_json.py helper’s json.load of the runner’s --format json object (reads the pinned aggregate counts + per-test/per-file payload from s-4aa2d17b’s schema). The count table display reads the same parsed object; no second text path.
  • The hand-rolled bash JSON emission in emit_json’s failure-message handling — replaced by the same helper’s json.dumps, which escapes control chars by construction.
  • parse_ori_failures’ bash escaper of failure-message text — superseded by the runner’s serde-escaped JSON failure messages (s-4aa2d17b emits them already-escaped).

Out of scope (stays text — different SSOT, not the runner’s JSON):

  • run_ori_interpreter / run_ori_llvm (test-all.sh:281-327) invoke the runner WITHOUT --format json and capture human output to $ORI_INTERP_OUTPUT / $ORI_LLVM_OUTPUT; this section adds the --format json invocation feeding parse_ori_results, it does not delete the human-output capture (the human summary line is still surfaced to the console).
  • parse_rust_results (test-all.sh:331-342) scrapes cargo test’s ^test result: lines — that is rustc/libtest output, NOT the Ori runner’s JSON; it is unaffected by this section and keeps its text parse.
  • crash detection on signal-terminated exit (parse_ori_results:350-357) keys on the exit code, not on output text; it is preserved unchanged.

Spec References

compiler_repo/test-all.sh (parse_ori_results, parse_rust_failures, parse_ori_failures, suite_status); the walking-skeleton THROWAWAY SCAFFOLD it retires — walking_skeleton_json_probe (test-all.sh:131-152), WALKING_SKELETON_JSON_SUITE/WALKING_SKELETON_JSON_RECORD/WALKING_SKELETON_JSON_EXIT (test-all.sh:111,129-130), and the probe’s call site + result-record handling — whose own comment (test-all.sh:121-128) names this durable all-suites JSON-consumption path as the scaffold’s retirement. The superseded plans/test-all-metrics-ledger section-01 (json-emission-foundation) absorbs here.

Dependencies

This section CONSUMES the runner JSON schema emitted-and-pinned by the sibling section runner-json-output (s-4aa2d17b). s-4aa2d17b is the predecessor: it derives Serialize on TestSummary/FileSummary/TestResult, resolves interned Name fields to strings, pins duration_ns as integer nanoseconds, and pins per-test outcome variants (Passed/Failed/Skipped/SkippedUnchanged/LlvmCompileFail) + error-file count. The python3 json.load helper here reads exactly that schema; the field names + types this section parses are the contract s-4aa2d17b owns. s-4aa2d17b’s own Scope Boundary names this section (s-bec662fc) as the consumption successor. Transitively, s-4aa2d17b depends on the walking-skeleton (s-7a95df82) that landed the OutputFormat::Json branch and the --format flag this section invokes. Both prerequisites (s-4aa2d17b, s-7a95df82) are already complete, so this dependency is not blocking — but the predecessor->successor DAG edge MUST be recorded so a schema change in s-4aa2d17b surfaces against this consumer.

The prerequisite edge exists: node s-bec662fc records hint_needs: , and s-7a95df82 is the transitive predecessor through s-4aa2d17b’s own hint_needs, reachable without a duplicate direct edge here.

Scope — retire the walking-skeleton scaffold

This section IS the durable all-suites JSON-consumption path that test-all.sh:121-128 declares retires walking_skeleton_json_probe. The scaffold proved the JSON spine end-to-end on ONE suite (tests/spec/test_coalesce_copy.ori): runner-emit behind --format=json, harness-consume via python3 json.load of top-level passed, parity-assert against the text leg, and record to a disc-backed artifact. Once parse_ori_results consumes the runner’s --format json for ALL Ori suites, the single-suite probe is redundant proof and its retained text-leg parity comparison reintroduces the very text scrape this section eliminates.

Retiring the scaffold means deleting walking_skeleton_json_probe + its call site + WALKING_SKELETON_JSON_SUITE / WALKING_SKELETON_JSON_RECORD / WALKING_SKELETON_JSON_EXIT + the probe’s result-record handling, and removing the build/walking-skeleton-json-probe.json artifact path. The durable path’s adversarial-fixture jq-empty gate (the existing success criterion) subsumes the scaffold’s single-suite parity proof: the durable consumption covers every suite, and the adversarial fixture proves correct escaping the single-suite integer-only probe never exercised.

python3 JSON helper — canonical location + sibling-reuse contract

The python3 helper that owns json.load of the runner --format json payload and json.dumps of the harness summary is a SINGLE module: compiler_repo/diagnostics/parse_test_json.py, with its own pytest at compiler_repo/diagnostics/tests/test_parse_test_json.py. test-all.sh invokes this one helper; no inline python3 -c JSON parsing is authored in test-all.sh (a second inline parser would be a second text-adjacent path, re-opening the SSOT split this section closes).

Sibling-reuse contract (resolves cc-1 + dd-1):

  • The metrics-ledger section (s-5494ed74) REUSES diagnostics/parse_test_json.py — it does NOT declare its own JSON parse path (a second ledger parser would violate this section’s “no second text path” SSOT principle).
  • s-5494ed74 consumes this helper for its delta computation; its hint_needs therefore include s-bec662fc (recorded in plan.json).
  • The round-trip-equality check for the adversarial fixture (w-38896e3b) lives as a thin probe helper compiler_repo/diagnostics/parse_test_json_roundtrip_check.py that imports parse_test_json and asserts the parsed failure-message entry equals the injected adversarial bytes.

JSON parse-error fallback contract

When the runner emits non-JSON (crash, stderr leakage into stdout, SIGPIPE partial output), the python3 helper MUST NOT silently default-zero the way the retired text-grep did (default-zero is exactly what masked the parse-error bug). The contract:

  • parse_test_json.py raises a non-zero exit when its stdin/argument is not valid JSON.
  • test-all.sh detects the non-zero helper exit and marks that suite test_suite.status=parse-error in test-all-summary.json (the SAME status field this section’s deliverable fixes for the escaping case), with the raw runner output captured to the suite’s failure-detail field.
  • The ${prefix}_{PASSED,FAILED,SKIPPED,LCFAIL,CRASHED} count vars are set to a sentinel that test-all.sh’s arithmetic treats as a hard suite failure (never bare-empty — bare-empty causes the $(( )) syntax errors agy-F2 flagged). The suite_status table renders parse-error, and the overall run exits non-zero.
  • The python3 helper’s non-zero exit MUST propagate to the suite’s exit-code variable (ORI_INTERP_EXIT) — not only to the sentinel count vars — so ANY_CORE_FAILED (test-all.sh:977) catches the parser failure and fails the overall run; a parse-error that updated only the count sentinel without setting ORI_INTERP_EXIT would still leave ANY_CORE_FAILED green.

A runner crash on signal-terminated exit stays on the exit-code crash-detection path (parse_ori_results:350-357), unchanged; the parse-error fallback governs the case where the runner exits 0 but emits malformed JSON.

Double spec-suite execution — single-run, file-backed

run_ori_interpreter / run_ori_llvm (test-all.sh:281-327) currently invoke the runner WITHOUT --format json and capture human output. This section does NOT add a SECOND runner invocation:

  • The runner is invoked ONCE per backend with --format json writing to a file (build/ori-interp-results.json / build/ori-llvm-results.json).
  • The human progress line is reconstructed FROM the parsed JSON for the console table — printed from the parsed object, preserving the console UX without re-execution.
  • parse_ori_results reads that file via parse_test_json.py. No suite runs twice.

Rust failure parsing — scoped out, with anchor

parse_rust_results (test-all.sh:331-342) scrapes cargo test’s ^test result: lines — rustc/libtest output, NOT the Ori runner’s JSON.

  • The control-character parse-error vulnerability this section fixes is specific to the Ori runner’s failure-message escaping path; cargo/libtest output is outside this section’s migration boundary.
  • Migrating Rust failure parsing to JSON requires cargo test --message-format json (libtest JSON), a distinct schema with its own scope — NOT folded here.
  • Deferral anchor (concrete registered work item): w-1b3d6409 in the metrics-ledger section (s-5494ed74) — “migrate parse_rust_results to cargo test —message-format json via the shared diagnostics/parse_test_json.py helper”. The ledger is the home because it already consumes the shared helper (s-5494ed74 hint_needs: ).
  • Until that item lands, parse_rust_results keeps its text parse, unchanged by this section.

Work Items

  • Replace test-all.sh’s text-scrape (parse_ori_results at test-all.sh:359) + hand-rolled bash JSON escaper with consumption of the runner’s —format json output via a python3 json.load/json.dumps helper; both the per-suite count display table AND emit_json’s test-all-summary.json read the single parsed object (no second text path), reading s-4aa2d17b’s pinned schema.
  • Add an adversarial-failure-message fixture (tab/newline/quote/unicode) and assert jq empty + json.load pass on build/test-all-summary.json (round-trip equality, not parse-success alone).
  • Retire the walking-skeleton THROWAWAY SCAFFOLD now that the durable all-suites path lands: delete walking_skeleton_json_probe (test-all.sh:131-152), its call site, WALKING_SKELETON_JSON_SUITE/WALKING_SKELETON_JSON_RECORD/WALKING_SKELETON_JSON_EXIT (test-all.sh:111,129-130), the probe result-record handling, and the build/walking-skeleton-json-probe.json artifact path, per the scaffold retirement contract at test-all.sh:121-128.