100%

runner-json-output

Goal

Give the Ori test runner full structured/machine-readable output: --format json serializing the complete TestSummary + per-file FileSummary + per-test outcome (Passed/Failed/Skipped/SkippedUnchanged/LlvmCompileFail) + durations. This is the genuine runner gap - parallelism (rayon) and incremental change-detection already exist.

Implementation Sketch

This section EXTENDS the walking-skeleton’s already-landed OutputFormat::Json branch — it does NOT re-add a --format flag. The OutputFormat enum + the run_tests format match already exist from the walking-skeleton; the skeleton’s TestSummary::render_json currently hand-renders three integers ({passed, failed, skipped}), which is safe ONLY because integers carry no escaping hazard. This section replaces that minimal render with the full payload.

Route the existing OutputFormat::Json branch through a serde_json serializer over TestSummary/FileSummary/TestResult in test/result/mod.rs. Keep text the default; JSON is opt-in. Text output stays byte-identical when the format is text.

Serializer commitment — serde_json, not hand-rendering (ABSOLUTE):

  • Derive Serialize on TestSummary, FileSummary, and the per-test result/outcome types. The “hand-write the JSON” option is BANNED for this payload.
  • Interned-name resolution (ABSOLUTE): TestResult.name is a Name and TestResult.targets is a Vec<Name>, where Name is a #[repr(transparent)] interned u32 index (compiler_repo/compiler/ori_ir/src/name/mod.rs). A naive #[derive(Serialize)] on TestResult would emit those integer indices, NOT the resolved test-name strings — the JSON consumer would get opaque numbers. The serializer MUST resolve every Name through the StringInterner to its string form before emission: serialize through a borrowed-DTO (a Serialize view holding &StringInterner that maps each Name to interner.lookup(name)), or run a name_str resolution pre-pass that materializes the strings. A round-trip test MUST assert the emitted name/targets fields are the test-name strings, never bare integers.
  • Rationale: Failed(String), Skipped(String), and LlvmCompileFail(String) carry arbitrary test-failure messages. Those messages routinely contain control characters — tabs, newlines, backslashes, double-quotes, and non-ASCII unicode. Hand-rendering those strings without correct JSON escaping reproduces the exact “control chars produce invalid JSON” parse-error class the sibling consumption section (s-bec662fc) exists to fix. serde_json escapes every string field correctly by construction; the hand-render path does not.
  • Duration serialization (ABSOLUTE): TestSummary, FileSummary, and TestResult each carry a pub duration: std::time::Duration field (compiler_repo/compiler/oric/src/test/result/mod.rs). std::time::Duration has NO default serde::Serialize impl in serde 1.x, so a bare #[derive(Serialize)] will NOT compile. Serialize each duration explicitly via a #[serde(serialize_with = ...)] attribute. The emitted unit is PINNED to integer nanoseconds (the field name in the JSON schema is duration_ns, an unsigned 64-bit integer) — this is Duration’s native 64-bit-ns representation, so the conversion is lossless and the schema contract is deterministic (the implementer does NOT choose the unit; the sibling consumption section s-bec662fc reads duration_ns as an integer). Same residual-field-type class as the interned-Name resolution above.
  • serde dependency (ABSOLUTE): the oric crate (compiler_repo/compiler/oric/Cargo.toml) does not currently depend on serde/serde_json. This section MUST add serde = { version = "1", features = ["derive"] } and serde_json = "1" to [dependencies] (pin to the workspace versions already used elsewhere in the workspace if a workspace-level pin exists). The #[derive(Serialize)] + serde_json::to_string calls above do not compile without these deps; adding them is part of this section’s deliverable.
  • Mandate a unit test asserting that a failure message containing control characters serializes to valid JSON (the emitted bytes parse back via a JSON parser without error).

Spec References

compiler_repo/compiler/oric/src/commands/test.rs; compiler_repo/compiler/oric/src/test/result/mod.rs (TestSummary/FileSummary/TestOutcome); compiler_repo/compiler/oric/src/test/runner/mod.rs.

Dependencies

This section depends on the walking-skeleton section (s-7a95df82), which landed the OutputFormat enum, the run_tests format match, and the minimal TestSummary::render_json. This section consumes that existing branch; it does not re-introduce the --format flag. The walking-skeleton is the predecessor and a prerequisite for this section’s work.

Scope Boundary

This section owns the RUNNER side only: the full JSON schema + its emission from the ori test JSON branch. It stops at runner JSON emission.

The durable test-all.sh consumption migration — replacing parse_ori_results with a json.dumps-fed parse of the runner’s JSON — is OUT OF SCOPE here and is owned by the sibling consumption section (s-bec662fc). The boundary: this section emits and pins the schema; the consumption sibling reads it in the harness.

Acceptance

The deliverable is satisfied only when the emitted JSON carries the full per-test payload, not a presence-only top-level shape. A minimal {passed, failed, skipped} integer render does NOT satisfy this section. Concretely, the parsed JSON object d MUST satisfy ALL of:

  • isinstance(d['files'], list) and len(d['files']) > 0 — at least one per-file entry is present.
  • Each per-file entry exposes its per-test results, and at least one per-test outcome key from {Passed, Failed, Skipped, SkippedUnchanged, LlvmCompileFail} appears across the payload — outcome variants are carried, not collapsed.
  • A duration field is present (per-test and/or per-file/summary durations).
  • The error-file count is represented.

This Acceptance intent strengthens the section’s success criterion beyond the presence-only probe ('files' in d or 'summary' in d) so the criterion is not satisfiable by the walking-skeleton’s minimal render. The plan.json criterion rewrite (assert isinstance(d['files'], list) and len(d['files'])>0, at least one per-test outcome key, and a duration field) is script-mediated; this body section is the authored intent for that rewrite.

Work Items

  • Extend the existing OutputFormat::Json branch of ori test to emit the full TestSummary + per-file + per-test-outcome + durations as serde_json-serialized machine-readable JSON; text remains the default and stays byte-identical. Derive Serialize on the result types; do not hand-render.
  • Cover every TestOutcome variant (Passed/Failed/Skipped/SkippedUnchanged/LlvmCompileFail) + error-file count in the JSON schema with a round-trip parse test. The round-trip test MUST include at least one Failed/Skipped message containing control characters — tabs, newlines, backslashes, double-quotes, and unicode — and assert the serializer escapes them so the output round-trips through a JSON parser cleanly. This control-char adversarial gate lives at the emission layer so escape bugs are caught before any harness consumer trips on them.