100%

cargo-nextest

Goal

Route the Rust unit/integration/AOT legs of test-all.sh through cargo-nextest for its intra-binary process-per-test scheduling and native machine-readable output, replacing plain cargo test for those legs while keeping cargo test --doc for doctests (nextest cannot run doctests).

Scope distinction (no overlap, no contradiction):

  • test-all.sh ALREADY runs every Rust leg in parallel at the LEG level (Phase 2 background & + wait, compiler_repo/test-all.sh:527-564). nextest does NOT add leg-level parallelism — that exists today.
  • What nextest adds is INTRA-binary process-per-test scheduling: within a single test binary, one slow or hanging #[test] no longer blocks the rest of that binary’s tests, and a crashing test is isolated to its own process rather than aborting the whole binary’s run.
  • This is distinct from the LLVM Context-lock parallelism owned by section incremental-and-llvm-parallelism (s-… — LLVM-backend ori test worker scheduling). That section parallelizes the Ori-LLVM leg; this section parallelizes within the Rust-cargo legs. The two do not touch the same code.

Implementation Sketch

The naive swap — replacing each cargo test ... call with cargo nextest run ... — is NOT correct and reintroduces a known race. test-all.sh deliberately splits into two phases (compiler_repo/test-all.sh:480-564):

  • Phase 1 (serial, compiler_repo/test-all.sh:482): cargo test --no-run -q --workspace --lib --bins --tests pre-builds ALL debug test binaries serially, plus the oric/ori_rt builds (L492) and the LLVM release build (L499).
  • Phase 2 (parallel, compiler_repo/test-all.sh:527-564): every run_rust_* function is RUN-ONLY — it executes the already-built binaries and never compiles into the shared target/. This is what prevents the failed to write ...rmeta: No such file (os error 2) build-artifact race (documented at compiler_repo/test-all.sh:475-476) that concurrent compiling cargo invocations trigger.

A bare cargo nextest run COMPILES if anything is stale, so dropping it into Phase 2 reintroduces exactly that race. The correct shape preserves the pre-build/run split:

  1. Phase 1 (serial pre-build): build the nextest binaries ONCE via cargo nextest archive --archive-file build/nextest-archive.tar.zst --workspace --exclude ori_llvm --lib --bins --tests (plus the per-leg selectors for ori_rt, ori_llvm --lib, ori_llvm --test aot), guarded on command -v cargo-nextest. archive is nextest’s build-once-run-many mechanism (cargo nextest run has NO --no-build flag — the archive IS how compilation is hoisted out of the run phase). When cargo-nextest is absent, keep the existing cargo test --no-run pre-build unchanged.

  2. Phase 2 (run-only): rewrite run_rust_workspace (compiler_repo/test-all.sh:161), run_rust_rt (:190), run_rust_llvm (:201), and run_aot (:214) to call cargo nextest run --no-fail-fast --archive-file build/nextest-archive.tar.zst <same leg partition> when cargo-nextest is present — --archive-file runs the pre-built archive with ZERO compilation (the true run-only step that preserves the race-free split), falling back to the current cargo_race_retry "$OUTPUT" test <selector> wrapper form (retry-on-rmeta-race preserved) when absent. --no-fail-fast keeps full-suite-run semantics (no leg aborts early on first failure).

  3. Leave run_rust_doctests (compiler_repo/test-all.sh:175) on cargo test --workspace --doc unchanged — nextest cannot run doctests, and the harness already documents this as the lone compiling invocation of Phase 2 (compiler_repo/test-all.sh:177-180).

  4. Summary parsing: parse_rust_results (compiler_repo/test-all.sh:319-330) scrapes plain cargo test’s human ^test result: lines via grep/sed/awk. cargo-nextest’s human summary uses a different shape (Summary [...] N tests run: N passed ...), so the existing parser returns zeros for nextest legs. The fix consumes nextest’s machine output instead: run with cargo nextest run --message-format libtest-json (libtest-compatible JSON, stable across nextest’s NEXTEST_EXPERIMENTAL_LIBTEST_JSON surface) OR parse nextest’s own summary line, and route the per-leg counts into the same ${prefix}_{PASSED,FAILED,IGNORED} variables parse_rust_results already populates. The doctest leg keeps the existing cargo test --doc parse path.

    Two boundary constraints on this parse-path change:

    • BUG-07-241 (live parse-path correctness gap): the existing parse_rust_results scrape carries an open correctness gap tracked as BUG-07-241; the nextest libtest-json consumer added here is a SEPARATE machine-output path for the four nextest legs and MUST NOT inherit the text-scrape’s gap. Whichever path is active (nextest-json when present, the existing text scrape on fallback), a parse failure / unrecognized shape sets a HARD-failure sentinel + non-zero exit (never silent zero) — the same parse-error contract s-bec662fc established for the Ori runner. This section’s parse change is scoped to the Rust-leg nextest path only; it does NOT reach into the BUG-07-241 cargo-text-scrape fix (that remains its own tracked work).
    • Boundary vs s-5494ed74 w-1b3d6409: the metrics-ledger section’s deferred w-1b3d6409 (“migrate parse_rust_results to cargo test --message-format json”) targets the CARGO text path; this section adds the NEXTEST libtest-json path. The two COEXIST (different runners, different machine formats) — this section does NOT subsume w-1b3d6409; both feed the same ${prefix}_* vars via distinct parse branches selected by command -v cargo-nextest.

No test is dropped: the same --workspace --exclude ori_llvm --lib --bins --tests, -p ori_rt, -p ori_llvm --lib, and -p ori_llvm --test aot selectors are preserved; only the runner and the summary-parse path for those four legs change.

Spec References

compiler_repo/test-all.sh:

  • run_rust_workspace — compiler_repo/test-all.sh:161
  • run_rust_doctests — compiler_repo/test-all.sh:175 (kept on cargo test --doc)
  • run_rust_rt — compiler_repo/test-all.sh:190
  • run_rust_llvm — compiler_repo/test-all.sh:201
  • run_aot — compiler_repo/test-all.sh:214
  • Phase 1 serial pre-build — compiler_repo/test-all.sh:480-503
  • Phase 2 parallel run-only — compiler_repo/test-all.sh:507-564
  • rmeta build-artifact race rationale — compiler_repo/test-all.sh:475-479
  • parse_rust_results summary parser — compiler_repo/test-all.sh:319-330

cargo-nextest docs: process-per-test execution model; cargo nextest archive --archive-file build-once-run-many (nextest has NO --no-build run flag); libtest-json machine output (NEXTEST_EXPERIMENTAL_LIBTEST_JSON).

Success Criteria

  • [sc-187d8596] test-all.sh routes the Rust unit (run_rust_workspace, compiler_repo/test-all.sh:161), runtime (run_rust_rt, :190), LLVM-lib (run_rust_llvm, :201), and AOT (run_aot, :214) legs through cargo nextest run --archive-file when command -v cargo-nextest succeeds, falling back to cargo test when it does not, with run_rust_doctests (:175) left on cargo test --doc.
  • The nextest run-only invocations live in Phase 2 (compiler_repo/test-all.sh:507-564) and their compilation is hoisted into the Phase 1 serial pre-build (compiler_repo/test-all.sh:480-503) via cargo nextest archive --archive-file (Phase 1) consumed by cargo nextest run --archive-file (Phase 2 — zero compilation at run time), preserving the race-free pre-build/run split — verified by a full ./test-all.sh run completing with NO failed to write ...rmeta error and NO suite CRASHED/ERR leg.
  • The harness summary reports per-leg PASSED/FAILED/IGNORED counts for the nextest legs equal to the counts the cargo test path reported before the swap (no test dropped, no count silently zeroed). Two granularities verify this: (a) the deterministic sc-187d8596 probe asserts cargo test --list count == cargo nextest list count across the changed legs (no full run needed); (b) the L12 production-entry-point evidence is a full ./test-all.sh run whose compiler_repo/build/test-all-summary.json Rust-leg totals match the pre-swap run on the same HEAD. The probe is the cheap per-leg gate; the live test-all run is the four-leg L12 confirmation.
  • With cargo-nextest uninstalled (or PATH-masked), ./test-all.sh falls back to the existing cargo test legs and produces identical pass/fail verdicts — verified by a fallback-path run.

Work Items

  • Route the Rust unit/integration/AOT legs through cargo-nextest with a guarded command -v cargo-nextest fallback to cargo test, preserving the Phase 1 serial pre-build / Phase 2 run-only split via nextest’s build-once-run-many archive (Phase 1 cargo nextest archive --archive-file build/nextest-archive.tar.zst, Phase 2 cargo nextest run --archive-file ... = zero-compile run-only; nextest has NO --no-build run flag), keeping run_rust_doctests on cargo test --doc, and consuming nextest’s libtest-json machine output so parse_rust_results reports correct per-leg counts (the nextest parse path enforces the hard-failure-sentinel parse-error contract, scoped separate from the BUG-07-241 cargo-text-scrape gap, COEXISTing with s-5494ed74 w-1b3d6409). Anchors: compiler_repo/test-all.sh:161,175,190,201,214,319-330,480-564.