Measurement + Benchmark Engine + Fairness
Goal
Deterministic build/run/measure of all 4 variants with pinned flags, hyperfine + portable fallback, 3-way checksum + both STRICT hard perf gates, determinism metadata recorded, adversarial C++ baseline checkpoint.
Implementation Sketch
Scripts own every mechanical action; the LLM authors programs/tests and analyzes only. All mechanical work is a single python -m scripts.rosetta_test_runtime <subcommand> invocation directed by a work_item body line.
Work Items
- Implement
build.py— build Ori AOT debug + release with correct flag order (ori build <file> --release -o <out>), C++clang++ -O3 -march=native -flto, Python (no build); eliminates thebuild --release <file>flag-order failure class. - Implement
measure.py— run each variant underhyperfine(warmup + runs + median +--export-json) with a portable Python median-of-N fallback when hyperfine is absent; capture per-variant stdout (= checksum) + median ms + run-count/env metadata. - Compute 3-way checksum agreement (reuse
checksums_agree) + both perf gates (reuseperf_gate.evaluate_perf_gate): Compiled (Ori release <= C++ -O3, 5% tol) AND Interpreted (ori run<= Python3, 0 tol); STRICT hard gate — a loss is a fail with NO skip-escape. - Record determinism metadata (run count, warmup, env, CPU-pin note) + an adversarial C++ baseline review checkpoint (emit a needs-review flag on the c++ baseline before recording the gate) into
benchmark_results.jsonvia the §01 API. - pytest covers build-flag composition, the hyperfine + fallback timing path, and checksum + gate verdicts;
--self-testgreen.