100%

Benchmark Infrastructure + Ownership Reconciliation

Goal

Establish reproducible correctness and performance oracles before implementation, preserving verified benchmark provenance from the retired plans/perf-engineering plan without recreating its overlapping VM work.

Implementation Sketch

  • The benchmark harness already exists on disk as a three-lane oracle stack: the generic scripts/bench_probes/measure.sh + registry.json probe, the shared vm_benchmark_cases.rs correctness registry driving the Criterion timing lane and the exact-result admission test, and the compiler/oric/examples/vm_breadth/ applicability-sweep + differential-parity harness over a frozen manifest. The work is hardening + comparator provisioning + memory accounting, not greenfield. Record provenance from verified source commit 28d01e497, verify each lane against current source, and rerun its tests before freezing a baseline; do not assume any specific file layout at the target compiler HEAD.
  • Extend the generic bench system by DATA, never by forking it. scripts/bench_probes/measure.sh + registry.json is the single harness (“one system, subjects are data rows”); VM subjects are added as registry.json rows carrying their own external_baseline, kind, direction, and noise_threshold_pct. A second per-subject bench script or a VM-specific runner that forks measure.sh is LEAK:algorithmic-duplication.
  • Keep every shared harness plan-agnostic. scripts/bench_probes/** and any generic runtime must not hard-code the bytecode-vm plan name, a subsystem term list, or a per-plan heuristic (STRUCTURE:plan-specific-tooling-leak, Critical). Plan-specific knowledge lives in the data row, not the runtime.
  • Preserve the micro ladder and macro runner, but add a versioned environment manifest, output checks, warmup policy, sample counts, noise bounds, and separate process-startup from VM steady-state measurements. Capture timing and memory in the same run identity.
  • Provision two explicit comparators: Python 3 as the minimum viability floor (Ori/Python <= 0.50 per program and by corpus geometric mean) and a pinned LuaJIT build invoked with -joff as the north-star mission/completion gate. Programs must implement equivalent algorithms and produce checked output.
  • Prove comparator work-equivalence MECHANICALLY, not by inspection. Each comparator triple carries an output check plus a work-count pin (per benchmark-methodology.md BM-V03/BM-V04), so a subtle algorithm divergence between the hand-written Ori/Python/LuaJIT programs is caught by the harness rather than assumed away. Equal source intent is not equal work.
  • Characterize every frozen workload with language-independent dynamic metrics (allocation rate, call polymorphism degree, basic-operation frequency mix) and record them beside the timing. The breadth matrix must be PROVABLY diverse from those metrics, not asserted diverse; the observed “repeated 4-16-operation regions rather than one universal hot opcode” is the diversity claim this metadata has to substantiate.
  • Freeze a breadth matrix before optimizing: isolated dispatch/primitive probes; numeric loops and branches; direct, deep, recursive, and higher-order calls; allocation/COW and nested collections; iterator/reduction pipelines; strings and runtime services; panic/overflow/resource-limit paths; and application-shaped programs that combine several of those semantics. Exercise multiple input scales, call depths, nesting depths, and working-set sizes.
  • Treat combined applications as their own gate. Include graph traversal, parse/filter/transform-style string pipelines, sort/deduplicate or indexing workloads, and at least one larger mixed program whose hot path crosses calls, control flow, collections, allocation/ownership, and runtime operations. A bundle of disconnected microkernels does not satisfy this gate.
  • Report every frozen workload individually, the corpus geometric mean, the slowest ratio, distribution summaries, and opcode/runtime/ownership mixes. A geometric mean cannot hide a program that misses the Python floor, and a win on one isolation probe cannot promote an ISA, dispatch, value, frame, quickening, or tiering choice.
  • Record unadjusted and idle-baseline-subtracted peak RSS, final and peak managed objects/bytes, cumulative allocation count/bytes, peak frame/register/scratch bytes, immutable program/bytecode/metadata bytes, and JIT code/tier-state bytes when applicable. Measure growth slopes across the declared scale and repeated-run axes; final live VM-owned objects and bytes must be zero on success, error, panic, and quota exit.
  • Freeze memory comparators beside timing comparators: the prototype floor is workload-attributable peak memory at most half Python’s per program and by geometric mean with unadjusted peak RSS no worse than Python; pinned LuaJIT -joff is the strict-interpreter memory north star. Report raw values and ratios so baseline subtraction cannot conceal a large resident runtime.
  • Run an applicability sweep over the existing Ori fixture corpus and report entered_vm, fallback, unsupported, and failed counts plus stable reason classes. Unsupported combined programs stay in the denominator and drive closure work; the harness must not silently narrow the corpus to currently implemented operations.
  • Create the differential harness shape early so later sections can select fixtures and run VM/tree-walker/LLVM parity plus leak checks through one command.
  • Collapse the harness’s duplicated contract types while hardening it: contract.rs carries two parity-status types (BackendParityStatus, ParityStatus) and two per-lane disposition enums (OracleWorkerDisposition, VmWorkerDisposition) that share structure and can drift apart. Reduce each pair to one type parameterized by lane. The remaining near-duplicate pairs (Supervised/Supervisor, CompileErrorKind/ExecutionErrorKind, the FROZEN_* constants, the *Kind taxonomy) are intentional complements and stay as they are.
  • Make native-runtime identity part of every AOT record and cache key. Before debug or release AOT comparison, explicitly build the matching ori_rt static archive, record its canonical path and content hash, and force relinking when that identity changes. Building oric alone is not proof that target/{profile}/libori_rt.a is current.
  • Treat the deleted plans/perf-engineering node as historical provenance, not a live dependency. Preserve its verified commit/evidence citations in this section; all continuing VM benchmark ownership lives here, and no unresolved cross-plan edge may target the retired plan.

Spec & Rules

  • Never weaken a frozen gate to make a result green. Environment drift produces a new labeled baseline; it does not overwrite history.
  • A missing LuaJIT executable blocks only the LuaJIT reading, not implementation or the Python floor; the plan cannot complete while the north-star mission criterion is unsatisfied.
  • A parity mismatch invalidates every performance sample from the affected executable. The harness may normalize only a frozen, reviewed set of inherently nondeterministic presentation fields; it must never normalize semantic values, diagnostic classes, output order, exit behavior, panics, or ownership/leak results.
  • A stale compiler binary, runtime archive, cached object, or linked executable is an invalid run, not a backend failure or pass. The harness must emit and compare compiler/runtime/artifact identities before interpreting behavior.
  • Promotion requires both speed and memory breadth/interaction evidence: the candidate must survive the frozen semantic slices and combined applications at every declared scale. Results must label compilation/startup, warm interpreter-only execution, and any quickened/JIT tier separately. Pareto tradeoffs are recorded explicitly; leak, unbounded growth, or worse-than-floor memory rejects a candidate regardless of speed.
  • The evaluator is the immutable semantic oracle. Editing ori_eval, or normalizing semantic values, output order, or exit behavior, to make a parity comparison green is the canonical INVERTED-TDD trap this section exists to prevent. A parity mismatch is evidence about the VM, never a defect in the oracle.
  • Every frozen artifact is append-only with provenance. The <= 0.50 Python floor, the LuaJIT north star, the manifest FROZEN_* digests, and each labeled baseline are never edited to absorb drift or a regression; drift produces a NEW labeled record beside the old one.
  • Guard the classification surface, not just the timing surface. Risk concentrates in worker.rs::compile_source and the *_error_kind taxonomies in worker/; every historically hard repair here was a CLASSIFICATION defect (the worker_record_missing misclassification, the ARC class-ledger panic versus harness/process_crash, the tail-call ()-versus-Unit mismatch). A change to those symbols carries a typed-classification regression pin.
  • Benchmark harness work changes measurement, not language semantics.

Live Prototype Harness Evidence — 2026-07-14

  • One shared fifteen-case registry drives the compile-once/run-many Criterion lane and the real-VM admission test.
  • The registry covers exact nonzero return/output pins, numeric calls, strings, COW snapshots, nested lists, file-style pipelines, JSON-style construction, call chains, and looped/nested aggregate escapes.
  • The admission test compiles explicit typed and untyped bytecode, executes each twice, and pins exact result/output plus deterministic metrics and zero final session state.
  • Every case requires zero exit-live heap objects/bytes before forced cleanup; teardown zero is not accepted as a logical leak-freedom oracle. Nested-list aliasing, nested string COW forks, and split-string release fixtures pin transitive ownership and exact evaluator/LLVM/typed-VM/untyped-VM results 61, 67, and 71.
  • The thin pathfinder reports result, steps, cumulative heap allocations/payload lower bound, exit/final/peak heap state, heap-table capacity, frame/register/scratch capacity, output capacity, and phase timing.
  • Two aggregate-escape fixtures return exact codes 37 and 73 through evaluator and LLVM AOT, while the saved pre-repair VM parent and lazy-frame candidate return 99; the red pins block promotion until identity is stable.
  • The first mixed readings exposed distinct resource pressure: the small numeric case allocates no heap objects; the string case performs 127,000 logical allocations; and the repaired file_pipeline and json_builder paths now exit bytecode with zero live heap objects after transitive AIMS-driven release. The ownership worklist is session-owned, reusable, and measured independently from frames and value storage.
  • A sustained recursion diagnostic returned exact Int(0) after 1,752,452,642 dispatches. One-shot whole-process readings were 12.05 s / 7,392 KiB for the current pre-reclamation VM candidate, 4.60 s / 10,576 KiB for Python 3, and 1.44 s / 2,600 KiB for pinned LuaJIT under -joff: Ori/Python time was 2.61956521739, Ori/LuaJIT time was 8.36805555555, and Ori/Python RSS was 0.698940998487. This candidate therefore fails both the <= 0.50 Python admission floor and the strict-interpreter north star; its lower raw RSS does not satisfy the half-Python attributable-memory floor.
  • Dynamic profiles across recursion, strings, file transformation, JSON construction, sharing/calls, and nested loops show repeated 4-16-operation regions rather than one universal hot opcode. The recursion profile alone executes 1,752,452,642 operations dominated by Copy, IntBinary, Const, Jump, Branch, Return, and Call; portable verified regions must therefore be tested across the mixed corpus rather than fitted to it.
  • The first frozen applicability sweep selects every real @main under tests/{run-pass,spec,benchmarks,valgrind} and excludes only generated /_test/ modules. Of 207 unique programs, 13 are intentionally frontend-invalid; the 194-program executable denominator yields 47 evaluator-correct full-VM runs (24.2268041237%), one wrong result (0.515463917526%), 122 realization/unsupported results (62.8865979381%), 21 post-entry runtime failures (10.8247422680%), three five-second timeouts (1.54639175258%), zero verifier failures, and zero fallback.
  • Strict majority is 98 of 194, leaving a 51-program closure gap. First-failure classes are 94 unresolved runtime operations (length masks 45), 14 aggregate-versus-heap projection failures, ten AIMS contract-coherence failures, ten map constructors, five executed-unreachable failures, and smaller closure/partial-RC, aggregate arity, switch, and class-ledger gaps. Keep the frozen denominator and rerun it after every closure family; the upper bound from first errors is a prioritization signal, never a forecast of downstream success.
  • A provenance audit tightens the interpretation of that snapshot: 194 means frontend-valid, not evaluator-correct. The old oracle executed only the 48 nominal full_vm rows, so unsupported, runtime, timeout, and verifier rows have no evaluator observation; its evaluator binary was not hashed and its process exit status cannot represent arbitrary signed i64 results. Preserve the counts as historical evidence, but do not reuse the TSV workflow as an acceptance harness.
  • Freeze the 207-case denominator in a versioned manifest rather than rerunning a live glob. The recovered ordered path-list digest is 5c001b9b79bfa41620c9c59bef988077172cd10e3d255cf40c433d04c8f267f6; the current ordered path-and-content digest is 810d99140c99164c853436888aa9163964298f2d65b1270885189f5ff16a194e. New or changed fixtures produce explicit manifest drift and never silently change the denominator.
  • The production harness now validates a checked-in 207-case manifest with raw-file SHA 95d7d9ece593f00e65b5b7d926365b5730c42957d96f2776b9e756d9897b2500. Its canonical source index hashes each ordered path NUL source_sha256 LF record to 32d847df4ceb2ac66d451570e4e633bb2fc611808f9d2c33537d8a2a0ae26d95; this is a specified serialization and is not interchangeable with the earlier raw path-and-content digest. Validation pins 207 selected, 194 frontend-valid, 13 frontend-rejected/not-run, 48 historical full-VM rows, 122 unsupported, 21 runtime failures, three timeouts, and the independent 47-match/1-mismatch historical oracle.
  • Replace display-text classification with deterministic typed JSONL: one identity/config record, exactly one case record per manifest entry in manifest order, and one reconciling summary. Record evaluator and each requested VM plan independently; exact tagged result, output bytes/digest, typed phase/error, canonical steps, complete success-or-failure ExecutionMetrics, bytecode/plan bytes, cleanup state, process RSS, timeout phase, and requested-versus-executed plan are mandatory. Any plan substitution is fallback, even when output matches.
  • That JSONL contract is executable. It supports historical-48 and all-frontend-valid oracle policies, always attempts the generic VM on all 194 frozen frontend-valid cases, records physical execution as explicitly unavailable rather than falling back, supervises each worker in an isolated process group, bounds exact output capture, samples Linux peak RSS, and reconciles every denominator. Eight focused tests, manifest validation, full-dependency warning-denied checking, and workspace formatting pass.
  • The first all-frontend-valid run used a 5,000 ms per-worker timeout, manifest SHA above, and executable SHA 5467fe850fc96f8e8ed14a4a54d6f3f65ea000011dbd43d37460f9a3ff2af433. All denominator checks reconciled. The evaluator produced 182 successes, three runtime errors, and nine timeouts. The generic VM produced 72 exact value/output matches, zero mismatches, one comparison-unavailable result, 85 unsupported results, ten realization errors, 18 runtime errors, five timeouts, three internal errors, and zero compile/verifier/fallback results. Exact coverage is 37.1134020618557% of the 194 executable denominator: 25 more correct programs than the frozen 47-case snapshot, but still 26 short of the strict-majority threshold of 98. Physical execution remained typed unavailable for all 194 cases. This is semantic-closure evidence, not a timing benchmark.
  • Typed failure grouping makes the next closure order explicit. The 85 unsupported rows contain 53 missing callable/runtime identities, 19 unsupported constructors, 11 indirect-closure instructions, and two constructors wider than the four-field prototype arena. Runtime failures split into six unsupported sum-discriminant operations, six executed-unreachable paths, five Heap-versus-Int type mismatches, and one aggregate-field bounds error. Ten programs fail the shared post-AIMS contract-coherence seam; five time out; three workers terminate abnormally; and the repaired historical tail-call mismatch is now comparison-unavailable rather than wrong. Treat these as independent typed families. The 53-row missing-callable group is the first breadth target, but no row is credited until its actual program produces evaluator-equal value/output with zero fallback and clean final state.
  • The runtime grouping exposed one shared logical-sum defect across 17 programs: six unsupported discriminants, six default-arm Unreachable executions, four ConstantString-versus-Int tag reads, and one field-bounds failure. ARC’s executable projection contract places a logical enum tag at field 0 and payloads at fields 1..N, while AIMS Set mutates zero-based payload slots and SetTag mutates the tag separately. The VM already stored tag and payload separately but projected the payload array as though every aggregate were a product. An explicit product/variant aggregate kind now preserves those two logical views without importing LLVM niche/tag offsets. All 17 programs complete as exact Int(0) in typed and untyped modes with zero fallback, unsupported execution, final heap state, or final arena state. The 100,000-iteration sum-payload case executes 3,950,024 logical steps while bounding the arena to 1,025 slots and 188,448 measured bytes through 97 collections, 99,327 reclaims, and 98,975 reuses. Keep this as targeted closure evidence until the frozen 194-case sweep independently credits each row. The evaluator and fresh LLVM debug/release AOT return zero for the 100,000-iteration representative, and both AOT profiles pass full definite/indirect Valgrind leak checks with zero errors.
  • The three apparent worker_record_missing internal failures were harness misclassification, not silent compiler outcomes. All panic in the ARC class-ledger readiness invariant and exit 101 before writing a worker record. The supervisor now gives abnormal process termination authority over the derivative missing-record symptom, emitting typed harness/process_crash, executed_plan: null, fallback: false, exact exit status, and bounded stderr. Successful exits with no record remain worker_record_missing; spawn failures remain process_spawn. Eleven harness tests and an end-to-end rerun of all three cases pin the distinction. The ARC panics remain semantic work and cannot be credited as VM support.
  • The sweep’s sole wrong result was tests/spec/codegen/tail_call_test.ori: source () reached ARC as an empty tuple construction, so the VM returned an empty aggregate while evaluator and LLVM returned Unit. ARC now canonicalizes the empty tuple to LitValue::Unit; typed/untyped VM, evaluator, LLVM debug/release, AOT, and leak checks are green. Keep the recorded snapshot unchanged and require the next full sweep to remove the mismatch.
  • Initial receiver-aware len/length closure used the registry’s Len protocol; the explicit MethodRuntime identity below supersedes that provisional inference while preserving one backend-neutral RuntimeCall::Length. Preserve the original 45-program first-blocker count until the frozen sweep is rerun, and count unsupported Map/Set execution explicitly.
  • Runtime-capable methods now carry an explicit registry MethodRuntime identity instead of deriving execution from trait or source spelling. An authoritative replay of all 53 first-run missing-callable rows blocked by insert/push/set/prepend/remove/updated/len/length executed 106 typed/untyped plans with identical classifications: 23 full-VM results, 17 explicit unsupported results, 13 later independent failures, and zero fallback in each mode. Every full-VM run ended with zero final heap and value-arena state. The remaining insert rows are Map operations; the List insert/remove benchmark advances to the separately missing List.last call. Keep these as targeted closure results until a fresh all-194 sweep credits exact evaluator-equal rows.
  • The broad 10,000-door stress case first failed at step 2,534,418 because Range<int>.iter and [bool].iter had both become an unqualified runtime call and bytecode had discarded the post-AIMS receiver ownership. After repairing both shared seams, typed and untyped VM runs complete as exact Int(100) after 2,714,457 steps with zero fallback/unsupported and zero exit/final heap or arena state; evaluator and LLVM debug/release AOT return 100. A real Invoke/unwind regression releases the owned list source, and scalar-child filtering reduces measured ownership scratch from 160,096 B to 160 B. Keep the original failure coordinate as provenance, retain range/list plus borrowed/owned and unwind pins, and extend the gate through managed/nested yielded elements, every admitted source kind, early exit, double drop, AOT, and Valgrind.
  • The first session-stable append-only value arena passes the exact escape pins but is rejected as a memory design: 100-doors exhausts its 1,000,000-entry cap after about 181 million dispatches, with 980,953 aggregates and 19,047 iterators occupying 83,886,080 measured bytes. Reclamation/reuse must preserve generation-safe identity, trace all VM roots, and report live/peak/reuse/collection metrics before the identity repair can be promoted.
  • The generation-safe trace/sweep candidate closes that failure on the 50,000-iteration 100-doors variant: exact checksum 540000, 952,600,025 steps, 5,250,001 allocations, 5,249,960 reclaimed entries, 5,248,971 reused entries, 1,030 peak live/physical slots, 193,472 measured peak arena bytes, zero exit-live heap state, and 7,164 KiB unadjusted peak RSS. Its sustained-recursion mean is 13.110 s versus Python’s 4.485 s and LuaJIT -joff’s 1.428 s, so Ori/Python is 2.92307692308, the Python-floor speedup gap is 5.84615384615, and Ori/LuaJIT is 9.18067226891. Raw recursion RSS is 7,644 KiB versus Python’s 10,576 KiB and LuaJIT’s 2,600 KiB.
  • On the shorter output-checked 100-doors comparator, thirty-run means are 37.2 ms Ori, 18.4 ms Python, and 2.1 ms LuaJIT -joff; the ratios are 2.02173913043, a 4.04347826087 speedup gap to the Python floor, and 17.7142857143 respectively. One-shot raw RSS is 6,976 KiB Ori, 10,336 KiB Python, and 2,548 KiB LuaJIT: Ori/Python is 0.674922600619 and Ori/LuaJIT is 2.73783359498, so neither memory target is satisfied.
  • These are diagnostic readings, not a frozen benchmark verdict. Remaining gaps include failure-path cases, unsupported-case classification, declared scales, fresh-process RSS, and automated evaluator/LLVM/AOT admission.

Oracle Commands

PurposeCommandYields
Grind single-number probescripts/bench_probes/measure.sh evalMIN wall-clock seconds for the row’s Ori program
Comparator readingscripts/bench_probes/measure.sh eval --baseline-refthe row’s external_baseline reading — the reference the grind must beat
Applicability + parity sweepcargo run --example vm_breadth -- <manifest>typed-JSONL per-case records: tagged result, output bytes/digest, typed phase/error, canonical steps, ExecutionMetrics, cleanup state, RSS, timeout phase, requested-versus-executed plan, reconciled denominators
Timing lanecargo bench -p oric --bench vmcompile-once/run-many median per registry case
Exact-result admissioncargo test -p oric --test vm_benchmark_corpusexact result/output plus zero-final-session-state pins

This section OBSERVES ownership; it does not emit it. Final-live-zero and Valgrind readings are admissible harness evidence, but a default-path RC count is never an RC verdict — that verdict belongs to the AIMS sections this harness measures.

Sibling Coordination

  • prior-art-ingestion (s-fa0fd48c) registers this section as a downstream consumer of the Dispatch and Portability decision rows under the strict lane. The portability gate is inherited, not restated: a portable winner must hold on every supported target class jointly on timing, host instructions, code/metadata bytes, and RSS, with foreign assembly and unsafe rejected.
  • perf-grind-luajit-class (s-07271bdd) co-owns the interpreted-gate corpus ratio and carries its own dossier. Coordinate on the shared metric; do not duplicate its LuaJIT-class recon.
  • parity-verification (s-4b00387e) owns the L9 cross-executor parity obligation. This section builds the harness that obligation runs through and certifies none of it.
  • register-vm-dispatch (s-d34c4229) and walking-skeleton (s-0279c4d5) are the measured subjects whose steady-state and end-to-end path this harness exercises.
  • scripts/bench_probes/ is SHARED tooling owned by the benchmark-grind contract, not by this plan. This section is a consumer that adds data rows.

Declared Recon Limits

  • Derived graph facets are stale for this terrain: embeddings, insights, co-change, test-topology, and sibling-divergence have not consumed the current code import. Treat symbol-terrain metrics, drift, complexity, co-change, and confirmed-duplication as UNKNOWN rather than clean; ground structural claims by reading the harness sources directly.
  • The duplication pairs named in the Implementation Sketch are graph NOMINATIONS. Confirm each by reading both endpoints before collapsing them.
  • No issue-sentiment corpus exists for this terrain — the reference perf runtimes are code-only in the graph. The external methodology literature is the community bar instead, and the reviewer standard for this harness is that it survives a published benchmarking-crimes checklist: no selective subsetting, no improper baseline, no variance-free reporting, no comparison only against itself, no micro presented as macro, and no arithmetic mean of normalized scores.
  • Plan-to-code ownership edges are absent for example and test code, so ownership above is derived from plan structure rather than symbol references.

Layer Coverage

LayerDisposition
L1N/A: measurement tooling does not change parsing.
L2N/A: measurement tooling does not change typechecking.
L3N/A: measurement tooling does not change canonicalization.
L4N/A: measurement tooling does not change ARC/AIMS.
L5N/A: the harness observes evaluators but does not alter them.
L6N/A: the harness observes LLVM debug but does not alter it.
L7N/A: the harness observes LLVM release but does not alter it.
L8N/A: the harness observes AOT but does not alter it.
L9N/A: this section builds the oracle; parity obligations belong to VM implementation sections.
L10N/A: this section builds the leak probe; it introduces no runtime allocation behavior.
L11N/A: no language behavior is added; benchmark fixture validity is covered by output oracles.
L12N/A: the harness is developer tooling, not a new compiler production entry point.

Work Items

  • Port/reuse the verified perf-engineering benchmark implementation, add the differential harness and applicability classifier, and freeze the breadth matrix plus output-checked tree-walker, Python 3, and pinned LuaJIT -joff speed/memory baselines with environment provenance, per-program/geometric-mean/worst-case ratio reporting, scale sweeps, allocation/footprint/growth accounting, final-live-zero checks, and separately labeled interpreter/tiered results.
  • Prove comparator work-equivalence mechanically: pair every Ori/Python-3/LuaJIT -joff comparator triple with an output check plus a work-count pin (per benchmark-methodology.md BM-V03/BM-V04), so a subtle algorithm divergence between the hand-written programs is caught by the harness rather than assumed away from equal source intent alone. This is the section’s own explicit Implementation Sketch bullet and the dossier’s identified hard obligation for guaranteeing the equal-work precondition external methodology (AWFY) requires.
  • Characterize every frozen workload with language-independent dynamic metrics (allocation rate, call-polymorphism degree, basic-operation frequency mix) recorded beside the timing data, so the breadth matrix’s claimed diversity (“repeated 4-16-operation regions rather than one universal hot opcode”) is PROVABLY substantiated rather than merely asserted, per the section’s own Implementation Sketch and the dossier’s recommended AWFY-style sharpening.
  • Collapse contract.rs’s duplicated parity-status types (BackendParityStatus/ParityStatus) and per-lane disposition enums (OracleWorkerDisposition/VmWorkerDisposition) into one type each, parameterized by lane, while hardening the harness; leave the intentional-complement pairs (Supervised/Supervisor, CompileErrorKind/ExecutionErrorKind, FROZEN_* constants, the *Kind taxonomy) unchanged. Named explicitly in the section’s Implementation Sketch and independently confirmed as a live LEAK:algorithmic-duplication/DESIGN:low-cohesion risk by the dossier’s conformance audit (score 0.968 and 0.962).
  • Make native-runtime identity (the ori_rt static archive’s canonical path and content hash) part of every AOT benchmark record and cache key; explicitly build the matching archive before debug or release AOT comparison and force relinking when that identity changes, treating a stale binary/archive/cached object as an invalid run rather than a backend pass or fail. Named explicitly in both the Implementation Sketch and the section’s Spec & Rules invariants.
  • Preserve the retired perf-engineering benchmark provenance and history in this section, clear its obsolete structural dependency through the sanctioned plan-corpus API, and verify this plan is the sole live owner of continuing VM benchmark work.

Intel dossier — pointer and tier disposition

Dossier: benchmark-infrastructure--s-b5d1f822.intel.md (this section’s content/intel/ sidecar)

Read that dossier from line 1 through EOF before acting on this section. This block is a pointer and an audit record, never a substitute: not the BLUF, not a summary, not selected tiers.

Dossier tierTitleLinesDispositionWhere / why
00. Bottom line up front56integratedImplementation Sketch (new ‘Extend the generic bench system by DATA, never by forking it’ + ‘Keep every shared harness plan-agnostic’ bullets); Spec & Rules (new ‘evaluator is the immutable semantic oracle’ + ‘Every frozen artifact is append-only with provenance’ rules)
11. The target — verbatim from the section54not_applicableTier 1 is the section’s own Goal, Implementation Sketch, Spec & Rules, and work items quoted back verbatim. It imposes no constraint the section does not already state in those headings, so no heading changes.
22. The terrain — symbols + files in the objective area61integratedImplementation Sketch (first bullet names the three-lane oracle stack and its exact paths); Oracle Commands (new heading binding each lane to its invocation plus the ExecutionMetrics VM seam the harness consumes)
33. Code-graph recon56integratedSpec & Rules (new ‘Guard the classification surface, not just the timing surface’ rule naming worker.rs::compile_source and the *_error_kind taxonomies as the churn-concentrated, historically-repaired surface requiring a typed-classification regression pin)
3D3D. Diagnostic / observability surface31integratedOracle Commands (new heading: measure.sh probe and —baseline-ref comparator, vm_breadth sweep, Criterion lane, admission test, plus the ‘a default-path RC count is never an RC verdict’ boundary)
3H3H. Hygiene constraints (design VERDICTs inherit these)37integratedImplementation Sketch (no-fork/data-row rule, plan-agnostic-runtime rule, duplicated parity/disposition collapse); Spec & Rules (frozen-artifact append-only rule, evaluator-as-oracle rule)
44. Cluster / family — sibling sections in the bytecode-vm plan27integratedSibling Coordination (new heading: prior-art-ingestion strict-lane Dispatch and Portability inheritance, perf-grind-luajit-class shared-metric coordination, parity-verification owning L9, register-vm-dispatch and walking-skeleton as measured subjects)
55. Conformance audit — MISSING_ABSTRACTION twins over the harness30integratedImplementation Sketch (new bullet collapsing BackendParityStatus/ParityStatus and OracleWorkerDisposition/VmWorkerDisposition to one lane-parameterized type each, retaining the intentional complements); Declared Recon Limits (records these as graph nominations requiring both endpoints read before collapsing)
66. Plan ownership — who owns this code now18integratedSibling Coordination (scripts/bench_probes/ is shared tooling owned by the benchmark-grind contract, this section is a consumer that adds data rows); Declared Recon Limits (ownership derived from plan structure because plan-to-code edges are absent for example and test code)
77. External prior art (web) — OPEN-HARD problem class85integratedImplementation Sketch (new ‘Prove comparator work-equivalence MECHANICALLY’ bullet carrying the BM-V03/BM-V04 work-count pin, and new ‘Characterize every frozen workload with language-independent dynamic metrics’ bullet making breadth-matrix diversity provable rather than asserted); Declared Recon Limits (the published benchmarking-crimes checklist as the reviewer standard)
88. Sentiment — community heat around this terrain20not_applicableNo issue-sentiment corpus exists for this terrain: the reference perf runtimes (cpython, luajit, lua, quickjs, mruby, wasm3) are code-only in the graph with zero issues and zero PRs, so the tier yields no community signal. Its only consequence, that the external methodology literature becomes the reviewer bar, is already integrated from tier 7 into Declared Recon Limits.
99. Declared coverage gaps31integratedDeclared Recon Limits (new heading: stale embeddings/insights/co-change/test-topology/sibling-divergence facets treated as UNKNOWN rather than clean, duplication pairs as nominations, absent sentiment corpus, plan-to-code edge absence for example and test code)
1010. Recommended recon entry points (for the consumer)34not_applicableTier 10 is consumer reading-order navigation into artifacts the section’s own headings now name directly: the three-lane stack and its invocations (Oracle Commands), the sibling sections and their inherited gates (Sibling Coordination), and the benchmark-methodology ruleset (Spec & Rules). It adds no contract, constraint, or verdict beyond pointing at those.