50%

Goal

Implement the fused chunk-pool-store + flat-arena-resolve InternShard

Implementation Sketch

See the work items below; each is the bounded deliverable.

Spec References

  • ledger evolution ev14 (eligible) in plans/benchmark-grind-lexer (the hypothesis + kill_criterion).

  • compiler_repo/compiler/ori_ir/src/interner/mod.rs (the InternShard surface).

Work Items

  • Rewrite InternShard to { bytes: Vec, spans: Vec<(u32,u32)>, map: FxHashMap<&‘static str,u32> }: try_intern appends s.as_bytes() to bytes (no Box::leak per ident, no to_owned), records (offset,len), inserts the cycle-10 chunk-pool &‘static str key; lookup(Name) resolves via from_utf8(&bytes[off..off+len]).unwrap() (ori_ir deny-unsafe: NO from_utf8_unchecked). One contiguous per-shard append on write, base+offset read on lookup.

Items

  • Ensure InternShard uses a safe, deny-unsafe, Sync-shared chunk-pool layout: { map: FxHashMap<&‘static str, u32>, spans: Vec<&‘static str> } backed by a per-shard pool of individually-leaked (Vec::leak), fixed-capacity byte chunks that interned bytes are copied into on insert — growing by leaking a NEW, larger chunk when the current one is exhausted, never reallocating (and thereby invalidating) a single shared buffer. try_intern copies s.as_bytes() into the current chunk’s remaining capacity (or a freshly leaked chunk), records the resulting &‘static str as both the map key (the cycle-10 chunk-pool key) and the spans entry — no Box::leak per ident via to_owned, no per-ident heap allocation. Preserve the EXISTING public contract exactly: lookup(&self, Name) -> &str, try_lookup(&self, Name) -> Option<&str>, lookup_static(&self, Name) -> &‘static str — do NOT introduce a lock-held guard/wrapper return type (DO-NOT-RE-TRY per this section’s own ledger attempt 2: a guard-backed InternedStr caused reentrant shard-lock hangs in ori_arc/ori_eval). No unsafe (ori_ir #![deny(unsafe_code)]: no from_utf8_unchecked, no transmute). This design is already realized in compiler/ori_ir/src/interner/shard.rs and validated (6 interner unit tests, ori_ir + ori_patterns suites, clippy -D warnings clean per this section’s ledger attempt 3, CONFIRMED). Confirm the live implementation matches this contract and that the section’s owned probes (cargo build -p ori_ir debug + release; sc-b3a93ca5) are green; do not regress it toward the unsound single-Vec-with-offsets shape.

Fresh intel (regenerated)

VERDICT: This is NOT an unstarted section. The ledger for THIS child plan already records three attempt cycles against s-d483addc (dates 2026-07-01, today), and the current working tree is DIRTY with an UNCOMMITTED partial implementation that only delivers HALF of ev14’s fused hypothesis. Cycle 1 implemented the chunk-pool-store half (allocation-count collapse, matching parent-plan cycle 10’s proven-safe design). Cycle 2 attempted the flat-arena-resolve half via a lock-returning InternedStr guard API and was REVERTED (verdict: WORSE) after it caused REAL reentrant parking_lot::RwLock deadlocks in ori_arc/ori_eval test sweeps. Cycle 3 retreated to the safe chunk-pool-only shape and is KEPT/CONFIRMED — but that retreat means the flat-arena-resolve half of ev14’s hypothesis (the (offset,len) contiguous-byte-arena piece the section body’s w-fb942b5b asks for) has NOT been implemented; it was explicitly abandoned as architecturally incompatible with the existing lock-free &'static str-returning lookup/lookup_static API.

The decisive cross-cutting [JOIN] — five surfaces converge on the SAME structural blocker:

  • [JOIN] git state x ledger: git diff HEAD -- compiler/ori_ir/src/interner/ shows mod.rs modified (-114/+ref) and a NEW untracked shard.rs — this is cycle-3’s KEPT chunk-pool rewrite, uncommitted. HEAD’s committed interner/mod.rs still has the ORIGINAL per-ident Box::leak design (strings: Vec<&'static str>, no chunk pool at all). The live working file I read directly (shard.rs) confirms: map: FxHashMap<&'static str, u32>, spans: Vec<&'static str> (STILL a fat-pointer vec, NOT (u32,u32) offsets), key_remaining: &'static mut [u8] growing leaked chunks via Vec::new().leak(). The section’s own success criterion sc-b3a93ca5 (“InternShard rewritten to bytes-arena + spans + chunk-pool key”) is UNMET by the current live code.
  • [JOIN] blast radius x lock-safety tests: lookup_static has 30 direct callers across ori_arc (AIMS realize, decision-tree flatten, lower), ori_eval (every method-dispatch table), and ori_llvm (codegen dispatch) — and a graph-computed 101 distinct callers transitively into the lookup_static/lookup family. TWO existing interner unit tests (test_lookup_does_not_hold_shard_lock, test_try_lookup_does_not_hold_shard_lock — ori_ir/src/interner/tests.rs:108-133) MECHANICALLY assert the shard RwLock is free immediately after lookup/try_lookup return. Any resolve path that borrows &bytes[off..off+len] from a Vec<u8> living inside RwLock<InternShard> is tied to the guard’s lifetime and CANNOT satisfy both the &'static str return type AND “lock released before return” in safe Rust — which is exactly what cycle 2 discovered empirically (a real deadlock, not a hypothetical).
  • [JOIN] internal prior art x external prior art: the graph’s closest embedding twins for try_intern (Roc SmallStringInterner, rustc Interner, Zig InternPool) and freshly-fetched rustc/matklad/lasso sources ALL confirm the same resolution pattern: a 'static-safe resolve requires the BACKING BYTES to be leaked/arena-stable (never moved, never freed) — a lock is acquired only TRANSIENTLY during the interning/lookup call and is never held across the returned reference. None of the three reference designs use a single reallocating Vec<u8> as the resolve backing store for a lock-shared interner; all three use a chunked/bump arena that never invalidates prior allocations. This is EXACTLY what Ori’s key_remaining chunk pool already does — for the MAP KEY only. The unimplemented half needs the SAME discipline applied a second time to the SPANS structure, not a plain Vec<u8>.
  • [JOIN] parent-plan mission x this plan’s gate: the parent benchmark-grind-lexer ledger (cycles 10/12/13, all REJECTED/INERT) already proved the ALLOCATION-COUNT axis wall-clock-INERT (+0.8%, inside the ±5% noise floor) and the mission was RE-GROUNDED to the reachable ceiling (202.70 MiB/s, sc-f1c702 marked satisfied). fused-interner-ev14’s own gate_metric still targets lexer_throughput unqualified with evolution_n 14’s predicted 213.00 — i.e., this child plan inherits the SAME unreached +10% target (212.762) the parent already declared unreachable via any single-axis lexer lever, betting the JOINT (allocation+indirection) removal crosses the noise floor where NEITHER axis did alone. evolution_n 14’s own kill_criterion text concedes this is the open, unproven question.

Decisions this package forces before continuing s-d483addc:

  1. Do not re-attempt cycle 2’s shape. A lock-returning resolve API (InternedStr guard, or any function whose return borrows a Vec<u8> living behind the shard RwLock) is a proven DO-NOT-RE-TRY dead end — verbatim teeth in ledger cycle 2 (REVERTED, verdict: WORSE, real hangs in ori_arc/ori_eval). Per learning-ledger.md §6, consult this before forming the next hypothesis.
  2. The only safe path to a true flat/compact spans structure is a SECOND leaked chunked arena mirroring key_remaining’s discipline (never move, never realloc-in-place, chunk boundaries tracked), addressed by (chunk_idx, offset, len) or an equivalent compact index — NOT a single reallocating Vec<u8> resolved via a live borrow. This is a genuinely un-attempted shape (cycles 1-3 tried allocation-collapse-only and lock-returning-borrow; neither is this).
  3. Reconcile sc-b3a93ca5’s literal text against what cycle 3 actually proved safe. The criterion still says “bytes-arena + spans” as if this is a simple rewrite; the ledger’s own generalizes_to on cycle 3 says the opposite (“do not make downstream compiler layers depend on private lifetime or lock mechanics… preserve the existing public seam”). This is a state-discipline.md §5 plan-altering-change the section owner must resolve (update the criterion to name the two-arena shape, or accept the chunk-pool-only shape already KEPT as the section’s final deliverable and re-scope the criterion).
  4. The measurement question is genuinely unresolved and may still land inside the noise floor — even the correct two-arena implementation only removes cache-locality overhead on an axis (allocation count) the parent ledger already measured as wall-clock-INERT under a SIMPLER version of this same change. Budget for a refuted outcome per evolution_n 14’s own kill_criterion; the section s-24059b3f measurement work item exists precisely to make this determination and flip the ledger either way.

Do NOT conclude the section is not-started from the plan.json status field alone — that field is stale relative to the ledger + working-tree state; the very first action on this section MUST be reading the FULL ledger (python -m scripts.plan_corpus.ledger show plans/fused-interner-ev14) and running git status/git diff on compiler/ori_ir/src/interner/ before forming any next step.

DIG DEEPER

python -m scripts.plan_corpus.ledger show plans/fused-interner-ev14 --rank --all
cd compiler_repo && git status --short compiler/ori_ir/src/interner/ && git diff --stat HEAD -- compiler/ori_ir/src/interner/
python -m scripts.plan_corpus.read plans/fused-interner-ev14

(full dossier: implement-fused-internshard—s-d483addc.intel.md)