0%

structural lexer throughput levers

Goal

Lift lexer_throughput toward the target (212.762 MiB/s; baseline 193.42, current best 202.70) by working the profile-confirmed structural hot loop, now that the inline micro-opt surface is exhausted. This is a new grind DIRECTION on the SAME gate_metric and the SAME plan-scoped plan.json:ledger[] — not a spin-off. Each work item is one grind lever worked as a ledger cycle.

Profile grounding

Callgrind (/tmp/cg-lexer.out) localizes the cost to the scan+dispatch+driver hot loop:

  • ori_lexer::driver::lex_driver — 17.6% self Ir (biggest single self-cost; only token_span has been sunk so far, in cycle 1).
  • ori_lexer_core::raw_scanner::next_token — 10.8% self Ir.
  • ori_lexer::cooker::cook — 10.6% self Ir.
  • Interner HASHING is only ~3.4% — the interner-hash-once hypothesis is WRONG and is a recorded dead-end. The one real interner lever is ALLOCATION: ori_ir try_intern does Box::leak(s.to_owned()) per unique identifier (5000 heap allocations for the func{i} corpus).

The fmt/u64 ~20% in the raw profile is corpus-generation contamination, not lexer cost — do NOT chase it.

Dead-ends inherited from the plan ledger (do NOT re-try)

  • Cycle 2 INERT: cook_int / numeric-literal no-underscore fast paths — int literals are too sparse in the measured corpus to move the metric.
  • Interner hash-once (hash computed twice on insert) — profile shows hashing is ~3.4%, not the bottleneck.

Per-lever rigor (every work item)

  • Graph-first recon (/query-intel callers/callees/similar on the target symbol) BEFORE editing; record graph-recon: in the cycle clause.
  • Behavior-neutral edit (the lexer output — tokens/tags/flags/spans — is byte-identical); the lever is a layout / allocation / dispatch change, never a semantics change.
  • Measure lexer_throughput via compiler_repo/scripts/bench_probes/measure.sh lexer (best-of-N, regression-thresholded).
  • Full ./test-all.sh: no new failures over the 1803 floor; no CRASHED/ERR suite.
  • Dual debug + release build green; interp/LLVM parity unaffected (lexer is pre-typeck, but the gate still runs).
  • Land each lever as a ledger cycle: KEPT on BETTER + floor held (verbatim before->after teeth); REVERTED (verdict WORSE + teeth) or INERT (verdict NEUTRAL) otherwise, with a generalizes_to dead-end clause when the verdict generalizes.

Spec / file references

  • compiler_repo/compiler/ori_lexer/src/driver.rs — lex_driver main loop.
  • compiler_repo/compiler/ori_lexer_core/src/raw_scanner — next_token.
  • compiler_repo/compiler/ori_lexer/src/cooker — cook + try_trivial fast path.
  • compiler_repo/compiler/ori_ir interner — try_intern / Box::leak allocation site.
  • Parser/lexer rules: .claude/rules/parse.md (LB-2/LB-3/LB-4 token + interning invariants the levers must preserve).

Items

  • Driver hot-loop per-token bookkeeping (ori_lexer::driver::lex_driver): reorder / sink unconditional per-iteration work into the arms that consume it, and tighten the match-arm + pending_flags + offset update sequence. Same behavior-neutral ‘sink unconditional loop-head work’ class that banked cycle 1 (lazy token_span, +3.17%). Measure lexer_throughput; KEPT only on BETTER + floor held.
  • Raw scanner next_token dispatch + branch layout (ori_lexer_core::raw_scanner::next_token, 10.8% self Ir): reduce per-token dispatch cost via branch reordering by token frequency / table-driven classification, keeping (tag,len) output identical. Measure; KEPT on BETTER + floor held.
  • Cook dispatch reduction (ori_lexer::cooker::cook, 10.6% self Ir): widen the try_trivial fast path and shrink the cooked-token dispatch so frequent operator/delimiter/identifier tokens bypass more of cook(). Behavior-neutral. Measure; KEPT on BETTER + floor held.
  • Interner per-unique allocation arena (ori_ir try_intern Box::leak(s.to_owned()) per unique identifier): replace per-unique heap leak with a bump/string-arena so 5000 unique idents on the func{i} corpus do not each hit the allocator. The ONE profile-confirmed interner lever (allocation, NOT hashing — the hash-once hypothesis is a recorded dead-end). Measure; KEPT on BETTER + floor held.

Fresh intel (regenerated)

{ “schema_version”: 1, “target”: { “kind”: “plan-section”, “ref”: “benchmark-grind-lexer/s-d27fc95d” }, “generated_at”: “2026-06-19T15:10:40.668648+00:00”, “graph_state”: { “head_sha”: “a1b25f7a”, “last_code_import_at”: “2026-06-19T15:04:37.956Z”, “embedding_stale”: true, “insights_stale”: false, “cpg_stale”: false }, “surfaces”: {}, “summary”: “intel-package[plan-section:benchmark-grind-lexer/s-d27fc95d] agent-authored dossier”, “degraded”: false, “dossier”: { “objective_symbols”: [], “tiers”: {}, “agent_authored”: true, “difficulty”: { “class”: “open-hard”, “research_online”: true, “research_mode”: “auto”, “signals”: [ “grind-plan:gate_metric=lexer_throughput”, “ledger-evolution:11-entries”, “dead-end-density:4-reverted-or-inert(>= 3)” ], “plan_dir”: “/home/eric/projects/ori_lang/plans/benchmark-grind-lexer” } } }

(full dossier: structural-lexer-throughput-levers—s-d27fc95d.intel.json)