Lexer Optimization (ledger grind)

Goal

LEDGER GRIND-CYCLE section. Each work item is an iterative ratchet: hypothesize a tuning -> apply -> MICRO-benchmark (fast feedback) -> append a plan.json:ledger[] cycle entry via python -m scripts.plan_corpus.ledger (NEVER a hand-edit; the ledger is plan.json:ledger[] per learning-ledger.md §7, NOT decisions/LEDGER.md) -> consult dead-ends -> repeat until the micro-benchmark plateaus. /continue-roadmap fires the per-cycle ledger_ratchet_directive from cycle 1 (the plan carries a gate_metric + ledger). Regression detection uses —baseline parser-perf-v0 deltas, not absolute MiB/s.

Work Items

• GRIND: #[inline] audit on cross-crate lexer hot fns -> micro-bench (lexer_core/lexer) -> append plan.json:ledger[] cycle (KEPT if faster, REVERTED with WORSE before->after teeth if slower, INERT if NEUTRAL) -> repeat to optimal.
• GRIND: arena/buffer pre-sizing (with_capacity tuning) -> micro-bench -> ledger cycle -> repeat.
• GRIND: cooker fast-path tuning (try_trivial, IdentCache, WITH_METADATA monomorphization) -> micro-bench -> ledger cycle -> repeat.