0%

Section 03: Bootstrap Importer

Goal: A one-time importer parses the current 34 markdown files at compiler_repo/docs/ori_lang/v2026/spec/ into the Section 02 JSON schema, writing the initial JSON corpus.

Success Criteria:

  • Importer parses all 34 files with zero OpaqueMarkdownFragment fallbacks needed for known-recurring structures (NOTE/EXAMPLE/Table/RuleTable/Diagram) — a fallback firing there is a schema gap, not expected behavior
  • Initial JSON corpus written to compiler_repo/docs/ori_lang/v2026/spec/corpus/ (requires the Section 01 approved-proposal bypass, since this write lands inside the hook-gated tree)
  • grammar.ebnf’s 259 productions and operator-rules.md’s per-operator rules both parse into their respective flat models with zero raw-text loss

Context: This section BLOCKS on Section 01 (proposal approval) for its actual corpus write — the write target is inside compiler_repo/docs/ori_lang/v2026/spec/, which .claude/hooks/block-spec-edits.sh gates unconditionally. Reading the existing 34 files is always allowed and may start immediately; only the corpus WRITE needs the bypass.

Depends on: Section 01 (proposal approval, for the write); Section 02 (schema).

Intelligence Reconnaissance

Queries run 2026-07-09: same corpus deep-read as Section 02 (08-types.md, 15-patterns.md, 20-capabilities.md, annex-e-system-considerations.md, grammar.ebnf, operator-rules.md).

Results summary [ori]: the importer’s structural-coverage bar is set directly by the counts found during authoring (17 RuleTable rows in annex-e; 5-file Diagram recurrence; 259 grammar productions; H1-H4 heading depth) — these become the importer’s own test fixtures.

03.1 Markdown Block Parser

File(s): scripts/spec_corpus/importer/markdown_parser.py

  • Implement the markdown -> Block model parser covering every structural element found in research: numbered/bare NOTE, numbered/bare EXAMPLE with nested code fences, tables, the annex-e RuleTable rows, the cross-5-file Diagram structures (with a kind discriminator), and all 3+ observed cross-ref-callout formats (> **Grammar:**, > **Proposal:**, inline markdown links). clause_id is derived from the H1/H2/H3/H4 heading’s leading numeric token (the N/N.M/N.M.P/N.M.P.Q prefix immediately after the #/##/###/#### marker, matching spec_adapter.py’s existing heading-extraction regex ^(#{1,4})\s+(\d+(?:\.\d+)*)\s+(.+?)\s*$) — the SAME derivation the live graph already uses, not a new convention

  • Subsection close-out (03.1)

    • Parser handles all 4 research-sample files + a spot-check of 6 more, zero unexpected OpaqueMarkdownFragment fallbacks
    • A sample of >=20 imported clause_id values is diffed against scripts/intel-query.sh cypher 'MATCH (s:SpecClause) RETURN s.clause_id LIMIT 20' and confirmed identical before Section 05 relies on the MERGE-key match

03.2 grammar.ebnf + operator-rules.md Parsers

File(s): scripts/spec_corpus/importer/grammar_parser.py, scripts/spec_corpus/importer/operator_parser.py

  • Implement a line-oriented parser extracting the 259 productions into GrammarProduction records (name, section_grouping from the preceding // See: comment block, raw_rhs verbatim) — NOT an EBNF grammar parser, per content/decisions/01-grammar-flat-string-not-parsed.md. Populate cross_refs by matching each production’s name against CrossRefCallout{kind: grammar} blocks found on the clause side during 03.1’s parse (a clause’s > **Grammar:** See [grammar.ebnf](grammar.md) § SECTION_NAME callout cross-links to every production whose section_grouping equals SECTION_NAME) — this is the field Section 05.3’s clause-to-grammar-production edges read from; cross_refs is never independently populated from grammar.ebnf’s own text (it has no clause-id syntax of its own beyond the // See: comment)

  • Implement the operator-rules.md parser extracting per-operator OperatorRule records from the TYPE RULES/COMPILE ERRORS/DESUGARING fenced-code subsections

  • Subsection close-out (03.2)

    • Both parsers produce records for all productions/operators; raw_rhs/text fields verified byte-identical to source spans

03.3 Write Initial Corpus (blocked by Section 01)

  • With ORI_SPEC_PROPOSAL=<approved-proposal-filename> set, run the importer end-to-end and write the JSON corpus to compiler_repo/docs/ori_lang/v2026/spec/corpus/

  • Subsection close-out (03.3)

    • Corpus written; python -m scripts.spec_corpus check (Section 02’s validator) passes clean against it

03.R Third Party Review Findings

  • None.

03.N Completion Checklist

  • All subsections (03.1-03.3) [x], status complete
  • Initial JSON corpus exists and validates clean
  • ./test-all.sh green — no regressions
  • /tpr-review passed
  • /impl-hygiene-review passed

Exit Criteria: The 34-file spec corpus is fully represented as validated JSON at compiler_repo/docs/ori_lang/v2026/spec/corpus/.