0%

Section 02: JSON Schema Design

Goal: A versioned, dataclass-based schema package for the spec JSON corpus exists, mirroring scripts/plan_corpus’s own exact schema-versioning pattern, with every structural element found in the 34-file corpus (NOTE/EXAMPLE numbering, tables, recurring RuleTable and Diagram structures, multiple cross-ref-callout formats, H1-H4 headings) representable — grounded in the concrete counts found during authoring (e.g. RuleTable-style rows recur 17x/46 rows in annex-e alone; Diagram-style box-drawing recurs across 5 files, not 1).

Success Criteria:

  • scripts/spec_corpus/schemas/ package exists with a recursive Block union type (Paragraph | Note | Example | Table | CodeBlock | CrossRefCallout | RuleTable | Diagram | OpaqueMarkdownFragment) as hand-rolled frozen dataclasses (NOT Pydantic), matching scripts/plan_corpus/schemas/’s exact style
  • scripts/spec_corpus/version.py exists: bare CURRENT_SCHEMA_VERSION: int = 1, pkgutil-discovered migrations/ package (no manual registry list), consumer pin at .claude/.sync-ref/spec-corpus-version.json, matching plan_corpus/version.py’s MINOR/MAJOR bump-policy text verbatim in a header comment
  • GrammarProduction{name, section_grouping, raw_rhs} is a FLAT model (no parse tree) per content/decisions/01-grammar-flat-string-not-parsed.md
  • clause_id format pinned as bare dotted-numeric string (e.g. "14.3"), globally unique, matching the 822 existing :SpecClause nodes’ MERGE key — a unit test asserts the format regex directly against a sample of real existing clause_ids read from the live graph

Context: Deep-read research (this plan’s authoring) found: NOTE blocks (bare and numbered NOTE 1/NOTE 2), EXAMPLE blocks (bare and numbered, with nested ori code fences), markdown tables, inline ebnf fences mid-prose (15-patterns.md), an ASCII box-drawing diagram recurring across 5 files (operator-rules.md, 19-testing.md, 21-memory-model.md, 18-modules.md, annex-e), and a distinct rule-ID table structure (| Rule | Constraint | rows, e.g. RL-1/TF-1) recurring 17x within annex-e alone. A generic “opaque markdown” escape hatch was considered for these but REJECTED for RuleTable/Diagram specifically since both recur — they get real typed Block variants; OpaqueMarkdownFragment remains only as a genuine escape hatch for whatever the importer (Section 03) encounters that fits no known pattern.

Reference implementations:

  • plan_corpus scripts/plan_corpus/version.py: CURRENT_SCHEMA_VERSION int + pkgutil-discovered migrations + consumer pin — the exact pattern to mirror
  • plan_corpus scripts/plan_corpus/schemas/section_schemas.py: hand-rolled frozen @dataclass style (not Pydantic) — the exact style to mirror

Depends on: None for the safe subset (clause/annex/frontmatter schema); the grammar.ebnf production-schema shape and cross-ref-edge schema wait on Section 01’s proposal settling scope (per /tp-help round-1 nuance).

Intelligence Reconnaissance

Queries run 2026-07-09:

  • Deep-read of scripts/plan_corpus/version.py, schemas/section_schemas.py, migrations/__init__.py, migrations/001_owns_crates_uniqueness.py, migrations/012_binding_constraints.py — exact pattern captured (see Success Criteria).
  • Deep-read of 4 structurally-diverse spec files (08-types.md, 15-patterns.md, 20-capabilities.md, annex-e-system-considerations.md) — structural inventory captured in Context above.
  • scripts/intel-query.sh cypher 'MATCH (s:Symbol)-[r:REALIZES_CLAUSE]->(sc:SpecClause) RETURN count(r)' -> 99 (corrected from an initial mis-query against the wrong node label) — confirms REALIZES_CLAUSE is a live, separate mechanism, out of this plan’s scope (see Section 05).

Results summary [ori]: plan_corpus’s schema+versioning pattern is concrete and mirror-able; spec corpus structural diversity requires ~9 typed Block variants, not a flat string; clause_id MERGE-key format is load-bearing and independently verified.

02.1 Content-Block Schema Package

File(s): scripts/spec_corpus/schemas/blocks.py, scripts/spec_corpus/schemas/clause.py

  • Author scripts/spec_corpus/schemas/ package: Clause{id, title, heading_depth, frontmatter, blocks, subclauses}, Annex{..., informative}, and the Block union (Paragraph, Note{numbered, ordinal}, Example{numbered, ordinal, code_blocks}, Table{rows}, CodeBlock{lang}, CrossRefCallout{kind: grammar|proposal|rule|annex}, RuleTable{rows: {rule_id, text}}, Diagram{kind: pipeline|tree|graph|inference_rule, raw}, OpaqueMarkdownFragment{raw}) as frozen dataclasses per plan_corpus/schemas/’s style

    @dataclass(frozen=True)
class RuleTable:
    rows: tuple[tuple[str, str], ...]  # (rule_id, text)

  • Subsection close-out (02.1)

    • Package authored; unit tests cover each Block variant round-tripping to/from dict

02.2 Versioning + Migrations Pattern

File(s): scripts/spec_corpus/version.py, scripts/spec_corpus/migrations/__init__.py

  • Author version.py mirroring plan_corpus/version.py exactly: CURRENT_SCHEMA_VERSION: int = 1, consumer_pin_path/read_consumer_pin/write_consumer_pin at .claude/.sync-ref/spec-corpus-version.json, and the MINOR/MAJOR bump-policy comment block verbatim-adapted; author migrations/__init__.py with the same pkgutil-discovery available_migrations()/latest_registered_version() pattern (no manual registry list)

  • Subsection close-out (02.2)

    • version.py + migrations package authored; unit tests mirror plan_corpus’s migration-discovery test shape

02.3 Grammar + Operator Models (flat, not parsed)

File(s): scripts/spec_corpus/schemas/grammar.py, scripts/spec_corpus/schemas/operators.py

  • Define GrammarProduction{name, section_grouping, raw_rhs, cross_refs} as a FLAT model (raw_rhs is a verbatim string, no parse tree) and OperatorRule{name, symbol, type_rules, compile_errors, desugaring} matching operator-rules.md’s per-operator TYPE RULES/COMPILE ERRORS/DESUGARING structure

  • Subsection close-out (02.3)

    • Models authored; unit tests round-trip a sample production/operator-rule to dict

02.4 clause_id Invariant + Shared Citation Field

File(s): scripts/spec_corpus/schemas/clause.py, cross-plan coordination note

  • Pin clause_id as bare dotted-numeric string (regex ^\d+(\.\d+)*$); write a unit test asserting this regex matches a live sample of clause_id values read via scripts/intel-query.sh cypher 'MATCH (s:SpecClause) RETURN s.clause_id LIMIT 20'

  • Document the shared spec_clause_refs[] field shape ({clause_id, corpus: "language"|"compiler"}) here as the design both this plan’s Section 06 and the sibling plans/compiler-spec-system plan converge on — a single field, not two parallel per-corpus fields

  • Write content/decisions/01-grammar-flat-string-not-parsed.md recording the flat-string decision (content provided; see plan’s decisions/ dir)

  • Subsection close-out (02.4)

    • Invariant test passes against live graph sample; ADR filed; citation-field shape documented

02.R Third Party Review Findings

  • None. (Both /tp-help consensus rounds already ran during plan authoring; see content/decisions/01-grammar-flat-string-not-parsed.md for the round-2 resolution this section’s design incorporates.)

02.N Completion Checklist

  • All subsections (02.1-02.4) [x], status complete
  • Unit tests for every schema module pass
  • content/decisions/01-grammar-flat-string-not-parsed.md filed
  • Plan sync: status -> complete; Section 03 unblocked (pending Section 01 too)
  • /tpr-review passed
  • /impl-hygiene-review passed

Exit Criteria: scripts/spec_corpus/schemas/ + version.py + migrations/ exist, unit-tested, mirroring plan_corpus’s exact versioning pattern; clause_id invariant verified against the live graph.