100%

Semantic-dup correctness (Defect 2)

Goal

Make MISSING_ABSTRACTION a real semantic-dup signal by routing it through the existing-but-bypassed HNSW cosine index, with structural fusion to kill false positives.

Implementation Sketch

Score B does NOT start from zero cosine integration:

  • metrics.py:478-543 already carries a per-pair embedding-cosine correction surface (corrected_similarity:483, embedding_cosine_for_pair:526, correct_pair_missing_abstraction:536) over gds.similarity.cosine against the symbol_embedding vectors, tagged “proof-of-pipe; full swap is s-a04ca9c6” (metrics.py:479).
  • This section is that full swap.

What it REPLACES: the proof-of-pipe re-scores ONE Jaccard-generated pair after the fact — the candidate set still comes from the gds.nodeSimilarity Jaccard projection (_NODE_SIMILARITY_STREAM, metrics.py:294-305), and the cosine only corrects the stored similarity / drops sub-floor pairs per pair. Jaccard saturates at 1.0 on token-overlapping but semantically-unrelated pairs (the verify_trmc_soundness ~ linker_args false pair), so a Jaccard-first candidate set admits false positives the per-pair correction can only prune one at a time.

What it ADDS — candidate GENERATION from the HNSW index, so false pairs are never proposed:

  • Replace the gds.nodeSimilarity Jaccard candidate stream with the native Neo4j vector index db.index.vector.queryNodes('symbol_embedding', k, $query_vector) (the HNSW index already exists per enrich_embeddings.py:449-461; intel_repo/CLAUDE.md §Embedding).
  • Keep neighbors with cosine >= _load_cosine_floor() (the proof-of-pipe’s corrected_similarity(jaccard, cosine, floor) keep/drop verdict is reused on the GENERATED set); store similarity = cosine (the property stops lying).
  • Fuse the GDS structural node-similarity term by a BANDED rule (w-6314a776), NOT a hard intersection: high-cosine pairs survive unconditionally; the structural term gates only the ambiguous near-floor band where cosine-false-positives cluster.

What it LEVERAGES unchanged: the existing _PAIR_MA_DELETE / _PAIR_MA_SET_COSINE write pipeline (metrics.py:511-523), the declarative per-run purge/set transaction (metrics.py:470-472), the canonical_pair endpoint ordering (metrics.py:282), and the degree_penalized_score hub penalty (metrics.py:275).

KNN backend decision — native Neo4j vector index

Use db.index.vector.queryNodes('symbol_embedding', k, $vec) for candidate generation, NOT a GDS KNN in-memory projection:

  • The HNSW symbol_embedding index already exists and is enriched per-commit (intel_repo/CLAUDE.md §Embedding); querying it directly is the lower-complexity path and bypasses the GDS in-memory-projection build/drop overhead the Jaccard gds.nodeSimilarity path pays.
  • GDS is retained only for the structural node-similarity fusion term (w-6314a776), not for candidate generation.

Threshold marker — cosine_floor (no hardcoded float)

Both cosine cutoffs are parseable bold markers — **cosine_floor = 0.80** (keep/drop) and **cosine_high_band = 0.92** (the w-6314a776 banded-fusion survive-unconditionally cutoff) — added to intel_repo/CLAUDE.md (GDS Graph Insights → Metrics-layer thresholds table) and read by new _load_cosine_floor() / _load_cosine_high_band() helpers in intel_repo/neo4j/metrics_thresholds.py, each mirroring _load_jaccard_floor (e.g. _COSINE_FLOOR_RE = re.compile(r"\*\*cosine_floor = ([0-9]+\.?[0-9]*)\*\*"), bootstrap default 0.80, dispatched through _load_float_marker). A hardcoded 0.80 / 0.92 in the score code is a DRIFT:hardcoded-threshold violation per the section-05 threshold contract; the proof-of-pipe corrected_similarity already takes floor as a parameter, so the score code threads the loaders in.

Edit-sequence + sync-order coordination

  • Edit-sequence vs §4 (gds-activation, s-4e62453e): §4 is a LATER route-order section whose work items add GDS centrality/community projections (WCC, Bridges/SCC, betweenness, Leiden, Symbol.crate) to gds_enrich.py + metrics.py. §4 does NOT touch the gds.nodeSimilarity candidate-generation surface — that surface is owned by THIS section’s Score B (gds_enrich.py has zero nodeSimilarity refs; the metrics.py nodeSimilarity term is this section’s). The two sections edit the same metrics.py FILE but DIFFERENT surfaces (nodeSimilarity vs centrality/community), so there is no shared-surface clobber.
  • Route order IS the edit sequence: THIS section (§3) lands FIRST and moves Score B candidate generation off gds.nodeSimilarity onto the vector index; the later §4 rebases its centrality/community additions onto this section’s metrics.py state (a forward dependency: §4 depends on §3, never the reverse). The w-6314a776 structural-intersection term still consumes the EXISTING gds.nodeSimilarity projection (this section’s surface) — only the candidate-generation call moves off it.
  • A backward hint_needs: s-4e62453e / depends_on: s-4e62453e on THIS section is BANNED: declaring a downstream section as predecessor violates the strict linear DAG (INV-19, v7-plan-system.md §3). The coordination is route-order-determined (§3 before §4), not a declared dependency edge.
  • Embedding freshness (NOT a sync-script serialization): enrich_embeddings.py is NOT run by sync-ori-graph.sh. It is a SEPARATE backgrounded post-commit hook (refresh-embeddings, wrapper lefthook.yml) running in PARALLEL with refresh-intel-graph (which runs sync-ori-graph.sh → gds_enrich.py → metrics.py). The two race with a documented ONE-COMMIT lag (intel_repo/CLAUDE.md §Cadence): a freshly-imported symbol may carry no symbol_embedding when KNN candidate generation reads it.
  • This section TOLERATES the lag rather than serializing the hooks — serializing embedding-before-GDS inside sync-ori-graph.sh would regress the established parallel-cadence design (intel_repo/CLAUDE.md §Cadence §06.2). KNN candidate generation SKIPS a symbol whose symbol_embedding is absent/null that commit; the next embedding refresh re-embeds it and its :MISSING_ABSTRACTION edges appear one commit later — consistent with the existing similar-query staleness rationale. The validation cell pins graceful null-embedding handling (skip, not crash), NOT a sync-script ordering assertion.

Work Items

  • Replace the gds.nodeSimilarity Jaccard candidate stream (metrics.py:_NODE_SIMILARITY_STREAM, 294-305) with native-vector-index candidate generation via db.index.vector.queryNodes('symbol_embedding', k, $vec) over the existing HNSW index (enrich_embeddings.py:449-461). Reuse the proof-of-pipe embedding_cosine_for_pair / _PAIR_MA_* write pipeline (metrics.py:511-533) on the generated set rather than re-implementing it.
  • Add the **cosine_floor = 0.80** AND **cosine_high_band = 0.92** markers to intel_repo/CLAUDE.md (Metrics-layer thresholds table) + _load_cosine_floor() / _load_cosine_high_band() in metrics_thresholds.py (mirror _load_jaccard_floor); thread cosine_floor into corrected_similarity(jaccard, cosine, floor) and cosine_high_band into the w-6314a776 banded fusion; store similarity = cosine on the kept :MISSING_ABSTRACTION edges (the property name stops lying). No hardcoded float in score code — both bands are parseable markers.
  • Fuse GDS structural node-similarity (this section’s EXISTING gds.nodeSimilarity projection term in metrics.py, NOT a §4 deliverable) with the embedding-KNN candidate set by a BANDED rule, NOT a hard intersection. A pair with cosine >= _load_cosine_high_band() SURVIVES unconditionally — a strong type-4 semantic-clone signal must NOT be dropped for low structural node-similarity, because structural divergence (different call patterns) is the DEFINING property of a type-4 clone; a hard intersection would drop exactly the clones the cosine signal exists to catch. The structural-agreement intersection gates ONLY the ambiguous band [cosine_floor, cosine_high_band), where cosine-similar-but-unrelated false positives cluster and structural agreement disambiguates them. Boundary vs sibling s-71256fd2 (embedding-quality, w-3d19af41): that section adds AST Merkle clone fingerprints for type-4 semantic clones LATER; this section’s structural term is the existing GDS node-similarity measure, not the Merkle fingerprint. The two do not overlap in implementation — this section ships banded structural-fusion on the current graph; s-71256fd2 adds the Merkle structural term as a follow-on.
  • Pin a durable pytest regression cell test_missing_abstraction_cosine_cutoff in intel_repo/tests/test_metrics_layer_unit.py (the existing metrics unit surface; test_metrics_quality.py for the live-graph assertion), authored test-first per the completed sibling calls-resolution-precision (s-1209f1e9, w-a48c19b9 TDD pattern). The cell asserts: (1) corrected_similarity(jaccard=1.0, cosine=below-floor, floor) returns (False, _) (the verify_trmc_soundness ~ linker_args false-pair class is dropped); (2) corrected_similarity(_, cosine>=floor, floor) returns (True, cosine) (stored similarity is the cosine, not the Jaccard); (3) _load_cosine_floor() reads the live marker (drift pin, mirroring test_threshold_loaders / test_hub_threshold_drift_pin); (4) KNN candidate generation SKIPS a symbol whose symbol_embedding is absent/null (the one-commit embedding-lag case) and returns the kept set minus the unembedded symbol — never raises (pins the lag-tolerance contract, NOT a sync-script ordering assertion); (5) the BANDED fusion (w-6314a776) KEEPS a high-cosine (cosine >= cosine_high_band) low-structural-node-similarity pair — a type-4 true-positive SURVIVES the fusion (positive survival pin), paired with the negative near-floor-low-structural drop — so the structural term cannot silently drop genuine semantic clones. Live-graph validation (cypher count: the false pair is absent from :MISSING_ABSTRACTION; labeled true-positive spot-check) supplements but does not replace the durable cell.

References

  • intel_repo/neo4j/metrics.py — Score B writer: write_missing_abstractions (271-475, _NODE_SIMILARITY_STREAM Jaccard candidate stream at 294-305); the per-pair cosine-correction proof-of-pipe this section supersedes (corrected_similarity:483, embedding_cosine_for_pair:526, correct_pair_missing_abstraction:536, _PAIR_MA_DELETE/_PAIR_MA_SET_COSINE write pipeline 511-523).
  • intel_repo/neo4j/metrics_thresholds.py — threshold-loader SSOT; new _load_cosine_floor() + _load_cosine_high_band() mirror _load_jaccard_floor (103-106).
  • intel_repo/neo4j/enrich_embeddings.py — the symbol_embedding HNSW index (449-461) consumed by db.index.vector.queryNodes for candidate generation.
  • intel_repo/tests/test_metrics_layer_unit.py + test_metrics_quality.py — durable pytest regression surface (cosine-cutoff cell + drift pin).
  • intel_repo/CLAUDE.md §Embedding + §Metrics-layer thresholds — HNSW index contract + the **marker = N** / _load_* threshold discipline.
  • Sibling boundary: plans/intel-graph-precision/content/embedding-quality--s-71256fd2.md (w-3d19af41 AST Merkle clone fingerprints — follow-on structural term); coordination: §4 gds-activation (s-4e62453e) edits the same metrics.py file but different GDS surfaces (centrality/community, not the gds.nodeSimilarity candidate-gen surface this section owns).