50%

refresh-since-manifest-race-deflake

Goal

refresh --since routes the —changed scope deterministically under parallel-session background syncs; the CLI harness case never flakes.

Diagnosis

  • scripts/intel-query.sh --json refresh --since HEAD~3 --dry-run falls back to CODE_ARGS=(--full) when _refresh_manifest_compiler_sha returns empty or an unresolvable SHA (scripts/intel_query/02b-refresh.sh).
  • NOT a manifest-write race: writes are atomic (graph_manifest.py _atomic_write mkstemp + os.replace under _manifest_lock flock) — torn/mid-rewrite reads are impossible.
  • The pytest failure is DETERMINISTIC: conftest.py’s autouse _isolate_graph_manifest pins ORI_GRAPH_MANIFEST to a never-created tmp path that leaks through subprocess.run into the bash harness — the manifest is genuinely absent for the whole subprocess lifetime. Cure: harness-case hermetic manifest fixture.
  • The separate PRODUCTION hazard is write-side semantic nulling: dirty-tree/errored background syncs passed None (not the _UNSET sentinel) into the manifest writers, clobbering the last-known-good compiler_sha for minutes-to-hours. Cure: PRESERVE_COMPILER_SHA on unverifiable generations.
  • The —full fail-safe stays correct production behavior for a GENUINELY absent manifest.

Work Items

  • Root-cause and cure the deterministic refresh --since <SHA> fallback to --full: empirical reproduction shows this is NOT a manifest-write race — writes are atomic via graph_manifest.py’s _atomic_write (mkstemp + os.replace) under _manifest_lock (flock), so torn/mid-rewrite reads are impossible. The pytest failure of tests/intel_query_cli/04-refresh.sh::test_refresh_since_dry_run_forwards_changed_scope (under test_intel_query_cli_runner.py) is deterministic: the autouse _isolate_graph_manifest fixture (intel_repo/tests/conftest.py:83-94) pins ORI_GRAPH_MANIFEST to a per-test tmp path that leaks through subprocess.run’s inherited env into the bash harness, so the manifest is genuinely (not transiently) absent for the whole subprocess lifetime. Cure has TWO required surfaces: (i) harness hermeticity — give the harness case (or the runner invoking it) its own valid manifest fixture by composing ORI_GRAPH_MANIFEST + ORI_COMPILER_ROOT, per the existing pattern already used in intel_repo/tests/intel_query_cli/06-ensure-fresh-tracing.sh:100, instead of relying on whatever the ambient/inherited env happens to provide; (ii) write-side compiler_sha preservation — change the two call sites sync_ori_graph.py:852-855 (incremental, stats["errors"] != 0) and :1085-1086 (full sync, committed_scope_matches failure) to pass the existing _UNSET sentinel instead of None into update_manifest, so a merely-unverifiable generation (dirty tree / sync error) PRESERVES the last-known-good sha rather than nulling it — an older preserved sha only over-collects the --changed scope for --since, never under-collects, so this is safe for every consumer. As part of this cure, fold the ad hoc runpy heredoc in _refresh_manifest_compiler_sha (scripts/intel_query/02b-refresh.sh:198-210) into a graph_manifest.py CLI subcommand (sibling of the existing ensure-fresh subcommand) so the shell has one SSOT read API instead of a second embedded manifest reader (closes the latent LEAK:algorithmic-duplication). NEVER weaken, skip, or make the harness pin tolerant of --full; NEVER cure via a bare retry loop around the read — retry fixes neither mechanism (the env-pinned absence is permanent for the subprocess’s whole lifetime; the write-side null persists until the next clean-tree sync, not until a retry succeeds).
  • Regression coverage for BOTH real cure surfaces from w-e8f0638e (the recorded ‘mid-rewrite window during the read’ premise is false — manifest writes are atomic; do not attempt to reproduce it): (1) a hermetic version of the --since harness case (or the case itself, made hermetic) that composes its own valid ORI_GRAPH_MANIFEST + ORI_COMPILER_ROOT fixture instead of relying on ambient/inherited env, proving test_refresh_since_dry_run_forwards_changed_scope passes deterministically under BOTH pytest and standalone invocation — proof the fix closes the harness-hermeticity gap, not that the assertion was loosened; (2) a Python-side test (alongside test_manifest_reconcile.py / test_ensure_fresh.py) asserting update_manifest / run_incremental PRESERVE a previously-recorded compiler_sha via _UNSET rather than nulling it via None when a sync is dirty-tree-unverifiable (committed_paths_match/committed_scope_matches fails) or errored, closing the currently-unpinned write-side policy gap. The existing harness case (04-refresh.sh:76-102) stays as the production-path pin verbatim — its assertion is never loosened, only its execution environment is made hermetic.

Fresh intel (regenerated)

The section’s recorded diagnosis is attributing the flake to the WRONG mechanism, and the prescribed cure (bounded-retry manifest read) will NOT fix the failing harness case. Empirical reproduction today (2026-07-14):

  • timeout 150 intel_repo/.venv/bin/python -m pytest intel_repo/tests/test_intel_query_cli_runner.py::test_intel_query_cli_harness_passes — FAILED (rc=1, 98s), at exactly the refresh group.
  • ORI_GRAPH_MANIFEST=/tmp/.../nonexistent.json bash intel_repo/tests/test_intel_query_cli.sh — isolates the failure to exactly test_refresh_since_dry_run_forwards_changed_scope and nothing else.
  • ORI_GRAPH_MANIFEST=<nonexistent> scripts/intel-query.sh --json refresh --since HEAD~3 --dry-run → code_cmd: ["sync-ori-graph.sh","--full"]; without the override → --changed <scope>.

[JOIN] (the decisive one): intel_repo/tests/conftest.py:83-94 defines _isolate_graph_manifest — an autouse=True fixture that monkeypatch.setenv("ORI_GRAPH_MANIFEST", str(tmp_path / "test-graph-manifest.json")) for EVERY test in the suite, including test_intel_query_cli_harness_passes (test_intel_query_cli_runner.py:36). That test subprocess.run(["bash", ...])s the CLI harness (:28-33), and the subprocess INHERITS os.environ — so inside the pytest run the bash harness’s refresh --since reads a manifest path that NEVER EXISTS, read_manifest() returns None, _refresh_manifest_compiler_sha prints nothing, and 02b-refresh.sh:216 routes CODE_ARGS=(--full). This is DETERMINISTIC under the pytest runner and impossible standalone — which exactly matches the observed “two consecutive pytest runs failed while every standalone invocation passed”. It is a test-isolation boundary leak, not a race. A retried/atomic manifest read (w-e8f0638e’s prescribed cure) cannot fix it: the manifest is genuinely absent for the entire subprocess lifetime, and --full is the CORRECT production fallback for genuinely-absent.

A real production hazard DOES exist, but it is not a torn read either. All manifest writes go through _atomic_write (graph_manifest.py:89, temp-file + os.replace) under _manifest_lock (:104), so a reader can never observe partial JSON. What a reader CAN observe is a manifest whose compiler_sha was semantically nulled by the parallel post-commit background sync (lefthook.yml:419, nohup scripts/intel-query.sh refresh):

  • Incremental error path: sync_ori_graph.py:852-855 calls _maybe_update_manifest(..., None) whenever stats["errors"] != 0 — and update_manifest (graph_manifest.py:187-188) OVERWRITES compiler_sha with None because None is not _UNSET.
  • Dirty-tree path: sync_ori_graph.py:827-835 sets generation_sha = None whenever committed_paths_match (rust_scip_snapshot.py:85-97) sees ANY synced path dirty/untracked vs captured HEAD; the full path does the same via committed_scope_matches (:100) at sync_ori_graph.py:1003-1005 before write_full_manifest(source_root, compiler_sha=generation_sha) (:1085-1086). In this project’s normal multi-session environment (“dirty/shared tree is NORMAL”), post-commit syncs routinely stamp compiler_sha: null, and the null PERSISTS until the next clean-tree sync — a window of minutes to hours, not microseconds.

Decisions this package forces before any work:

  1. Re-ground both work items. w-e8f0638e’s “atomic or retried manifest read” premise is falsified: the read IS atomic; retry cannot cure either mechanism (env-pinned absence is permanent for the subprocess; compiler_sha-null persists until the next clean sync). The cure splits into two surfaces: (i) harness hermeticity at the pytest→subprocess boundary (give the harness case a real manifest fixture, or have the runner/case compose its own ORI_GRAPH_MANIFEST + ORI_COMPILER_ROOT pointing at a valid fixture — the exact pattern tests/intel_query_cli/06-ensure-fresh-tracing.sh:100 already uses); (ii) write-side last-known-good compiler_sha preservation (pass _UNSET, not None, when the generation is merely unverifiable — safe for the --since consumer because an OLDER baseline only over-collects the changed set, never under-collects).
  2. Do not weaken the pin — test_refresh_since_dry_run_forwards_changed_scope (tests/intel_query_cli/04-refresh.sh:76-102) is the production-path pin the section explicitly protects; both cures must make it green under pytest AND standalone (INVERTED-TDD:positive-test-modification / :disabled-negative-pin otherwise).
  3. This terrain is graph-dark. Zero :Symbol/:File rows exist for scripts/** or intel_repo/** (the code graph indexes compiler/, tests/, library/ only) — the platform-estate ingestion is the deliverable of sibling sections s-b9cacd0d (platform-domain-python-estate) and s-3a837610 (platform-domain-artifact-fabric), both not-started. All code recon below is filesystem-grounded and every claim carries a runnable command.

(full dossier: refresh-since-manifest-race-deflake—s-031257a4.intel.md)