docs(04-02): complete golden-corpus + bake-off-close plan

- SUMMARY: 3/3 tasks, 5 commits, corpus gate green at 100% top-3 / 7-7 refusals, lever None; deviations documented (fixture-source implication, bbox pin, pavilion-kl coordinate fix)
2026-10-24 01:05:25 +08:00
parent 99bc9c9a09
commit 3aec9542a3
1 changed files with 200 additions and 0 deletions
@@ -0,0 +1,200 @@
 ---
 phase: 04-destination-search-kl-gazetteer
 plan: 02
 subsystem: search
 tags: [search, golden-corpus, d14-gate, overpass, sqlite, nominatim, tdd]
 requires:
  - "plan 04-01 search workspace (contracts, normalizer, scorer, gazetteer, loader)"
  - "Node 24 type-stripping (.mjs script imports ../src/index.ts)"
  - "node:sqlite (built into Node 24.18.0 — converter output path)"
 provides:
  - "D-13 golden corpus v1 — 75 pinned hand-written cases in 6 categories with real negatives"
  - "D-14 gate — `npm run search:corpus` (exit 0 only on top-3 >= 0.9 + zero leaks) and the in-process CO1-CO8 gate inside `npm run search:test`"
  - "D-06 tool — search/scripts/overpass-topup.mjs (one UA'd request, PROPOSAL output, soft-exit 0 on busy, --source=overpass|nominatim|fixture)"
  - "D-05 tool — search/scripts/json-to-sqlite.mjs (same id set, real .sqlite via node:sqlite, ODbL notice embedded)"
  - "docs/SEARCH-QUALITY.md — the bake-off close: measured rates, lever None, coordinate verification"
 affects:
  - "plan 04-03 (bot /cari handler ships against a closed, green corpus gate)"
  - "plan 04-04 (web deep-link reader — same module, corpus unchanged)"
  - "future top-ups (the Overpass proposer + SEARCH-QUALITY record define the maintenance loop)"
 tech-stack:
  added:
    - "node:sqlite DatabaseSync (converter only — repo tool, zero added deps)"
  patterns:
    - "measure-once/report-twice: the runner exports measureCorpus(); the suite asserts the same numbers in-process"
    - "main-guarded .mjs scripts (CLI when invoked, importable library otherwise)"
    - "source-introspection law gates (TP1/TP6/TP7, CV2/CV3) following bot/test/privacy.test.mjs"
 key-files:
  created:
    - search/test/fixtures/golden-corpus.json
    - search/scripts/gazetteer-corpus.mjs
    - search/test/corpus.test.mjs
    - search/scripts/overpass-topup.mjs
    - search/test/fixtures/overpass-response.json
    - search/scripts/json-to-sqlite.mjs
    - search/test/topup.test.mjs
    - search/test/converter.test.mjs
    - docs/SEARCH-QUALITY.md
  modified:
    - search/package.json (corpus delegate)
    - package.json (search:corpus delegate)
    - search/src/data/kl-gazetteer.json (pavilion-kl coordinate fix)
 decisions:
  - "Corpus authored to the shipped 30-entry gazetteer per the plan's ILLUSTRATIVE clause: dropped non-existent targets (Intermark, Cheras South, MV — 2-char queries sit below the scorer's min-length guard by design) and pinned equivalents that exist (IOI, Pudu, Tmn Perdana, Kl Towr, ...)"
  - "Near-miss negatives chosen with measured margins: Penang 0.625 and Kampung Cina ~0.58, both in [0.55, 0.72) — lowering the admission line leaks them and fails CO2+CO8"
  - "measureCorpus() exported from the runner and reused by corpus.test.mjs — one measurement, two reporters, no drift"
  - "--fixture=<path> implies --source=fixture (a fixture flag silently hitting the live network would be a trap)"
  - "Converter emits a REAL .sqlite via node:sqlite (probed available in Node 24.18.0) — no dependency added; ODbL notice embedded in a metadata table + NOTICE sidecar"
 metrics:
  duration: "~14 min executor time"
  completed: 2026-10-23
  tasks: 3
  filesCreated: 9
  filesModified: 3
 ---
 # Phase 4 Plan 2: Golden Corpus + Bake-off Close Summary
 **One-liner:** the D-13/D-14 experiment closed mechanically — a 75-case pinned corpus measured
 top-3 68/68 (100%) with 7/7 negatives refused at margin 0.643 < 0.72 on the FIRST run
 (lever: None), enforced twice (`npm run search:corpus` exit-code gate + CO1-CO8 inside
 `npm run search:test`), alongside the two forkability tools: an Overpass top-up proposer
 (one UA'd request, PROPOSAL-only output, exit-0-on-busy) and a JSON→SQLite converter that
 preserves the id set exactly.
 ## What Was Built
 ### Task 1 — pinned golden corpus + D-14 runner + Nyquist gate (RED `dea7971` → GREEN `b2689f7`)
 - `golden-corpus.json`: 75 hand-written cases — landmark 17, abbreviation 15, glued 7,
  typo 14, alias 15, negative 7 — every positive pinning a real gazetteer id
  (start_param-safe charset), `threshold.top3_hit_rate: 0.9`. 16 non-negative cases are not
  string-equal to any alias/name (tautology floor is 15). Negatives include two MEASURED
  near-misses (Penang 0.625, Kampung Cina ~0.58) so a lowered admission line fails CO2+CO8.
 - `gazetteer-corpus.mjs`: imports the SAME `src/index.ts` the bot uses (D-03), measures
  top-1/top-3 + per-category rates + avg ms/query + the refusal margin, prints MISSED/LEAKED
  blocks, the threshold line, and the verbatim escalation ladder on failure; exits 0 only on
  threshold + zero leaks. Exports `measureCorpus()`; main-guarded so the suite can import it.
 - `corpus.test.mjs`: CO1-CO8 — threshold, zero leaks, tautology floor, dangling-id charset,
  size+category floors, per-category reporting, and the pinned refusal margin (replicating
  the scorer's two-level whole-string+token fuzzy shape over `damerau`).
 - Delegates: `search:corpus` (root), `corpus` (search workspace).
 - Measured: **top-3 68/68 (100.0%), negatives refused 7/7, margin 0.643 < 0.72, 1.07 ms/query —
  threshold met on the first run.** The two top-1 misses ("KCLC" 3-way fuzzy tie broken
  id-ascending; "Sunway Vellocity" legitimately riding PREFIX via the "Sunway" alias) are
  documented expected behaviour.
 ### Task 2 — Overpass top-up (D-06) + JSON→SQLite converter (D-05) (RED `ec98c8d` → GREEN `c6a5f2b`)
 - `overpass-topup.mjs`: maintainer-time repo tool under the six hard laws — never a runtime
  dependency (TP1), ONE request per run via a single shared `requestJson` helper for both the
  Overpass and Nominatim paths (TP7), mandatory identifying User-Agent (TP2 — live-verified
  406-without), bbox-bounded single-family query pinned verbatim (TP3), busy/502/429/timeout/
  unparseable → `Overpass instance busy — retry later` + exit 0 (TP4), PROPOSAL-only output
  whose source never names the curated file (TP6). Flags: `--source=overpass|nominatim|fixture`,
  `--endpoint=`/`OVERPASS_ENDPOINT`, `--bbox=`, `--q=`, `--tag-family=`, `--fixture=`, `--out=`.
  `--fixture` implies the fixture source (offline by default when a fixture is named).
 - `overpass-response.json`: committed real-shape Overpass response (6 KL elements — nodes with
  lat/lon, ways/relation with center) for deterministic offline tests.
 - `json-to-sqlite.mjs`: reads the same committed `kl-gazetteer.json`, emits
  `kl-gazetteer.sqlite` (places + metadata tables) via `node:sqlite` — zero added
  dependencies — with the SAME id set (CV1 reads it back through DatabaseSync) and the ODbL
  notice embedded in the metadata table + a NOTICE sidecar (ODbL §4.4, T-04-20). Deterministic:
  dataset order, no timestamps.
 - Suites: TP1-TP7 + CV1-CV3 all green; grep gates confirm zero `overpass`/`sqlite` mentions
  under `bot/src/` and `web/src/`.
 ### Task 3 — bake-off close + coordinate verification (`99bc9c9`)
 - `docs/SEARCH-QUALITY.md`: the close record — metric table, per-category table, escalation
  lever section naming exactly one lever (**None — met on the first run; no
  abbreviations.ts row, no alias, no corpus edit**), corpus corrections (none, threshold
  still 0.9), reproduction commands, known limitations (hand-written corpus scope,
  refusal-not-routing in v1, the recorded fuzzy-tier scaling decision).
 - Coordinate verification (live Nominatim, identifying UA, 1 req/s, 10 seed entries):
  **8/10 within 0.01°.** `bukit-bintang` outlier = area-vs-point discrepancy (no change).
  `pavilion-kl` = real ~2 km coordinate error in the 04-01 seed → corrected to
  (3.149, 101.7132), post-fix delta 0.0002° (Rule 1 fix below).
 - Full sweep green: `search:test` 57/57, `search:typecheck`, `bot:test` 50/50, `web:test` 33/33.
 ## Commits
 | task | commit | message |
 | ---- | ------ | ------- |
 | 1 RED | `dea7971` | test(04-02): RED — golden-corpus D-14 gate suite (CO1-CO8) |
 | 1 GREEN | `b2689f7` | feat(04-02): pinned golden corpus + D-14 corpus runner + Nyquist gate |
 | 2 RED | `ec98c8d` | test(04-02): RED — D-06 topup law suite (TP1-TP7) + D-05 converter suite (CV1-CV3) |
 | 2 GREEN | `c6a5f2b` | feat(04-02): Overpass top-up proposer (D-06) + JSON-to-SQLite converter (D-05) |
 | 3 | `99bc9c9` | docs(04-02): close the D-14 bake-off — threshold met on the first run, lever None |
 ## Verification Results
 - `npm run search:corpus` — exit 0: `top-3 68/68 (100.0%)`, `negatives refused 7/7`,
  `refusal margin 0.643 < 0.72`, `threshold top3 >= 90.0% — PASS`
 - `npm run search:test` — **57 passed, 0 failed** (CO1-CO8 + TP1-TP7 + CV1-CV3 + the 39 from 04-01)
 - `npm run search:typecheck` — green
 - `npm test -w bot` (50) / `npm test -w web` (33) — green, untouched
 - `grep -rl 'overpass' bot/src/ web/src/` → 0; `grep -rl 'sqlite' bot/src/ web/src/` → 0
 - Overpass script proven offline: committed fixture → 6-candidate PROPOSAL, exit 0;
  busy-shaped fixture → exactly the soft-exit line, exit 0
 - Converter acceptance probe: id set preserved exactly (sqlite round-trip)
 ## Deviations from Plan
 **1. [Rule 1 - Bug] `--fixture=` silently hit the live network**
 - **Found during:** task 2 (GREEN probes)
 - **Issue:** the first manual probe passed `--fixture=...` without `--source=fixture`; the
  script fell through to a live Overpass request, so the busy-fixture acceptance check would
  have printed a live PROPOSAL instead of the soft-exit line.
 - **Fix:** presence of `--fixture` now implies the fixture source. (The accidental live run
  made exactly one UA'd request — LAW 2 was never violated — and usefully proved the live
  Overpass path works from this environment with the mandated User-Agent.)
 - **Files:** search/scripts/overpass-topup.mjs
 **2. [Rule 1 - Bug] TP3 failed — no literal bbox tuple in source**
 - **Found during:** task 2 (GREEN)
 - **Issue:** the query interpolated `(${bbox})`, so the source contained no literal
  parenthesized bbox tuple for TP3's regex.
 - **Fix:** the canonical default query is now pinned verbatim —
  `nwr["tourism"="attraction"](3.05,101.55,3.25,101.80)` — with flag substitutions building
  the same shape for overrides.
 - **Files:** search/scripts/overpass-topup.mjs
 **3. [Rule 1 - Bug] pavilion-kl committed coordinate ~2 km west of the mall**
 - **Found during:** task 3 (the plan's prescribed Nominatim spot-verification)
 - **Issue:** seed entry carried lng 101.695; Nominatim (and the mall's true position) place
  Pavilion Kuala Lumpur at ~(3.1492, 101.7130).
 - **Fix:** entry corrected to (3.149, 101.7132); recorded in SEARCH-QUALITY.md under
  "Coordinate verification". Corpus is coordinate-blind; all suites stayed green.
 - **Files:** search/src/data/kl-gazetteer.json, docs/SEARCH-QUALITY.md
 **4. [Planner-note] corpus illustrative strings resolved against the real gazetteer**
 - The plan marks its query strings ILLUSTRATIVE and requires pinning to existing ids. Targets
  with no gazetteer entry (The Intermark, Cheras South) and queries the scorer must refuse by
  design ("MV" — 2 chars, below the min-length guard) were replaced with equivalent pinned
  cases (IOI, Pudu, Tmn Perdana, Kl Towr, Merdeaka 118, ...). Floors all exceeded.
 ## Auth Gates
 None — no secrets, no auth flows. The only network surfaces are maintainer-time repo tools
 (Overpass/Nominatim/Nominatim-verify), all of which worked from this environment with
 identifying User-Agents.
 ## Known Stubs
 None in the shipped-search sense — every artifact is wired to real behavior. The plan's
 "stub" language for the converter means backend #2 itself is deliberately unbuilt (D-05,
 post-v1); the converter that keeps its format reachable is a full, working implementation.
 ## Threat Flags
 None — the built surface is exactly the plan's threat model (T-04-12…T-04-20): the corpus
 tampering gates (CO3/CO6 + the 0.9 threshold grep), the Overpass runtime-dependency and
 never-writes laws (TP1/TP6/TP7), the busy-soft-exit (TP4), and the ODbL notice traveling
 with converter output (T-04-20, embedded in the metadata table).
 ## TDD Gate Compliance
 PASSED — task 1: RED `dea7971` (8 CO cases failing on missing corpus) → GREEN `b2689f7`;
 task 2: RED `ec98c8d` (8 TP/CV cases failing on missing scripts) → GREEN `c6a5f2b`. Task 3
 is a non-TDD measurement/record task (no RED/GREEN applies). RED failures were verified
 before each GREEN commit.
 ## Self-Check: PASSED