arah2/.planning/phases/04-destination-search-kl-gazetteer/04-02-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, decisions, metrics

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	decisions	metrics
04-destination-search-kl-gazetteer	02	search	search golden-corpus d14-gate overpass sqlite nominatim tdd	plan 04-01 search workspace (contracts, normalizer, scorer, gazetteer, loader) Node 24 type-stripping (.mjs script imports ../src/index.ts) node:sqlite (built into Node 24.18.0 — converter output path)	D-13 golden corpus v1 — 75 pinned hand-written cases in 6 categories with real negatives D-14 gate — `npm run search:corpus` (exit 0 only on top-3 >= 0.9 + zero leaks) and the in-process CO1-CO8 gate inside `npm run search:test` D-06 tool — search/scripts/overpass-topup.mjs (one UA'd request, PROPOSAL output, soft-exit 0 on busy, --source=overpass|nominatim|fixture) D-05 tool — search/scripts/json-to-sqlite.mjs (same id set, real .sqlite via node:sqlite, ODbL notice embedded) docs/SEARCH-QUALITY.md — the bake-off close: measured rates, lever None, coordinate verification	plan 04-03 (bot /cari handler ships against a closed, green corpus gate) plan 04-04 (web deep-link reader — same module, corpus unchanged) future top-ups (the Overpass proposer + SEARCH-QUALITY record define the maintenance loop)	added patterns node:sqlite DatabaseSync (converter only — repo tool, zero added deps) measure-once/report-twice: the runner exports measureCorpus(); the suite asserts the same numbers in-process main-guarded .mjs scripts (CLI when invoked, importable library otherwise) source-introspection law gates (TP1/TP6/TP7, CV2/CV3) following bot/test/privacy.test.mjs	created modified search/test/fixtures/golden-corpus.json search/scripts/gazetteer-corpus.mjs search/test/corpus.test.mjs search/scripts/overpass-topup.mjs search/test/fixtures/overpass-response.json search/scripts/json-to-sqlite.mjs search/test/topup.test.mjs search/test/converter.test.mjs docs/SEARCH-QUALITY.md search/package.json (corpus delegate) package.json (search:corpus delegate) search/src/data/kl-gazetteer.json (pavilion-kl coordinate fix)	Corpus authored to the shipped 30-entry gazetteer per the plan's ILLUSTRATIVE clause: dropped non-existent targets (Intermark, Cheras South, MV — 2-char queries sit below the scorer's min-length guard by design) and pinned equivalents that exist (IOI, Pudu, Tmn Perdana, Kl Towr, ...) Near-miss negatives chosen with measured margins: Penang 0.625 and Kampung Cina ~0.58, both in [0.55, 0.72) — lowering the admission line leaks them and fails CO2+CO8 measureCorpus() exported from the runner and reused by corpus.test.mjs — one measurement, two reporters, no drift --fixture=<path> implies --source=fixture (a fixture flag silently hitting the live network would be a trap) Converter emits a REAL .sqlite via node:sqlite (probed available in Node 24.18.0) — no dependency added; ODbL notice embedded in a metadata table + NOTICE sidecar	duration completed tasks filesCreated filesModified ~14 min executor time 2026-10-23 3 9 3

Phase 4 Plan 2: Golden Corpus + Bake-off Close Summary

One-liner: the D-13/D-14 experiment closed mechanically — a 75-case pinned corpus measured top-3 68/68 (100%) with 7/7 negatives refused at margin 0.643 < 0.72 on the FIRST run (lever: None), enforced twice (npm run search:corpus exit-code gate + CO1-CO8 inside npm run search:test), alongside the two forkability tools: an Overpass top-up proposer (one UA'd request, PROPOSAL-only output, exit-0-on-busy) and a JSON→SQLite converter that preserves the id set exactly.

What Was Built

Task 1 — pinned golden corpus + D-14 runner + Nyquist gate (RED dea7971 → GREEN b2689f7)

• golden-corpus.json: 75 hand-written cases — landmark 17, abbreviation 15, glued 7, typo 14, alias 15, negative 7 — every positive pinning a real gazetteer id (start_param-safe charset), threshold.top3_hit_rate: 0.9. 16 non-negative cases are not string-equal to any alias/name (tautology floor is 15). Negatives include two MEASURED near-misses (Penang 0.625, Kampung Cina ~0.58) so a lowered admission line fails CO2+CO8.
• gazetteer-corpus.mjs: imports the SAME src/index.ts the bot uses (D-03), measures top-1/top-3 + per-category rates + avg ms/query + the refusal margin, prints MISSED/LEAKED blocks, the threshold line, and the verbatim escalation ladder on failure; exits 0 only on threshold + zero leaks. Exports measureCorpus(); main-guarded so the suite can import it.
• corpus.test.mjs: CO1-CO8 — threshold, zero leaks, tautology floor, dangling-id charset, size+category floors, per-category reporting, and the pinned refusal margin (replicating the scorer's two-level whole-string+token fuzzy shape over damerau).
• Delegates: search:corpus (root), corpus (search workspace).
• Measured: top-3 68/68 (100.0%), negatives refused 7/7, margin 0.643 < 0.72, 1.07 ms/query — threshold met on the first run. The two top-1 misses ("KCLC" 3-way fuzzy tie broken id-ascending; "Sunway Vellocity" legitimately riding PREFIX via the "Sunway" alias) are documented expected behaviour.

Task 2 — Overpass top-up (D-06) + JSON→SQLite converter (D-05) (RED ec98c8d → GREEN c6a5f2b)

• overpass-topup.mjs: maintainer-time repo tool under the six hard laws — never a runtime dependency (TP1), ONE request per run via a single shared requestJson helper for both the Overpass and Nominatim paths (TP7), mandatory identifying User-Agent (TP2 — live-verified 406-without), bbox-bounded single-family query pinned verbatim (TP3), busy/502/429/timeout/ unparseable → Overpass instance busy — retry later + exit 0 (TP4), PROPOSAL-only output whose source never names the curated file (TP6). Flags: --source=overpass|nominatim|fixture, --endpoint=/OVERPASS_ENDPOINT, --bbox=, --q=, --tag-family=, --fixture=, --out=. --fixture implies the fixture source (offline by default when a fixture is named).
• overpass-response.json: committed real-shape Overpass response (6 KL elements — nodes with lat/lon, ways/relation with center) for deterministic offline tests.
• json-to-sqlite.mjs: reads the same committed kl-gazetteer.json, emits kl-gazetteer.sqlite (places + metadata tables) via node:sqlite — zero added dependencies — with the SAME id set (CV1 reads it back through DatabaseSync) and the ODbL notice embedded in the metadata table + a NOTICE sidecar (ODbL §4.4, T-04-20). Deterministic: dataset order, no timestamps.
• Suites: TP1-TP7 + CV1-CV3 all green; grep gates confirm zero overpass/sqlite mentions under bot/src/ and web/src/.

Task 3 — bake-off close + coordinate verification (99bc9c9)

• docs/SEARCH-QUALITY.md: the close record — metric table, per-category table, escalation lever section naming exactly one lever (None — met on the first run; no abbreviations.ts row, no alias, no corpus edit), corpus corrections (none, threshold still 0.9), reproduction commands, known limitations (hand-written corpus scope, refusal-not-routing in v1, the recorded fuzzy-tier scaling decision).
• Coordinate verification (live Nominatim, identifying UA, 1 req/s, 10 seed entries): 8/10 within 0.01°. bukit-bintang outlier = area-vs-point discrepancy (no change). pavilion-kl = real ~2 km coordinate error in the 04-01 seed → corrected to (3.149, 101.7132), post-fix delta 0.0002° (Rule 1 fix below).
• Full sweep green: search:test 57/57, search:typecheck, bot:test 50/50, web:test 33/33.

Commits

task	commit	message
1 RED	dea7971	test(04-02): RED — golden-corpus D-14 gate suite (CO1-CO8)
1 GREEN	b2689f7	feat(04-02): pinned golden corpus + D-14 corpus runner + Nyquist gate
2 RED	ec98c8d	test(04-02): RED — D-06 topup law suite (TP1-TP7) + D-05 converter suite (CV1-CV3)
2 GREEN	c6a5f2b	feat(04-02): Overpass top-up proposer (D-06) + JSON-to-SQLite converter (D-05)
3	99bc9c9	docs(04-02): close the D-14 bake-off — threshold met on the first run, lever None

Verification Results

• npm run search:corpus — exit 0: top-3 68/68 (100.0%), negatives refused 7/7, refusal margin 0.643 < 0.72, threshold top3 >= 90.0% — PASS
• npm run search:test — 57 passed, 0 failed (CO1-CO8 + TP1-TP7 + CV1-CV3 + the 39 from 04-01)
• npm run search:typecheck — green
• npm test -w bot (50) / npm test -w web (33) — green, untouched
• grep -rl 'overpass' bot/src/ web/src/ → 0; grep -rl 'sqlite' bot/src/ web/src/ → 0
• Overpass script proven offline: committed fixture → 6-candidate PROPOSAL, exit 0; busy-shaped fixture → exactly the soft-exit line, exit 0
• Converter acceptance probe: id set preserved exactly (sqlite round-trip)

Deviations from Plan

1. [Rule 1 - Bug] --fixture= silently hit the live network

• Found during: task 2 (GREEN probes)
• Issue: the first manual probe passed --fixture=... without --source=fixture; the script fell through to a live Overpass request, so the busy-fixture acceptance check would have printed a live PROPOSAL instead of the soft-exit line.
• Fix: presence of --fixture now implies the fixture source. (The accidental live run made exactly one UA'd request — LAW 2 was never violated — and usefully proved the live Overpass path works from this environment with the mandated User-Agent.)
• Files: search/scripts/overpass-topup.mjs

2. [Rule 1 - Bug] TP3 failed — no literal bbox tuple in source

• Found during: task 2 (GREEN)
• Issue: the query interpolated (${bbox}), so the source contained no literal parenthesized bbox tuple for TP3's regex.
• Fix: the canonical default query is now pinned verbatim — nwr["tourism"="attraction"](3.05,101.55,3.25,101.80) — with flag substitutions building the same shape for overrides.
• Files: search/scripts/overpass-topup.mjs

3. [Rule 1 - Bug] pavilion-kl committed coordinate ~2 km west of the mall

• Found during: task 3 (the plan's prescribed Nominatim spot-verification)
• Issue: seed entry carried lng 101.695; Nominatim (and the mall's true position) place Pavilion Kuala Lumpur at ~(3.1492, 101.7130).
• Fix: entry corrected to (3.149, 101.7132); recorded in SEARCH-QUALITY.md under "Coordinate verification". Corpus is coordinate-blind; all suites stayed green.
• Files: search/src/data/kl-gazetteer.json, docs/SEARCH-QUALITY.md

4. [Planner-note] corpus illustrative strings resolved against the real gazetteer

• The plan marks its query strings ILLUSTRATIVE and requires pinning to existing ids. Targets with no gazetteer entry (The Intermark, Cheras South) and queries the scorer must refuse by design ("MV" — 2 chars, below the min-length guard) were replaced with equivalent pinned cases (IOI, Pudu, Tmn Perdana, Kl Towr, Merdeaka 118, ...). Floors all exceeded.

Auth Gates

None — no secrets, no auth flows. The only network surfaces are maintainer-time repo tools (Overpass/Nominatim/Nominatim-verify), all of which worked from this environment with identifying User-Agents.

Known Stubs

None in the shipped-search sense — every artifact is wired to real behavior. The plan's "stub" language for the converter means backend #2 itself is deliberately unbuilt (D-05, post-v1); the converter that keeps its format reachable is a full, working implementation.

Threat Flags

None — the built surface is exactly the plan's threat model (T-04-12…T-04-20): the corpus tampering gates (CO3/CO6 + the 0.9 threshold grep), the Overpass runtime-dependency and never-writes laws (TP1/TP6/TP7), the busy-soft-exit (TP4), and the ODbL notice traveling with converter output (T-04-20, embedded in the metadata table).

TDD Gate Compliance

PASSED — task 1: RED dea7971 (8 CO cases failing on missing corpus) → GREEN b2689f7; task 2: RED ec98c8d (8 TP/CV cases failing on missing scripts) → GREEN c6a5f2b. Task 3 is a non-TDD measurement/record task (no RED/GREEN applies). RED failures were verified before each GREEN commit.

Self-Check: PASSED