docs(04): plan revision 1 — fix wave-1 bundle proof + acceptance-criteria corrections

.planning/phases/04-destination-search-kl-gazetteer/04-02-PLAN.md

@@ -31,7 +31,7 @@ must_haves:
   truths:
     - "The golden corpus contains at least 60 hand-written cases across 6 categories: golden landmarks, abbreviations, glued words, typos, aliases, and deliberate out-of-KL negatives"
     - "At least 15 corpus queries are NOT string-equal to any alias in the gazetteer — the corpus measures something real, it is not tautological"
-    - "The corpus runner exits 0 only when top-3 >= 90% on positives AND every negative returned an empty shortlist"
+    - "The corpus runner exits 0 only when top-3 >= 90% on positives AND every negative returned an empty shortlist with its best fuzzy similarity below the 0.72 admission line (the margin is asserted, not hoped for)"
     - "The corpus runner reports per-category rates so a regression is diagnosable, not a diffuse number"
     - "The Overpass top-up script produces a valid-shaped proposal file from a COMMITTED FIXTURE, and treats 502/429/timeout as a normal retryable outcome that EXITS 0"
     - "The Overpass script sends an identifying User-Agent (a request without one gets HTTP 406) and makes exactly ONE request per run"
@@ -162,11 +162,11 @@ Rules:
 - `landmark` — >= 12 (the golden landmarks and other well-known KL places: "KLCC", "Sunway Pyramid", "DUKE", "Pavilion", "Mid Valley", "Berjaya Times Square", "IOI City Mall", "Merdeka 118", "The Intermark", "KL Sentral", "Nu Sentral", "PWTC", "IJN", "Taman Bangsar")
 - `abbreviation` — >= 15 ("Jln Ampang", "Jalan Ampang", "Tmn Bangsar", "Taman Bangsar", "Kg Ampang", "Kampung Ampang", "Bdr Baru", "Bandar Baru", "Lbh Ampang", "Lebuh Ampang", "Psrn KL", "Persiaran KL", "Sg Wang", "Sungai Wang", "Blk A")
 - `glued` — >= 6 ("tamanbangsar", "jalankanpar", "kampungbarra", "bandarbaru", "sungaiwang", "bukitbintang")
-- `typo` — >= 12 ("KCLC", "KLCC" transposed, "Sunwya Pyarmid", "Pavillion", "Mid Vally", "Berjaya Time Square", "IJN" → "IJNP", "Sentral" → "Sentrl", "Merdeka 118" → "Merdeka 118", "Tmn Bangsr", "Pavilon", "Intermrk")
+- `typo` — >= 12 ("KCLC", "KLCC" transposed, "Sunwya Pyarmid", "Pavillion", "Mid Vally", "Berjaya Time Square", "IJN" → "IJNP", "Sentral" → "Sentrl", "Tmn Bangsr", "Pavilon", "Intermrk", "Sunway Velocity" → "Sunway Vellocity") — every typo case is a NON-string-equal misspelling; the bucket resolves through plan 04-01's FUZZY residual tier (each illustrative's similarity is >= `FUZZY_ADMIT_SIMILARITY` 0.72), never through alias edits
 - `alias` — >= 12 ("IJN", "PWTC", "Stesen KL Sentral", "KL Central", "BTS", "MV", "IOI", "Sunway Velocity", "Velocity", "Cheras South", "Pudu", "Intermark")
-- `negative` — >= 6 ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") — these MUST be refused with an empty shortlist, never a fabricated pin
+- `negative` — >= 6 ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") — these MUST be refused with an empty shortlist, never a fabricated pin; the bucket MUST ALSO include at least one NEAR-MISS negative — a genuinely out-of-KL query whose best fuzzy similarity to some gazetteer alias lands in `[0.55, FUZZY_ADMIT_SIMILARITY)` (illustrative: "Kampung Cina" vs the `kampung-ampang` alias "Kampung Ampang" — the "kampung" token matches exactly, "cina" drags the token-level mean to ~0.6) — a query a LOWERED admission threshold would wrongly admit. CO8 pins this margin mechanically
 
-**TAUTOLOGY LAW (D-13 / RESEARCH Pitfall 7):** the corpus must contain queries that are NOT string-equal to any alias in the gazetteer. The typo and glued buckets are exactly those queries — and at least 15 cases overall must not equal any alias. The runner ASSERTS this (`cases.filter(c => c.kind !== 'negative' && !aliases.has(normalize(c.q))).length >= 15`), so a future regeneration that makes the corpus tautological fails the suite.
+**TAUTOLOGY LAW (D-13 / RESEARCH Pitfall 7):** the corpus must contain queries that are NOT string-equal to any alias in the gazetteer. The typo and glued buckets are exactly those queries — and at least 15 cases overall must not equal any alias. The runner ASSERTS this (`cases.filter(c => c.kind !== 'negative' && !aliases.has(normalize(c.q))).length >= 15`), so a future regeneration that makes the corpus tautological fails the suite. JOINT ARITHMETIC with the D-14 escalation ladder: the typo bucket resolves through plan 04-01's FUZZY residual tier (`FUZZY_ADMIT_SIMILARITY` 0.72 × the 0.60 ceiling — "KCLC" sim 0.75 → score 0.45), so lever 1 is NEVER needed for a typo. Adding a typo string as an alias would both defeat the case's purpose and erode the tautology floor below CO3's 15 — reserve lever 1 for genuine alias gaps (a missing landmark, a missing local shorthand). With the typo bucket served by the residual net, the ~12 typo cases plus the alias-bucket queries that are not literal alias strings keep the non-tautological count comfortably above 15.
 
 **Every `expectId` MUST match `/^[A-Za-z0-9_-]{1,64}$/`** — the start_param-safe charset (RESEARCH: "The corpus runner should assert every corpus `expectId` matches `/^[A-Za-z0-9_-]{1,64}$/`").
 
@@ -183,9 +183,9 @@ Behaviour:
 4. For each positive: `search(c.q, 3)` → record whether `hits[0].entry.id === c.expectId` (top-1) and whether `hits.map(h => h.entry.id).includes(c.expectId)` (top-3). Record the miss as `"<query> -> [<ids or —>] (expected <id>)"`.
 5. For each negative: `search(c.q, 3)` → if `hits.length > 0`, record the leak as `"<query> -> <first id>"`.
 6. Print a report to stdout:
-   - the overall line: `top-1 <n>/<N>  top-3 <n>/<N> (<pct>%)`
+   - the overall line: `top-1 <n>/<N>  top-3 <n>/<N> (<pct>%)  <avg> ms/query` — the per-query average gives the 10 ms Workers CPU-budget claim a measuring hook at every run
    - a PER-CATEGORY breakdown: one line per `kind` with that category's top-3 rate
-   - `negatives refused <n>/<N>`
+   - `negatives refused <n>/<N>` — followed by the refusal-margin line: `refusal margin <best negative sim> < 0.72` (the highest fuzzy similarity any negative reached against any alias, and its gap to `FUZZY_ADMIT_SIMILARITY` — the margin that makes the refusals honest is VISIBLE, not accidental)
    - the `MISSED:` block (one line per miss) and the `LEAKED (must refuse):` block (one line per leak)
    - the threshold line: `threshold top3 >= 90.0% — PASS|FAIL`
 7. Exit code: `0` only when `top3Rate >= threshold.top3_hit_rate` AND `leaked.length === 0`; `1` otherwise.
@@ -228,7 +228,7 @@ and exits **0** (never a stack trace, never a non-zero code). Nothing in the rep
 
 **LAW 5 — it NEVER writes to `kl-gazetteer.json`.** Output is a PRINTED candidate list (or a proposal file at an explicit `--out` path that the maintainer diffs and merges by hand). The script writes nothing to the curated dataset under any flag. The word "proposal" must appear in the output header.
 
-**LAW 6 — the source and the endpoint are overridable.** `--source=overpass|nominatim|fixture` selects the candidate source (default `overpass`); `--endpoint=<url>` and the `OVERPASS_ENDPOINT` env var override the Overpass endpoint (default `https://overpass-api.de/api/interpreter`); `--fixture=<path>` names the fixture file for `--source=fixture`. `--source=nominatim --q=<term>` queries the reachable Nominatim search endpoint bounded to the KL viewbox instead (Nominatim was live-verified reachable 2026-10-23 while the Overpass mirrors were not). This is the resolved offline posture: the committed fixture is the deterministic test path, Nominatim is the alternative live path, and neither is ever a runtime dependency.
+**LAW 6 — the source and the endpoint are overridable.** `--source=overpass|nominatim|fixture` selects the candidate source (default `overpass`); `--endpoint=<url>` and the `OVERPASS_ENDPOINT` env var override the Overpass endpoint (default `https://overpass-api.de/api/interpreter`); `--fixture=<path>` names the fixture file for `--source=fixture`. `--source=nominatim --q=<term>` queries the reachable Nominatim search endpoint bounded to the KL viewbox instead (Nominatim was live-verified reachable 2026-10-23 while the Overpass mirrors were not). This is the resolved offline posture: the committed fixture is the deterministic test path, Nominatim is the alternative live path, and neither is ever a runtime dependency. **Both live paths (Overpass and Nominatim) MUST be served by ONE shared `fetch` helper**, so LAW 2's exactly-one-request invariant is structural — exactly ONE `fetch(` call site in the source, whichever source is selected (this is what TP7 asserts).
 
 **The query shape** — a NARROW, single-tag-family query bounded to the KL bbox (Overpass bbox order is `s,w,n,e` = south, west, north, east):
 
@@ -246,7 +246,7 @@ The bbox MUST be present (RESEARCH §Anti-Patterns: "An Overpass query with no b
 
 ```
 PROPOSAL — 7 candidate(s) from tourism=attraction in bbox 3.05,101.55,3.25,101.80
-Review and hand-merge into search/src/data/kl-gazetteer.json — this script NEVER writes it.
+Review and hand-merge into the curated dataset file — this script NEVER writes it.
   name=Merdeka 118           osm=way/12345678  lat=3.1418  lon=101.6943  category=landmark
   name=National Monument     osm=node/87654321  lat=3.1489  lon=101.6936  category=landmark
 ```
@@ -260,7 +260,7 @@ Review and hand-merge into search/src/data/kl-gazetteer.json — this script NEV
 - `TP4 the script exits 0 on a busy instance` — run the script with `--fixture=` pointing at a fixture whose content is NOT an Overpass response (e.g. a JSON object with `"error"`), assert the process exit code is 0 and stdout contains "Overpass instance busy".
 - `TP5 the script produces a valid-shaped proposal from the committed fixture` — run the script with `--fixture=search/test/fixtures/overpass-response.json`, assert exit 0 and that stdout contains "PROPOSAL", at least one `name=` line, and the literal text "NEVER writes".
 - `TP6 the script never writes the curated file` — assert the script source contains no `writeFileSync` targeting `kl-gazetteer.json` (the only permitted write is an explicit `--out` proposal path, and that path must not be the curated file — assert the source does not contain the literal `kl-gazetteer.json` at all).
-- `TP7 exactly one request per run` — assert the script source contains exactly ONE `fetch(` call.
+- `TP7 exactly one request per run` — the Overpass and Nominatim live paths (LAW 6) share ONE `fetch` helper, so the source contains exactly ONE `fetch(` call site regardless of the selected source; assert that count is exactly 1.
 
 **`search/scripts/json-to-sqlite.mjs`** — the D-05 backend-#2 format converter. It reads `search/src/data/kl-gazetteer.json` and emits a SQLite file with the SAME ids, so backend #2's dataset format is reachable today (D-05: "The JSON↔SQLite converter script keeps its dataset format reachable from day one"). It is a STUB in the sense that the SQLite WRITER is a minimal, dependency-free implementation (Node 24 has `node:sqlite` behind a flag — check whether `node:sqlite` is available in Node 24.18.0; if it is, use it; if not, emit a deterministic, documented, plain-text `.sql` DUMP file plus a JSON sidecar and say so in the header — do NOT add a dependency).
 
@@ -307,6 +307,13 @@ Review and hand-merge into search/src/data/kl-gazetteer.json — this script NEV
 - The corpus is hand-written and small; it measures the documented local shorthand, not every
   phrasing a KL driver might use.
 - Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1.
+- The fuzzy tier performs a full-candidate OSA scan when a query has no cheap-tier hit (the
+  pure-typo path) — measured at ~8.2 ms/query against ~2 000 alias pairs (RESEARCH §Verification
+  A1), inside the 10 ms Workers Free CPU limit at the seed scale. This is a RECORDED scaling
+  decision, not an oversight (RESEARCH directive #1): if a future top-up pushes the dataset toward
+  D-07's 500-entry ceiling and CPU approaches the limit, the documented next step is a cheap
+  blocking key (first-letter + length band) in front of the OSA tier — measured against this
+  corpus before it ships.
 ```
 
 **Acceptance:** `npm run search:corpus` exits 0 with top-3 >= 90% and zero negative leaks; `npm run search:test` green; `docs/SEARCH-QUALITY.md` records the measured numbers and the lever.
@@ -352,10 +359,11 @@ and to the root `package.json`:
 - `CO5 every expectId matches the start_param-safe charset` — `/^[A-Za-z0-9_-]{1,64}$/`.
 - `CO6 the corpus meets its size and category floors` — >= 60 cases; landmark >= 12, abbreviation >= 15, glued >= 6, typo >= 12, alias >= 12, negative >= 6.
 - `CO7 every category's top-3 rate is reported` — the suite prints (not asserts) the per-category rates so the runner output and the suite output agree.
+- `CO8 the refusal margin is pinned` — for every negative, compute its best fuzzy similarity against every gazetteer alias using the shared `damerau` and the SAME two-level (whole-string + token) shape the scorer uses; assert ALL negatives sit strictly below `FUZZY_ADMIT_SIMILARITY` (0.72), and assert at least one near-miss negative sits in `[0.55, 0.72)` (the corpus contract's near-miss requirement) — so lowering the admission threshold would leak a real negative and fail BOTH CO2 and CO8. The floor is pinned by test, not by hope.
 
 The suite imports the shared module and the corpus directly (in-process), so `npm run search:test` fails on a corpus regression.
 
-**Acceptance:** `npm run search:corpus` exits 0; `npm run search:test` green with `PASS CO1`…`PASS CO7`.
+**Acceptance:** `npm run search:corpus` exits 0; `npm run search:test` green with `PASS CO1`…`PASS CO8`.
   </action>
   <verify>
     <automated>cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -30 ; npm run search:test 2>&1 | grep -E 'PASS CO|runner' ; node -e "const c=require('./search/test/fixtures/golden-corpus.json'); if(c.cases.length<60) throw new Error('corpus too small: '+c.cases.length); console.log('corpus OK '+c.cases.length+' cases')"</automated>
@@ -364,7 +372,7 @@ The suite imports the shared module and the corpus directly (in-process), so `np
     - `npm run search:corpus` exits 0
     - `npm run search:corpus` output contains a `top-3` line with a percentage >= 90.0% and a `negatives refused` line showing all negatives refused
     - `npm run search:corpus` output contains one `PASS`/`FAIL` threshold line naming the 90.0% gate
-    - `npm run search:test` exits 0 and the output contains `PASS CO1` through `PASS CO7`
+    - `npm run search:test` exits 0 and the output contains `PASS CO1` through `PASS CO8`
     - `node -e "const c=require('./search/test/fixtures/golden-corpus.json'); const g=require('./search/src/data/kl-gazetteer.json'); const ids=new Set(g.entries.map(e=>e.id)); const kinds={}; let bad=0; for(const x of c.cases){ kinds[x.kind]=(kinds[x.kind]||0)+1; if(x.expectId!==null && !ids.has(x.expectId)) bad++; if(x.expectId!==null && !/^[A-Za-z0-9_-]{1,64}$/.test(x.expectId)) bad++; } if(c.cases.length<60) throw new Error('size'); if(bad) throw new Error('dangling ids'); console.log(JSON.stringify(kinds))"` exits 0 and prints all six `kind` keys with landmark>=12, abbreviation>=15, glued>=6, typo>=12, alias>=12, negative>=6
     - `grep -q '"search:corpus": "node search/scripts/gazetteer-corpus.mjs"' package.json` exits 0 AND `grep -q '"corpus": "node scripts/gazetteer-corpus.mjs"' search/package.json` exits 0
     - `grep -q 'D-14 threshold not met' search/scripts/gazetteer-corpus.mjs` exits 0 AND `grep -q 'backend #2' search/scripts/gazetteer-corpus.mjs` exits 0 AND `grep -q 'Photon-on-VPS' search/scripts/gazetteer-corpus.mjs` exits 0 (the fixed escalation ladder is printed verbatim on failure)
@@ -399,7 +407,7 @@ Header contract block (copy `set-webhook.mjs`'s shape — purpose, inputs, what
  * it succeeding. The Worker only ever reads the COMMITTED curated JSON.
  *
  * Output is a PROPOSAL — a printed candidate list (or a file at an explicit --out
- * path) that a human diffs and hand-merges into search/src/data/kl-gazetteer.json.
+ * path) that a human diffs and hand-merges into the curated dataset file.
  * THIS SCRIPT NEVER WRITES THE CURATED FILE.
  *
  * Exit codes: 0 on success AND on a busy instance (Overpass capacity saturation
@@ -459,7 +467,7 @@ and `process.exit(0)`.
     - `cd search && node scripts/overpass-topup.mjs --fixture=<a non-Overpass fixture>` exits 0 and its stdout contains exactly the line `Overpass instance busy — retry later`
     - `grep -q 'User-Agent: araharah-gazetteer-topup/1.0' search/scripts/overpass-topup.mjs` exits 0 (HTTP 406 without it — live-verified)
     - `grep -q '\-\-source' search/scripts/overpass-topup.mjs` exits 0 (the source flag exists: overpass | nominatim | fixture — the resolved-question posture)
-    - `grep -c 'await fetch(\|fetch(' search/scripts/overpass-topup.mjs | head -1` returns 1 (EXACTLY ONE request per run)
+    - `grep -c 'await fetch(\|fetch(' search/scripts/overpass-topup.mjs | head -1` returns 1 (EXACTLY ONE request per run — the Overpass and Nominatim paths share this single fetch helper, per LAW 6)
     - `grep -c 'kl-gazetteer.json' search/scripts/overpass-topup.mjs | head -1` returns 0 (the script never names the curated file — it cannot write it)
     - `grep -rl 'overpass' bot/src/ web/src/ 2>/dev/null | wc -l` returns 0 (the script is never a runtime dependency — RESEARCH Pitfall 6)
     - `grep -rl 'sqlite\|json-to-sqlite' bot/src/ web/src/ 2>/dev/null | wc -l` returns 0
@@ -470,7 +478,7 @@ and `process.exit(0)`.
 
 <task type="auto">
   <name>task 3: run the bake-off, close it, and record the decision</name>
-  <files>docs/SEARCH-QUALITY.md, search/test/fixtures/golden-corpus.json, search/src/data/kl-gazetteer.json, search/src/rules/abbreviations.ts</files>
+  <files>docs/SEARCH-QUALITY.md, search/test/fixtures/golden-corpus.json, search/src/data/kl-gazetteer.json, search/src/rules/abbreviations.ts, search/test/normalizer.test.mjs</files>
   <read_first>
     - `search/scripts/gazetteer-corpus.mjs` (task 1's output — the instrument that measures the bake-off)
     - `search/test/fixtures/golden-corpus.json` (task 1's output — the pinned corpus being measured)