From 99bc9c9a0914a931044cfd42e4c56d4f472644d5 Mon Sep 17 00:00:00 2001 From: araharah-executor Date: Sat, 24 Oct 2026 01:03:39 +0800 Subject: [PATCH] =?UTF-8?q?docs(04-02):=20close=20the=20D-14=20bake-off=20?= =?UTF-8?q?=E2=80=94=20threshold=20met=20on=20the=20first=20run,=20lever?= =?UTF-8?q?=20None?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/SEARCH-QUALITY.md: the bake-off close record — corpus v1 (75 cases) x gazetteer v1 (30 entries), measured 2026-10-23T17:02Z via npm run search:corpus - top-1 66/68 (97.1%), top-3 68/68 (100.0%) >= 90% threshold - negatives refused 7/7, refusal margin 0.643 < 0.72 admission line - per-category table (landmark/abbreviation/glued/typo/alias all 100%) - escalation lever: None — met on first run; no abbreviations.ts row, no alias, no corpus edit; the fixed ladder was never climbed - coordinate verification: 10 seed entries spot-checked against Nominatim (identifying UA, 1 req/s): 8/10 within 0.01°; bukit-bintang outlier is an area-vs-point discrepancy (no change); pavilion-kl was a real ~2 km error - both top-1 misses documented as expected disjoint-tier behaviour - kl-gazetteer.json: pavilion-kl coordinate corrected (3.1478,101.695) -> (3.149,101.7132) — Rule 1 data bug found by the prescribed Nominatim verification; post-fix delta 0.0002°; corpus is coordinate-blind, all suites green --- docs/SEARCH-QUALITY.md | 99 +++++++++++++++++++++++++++++++ search/src/data/kl-gazetteer.json | 4 +- 2 files changed, 101 insertions(+), 2 deletions(-) create mode 100644 docs/SEARCH-QUALITY.md diff --git a/docs/SEARCH-QUALITY.md b/docs/SEARCH-QUALITY.md new file mode 100644 index 0000000..1ef99e2 --- /dev/null +++ b/docs/SEARCH-QUALITY.md @@ -0,0 +1,99 @@ +# Search quality — Phase 4 bake-off close + +**Corpus:** `search/test/fixtures/golden-corpus.json` version 1 — 75 hand-written cases +(68 positive / 7 negative) +**Gazetteer:** `search/src/data/kl-gazetteer.json` version 1 — 30 entries +**Measured:** 2026-10-23T17:02Z via `npm run search:corpus` + +| Metric | Value | Threshold | +|---|---|---| +| top-1 hit rate (positives) | 66/68 (97.1%) | — (reported) | +| top-3 hit rate (positives) | 68/68 (100.0%) | >= 90% | +| negatives refused | 7/7 | 100% (zero leaks) | +| refusal margin | best negative sim 0.643 < 0.72 | every negative strictly below `FUZZY_ADMIT_SIMILARITY` | +| cost | 1.07 ms/query | 10 ms Workers Free CPU budget | + +### Per-category + +| Category | Cases | top-3 | +|---|---|---| +| landmark | 17 | 17/17 (100.0%) | +| abbreviation | 15 | 15/15 (100.0%) | +| glued | 7 | 7/7 (100.0%) | +| typo | 14 | 14/14 (100.0%) | +| alias | 15 | 15/15 (100.0%) | +| negative | 7 | refused 7/7 (100.0%) | + +The gate closes mechanically: `npm run search:corpus` exits 0 only when top-3 >= the +corpus's pinned `top3_hit_rate` (0.9) AND every negative is refused; the same measurements +are asserted in-process by `search/test/corpus.test.mjs` (CO1–CO8) inside +`npm run search:test`, so a regression fails the suite, not just the standalone runner. + +### Escalation lever pulled (D-14) + +1. **None — the threshold was met on the first run.** No alias/normalizer table edits were + needed: no rows were added to `search/src/rules/abbreviations.ts`, no aliases were added + to the gazetteer, and the corpus was not edited. The numbers above ARE the first-run + measurement. (The ladder — lever 1 table edits → lever 2 WASM geocoder → lever 3 + Photon-on-VPS — is printed by the runner whenever a future run fails; it was never + climbed this phase.) + +The two top-1 misses (both still top-3 hits) are expected scorer behaviour, not gaps: +"KCLC" lands in a three-way 0.45 fuzzy tie resolved id-ascending (suria-klcc third), and +"Sunway Vellocity" legitimately rides the PREFIX tier via the "Sunway" alias, which outranks +the fuzzy hit on the full name — the disjoint-tier design doing what it is specified to do. + +### Corpus corrections + +None — every case passed as pinned; no `expectId` was edited and `threshold.top3_hit_rate` +is still `0.9`. + +### Coordinate verification + +Method: one-shot maintainer-time check (not a test, never a runtime path). For 10 sampled +seed entries, `GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=, Kuala +Lumpur&limit=1` with the identifying User-Agent `araharah-coordinate-verify/1.0 +(+https://github.com/araharah)` (a missing UA is the 406 trap — the same discipline as the +Overpass script), comparing max(|Δlat|, |Δlng|) against a 0.01° tolerance. + +**Outcome: 8 of 10 sampled within 0.01°** (deltas 0.0001°–0.0047°). + +- `pavilion-kl` — OUT by 0.018°: the committed coordinate was wrong (lng 101.695, ~2 km + west of the mall). Nominatim places Pavilion Kuala Lumpur at (3.1492, 101.7130); the + entry was corrected to (3.149, 101.7132) — post-fix delta 0.0002°. This is a dataset + correction, not a corpus edit; the search corpus is coordinate-blind and all suites stayed + green. +- `bukit-bintang` — OUT by 0.017°: an area-vs-point discrepancy. "Bukit Bintang" is a + district; Nominatim's first hit is a district centroid, while the committed point is the + Jalan Bukit Bintang shopping strip itself. No change — the committed point is the + navigation-relevant landmark. + +This verified 10 of 30 seed entries; a full 30-entry pass is worth doing after the next +top-up lands (see Known limitations). Fallback invariant if Nominatim is unreachable at a +future re-check: the committed-bbox gate (G5, `search/test/gazetteer.test.mjs`) still +guarantees every entry sits inside the KL metro box. + +### Reproduce + +```sh +npm run search:corpus # the D-14 gate: exit 0 only on threshold + zero leaks +npm run search:test # CO1–CO8 assert the same measurements in-process +``` + +### Known limitations + +- The corpus is hand-written and small; it measures the documented local shorthand, not every + phrasing a KL driver might use. +- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1. +- The fuzzy tier performs a full-candidate OSA scan when a query has no cheap-tier hit (the + pure-typo path) — measured at ~8.2 ms/query against ~2 000 alias pairs at D-07's 500-entry + scale (RESEARCH §Verification A1), and at 1.07 ms/query against this 30-entry seed corpus, + inside the 10 ms Workers Free CPU limit. This is a RECORDED scaling decision, not an + oversight (RESEARCH directive #1): if a future top-up pushes the dataset toward D-07's + 500-entry ceiling and CPU approaches the limit, the documented next step is a cheap + blocking key (first-letter + length band) in front of the OSA tier — measured against this + corpus before it ships. +- The two top-1 misses show the id-ascending tie-break can put a same-tier neighbour first + on short fuzzy ties; top-3 (the D-14 gate) is unaffected. If top-1 quality ever matters + beyond this, the lever is a deterministic secondary signal (e.g. category boost), not a + threshold change. diff --git a/search/src/data/kl-gazetteer.json b/search/src/data/kl-gazetteer.json index 13fd458..e1bd800 100644 --- a/search/src/data/kl-gazetteer.json +++ b/search/src/data/kl-gazetteer.json @@ -35,8 +35,8 @@ "id": "pavilion-kl", "name": "Pavilion Kuala Lumpur", "aliases": ["Pavilion Kuala Lumpur", "Pavilion", "Pavilion KL", "KL Pavilion"], - "lat": 3.1478, - "lng": 101.695, + "lat": 3.149, + "lng": 101.7132, "category": "mall", "address": "Jalan Bukit Bintang, 55100 Kuala Lumpur" },