araharah-executor
99bc9c9a09
- docs/SEARCH-QUALITY.md: the bake-off close record — corpus v1 (75 cases) x
gazetteer v1 (30 entries), measured 2026-10-23T17:02Z via npm run search:corpus
- top-1 66/68 (97.1%), top-3 68/68 (100.0%) >= 90% threshold
- negatives refused 7/7, refusal margin 0.643 < 0.72 admission line
- per-category table (landmark/abbreviation/glued/typo/alias all 100%)
- escalation lever: None — met on first run; no abbreviations.ts row, no
alias, no corpus edit; the fixed ladder was never climbed
- coordinate verification: 10 seed entries spot-checked against Nominatim
(identifying UA, 1 req/s): 8/10 within 0.01°; bukit-bintang outlier is an
area-vs-point discrepancy (no change); pavilion-kl was a real ~2 km error
- both top-1 misses documented as expected disjoint-tier behaviour
- kl-gazetteer.json: pavilion-kl coordinate corrected (3.1478,101.695) ->
(3.149,101.7132) — Rule 1 data bug found by the prescribed Nominatim
verification; post-fix delta 0.0002°; corpus is coordinate-blind, all suites green
5.1 KiB
Search quality — Phase 4 bake-off close
Corpus: search/test/fixtures/golden-corpus.json version 1 — 75 hand-written cases
(68 positive / 7 negative)
Gazetteer: search/src/data/kl-gazetteer.json version 1 — 30 entries
Measured: 2026-10-23T17:02Z via npm run search:corpus
| Metric | Value | Threshold |
|---|---|---|
| top-1 hit rate (positives) | 66/68 (97.1%) | — (reported) |
| top-3 hit rate (positives) | 68/68 (100.0%) | >= 90% |
| negatives refused | 7/7 | 100% (zero leaks) |
| refusal margin | best negative sim 0.643 < 0.72 | every negative strictly below FUZZY_ADMIT_SIMILARITY |
| cost | 1.07 ms/query | 10 ms Workers Free CPU budget |
Per-category
| Category | Cases | top-3 |
|---|---|---|
| landmark | 17 | 17/17 (100.0%) |
| abbreviation | 15 | 15/15 (100.0%) |
| glued | 7 | 7/7 (100.0%) |
| typo | 14 | 14/14 (100.0%) |
| alias | 15 | 15/15 (100.0%) |
| negative | 7 | refused 7/7 (100.0%) |
The gate closes mechanically: npm run search:corpus exits 0 only when top-3 >= the
corpus's pinned top3_hit_rate (0.9) AND every negative is refused; the same measurements
are asserted in-process by search/test/corpus.test.mjs (CO1–CO8) inside
npm run search:test, so a regression fails the suite, not just the standalone runner.
Escalation lever pulled (D-14)
- None — the threshold was met on the first run. No alias/normalizer table edits were
needed: no rows were added to
search/src/rules/abbreviations.ts, no aliases were added to the gazetteer, and the corpus was not edited. The numbers above ARE the first-run measurement. (The ladder — lever 1 table edits → lever 2 WASM geocoder → lever 3 Photon-on-VPS — is printed by the runner whenever a future run fails; it was never climbed this phase.)
The two top-1 misses (both still top-3 hits) are expected scorer behaviour, not gaps: "KCLC" lands in a three-way 0.45 fuzzy tie resolved id-ascending (suria-klcc third), and "Sunway Vellocity" legitimately rides the PREFIX tier via the "Sunway" alias, which outranks the fuzzy hit on the full name — the disjoint-tier design doing what it is specified to do.
Corpus corrections
None — every case passed as pinned; no expectId was edited and threshold.top3_hit_rate
is still 0.9.
Coordinate verification
Method: one-shot maintainer-time check (not a test, never a runtime path). For 10 sampled
seed entries, GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=<name>, Kuala Lumpur&limit=1 with the identifying User-Agent araharah-coordinate-verify/1.0 (+https://github.com/araharah) (a missing UA is the 406 trap — the same discipline as the
Overpass script), comparing max(|Δlat|, |Δlng|) against a 0.01° tolerance.
Outcome: 8 of 10 sampled within 0.01° (deltas 0.0001°–0.0047°).
pavilion-kl— OUT by 0.018°: the committed coordinate was wrong (lng 101.695, ~2 km west of the mall). Nominatim places Pavilion Kuala Lumpur at (3.1492, 101.7130); the entry was corrected to (3.149, 101.7132) — post-fix delta 0.0002°. This is a dataset correction, not a corpus edit; the search corpus is coordinate-blind and all suites stayed green.bukit-bintang— OUT by 0.017°: an area-vs-point discrepancy. "Bukit Bintang" is a district; Nominatim's first hit is a district centroid, while the committed point is the Jalan Bukit Bintang shopping strip itself. No change — the committed point is the navigation-relevant landmark.
This verified 10 of 30 seed entries; a full 30-entry pass is worth doing after the next
top-up lands (see Known limitations). Fallback invariant if Nominatim is unreachable at a
future re-check: the committed-bbox gate (G5, search/test/gazetteer.test.mjs) still
guarantees every entry sits inside the KL metro box.
Reproduce
npm run search:corpus # the D-14 gate: exit 0 only on threshold + zero leaks
npm run search:test # CO1–CO8 assert the same measurements in-process
Known limitations
- The corpus is hand-written and small; it measures the documented local shorthand, not every phrasing a KL driver might use.
- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1.
- The fuzzy tier performs a full-candidate OSA scan when a query has no cheap-tier hit (the pure-typo path) — measured at ~8.2 ms/query against ~2 000 alias pairs at D-07's 500-entry scale (RESEARCH §Verification A1), and at 1.07 ms/query against this 30-entry seed corpus, inside the 10 ms Workers Free CPU limit. This is a RECORDED scaling decision, not an oversight (RESEARCH directive #1): if a future top-up pushes the dataset toward D-07's 500-entry ceiling and CPU approaches the limit, the documented next step is a cheap blocking key (first-letter + length band) in front of the OSA tier — measured against this corpus before it ships.
- The two top-1 misses show the id-ascending tie-break can put a same-tier neighbour first on short fuzzy ties; top-3 (the D-14 gate) is unaffected. If top-1 quality ever matters beyond this, the lever is a deterministic secondary signal (e.g. category boost), not a threshold change.