araharah-executor
99bc9c9a09
- docs/SEARCH-QUALITY.md: the bake-off close record — corpus v1 (75 cases) x
gazetteer v1 (30 entries), measured 2026-10-23T17:02Z via npm run search:corpus
- top-1 66/68 (97.1%), top-3 68/68 (100.0%) >= 90% threshold
- negatives refused 7/7, refusal margin 0.643 < 0.72 admission line
- per-category table (landmark/abbreviation/glued/typo/alias all 100%)
- escalation lever: None — met on first run; no abbreviations.ts row, no
alias, no corpus edit; the fixed ladder was never climbed
- coordinate verification: 10 seed entries spot-checked against Nominatim
(identifying UA, 1 req/s): 8/10 within 0.01°; bukit-bintang outlier is an
area-vs-point discrepancy (no change); pavilion-kl was a real ~2 km error
- both top-1 misses documented as expected disjoint-tier behaviour
- kl-gazetteer.json: pavilion-kl coordinate corrected (3.1478,101.695) ->
(3.149,101.7132) — Rule 1 data bug found by the prescribed Nominatim
verification; post-fix delta 0.0002°; corpus is coordinate-blind, all suites green
100 lines
5.1 KiB
Markdown
100 lines
5.1 KiB
Markdown
# Search quality — Phase 4 bake-off close
|
||
|
||
**Corpus:** `search/test/fixtures/golden-corpus.json` version 1 — 75 hand-written cases
|
||
(68 positive / 7 negative)
|
||
**Gazetteer:** `search/src/data/kl-gazetteer.json` version 1 — 30 entries
|
||
**Measured:** 2026-10-23T17:02Z via `npm run search:corpus`
|
||
|
||
| Metric | Value | Threshold |
|
||
|---|---|---|
|
||
| top-1 hit rate (positives) | 66/68 (97.1%) | — (reported) |
|
||
| top-3 hit rate (positives) | 68/68 (100.0%) | >= 90% |
|
||
| negatives refused | 7/7 | 100% (zero leaks) |
|
||
| refusal margin | best negative sim 0.643 < 0.72 | every negative strictly below `FUZZY_ADMIT_SIMILARITY` |
|
||
| cost | 1.07 ms/query | 10 ms Workers Free CPU budget |
|
||
|
||
### Per-category
|
||
|
||
| Category | Cases | top-3 |
|
||
|---|---|---|
|
||
| landmark | 17 | 17/17 (100.0%) |
|
||
| abbreviation | 15 | 15/15 (100.0%) |
|
||
| glued | 7 | 7/7 (100.0%) |
|
||
| typo | 14 | 14/14 (100.0%) |
|
||
| alias | 15 | 15/15 (100.0%) |
|
||
| negative | 7 | refused 7/7 (100.0%) |
|
||
|
||
The gate closes mechanically: `npm run search:corpus` exits 0 only when top-3 >= the
|
||
corpus's pinned `top3_hit_rate` (0.9) AND every negative is refused; the same measurements
|
||
are asserted in-process by `search/test/corpus.test.mjs` (CO1–CO8) inside
|
||
`npm run search:test`, so a regression fails the suite, not just the standalone runner.
|
||
|
||
### Escalation lever pulled (D-14)
|
||
|
||
1. **None — the threshold was met on the first run.** No alias/normalizer table edits were
|
||
needed: no rows were added to `search/src/rules/abbreviations.ts`, no aliases were added
|
||
to the gazetteer, and the corpus was not edited. The numbers above ARE the first-run
|
||
measurement. (The ladder — lever 1 table edits → lever 2 WASM geocoder → lever 3
|
||
Photon-on-VPS — is printed by the runner whenever a future run fails; it was never
|
||
climbed this phase.)
|
||
|
||
The two top-1 misses (both still top-3 hits) are expected scorer behaviour, not gaps:
|
||
"KCLC" lands in a three-way 0.45 fuzzy tie resolved id-ascending (suria-klcc third), and
|
||
"Sunway Vellocity" legitimately rides the PREFIX tier via the "Sunway" alias, which outranks
|
||
the fuzzy hit on the full name — the disjoint-tier design doing what it is specified to do.
|
||
|
||
### Corpus corrections
|
||
|
||
None — every case passed as pinned; no `expectId` was edited and `threshold.top3_hit_rate`
|
||
is still `0.9`.
|
||
|
||
### Coordinate verification
|
||
|
||
Method: one-shot maintainer-time check (not a test, never a runtime path). For 10 sampled
|
||
seed entries, `GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=<name>, Kuala
|
||
Lumpur&limit=1` with the identifying User-Agent `araharah-coordinate-verify/1.0
|
||
(+https://github.com/araharah)` (a missing UA is the 406 trap — the same discipline as the
|
||
Overpass script), comparing max(|Δlat|, |Δlng|) against a 0.01° tolerance.
|
||
|
||
**Outcome: 8 of 10 sampled within 0.01°** (deltas 0.0001°–0.0047°).
|
||
|
||
- `pavilion-kl` — OUT by 0.018°: the committed coordinate was wrong (lng 101.695, ~2 km
|
||
west of the mall). Nominatim places Pavilion Kuala Lumpur at (3.1492, 101.7130); the
|
||
entry was corrected to (3.149, 101.7132) — post-fix delta 0.0002°. This is a dataset
|
||
correction, not a corpus edit; the search corpus is coordinate-blind and all suites stayed
|
||
green.
|
||
- `bukit-bintang` — OUT by 0.017°: an area-vs-point discrepancy. "Bukit Bintang" is a
|
||
district; Nominatim's first hit is a district centroid, while the committed point is the
|
||
Jalan Bukit Bintang shopping strip itself. No change — the committed point is the
|
||
navigation-relevant landmark.
|
||
|
||
This verified 10 of 30 seed entries; a full 30-entry pass is worth doing after the next
|
||
top-up lands (see Known limitations). Fallback invariant if Nominatim is unreachable at a
|
||
future re-check: the committed-bbox gate (G5, `search/test/gazetteer.test.mjs`) still
|
||
guarantees every entry sits inside the KL metro box.
|
||
|
||
### Reproduce
|
||
|
||
```sh
|
||
npm run search:corpus # the D-14 gate: exit 0 only on threshold + zero leaks
|
||
npm run search:test # CO1–CO8 assert the same measurements in-process
|
||
```
|
||
|
||
### Known limitations
|
||
|
||
- The corpus is hand-written and small; it measures the documented local shorthand, not every
|
||
phrasing a KL driver might use.
|
||
- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1.
|
||
- The fuzzy tier performs a full-candidate OSA scan when a query has no cheap-tier hit (the
|
||
pure-typo path) — measured at ~8.2 ms/query against ~2 000 alias pairs at D-07's 500-entry
|
||
scale (RESEARCH §Verification A1), and at 1.07 ms/query against this 30-entry seed corpus,
|
||
inside the 10 ms Workers Free CPU limit. This is a RECORDED scaling decision, not an
|
||
oversight (RESEARCH directive #1): if a future top-up pushes the dataset toward D-07's
|
||
500-entry ceiling and CPU approaches the limit, the documented next step is a cheap
|
||
blocking key (first-letter + length band) in front of the OSA tier — measured against this
|
||
corpus before it ships.
|
||
- The two top-1 misses show the id-ascending tie-break can put a same-tier neighbour first
|
||
on short fuzzy ties; top-3 (the D-14 gate) is unaffected. If top-1 quality ever matters
|
||
beyond this, the lever is a deterministic secondary signal (e.g. category boost), not a
|
||
threshold change.
|