docs(04-02): close the D-14 bake-off — threshold met on the first run, lever None
- docs/SEARCH-QUALITY.md: the bake-off close record — corpus v1 (75 cases) x
gazetteer v1 (30 entries), measured 2026-10-23T17:02Z via npm run search:corpus
- top-1 66/68 (97.1%), top-3 68/68 (100.0%) >= 90% threshold
- negatives refused 7/7, refusal margin 0.643 < 0.72 admission line
- per-category table (landmark/abbreviation/glued/typo/alias all 100%)
- escalation lever: None — met on first run; no abbreviations.ts row, no
alias, no corpus edit; the fixed ladder was never climbed
- coordinate verification: 10 seed entries spot-checked against Nominatim
(identifying UA, 1 req/s): 8/10 within 0.01°; bukit-bintang outlier is an
area-vs-point discrepancy (no change); pavilion-kl was a real ~2 km error
- both top-1 misses documented as expected disjoint-tier behaviour
- kl-gazetteer.json: pavilion-kl coordinate corrected (3.1478,101.695) ->
(3.149,101.7132) — Rule 1 data bug found by the prescribed Nominatim
verification; post-fix delta 0.0002°; corpus is coordinate-blind, all suites green
This commit is contained in:
araharah-executor
@@ -0,0 +1,99 @@
|
||||
# Search quality — Phase 4 bake-off close
|
||||
|
||||
**Corpus:** `search/test/fixtures/golden-corpus.json` version 1 — 75 hand-written cases
|
||||
(68 positive / 7 negative)
|
||||
**Gazetteer:** `search/src/data/kl-gazetteer.json` version 1 — 30 entries
|
||||
**Measured:** 2026-10-23T17:02Z via `npm run search:corpus`
|
||||
|
||||
| Metric | Value | Threshold |
|
||||
|---|---|---|
|
||||
| top-1 hit rate (positives) | 66/68 (97.1%) | — (reported) |
|
||||
| top-3 hit rate (positives) | 68/68 (100.0%) | >= 90% |
|
||||
| negatives refused | 7/7 | 100% (zero leaks) |
|
||||
| refusal margin | best negative sim 0.643 < 0.72 | every negative strictly below `FUZZY_ADMIT_SIMILARITY` |
|
||||
| cost | 1.07 ms/query | 10 ms Workers Free CPU budget |
|
||||
|
||||
### Per-category
|
||||
|
||||
| Category | Cases | top-3 |
|
||||
|---|---|---|
|
||||
| landmark | 17 | 17/17 (100.0%) |
|
||||
| abbreviation | 15 | 15/15 (100.0%) |
|
||||
| glued | 7 | 7/7 (100.0%) |
|
||||
| typo | 14 | 14/14 (100.0%) |
|
||||
| alias | 15 | 15/15 (100.0%) |
|
||||
| negative | 7 | refused 7/7 (100.0%) |
|
||||
|
||||
The gate closes mechanically: `npm run search:corpus` exits 0 only when top-3 >= the
|
||||
corpus's pinned `top3_hit_rate` (0.9) AND every negative is refused; the same measurements
|
||||
are asserted in-process by `search/test/corpus.test.mjs` (CO1–CO8) inside
|
||||
`npm run search:test`, so a regression fails the suite, not just the standalone runner.
|
||||
|
||||
### Escalation lever pulled (D-14)
|
||||
|
||||
1. **None — the threshold was met on the first run.** No alias/normalizer table edits were
|
||||
needed: no rows were added to `search/src/rules/abbreviations.ts`, no aliases were added
|
||||
to the gazetteer, and the corpus was not edited. The numbers above ARE the first-run
|
||||
measurement. (The ladder — lever 1 table edits → lever 2 WASM geocoder → lever 3
|
||||
Photon-on-VPS — is printed by the runner whenever a future run fails; it was never
|
||||
climbed this phase.)
|
||||
|
||||
The two top-1 misses (both still top-3 hits) are expected scorer behaviour, not gaps:
|
||||
"KCLC" lands in a three-way 0.45 fuzzy tie resolved id-ascending (suria-klcc third), and
|
||||
"Sunway Vellocity" legitimately rides the PREFIX tier via the "Sunway" alias, which outranks
|
||||
the fuzzy hit on the full name — the disjoint-tier design doing what it is specified to do.
|
||||
|
||||
### Corpus corrections
|
||||
|
||||
None — every case passed as pinned; no `expectId` was edited and `threshold.top3_hit_rate`
|
||||
is still `0.9`.
|
||||
|
||||
### Coordinate verification
|
||||
|
||||
Method: one-shot maintainer-time check (not a test, never a runtime path). For 10 sampled
|
||||
seed entries, `GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=<name>, Kuala
|
||||
Lumpur&limit=1` with the identifying User-Agent `araharah-coordinate-verify/1.0
|
||||
(+https://github.com/araharah)` (a missing UA is the 406 trap — the same discipline as the
|
||||
Overpass script), comparing max(|Δlat|, |Δlng|) against a 0.01° tolerance.
|
||||
|
||||
**Outcome: 8 of 10 sampled within 0.01°** (deltas 0.0001°–0.0047°).
|
||||
|
||||
- `pavilion-kl` — OUT by 0.018°: the committed coordinate was wrong (lng 101.695, ~2 km
|
||||
west of the mall). Nominatim places Pavilion Kuala Lumpur at (3.1492, 101.7130); the
|
||||
entry was corrected to (3.149, 101.7132) — post-fix delta 0.0002°. This is a dataset
|
||||
correction, not a corpus edit; the search corpus is coordinate-blind and all suites stayed
|
||||
green.
|
||||
- `bukit-bintang` — OUT by 0.017°: an area-vs-point discrepancy. "Bukit Bintang" is a
|
||||
district; Nominatim's first hit is a district centroid, while the committed point is the
|
||||
Jalan Bukit Bintang shopping strip itself. No change — the committed point is the
|
||||
navigation-relevant landmark.
|
||||
|
||||
This verified 10 of 30 seed entries; a full 30-entry pass is worth doing after the next
|
||||
top-up lands (see Known limitations). Fallback invariant if Nominatim is unreachable at a
|
||||
future re-check: the committed-bbox gate (G5, `search/test/gazetteer.test.mjs`) still
|
||||
guarantees every entry sits inside the KL metro box.
|
||||
|
||||
### Reproduce
|
||||
|
||||
```sh
|
||||
npm run search:corpus # the D-14 gate: exit 0 only on threshold + zero leaks
|
||||
npm run search:test # CO1–CO8 assert the same measurements in-process
|
||||
```
|
||||
|
||||
### Known limitations
|
||||
|
||||
- The corpus is hand-written and small; it measures the documented local shorthand, not every
|
||||
phrasing a KL driver might use.
|
||||
- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1.
|
||||
- The fuzzy tier performs a full-candidate OSA scan when a query has no cheap-tier hit (the
|
||||
pure-typo path) — measured at ~8.2 ms/query against ~2 000 alias pairs at D-07's 500-entry
|
||||
scale (RESEARCH §Verification A1), and at 1.07 ms/query against this 30-entry seed corpus,
|
||||
inside the 10 ms Workers Free CPU limit. This is a RECORDED scaling decision, not an
|
||||
oversight (RESEARCH directive #1): if a future top-up pushes the dataset toward D-07's
|
||||
500-entry ceiling and CPU approaches the limit, the documented next step is a cheap
|
||||
blocking key (first-letter + length band) in front of the OSA tier — measured against this
|
||||
corpus before it ships.
|
||||
- The two top-1 misses show the id-ascending tie-break can put a same-tier neighbour first
|
||||
on short fuzzy ties; top-3 (the D-14 gate) is unaffected. If top-1 quality ever matters
|
||||
beyond this, the lever is a deterministic secondary signal (e.g. category boost), not a
|
||||
threshold change.
|
||||
@@ -35,8 +35,8 @@
|
||||
"id": "pavilion-kl",
|
||||
"name": "Pavilion Kuala Lumpur",
|
||||
"aliases": ["Pavilion Kuala Lumpur", "Pavilion", "Pavilion KL", "KL Pavilion"],
|
||||
"lat": 3.1478,
|
||||
"lng": 101.695,
|
||||
"lat": 3.149,
|
||||
"lng": 101.7132,
|
||||
"category": "mall",
|
||||
"address": "Jalan Bukit Bintang, 55100 Kuala Lumpur"
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user