54 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 04-destination-search-kl-gazetteer | 02 | execute | 2 |
|
|
true |
|
|
Purpose: D-14 says the phase ends with "a documented decision, not an open question" — the corpus must show top-3 ≥ 90% on positives with every negative refused, and the escalation lever that was pulled must be recorded. This plan builds the instrument that measures it AND the two repo tools the forkability law (D-04, D-06) demands: the Overpass script proposes top-up candidates for human approval (never a runtime dependency), and the JSON→SQLite converter keeps backend #2's dataset format reachable from day one (D-05).
Output: search/test/fixtures/golden-corpus.json (60+ hand-written cases), search/scripts/gazetteer-corpus.mjs (the D-14 runner), search/test/corpus.test.mjs (the Nyquist gate inside npm run search:test), search/scripts/overpass-topup.mjs + a committed Overpass fixture + search/test/topup.test.mjs, search/scripts/json-to-sqlite.mjs + search/test/converter.test.mjs, and docs/SEARCH-QUALITY.md.
<execution_context> @$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md @$HOME/.config/opencode/get-shit-done/templates/summary.md </execution_context>
@AGENTS.md @.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md @.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md @.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md @search/src/index.ts @search/src/types.ts @search/src/data/kl-gazetteer.json @bot/scripts/register-commands.mjs @bot/scripts/set-webhook.mjs @bot/test/privacy.test.mjs @bot/test/harness.mjs @bot/test/runner.mjs From `search/src/index.ts` (plan 04-01's output — the surface every script in this plan imports):export type { GazetteerEntry, SearchQuery, SearchHit, SearchSource } from "./types.ts";
export { normalize } from "./normalize.ts";
export { TIER } from "./score.ts";
export { damerau } from "./distance.ts";
export { getEntries, getEntryById, getIndex } from "./gazetteer.ts";
export { createGazetteerSource, search } from "./source.ts";
// search(text: string, limit?: number): SearchHit[] — synchronous convenience wrapper
// getEntryById(id: string): GazetteerEntry | null — honest null on unknown
// getEntries(): readonly GazetteerEntry[]
From search/src/types.ts:
export interface SearchHit {
entry: GazetteerEntry;
score: number;
matchedAlias: string;
}
The corpus runner and both scripts import from ../src/index.ts — Node 24 strips the types by default, so a .mjs script CAN import the .ts source directly (the repo's existing pattern; bot/test/commands.test.mjs line 27 does exactly this and passes).
<corpus_format_contract>
search/test/fixtures/golden-corpus.json — the pinned D-13 corpus. The schema carries a per-case kind so the runner can report per-category rates (RESEARCH Open Question 3: "measure and report per-category rates … so a regression is diagnosable").
{
"version": 1,
"note": "Hand-written and pinned (D-13). NOT generated from the gazetteer — a generated corpus is tautological. Reviewers extend this file by editing it.",
"threshold": { "top3_hit_rate": 0.9 },
"cases": [
{ "q": "KLCC", "expectId": "suria-klcc", "kind": "landmark" },
{ "q": "Jln Ampang", "expectId": "suria-klcc", "kind": "abbreviation" },
{ "q": "tamanbangsar", "expectId": "bangsar", "kind": "glued" },
{ "q": "KCLC", "expectId": "suria-klcc", "kind": "typo" },
{ "q": "IJN", "expectId": "ijnp", "kind": "alias" },
{ "q": "Kuching", "expectId": null, "kind": "negative" }
]
}
Rules:
q— the raw query text exactly as a local would type it (mixed case, abbreviations, typos, glued words).expectId— the entry id for positives;nullfor negatives.kind— one oflandmark | abbreviation | glued | typo | alias | negative.threshold.top3_hit_rate—0.9(D-14's fixed number; the runner reads it from the file so a future edit is a file edit).
Minimum size: 60 cases. (The specific query strings below are ILLUSTRATIVE — pin every case to an entry id that actually exists in plan 04-01's shipped search/src/data/kl-gazetteer.json; a case whose expectId has no entry fails the runner.) Category floors:
landmark— >= 12 (the golden landmarks and other well-known KL places: "KLCC", "Sunway Pyramid", "DUKE", "Pavilion", "Mid Valley", "Berjaya Times Square", "IOI City Mall", "Merdeka 118", "The Intermark", "KL Sentral", "Nu Sentral", "PWTC", "IJN", "Taman Bangsar")abbreviation— >= 15 ("Jln Ampang", "Jalan Ampang", "Tmn Bangsar", "Taman Bangsar", "Kg Ampang", "Kampung Ampang", "Bdr Baru", "Bandar Baru", "Lbh Ampang", "Lebuh Ampang", "Psrn KL", "Persiaran KL", "Sg Wang", "Sungai Wang", "Blk A")glued— >= 6 ("tamanbangsar", "jalankanpar", "kampungbarra", "bandarbaru", "sungaiwang", "bukitbintang")typo— >= 12 ("KCLC", "KLCC" transposed, "Sunwya Pyarmid", "Pavillion", "Mid Vally", "Berjaya Time Square", "IJN" → "IJNP", "Sentral" → "Sentrl", "Tmn Bangsr", "Pavilon", "Intermrk", "Sunway Velocity" → "Sunway Vellocity") — every typo case is a NON-string-equal misspelling; the bucket resolves through plan 04-01's FUZZY residual tier (each illustrative's similarity is >=FUZZY_ADMIT_SIMILARITY0.72), never through alias editsalias— >= 12 ("IJN", "PWTC", "Stesen KL Sentral", "KL Central", "BTS", "MV", "IOI", "Sunway Velocity", "Velocity", "Cheras South", "Pudu", "Intermark")negative— >= 6 ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") — these MUST be refused with an empty shortlist, never a fabricated pin; the bucket MUST ALSO include at least one NEAR-MISS negative — a genuinely out-of-KL query whose best fuzzy similarity to some gazetteer alias lands in[0.55, FUZZY_ADMIT_SIMILARITY)(illustrative: "Kampung Cina" vs thekampung-ampangalias "Kampung Ampang" — the "kampung" token matches exactly, "cina" drags the token-level mean to ~0.6) — a query a LOWERED admission threshold would wrongly admit. CO8 pins this margin mechanically
TAUTOLOGY LAW (D-13 / RESEARCH Pitfall 7): the corpus must contain queries that are NOT string-equal to any alias in the gazetteer. The typo and glued buckets are exactly those queries — and at least 15 cases overall must not equal any alias. The runner ASSERTS this (cases.filter(c => c.kind !== 'negative' && !aliases.has(normalize(c.q))).length >= 15), so a future regeneration that makes the corpus tautological fails the suite. JOINT ARITHMETIC with the D-14 escalation ladder: the typo bucket resolves through plan 04-01's FUZZY residual tier (FUZZY_ADMIT_SIMILARITY 0.72 × the 0.60 ceiling — "KCLC" sim 0.75 → score 0.45), so lever 1 is NEVER needed for a typo. Adding a typo string as an alias would both defeat the case's purpose and erode the tautology floor below CO3's 15 — reserve lever 1 for genuine alias gaps (a missing landmark, a missing local shorthand). With the typo bucket served by the residual net, the ~12 typo cases plus the alias-bucket queries that are not literal alias strings keep the non-tautological count comfortably above 15.
Every expectId MUST match /^[A-Za-z0-9_-]{1,64}$/ — the start_param-safe charset (RESEARCH: "The corpus runner should assert every corpus expectId matches /^[A-Za-z0-9_-]{1,64}$/").
Every positive's expectId MUST exist in the gazetteer — asserted by the runner. A corpus case pointing at a non-existent entry fails the suite immediately (that is a corpus bug, not a search bug).
</corpus_format_contract>
<corpus_runner_contract>
search/scripts/gazetteer-corpus.mjs — the D-14 bake-off close. Copy bot/scripts/register-commands.mjs's script conventions (shebang + a header contract block stating purpose, inputs, what is echoed, and the exit-code contract) and bot/scripts/set-webhook.mjs's single-fetch-then-report shape.
Behaviour:
- Import
searchfrom../src/index.ts(the same module the bot uses). - Read the corpus JSON from
search/test/fixtures/golden-corpus.jsonviareadFileSync+JSON.parse(a script may usefs— only the WORKER may not). - Split the cases into positives (
expectId !== null) and negatives (expectId === null). - For each positive:
search(c.q, 3)→ record whetherhits[0].entry.id === c.expectId(top-1) and whetherhits.map(h => h.entry.id).includes(c.expectId)(top-3). Record the miss as"<query> -> [<ids or —>] (expected <id>)". - For each negative:
search(c.q, 3)→ ifhits.length > 0, record the leak as"<query> -> <first id>". - Print a report to stdout:
- the overall line:
top-1 <n>/<N> top-3 <n>/<N> (<pct>%) <avg> ms/query— the per-query average gives the 10 ms Workers CPU-budget claim a measuring hook at every run - a PER-CATEGORY breakdown: one line per
kindwith that category's top-3 rate negatives refused <n>/<N>— followed by the refusal-margin line:refusal margin <best negative sim> < 0.72(the highest fuzzy similarity any negative reached against any alias, and its gap toFUZZY_ADMIT_SIMILARITY— the margin that makes the refusals honest is VISIBLE, not accidental)- the
MISSED:block (one line per miss) and theLEAKED (must refuse):block (one line per leak) - the threshold line:
threshold top3 >= 90.0% — PASS|FAIL
- the overall line:
- Exit code:
0only whentop3Rate >= threshold.top3_hit_rateANDleaked.length === 0;1otherwise.
The runner MAY print query text — it is a local dev tool over a committed file (RESEARCH: "The corpus runner (a local dev tool over a committed file) MAY print queries; the Worker NEVER may"). The runner MUST print the ESCALATION LADDER when it fails, verbatim:
D-14 threshold not met. Escalation order is FIXED:
(1) alias/normalizer table edits in search/ -> re-run this runner
(2) SearchSource backend #2 — the Malaysia-only WASM geocoder (D-05, post-v1)
(3) Photon-on-VPS (D-05, Phase 5)
Record which lever was pulled in docs/SEARCH-QUALITY.md.
Also add a root delegate "search:corpus": "node search/scripts/gazetteer-corpus.mjs" so the D-14 gate is one command.
The runner must ALSO be assertable in-process. search/test/corpus.test.mjs re-runs the same measurements by importing the shared module and the corpus directly (not by spawning the script) — so npm run search:test fails when the corpus regresses (the Nyquist requirement: "the corpus must be ≥ threshold at every wave merge, not just at the end"). The suite must assert the SAME things the script does: top-3 rate >= threshold, zero negative leaks, the tautology floor, the id charset, and that every positive's expectId exists in the gazetteer.
</corpus_runner_contract>
<overpass_script_contract>
search/scripts/overpass-topup.mjs — the D-06 maintainer tool. HARD LAWS:
LAW 1 — it is a REPO TOOL, never a runtime dependency. It lives under search/scripts/, is run manually, and is NEVER imported by bot/src/ or web/src/. A grep gate asserts no Overpass/OSM URL appears under either source tree (RESEARCH Pitfall 6: "Warning signs: fetch("https://overpass-api.de/…") anywhere under bot/src/ or web/src/").
LAW 2 — ONE request per run. The script makes exactly ONE Overpass request and prints the candidate list. It never loops over tag families.
LAW 3 — an identifying User-Agent is MANDATORY. A request without a UA gets HTTP 406 (live-verified 2026-10-23: curl https://overpass-api.de/api/status with no UA → HTTP 406). The header is exactly:
User-Agent: araharah-gazetteer-topup/1.0 (+https://github.com/araharah)
LAW 4 — a busy instance is a NORMAL outcome, and the script EXITS 0. Overpass capacity saturation is intermittent (RESEARCH §Verification A4 observed 502 and 200 minutes apart). On 502 / 429 / a network timeout / a non-200 status the script prints exactly one line:
Overpass instance busy — retry later
and exits 0 (never a stack trace, never a non-zero code). Nothing in the repo — no deploy, no test run, no corpus gate — may depend on this script succeeding (RESEARCH §Verification A4 item 4).
LAW 5 — it NEVER writes to kl-gazetteer.json. Output is a PRINTED candidate list (or a proposal file at an explicit --out path that the maintainer diffs and merges by hand). The script writes nothing to the curated dataset under any flag. The word "proposal" must appear in the output header.
LAW 6 — the source and the endpoint are overridable. --source=overpass|nominatim|fixture selects the candidate source (default overpass); --endpoint=<url> and the OVERPASS_ENDPOINT env var override the Overpass endpoint (default https://overpass-api.de/api/interpreter); --fixture=<path> names the fixture file for --source=fixture. --source=nominatim --q=<term> queries the reachable Nominatim search endpoint bounded to the KL viewbox instead (Nominatim was live-verified reachable 2026-10-23 while the Overpass mirrors were not). This is the resolved offline posture: the committed fixture is the deterministic test path, Nominatim is the alternative live path, and neither is ever a runtime dependency. Both live paths (Overpass and Nominatim) MUST be served by ONE shared fetch helper, so LAW 2's exactly-one-request invariant is structural — exactly ONE fetch( call site in the source, whichever source is selected (this is what TP7 asserts).
The query shape — a NARROW, single-tag-family query bounded to the KL bbox (Overpass bbox order is s,w,n,e = south, west, north, east):
[out:json][timeout:25];
nwr["tourism"="attraction"](3.05,101.55,3.25,101.80);
out center 40;
The bbox MUST be present (RESEARCH §Anti-Patterns: "An Overpass query with no bbox… will time out and is a well-known way to get rate-limited"). The default tag family is tourism=attraction; --tag-family= may override it with ONE family (never a union of families).
The offline + alternate paths. --source=fixture --fixture=<path> reads a committed Overpass JSON response instead of making a request — this is how the test exercises the script deterministically (RESEARCH §Verification A4: the committed fixture is the reliability anchor; the live primary instance works only with a User-Agent and saturates intermittently). --source=nominatim --q=<term> queries https://nominatim.openstreetmap.org/search?format=jsonv2&q=<term>&viewbox=101.55,3.25,101.80,3.05&bounded=1&limit=40 with the same identifying-User-Agent discipline (Nominatim's usage policy requires one too) and emits the same PROPOSAL shape.
The output format — one candidate per line, with the fields a maintainer needs to decide:
PROPOSAL — 7 candidate(s) from tourism=attraction in bbox 3.05,101.55,3.25,101.80
Review and hand-merge into the curated dataset file — this script NEVER writes it.
name=Merdeka 118 osm=way/12345678 lat=3.1418 lon=101.6943 category=landmark
name=National Monument osm=node/87654321 lat=3.1489 lon=101.6936 category=landmark
search/test/fixtures/overpass-response.json — a committed, REAL Overpass JSON response (the elements array shape: each element has type, id, lat/lon for nodes or center: {lat, lon} for ways/relations, and tags). Use 5-8 elements with realistic KL tag data. The fixture makes the top-up test deterministic — no live network in the test suite.
search/test/topup.test.mjs — cases:
TP1 the script is not imported by the Worker— grepbot/src/**andweb/src/**foroverpass/overpass-api.de/overpass-api→ ZERO hits.TP2 the script sends an identifying User-Agent— the script source containsUser-Agent: araharah-gazetteer-topup/1.0.TP3 the query is bbox-bounded— the script source contains a bbox tuple matching/\(\d+\.\d+,\s*\d+\.\d+,\s*\d+\.\d+,\s*\d+\.\d+\)/.TP4 the script exits 0 on a busy instance— run the script with--fixture=pointing at a fixture whose content is NOT an Overpass response (e.g. a JSON object with"error"), assert the process exit code is 0 and stdout contains "Overpass instance busy".TP5 the script produces a valid-shaped proposal from the committed fixture— run the script with--fixture=search/test/fixtures/overpass-response.json, assert exit 0 and that stdout contains "PROPOSAL", at least onename=line, and the literal text "NEVER writes".TP6 the script never writes the curated file— assert the script source contains nowriteFileSynctargetingkl-gazetteer.json(the only permitted write is an explicit--outproposal path, and that path must not be the curated file — assert the source does not contain the literalkl-gazetteer.jsonat all).TP7 exactly one request per run— the Overpass and Nominatim live paths (LAW 6) share ONEfetchhelper, so the source contains exactly ONEfetch(call site regardless of the selected source; assert that count is exactly 1.
search/scripts/json-to-sqlite.mjs — the D-05 backend-#2 format converter. It reads search/src/data/kl-gazetteer.json and emits a SQLite file with the SAME ids, so backend #2's dataset format is reachable today (D-05: "The JSON↔SQLite converter script keeps its dataset format reachable from day one"). It is a STUB in the sense that the SQLite WRITER is a minimal, dependency-free implementation (Node 24 has node:sqlite behind a flag — check whether node:sqlite is available in Node 24.18.0; if it is, use it; if not, emit a deterministic, documented, plain-text .sql DUMP file plus a JSON sidecar and say so in the header — do NOT add a dependency).
search/test/converter.test.mjs — cases:
CV1 the converter preserves the id set— run the converter into a temp directory, read the output's id list, and assert it is EXACTLY the gazetteer's id set (same members, same count).CV2 the converter reads the committed dataset— the script source referenceskl-gazetteer.json.CV3 the converter is not a runtime dependency— grepbot/src/**+web/src/**forjson-to-sqlite/sqlite→ ZERO hits.
docs/SEARCH-QUALITY.md — the bake-off close record. Structure:
# Search quality — Phase 4 bake-off close
**Corpus:** `search/test/fixtures/golden-corpus.json` version 1 — <N> hand-written cases
**Gazetteer:** `search/src/data/kl-gazetteer.json` version 1 — <M> entries
**Measured:** <date>
| Metric | Value | Threshold |
|---|---|---|
| top-1 hit rate (positives) | <n>/<N> (<pct>%) | — (reported) |
| top-3 hit rate (positives) | <n>/<N> (<pct>%) | >= 90% |
| negatives refused | <n>/<N> | 100% (zero leaks) |
### Per-category
| Category | Cases | top-3 |
|---|---|---|
| landmark | … | … |
| abbreviation | … | … |
| glued | … | … |
| typo | … | … |
| alias | … | … |
### Escalation lever pulled (D-14)
<Exactly one of:>
1. **None — the threshold was met on the first run.** No table edits were needed.
2. **Lever 1 — alias/normalizer table edits.** <Which rows were added to `search/src/rules/abbreviations.ts` / which aliases were added to the gazetteer. The corpus was re-run; the result above is the post-edit measurement.>
3. **Lever 2 — SearchSource backend #2 (WASM geocoder).** <…>
4. **Lever 3 — Photon-on-VPS.** <…>
### Known limitations
- The corpus is hand-written and small; it measures the documented local shorthand, not every
phrasing a KL driver might use.
- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1.
- The fuzzy tier performs a full-candidate OSA scan when a query has no cheap-tier hit (the
pure-typo path) — measured at ~8.2 ms/query against ~2 000 alias pairs (RESEARCH §Verification
A1), inside the 10 ms Workers Free CPU limit at the seed scale. This is a RECORDED scaling
decision, not an oversight (RESEARCH directive #1): if a future top-up pushes the dataset toward
D-07's 500-entry ceiling and CPU approaches the limit, the documented next step is a cheap
blocking key (first-letter + length band) in front of the OSA tier — measured against this
corpus before it ships.
Acceptance: npm run search:corpus exits 0 with top-3 >= 90% and zero negative leaks; npm run search:test green; docs/SEARCH-QUALITY.md records the measured numbers and the lever.
Every expectId MUST be a real id in search/src/data/kl-gazetteer.json. The corpus's negatives MUST be genuinely out-of-area ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") so the refusal path is exercised.
search/scripts/gazetteer-corpus.mjs — the runner, EXACTLY per the <corpus_runner_contract> block: shebang + header contract block, the report format (overall line, PER-CATEGORY breakdown, negatives-refused line, MISSED: block, LEAKED (must refuse): block, threshold line), the fixed escalation ladder printed on failure, and the exit-code contract (0 only when top-3 >= threshold AND zero leaks).
Add to search/package.json:
"corpus": "node scripts/gazetteer-corpus.mjs"
and to the root package.json:
"search:corpus": "node search/scripts/gazetteer-corpus.mjs",
search/test/corpus.test.mjs — the Nyquist gate (RED-first). Cases:
CO1 top-3 hit rate meets the D-14 threshold—top3 / positives.length >= corpus.threshold.top3_hit_rate; the failure message names the D-14 threshold and the escalation ladder's first lever.CO2 every negative is refused— zero leaks; the message names the leaking query.CO3 the corpus is not tautological— at least 15 non-negative cases whose normalized query is NOT string-equal to any alias or name in the gazetteer (RESEARCH Pitfall 7).CO4 every positive's expectId exists in the gazetteer— no dangling ids.CO5 every expectId matches the start_param-safe charset—/^[A-Za-z0-9_-]{1,64}$/.CO6 the corpus meets its size and category floors— >= 60 cases; landmark >= 12, abbreviation >= 15, glued >= 6, typo >= 12, alias >= 12, negative >= 6.CO7 every category's top-3 rate is reported— the suite prints (not asserts) the per-category rates so the runner output and the suite output agree.CO8 the refusal margin is pinned— for every negative, compute its best fuzzy similarity against every gazetteer alias using the shareddamerauand the SAME two-level (whole-string + token) shape the scorer uses; assert ALL negatives sit strictly belowFUZZY_ADMIT_SIMILARITY(0.72), and assert at least one near-miss negative sits in[0.55, 0.72)(the corpus contract's near-miss requirement) — so lowering the admission threshold would leak a real negative and fail BOTH CO2 and CO8. The floor is pinned by test, not by hope.
The suite imports the shared module and the corpus directly (in-process), so npm run search:test fails on a corpus regression.
Acceptance: npm run search:corpus exits 0; npm run search:test green with PASS CO1…PASS CO8.
cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -30 ; npm run search:test 2>&1 | grep -E 'PASS CO|runner' ; node -e "const c=require('./search/test/fixtures/golden-corpus.json'); if(c.cases.length<60) throw new Error('corpus too small: '+c.cases.length); console.log('corpus OK '+c.cases.length+' cases')"
<acceptance_criteria>
- npm run search:corpus exits 0
- npm run search:corpus output contains a top-3 line with a percentage >= 90.0% and a negatives refused line showing all negatives refused
- npm run search:corpus output contains one PASS/FAIL threshold line naming the 90.0% gate
- npm run search:test exits 0 and the output contains PASS CO1 through PASS CO8
- node -e "const c=require('./search/test/fixtures/golden-corpus.json'); const g=require('./search/src/data/kl-gazetteer.json'); const ids=new Set(g.entries.map(e=>e.id)); const kinds={}; let bad=0; for(const x of c.cases){ kinds[x.kind]=(kinds[x.kind]||0)+1; if(x.expectId!==null && !ids.has(x.expectId)) bad++; if(x.expectId!==null && !/^[A-Za-z0-9_-]{1,64}$/.test(x.expectId)) bad++; } if(c.cases.length<60) throw new Error('size'); if(bad) throw new Error('dangling ids'); console.log(JSON.stringify(kinds))" exits 0 and prints all six kind keys with landmark>=12, abbreviation>=15, glued>=6, typo>=12, alias>=12, negative>=6
- grep -q '"search:corpus": "node search/scripts/gazetteer-corpus.mjs"' package.json exits 0 AND grep -q '"corpus": "node scripts/gazetteer-corpus.mjs"' search/package.json exits 0
- grep -q 'D-14 threshold not met' search/scripts/gazetteer-corpus.mjs exits 0 AND grep -q 'backend #2' search/scripts/gazetteer-corpus.mjs exits 0 AND grep -q 'Photon-on-VPS' search/scripts/gazetteer-corpus.mjs exits 0 (the fixed escalation ladder is printed verbatim on failure)
- grep -c 'expectId' search/test/fixtures/golden-corpus.json | head -1 returns >= 60 (each case pins an expected id or null)
</acceptance_criteria>
The golden corpus is pinned at 60+ hand-written cases across 6 categories with real negatives, the runner mechanically enforces D-14's top-3 >= 90% + zero-leak threshold, and npm run search:test fails whenever the corpus regresses.
Header contract block (copy set-webhook.mjs's shape — purpose, inputs, what is echoed/never echoed, exit-code contract):
#!/usr/bin/env node
/*
* overpass-topup.mjs — propose KL gazetteer top-up candidates from OpenStreetMap (D-06).
* -------------------------------------------------------------------------------------
* A MAINTAINER-TIME REPO TOOL. It is NOT a runtime dependency: nothing in
* bot/src/ or web/src/ imports it, and no deploy, test, or corpus gate depends on
* it succeeding. The Worker only ever reads the COMMITTED curated JSON.
*
* Output is a PROPOSAL — a printed candidate list (or a file at an explicit --out
* path) that a human diffs and hand-merges into the curated dataset file.
* THIS SCRIPT NEVER WRITES THE CURATED FILE.
*
* Exit codes: 0 on success AND on a busy instance (Overpass capacity saturation
* is intermittent — 502/429/timeout is a NORMAL, retryable outcome). Non-zero
* only on a usage error (no bbox resolvable / unreadable fixture).
*
* Inputs: --source=overpass|nominatim|fixture (default overpass), --endpoint=<url>
* (or OVERPASS_ENDPOINT), --bbox=s,w,n,e, --q=<term> (nominatim source),
* --tag-family=<key>=<value>, --fixture=<path> (offline input),
* --out=<path> (write the proposal here — never the curated file).
*/
The request carries EXACTLY this header set:
const headers = {
"User-Agent": "araharah-gazetteer-topup/1.0 (+https://github.com/araharah)",
"Content-Type": "application/x-www-form-urlencoded",
};
The Overpass QL query is single-family and bbox-bounded, e.g.:
[out:json][timeout:25];
nwr["tourism"="attraction"](3.05,101.55,3.25,101.80);
out center 40;
On a non-200 status, a network error, or an unparseable response, print exactly:
Overpass instance busy — retry later
and process.exit(0).
search/test/fixtures/overpass-response.json — a committed, realistic Overpass JSON response with 6 elements: a mix of node elements (with lat/lon) and way/relation elements (with center: {lat, lon}), each carrying tags with name and name:en plus a category-ish tag (tourism, amenity, railway, office, leisure). The elements must be real KL places (e.g. "Merdeka 118", "National Monument", "Masjid Jamek", "KL Tower", "Central Market", "Petaling Street") with plausible coordinates inside the KL bbox.
search/test/topup.test.mjs — the seven cases TP1…TP7 from the <overpass_script_contract> block. TP4 and TP5 RUN the script as a child process (node scripts/overpass-topup.mjs ...) from the search/ workspace directory and assert the exit code and the stdout shape. TP4 uses a temp fixture that is NOT an Overpass response (e.g. {"error":"busy"}) to exercise the soft-exit path.
search/scripts/json-to-sqlite.mjs — the D-05 converter. It reads search/src/data/kl-gazetteer.json and emits a dataset with the SAME ids so backend #2's format is reachable from day one (D-05). The header block states that it is a repo tool, never a runtime dependency, and that the ODbL notice MUST travel with the output (ODbL §4.4 share-alike on a derived database).
Node's SQLite availability: Node 24.18.0 ships node:sqlite (stable-ish behind no flag in Node 22.5+/24). PROBE IT FIRST in this task with node -e "const {DatabaseSync}=require('node:sqlite'); console.log('sqlite OK')". If it works, emit a real .sqlite file with a places table (id TEXT PRIMARY KEY, name TEXT, aliases TEXT, lat REAL, lng REAL, category TEXT, address TEXT). If it does NOT work, emit a deterministic .sql dump file plus a JSON sidecar and say so plainly in the header — do NOT add a dependency.
search/test/converter.test.mjs — the three cases CV1…CV3 from the <overpass_script_contract> block. CV1 runs the converter into a temp directory and asserts the emitted id set is EXACTLY the gazetteer's id set (same members, same count) — reading the ids back out of whatever format was emitted (SQLite via node:sqlite, or the .sql dump's INSERT lines).
Acceptance: npm run search:test green with PASS TP1…PASS TP7 and PASS CV1…PASS CV3; the top-up script exits 0 on the fixture AND on a busy-shaped input.
cd /home/C-Fu/araharah && npm run search:test 2>&1 | grep -E 'PASS TP|PASS CV|runner' ; cd search && node scripts/overpass-topup.mjs --fixture=test/fixtures/overpass-response.json 2>&1 | head -12 ; echo "exit=$?" ; node -e "const {DatabaseSync}=require('node:sqlite'); console.log('node:sqlite available')" 2>&1 | tail -1
<acceptance_criteria>
- npm run search:test exits 0 and the output contains PASS TP1 through PASS TP7 and PASS CV1 through PASS CV3
- cd search && node scripts/overpass-topup.mjs --fixture=test/fixtures/overpass-response.json exits 0 and its stdout contains PROPOSAL, at least one name= line, and the literal text NEVER writes
- cd search && node scripts/overpass-topup.mjs --fixture=<a non-Overpass fixture> exits 0 and its stdout contains exactly the line Overpass instance busy — retry later
- grep -q 'User-Agent: araharah-gazetteer-topup/1.0' search/scripts/overpass-topup.mjs exits 0 (HTTP 406 without it — live-verified)
- grep -q '\-\-source' search/scripts/overpass-topup.mjs exits 0 (the source flag exists: overpass | nominatim | fixture — the resolved-question posture)
- grep -c 'await fetch(\|fetch(' search/scripts/overpass-topup.mjs | head -1 returns 1 (EXACTLY ONE request per run — the Overpass and Nominatim paths share this single fetch helper, per LAW 6)
- grep -c 'kl-gazetteer.json' search/scripts/overpass-topup.mjs | head -1 returns 0 (the script never names the curated file — it cannot write it)
- grep -rl 'overpass' bot/src/ web/src/ 2>/dev/null | wc -l returns 0 (the script is never a runtime dependency — RESEARCH Pitfall 6)
- grep -rl 'sqlite\|json-to-sqlite' bot/src/ web/src/ 2>/dev/null | wc -l returns 0
- node -e "const g=require('./search/src/data/kl-gazetteer.json'); const {execFileSync}=require('node:child_process'); const fs=require('node:fs'); const os=require('node:os'); const path=require('node:path'); const out=fs.mkdtempSync(path.join(os.tmpdir(),'araharah-conv-')); execFileSync('node',['search/scripts/json-to-sqlite.mjs','--out',out],{cwd:process.cwd()}); const files=fs.readdirSync(out); if(!files.length) throw new Error('converter emitted nothing'); const blob=files.map(f=>fs.readFileSync(path.join(out,f),'utf8')).join('\n'); const ids=g.entries.map(e=>e.id); const missing=ids.filter(id=>!blob.includes(id)); if(missing.length) throw new Error('id set not preserved: '+missing.join(',')); console.log('converter OK -> '+files.join(','));" exits 0
</acceptance_criteria>
Both maintainer-time repo tools ship and are proven offline: the Overpass top-up makes exactly one identified request, prints a proposal, never writes the curated file, and exits 0 on a busy instance; the JSON→SQLite converter preserves the gazetteer's id set exactly, so backend #2's dataset format is reachable today.
If the threshold is met (top-3 >= 90% on positives AND zero negative leaks): the escalation lever is "None". Record that in docs/SEARCH-QUALITY.md and make NO corpus/table/data edits to force a pass.
If the threshold is NOT met: pull the escalation ladder in the FIXED order (D-14) and re-run after each pull:
- Lever 1 — alias/normalizer table edits. For each MISSED case, decide whether the miss is an ALIAS gap (add the local shorthand to the entry's
aliasesinsearch/src/data/kl-gazetteer.json) or an ABBREVIATION gap (add a row tosearch/src/rules/abbreviations.ts— a table edit, never a code change). Add a per-rule test vector tosearch/test/normalizer.test.mjsfor every new abbreviation row (D-09: "Per-rule test vectors; extends by table edit, not code change"). Re-runnpm run search:corpusandnpm run search:test. - Lever 2 — SearchSource backend #2 (the Malaysia-only WASM geocoder). This backend does NOT ship in v1 (D-05) and MUST NOT be implemented here. If lever 1 cannot reach the threshold, RECORD that fact in
docs/SEARCH-QUALITY.mdas the documented outcome and leave the threshold unmet with the lever named — the phase's experiment ends with a documented decision (D-14), not with an open question and not with an unbuilt backend smuggled in. - Lever 3 — Photon-on-VPS. Same handling as lever 2 — record, do not build (D-03 defers VPS provisioning to Phase 5).
NEVER edit the corpus to make it pass. The corpus is PINNED (D-13). A missed case is fixed by improving the DATA or the RULES, never by changing the expectation. If a corpus case is simply WRONG (its expectId points at an entry that is not the right landmark), fix the corpus's expectId and say so explicitly in docs/SEARCH-QUALITY.md under "Corpus corrections" — that is a corpus bug, and it must be visible.
Do NOT weaken the admission rule to make positives pass. The tiered admission (non-fuzzy tiers always; fuzzy only at similarity >= FUZZY_ADMIT_SIMILARITY 0.72 — plan 04-01's contract) is what makes the negatives honest (D-13/D-14). If a positive can only be found by lowering that number, the correct fix is an alias or a table row.
Coordinate verification (the resolved-question offline posture). Spot-verify at least 8 of the curated seed entries' coordinates against Nominatim (reachable per RESEARCH §Environment Availability): for each sampled entry, GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=<entry name>, Kuala Lumpur&limit=1 with an identifying User-Agent (the same discipline as the Overpass script — a missing UA is the 406 trap), and confirm the returned lat/lon lands within ~0.01° of the entry's committed coordinates. Record the sample size, the method, and the outcome in docs/SEARCH-QUALITY.md under a "Coordinate verification" section (N of M sampled within tolerance — or, if Nominatim is unreachable at run time, record that honestly and name the committed-bbox gate G5 as the fallback invariant). This is a one-shot maintainer-time check recorded in the doc — NOT a test-suite dependency and never a runtime path.
Write docs/SEARCH-QUALITY.md — EXACTLY per the <overpass_script_contract> block's structure: the header (corpus version + case count, gazetteer version + entry count, the measurement date), the metric table (top-1, top-3 vs the 90% threshold, negatives refused vs the zero-leak requirement), the per-category table, the "Escalation lever pulled" section naming exactly one lever (or "None"), the "Corpus corrections" section (empty if none), the "Coordinate verification" section, and the "Known limitations" section (the hand-written corpus measures the documented local shorthand, not every phrasing; out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1).
Acceptance: npm run search:corpus exits 0; docs/SEARCH-QUALITY.md records the real measured numbers; the lever section names one lever honestly.
cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -25 ; npm run search:test 2>&1 | tail -3 ; test -f docs/SEARCH-QUALITY.md && grep -c 'top-3' docs/SEARCH-QUALITY.md && grep -c 'Escalation lever pulled' docs/SEARCH-QUALITY.md && grep -c 'Known limitations' docs/SEARCH-QUALITY.md
<acceptance_criteria>
- npm run search:corpus exits 0
- npm run search:test exits 0 (any lever-1 edit kept every other suite green)
- test -f docs/SEARCH-QUALITY.md exits 0 AND grep -q 'top-3' docs/SEARCH-QUALITY.md exits 0 AND grep -q 'negatives refused' docs/SEARCH-QUALITY.md exits 0 AND grep -q 'Escalation lever pulled' docs/SEARCH-QUALITY.md exits 0 AND grep -q 'Coordinate verification' docs/SEARCH-QUALITY.md exits 0 AND grep -q 'Known limitations' docs/SEARCH-QUALITY.md exits 0
- grep -q 'Lever 1\|Lever 2\|Lever 3\|None — the threshold was met' docs/SEARCH-QUALITY.md exits 0 (exactly one lever is named)
- grep -c 'search:corpus' docs/SEARCH-QUALITY.md | head -1 returns >= 1 (the record names how to reproduce the measurement)
- git diff --stat search/test/fixtures/golden-corpus.json shows NO expectation edits that lowered a threshold — the corpus's threshold.top3_hit_rate is still 0.9: grep -q '"top3_hit_rate": 0.9' search/test/fixtures/golden-corpus.json exits 0
- grep -q 'search/src/rules/abbreviations.ts' docs/SEARCH-QUALITY.md exits 0 when lever 1 was pulled, OR the doc names lever 2/3 instead (the lever section is specific)
</acceptance_criteria>
The bake-off is closed with a measured, recorded, honest outcome: top-3 >= 90% on positives, every negative refused, and exactly one named escalation lever (or "None") recorded in docs/SEARCH-QUALITY.md with the reproduction command.
<threat_model>
Trust Boundaries
| Boundary | Description |
|---|---|
| Committed corpus file → the measured quality claim | The corpus is the instrument that decides whether the phase ships. A corpus edited to pass is a falsified instrument. |
| Live Overpass endpoint → the maintainer's terminal | A third-party service with no SLA, intermittent capacity, and an egress filter that 406s requests without a User-Agent. |
| Overpass response → the curated dataset | A candidate list is UNTRUSTED third-party data. It must never be written straight into the curated file. |
| Corpus runner (local tool) → stdout | The runner MAY print query text (it is a local dev tool over committed files); the Worker never may. The runner's output must not be shipped anywhere. |
STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|---|---|---|---|---|
| T-04-12 | Tampering | The corpus being edited so the gate always passes (RESEARCH Pitfall 7 — "the corpus becomes a generated artifact") | mitigate | The corpus is pinned and hand-written; the tautology gate (CO3) requires >= 15 non-negative cases whose normalized query is NOT string-equal to any gazetteer alias — a regenerated corpus fails it. threshold.top3_hit_rate stays 0.9 (asserted in the acceptance criteria). The plan explicitly forbids editing a case's expectId to make it pass, and any legitimate corpus correction must be listed under "Corpus corrections" in docs/SEARCH-QUALITY.md so it is visible. |
| T-04-13 | Repudiation | The phase closing with an unmeasured or unfalsified quality claim (D-14: "the experiment ends with a documented decision, not an open question") | mitigate | npm run search:corpus is a single command with a mechanical exit code; docs/SEARCH-QUALITY.md must carry the measured top-1/top-3/refusal numbers, the per-category table, exactly one named lever, and the reproduction command. The "Escalation lever pulled" section is mandatory content, not an optional note. |
| T-04-14 | Denial of Service | The Overpass script becoming a runtime dependency (SSRF-ish: the Worker fetching a third-party service on every cold start) | mitigate | LAW 1 plus the TP1 grep gate: bot/src/ and web/src/ contain zero occurrences of overpass. The script lives under search/scripts/, is never imported by product code, and the curated JSON is the only thing the Worker reads. |
| T-04-15 | Denial of Service | The Overpass script blocking a build, a test, or a deploy because the endpoint is down or busy | mitigate | LAW 4: 502 / 429 / timeout / non-200 / unparseable → print Overpass instance busy — retry later and exit 0. The TP4 case asserts the exit code is 0 on a busy-shaped input. No repo script depends on this script's success (RESEARCH §Verification A4 item 4). |
| T-04-16 | Denial of Service | An Overpass query without a bbox causing a global scan that times out and gets the maintainer's IP rate-limited | mitigate | LAW: the query is always bbox-bounded to the KL box ((3.05,101.55,3.25,101.80)), and TP3 asserts the bbox tuple is present in the source. The query is single-family and narrow by default, which is what fits a free worker slot. |
| T-04-17 | Spoofing | The Overpass endpoint serving a response that makes the maintainer merge hostile data into the curated dataset | mitigate | The script OUTPUT is a printed proposal only — LAW 5 plus the TP6 assertion that the script's source never names kl-gazetteer.json at all, so it CANNOT write it. A human diffs and hand-merges. The curated dataset's own invariants (unique ids, slug charset, KL-bbox coordinates, non-empty aliases) are asserted by search/test/gazetteer.test.mjs (plan 04-01), so hostile coordinates or a duplicate id would fail the suite even if a human merged them by mistake. |
| T-04-18 | Information Disclosure | The corpus runner's query-printing leaking into a shipped log | mitigate | The runner is a local dev tool under search/scripts/; it is never imported by bot/src/ or web/src/ and no product code calls it. The Worker-side logging law (category-only) is enforced separately in plan 04-03's source-introspection gate. The runner's output goes to a maintainer's terminal, not to a log aggregator. |
| T-04-19 | Tampering | The D-14 threshold being lowered (e.g. 0.9 → 0.8) to make a failing corpus pass |
mitigate | The threshold lives in the corpus file as a single literal 0.9, and the acceptance criteria assert grep -q '"top3_hit_rate": 0.9' still holds after the bake-off. D-14 fixes the number as 90%; changing it is a decision change, not a tuning knob. |
| T-04-20 | Information Disclosure | The JSON→SQLite converter emitting the dataset WITHOUT the ODbL notice, producing an unattributed derivative database (ODbL §4.4 share-alike) | mitigate | The converter's header block states that the ODbL notice MUST travel with the output, and the emitted artifact carries the _attribution / _license / _notice fields in a metadata table (or as a header comment in the .sql dump). The converter is a repo tool — the notice travels with the data by construction. |
| </threat_model> |
<success_criteria>
- D-13: a pinned golden corpus of 60+ hand-written cases exists at
search/test/fixtures/golden-corpus.json, with 6 categories and at least 15 non-tautological cases, each positive pinning its expected entry id - D-14: the bake-off CLOSES on a mechanical threshold —
npm run search:corpusexits 0 only when top-3 >= 90% on positives AND every negative is refused — and the measured outcome plus the lever pulled are recorded indocs/SEARCH-QUALITY.md - D-06: the Overpass top-up is a repo tool that proposes candidates for human approval, never writes the curated file, sends an identifying User-Agent, makes exactly one request, and exits 0 on a busy instance — proven offline against a committed fixture
- D-05: the JSON→SQLite converter exists, reads the same committed dataset, and preserves the id set exactly — backend #2's dataset format is reachable today
- SRCH-01/SRCH-02 are MEASURED: the golden landmarks and the abbreviation/typo/glued/alias buckets each have a measured top-3 rate, so a later regression is diagnosable per category rather than as one diffuse number </success_criteria>