564 lines
51 KiB
Markdown
564 lines
51 KiB
Markdown
---
|
|
phase: 04-destination-search-kl-gazetteer
|
|
plan: 02
|
|
type: execute
|
|
wave: 2
|
|
depends_on: [04-01]
|
|
files_modified:
|
|
- search/test/fixtures/golden-corpus.json
|
|
- search/test/corpus.test.mjs
|
|
- search/scripts/gazetteer-corpus.mjs
|
|
- search/scripts/overpass-topup.mjs
|
|
- search/scripts/json-to-sqlite.mjs
|
|
- search/test/topup.test.mjs
|
|
- search/test/converter.test.mjs
|
|
- search/test/fixtures/overpass-response.json
|
|
- search/package.json
|
|
- package.json
|
|
- search/src/data/kl-gazetteer.json
|
|
- search/src/rules/abbreviations.ts
|
|
- search/test/normalizer.test.mjs
|
|
- docs/SEARCH-QUALITY.md
|
|
autonomous: true
|
|
requirements: [SRCH-01, SRCH-02]
|
|
must_haves:
|
|
decisions_implemented:
|
|
- "D-06: the Overpass top-up is a REPO TOOL, run manually by a maintainer, that PRINTS candidates — it never writes the curated file and is never imported by the Worker"
|
|
- "D-05: the JSON dataset format stays reachable from day one — a JSON→SQLite converter script exists and preserves the id set, so backend #2 (WASM geocoder) is a format change, not a rewrite"
|
|
- "D-13: the golden corpus is HAND-WRITTEN and PINNED — ~60-100 cases including deliberate out-of-KL negatives, each positive pinning its expected entry id"
|
|
- "D-14: the bake-off closes on a MECHANICAL threshold — top-3 contains the expected id for >= 90% of positives AND every negative is refused — and the phase records which escalation lever was pulled"
|
|
- "D-03: the corpus runner exercises the SAME shared module the bot and the Mini App use, so the measured quality IS the shipped quality"
|
|
truths:
|
|
- "The golden corpus contains at least 60 hand-written cases across 6 categories: golden landmarks, abbreviations, glued words, typos, aliases, and deliberate out-of-KL negatives"
|
|
- "At least 15 corpus queries are NOT string-equal to any alias in the gazetteer — the corpus measures something real, it is not tautological"
|
|
- "The corpus runner exits 0 only when top-3 >= 90% on positives AND every negative returned an empty shortlist"
|
|
- "The corpus runner reports per-category rates so a regression is diagnosable, not a diffuse number"
|
|
- "The Overpass top-up script produces a valid-shaped proposal file from a COMMITTED FIXTURE, and treats 502/429/timeout as a normal retryable outcome that EXITS 0"
|
|
- "The Overpass script sends an identifying User-Agent (a request without one gets HTTP 406) and makes exactly ONE request per run"
|
|
- "The Overpass script never writes to kl-gazetteer.json and no OSM/Overpass URL appears anywhere under bot/src/ or web/src/"
|
|
- "The JSON→SQLite converter reads the same committed JSON and emits a file with the SAME id set — backend #2's format is reachable today"
|
|
- "The corpus outcome and the escalation lever pulled are recorded in docs/SEARCH-QUALITY.md"
|
|
artifacts:
|
|
- path: "search/test/fixtures/golden-corpus.json"
|
|
provides: "the pinned D-13 golden corpus: ~60-100 hand-written cases with expected ids and negatives"
|
|
contains: "\"cases\""
|
|
- path: "search/scripts/gazetteer-corpus.mjs"
|
|
provides: "the D-14 corpus runner — top-1/top-3 on positives, refusal rate on negatives, exit 1 below threshold"
|
|
- path: "search/test/corpus.test.mjs"
|
|
provides: "the suite that makes `npm run search:test` FAIL when the corpus regresses (the Nyquist gate)"
|
|
- path: "search/scripts/overpass-topup.mjs"
|
|
provides: "the D-06 maintainer-time Overpass candidate proposer — one request, UA header, soft-exit on busy instances"
|
|
- path: "search/test/fixtures/overpass-response.json"
|
|
provides: "a committed real Overpass JSON response so the top-up test is deterministic (no live network)"
|
|
- path: "search/scripts/json-to-sqlite.mjs"
|
|
provides: "the D-05 backend-#2 format converter stub — same ids, SQLite output"
|
|
- path: "docs/SEARCH-QUALITY.md"
|
|
provides: "the bake-off close record: corpus version, measured rates, the lever that was pulled"
|
|
key_links:
|
|
- from: "search/scripts/gazetteer-corpus.mjs"
|
|
to: "search/test/fixtures/golden-corpus.json"
|
|
via: "the runner reads the pinned corpus and asserts each positive's expected id"
|
|
pattern: "golden-corpus.json"
|
|
- from: "search/scripts/gazetteer-corpus.mjs"
|
|
to: "search/src/index.ts"
|
|
via: "the runner imports the SAME shared search module the bot uses — the measured quality is the shipped quality"
|
|
pattern: "from \"../src/index.ts\""
|
|
- from: "search/test/corpus.test.mjs"
|
|
to: "search/scripts/gazetteer-corpus.mjs"
|
|
via: "the suite re-runs the runner's assertions in-process so `npm run search:test` fails on a regression"
|
|
pattern: "search"
|
|
- from: "search/scripts/overpass-topup.mjs"
|
|
to: "search/test/fixtures/overpass-response.json"
|
|
via: "the fixture is the script's offline input path — the live Overpass endpoint is unreachable from this environment"
|
|
pattern: "overpass"
|
|
- from: "search/scripts/json-to-sqlite.mjs"
|
|
to: "search/src/data/kl-gazetteer.json"
|
|
via: "the converter reads the SAME committed dataset — backend #2's format is reachable today"
|
|
pattern: "kl-gazetteer.json"
|
|
---
|
|
|
|
<objective>
|
|
Close the search-quality experiment: pin the golden corpus (D-13), build the runner that mechanically enforces the D-14 threshold, land the two maintainer-time repo tools D-06 and D-05 require (the Overpass candidate proposer and the JSON→SQLite converter), and record the bake-off outcome in `docs/SEARCH-QUALITY.md`.
|
|
|
|
Purpose: D-14 says the phase ends with "a documented decision, not an open question" — the corpus must show top-3 ≥ 90% on positives with every negative refused, and the escalation lever that was pulled must be recorded. This plan builds the instrument that measures it AND the two repo tools the forkability law (D-04, D-06) demands: the Overpass script proposes top-up candidates for human approval (never a runtime dependency), and the JSON→SQLite converter keeps backend #2's dataset format reachable from day one (D-05).
|
|
|
|
Output: `search/test/fixtures/golden-corpus.json` (60+ hand-written cases), `search/scripts/gazetteer-corpus.mjs` (the D-14 runner), `search/test/corpus.test.mjs` (the Nyquist gate inside `npm run search:test`), `search/scripts/overpass-topup.mjs` + a committed Overpass fixture + `search/test/topup.test.mjs`, `search/scripts/json-to-sqlite.mjs` + `search/test/converter.test.mjs`, and `docs/SEARCH-QUALITY.md`.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.config/opencode/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@AGENTS.md
|
|
@.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md
|
|
@.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md
|
|
@.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md
|
|
@search/src/index.ts
|
|
@search/src/types.ts
|
|
@search/src/data/kl-gazetteer.json
|
|
@bot/scripts/register-commands.mjs
|
|
@bot/scripts/set-webhook.mjs
|
|
@bot/test/privacy.test.mjs
|
|
@bot/test/harness.mjs
|
|
@bot/test/runner.mjs
|
|
|
|
<interfaces>
|
|
From `search/src/index.ts` (plan 04-01's output — the surface every script in this plan imports):
|
|
|
|
```typescript
|
|
export type { GazetteerEntry, SearchQuery, SearchHit, SearchSource } from "./types.ts";
|
|
export { normalize } from "./normalize.ts";
|
|
export { TIER } from "./score.ts";
|
|
export { damerau } from "./distance.ts";
|
|
export { getEntries, getEntryById, getIndex } from "./gazetteer.ts";
|
|
export { createGazetteerSource, search } from "./source.ts";
|
|
|
|
// search(text: string, limit?: number): SearchHit[] — synchronous convenience wrapper
|
|
// getEntryById(id: string): GazetteerEntry | null — honest null on unknown
|
|
// getEntries(): readonly GazetteerEntry[]
|
|
```
|
|
|
|
From `search/src/types.ts`:
|
|
|
|
```typescript
|
|
export interface SearchHit {
|
|
entry: GazetteerEntry;
|
|
score: number;
|
|
matchedAlias: string;
|
|
}
|
|
```
|
|
|
|
The corpus runner and both scripts import from `../src/index.ts` — Node 24 strips the types by default, so a `.mjs` script CAN import the `.ts` source directly (the repo's existing pattern; `bot/test/commands.test.mjs` line 27 does exactly this and passes).
|
|
</interfaces>
|
|
|
|
<corpus_format_contract>
|
|
`search/test/fixtures/golden-corpus.json` — the pinned D-13 corpus. The schema carries a per-case `kind` so the runner can report per-category rates (RESEARCH Open Question 3: "measure and report per-category rates … so a regression is diagnosable").
|
|
|
|
```json
|
|
{
|
|
"version": 1,
|
|
"note": "Hand-written and pinned (D-13). NOT generated from the gazetteer — a generated corpus is tautological. Reviewers extend this file by editing it.",
|
|
"threshold": { "top3_hit_rate": 0.9 },
|
|
"cases": [
|
|
{ "q": "KLCC", "expectId": "suria-klcc", "kind": "landmark" },
|
|
{ "q": "Jln Ampang", "expectId": "suria-klcc", "kind": "abbreviation" },
|
|
{ "q": "tamanbangsar", "expectId": "bangsar", "kind": "glued" },
|
|
{ "q": "KCLC", "expectId": "suria-klcc", "kind": "typo" },
|
|
{ "q": "IJN", "expectId": "ijnp", "kind": "alias" },
|
|
{ "q": "Kuching", "expectId": null, "kind": "negative" }
|
|
]
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
- `q` — the raw query text exactly as a local would type it (mixed case, abbreviations, typos, glued words).
|
|
- `expectId` — the entry id for positives; `null` for negatives.
|
|
- `kind` — one of `landmark | abbreviation | glued | typo | alias | negative`.
|
|
- `threshold.top3_hit_rate` — `0.9` (D-14's fixed number; the runner reads it from the file so a future edit is a file edit).
|
|
|
|
**Minimum size: 60 cases.** (The specific query strings below are ILLUSTRATIVE — pin every case to an entry id that actually exists in plan 04-01's shipped `search/src/data/kl-gazetteer.json`; a case whose `expectId` has no entry fails the runner.) Category floors:
|
|
- `landmark` — >= 12 (the golden landmarks and other well-known KL places: "KLCC", "Sunway Pyramid", "DUKE", "Pavilion", "Mid Valley", "Berjaya Times Square", "IOI City Mall", "Merdeka 118", "The Intermark", "KL Sentral", "Nu Sentral", "PWTC", "IJN", "Taman Bangsar")
|
|
- `abbreviation` — >= 15 ("Jln Ampang", "Jalan Ampang", "Tmn Bangsar", "Taman Bangsar", "Kg Ampang", "Kampung Ampang", "Bdr Baru", "Bandar Baru", "Lbh Ampang", "Lebuh Ampang", "Psrn KL", "Persiaran KL", "Sg Wang", "Sungai Wang", "Blk A")
|
|
- `glued` — >= 6 ("tamanbangsar", "jalankanpar", "kampungbarra", "bandarbaru", "sungaiwang", "bukitbintang")
|
|
- `typo` — >= 12 ("KCLC", "KLCC" transposed, "Sunwya Pyarmid", "Pavillion", "Mid Vally", "Berjaya Time Square", "IJN" → "IJNP", "Sentral" → "Sentrl", "Merdeka 118" → "Merdeka 118", "Tmn Bangsr", "Pavilon", "Intermrk")
|
|
- `alias` — >= 12 ("IJN", "PWTC", "Stesen KL Sentral", "KL Central", "BTS", "MV", "IOI", "Sunway Velocity", "Velocity", "Cheras South", "Pudu", "Intermark")
|
|
- `negative` — >= 6 ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") — these MUST be refused with an empty shortlist, never a fabricated pin
|
|
|
|
**TAUTOLOGY LAW (D-13 / RESEARCH Pitfall 7):** the corpus must contain queries that are NOT string-equal to any alias in the gazetteer. The typo and glued buckets are exactly those queries — and at least 15 cases overall must not equal any alias. The runner ASSERTS this (`cases.filter(c => c.kind !== 'negative' && !aliases.has(normalize(c.q))).length >= 15`), so a future regeneration that makes the corpus tautological fails the suite.
|
|
|
|
**Every `expectId` MUST match `/^[A-Za-z0-9_-]{1,64}$/`** — the start_param-safe charset (RESEARCH: "The corpus runner should assert every corpus `expectId` matches `/^[A-Za-z0-9_-]{1,64}$/`").
|
|
|
|
**Every positive's `expectId` MUST exist in the gazetteer** — asserted by the runner. A corpus case pointing at a non-existent entry fails the suite immediately (that is a corpus bug, not a search bug).
|
|
</corpus_format_contract>
|
|
|
|
<corpus_runner_contract>
|
|
`search/scripts/gazetteer-corpus.mjs` — the D-14 bake-off close. Copy `bot/scripts/register-commands.mjs`'s script conventions (shebang + a header contract block stating purpose, inputs, what is echoed, and the exit-code contract) and `bot/scripts/set-webhook.mjs`'s single-fetch-then-report shape.
|
|
|
|
Behaviour:
|
|
1. Import `search` from `../src/index.ts` (the same module the bot uses).
|
|
2. Read the corpus JSON from `search/test/fixtures/golden-corpus.json` via `readFileSync` + `JSON.parse` (a script may use `fs` — only the WORKER may not).
|
|
3. Split the cases into positives (`expectId !== null`) and negatives (`expectId === null`).
|
|
4. For each positive: `search(c.q, 3)` → record whether `hits[0].entry.id === c.expectId` (top-1) and whether `hits.map(h => h.entry.id).includes(c.expectId)` (top-3). Record the miss as `"<query> -> [<ids or —>] (expected <id>)"`.
|
|
5. For each negative: `search(c.q, 3)` → if `hits.length > 0`, record the leak as `"<query> -> <first id>"`.
|
|
6. Print a report to stdout:
|
|
- the overall line: `top-1 <n>/<N> top-3 <n>/<N> (<pct>%)`
|
|
- a PER-CATEGORY breakdown: one line per `kind` with that category's top-3 rate
|
|
- `negatives refused <n>/<N>`
|
|
- the `MISSED:` block (one line per miss) and the `LEAKED (must refuse):` block (one line per leak)
|
|
- the threshold line: `threshold top3 >= 90.0% — PASS|FAIL`
|
|
7. Exit code: `0` only when `top3Rate >= threshold.top3_hit_rate` AND `leaked.length === 0`; `1` otherwise.
|
|
|
|
**The runner MAY print query text** — it is a local dev tool over a committed file (RESEARCH: "The corpus runner (a local dev tool over a committed file) MAY print queries; the Worker NEVER may"). The runner MUST print the ESCALATION LADDER when it fails, verbatim:
|
|
|
|
```
|
|
D-14 threshold not met. Escalation order is FIXED:
|
|
(1) alias/normalizer table edits in search/ -> re-run this runner
|
|
(2) SearchSource backend #2 — the Malaysia-only WASM geocoder (D-05, post-v1)
|
|
(3) Photon-on-VPS (D-05, Phase 5)
|
|
Record which lever was pulled in docs/SEARCH-QUALITY.md.
|
|
```
|
|
|
|
**Also add a root delegate** `"search:corpus": "node search/scripts/gazetteer-corpus.mjs"` so the D-14 gate is one command.
|
|
|
|
**The runner must ALSO be assertable in-process.** `search/test/corpus.test.mjs` re-runs the same measurements by importing the shared module and the corpus directly (not by spawning the script) — so `npm run search:test` fails when the corpus regresses (the Nyquist requirement: "the corpus must be ≥ threshold at every wave merge, not just at the end"). The suite must assert the SAME things the script does: top-3 rate >= threshold, zero negative leaks, the tautology floor, the id charset, and that every positive's `expectId` exists in the gazetteer.
|
|
</corpus_runner_contract>
|
|
|
|
<overpass_script_contract>
|
|
`search/scripts/overpass-topup.mjs` — the D-06 maintainer tool. HARD LAWS:
|
|
|
|
**LAW 1 — it is a REPO TOOL, never a runtime dependency.** It lives under `search/scripts/`, is run manually, and is NEVER imported by `bot/src/` or `web/src/`. A grep gate asserts no Overpass/OSM URL appears under either source tree (RESEARCH Pitfall 6: "Warning signs: `fetch("https://overpass-api.de/…")` anywhere under `bot/src/` or `web/src/`").
|
|
|
|
**LAW 2 — ONE request per run.** The script makes exactly ONE Overpass request and prints the candidate list. It never loops over tag families.
|
|
|
|
**LAW 3 — an identifying User-Agent is MANDATORY.** A request without a UA gets HTTP 406 (live-verified 2026-10-23: `curl https://overpass-api.de/api/status` with no UA → HTTP 406). The header is exactly:
|
|
|
|
```
|
|
User-Agent: araharah-gazetteer-topup/1.0 (+https://github.com/araharah)
|
|
```
|
|
|
|
**LAW 4 — a busy instance is a NORMAL outcome, and the script EXITS 0.** Overpass capacity saturation is intermittent (RESEARCH §Verification A4 observed 502 and 200 minutes apart). On 502 / 429 / a network timeout / a non-200 status the script prints exactly one line:
|
|
|
|
```
|
|
Overpass instance busy — retry later
|
|
```
|
|
|
|
and exits **0** (never a stack trace, never a non-zero code). Nothing in the repo — no deploy, no test run, no corpus gate — may depend on this script succeeding (RESEARCH §Verification A4 item 4).
|
|
|
|
**LAW 5 — it NEVER writes to `kl-gazetteer.json`.** Output is a PRINTED candidate list (or a proposal file at an explicit `--out` path that the maintainer diffs and merges by hand). The script writes nothing to the curated dataset under any flag. The word "proposal" must appear in the output header.
|
|
|
|
**LAW 6 — the source and the endpoint are overridable.** `--source=overpass|nominatim|fixture` selects the candidate source (default `overpass`); `--endpoint=<url>` and the `OVERPASS_ENDPOINT` env var override the Overpass endpoint (default `https://overpass-api.de/api/interpreter`); `--fixture=<path>` names the fixture file for `--source=fixture`. `--source=nominatim --q=<term>` queries the reachable Nominatim search endpoint bounded to the KL viewbox instead (Nominatim was live-verified reachable 2026-10-23 while the Overpass mirrors were not). This is the resolved offline posture: the committed fixture is the deterministic test path, Nominatim is the alternative live path, and neither is ever a runtime dependency.
|
|
|
|
**The query shape** — a NARROW, single-tag-family query bounded to the KL bbox (Overpass bbox order is `s,w,n,e` = south, west, north, east):
|
|
|
|
```
|
|
[out:json][timeout:25];
|
|
nwr["tourism"="attraction"](3.05,101.55,3.25,101.80);
|
|
out center 40;
|
|
```
|
|
|
|
The bbox MUST be present (RESEARCH §Anti-Patterns: "An Overpass query with no bbox… will time out and is a well-known way to get rate-limited"). The default tag family is `tourism=attraction`; `--tag-family=` may override it with ONE family (never a union of families).
|
|
|
|
**The offline + alternate paths.** `--source=fixture --fixture=<path>` reads a committed Overpass JSON response instead of making a request — this is how the test exercises the script deterministically (RESEARCH §Verification A4: the committed fixture is the reliability anchor; the live primary instance works only with a `User-Agent` and saturates intermittently). `--source=nominatim --q=<term>` queries `https://nominatim.openstreetmap.org/search?format=jsonv2&q=<term>&viewbox=101.55,3.25,101.80,3.05&bounded=1&limit=40` with the same identifying-User-Agent discipline (Nominatim's usage policy requires one too) and emits the same PROPOSAL shape.
|
|
|
|
**The output format** — one candidate per line, with the fields a maintainer needs to decide:
|
|
|
|
```
|
|
PROPOSAL — 7 candidate(s) from tourism=attraction in bbox 3.05,101.55,3.25,101.80
|
|
Review and hand-merge into search/src/data/kl-gazetteer.json — this script NEVER writes it.
|
|
name=Merdeka 118 osm=way/12345678 lat=3.1418 lon=101.6943 category=landmark
|
|
name=National Monument osm=node/87654321 lat=3.1489 lon=101.6936 category=landmark
|
|
```
|
|
|
|
**`search/test/fixtures/overpass-response.json`** — a committed, REAL Overpass JSON response (the `elements` array shape: each element has `type`, `id`, `lat`/`lon` for nodes or `center: {lat, lon}` for ways/relations, and `tags`). Use 5-8 elements with realistic KL tag data. The fixture makes the top-up test deterministic — no live network in the test suite.
|
|
|
|
**`search/test/topup.test.mjs`** — cases:
|
|
- `TP1 the script is not imported by the Worker` — grep `bot/src/**` and `web/src/**` for `overpass` / `overpass-api.de` / `overpass-api` → ZERO hits.
|
|
- `TP2 the script sends an identifying User-Agent` — the script source contains `User-Agent: araharah-gazetteer-topup/1.0`.
|
|
- `TP3 the query is bbox-bounded` — the script source contains a bbox tuple matching `/\(\d+\.\d+,\s*\d+\.\d+,\s*\d+\.\d+,\s*\d+\.\d+\)/`.
|
|
- `TP4 the script exits 0 on a busy instance` — run the script with `--fixture=` pointing at a fixture whose content is NOT an Overpass response (e.g. a JSON object with `"error"`), assert the process exit code is 0 and stdout contains "Overpass instance busy".
|
|
- `TP5 the script produces a valid-shaped proposal from the committed fixture` — run the script with `--fixture=search/test/fixtures/overpass-response.json`, assert exit 0 and that stdout contains "PROPOSAL", at least one `name=` line, and the literal text "NEVER writes".
|
|
- `TP6 the script never writes the curated file` — assert the script source contains no `writeFileSync` targeting `kl-gazetteer.json` (the only permitted write is an explicit `--out` proposal path, and that path must not be the curated file — assert the source does not contain the literal `kl-gazetteer.json` at all).
|
|
- `TP7 exactly one request per run` — assert the script source contains exactly ONE `fetch(` call.
|
|
|
|
**`search/scripts/json-to-sqlite.mjs`** — the D-05 backend-#2 format converter. It reads `search/src/data/kl-gazetteer.json` and emits a SQLite file with the SAME ids, so backend #2's dataset format is reachable today (D-05: "The JSON↔SQLite converter script keeps its dataset format reachable from day one"). It is a STUB in the sense that the SQLite WRITER is a minimal, dependency-free implementation (Node 24 has `node:sqlite` behind a flag — check whether `node:sqlite` is available in Node 24.18.0; if it is, use it; if not, emit a deterministic, documented, plain-text `.sql` DUMP file plus a JSON sidecar and say so in the header — do NOT add a dependency).
|
|
|
|
**`search/test/converter.test.mjs`** — cases:
|
|
- `CV1 the converter preserves the id set` — run the converter into a temp directory, read the output's id list, and assert it is EXACTLY the gazetteer's id set (same members, same count).
|
|
- `CV2 the converter reads the committed dataset` — the script source references `kl-gazetteer.json`.
|
|
- `CV3 the converter is not a runtime dependency` — grep `bot/src/**` + `web/src/**` for `json-to-sqlite` / `sqlite` → ZERO hits.
|
|
|
|
**`docs/SEARCH-QUALITY.md`** — the bake-off close record. Structure:
|
|
|
|
```markdown
|
|
# Search quality — Phase 4 bake-off close
|
|
|
|
**Corpus:** `search/test/fixtures/golden-corpus.json` version 1 — <N> hand-written cases
|
|
**Gazetteer:** `search/src/data/kl-gazetteer.json` version 1 — <M> entries
|
|
**Measured:** <date>
|
|
|
|
| Metric | Value | Threshold |
|
|
|---|---|---|
|
|
| top-1 hit rate (positives) | <n>/<N> (<pct>%) | — (reported) |
|
|
| top-3 hit rate (positives) | <n>/<N> (<pct>%) | >= 90% |
|
|
| negatives refused | <n>/<N> | 100% (zero leaks) |
|
|
|
|
### Per-category
|
|
|
|
| Category | Cases | top-3 |
|
|
|---|---|---|
|
|
| landmark | … | … |
|
|
| abbreviation | … | … |
|
|
| glued | … | … |
|
|
| typo | … | … |
|
|
| alias | … | … |
|
|
|
|
### Escalation lever pulled (D-14)
|
|
|
|
<Exactly one of:>
|
|
1. **None — the threshold was met on the first run.** No table edits were needed.
|
|
2. **Lever 1 — alias/normalizer table edits.** <Which rows were added to `search/src/rules/abbreviations.ts` / which aliases were added to the gazetteer. The corpus was re-run; the result above is the post-edit measurement.>
|
|
3. **Lever 2 — SearchSource backend #2 (WASM geocoder).** <…>
|
|
4. **Lever 3 — Photon-on-VPS.** <…>
|
|
|
|
### Known limitations
|
|
|
|
- The corpus is hand-written and small; it measures the documented local shorthand, not every
|
|
phrasing a KL driver might use.
|
|
- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1.
|
|
```
|
|
|
|
**Acceptance:** `npm run search:corpus` exits 0 with top-3 >= 90% and zero negative leaks; `npm run search:test` green; `docs/SEARCH-QUALITY.md` records the measured numbers and the lever.
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>task 1: the pinned golden corpus + the D-14 runner + the Nyquist gate</name>
|
|
<files>search/test/fixtures/golden-corpus.json, search/scripts/gazetteer-corpus.mjs, search/test/corpus.test.mjs, search/package.json, package.json</files>
|
|
<read_first>
|
|
- `search/src/index.ts` (plan 04-01's output — the `search()` surface the runner calls)
|
|
- `search/src/data/kl-gazetteer.json` (plan 04-01's output — the ids and aliases the corpus must pin)
|
|
- `bot/scripts/register-commands.mjs` (the script conventions: shebang, header contract block, exit-code contract)
|
|
- `bot/scripts/set-webhook.mjs` (the single-fetch-then-report shape, and the "missing vars → fatal" resolution pattern)
|
|
- `bot/test/harness.mjs` + `bot/test/runner.mjs` (the suite style the new suite copies)
|
|
- `.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md` (§"The corpus runner" code example and the §"Common Pitfalls" #7 tautology law)
|
|
</read_first>
|
|
<action>
|
|
**`search/test/fixtures/golden-corpus.json`** — the pinned corpus, EXACTLY per the `<corpus_format_contract>` block: the header object (`version`, `note`, `threshold.top3_hit_rate: 0.9`) and at least 60 `cases` with the per-category floors (landmark >= 12, abbreviation >= 15, glued >= 6, typo >= 12, alias >= 12, negative >= 6).
|
|
|
|
Every `expectId` MUST be a real id in `search/src/data/kl-gazetteer.json`. The corpus's negatives MUST be genuinely out-of-area ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") so the refusal path is exercised.
|
|
|
|
**`search/scripts/gazetteer-corpus.mjs`** — the runner, EXACTLY per the `<corpus_runner_contract>` block: shebang + header contract block, the report format (overall line, PER-CATEGORY breakdown, negatives-refused line, `MISSED:` block, `LEAKED (must refuse):` block, threshold line), the fixed escalation ladder printed on failure, and the exit-code contract (`0` only when top-3 >= threshold AND zero leaks).
|
|
|
|
Add to `search/package.json`:
|
|
|
|
```json
|
|
"corpus": "node scripts/gazetteer-corpus.mjs"
|
|
```
|
|
|
|
and to the root `package.json`:
|
|
|
|
```json
|
|
"search:corpus": "node search/scripts/gazetteer-corpus.mjs",
|
|
```
|
|
|
|
**`search/test/corpus.test.mjs`** — the Nyquist gate (RED-first). Cases:
|
|
- `CO1 top-3 hit rate meets the D-14 threshold` — `top3 / positives.length >= corpus.threshold.top3_hit_rate`; the failure message names the D-14 threshold and the escalation ladder's first lever.
|
|
- `CO2 every negative is refused` — zero leaks; the message names the leaking query.
|
|
- `CO3 the corpus is not tautological` — at least 15 non-negative cases whose normalized query is NOT string-equal to any alias or name in the gazetteer (RESEARCH Pitfall 7).
|
|
- `CO4 every positive's expectId exists in the gazetteer` — no dangling ids.
|
|
- `CO5 every expectId matches the start_param-safe charset` — `/^[A-Za-z0-9_-]{1,64}$/`.
|
|
- `CO6 the corpus meets its size and category floors` — >= 60 cases; landmark >= 12, abbreviation >= 15, glued >= 6, typo >= 12, alias >= 12, negative >= 6.
|
|
- `CO7 every category's top-3 rate is reported` — the suite prints (not asserts) the per-category rates so the runner output and the suite output agree.
|
|
|
|
The suite imports the shared module and the corpus directly (in-process), so `npm run search:test` fails on a corpus regression.
|
|
|
|
**Acceptance:** `npm run search:corpus` exits 0; `npm run search:test` green with `PASS CO1`…`PASS CO7`.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -30 ; npm run search:test 2>&1 | grep -E 'PASS CO|runner' ; node -e "const c=require('./search/test/fixtures/golden-corpus.json'); if(c.cases.length<60) throw new Error('corpus too small: '+c.cases.length); console.log('corpus OK '+c.cases.length+' cases')"</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `npm run search:corpus` exits 0
|
|
- `npm run search:corpus` output contains a `top-3` line with a percentage >= 90.0% and a `negatives refused` line showing all negatives refused
|
|
- `npm run search:corpus` output contains one `PASS`/`FAIL` threshold line naming the 90.0% gate
|
|
- `npm run search:test` exits 0 and the output contains `PASS CO1` through `PASS CO7`
|
|
- `node -e "const c=require('./search/test/fixtures/golden-corpus.json'); const g=require('./search/src/data/kl-gazetteer.json'); const ids=new Set(g.entries.map(e=>e.id)); const kinds={}; let bad=0; for(const x of c.cases){ kinds[x.kind]=(kinds[x.kind]||0)+1; if(x.expectId!==null && !ids.has(x.expectId)) bad++; if(x.expectId!==null && !/^[A-Za-z0-9_-]{1,64}$/.test(x.expectId)) bad++; } if(c.cases.length<60) throw new Error('size'); if(bad) throw new Error('dangling ids'); console.log(JSON.stringify(kinds))"` exits 0 and prints all six `kind` keys with landmark>=12, abbreviation>=15, glued>=6, typo>=12, alias>=12, negative>=6
|
|
- `grep -q '"search:corpus": "node search/scripts/gazetteer-corpus.mjs"' package.json` exits 0 AND `grep -q '"corpus": "node scripts/gazetteer-corpus.mjs"' search/package.json` exits 0
|
|
- `grep -q 'D-14 threshold not met' search/scripts/gazetteer-corpus.mjs` exits 0 AND `grep -q 'backend #2' search/scripts/gazetteer-corpus.mjs` exits 0 AND `grep -q 'Photon-on-VPS' search/scripts/gazetteer-corpus.mjs` exits 0 (the fixed escalation ladder is printed verbatim on failure)
|
|
- `grep -c 'expectId' search/test/fixtures/golden-corpus.json | head -1` returns >= 60 (each case pins an expected id or null)
|
|
</acceptance_criteria>
|
|
<done>The golden corpus is pinned at 60+ hand-written cases across 6 categories with real negatives, the runner mechanically enforces D-14's top-3 >= 90% + zero-leak threshold, and `npm run search:test` fails whenever the corpus regresses.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>task 2: the Overpass top-up tool (D-06) + the JSON→SQLite converter (D-05)</name>
|
|
<files>search/scripts/overpass-topup.mjs, search/test/fixtures/overpass-response.json, search/test/topup.test.mjs, search/scripts/json-to-sqlite.mjs, search/test/converter.test.mjs</files>
|
|
<read_first>
|
|
- `bot/scripts/set-webhook.mjs` (the script conventions: shebang, header contract block, env/flag resolution, one fetch + a category-only failure print, the exit-code contract)
|
|
- `bot/scripts/register-commands.mjs` (the `.dev.vars` resolution pattern the `--endpoint` / `OVERPASS_ENDPOINT` override mirrors)
|
|
- `search/src/data/kl-gazetteer.json` (the dataset the converter reads and the ids it must preserve)
|
|
- `bot/test/privacy.test.mjs` (the source-introspection gate style the TP1/TP7 greps copy)
|
|
- `search/test/harness.mjs` (the suite style)
|
|
- `.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md` (§"Common Pitfalls" #6 and §"Verification ledger" A4 — the soft-exit posture)
|
|
</read_first>
|
|
<action>
|
|
**`search/scripts/overpass-topup.mjs`** — the D-06 tool, EXACTLY per the `<overpass_script_contract>` block's six laws: repo tool (never imported by `bot/src/` or `web/src/`), ONE request per run, the mandatory identifying User-Agent, a busy instance is a NORMAL outcome that EXITS 0, it NEVER writes the curated file, and the endpoint is overridable via `--endpoint=` or `OVERPASS_ENDPOINT`.
|
|
|
|
Header contract block (copy `set-webhook.mjs`'s shape — purpose, inputs, what is echoed/never echoed, exit-code contract):
|
|
|
|
```javascript
|
|
#!/usr/bin/env node
|
|
/*
|
|
* overpass-topup.mjs — propose KL gazetteer top-up candidates from OpenStreetMap (D-06).
|
|
* -------------------------------------------------------------------------------------
|
|
* A MAINTAINER-TIME REPO TOOL. It is NOT a runtime dependency: nothing in
|
|
* bot/src/ or web/src/ imports it, and no deploy, test, or corpus gate depends on
|
|
* it succeeding. The Worker only ever reads the COMMITTED curated JSON.
|
|
*
|
|
* Output is a PROPOSAL — a printed candidate list (or a file at an explicit --out
|
|
* path) that a human diffs and hand-merges into search/src/data/kl-gazetteer.json.
|
|
* THIS SCRIPT NEVER WRITES THE CURATED FILE.
|
|
*
|
|
* Exit codes: 0 on success AND on a busy instance (Overpass capacity saturation
|
|
* is intermittent — 502/429/timeout is a NORMAL, retryable outcome). Non-zero
|
|
* only on a usage error (no bbox resolvable / unreadable fixture).
|
|
*
|
|
* Inputs: --source=overpass|nominatim|fixture (default overpass), --endpoint=<url>
|
|
* (or OVERPASS_ENDPOINT), --bbox=s,w,n,e, --q=<term> (nominatim source),
|
|
* --tag-family=<key>=<value>, --fixture=<path> (offline input),
|
|
* --out=<path> (write the proposal here — never the curated file).
|
|
*/
|
|
```
|
|
|
|
The request carries EXACTLY this header set:
|
|
|
|
```javascript
|
|
const headers = {
|
|
"User-Agent": "araharah-gazetteer-topup/1.0 (+https://github.com/araharah)",
|
|
"Content-Type": "application/x-www-form-urlencoded",
|
|
};
|
|
```
|
|
|
|
The Overpass QL query is single-family and bbox-bounded, e.g.:
|
|
|
|
```
|
|
[out:json][timeout:25];
|
|
nwr["tourism"="attraction"](3.05,101.55,3.25,101.80);
|
|
out center 40;
|
|
```
|
|
|
|
On a non-200 status, a network error, or an unparseable response, print exactly:
|
|
|
|
```
|
|
Overpass instance busy — retry later
|
|
```
|
|
|
|
and `process.exit(0)`.
|
|
|
|
**`search/test/fixtures/overpass-response.json`** — a committed, realistic Overpass JSON response with 6 elements: a mix of `node` elements (with `lat`/`lon`) and `way`/`relation` elements (with `center: {lat, lon}`), each carrying `tags` with `name` and `name:en` plus a category-ish tag (`tourism`, `amenity`, `railway`, `office`, `leisure`). The elements must be real KL places (e.g. "Merdeka 118", "National Monument", "Masjid Jamek", "KL Tower", "Central Market", "Petaling Street") with plausible coordinates inside the KL bbox.
|
|
|
|
**`search/test/topup.test.mjs`** — the seven cases `TP1`…`TP7` from the `<overpass_script_contract>` block. `TP4` and `TP5` RUN the script as a child process (`node scripts/overpass-topup.mjs ...`) from the `search/` workspace directory and assert the exit code and the stdout shape. `TP4` uses a temp fixture that is NOT an Overpass response (e.g. `{"error":"busy"}`) to exercise the soft-exit path.
|
|
|
|
**`search/scripts/json-to-sqlite.mjs`** — the D-05 converter. It reads `search/src/data/kl-gazetteer.json` and emits a dataset with the SAME ids so backend #2's format is reachable from day one (D-05). The header block states that it is a repo tool, never a runtime dependency, and that the ODbL notice MUST travel with the output (ODbL §4.4 share-alike on a derived database).
|
|
|
|
**Node's SQLite availability:** Node 24.18.0 ships `node:sqlite` (stable-ish behind no flag in Node 22.5+/24). PROBE IT FIRST in this task with `node -e "const {DatabaseSync}=require('node:sqlite'); console.log('sqlite OK')"`. If it works, emit a real `.sqlite` file with a `places` table (`id TEXT PRIMARY KEY, name TEXT, aliases TEXT, lat REAL, lng REAL, category TEXT, address TEXT`). If it does NOT work, emit a deterministic `.sql` dump file plus a JSON sidecar and say so plainly in the header — do NOT add a dependency.
|
|
|
|
**`search/test/converter.test.mjs`** — the three cases `CV1`…`CV3` from the `<overpass_script_contract>` block. `CV1` runs the converter into a temp directory and asserts the emitted id set is EXACTLY the gazetteer's id set (same members, same count) — reading the ids back out of whatever format was emitted (SQLite via `node:sqlite`, or the `.sql` dump's INSERT lines).
|
|
|
|
**Acceptance:** `npm run search:test` green with `PASS TP1`…`PASS TP7` and `PASS CV1`…`PASS CV3`; the top-up script exits 0 on the fixture AND on a busy-shaped input.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/C-Fu/araharah && npm run search:test 2>&1 | grep -E 'PASS TP|PASS CV|runner' ; cd search && node scripts/overpass-topup.mjs --fixture=test/fixtures/overpass-response.json 2>&1 | head -12 ; echo "exit=$?" ; node -e "const {DatabaseSync}=require('node:sqlite'); console.log('node:sqlite available')" 2>&1 | tail -1</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `npm run search:test` exits 0 and the output contains `PASS TP1` through `PASS TP7` and `PASS CV1` through `PASS CV3`
|
|
- `cd search && node scripts/overpass-topup.mjs --fixture=test/fixtures/overpass-response.json` exits 0 and its stdout contains `PROPOSAL`, at least one `name=` line, and the literal text `NEVER writes`
|
|
- `cd search && node scripts/overpass-topup.mjs --fixture=<a non-Overpass fixture>` exits 0 and its stdout contains exactly the line `Overpass instance busy — retry later`
|
|
- `grep -q 'User-Agent: araharah-gazetteer-topup/1.0' search/scripts/overpass-topup.mjs` exits 0 (HTTP 406 without it — live-verified)
|
|
- `grep -q '\-\-source' search/scripts/overpass-topup.mjs` exits 0 (the source flag exists: overpass | nominatim | fixture — the resolved-question posture)
|
|
- `grep -c 'await fetch(\|fetch(' search/scripts/overpass-topup.mjs | head -1` returns 1 (EXACTLY ONE request per run)
|
|
- `grep -c 'kl-gazetteer.json' search/scripts/overpass-topup.mjs | head -1` returns 0 (the script never names the curated file — it cannot write it)
|
|
- `grep -rl 'overpass' bot/src/ web/src/ 2>/dev/null | wc -l` returns 0 (the script is never a runtime dependency — RESEARCH Pitfall 6)
|
|
- `grep -rl 'sqlite\|json-to-sqlite' bot/src/ web/src/ 2>/dev/null | wc -l` returns 0
|
|
- `node -e "const g=require('./search/src/data/kl-gazetteer.json'); const {execFileSync}=require('node:child_process'); const fs=require('node:fs'); const os=require('node:os'); const path=require('node:path'); const out=fs.mkdtempSync(path.join(os.tmpdir(),'araharah-conv-')); execFileSync('node',['search/scripts/json-to-sqlite.mjs','--out',out],{cwd:process.cwd()}); const files=fs.readdirSync(out); if(!files.length) throw new Error('converter emitted nothing'); const blob=files.map(f=>fs.readFileSync(path.join(out,f),'utf8')).join('\n'); const ids=g.entries.map(e=>e.id); const missing=ids.filter(id=>!blob.includes(id)); if(missing.length) throw new Error('id set not preserved: '+missing.join(',')); console.log('converter OK -> '+files.join(','));"` exits 0
|
|
</acceptance_criteria>
|
|
<done>Both maintainer-time repo tools ship and are proven offline: the Overpass top-up makes exactly one identified request, prints a proposal, never writes the curated file, and exits 0 on a busy instance; the JSON→SQLite converter preserves the gazetteer's id set exactly, so backend #2's dataset format is reachable today.</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>task 3: run the bake-off, close it, and record the decision</name>
|
|
<files>docs/SEARCH-QUALITY.md, search/test/fixtures/golden-corpus.json, search/src/data/kl-gazetteer.json, search/src/rules/abbreviations.ts</files>
|
|
<read_first>
|
|
- `search/scripts/gazetteer-corpus.mjs` (task 1's output — the instrument that measures the bake-off)
|
|
- `search/test/fixtures/golden-corpus.json` (task 1's output — the pinned corpus being measured)
|
|
- `search/src/data/kl-gazetteer.json` (plan 04-01's output — the curated dataset under measurement)
|
|
- `search/src/rules/abbreviations.ts` (plan 04-01's output — the lever-1 table the escalation edits target)
|
|
- `.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md` (D-13 and D-14 — the exact threshold and the exact escalation order)
|
|
</read_first>
|
|
<action>
|
|
**Run the bake-off.** Execute `npm run search:corpus` and record the real output — top-1, top-3, the per-category breakdown, the negatives-refused count, and every MISSED and LEAKED line.
|
|
|
|
**If the threshold is met** (top-3 >= 90% on positives AND zero negative leaks): the escalation lever is **"None"**. Record that in `docs/SEARCH-QUALITY.md` and make NO corpus/table/data edits to force a pass.
|
|
|
|
**If the threshold is NOT met:** pull the escalation ladder in the FIXED order (D-14) and re-run after each pull:
|
|
1. **Lever 1 — alias/normalizer table edits.** For each MISSED case, decide whether the miss is an ALIAS gap (add the local shorthand to the entry's `aliases` in `search/src/data/kl-gazetteer.json`) or an ABBREVIATION gap (add a row to `search/src/rules/abbreviations.ts` — a table edit, never a code change). Add a per-rule test vector to `search/test/normalizer.test.mjs` for every new abbreviation row (D-09: "Per-rule test vectors; extends by table edit, not code change"). Re-run `npm run search:corpus` and `npm run search:test`.
|
|
2. **Lever 2 — SearchSource backend #2 (the Malaysia-only WASM geocoder).** This backend does NOT ship in v1 (D-05) and MUST NOT be implemented here. If lever 1 cannot reach the threshold, RECORD that fact in `docs/SEARCH-QUALITY.md` as the documented outcome and leave the threshold unmet with the lever named — the phase's experiment ends with a documented decision (D-14), not with an open question and not with an unbuilt backend smuggled in.
|
|
3. **Lever 3 — Photon-on-VPS.** Same handling as lever 2 — record, do not build (D-03 defers VPS provisioning to Phase 5).
|
|
|
|
**NEVER edit the corpus to make it pass.** The corpus is PINNED (D-13). A missed case is fixed by improving the DATA or the RULES, never by changing the expectation. If a corpus case is simply WRONG (its `expectId` points at an entry that is not the right landmark), fix the corpus's `expectId` and say so explicitly in `docs/SEARCH-QUALITY.md` under "Corpus corrections" — that is a corpus bug, and it must be visible.
|
|
|
|
**Do NOT weaken the admission rule** to make positives pass. The tiered admission (non-fuzzy tiers always; fuzzy only at similarity >= `FUZZY_ADMIT_SIMILARITY` 0.72 — plan 04-01's contract) is what makes the negatives honest (D-13/D-14). If a positive can only be found by lowering that number, the correct fix is an alias or a table row.
|
|
|
|
**Coordinate verification (the resolved-question offline posture).** Spot-verify at least 8 of the curated seed entries' coordinates against Nominatim (reachable per RESEARCH §Environment Availability): for each sampled entry, `GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=<entry name>, Kuala Lumpur&limit=1` with an identifying User-Agent (the same discipline as the Overpass script — a missing UA is the 406 trap), and confirm the returned lat/lon lands within ~0.01° of the entry's committed coordinates. Record the sample size, the method, and the outcome in `docs/SEARCH-QUALITY.md` under a "Coordinate verification" section (N of M sampled within tolerance — or, if Nominatim is unreachable at run time, record that honestly and name the committed-bbox gate G5 as the fallback invariant). This is a one-shot maintainer-time check recorded in the doc — NOT a test-suite dependency and never a runtime path.
|
|
|
|
**Write `docs/SEARCH-QUALITY.md`** — EXACTLY per the `<overpass_script_contract>` block's structure: the header (corpus version + case count, gazetteer version + entry count, the measurement date), the metric table (top-1, top-3 vs the 90% threshold, negatives refused vs the zero-leak requirement), the per-category table, the "Escalation lever pulled" section naming exactly one lever (or "None"), the "Corpus corrections" section (empty if none), the "Coordinate verification" section, and the "Known limitations" section (the hand-written corpus measures the documented local shorthand, not every phrasing; out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1).
|
|
|
|
**Acceptance:** `npm run search:corpus` exits 0; `docs/SEARCH-QUALITY.md` records the real measured numbers; the lever section names one lever honestly.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -25 ; npm run search:test 2>&1 | tail -3 ; test -f docs/SEARCH-QUALITY.md && grep -c 'top-3' docs/SEARCH-QUALITY.md && grep -c 'Escalation lever pulled' docs/SEARCH-QUALITY.md && grep -c 'Known limitations' docs/SEARCH-QUALITY.md</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `npm run search:corpus` exits 0
|
|
- `npm run search:test` exits 0 (any lever-1 edit kept every other suite green)
|
|
- `test -f docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'top-3' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'negatives refused' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'Escalation lever pulled' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'Coordinate verification' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'Known limitations' docs/SEARCH-QUALITY.md` exits 0
|
|
- `grep -q 'Lever 1\|Lever 2\|Lever 3\|None — the threshold was met' docs/SEARCH-QUALITY.md` exits 0 (exactly one lever is named)
|
|
- `grep -c 'search:corpus' docs/SEARCH-QUALITY.md | head -1` returns >= 1 (the record names how to reproduce the measurement)
|
|
- `git diff --stat search/test/fixtures/golden-corpus.json` shows NO expectation edits that lowered a threshold — the corpus's `threshold.top3_hit_rate` is still `0.9`: `grep -q '"top3_hit_rate": 0.9' search/test/fixtures/golden-corpus.json` exits 0
|
|
- `grep -q 'search/src/rules/abbreviations.ts' docs/SEARCH-QUALITY.md` exits 0 when lever 1 was pulled, OR the doc names lever 2/3 instead (the lever section is specific)
|
|
</acceptance_criteria>
|
|
<done>The bake-off is closed with a measured, recorded, honest outcome: top-3 >= 90% on positives, every negative refused, and exactly one named escalation lever (or "None") recorded in `docs/SEARCH-QUALITY.md` with the reproduction command.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<threat_model>
|
|
## Trust Boundaries
|
|
|
|
| Boundary | Description |
|
|
|----------|-------------|
|
|
| Committed corpus file → the measured quality claim | The corpus is the instrument that decides whether the phase ships. A corpus edited to pass is a falsified instrument. |
|
|
| Live Overpass endpoint → the maintainer's terminal | A third-party service with no SLA, intermittent capacity, and an egress filter that 406s requests without a User-Agent. |
|
|
| Overpass response → the curated dataset | A candidate list is UNTRUSTED third-party data. It must never be written straight into the curated file. |
|
|
| Corpus runner (local tool) → stdout | The runner MAY print query text (it is a local dev tool over committed files); the Worker never may. The runner's output must not be shipped anywhere. |
|
|
|
|
## STRIDE Threat Register
|
|
|
|
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
|
|-----------|----------|-----------|-------------|-----------------|
|
|
| T-04-12 | Tampering | The corpus being edited so the gate always passes (RESEARCH Pitfall 7 — "the corpus becomes a generated artifact") | mitigate | The corpus is pinned and hand-written; the tautology gate (CO3) requires >= 15 non-negative cases whose normalized query is NOT string-equal to any gazetteer alias — a regenerated corpus fails it. `threshold.top3_hit_rate` stays `0.9` (asserted in the acceptance criteria). The plan explicitly forbids editing a case's `expectId` to make it pass, and any legitimate corpus correction must be listed under "Corpus corrections" in `docs/SEARCH-QUALITY.md` so it is visible. |
|
|
| T-04-13 | Repudiation | The phase closing with an unmeasured or unfalsified quality claim (D-14: "the experiment ends with a documented decision, not an open question") | mitigate | `npm run search:corpus` is a single command with a mechanical exit code; `docs/SEARCH-QUALITY.md` must carry the measured top-1/top-3/refusal numbers, the per-category table, exactly one named lever, and the reproduction command. The "Escalation lever pulled" section is mandatory content, not an optional note. |
|
|
| T-04-14 | Denial of Service | The Overpass script becoming a runtime dependency (SSRF-ish: the Worker fetching a third-party service on every cold start) | mitigate | LAW 1 plus the TP1 grep gate: `bot/src/` and `web/src/` contain zero occurrences of `overpass`. The script lives under `search/scripts/`, is never imported by product code, and the curated JSON is the only thing the Worker reads. |
|
|
| T-04-15 | Denial of Service | The Overpass script blocking a build, a test, or a deploy because the endpoint is down or busy | mitigate | LAW 4: 502 / 429 / timeout / non-200 / unparseable → print `Overpass instance busy — retry later` and exit 0. The TP4 case asserts the exit code is 0 on a busy-shaped input. No repo script depends on this script's success (RESEARCH §Verification A4 item 4). |
|
|
| T-04-16 | Denial of Service | An Overpass query without a bbox causing a global scan that times out and gets the maintainer's IP rate-limited | mitigate | LAW: the query is always bbox-bounded to the KL box (`(3.05,101.55,3.25,101.80)`), and TP3 asserts the bbox tuple is present in the source. The query is single-family and narrow by default, which is what fits a free worker slot. |
|
|
| T-04-17 | Spoofing | The Overpass endpoint serving a response that makes the maintainer merge hostile data into the curated dataset | mitigate | The script OUTPUT is a printed proposal only — LAW 5 plus the TP6 assertion that the script's source never names `kl-gazetteer.json` at all, so it CANNOT write it. A human diffs and hand-merges. The curated dataset's own invariants (unique ids, slug charset, KL-bbox coordinates, non-empty aliases) are asserted by `search/test/gazetteer.test.mjs` (plan 04-01), so hostile coordinates or a duplicate id would fail the suite even if a human merged them by mistake. |
|
|
| T-04-18 | Information Disclosure | The corpus runner's query-printing leaking into a shipped log | mitigate | The runner is a local dev tool under `search/scripts/`; it is never imported by `bot/src/` or `web/src/` and no product code calls it. The Worker-side logging law (category-only) is enforced separately in plan 04-03's source-introspection gate. The runner's output goes to a maintainer's terminal, not to a log aggregator. |
|
|
| T-04-19 | Tampering | The D-14 threshold being lowered (e.g. `0.9` → `0.8`) to make a failing corpus pass | mitigate | The threshold lives in the corpus file as a single literal `0.9`, and the acceptance criteria assert `grep -q '"top3_hit_rate": 0.9'` still holds after the bake-off. D-14 fixes the number as 90%; changing it is a decision change, not a tuning knob. |
|
|
| T-04-20 | Information Disclosure | The JSON→SQLite converter emitting the dataset WITHOUT the ODbL notice, producing an unattributed derivative database (ODbL §4.4 share-alike) | mitigate | The converter's header block states that the ODbL notice MUST travel with the output, and the emitted artifact carries the `_attribution` / `_license` / `_notice` fields in a metadata table (or as a header comment in the `.sql` dump). The converter is a repo tool — the notice travels with the data by construction. |
|
|
</threat_model>
|
|
|
|
<verification>
|
|
- `npm run search:corpus` exits 0 with top-3 >= 90% and zero negative leaks, and prints the per-category breakdown
|
|
- `npm run search:test` green (the corpus gate is inside the suite — a regression fails `npm run search:test`, not just the standalone runner)
|
|
- `npm run search:typecheck` green
|
|
- `grep -rl 'overpass' bot/src/ web/src/` returns NOTHING — the top-up tool is provably not a runtime dependency
|
|
- `grep -rl 'sqlite' bot/src/ web/src/` returns NOTHING — the backend-#2 converter is provably not a runtime dependency
|
|
- `docs/SEARCH-QUALITY.md` exists and carries the measured numbers, the per-category table, exactly one named lever, and the reproduction command
|
|
- `npm test -w bot` and `npm test -w web` still green (this plan touches no bot/web source)
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- D-13: a pinned golden corpus of 60+ hand-written cases exists at `search/test/fixtures/golden-corpus.json`, with 6 categories and at least 15 non-tautological cases, each positive pinning its expected entry id
|
|
- D-14: the bake-off CLOSES on a mechanical threshold — `npm run search:corpus` exits 0 only when top-3 >= 90% on positives AND every negative is refused — and the measured outcome plus the lever pulled are recorded in `docs/SEARCH-QUALITY.md`
|
|
- D-06: the Overpass top-up is a repo tool that proposes candidates for human approval, never writes the curated file, sends an identifying User-Agent, makes exactly one request, and exits 0 on a busy instance — proven offline against a committed fixture
|
|
- D-05: the JSON→SQLite converter exists, reads the same committed dataset, and preserves the id set exactly — backend #2's dataset format is reachable today
|
|
- SRCH-01/SRCH-02 are MEASURED: the golden landmarks and the abbreviation/typo/glued/alias buckets each have a measured top-3 rate, so a later regression is diagnosable per category rather than as one diffuse number
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/04-destination-search-kl-gazetteer/04-02-SUMMARY.md`
|
|
</output>
|