araharah-executor
3aec9542a3
- SUMMARY: 3/3 tasks, 5 commits, corpus gate green at 100% top-3 / 7-7 refusals, lever None; deviations documented (fixture-source implication, bbox pin, pavilion-kl coordinate fix)
201 lines
12 KiB
Markdown
201 lines
12 KiB
Markdown
---
|
|
phase: 04-destination-search-kl-gazetteer
|
|
plan: 02
|
|
subsystem: search
|
|
tags: [search, golden-corpus, d14-gate, overpass, sqlite, nominatim, tdd]
|
|
requires:
|
|
- "plan 04-01 search workspace (contracts, normalizer, scorer, gazetteer, loader)"
|
|
- "Node 24 type-stripping (.mjs script imports ../src/index.ts)"
|
|
- "node:sqlite (built into Node 24.18.0 — converter output path)"
|
|
provides:
|
|
- "D-13 golden corpus v1 — 75 pinned hand-written cases in 6 categories with real negatives"
|
|
- "D-14 gate — `npm run search:corpus` (exit 0 only on top-3 >= 0.9 + zero leaks) and the in-process CO1-CO8 gate inside `npm run search:test`"
|
|
- "D-06 tool — search/scripts/overpass-topup.mjs (one UA'd request, PROPOSAL output, soft-exit 0 on busy, --source=overpass|nominatim|fixture)"
|
|
- "D-05 tool — search/scripts/json-to-sqlite.mjs (same id set, real .sqlite via node:sqlite, ODbL notice embedded)"
|
|
- "docs/SEARCH-QUALITY.md — the bake-off close: measured rates, lever None, coordinate verification"
|
|
affects:
|
|
- "plan 04-03 (bot /cari handler ships against a closed, green corpus gate)"
|
|
- "plan 04-04 (web deep-link reader — same module, corpus unchanged)"
|
|
- "future top-ups (the Overpass proposer + SEARCH-QUALITY record define the maintenance loop)"
|
|
tech-stack:
|
|
added:
|
|
- "node:sqlite DatabaseSync (converter only — repo tool, zero added deps)"
|
|
patterns:
|
|
- "measure-once/report-twice: the runner exports measureCorpus(); the suite asserts the same numbers in-process"
|
|
- "main-guarded .mjs scripts (CLI when invoked, importable library otherwise)"
|
|
- "source-introspection law gates (TP1/TP6/TP7, CV2/CV3) following bot/test/privacy.test.mjs"
|
|
key-files:
|
|
created:
|
|
- search/test/fixtures/golden-corpus.json
|
|
- search/scripts/gazetteer-corpus.mjs
|
|
- search/test/corpus.test.mjs
|
|
- search/scripts/overpass-topup.mjs
|
|
- search/test/fixtures/overpass-response.json
|
|
- search/scripts/json-to-sqlite.mjs
|
|
- search/test/topup.test.mjs
|
|
- search/test/converter.test.mjs
|
|
- docs/SEARCH-QUALITY.md
|
|
modified:
|
|
- search/package.json (corpus delegate)
|
|
- package.json (search:corpus delegate)
|
|
- search/src/data/kl-gazetteer.json (pavilion-kl coordinate fix)
|
|
decisions:
|
|
- "Corpus authored to the shipped 30-entry gazetteer per the plan's ILLUSTRATIVE clause: dropped non-existent targets (Intermark, Cheras South, MV — 2-char queries sit below the scorer's min-length guard by design) and pinned equivalents that exist (IOI, Pudu, Tmn Perdana, Kl Towr, ...)"
|
|
- "Near-miss negatives chosen with measured margins: Penang 0.625 and Kampung Cina ~0.58, both in [0.55, 0.72) — lowering the admission line leaks them and fails CO2+CO8"
|
|
- "measureCorpus() exported from the runner and reused by corpus.test.mjs — one measurement, two reporters, no drift"
|
|
- "--fixture=<path> implies --source=fixture (a fixture flag silently hitting the live network would be a trap)"
|
|
- "Converter emits a REAL .sqlite via node:sqlite (probed available in Node 24.18.0) — no dependency added; ODbL notice embedded in a metadata table + NOTICE sidecar"
|
|
metrics:
|
|
duration: "~14 min executor time"
|
|
completed: 2026-10-23
|
|
tasks: 3
|
|
filesCreated: 9
|
|
filesModified: 3
|
|
---
|
|
|
|
# Phase 4 Plan 2: Golden Corpus + Bake-off Close Summary
|
|
|
|
**One-liner:** the D-13/D-14 experiment closed mechanically — a 75-case pinned corpus measured
|
|
top-3 68/68 (100%) with 7/7 negatives refused at margin 0.643 < 0.72 on the FIRST run
|
|
(lever: None), enforced twice (`npm run search:corpus` exit-code gate + CO1-CO8 inside
|
|
`npm run search:test`), alongside the two forkability tools: an Overpass top-up proposer
|
|
(one UA'd request, PROPOSAL-only output, exit-0-on-busy) and a JSON→SQLite converter that
|
|
preserves the id set exactly.
|
|
|
|
## What Was Built
|
|
|
|
### Task 1 — pinned golden corpus + D-14 runner + Nyquist gate (RED `dea7971` → GREEN `b2689f7`)
|
|
- `golden-corpus.json`: 75 hand-written cases — landmark 17, abbreviation 15, glued 7,
|
|
typo 14, alias 15, negative 7 — every positive pinning a real gazetteer id
|
|
(start_param-safe charset), `threshold.top3_hit_rate: 0.9`. 16 non-negative cases are not
|
|
string-equal to any alias/name (tautology floor is 15). Negatives include two MEASURED
|
|
near-misses (Penang 0.625, Kampung Cina ~0.58) so a lowered admission line fails CO2+CO8.
|
|
- `gazetteer-corpus.mjs`: imports the SAME `src/index.ts` the bot uses (D-03), measures
|
|
top-1/top-3 + per-category rates + avg ms/query + the refusal margin, prints MISSED/LEAKED
|
|
blocks, the threshold line, and the verbatim escalation ladder on failure; exits 0 only on
|
|
threshold + zero leaks. Exports `measureCorpus()`; main-guarded so the suite can import it.
|
|
- `corpus.test.mjs`: CO1-CO8 — threshold, zero leaks, tautology floor, dangling-id charset,
|
|
size+category floors, per-category reporting, and the pinned refusal margin (replicating
|
|
the scorer's two-level whole-string+token fuzzy shape over `damerau`).
|
|
- Delegates: `search:corpus` (root), `corpus` (search workspace).
|
|
- Measured: **top-3 68/68 (100.0%), negatives refused 7/7, margin 0.643 < 0.72, 1.07 ms/query —
|
|
threshold met on the first run.** The two top-1 misses ("KCLC" 3-way fuzzy tie broken
|
|
id-ascending; "Sunway Vellocity" legitimately riding PREFIX via the "Sunway" alias) are
|
|
documented expected behaviour.
|
|
|
|
### Task 2 — Overpass top-up (D-06) + JSON→SQLite converter (D-05) (RED `ec98c8d` → GREEN `c6a5f2b`)
|
|
- `overpass-topup.mjs`: maintainer-time repo tool under the six hard laws — never a runtime
|
|
dependency (TP1), ONE request per run via a single shared `requestJson` helper for both the
|
|
Overpass and Nominatim paths (TP7), mandatory identifying User-Agent (TP2 — live-verified
|
|
406-without), bbox-bounded single-family query pinned verbatim (TP3), busy/502/429/timeout/
|
|
unparseable → `Overpass instance busy — retry later` + exit 0 (TP4), PROPOSAL-only output
|
|
whose source never names the curated file (TP6). Flags: `--source=overpass|nominatim|fixture`,
|
|
`--endpoint=`/`OVERPASS_ENDPOINT`, `--bbox=`, `--q=`, `--tag-family=`, `--fixture=`, `--out=`.
|
|
`--fixture` implies the fixture source (offline by default when a fixture is named).
|
|
- `overpass-response.json`: committed real-shape Overpass response (6 KL elements — nodes with
|
|
lat/lon, ways/relation with center) for deterministic offline tests.
|
|
- `json-to-sqlite.mjs`: reads the same committed `kl-gazetteer.json`, emits
|
|
`kl-gazetteer.sqlite` (places + metadata tables) via `node:sqlite` — zero added
|
|
dependencies — with the SAME id set (CV1 reads it back through DatabaseSync) and the ODbL
|
|
notice embedded in the metadata table + a NOTICE sidecar (ODbL §4.4, T-04-20). Deterministic:
|
|
dataset order, no timestamps.
|
|
- Suites: TP1-TP7 + CV1-CV3 all green; grep gates confirm zero `overpass`/`sqlite` mentions
|
|
under `bot/src/` and `web/src/`.
|
|
|
|
### Task 3 — bake-off close + coordinate verification (`99bc9c9`)
|
|
- `docs/SEARCH-QUALITY.md`: the close record — metric table, per-category table, escalation
|
|
lever section naming exactly one lever (**None — met on the first run; no
|
|
abbreviations.ts row, no alias, no corpus edit**), corpus corrections (none, threshold
|
|
still 0.9), reproduction commands, known limitations (hand-written corpus scope,
|
|
refusal-not-routing in v1, the recorded fuzzy-tier scaling decision).
|
|
- Coordinate verification (live Nominatim, identifying UA, 1 req/s, 10 seed entries):
|
|
**8/10 within 0.01°.** `bukit-bintang` outlier = area-vs-point discrepancy (no change).
|
|
`pavilion-kl` = real ~2 km coordinate error in the 04-01 seed → corrected to
|
|
(3.149, 101.7132), post-fix delta 0.0002° (Rule 1 fix below).
|
|
- Full sweep green: `search:test` 57/57, `search:typecheck`, `bot:test` 50/50, `web:test` 33/33.
|
|
|
|
## Commits
|
|
|
|
| task | commit | message |
|
|
| ---- | ------ | ------- |
|
|
| 1 RED | `dea7971` | test(04-02): RED — golden-corpus D-14 gate suite (CO1-CO8) |
|
|
| 1 GREEN | `b2689f7` | feat(04-02): pinned golden corpus + D-14 corpus runner + Nyquist gate |
|
|
| 2 RED | `ec98c8d` | test(04-02): RED — D-06 topup law suite (TP1-TP7) + D-05 converter suite (CV1-CV3) |
|
|
| 2 GREEN | `c6a5f2b` | feat(04-02): Overpass top-up proposer (D-06) + JSON-to-SQLite converter (D-05) |
|
|
| 3 | `99bc9c9` | docs(04-02): close the D-14 bake-off — threshold met on the first run, lever None |
|
|
|
|
## Verification Results
|
|
|
|
- `npm run search:corpus` — exit 0: `top-3 68/68 (100.0%)`, `negatives refused 7/7`,
|
|
`refusal margin 0.643 < 0.72`, `threshold top3 >= 90.0% — PASS`
|
|
- `npm run search:test` — **57 passed, 0 failed** (CO1-CO8 + TP1-TP7 + CV1-CV3 + the 39 from 04-01)
|
|
- `npm run search:typecheck` — green
|
|
- `npm test -w bot` (50) / `npm test -w web` (33) — green, untouched
|
|
- `grep -rl 'overpass' bot/src/ web/src/` → 0; `grep -rl 'sqlite' bot/src/ web/src/` → 0
|
|
- Overpass script proven offline: committed fixture → 6-candidate PROPOSAL, exit 0;
|
|
busy-shaped fixture → exactly the soft-exit line, exit 0
|
|
- Converter acceptance probe: id set preserved exactly (sqlite round-trip)
|
|
|
|
## Deviations from Plan
|
|
|
|
**1. [Rule 1 - Bug] `--fixture=` silently hit the live network**
|
|
- **Found during:** task 2 (GREEN probes)
|
|
- **Issue:** the first manual probe passed `--fixture=...` without `--source=fixture`; the
|
|
script fell through to a live Overpass request, so the busy-fixture acceptance check would
|
|
have printed a live PROPOSAL instead of the soft-exit line.
|
|
- **Fix:** presence of `--fixture` now implies the fixture source. (The accidental live run
|
|
made exactly one UA'd request — LAW 2 was never violated — and usefully proved the live
|
|
Overpass path works from this environment with the mandated User-Agent.)
|
|
- **Files:** search/scripts/overpass-topup.mjs
|
|
|
|
**2. [Rule 1 - Bug] TP3 failed — no literal bbox tuple in source**
|
|
- **Found during:** task 2 (GREEN)
|
|
- **Issue:** the query interpolated `(${bbox})`, so the source contained no literal
|
|
parenthesized bbox tuple for TP3's regex.
|
|
- **Fix:** the canonical default query is now pinned verbatim —
|
|
`nwr["tourism"="attraction"](3.05,101.55,3.25,101.80)` — with flag substitutions building
|
|
the same shape for overrides.
|
|
- **Files:** search/scripts/overpass-topup.mjs
|
|
|
|
**3. [Rule 1 - Bug] pavilion-kl committed coordinate ~2 km west of the mall**
|
|
- **Found during:** task 3 (the plan's prescribed Nominatim spot-verification)
|
|
- **Issue:** seed entry carried lng 101.695; Nominatim (and the mall's true position) place
|
|
Pavilion Kuala Lumpur at ~(3.1492, 101.7130).
|
|
- **Fix:** entry corrected to (3.149, 101.7132); recorded in SEARCH-QUALITY.md under
|
|
"Coordinate verification". Corpus is coordinate-blind; all suites stayed green.
|
|
- **Files:** search/src/data/kl-gazetteer.json, docs/SEARCH-QUALITY.md
|
|
|
|
**4. [Planner-note] corpus illustrative strings resolved against the real gazetteer**
|
|
- The plan marks its query strings ILLUSTRATIVE and requires pinning to existing ids. Targets
|
|
with no gazetteer entry (The Intermark, Cheras South) and queries the scorer must refuse by
|
|
design ("MV" — 2 chars, below the min-length guard) were replaced with equivalent pinned
|
|
cases (IOI, Pudu, Tmn Perdana, Kl Towr, Merdeaka 118, ...). Floors all exceeded.
|
|
|
|
## Auth Gates
|
|
|
|
None — no secrets, no auth flows. The only network surfaces are maintainer-time repo tools
|
|
(Overpass/Nominatim/Nominatim-verify), all of which worked from this environment with
|
|
identifying User-Agents.
|
|
|
|
## Known Stubs
|
|
|
|
None in the shipped-search sense — every artifact is wired to real behavior. The plan's
|
|
"stub" language for the converter means backend #2 itself is deliberately unbuilt (D-05,
|
|
post-v1); the converter that keeps its format reachable is a full, working implementation.
|
|
|
|
## Threat Flags
|
|
|
|
None — the built surface is exactly the plan's threat model (T-04-12…T-04-20): the corpus
|
|
tampering gates (CO3/CO6 + the 0.9 threshold grep), the Overpass runtime-dependency and
|
|
never-writes laws (TP1/TP6/TP7), the busy-soft-exit (TP4), and the ODbL notice traveling
|
|
with converter output (T-04-20, embedded in the metadata table).
|
|
|
|
## TDD Gate Compliance
|
|
|
|
PASSED — task 1: RED `dea7971` (8 CO cases failing on missing corpus) → GREEN `b2689f7`;
|
|
task 2: RED `ec98c8d` (8 TP/CV cases failing on missing scripts) → GREEN `c6a5f2b`. Task 3
|
|
is a non-TDD measurement/record task (no RED/GREEN applies). RED failures were verified
|
|
before each GREEN commit.
|
|
|
|
## Self-Check: PASSED
|