From a575ecd1398aa13322d54ae9a0a1e24b7a66e24c Mon Sep 17 00:00:00 2001 From: araharah-executor Date: Fri, 23 Oct 2026 23:40:41 +0800 Subject: [PATCH] =?UTF-8?q?docs(04):=20create=20phase=20plan=20=E2=80=94?= =?UTF-8?q?=204=20plans=20in=203=20waves=20(search=20module,=20corpus/bake?= =?UTF-8?q?-off,=20/cari=20surface,=20webview=20hand-off=20+=20device=20ga?= =?UTF-8?q?te)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .planning/ROADMAP.md | 25 +- .../04-01-PLAN.md | 860 ++++++++++++++++++ .../04-02-PLAN.md | 563 ++++++++++++ .../04-03-PLAN.md | 665 ++++++++++++++ .../04-04-PLAN.md | 629 +++++++++++++ .../04-VALIDATION.md | 106 +++ 6 files changed, 2845 insertions(+), 3 deletions(-) create mode 100644 .planning/phases/04-destination-search-kl-gazetteer/04-01-PLAN.md create mode 100644 .planning/phases/04-destination-search-kl-gazetteer/04-02-PLAN.md create mode 100644 .planning/phases/04-destination-search-kl-gazetteer/04-03-PLAN.md create mode 100644 .planning/phases/04-destination-search-kl-gazetteer/04-04-PLAN.md create mode 100644 .planning/phases/04-destination-search-kl-gazetteer/04-VALIDATION.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 2f104fc..1db8c2e 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -123,11 +123,30 @@ Notes: 3. Choosing a result sends a native venue card in chat confirming the destination before navigation opens (SRCH-03) 4. Tapping through from the search result launches the Mini App with the destination preloaded — no re-entry in the webview (ONBD-02); adding araharah to the phone home screen and launching from there lands in the app ready to search (ONBD-03) 5. The self-hosted Photon KL bake-off is concluded against the real-query acceptance corpus with the decision (and any gazetteer/fallback escalation) documented — the search-quality experiment closes, it doesn't stay open -**Plans**: TBD +**Plans**: 4 plans + +Plans: + +**Wave 1** *(the shared foundation — a fourth npm workspace, `search/`)*: +- [ ] 04-01-PLAN.md — `@araharah/search` workspace: the `SearchSource`/`GazetteerEntry` contract, the JUPEM-derived abbreviation table as editable DATA, the symmetric normalizer, the hand-rolled OSA distance, the disjoint-tier scorer, the curated KL gazetteer with locals' aliases, the lazy three-environment loader, plus the root GPL-3.0 `LICENSE` and the ODbL-1.0 data notice (D-04/05/06/07/08/09/15) + +**Wave 2** *(parallel — the measurement instrument and the chat surface)*: +- [ ] 04-02-PLAN.md — the pinned golden corpus (D-13), the D-14 bake-off runner + the in-suite Nyquist gate, the Overpass top-up tool (D-06), the JSON→SQLite converter (D-05), and `docs/SEARCH-QUALITY.md` recording the measured result and the escalation lever (SRCH-01, SRCH-02) +- [ ] 04-03-PLAN.md — the `/cari` command (D-01), the single-message `sendVenue` card with 2-4 inline alternates (D-02/D-10), the honest out-of-area refusal, the OPEN deep link carrying a plain gazetteer id with NO crypto (D-11), `MINI_APP_URL` config, and the four `commands.test.mjs` landmines defused in the same change (SRCH-03, ONBD-02) + +**Wave 3** *(the webview hand-off + the blocking device gate)*: +- [ ] 04-04-PLAN.md — the two-carrier destination reader (`?dest=` primary, `initData.start_param` secondary) with one shared validation, the memory-only resolver, the honest "destination unavailable" state, ONBD-03's minimal `addToHomeScreen`/`checkHomeScreenStatus` hooks behind a Bot API 8.0 guard, the build-artifact drift gate, the destination-preload capture, and the blocking Android/iOS/Desktop device gate (ONBD-02, ONBD-03) + +**Cross-cutting constraints** *(must_haves.truths asserted by 2+ plans)*: +- The deep-link carrier correction (plain id via `?dest=` URL param read from `location.search`, `start_param` secondary, NO crypto) — plans 03, 04 +- The D-14 corpus gate as an executable in-suite assertion (top-3 ≥ 90% on positives, negatives honestly refused) — plans 01 (admission rule), 02 (runner + suite) +- Category-only logging / no query text persisted (D-15) — plans 01 (no logging in the module), 03 (category-only literals), 04 (state-token capture signal) Notes: - Budgeted bake-off: research recommends `/gsd-research-phase` (gazetteer sourcing/structure ~200 entries, Malay normalizer, photon-vs-fallback decision; funds a GraphHopper MY dump on the VPS). -- VPS provisioning (Photon + Valhalla hosting) begins here — free-first infrastructure mandate (Oracle Cloud Always Free candidate). +- **D-03 supersedes the roadmap's "VPS provisioning begins here" note** — the search backend is the in-Worker curated gazetteer ($0 infrastructure); Photon-on-VPS is deferred to Phase 5 (Valhalla needs the VPS anyway) and stays behind the `SearchSource` interface as backend #3. +- **Mechanism correction (RESEARCH Pitfall 1, plan 04-03/04-04):** a `web_app` inline button's `WebAppInfo` carries ONLY a `url` — there is no `start_param` field (verified against the installed grammY typings), so the deep link's plain gazetteer id travels as `?dest=` in that URL and the webview reads `location.search` first, `initData.start_param` second. The D-10/D-11 intent (venue card → one tap → Mini App preloaded with a plain id, no crypto) is unchanged; a structural gate asserts no crypto in the deep-link path. +- The bilingual alias scheme: `/cari` + `/search` are registered together (the Phase 2 D-04 precedent of pairing `/start`↔`/mula`, `/privacy`↔`/privasi`); `/pergi_ke` stays reserved for Phase 5 and its future-stub ban is narrowed to `/pergi_ke`, not removed. **UI hint**: no ### Phase 5: Routing + Route Display @@ -205,7 +224,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 | 1. Design Tokens + Rams Foundation | 0/TBD | Not started | - | | 2. Bot Skeleton + initData Security | 0/TBD | Not started | - | | 3. Mini App Shell + Geolocation | 0/4 | Ready to execute | - | -| 4. Destination Search + KL Gazetteer | 0/TBD | Not started | - | +| 4. Destination Search + KL Gazetteer | 0/4 | Ready to execute | - | | 5. Routing + Route Display | 0/TBD | Not started | - | | 6. Turn-by-Turn Guidance (Live Driving) | 0/TBD | Not started | - | | 7. Hardening + Private Launch | 0/TBD | Not started | - | diff --git a/.planning/phases/04-destination-search-kl-gazetteer/04-01-PLAN.md b/.planning/phases/04-destination-search-kl-gazetteer/04-01-PLAN.md new file mode 100644 index 0000000..754edee --- /dev/null +++ b/.planning/phases/04-destination-search-kl-gazetteer/04-01-PLAN.md @@ -0,0 +1,860 @@ +--- +phase: 04-destination-search-kl-gazetteer +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - package.json + - search/package.json + - search/tsconfig.json + - search/src/types.ts + - search/src/rules/abbreviations.ts + - search/src/normalize.ts + - search/src/distance.ts + - search/src/score.ts + - search/src/data/kl-gazetteer.json + - search/src/gazetteer.ts + - search/src/source.ts + - search/src/index.ts + - search/test/runner.mjs + - search/test/harness.mjs + - search/test/normalizer.test.mjs + - search/test/distance.test.mjs + - search/test/score.test.mjs + - search/test/gazetteer.test.mjs + - search/test/loader-env.test.mjs + - LICENSE + - LICENSES.md + - README.md +autonomous: true +requirements: [SRCH-01, SRCH-02] +must_haves: + decisions_implemented: + - "D-04: GPL-3.0 LICENSE at the repo root; data as plain committed files; no hardcoded secrets (the search module reads no env at all)" + - "D-05: the JSON dataset format stays reachable from day one — the curated file is plain JSON, so backend #2 (WASM geocoder) is a format change, not a rewrite" + - "D-06: the curated gazetteer is hand-curated seed data committed to the repo (the Overpass top-up script is plan 04-02)" + - "D-07: canonical format is typed JSON in the repo — V8-native parse, no WASM/sqlite in the Worker bundle" + - "D-08: aliases live natively inside each entry as an `aliases` array — locals' shorthand is the corpus-quality lever" + - "D-09: the normalizer is a rule table + folding + fuzzy-residual module; the abbreviation table is DATA (a file edit extends it, never a code change); normalization is applied symmetrically to query AND entry aliases" + - "D-15: the shared module contains no logging and no storage — the query text never leaves the caller" + truths: + - "A user query 'KLCC' resolves to the Suria KLCC entry id 'suria-klcc' through the shared module, with no network call and no infrastructure" + - "'Jln Ampang' and 'Jalan Ampang' normalize to the IDENTICAL string; 'Tmn' expands to 'taman'; 'Kg' to 'kampung'; 'Prsn' to 'presint'" + - "A transposition typo ('KCLC') still resolves to the expected entry id, via the Damerau-Levenshtein residual tier" + - "An exact normalized match ALWAYS outranks any fuzzy match — the tier score bands are disjoint and asserted by a test" + - "The gazetteer JSON loads identically in three environments: the wrangler bundle, the Vite bundle, and bare Node 24 (the corpus runner)" + - "The Worker never builds an index at module scope — the index is built lazily on first search and memoized" + - "A search over the full gazetteer costs well under the Workers Free 10 ms CPU/request budget" + - "The repo carries a GPL-3.0 LICENSE and the gazetteer carries an ODbL-1.0 + OpenStreetMap attribution notice" + - "No log line anywhere in the shared module can carry query text — the module has no logging at all" + artifacts: + - path: "search/src/types.ts" + provides: "GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts (D-03, D-07, D-08)" + contains: "export interface SearchSource" + - path: "search/src/rules/abbreviations.ts" + provides: "the vendored JUPEM-derived bidirectional abbreviation table as DATA (D-09)" + contains: "ABBREVIATIONS" + - path: "search/src/normalize.ts" + provides: "pure normalize(): NFKD-fold → whitespace/punctuation collapse → whole-token abbreviation expansion → glued-word split (D-09)" + - path: "search/src/distance.ts" + provides: "hand-rolled Damerau-Levenshtein (OSA) distance — zero runtime dependencies" + - path: "search/src/score.ts" + provides: "the disjoint-tier scorer: EXACT > ALIAS_EXACT > PREFIX > TOKEN_SET > FUZZY (D-09)" + contains: "export const TIER" + - path: "search/src/data/kl-gazetteer.json" + provides: "the committed curated KL gazetteer with per-entry aliases (D-06, D-07, D-08)" + contains: "\"entries\"" + - path: "search/src/gazetteer.ts" + provides: "the lazy loader + index: parse-once, memoized, three-environment safe (D-07)" + - path: "search/src/index.ts" + provides: "the public API surface — search(), normalize(), byId(), entries — imported by bot, web, and the corpus runner" + - path: "LICENSE" + provides: "the GPL-3.0 license text (D-04)" + - path: "LICENSES.md" + provides: "the dual-license note: code GPL-3.0, gazetteer data ODbL-1.0 with OSM attribution (D-04, ODbL §4.4)" + - path: "search/test/normalizer.test.mjs" + provides: "per-rule vectors for the whole abbreviation table + idempotence + glued-split vectors (D-09)" + - path: "search/test/distance.test.mjs" + provides: "hand-computed OSA distance vectors (SRCH-02)" + - path: "search/test/score.test.mjs" + provides: "tier disjointness, min-length guard, exact-outranks-fuzzy invariants (SRCH-02)" + - path: "search/test/gazetteer.test.mjs" + provides: "entry schema, unique slug-safe ids, alias presence, ODbL header, LICENSE presence (D-07, D-08)" + - path: "search/test/loader-env.test.mjs" + provides: "the three-environment loader proof: bare Node import, wrangler dry-run bundle, Vite bundle (the phase's top mechanical risk)" + key_links: + - from: "search/src/normalize.ts" + to: "search/src/rules/abbreviations.ts" + via: "the whole-token expansion reads the imported table; a table edit extends behavior without a code change (D-09)" + pattern: "ABBREVIATIONS" + - from: "search/src/gazetteer.ts" + to: "search/src/data/kl-gazetteer.json" + via: "a static JSON import (build-time inlined), never a runtime fetch — Workers have no fs" + pattern: "kl-gazetteer.json" + - from: "search/src/gazetteer.ts" + to: "search/src/normalize.ts" + via: "every entry alias is normalized at index-build time, so both sides of a comparison are canonical (D-09 symmetry law)" + pattern: "normalize" + - from: "search/src/score.ts" + to: "search/src/distance.ts" + via: "the FUZZY tier's similarity is 1 - damerau/maxLen" + pattern: "damerau" + - from: "search/src/index.ts" + to: "search/src/gazetteer.ts" + via: "the public search()/byId() surface delegates to the lazily-built index" + pattern: "export" +--- + + +Create the shared, dependency-free search module that the bot, the Mini App, and the corpus runner all import — the pure foundation the whole phase rests on — and land the repo's licensing (GPL-3.0 code + ODbL-1.0 gazetteer data) in the same change. + +Purpose: D-03 makes the search backend a pluggable `SearchSource` seam, D-07 fixes the dataset format as typed JSON, and D-11 requires the bot and the webview to map the SAME gazetteer id → destination. All three are satisfied by one shared workspace package (`@araharah/search`) imported by both `bot/` and `web/`. This plan builds that package: the JUPEM-derived abbreviation table as editable DATA, the symmetric normalizer, the hand-rolled Damerau distance, the disjoint-tier scorer, the curated KL gazetteer with locals' aliases, and the lazy three-environment loader. It also de-risks the phase's #1 mechanical risk (RESEARCH: "the JSON loader must survive THREE environments — wrangler bundle / Vite bundle / bare Node corpus runner") with an explicit spike. + +Output: a fourth npm workspace `search/` (package `@araharah/search`) with types, rules, normalize, distance, score, gazetteer loader, the curated `kl-gazetteer.json`, four test suites, the root `LICENSE` (GPL-3.0), `LICENSES.md` (ODbL notice), and a corrected README §License. + + + +@$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md +@$HOME/.config/opencode/get-shit-done/templates/summary.md + + + +@AGENTS.md +@.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md +@.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md +@.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md +@bot/src/security/initdata.ts +@bot/src/i18n/strings.ts +@bot/src/kv/sessions.ts +@web/src/location/types.ts +@web/src/map/map.ts +@web/tsconfig.json +@theme/package.json +@package.json +@README.md + + +Key contracts the executor needs. `GazetteerEntry` / `SearchSource` are given verbatim by RESEARCH §Pattern 1 (04-RESEARCH.md lines 294-319) and must be implemented EXACTLY as shaped here — downstream plans (04-03 bot, 04-04 web) import these names. + +From `search/src/types.ts` (this plan's output — the contract everything else implements against): + +```typescript +export interface GazetteerEntry { + id: string; // stable, slug-shaped: "suria-klcc", "sunway-pyramid" + name: string; // canonical display name (the OSM landmark name, e.g. "Suria KLCC") + aliases: string[]; // locals' shorthand (D-08) — never empty + lat: number; + lng: number; + category: string; // "mall" | "office-tower" | "park" | "station" | ... + address?: string; // human address for sendVenue's `address` field — shown VERBATIM +} + +export interface SearchQuery { + text: string; // raw user query + limit?: number; // default 5 (1 best + 2-4 alternates, D-02) +} + +export interface SearchHit { + entry: GazetteerEntry; + score: number; // 0..1, higher is better + matchedAlias: string; // which alias matched — the corpus report needs this +} + +export interface SearchSource { + readonly name: string; // "gazetteer" | "wasm-geocoder" | "photon" + search(q: SearchQuery): Promise; // EMPTY array = honest refusal + byId(id: string): GazetteerEntry | null; // synchronous — the callback handler + webview both need it +} +``` + +`byId` is a SEPARATE SYNCHRONOUS method (not part of `search`): the alternates callback handler and the webview's deep-link reader both need an id → entry lookup and neither should await a network. RESEARCH Open Question 4 recommends exactly this shape. + +From `bot/src/security/initdata.ts` — the header-contract style every new module copies: + +```typescript +/* + * — (). + * ---... + * + */ +``` + + + +Verified in this session — do not re-derive, do not spend a task re-probing: + +- Node is **v24.18.0**, npm **11.16.0**. Node 24 strips TypeScript types by DEFAULT, so a `.mjs` script CAN `await import("../src/foo.ts")` — this is the repo's existing pattern (`bot/test/commands.test.mjs` line 27 does exactly this and passes). +- `import('./package.json', {with:{type:'json'}})` WORKS on this Node (probed: "json import OK"). +- `web/src/map/map.ts` line 20 already does `import ramsStyle from "@araharah/theme/basemap/liberty-rams.json";` — the build-time JSON-inlining precedent for a workspace subpath import. Vite and wrangler/esbuild both inline it. +- grammY 1.46.0 is installed. `ctx.replyWithVenue(latitude, longitude, title, address, other?)` exists (node_modules/grammy/out/context.d.ts:690). `InlineKeyboard.webApp(text, url)` exists (node_modules/grammy/out/convenience/keyboard.d.ts:594). +- **`WebAppInfo` has EXACTLY ONE field — `url`** (node_modules/@grammyjs/types/markup.d.ts:219-222). There is NO `start_param` field on a `web_app` inline button. This is the verified basis for the deep-link mechanism correction in plan 04-03. +- The `search/` workspace must be added to the root `package.json` `"workspaces"` array (currently `["theme","bot","web"]`). + + + +The curated gazetteer is the phase's product, not a generated artifact (RESEARCH §Verification A3: "KLCC" has no usable OSM landmark name in OSM — the landmark is `name="Suria KLCC"`, and "DUKE" is a highway acronym; therefore the alias list IS the product, and "KLCC" must be an explicit `aliases` entry, never a `name`). + +Minimum seed content (hand-authored, coordinates from the landmarks' real KL locations): + +Required entries (each with `aliases` covering the local shorthand, per D-08): +- `suria-klcc` — name "Suria KLCC", aliases `["KLCC","KLCC Park","KLCC Mall","Jalan Ampang"]`, category "mall", lat 3.1578, lng 101.7117, address "Jalan Ampang, 50088 Kuala Lumpur" — NOTE: "Jalan Ampang" is the complex's address street and a real local shorthand for it (the corpus's "Jln Ampang"/"Jalan Ampang" cases pin `suria-klcc`); "KL Sentral" must NEVER appear here — it is a different landmark with its own entry, and aliasing it here would be a cross-landmark collision +- `sunway-pyramid` — name "Sunway Pyramid", aliases `["Pyramid","Sunway","Sunway Pyramid Shopping Mall"]`, category "mall", lat 3.0726, lng 101.6069, address "Jalan PJS 11/15, Bandar Sunway, 47500 Subang Jaya" +- `the-duke` — name "Lebuhraya DUKE", aliases `["DUKE","Duta-Ulu Klang Expressway","Karak Link"]`, category "highway", lat 3.1357, lng 101.6864, address "Lebuhraya DUKE, 50000 Kuala Lumpur" +- `pavilion-kl` — name "Pavilion Kuala Lumpur", aliases `["Pavilion","Pavilion KL","KL Pavilion"]`, category "mall", lat 3.1478, lng 101.6950, address "Jalan Bukit Bintang, 55100 Kuala Lumpur" +- `ijnp` — name "Institut Jantung Negara", aliases `["IJN","IJN Hospital","Institut Jantung Negara Malaysia"]`, category "hospital", lat 3.1636, lng 101.7060, address "Jalan Tun Razak, 50400 Kuala Lumpur" +- `pwtc` — name "PWTC", aliases `["PWTC","Putra World Trade Centre"]`, category "convention", lat 3.1768, lng 101.6980, address "Jalan Tun Razak, 50400 Kuala Lumpur" — aliases must NOT invent a wrong expansion ("Pusat equated Trade Fair" is not a thing; PWTC = Putra World Trade Centre) +- `kl-sentral` — name "KL Sentral", aliases `["KL Sentral","Stesen KL Sentral","KL Central","Sentral Station","Stesen Sentral"]`, category "station", lat 3.1343, lng 101.6869, address "Jalan Stesen Sentral 5, 50470 Kuala Lumpur" +- `bangsar` — name "Taman Bangsar", aliases `["Bangsar","Taman Bangsar","Bangsar Park"]`, category "park", lat 3.1477, lng 101.6875, address "Jalan Maarof, 59000 Kuala Lumpur" +- `kampung-ampang` — name "Kampung Ampang", aliases `["Kg Ampang","Kampung Ampang","Ampang Village"]`, category "village", lat 3.1456, lng 101.7120, address "Jalan Ampang, 68000 Ampang" +- `tun-razak` — name "Jalan Tun Razak", aliases `["Tun Razak","Jalan Tun Razak"]`, category "road", lat 3.1560, lng 101.7100, address "Jalan Tun Razak, 50400 Kuala Lumpur" + +Add at least **20 more** hand-curated KL-metro landmarks to reach the seed floor of 30 entries (10 required + 20 more). Suggested additional KL landmarks with local aliases: `sungei-wang` (aliases "Sungei Wang","Sg Wang","Sungai Wang" — the corpus's "Sg Wang" cases need this entry), `berjaya-times-square` (aliases "Berjaya Times Square","Times Square","BTS"), `mid-valley` (aliases "Mid Valley","Mid Valley Megamall","MV"), `sunway-velocity` (aliases "Velocity","Sunway Velocity Mall"), `ioi-city-mall` (aliases "IOI City Mall","IOI City"), `nu-sentral` (aliases "Nu Sentral","KL Sentral Mall"), `merdeka-118`, `kl-tower` (aliases "KL Tower","Menara Kuala Lumpur"), `petronas-twin-towers` (aliases "Petronas Towers","Twin Towers","KLCC Towers"), `aquaria-klcc`, `central-market` (aliases "Central Market","Pasar Seni"), `petaling-street` (aliases "Petaling Street","Chi Cheong Kai"), `masjid-jamek`, `pudu-raya` (aliases "Pudu Raya","Puduraya"), `bukit-bintang` (aliases "Bukit Bintang","BB Park"), `chinatown`, `little-india-brickfields` (aliases "Brickfields","Little India"), `titiwangsa`, `lake-gardens` (aliases "Lake Gardens","Perdana Botanical Garden"), `national-museum` (aliases "National Museum","Muzium Negara"). + +Every entry MUST have: +- `id` matching `/^[a-z0-9-]{1,64}$/` (slug-shaped, unique, never renamed once shipped) +- `name` = the landmark's canonical OSM name (NOT the local shorthand) +- `aliases` = a NON-EMPTY array including the canonical name itself +- `lat` / `lng` = the real coordinates (finite numbers, KL metro bbox roughly lat 3.0-3.3, lng 101.5-101.9) +- `category` from a small closed set: `mall | office-tower | park | station | hospital | road | highway | landmark | convention | village | education | government` +- `address` = a human address string, shown VERBATIM on the venue card (never the normalized form — RESEARCH Pitfall 3) + +The JSON file ALSO carries an ODbL attribution header object, so the notice travels with the data (OSMF attribution guidelines require it "within the data or metadata"): + +```json +{ + "_license": "ODbL-1.0", + "_attribution": "© OpenStreetMap contributors", + "_notice": "Curated from OpenStreetMap data. The curated gazetteer and any derived index are made available under the Open Database License (ODbL) v1.0 — https://opendatacommons.org/licenses/odbl/1-0/", + "version": 1, + "entries": [ ... ] +} +``` + + + +The table is DATA, vendored from the JUPEM-normative list as transcribed by the OSM Nominatim project in `settings/icu-rules/variants-ms.yaml` (PR #4056, merged 2026-04-02; primary source: "JUPEM Toponymic Guidelines — 'Abbreviations of Geographical Terms and Other Words Used in Maps'"). Vendored wholesale — do NOT author a smaller list (RESEARCH: "the difference between the 8-entry original and the current 65+ entry table is the difference between 'Jln' working and 'Jln Ampang' working"). + +The file `search/src/rules/abbreviations.ts` holds a `ReadonlyArray` of `[fullForm, commaSeparatedAbbreviations]` pairs, with the header comment citing JUPEM + the Nominatim transcription. Minimum rows (the shared + MY sections of the upstream table): + +``` +Jalan → Jln +Lorong → Lrg, Lor +Lebuh → Lbh +Simpang → Spg +Kampung → Kg +Kampong → Kg +Taman → Tmn +Bangunan → Bgn +Bukit → Bkt +Batu → Bt +Sungai → Sg +Pulau → P, Pl +Tanjung → Tg +Tanjong → Tg +Teluk → Tlk +Telok → Tlk +Gunung → G, Gn +Gunong → G, Gn +Haji → Hj +Hajjah → Hjh +Abdul → Abd +Persiaran → Psrn +Lebuhraya → Lbh.Raya +Bandar → Bdr +Kondominium → Kondo +Bahagian → Bhg +Blok → Blk +Kuala → K, Kl +Alur → Alr +Bakau → Bak +Hulu → H +Hilir → Hr +Jeram → J, Jr +Lembah → Lmbh +Padang → Pdg +Parit → Pt +Permatang → Pmtg +Pelabuhan → Plbh +Kepulauan → Kep +Ladang → Ldg +Lapangan → Lpg +Gosong → Gsg +Sekolah kebangsaan → SK +Sekolah menengah → SM +Sekolah menengah kebangsaan → SMK +Sekolah rendah → SR +Hospital → Hosp +Dispensari → Disp +``` + +Plus the two explicit D-09 vowel-drop rows (RESEARCH: Putrajaya precincts are written both "Presint" and "Prsn" — the latter drops the internal `e`; `Prsn` is NOT a prefix abbreviation): + +``` +Presint → Prsn +``` + +"Presint" must map to itself so the canonical long form is always reachable. + +The module builds the BIDIRECTIONAL map ONCE at module load by inverting the table: every abbreviation maps to its full form(s), and every full form maps to itself. The canonical form is ALWAYS the long form (so "Jln Ampang" and "Jalan Ampang" both normalize to "jalan ampang" — the symmetry law D-09 requires). + +AMBIGUITY POLICY (RESEARCH §"Ambiguity to handle"): where one abbreviation maps to several full forms (`G` → Gunung/Gunong, `P` → Pulau/Parit…), the normalizer keeps a per-token ambiguity LIST rather than guessing, and the fuzzy-residual tier resolves what the table cannot. In practice: single-letter abbreviations (`G`, `P`, `H`, `J`, `K`, `L`) expand to their FIRST listed full form ONLY when the token is a standalone word AND the expansion is unambiguous in context; the normalizer must NOT expand a single letter that is part of a longer token. + +Whole-token-only expansion is a hard law (RESEARCH Pitfall 5): "Pdang" is never expanded, only "pdg". + + + +`normalize(input: string): string` — pure, no I/O, no imports beyond the rules table. Pipeline (each step separately testable): + +1. NFKD normalize + strip combining marks. +2. Lowercase (casefold). +3. Punctuation → space, but KEEP `/` (Malaysian address shorthand "Jln 3/1" and "Presint 8/1" need it) and keep `-`? — NO: strip `-` too (it is a separator in "Sunway-Pyramid"). Keep only letters, digits, spaces, and `/`. +4. Collapse whitespace + trim. +5. Per-token abbreviation expansion (whole-token only, per the table above). +6. Glued-word split — LONGEST-MATCH-FIRST against a starter list, requiring the remainder to be >= 3 characters (RESEARCH: "a naive 'insert a space before any known word' will corrupt words that legitimately contain a starter as a substring"). Starters: `jalan, taman, kampung, kampong, lorong, bukit, sungai, bandar, persiaran, lebuh, jalan, perkampungan, presint`. Longest starters first (`kampung` before `kampong` before `kg`-expanded forms are already separate tokens by step 5). +7. Collapse whitespace again + trim. + +IDEMPOTENCE LAW: `normalize(normalize(x)) === normalize(x)` for every vector — assert this in the normalizer suite (the repo's T7-style independence self-check, `bot/test/initdata.test.mjs` lines 240-243). + +The normalizer MUST NOT strip the `/` in "3/1" and MUST NOT expand a token that is a single letter unless it is in the table with an unambiguous mapping. + +Example vectors that MUST hold (these are the suite's minimum): +- "Jln Ampang" → "jalan ampang" +- "Jalan Ampang" → "jalan ampang" (IDENTICAL to the above — the symmetry law) +- "Jln Tmn Desa" → "jalan taman desa" (both sides expand; RESEARCH Pitfall 3: this is correct for MATCHING even though the DISPLAY form differs) +- "Tmn Bangsar" → "taman bangsar" +- "Kg Ampang" → "kampung ampang" +- "Kpg Ampang" → "kampung ampang" +- "Prsn 15" → "presint 15" +- "Presint 15" → "presint 15" +- "tamanbangsar" → "taman bangsar" (glued split) +- "Jln 3/1" → "jalan 3/1" +- "Bdr Baru" → "bandar baru" +- "Sunway Pyramid" → "sunway pyramid" +- " SUNWAY PYRAMID " → "sunway pyramid" +- "Kuala Lumpur" → "kuala lumpur" +- "PJ" → "pj" (city codes are ALIASES, not abbreviation rules — RESEARCH §"A ready-made city-code table": "PJ" → Petaling Jaya belongs in the gazetteer ALIAS table, not the abbreviation rule table) + + + +`score.ts` — the disjoint-tier scorer. The tiers MUST have strictly disjoint, descending score bands so an exact match can never be outranked by a fuzzy hit (RESEARCH Pitfall 4: "the ordering is a property of the TIER, not of the arithmetic"). + +```typescript +export const TIER = { + EXACT: 1.00, // normalized query === normalized alias or name + ALIAS_EXACT: 0.90, // normalized query === normalized alias (entry name is a different string) + PREFIX: 0.75, // one is a prefix of the other, >= 3 chars + TOKEN_SET: 0.65, // all query tokens present in the candidate's token set + FUZZY: 0.60, // the fuzzy band's CEILING — a fuzzy score is sim * 0.60, i.e. always in (0, 0.60) +} as const; + +/** A FUZZY hit is ADMITTED only at this similarity or above — this is the + * mechanism behind D-13's honest refusal, tuned between the corpus's two + * loads: "KCLC" vs "KLCC" is sim 0.75 (admitted — the typo bucket passes) + * while "Penang" vs "Padang" is sim 0.667 (refused — no fabricated pin). */ +export const FUZZY_ADMIT_SIMILARITY = 0.72; +``` + +NOTE the scaling: `FUZZY` is the band's CEILING (0.60), and a fuzzy score is `sim * 0.60` — a naive `TIER.FUZZY: 0.00` multiplied into the similarity would zero the whole tier. The tier values are exact literals; a test asserts them. + +`scoreOne(queryNorm: string, aliasNorm: string): number`: +- `queryNorm === aliasNorm` → `TIER.EXACT` +- either side shorter than 3 characters → `0` (the min-length guard — RESEARCH Pitfall 5: "KLCC" and "DUKE" must be ALIAS hits, not fuzzy hits) +- prefix (one starts with the other, both >= 3 chars) → `TIER.PREFIX` +- all query tokens present in the candidate's token set → `TIER.TOKEN_SET` +- otherwise → the FUZZY residual: `sim = 1 - damerau(a, b) / Math.max(a.length, b.length)`, computed at BOTH the whole-string level AND the token level. The token level compares each query token against each candidate token (both >= 3 chars) and takes the mean of the best per-token similarities, scaled by the fraction of query tokens that found any candidate token. The fuzzy similarity is `max(wholeStringSim, tokenLevelSim)`, and the fuzzy score is `fuzzySim * TIER.FUZZY` — a value in (0, 0.60). The token level is what lets "Sentrl" reach the "KL Sentral" alias (the error is in ONE word — RESEARCH's anti-pattern against whole-string-only fuzzy) while "Jln Tun Razak" still cannot reach "Jalan Ampang" (the non-matching tokens drag the token-level mean down). + +The FUZZY tier is the RESIDUAL net only (D-09: "fuzzy scoring (Damerau) is only the residual net for genuine typos on longer words"). The TOKEN_SET tier runs BEFORE fuzzy so a one-word error does not cause cross-street confusion ("Jln Tun Razak" must not match "Jalan Ampang"). + +`score.ts` MUST export the tier constants AND a `scoreCandidate` that takes the query's normalized form and an entry's normalized alias set, returning the BEST tier hit plus the `matchedAlias` string (the corpus report needs that provenance). + + + +`distance.ts` — hand-rolled Optimal String Alignment (Damerau-Levenshtein with adjacent transpositions), ~25 lines, ZERO dependencies (RESEARCH §"Don't Hand-Roll": this is the ONE place hand-rolling IS correct — a dependency would add package weight to a Worker bundle D-07 deliberately wants minimal, and every dedicated string-distance npm package is 2022-vintage and unmaintained). + +```typescript +export function damerau(a: string, b: string): number { + if (a === b) return 0; + if (a.length === 0) return b.length; + if (b.length === 0) return a.length; + // 3-row band OSA with the adjacent-transposition term + // (i > 1 && j > 1 && a[i-1] === b[j-2] && a[i-2] === b[j-1] → prev2[j-2] + 1) +} +``` + +Hand-computed vectors the suite MUST assert (from the literature / RESEARCH §Code Examples): +- `damerau("ab", "ba") === 1` (adjacent transposition) +- `damerau("a cat", "an act") === 2` (two transpositions) +- `damerau("", "") === 0` +- `damerau("", "abc") === 3` +- `damerau("abc", "") === 3` +- `damerau("same", "same") === 0` +- `damerau("klcc", "kclc") === 1` +- `damerau("sunway", "sunwya") === 1` (adjacent transposition of a real corpus typo) +- `damerau("kitten", "sitting") === 3` (classic Levenshtein sanity) + +The row-rotation in a 3-band OSA is easy to get wrong — write the vectors FIRST and make them pass (repo RED→GREEN convention). + + + +`search/src/gazetteer.ts` — the dataset loader + index. Two hard laws: + +**LAW 1 — lazy, memoized, NEVER at module scope (RESEARCH Pitfall 8, Workers 1 s startup limit / "avoid expensive work in global scope").** The pattern is `bot/test/initdata.test.mjs` lines 42-51 (the repo's only lazy-init precedent): + +```typescript +let index: Map | null = null; +function loadIndex(): Map { + if (index === null) { /* build once */ } + return index; +} +``` + +**LAW 2 — the loader must survive THREE environments**: the wrangler/esbuild bundle (Worker), the Vite bundle (browser), and bare Node 24 (the corpus runner, which imports the `.ts` source directly via type-stripping). The mechanism is a STATIC JSON IMPORT — `import raw from "./data/kl-gazetteer.json" with { type: "json" }` — because Workers have no `fs` and a runtime fetch would be a hard failure. esbuild and Vite both INLINE a static JSON import at build time. The `with { type: "json" }` attribute is valid on Node 24 (probed) and ignored/harmless under esbuild/Vite (verify with the loader-env suite). + +The `with { type: "json" }` attribute under **esbuild** is the one thing that must be proven, not assumed: if esbuild rejects the import attribute, fall back to `import raw from "./data/kl-gazetteer.json";` (esbuild inlines JSON by default without the attribute) and let the Node-side import assert via the loader-env suite. The loader-env suite is the arbiter. + +The module exports: +- `getEntries(): readonly GazetteerEntry[]` — the parsed entries +- `getEntryById(id: string): GazetteerEntry | null` — id → entry, null when unknown (the honest-null law, `bot/src/kv/sessions.ts` `getClaimByToken` precedent) +- `getIndex(): Map` — the memoized alias → entries map + +The index maps each entry's NORMALIZED name AND each NORMALIZED alias → the entry (D-09 symmetry: both sides normalized). Normalization happens ONCE at index-build time, never per query. + +**LAW 3 — no logging and no storage anywhere in the module** (D-15: the query text is user payload). The module contains NO `console.log` at all, and no storage API call of any kind. + + + +`search/src/index.ts` — the public API. Named exports ONLY (no default export). The package has NO `exports`/`main` field (the `@araharah/theme` precedent), so consumers MUST import the EXPLICIT SUBPATH WITH EXTENSION — `import { createGazetteerSource } from "@araharah/search/src/index.ts";` (both tsconfigs carry `allowImportingTsExtensions` + bundler resolution; esbuild and Vite compile the `.ts` source; Node 24 type-stripping handles the corpus runner's RELATIVE form `../src/index.ts`). A bare `@araharah/search` specifier resolves to NOTHING (there is no `index.js` for default resolution) — never use it: + +```typescript +export type { GazetteerEntry, SearchQuery, SearchHit, SearchSource } from "./types.ts"; +export { normalize } from "./normalize.ts"; +export { TIER } from "./score.ts"; +export { damerau } from "./distance.ts"; +export { getEntries, getEntryById, getIndex } from "./gazetteer.ts"; +export { createGazetteerSource, search } from "./source.ts"; +``` + +`search(text, limit?)` is the CONVENIENCE wrapper over `createGazetteerSource()` — a synchronous-search wrapper the corpus runner can call without await (the `SearchSource.search` method stays `async` per RESEARCH Open Question 4, because backend #3 (Photon-on-VPS) will need it). + +`createGazetteerSource(): SearchSource` returns the D-03 in-Worker implementation: `name: "gazetteer"`, an async `search(q)` that returns hits ranked by tier then score, and a synchronous `byId(id)`. + +**The refusal contract is load-bearing** (RESEARCH §Pattern 1): an EMPTY array is a legitimate, load-bearing return value. D-13's negatives ("Kuching", "Penang") MUST produce `[]`, never a nearest-match pin. The admission rule is TIERED, not a single floor: the EXACT / ALIAS_EXACT / PREFIX / TOKEN_SET tiers are always admitted; a FUZZY hit is admitted only when its similarity reaches `FUZZY_ADMIT_SIMILARITY` (0.72). The arithmetic: "KCLC" vs "KLCC" is sim 0.75 → admitted (score 0.45); "Sunwya Pyarmid" vs "Sunway Pyramid" is sim 0.857 → admitted (score 0.514); "Penang" vs "Padang" is sim 0.667 → REFUSED. This is what makes the negatives honest WITHOUT starving the corpus's typo bucket — a flat `score >= 0.60` floor would refuse every typo positive and make D-14 unreachable. A query whose best candidate does not clear its tier's admission returns `[]`. Asserted by the corpus runner in plan 04-02 and documented in the module header. + +Ranking: sort by tier band DESC, then by score DESC, then by entry id ASC (a deterministic tie-break — the corpus results must be reproducible). + + + + + + + task 1: shared workspace scaffold + types + the JUPEM abbreviation table as DATA + package.json, search/package.json, search/tsconfig.json, search/src/types.ts, search/src/rules/abbreviations.ts, search/test/normalizer.test.mjs + + - `package.json` (root — the `workspaces` array is `["theme","bot","web"]`; add `"search"`; keep every existing script) + - `theme/package.json` (the manifest shape to copy — `name`, `type: "module"`, no `exports`/`main`, so consumers import sub-paths directly) + - `web/src/location/types.ts` (the type-module style: type-only module, no runtime values, no imports) + - `bot/src/i18n/strings.ts` (the data-table-first module style — the `rules/abbreviations.ts` shape: an editable data structure with a header contract naming how to extend it) + - `bot/src/security/initdata.ts` (the header-comment contract every new module copies) + - `bot/test/initdata.test.mjs` (the lazy-module-load + T-numbered-vector suite style; the normalizer suite copies this shape) + + +**Root `package.json`.** Add `"search"` to `workspaces` (so it becomes `["theme","bot","web","search"]`) and add these delegate scripts, matching the existing `bot:test` / `web:test` naming: + +```json +"search:test": "npm test -w search", +"search:typecheck": "npm run typecheck -w search", +``` + +Do NOT change `build`, `deploy`, or `dev` — `search/` is buildless (pure `.ts` + `.json` sources imported directly by consumers), exactly like the `@araharah/theme/basemap/liberty-rams.json` precedent, so the existing deploy chain needs no change. + +**`search/package.json`** — copy `theme/package.json`'s manifest shape: + +```json +{ + "name": "@araharah/search", + "private": true, + "type": "module", + "scripts": { + "test": "node test/runner.mjs", + "typecheck": "tsc --noEmit" + }, + "devDependencies": { + "typescript": "^5.9.3" + } +} +``` + +NO `exports`/`main` field — consumers import sub-paths directly (the `@araharah/theme` precedent). NO runtime dependencies (D-07: zero package weight in the Worker bundle). + +**`search/tsconfig.json`** — copy `web/tsconfig.json`'s compilerOptions (it has `resolveJsonModule: true`, which this workspace needs for the JSON import) but with `"types": ["node"]` replaced by nothing (no DOM, no Workers types — this module is platform-neutral) and `"include": ["src", "test"]`: + +```json +{ + "compilerOptions": { + "target": "ES2023", + "lib": ["ES2023"], + "module": "ESNext", + "moduleResolution": "bundler", + "strict": true, + "noUncheckedIndexedAccess": true, + "verbatimModuleSyntax": true, + "allowImportingTsExtensions": true, + "noEmit": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": true + }, + "include": ["src", "test"] +} +``` + +**`search/test/runner.mjs` + `search/test/harness.mjs`** — copy `bot/test/runner.mjs` and `bot/test/harness.mjs` VERBATIM (both are 47 and 10 lines; the runner auto-discovers `*.test.mjs` and drains the registry). This is the repo-native zero-dependency harness — no Jest, no Vitest, no Mocha (RESEARCH §Validation Architecture: "No Jest/Vitest/Mocha"). The runner must resolve the suite directory from `import.meta.dirname` exactly as `bot/test/runner.mjs` does. + +**`search/src/types.ts`** — the contract from the `` block, implemented EXACTLY (the names and shapes are load-bearing for plans 04-03 and 04-04). Add the header comment: + +```typescript +/* + * types.ts — the search contracts (D-03, D-07, D-08, D-11). + * -------------------------------------------- + * GazetteerEntry / SearchQuery / SearchHit / SearchSource are the seam + * every backend implements: the in-Worker gazetteer (backend #1, this + * phase), the Malaysia-only WASM geocoder (backend #2, D-05, post-v1), + * and Photon-on-VPS (backend #3, D-05, Phase 5). Backend #2/#3 do NOT + * ship in this phase — the interface exists so they can be added without + * a rewrite. + * + * SearchSource.search() returns an EMPTY array as a legitimate, + * load-bearing value: "no match in the KL gazetteer" (D-13's + * out-of-area negatives must be refused honestly, never answered with + * the nearest KL pin). + * + * byId() is separate and SYNCHRONOUS: the alternates callback handler + * and the webview's deep-link reader both need an id -> entry lookup and + * neither should await a network. + * + * Type-only module: no runtime values, no imports, no storage (GEO-03). + */ +``` + +**`search/src/rules/abbreviations.ts`** — the vendored table as DATA, exactly per the `` block: a `ReadonlyArray` named `ABBREVIATION_ROWS` with all 47 rows listed there plus the explicit `Presint → Prsn` row, and an exported `buildAbbreviationMap(): Map` that inverts the table once (abbreviation → full forms; full form → itself) — the canonical form is ALWAYS the long form. The header comment MUST cite the source: + +```typescript +/* + * abbreviations.ts — the Malay abbreviation table, as DATA (D-09). + * -------------------------------------------------------------- + * PRIMARY SOURCE: JUPEM Toponymic Guidelines — "Abbreviations of + * Geographical Terms and Other Words Used in Maps" + * (Malaysia, 2017 ed.) — the normative national list. + * TRANSCRIPTION: https://github.com/osm-search/Nominatim/pull/4056 + * (settings/icu-rules/variants-ms.yaml, merged 2026-04-02) + * SECONDARY: https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations + * + * D-09's law: this table extends by FILE EDIT, never by code change. + * Add a row, add a vector to test/normalizer.test.mjs, done. + * + * "Presint → Prsn" is an explicit vowel-drop expansion (Putrajaya + * precincts are written both ways in live addresses: "JALAN P8G + * PRESINT 8", "PRSN RESIDEN DESA PARKCI"). "Prsn" is NOT a prefix + * abbreviation — it drops an interior "e". Do NOT generalize this into + * a generic vowel-drop rule (it over-expands legitimate names). + * + * City codes (PJ, JB, KB, KK) are deliberately ABSENT: they are place + * identities, not address-word forms, so they live in the gazetteer's + * alias table (D-08), not here. + */ +``` + +**`search/test/normalizer.test.mjs`** — RED-first. The suite imports the normalizer LAZILY (the `bot/test/initdata.test.mjs` lines 42-51 pattern) so a missing module reports per-case failures rather than a lost suite: + +```javascript +let normalizePromise = null; +function loadNormalizer() { + if (!normalizePromise) { + normalizePromise = import("../src/normalize.ts").then((m) => m.normalize); + } + return normalizePromise; +} +``` + +Cases (T-numbered, one per law): +- `T1 rule vectors` — every vector in the `` block, asserted one by one with a named message. +- `T2 symmetry law` — `normalize("Jln Ampang") === normalize("Jalan Ampang")` and `normalize("Tmn Bangsar") === normalize("Taman Bangsar")`. +- `T3 whole-token-only expansion` — `"Pdang"` does NOT contain "jalan"/"persiaran" (it stays "pdang"-like); `"pdg"` DOES expand to "padang". +- `T4 slash preservation` — `normalize("Jln 3/1") === "jalan 3/1"`. +- `T5 glued-word split` — `normalize("tamanbangsar") === "taman bangsar"`; `normalize("jalankanpar")` contains "jalan" as a separate token. +- `T6 vowel-drop` — `normalize("Prsn 15") === "presint 15"` and `normalize("Presint 15") === "presint 15"`. +- `T7 idempotence` — for every vector `v`: `normalize(normalize(v)) === normalize(v)`. +- `T8 case/whitespace folding` — `normalize(" SUNWAY PYRAMID ") === "sunway pyramid"`. +- `T9 the table is data` — read `search/src/rules/abbreviations.ts` as TEXT (comments stripped) and assert it contains no `function` keyword (the module is data + one builder function, not algorithm code) and that the row count is >= 47. + +**`search/src/normalize.ts`** — implement EXACTLY per the `` pipeline. The exported signature is `export function normalize(input: string): string`. Import the table via `import { buildAbbreviationMap } from "./rules/abbreviations.ts";` and build the map ONCE at module scope (the map is ~60 entries of plain data — cheap; the D-09 "lazy" law applies to the 500-entry gazetteer INDEX, not to this table). + +Write `search/src/normalize.ts` NOW as part of this task so the suite in this task can go GREEN — the RED phase is the suite itself. + +**Acceptance:** `npm run search:test` exits 0 and the output contains `PASS T1` … `PASS T9`; `npm run search:typecheck` exits 0. + + + cd /home/C-Fu/araharah && npm install --package-lock-only --workspaces 2>&1 | tail -3 ; npm run search:test 2>&1 | tail -20 ; npm run search:typecheck 2>&1 | tail -5 ; grep -c '"search"' package.json + + + - `grep -q '"search"' package.json` exits 0 AND `node -e "const w=require('./package.json').workspaces; if(!w.includes('search')) throw new Error('search workspace missing'); console.log('OK')"` exits 0 + - `grep -q '"search:test": "npm test -w search"' package.json` exits 0 + - `test -f search/package.json && test -f search/tsconfig.json && test -f search/test/runner.mjs && test -f search/test/harness.mjs` exits 0 + - `node -e "const p=require('./search/package.json'); if(p.name!=='@araharah/search') throw new Error(1); if(p.dependencies) throw new Error('search must have zero runtime dependencies'); console.log('OK')"` exits 0 (D-07: zero Worker bundle weight) + - `npm run search:test` exits 0 and the runner output contains `PASS T1` through `PASS T9` + - `npm run search:typecheck` exits 0 + - `grep -q 'export interface SearchSource' search/src/types.ts` exits 0 AND `grep -q 'byId(id: string)' search/src/types.ts` exits 0 (the synchronous id lookup is part of the contract — D-03 seam) + - `grep -q 'ABBREVIATION_ROWS' search/src/rules/abbreviations.ts` exits 0 (the table is DATA) + - `grep -c 'JUPEM' search/src/rules/abbreviations.ts | head -1` returns >= 1 (the vendored table cites its normative source) + - `grep -q 'Presint' search/src/rules/abbreviations.ts` exits 0 (the D-09 vowel-drop rule is present) + - `node --experimental-strip-types -e "1" 2>/dev/null; node -e "import('./search/src/normalize.ts').then(m=>{ if(m.normalize('Jln Ampang')!=='jalan ampang') throw new Error('expansion broken'); if(m.normalize('Jalan Ampang')!=='jalan ampang') throw new Error('symmetry broken'); if(m.normalize('tamanbangsar')!=='taman bangsar') throw new Error('glued split broken'); if(m.normalize('Prsn 15')!=='presint 15') throw new Error('vowel-drop broken'); console.log('normalize OK')})"` exits 0 + + The `search/` workspace exists as a buildless fourth npm workspace with the type contract, the JUPEM-derived abbreviation table as editable DATA, and a symmetric normalizer proven by 9 named test cases including the symmetry, whole-token-only, glued-split, vowel-drop, and idempotence laws. + + + + task 2: Damerau distance + disjoint-tier scorer (the residual net) + search/src/distance.ts, search/src/score.ts, search/test/distance.test.mjs, search/test/score.test.mjs + + - `search/src/normalize.ts` (task 1's output — the scorer consumes NORMALIZED strings; do not re-normalize inside the scorer) + - `search/src/rules/abbreviations.ts` (task 1's output — the rules table is the tier-1 lever, the fuzzy tier is only the residual) + - `bot/test/initdata.test.mjs` (the RED→GREEN vector-suite style the two new suites copy) + - `.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md` (the §"Common Pitfalls" #4 and #5 entries — the tier-disjointness and min-length-guard laws) + + +**`search/src/distance.ts`** — hand-rolled OSA Damerau-Levenshtein, per the `` block. ZERO imports, ZERO dependencies. Header comment: + +```typescript +/* + * distance.ts — Damerau-Levenshtein (OSA) distance, hand-rolled. + * ---------------------------------------------------------- + * The ONE place hand-rolling IS correct (RESEARCH §"Don't Hand-Roll"): + * at ~500 entries x ~5 aliases the comparison count is tiny, the + * algorithm is ~25 lines of standard OSA dynamic programming, and a + * dependency would add package weight to a Worker bundle D-07 + * deliberately wants minimal. Every dedicated string-distance npm + * package is 2022-vintage and unmaintained (verified 2026-10-23). + * + * Adjacent transpositions count as ONE edit — "KCLC" must reach + * "KLCC" (D-13's transposition typo bucket), and a Levenshtein-only + * distance would score it 2 and fall out of the FUZZY tier. + * + * Pure: no I/O, no imports, no third-party packages. + */ +``` + +**`search/src/score.ts`** — the disjoint-tier scorer, per the `` block. Exports: +- `export const TIER = { EXACT: 1.00, ALIAS_EXACT: 0.90, PREFIX: 0.75, TOKEN_SET: 0.65, FUZZY: 0.00 } as const;` +- `export function scoreOne(queryNorm: string, aliasNorm: string): number` +- `export function scoreCandidate(queryNorm: string, aliasNorms: readonly string[]): { score: number; matchedAlias: string } | null` — returns the BEST tier hit and WHICH alias matched (the corpus report needs that provenance), or `null` when nothing scores. + +Implementation notes: +- The min-length guard fires BEFORE the prefix/fuzzy tiers: if `queryNorm.length < 3 || aliasNorm.length < 3` → return `0` (RESEARCH Pitfall 5). +- The EXACT tier fires on `queryNorm === aliasNorm`. +- The ALIAS_EXACT tier exists for the case where the query equals an entry's normalized ALIAS but the comparison is being made against the entry NAME (the name-match and alias-match are distinct comparisons). `scoreCandidate` handles this by scoring against the name and each alias, and reporting the ALIAS_EXACT band when the alias match is exact but the name differs. +- The TOKEN_SET tier: every whitespace-separated token of the query appears in the candidate's token set. +- The FUZZY tier: `fuzzySim * TIER.FUZZY` where `fuzzySim = max(wholeStringSim, tokenLevelSim)` and each `sim = 1 - damerau(a, b) / Math.max(a.length, b.length)` (see the `` block for the exact two-level shape). +- `scoreCandidate` MUST return the alias that produced the score (the raw, UN-normalized alias string — the corpus report wants the human-readable alias, not the normalized form). + +**`search/test/distance.test.mjs`** — RED-first. One `test(...)` per vector from the `` block, with the hand-computed expected value in the message. Load the module lazily (task 1's pattern). + +**`search/test/score.test.mjs`** — RED-first. Cases: +- `S1 tier disjointness` — for each pair of adjacent tiers, `TIER[a] > TIER[b]`; and `TIER.FUZZY === 0` with every FUZZY score `<= 0.60` and `> 0` for a near miss. +- `S2 min-length guard` — `scoreOne("ab", "abcd") === 0` and `scoreOne("kl", "kl") === 0` (both shorter than 3 → 0, EVEN for an exact match — the guard fires first). +- `S3 exact outranks fuzzy` — `scoreOne("klcc", "klcc") > scoreOne("klcc", "klca")` and `scoreOne("klcc", "klcc") > TIER.FUZZY` (an exact match always beats any fuzzy hit). +- `S4 prefix tier` — `scoreOne("jalan amp", "jalan ampang") === TIER.PREFIX`. +- `S5 token-set tier` — `scoreOne("taman bangsar", "taman bangsar kib") === TIER.TOKEN_SET` (every query token present) and `scoreOne("taman bangsar", "taman bukit") === 0` (not a subset — must NOT be TOKEN_SET). +- `S6 fuzzy residual + admission boundary` — `scoreOne("kclc", "klcc")` is in `(0, 0.60)` and equals `0.75 * 0.60`; and the admission boundary: a similarity of 0.72 admits, 0.667 does not (a "penang"-vs-"padang"-shaped near-miss stays below admission — assert via the two similarities' relationship to `FUZZY_ADMIT_SIMILARITY`). +- `S7 matchedAlias provenance` — `scoreCandidate("klcc", ["Suria KLCC", "KLCC"])` returns `{ score: TIER.EXACT or ALIAS_EXACT, matchedAlias: "KLCC" }` — the matched alias is the RAW alias string. + +**Acceptance:** `npm run search:test` exits 0 and the output contains `PASS S1` … `PASS S7` and `PASS T1` (distance) — the runner names every case. + + + cd /home/C-Fu/araharah && npm run search:test 2>&1 | tail -30 ; npm run search:typecheck 2>&1 | tail -5 ; grep -c 'export const TIER' search/src/score.ts + + + - `npm run search:test` exits 0 and the output contains `PASS S1` through `PASS S7` + - `npm run search:typecheck` exits 0 + - `grep -q 'export const TIER' search/src/score.ts` exits 0 AND `grep -q 'EXACT: *1.00' search/src/score.ts` exits 0 AND `grep -q 'ALIAS_EXACT: *0.90' search/src/score.ts` exits 0 AND `grep -q 'PREFIX: *0.75' search/src/score.ts` exits 0 AND `grep -q 'TOKEN_SET: *0.65' search/src/score.ts` exits 0 AND `grep -q 'FUZZY: *0.00' search/src/score.ts` exits 0 (the disjoint bands are exact literals) + - `grep -q 'export function damerau' search/src/distance.ts` exits 0 + - `node -e "import('./search/src/distance.ts').then(m=>{const d=m.damerau; const cases=[['ab','ba',1],['a cat','an act',2],['','',0],['','abc',3],['abc','',3],['same','same',0],['klcc','kclc',1],['sunway','sunwya',1],['kitten','sitting',3]]; for(const [a,b,e] of cases){const g=d(a,b); if(g!==e) throw new Error(a+'/'+b+' => '+g+' expected '+e);} console.log('distance OK')})"` exits 0 + - `node -e "import('./search/src/score.ts').then(m=>{if(m.scoreOne('ab','abcd')!==0) throw new Error('min-length guard'); if(!(m.scoreOne('klcc','klcc')>m.scoreOne('klcc','klca'))) throw new Error('exact must outrank fuzzy'); if(m.scoreOne('taman bangsar','taman bukit')===m.TIER.TOKEN_SET) throw new Error('token-set false positive'); console.log('score OK')})"` exits 0 + - `grep -v '^\s*//' search/src/distance.ts | grep -c "require(\|from \"" | head -1` returns 0 (the distance module has ZERO imports — D-07) + + The residual net is in place: a hand-rolled OSA distance proven against 9 hand-computed vectors, and a scorer whose 5 tiers occupy strictly disjoint descending bands so an exact match can never be outranked by a fuzzy hit. + + + + task 3: curated gazetteer JSON + lazy loader + three-environment proof + GPL/ODbL licensing + search/src/data/kl-gazetteer.json, search/src/gazetteer.ts, search/src/source.ts, search/src/index.ts, search/test/gazetteer.test.mjs, search/test/loader-env.test.mjs, LICENSE, LICENSES.md, README.md + + - `search/src/types.ts` (task 1's output — the exact contract the loader and source implement) + - `search/src/normalize.ts` (task 1's output — the loader normalizes every alias at index-build time) + - `search/src/score.ts` (task 2's output — the source ranks with it) + - `web/src/map/map.ts` (lines 19-20 — the build-time JSON-import precedent for a workspace subpath) + - `bot/test/initdata.test.mjs` (lines 42-51 — the lazy module-scope memoization pattern the loader copies) + - `bot/src/kv/sessions.ts` (lines 99-103 — the honest-null contract: `getClaimByToken` returns null on unknown, never a fabricated value) + - `README.md` (the §License section, lines ~221-225, which currently says "Private repository. No license file is present" — it must be rewritten in THIS task) + - `web/src/i18n/strings.ts` (line 73 — the `attribution: "© OpenStreetMap contributors"` string precedent) + + +**`search/src/data/kl-gazetteer.json`** — the curated gazetteer, EXACTLY per the `` block: the `_license` / `_attribution` / `_notice` header object, `"version": 1`, and an `"entries"` array of at least 30 hand-curated KL-metro landmarks (the 10 required + at least 20 more from the suggested list). Every entry carries `id` (slug, `/^[a-z0-9-]{1,64}$/`), `name` (the canonical OSM landmark name), `aliases` (non-empty, includes the canonical name and the local shorthand), `lat`, `lng`, `category`, `address`. + +The JSON must be PRETTY-PRINTED with 2-space indent (diff-friendly PRs — D-07's stated rationale: "diff-friendly PRs"). + +**`search/src/gazetteer.ts`** — the loader, per the `` block's three laws. Header comment must state the three laws verbatim (lazy, static JSON import, no logging/storage). Exports `getEntries()`, `getEntryById(id)`, `getIndex()`. + +The JSON import line (the mechanism under test in task 4's loader-env suite): + +```typescript +import raw from "./data/kl-gazetteer.json" with { type: "json" }; +``` + +If `npm run search:typecheck` or the loader-env suite rejects the import attribute, drop the attribute (esbuild and Vite inline JSON without it) — the loader-env suite is the arbiter, and the FINAL committed form must pass all three environments. + +The index build: for each entry, normalize the entry's `name` and each of its `aliases`, and map each normalized string → the entry. The map is built ONCE, lazily, memoized in module scope (`let index: Map | null = null;`). + +**`search/src/source.ts`** — the D-03 `SearchSource` implementation. Exports `createGazetteerSource(): SearchSource` and a synchronous `search(text: string, limit?: number): SearchHit[]` convenience wrapper. Behaviour: +- Normalize the query text. +- For every candidate (each entry's normalized name + each alias), score with `scoreCandidate`. +- Keep only ADMITTED hits: EXACT / ALIAS_EXACT / PREFIX / TOKEN_SET always; a FUZZY hit only when its similarity >= `FUZZY_ADMIT_SIMILARITY` (0.72). Below admission, the query is "not in the v1 area" and the result is an EMPTY array, never a nearest-match pin (D-13/D-14). +- Sort by tier band DESC, then score DESC, then entry id ASC (deterministic). +- Return at most `limit ?? 5` hits. +- `byId(id)` validates `id` against `/^[a-z0-9-]{1,64}$/` and returns the entry or `null` (honest null). + +**`search/src/index.ts`** — the public API, EXACTLY per the `` block. Named exports only. + +**`search/test/gazetteer.test.mjs`** — RED-first. Cases: +- `G1 entry schema` — every entry has a non-empty `id`, `name`, a NON-EMPTY `aliases` array, finite numeric `lat`/`lng`, a `category` from the closed set, and a non-empty `address`. +- `G2 id uniqueness + charset` — no duplicate ids; every id matches `/^[a-z0-9-]{1,64}$/`. +- `G3 the landmark names are canonical, the shorthand is an ALIAS` — assert `suria-klcc`'s `name` is "Suria KLCC" AND that "KLCC" appears in its `aliases` (RESEARCH §Verification A3: "KLCC" has no usable OSM landmark name; the alias list IS the product). +- `G4 the alias table carries the local shorthand` — assert the entries for `the-duke` include "DUKE" as an alias and that no entry's `name` equals "DUKE" (it is a highway acronym, not a landmark name). +- `G5 coordinates are in the KL metro` — every `lat` in `[2.9, 3.4]` and every `lng` in `[101.4, 102.0]`. +- `G6 ODbL header` — the JSON carries `_license` containing "ODbL", `_attribution` containing "OpenStreetMap", and `_notice` containing "opendatacommons.org/licenses/odbl". +- `G7 the LICENSE file exists and is GPL-3.0` — `LICENSE` exists at the repo root and its text contains "GNU GENERAL PUBLIC LICENSE" and "Version 3". +- `G8 the data licence is documented` — `LICENSES.md` exists and names BOTH "GPL-3.0" and "ODbL" and "OpenStreetMap". +- `G9 the README licence section is no longer self-contradictory` — `README.md` contains "GPL-3.0" and does NOT contain "all rights reserved". +- `G10 the module has no logging and no storage` — read `search/src/*.ts` as text (comments stripped) and assert NO `console.log`, NO `localStorage`, NO `sessionStorage`, NO `indexedDB`, NO `document.cookie` (D-15). +- `G11 the loader is lazy` — read `search/src/gazetteer.ts` (comments stripped) and assert the index-building code is inside a function, NOT at module top level (RESEARCH Pitfall 8: Workers "avoid expensive work in global scope"). Practical assertion: the module's top-level statements (before the first `function`/`export function` declaration) contain no `new Map(` and no JSON traversal. + +**`search/test/loader-env.test.mjs`** — THE PHASE'S TOP MECHANICAL RISK, proven mechanically. Three environments, three assertions: +- `L1 bare Node (type-stripping)` — `await import("../src/index.ts")` works and `getEntries().length >= 30`; the runner file itself IS this environment (Node 24 strips types by default). +- `L2 the Vite bundle inlines the gazetteer` — the web bundle is built by `npm run build -w theme && npm run build -w web` and the emitted `web/dist/assets/*.js` contains a gazetteer id marker (e.g. `"suria-klcc"`) — proving the JSON was inlined at BUILD time, not fetched at runtime. (The web workspace imports `@araharah/search` starting in plan 04-04; until then this case asserts the SEARCH workspace's own module can be bundled — assert it against the search workspace's own esbuild-free check: `node -e "import('./search/src/index.ts')"` PLUS a static-text assertion that `search/src/gazetteer.ts` contains a STATIC import of the JSON, not a `fetch(` or `readFileSync`. If plan 04-04 has landed by the time this suite runs, ALSO assert the web bundle marker; otherwise skip that half with an explicit `test.skip`-style guard that still asserts the static-import half. Make the skip visible in the runner output.) +- `L3 the Worker bundle inlines the gazetteer` — `cd bot && npx wrangler deploy --dry-run` (wrangler 4.151.3 supports `--dry-run`; `wrangler dev` does NOT) must succeed, and the dry-run output must show the gazetteer id marker in the bundled Worker output. Capture the dry-run output to a temp file and grep it for `suria-klcc`. `wrangler deploy --dry-run` prints the bundled worker size and (with `--outdir`) writes the bundle — use `--outdir` to a temp directory and grep the emitted JS for the marker, which is the deterministic proof. + +**`LICENSE`** — the VERBATIM GPL-3.0 text from https://www.gnu.org/licenses/gpl-3.0.txt. Do NOT paraphrase, do NOT summarize, do NOT truncate the "how to apply" section. The file must contain the strings "GNU GENERAL PUBLIC LICENSE", "Version 3, 29 June 2007", and "TERMS AND CONDITIONS". + +**`LICENSES.md`** — the dual-license note: + +```markdown +# Licences + +## Code — GPL-3.0 + +All araharah source code (the `bot/`, `web/`, `theme/`, and `search/` workspaces) is licensed under +the GNU General Public License v3.0. The full text is in [LICENSE](./LICENSE). + +GPL-3.0 is a deliberate project law (D-04): a fork must be able to read, change, and redistribute the +whole system — including the parts that make it a working navigation product. + +## Gazetteer data — ODbL-1.0 + +`search/src/data/kl-gazetteer.json` (and any index derived from it) is curated from OpenStreetMap +data and is made available under the Open Database License (ODbL) v1.0: + + +Attribution: © OpenStreetMap contributors. + +The ODbL's share-alike obligation (ODbL §4.4) applies to a Derivative Database: the curated +gazetteer and any restructured form of it carry the same ODbL notice. The normalizer's index is a +derived, restructured form of the gazetteer and therefore carries the same notice. + +Code and data are separately licensed — ODbL §2.3(a) excludes computer programs from the ODbL. +``` + +**`README.md` §License** — replace the current "Private repository. No license file is present; all +rights reserved by the project owner." section with a short statement pointing at `LICENSE` (GPL-3.0) +and `LICENSES.md` (the ODbL notice for the gazetteer). Also update the "Repository layout" code block +to include the `search/` workspace line: + +``` +search/ @araharah/search — src/types.ts, src/normalize.ts, src/score.ts, src/gazetteer.ts, src/data/kl-gazetteer.json +``` + +**Acceptance:** `npm run search:test` exits 0 (including `PASS G1`…`PASS G11` and `PASS L1`…`PASS L3`); `npm run search:typecheck` exits 0; `npx wrangler deploy --dry-run` in `bot/` succeeds with the gazetteer marker inlined. + + + cd /home/C-Fu/araharah && npm run search:test 2>&1 | tail -40 ; npm run search:typecheck 2>&1 | tail -5 ; cd bot && npx wrangler deploy --dry-run --outdir /tmp/opencode/araharah-worker-dryrun 2>&1 | tail -5 ; grep -c 'suria-klcc' /tmp/opencode/araharah-worker-dryrun/index.js 2>/dev/null || grep -rc 'suria-klcc' /tmp/opencode/araharah-worker-dryrun/ 2>/dev/null | head -1 + + + - `npm run search:test` exits 0 and the output contains `PASS G1` through `PASS G11` and `PASS L1` through `PASS L3` + - `npm run search:typecheck` exits 0 + - `cd bot && npx wrangler deploy --dry-run --outdir /tmp/opencode/araharah-worker-dryrun` exits 0 (the Worker bundle builds — the L3 half of the three-environment proof) + - `grep -rl 'suria-klcc' /tmp/opencode/araharah-worker-dryrun/ | wc -l` returns >= 1 (the gazetteer JSON is INLINED into the Worker bundle — a runtime fetch would be a hard failure in a Worker) + - `node -e "const g=require('./search/src/data/kl-gazetteer.json'); if(g.entries.length<30) throw new Error('seed too small: '+g.entries.length); if(!g._license||!g._license.includes('ODbL')) throw new Error('no ODbL header'); if(!g._attribution||!g._attribution.includes('OpenStreetMap')) throw new Error('no OSM attribution'); console.log('gazetteer OK '+g.entries.length+' entries')"` exits 0 + - `node -e "const g=require('./search/src/data/kl-gazetteer.json'); const ids=g.entries.map(e=>e.id); if(new Set(ids).size!==ids.length) throw new Error('duplicate ids'); for(const e of g.entries){ if(!/^[a-z0-9-]{1,64}$/.test(e.id)) throw new Error('bad id '+e.id); if(!e.aliases||e.aliases.length===0) throw new Error('no aliases '+e.id); if(!Number.isFinite(e.lat)||!Number.isFinite(e.lng)) throw new Error('bad coords '+e.id); if(e.lat<2.9||e.lat>3.4||e.lng<101.4||e.lng>102.0) throw new Error('out of KL bbox '+e.id);} console.log('entries OK')"` exits 0 + - `test -f LICENSE && grep -c 'GNU GENERAL PUBLIC LICENSE' LICENSE | head -1` returns >= 1 AND `grep -q 'Version 3, 29 June 2007' LICENSE` exits 0 (the VERBATIM GPL-3.0 text — D-04) + - `test -f LICENSES.md && grep -q 'ODbL' LICENSES.md && grep -q 'OpenStreetMap' LICENSES.md && grep -q 'GPL-3.0' LICENSES.md` exits 0 + - `grep -q 'GPL-3.0' README.md` exits 0 AND `grep -c 'all rights reserved' README.md | head -1` returns 0 (the README no longer contradicts itself) + - `grep -v '^\s*\*\|^\s*//\|^/\*\|^\s*$' search/src/*.ts | grep -c 'console\.log\|localStorage\|sessionStorage\|indexedDB' | head -1` returns 0 (D-15: the shared module has no logging and no storage) + - `grep -q 'import raw from "./data/kl-gazetteer.json"' search/src/gazetteer.ts` exits 0 (the STATIC JSON import — never a runtime fetch, never `readFileSync`) + - `grep -c 'readFileSync\|fetch(' search/src/gazetteer.ts | head -1` returns 0 (Workers have no fs — the loader must be pure-import) + - `node -e "import('./search/src/index.ts').then(m=>{const hits=m.search('KLCC',5); if(!hits.length) throw new Error('KLCC found nothing'); if(hits[0].entry.id!=='suria-klcc') throw new Error('KLCC => '+hits[0].entry.id); if(m.search('Kuching',3).length!==0) throw new Error('negative leaked'); if(m.search('Penang',3).length!==0) throw new Error('negative leaked'); console.log('search OK: '+hits[0].entry.id)})"` exits 0 (SRCH-01 lands; the D-13 negatives are refused honestly) + - `node -e "import('./search/src/index.ts').then(m=>{const h=m.search('Jln Ampang',5); if(!h.some(x=>x.entry.id==='suria-klcc')) throw new Error('Jln Ampang missed KLCC'); const t=m.search('KCLC',5); if(!t.some(x=>x.entry.id==='suria-klcc')) throw new Error('typo missed'); console.log('SRCH-02 OK')})"` exits 0 + + The curated KL gazetteer ships as committed, ODbL-attributed JSON with locals' aliases; the loader parses it lazily and memoizes a normalized index that survives all three bundling environments with the JSON INLINED (proven by the Worker dry-run bundle containing the id marker); the repo carries the verbatim GPL-3.0 LICENSE and the ODbL data notice; and the README no longer says "all rights reserved". + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Raw user query → normalizer/scorer | The query is untrusted, arbitrary text of arbitrary length; it enters the shared module and must never reach a log or a storage surface. | +| Committed JSON file → runtime index | The gazetteer is trusted repo content, but a malformed entry (duplicate id, non-finite coordinate) would corrupt every downstream shortlist and every venue card. | +| Bundle-time JSON import → runtime | The Worker has no `fs`; a runtime fetch of the dataset would fail at request time in a way that looks like "no results" — an honesty failure, not just an outage. | +| Module scope → Workers startup budget | The Worker has a 1 s startup limit and a 10 ms CPU/request budget; index construction at module scope is an availability threat, not a style issue. | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-04-01 | Information Disclosure | Query text reaching a log line (D-15 / V7) | mitigate | The shared module contains **no logging at all** — `search/src/*.ts` must contain zero `console.log` (G10 asserts it, comments stripped). The module also takes no logger and no storage dependency, so there is no seam through which a query could escape. The bot-side logging law (category-only) is enforced in plan 04-03. | +| T-04-02 | Information Disclosure | Query text reaching a storage surface | mitigate | The module imports no storage API and calls none (G10 bans `localStorage`/`sessionStorage`/`indexedDB`/`document.cookie` in `search/src/*.ts`). The gazetteer is a build-time constant, not user data. | +| T-04-03 | Denial of Service | Hostile query text — long string, pathological repetition — forcing an unbounded scan or allocation | mitigate | The query is bounded BEFORE this module is reached (plan 04-03 caps the raw command argument at 128 chars and rejects over-cap input). Inside the module the normalized query is tokenized once and compared against a finite, pre-built alias index; the fuzzy tier runs only on survivors of the token/prefix filters and only for tokens >= 3 characters, so the per-query work is bounded by the index size, not by the query length. The scorer refuses to score any side shorter than 3 characters. | +| T-04-04 | Denial of Service | Index construction at module scope blowing the Workers 1 s startup budget (RESEARCH Pitfall 8) | mitigate | LAW 1 in the loader: the index is built lazily inside `loadIndex()` and memoized in module scope — never at module top level (G11 asserts the top level contains no `new Map(` and no JSON traversal). The same pattern as `bot/test/initdata.test.mjs`'s lazy validator load. | +| T-04-05 | Denial of Service | Naive full-scan Damerau over ~2500 alias pairs costing ~8 ms/query against a 10 ms Workers Free CPU limit (RESEARCH §Verification A1 — REPRODUCED at 8.243 ms) | mitigate | The scorer is TIERED: the EXACT / ALIAS_EXACT / PREFIX / TOKEN_SET tiers are string comparisons against a pre-built hash index (O(1) per candidate), and the Damerau FUZZY tier runs only on candidates that survive the cheap tiers AND have both sides >= 3 characters. The index maps each normalized alias directly to its entry, so an exact alias hit never touches the fuzzy tier at all. Plan 04-04's timing assertion measures the whole query path. | +| T-04-06 | Tampering | A malformed committed entry (duplicate id, non-finite coordinate, empty alias list) reaching a venue card as a fabricated or broken pin | mitigate | G1/G2/G5 assert the dataset's invariants as a gate: unique slug-safe ids, non-empty aliases, finite coordinates inside the KL bbox. A non-finite coordinate would surface as `NaN` on the venue card — the gate makes that impossible to ship. | +| T-04-07 | Tampering | An out-of-area query answered with the nearest KL pin (a fabricated location) — an honesty failure (RESEARCH §Security V8, D-13/D-14) | mitigate | The tiered admission rule: EXACT/ALIAS_EXACT/PREFIX/TOKEN_SET always admitted; a FUZZY hit only at similarity >= `FUZZY_ADMIT_SIMILARITY` (0.72). "Kuching" and "Penang" must yield `[]` — asserted in this plan's acceptance criteria and re-asserted by the corpus runner in plan 04-02. An empty shortlist is a legitimate, load-bearing return value of the `SearchSource` contract, and the bot renders the honest refusal copy (plan 04-03). | +| T-04-08 | Tampering | The rule table growing a code-shaped entry point — a rule that changes behavior via code rather than a table edit (D-09's "extends by table edit, not code change" law) | mitigate | T9 asserts the rules file contains no algorithm and the row count is >= 47; the table is a `ReadonlyArray` of pairs, inverted once by a single generic builder. Any new abbreviation is a row, not a branch. | +| T-04-09 | Repudiation | The dataset's licence obligation unmet — the gazetteer shipped without the ODbL notice (ODbL §4.4 share-alike; OSMF attribution guidelines) | mitigate | The `_license` / `_attribution` / `_notice` header travels INSIDE the JSON (G6), `LICENSES.md` names both licences and the OSM attribution (G8), and the README points at both (G9). Three surfaces, so no single edit can leave the notice missing. | +| T-04-10 | Spoofing | A caller substituting a different `SearchSource` implementation to bypass the refusal floor | mitigate | The `SearchSource` interface is the seam (D-03), but the refusal contract is part of the INTERFACE's documented semantics, and the corpus runner exercises the concrete gazetteer source. A future backend (#2/#3) is a documented escalation path (D-05), not a runtime switch — there is no env flag that swaps the backend in production. | +| T-04-11 | Elevation of Privilege | The shared module reading env/config to reach a privileged backend | mitigate | The module reads NO environment, NO secrets, and NO network. It has exactly two imports (its own rules table and its own JSON data). The forkability law (D-04: "no hardcoded secrets — env only") is trivially satisfied because there is nothing to configure. | + + + +- `npm run search:test` green — 9 normalizer vectors + 9 distance vectors + 7 score vectors + 11 gazetteer/license vectors + 3 loader-environment vectors, all named +- `npm run search:typecheck` green (`tsc --noEmit`, `strict` + `noUncheckedIndexedAccess`) +- `cd bot && npx wrangler deploy --dry-run` exits 0 AND the emitted bundle contains `suria-klcc` — the Worker bundle INLINES the dataset (never a runtime fetch) +- `npm test -w bot` and `npm test -w web` still green (this plan adds no bot/web source, so the existing suites must be unaffected) +- `node -e "import('./search/src/index.ts')..."` proves `search('KLCC')` → `suria-klcc` at rank 1 and `search('Kuching')` → `[]` +- `git status` shows no build output staged (`search/` is buildless — nothing under `search/dist/` may exist) + + + +- SRCH-01 (foundation): a query for "KLCC" resolves to the `suria-klcc` entry through the shared module with no network and no infrastructure; the same code path serves the bot, the Mini App, and the corpus runner +- SRCH-02 (foundation): "Jln Ampang" and "Jalan Ampang" normalize identically; "Tmn"→"taman"; "Kg"→"kampung"; "Prsn"→"presint"; "tamanbangsar" splits to "taman bangsar"; a transposition typo ("KCLC") still resolves to the right entry through the Damerau residual tier; an exact match always outranks a fuzzy one (disjoint tier bands, asserted) +- D-09: the abbreviation table is DATA with 47+ JUPEM-derived rows plus the explicit Presint/Prsn vowel-drop row; extending it is a file edit plus a vector, never a code change +- D-07/D-08: the dataset is typed, committed, diff-friendly JSON; each entry carries its locals' shorthand as native aliases ("KLCC" → suria-klcc, "DUKE" → the-duke) with the canonical OSM name as the `name` +- D-03: the `SearchSource` seam exists with an async `search()` and a synchronous `byId()`, so backend #2 (WASM geocoder) and #3 (Photon) are format changes rather than rewrites +- D-15: the shared module contains no logging and no storage surface at all +- D-04: the repo carries the verbatim GPL-3.0 LICENSE, `LICENSES.md` records the ODbL-1.0 + OpenStreetMap notice for the gazetteer, and the README no longer claims "all rights reserved" +- The three-environment loader risk is closed mechanically: bare Node, the Vite bundle, and the Worker dry-run bundle all resolve the dataset with the JSON INLINED at build time + + + +After completion, create `.planning/phases/04-destination-search-kl-gazetteer/04-01-SUMMARY.md` + diff --git a/.planning/phases/04-destination-search-kl-gazetteer/04-02-PLAN.md b/.planning/phases/04-destination-search-kl-gazetteer/04-02-PLAN.md new file mode 100644 index 0000000..b9e21e6 --- /dev/null +++ b/.planning/phases/04-destination-search-kl-gazetteer/04-02-PLAN.md @@ -0,0 +1,563 @@ +--- +phase: 04-destination-search-kl-gazetteer +plan: 02 +type: execute +wave: 2 +depends_on: [04-01] +files_modified: + - search/test/fixtures/golden-corpus.json + - search/test/corpus.test.mjs + - search/scripts/gazetteer-corpus.mjs + - search/scripts/overpass-topup.mjs + - search/scripts/json-to-sqlite.mjs + - search/test/topup.test.mjs + - search/test/converter.test.mjs + - search/test/fixtures/overpass-response.json + - search/package.json + - package.json + - search/src/data/kl-gazetteer.json + - search/src/rules/abbreviations.ts + - search/test/normalizer.test.mjs + - docs/SEARCH-QUALITY.md +autonomous: true +requirements: [SRCH-01, SRCH-02] +must_haves: + decisions_implemented: + - "D-06: the Overpass top-up is a REPO TOOL, run manually by a maintainer, that PRINTS candidates — it never writes the curated file and is never imported by the Worker" + - "D-05: the JSON dataset format stays reachable from day one — a JSON→SQLite converter script exists and preserves the id set, so backend #2 (WASM geocoder) is a format change, not a rewrite" + - "D-13: the golden corpus is HAND-WRITTEN and PINNED — ~60-100 cases including deliberate out-of-KL negatives, each positive pinning its expected entry id" + - "D-14: the bake-off closes on a MECHANICAL threshold — top-3 contains the expected id for >= 90% of positives AND every negative is refused — and the phase records which escalation lever was pulled" + - "D-03: the corpus runner exercises the SAME shared module the bot and the Mini App use, so the measured quality IS the shipped quality" + truths: + - "The golden corpus contains at least 60 hand-written cases across 6 categories: golden landmarks, abbreviations, glued words, typos, aliases, and deliberate out-of-KL negatives" + - "At least 15 corpus queries are NOT string-equal to any alias in the gazetteer — the corpus measures something real, it is not tautological" + - "The corpus runner exits 0 only when top-3 >= 90% on positives AND every negative returned an empty shortlist" + - "The corpus runner reports per-category rates so a regression is diagnosable, not a diffuse number" + - "The Overpass top-up script produces a valid-shaped proposal file from a COMMITTED FIXTURE, and treats 502/429/timeout as a normal retryable outcome that EXITS 0" + - "The Overpass script sends an identifying User-Agent (a request without one gets HTTP 406) and makes exactly ONE request per run" + - "The Overpass script never writes to kl-gazetteer.json and no OSM/Overpass URL appears anywhere under bot/src/ or web/src/" + - "The JSON→SQLite converter reads the same committed JSON and emits a file with the SAME id set — backend #2's format is reachable today" + - "The corpus outcome and the escalation lever pulled are recorded in docs/SEARCH-QUALITY.md" + artifacts: + - path: "search/test/fixtures/golden-corpus.json" + provides: "the pinned D-13 golden corpus: ~60-100 hand-written cases with expected ids and negatives" + contains: "\"cases\"" + - path: "search/scripts/gazetteer-corpus.mjs" + provides: "the D-14 corpus runner — top-1/top-3 on positives, refusal rate on negatives, exit 1 below threshold" + - path: "search/test/corpus.test.mjs" + provides: "the suite that makes `npm run search:test` FAIL when the corpus regresses (the Nyquist gate)" + - path: "search/scripts/overpass-topup.mjs" + provides: "the D-06 maintainer-time Overpass candidate proposer — one request, UA header, soft-exit on busy instances" + - path: "search/test/fixtures/overpass-response.json" + provides: "a committed real Overpass JSON response so the top-up test is deterministic (no live network)" + - path: "search/scripts/json-to-sqlite.mjs" + provides: "the D-05 backend-#2 format converter stub — same ids, SQLite output" + - path: "docs/SEARCH-QUALITY.md" + provides: "the bake-off close record: corpus version, measured rates, the lever that was pulled" + key_links: + - from: "search/scripts/gazetteer-corpus.mjs" + to: "search/test/fixtures/golden-corpus.json" + via: "the runner reads the pinned corpus and asserts each positive's expected id" + pattern: "golden-corpus.json" + - from: "search/scripts/gazetteer-corpus.mjs" + to: "search/src/index.ts" + via: "the runner imports the SAME shared search module the bot uses — the measured quality is the shipped quality" + pattern: "from \"../src/index.ts\"" + - from: "search/test/corpus.test.mjs" + to: "search/scripts/gazetteer-corpus.mjs" + via: "the suite re-runs the runner's assertions in-process so `npm run search:test` fails on a regression" + pattern: "search" + - from: "search/scripts/overpass-topup.mjs" + to: "search/test/fixtures/overpass-response.json" + via: "the fixture is the script's offline input path — the live Overpass endpoint is unreachable from this environment" + pattern: "overpass" + - from: "search/scripts/json-to-sqlite.mjs" + to: "search/src/data/kl-gazetteer.json" + via: "the converter reads the SAME committed dataset — backend #2's format is reachable today" + pattern: "kl-gazetteer.json" +--- + + +Close the search-quality experiment: pin the golden corpus (D-13), build the runner that mechanically enforces the D-14 threshold, land the two maintainer-time repo tools D-06 and D-05 require (the Overpass candidate proposer and the JSON→SQLite converter), and record the bake-off outcome in `docs/SEARCH-QUALITY.md`. + +Purpose: D-14 says the phase ends with "a documented decision, not an open question" — the corpus must show top-3 ≥ 90% on positives with every negative refused, and the escalation lever that was pulled must be recorded. This plan builds the instrument that measures it AND the two repo tools the forkability law (D-04, D-06) demands: the Overpass script proposes top-up candidates for human approval (never a runtime dependency), and the JSON→SQLite converter keeps backend #2's dataset format reachable from day one (D-05). + +Output: `search/test/fixtures/golden-corpus.json` (60+ hand-written cases), `search/scripts/gazetteer-corpus.mjs` (the D-14 runner), `search/test/corpus.test.mjs` (the Nyquist gate inside `npm run search:test`), `search/scripts/overpass-topup.mjs` + a committed Overpass fixture + `search/test/topup.test.mjs`, `search/scripts/json-to-sqlite.mjs` + `search/test/converter.test.mjs`, and `docs/SEARCH-QUALITY.md`. + + + +@$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md +@$HOME/.config/opencode/get-shit-done/templates/summary.md + + + +@AGENTS.md +@.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md +@.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md +@.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md +@search/src/index.ts +@search/src/types.ts +@search/src/data/kl-gazetteer.json +@bot/scripts/register-commands.mjs +@bot/scripts/set-webhook.mjs +@bot/test/privacy.test.mjs +@bot/test/harness.mjs +@bot/test/runner.mjs + + +From `search/src/index.ts` (plan 04-01's output — the surface every script in this plan imports): + +```typescript +export type { GazetteerEntry, SearchQuery, SearchHit, SearchSource } from "./types.ts"; +export { normalize } from "./normalize.ts"; +export { TIER } from "./score.ts"; +export { damerau } from "./distance.ts"; +export { getEntries, getEntryById, getIndex } from "./gazetteer.ts"; +export { createGazetteerSource, search } from "./source.ts"; + +// search(text: string, limit?: number): SearchHit[] — synchronous convenience wrapper +// getEntryById(id: string): GazetteerEntry | null — honest null on unknown +// getEntries(): readonly GazetteerEntry[] +``` + +From `search/src/types.ts`: + +```typescript +export interface SearchHit { + entry: GazetteerEntry; + score: number; + matchedAlias: string; +} +``` + +The corpus runner and both scripts import from `../src/index.ts` — Node 24 strips the types by default, so a `.mjs` script CAN import the `.ts` source directly (the repo's existing pattern; `bot/test/commands.test.mjs` line 27 does exactly this and passes). + + + +`search/test/fixtures/golden-corpus.json` — the pinned D-13 corpus. The schema carries a per-case `kind` so the runner can report per-category rates (RESEARCH Open Question 3: "measure and report per-category rates … so a regression is diagnosable"). + +```json +{ + "version": 1, + "note": "Hand-written and pinned (D-13). NOT generated from the gazetteer — a generated corpus is tautological. Reviewers extend this file by editing it.", + "threshold": { "top3_hit_rate": 0.9 }, + "cases": [ + { "q": "KLCC", "expectId": "suria-klcc", "kind": "landmark" }, + { "q": "Jln Ampang", "expectId": "suria-klcc", "kind": "abbreviation" }, + { "q": "tamanbangsar", "expectId": "bangsar", "kind": "glued" }, + { "q": "KCLC", "expectId": "suria-klcc", "kind": "typo" }, + { "q": "IJN", "expectId": "ijnp", "kind": "alias" }, + { "q": "Kuching", "expectId": null, "kind": "negative" } + ] +} +``` + +Rules: +- `q` — the raw query text exactly as a local would type it (mixed case, abbreviations, typos, glued words). +- `expectId` — the entry id for positives; `null` for negatives. +- `kind` — one of `landmark | abbreviation | glued | typo | alias | negative`. +- `threshold.top3_hit_rate` — `0.9` (D-14's fixed number; the runner reads it from the file so a future edit is a file edit). + +**Minimum size: 60 cases.** (The specific query strings below are ILLUSTRATIVE — pin every case to an entry id that actually exists in plan 04-01's shipped `search/src/data/kl-gazetteer.json`; a case whose `expectId` has no entry fails the runner.) Category floors: +- `landmark` — >= 12 (the golden landmarks and other well-known KL places: "KLCC", "Sunway Pyramid", "DUKE", "Pavilion", "Mid Valley", "Berjaya Times Square", "IOI City Mall", "Merdeka 118", "The Intermark", "KL Sentral", "Nu Sentral", "PWTC", "IJN", "Taman Bangsar") +- `abbreviation` — >= 15 ("Jln Ampang", "Jalan Ampang", "Tmn Bangsar", "Taman Bangsar", "Kg Ampang", "Kampung Ampang", "Bdr Baru", "Bandar Baru", "Lbh Ampang", "Lebuh Ampang", "Psrn KL", "Persiaran KL", "Sg Wang", "Sungai Wang", "Blk A") +- `glued` — >= 6 ("tamanbangsar", "jalankanpar", "kampungbarra", "bandarbaru", "sungaiwang", "bukitbintang") +- `typo` — >= 12 ("KCLC", "KLCC" transposed, "Sunwya Pyarmid", "Pavillion", "Mid Vally", "Berjaya Time Square", "IJN" → "IJNP", "Sentral" → "Sentrl", "Merdeka 118" → "Merdeka 118", "Tmn Bangsr", "Pavilon", "Intermrk") +- `alias` — >= 12 ("IJN", "PWTC", "Stesen KL Sentral", "KL Central", "BTS", "MV", "IOI", "Sunway Velocity", "Velocity", "Cheras South", "Pudu", "Intermark") +- `negative` — >= 6 ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") — these MUST be refused with an empty shortlist, never a fabricated pin + +**TAUTOLOGY LAW (D-13 / RESEARCH Pitfall 7):** the corpus must contain queries that are NOT string-equal to any alias in the gazetteer. The typo and glued buckets are exactly those queries — and at least 15 cases overall must not equal any alias. The runner ASSERTS this (`cases.filter(c => c.kind !== 'negative' && !aliases.has(normalize(c.q))).length >= 15`), so a future regeneration that makes the corpus tautological fails the suite. + +**Every `expectId` MUST match `/^[A-Za-z0-9_-]{1,64}$/`** — the start_param-safe charset (RESEARCH: "The corpus runner should assert every corpus `expectId` matches `/^[A-Za-z0-9_-]{1,64}$/`"). + +**Every positive's `expectId` MUST exist in the gazetteer** — asserted by the runner. A corpus case pointing at a non-existent entry fails the suite immediately (that is a corpus bug, not a search bug). + + + +`search/scripts/gazetteer-corpus.mjs` — the D-14 bake-off close. Copy `bot/scripts/register-commands.mjs`'s script conventions (shebang + a header contract block stating purpose, inputs, what is echoed, and the exit-code contract) and `bot/scripts/set-webhook.mjs`'s single-fetch-then-report shape. + +Behaviour: +1. Import `search` from `../src/index.ts` (the same module the bot uses). +2. Read the corpus JSON from `search/test/fixtures/golden-corpus.json` via `readFileSync` + `JSON.parse` (a script may use `fs` — only the WORKER may not). +3. Split the cases into positives (`expectId !== null`) and negatives (`expectId === null`). +4. For each positive: `search(c.q, 3)` → record whether `hits[0].entry.id === c.expectId` (top-1) and whether `hits.map(h => h.entry.id).includes(c.expectId)` (top-3). Record the miss as `" -> [] (expected )"`. +5. For each negative: `search(c.q, 3)` → if `hits.length > 0`, record the leak as `" -> "`. +6. Print a report to stdout: + - the overall line: `top-1 / top-3 / (%)` + - a PER-CATEGORY breakdown: one line per `kind` with that category's top-3 rate + - `negatives refused /` + - the `MISSED:` block (one line per miss) and the `LEAKED (must refuse):` block (one line per leak) + - the threshold line: `threshold top3 >= 90.0% — PASS|FAIL` +7. Exit code: `0` only when `top3Rate >= threshold.top3_hit_rate` AND `leaked.length === 0`; `1` otherwise. + +**The runner MAY print query text** — it is a local dev tool over a committed file (RESEARCH: "The corpus runner (a local dev tool over a committed file) MAY print queries; the Worker NEVER may"). The runner MUST print the ESCALATION LADDER when it fails, verbatim: + +``` +D-14 threshold not met. Escalation order is FIXED: + (1) alias/normalizer table edits in search/ -> re-run this runner + (2) SearchSource backend #2 — the Malaysia-only WASM geocoder (D-05, post-v1) + (3) Photon-on-VPS (D-05, Phase 5) +Record which lever was pulled in docs/SEARCH-QUALITY.md. +``` + +**Also add a root delegate** `"search:corpus": "node search/scripts/gazetteer-corpus.mjs"` so the D-14 gate is one command. + +**The runner must ALSO be assertable in-process.** `search/test/corpus.test.mjs` re-runs the same measurements by importing the shared module and the corpus directly (not by spawning the script) — so `npm run search:test` fails when the corpus regresses (the Nyquist requirement: "the corpus must be ≥ threshold at every wave merge, not just at the end"). The suite must assert the SAME things the script does: top-3 rate >= threshold, zero negative leaks, the tautology floor, the id charset, and that every positive's `expectId` exists in the gazetteer. + + + +`search/scripts/overpass-topup.mjs` — the D-06 maintainer tool. HARD LAWS: + +**LAW 1 — it is a REPO TOOL, never a runtime dependency.** It lives under `search/scripts/`, is run manually, and is NEVER imported by `bot/src/` or `web/src/`. A grep gate asserts no Overpass/OSM URL appears under either source tree (RESEARCH Pitfall 6: "Warning signs: `fetch("https://overpass-api.de/…")` anywhere under `bot/src/` or `web/src/`"). + +**LAW 2 — ONE request per run.** The script makes exactly ONE Overpass request and prints the candidate list. It never loops over tag families. + +**LAW 3 — an identifying User-Agent is MANDATORY.** A request without a UA gets HTTP 406 (live-verified 2026-10-23: `curl https://overpass-api.de/api/status` with no UA → HTTP 406). The header is exactly: + +``` +User-Agent: araharah-gazetteer-topup/1.0 (+https://github.com/araharah) +``` + +**LAW 4 — a busy instance is a NORMAL outcome, and the script EXITS 0.** Overpass capacity saturation is intermittent (RESEARCH §Verification A4 observed 502 and 200 minutes apart). On 502 / 429 / a network timeout / a non-200 status the script prints exactly one line: + +``` +Overpass instance busy — retry later +``` + +and exits **0** (never a stack trace, never a non-zero code). Nothing in the repo — no deploy, no test run, no corpus gate — may depend on this script succeeding (RESEARCH §Verification A4 item 4). + +**LAW 5 — it NEVER writes to `kl-gazetteer.json`.** Output is a PRINTED candidate list (or a proposal file at an explicit `--out` path that the maintainer diffs and merges by hand). The script writes nothing to the curated dataset under any flag. The word "proposal" must appear in the output header. + +**LAW 6 — the source and the endpoint are overridable.** `--source=overpass|nominatim|fixture` selects the candidate source (default `overpass`); `--endpoint=` and the `OVERPASS_ENDPOINT` env var override the Overpass endpoint (default `https://overpass-api.de/api/interpreter`); `--fixture=` names the fixture file for `--source=fixture`. `--source=nominatim --q=` queries the reachable Nominatim search endpoint bounded to the KL viewbox instead (Nominatim was live-verified reachable 2026-10-23 while the Overpass mirrors were not). This is the resolved offline posture: the committed fixture is the deterministic test path, Nominatim is the alternative live path, and neither is ever a runtime dependency. + +**The query shape** — a NARROW, single-tag-family query bounded to the KL bbox (Overpass bbox order is `s,w,n,e` = south, west, north, east): + +``` +[out:json][timeout:25]; +nwr["tourism"="attraction"](3.05,101.55,3.25,101.80); +out center 40; +``` + +The bbox MUST be present (RESEARCH §Anti-Patterns: "An Overpass query with no bbox… will time out and is a well-known way to get rate-limited"). The default tag family is `tourism=attraction`; `--tag-family=` may override it with ONE family (never a union of families). + +**The offline + alternate paths.** `--source=fixture --fixture=` reads a committed Overpass JSON response instead of making a request — this is how the test exercises the script deterministically (RESEARCH §Verification A4: the committed fixture is the reliability anchor; the live primary instance works only with a `User-Agent` and saturates intermittently). `--source=nominatim --q=` queries `https://nominatim.openstreetmap.org/search?format=jsonv2&q=&viewbox=101.55,3.25,101.80,3.05&bounded=1&limit=40` with the same identifying-User-Agent discipline (Nominatim's usage policy requires one too) and emits the same PROPOSAL shape. + +**The output format** — one candidate per line, with the fields a maintainer needs to decide: + +``` +PROPOSAL — 7 candidate(s) from tourism=attraction in bbox 3.05,101.55,3.25,101.80 +Review and hand-merge into search/src/data/kl-gazetteer.json — this script NEVER writes it. + name=Merdeka 118 osm=way/12345678 lat=3.1418 lon=101.6943 category=landmark + name=National Monument osm=node/87654321 lat=3.1489 lon=101.6936 category=landmark +``` + +**`search/test/fixtures/overpass-response.json`** — a committed, REAL Overpass JSON response (the `elements` array shape: each element has `type`, `id`, `lat`/`lon` for nodes or `center: {lat, lon}` for ways/relations, and `tags`). Use 5-8 elements with realistic KL tag data. The fixture makes the top-up test deterministic — no live network in the test suite. + +**`search/test/topup.test.mjs`** — cases: +- `TP1 the script is not imported by the Worker` — grep `bot/src/**` and `web/src/**` for `overpass` / `overpass-api.de` / `overpass-api` → ZERO hits. +- `TP2 the script sends an identifying User-Agent` — the script source contains `User-Agent: araharah-gazetteer-topup/1.0`. +- `TP3 the query is bbox-bounded` — the script source contains a bbox tuple matching `/\(\d+\.\d+,\s*\d+\.\d+,\s*\d+\.\d+,\s*\d+\.\d+\)/`. +- `TP4 the script exits 0 on a busy instance` — run the script with `--fixture=` pointing at a fixture whose content is NOT an Overpass response (e.g. a JSON object with `"error"`), assert the process exit code is 0 and stdout contains "Overpass instance busy". +- `TP5 the script produces a valid-shaped proposal from the committed fixture` — run the script with `--fixture=search/test/fixtures/overpass-response.json`, assert exit 0 and that stdout contains "PROPOSAL", at least one `name=` line, and the literal text "NEVER writes". +- `TP6 the script never writes the curated file` — assert the script source contains no `writeFileSync` targeting `kl-gazetteer.json` (the only permitted write is an explicit `--out` proposal path, and that path must not be the curated file — assert the source does not contain the literal `kl-gazetteer.json` at all). +- `TP7 exactly one request per run` — assert the script source contains exactly ONE `fetch(` call. + +**`search/scripts/json-to-sqlite.mjs`** — the D-05 backend-#2 format converter. It reads `search/src/data/kl-gazetteer.json` and emits a SQLite file with the SAME ids, so backend #2's dataset format is reachable today (D-05: "The JSON↔SQLite converter script keeps its dataset format reachable from day one"). It is a STUB in the sense that the SQLite WRITER is a minimal, dependency-free implementation (Node 24 has `node:sqlite` behind a flag — check whether `node:sqlite` is available in Node 24.18.0; if it is, use it; if not, emit a deterministic, documented, plain-text `.sql` DUMP file plus a JSON sidecar and say so in the header — do NOT add a dependency). + +**`search/test/converter.test.mjs`** — cases: +- `CV1 the converter preserves the id set` — run the converter into a temp directory, read the output's id list, and assert it is EXACTLY the gazetteer's id set (same members, same count). +- `CV2 the converter reads the committed dataset` — the script source references `kl-gazetteer.json`. +- `CV3 the converter is not a runtime dependency` — grep `bot/src/**` + `web/src/**` for `json-to-sqlite` / `sqlite` → ZERO hits. + +**`docs/SEARCH-QUALITY.md`** — the bake-off close record. Structure: + +```markdown +# Search quality — Phase 4 bake-off close + +**Corpus:** `search/test/fixtures/golden-corpus.json` version 1 — hand-written cases +**Gazetteer:** `search/src/data/kl-gazetteer.json` version 1 — entries +**Measured:** + +| Metric | Value | Threshold | +|---|---|---| +| top-1 hit rate (positives) | / (%) | — (reported) | +| top-3 hit rate (positives) | / (%) | >= 90% | +| negatives refused | / | 100% (zero leaks) | + +### Per-category + +| Category | Cases | top-3 | +|---|---|---| +| landmark | … | … | +| abbreviation | … | … | +| glued | … | … | +| typo | … | … | +| alias | … | … | + +### Escalation lever pulled (D-14) + + +1. **None — the threshold was met on the first run.** No table edits were needed. +2. **Lever 1 — alias/normalizer table edits.** +3. **Lever 2 — SearchSource backend #2 (WASM geocoder).** <…> +4. **Lever 3 — Photon-on-VPS.** <…> + +### Known limitations + +- The corpus is hand-written and small; it measures the documented local shorthand, not every + phrasing a KL driver might use. +- Out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1. +``` + +**Acceptance:** `npm run search:corpus` exits 0 with top-3 >= 90% and zero negative leaks; `npm run search:test` green; `docs/SEARCH-QUALITY.md` records the measured numbers and the lever. + + + + + + task 1: the pinned golden corpus + the D-14 runner + the Nyquist gate + search/test/fixtures/golden-corpus.json, search/scripts/gazetteer-corpus.mjs, search/test/corpus.test.mjs, search/package.json, package.json + + - `search/src/index.ts` (plan 04-01's output — the `search()` surface the runner calls) + - `search/src/data/kl-gazetteer.json` (plan 04-01's output — the ids and aliases the corpus must pin) + - `bot/scripts/register-commands.mjs` (the script conventions: shebang, header contract block, exit-code contract) + - `bot/scripts/set-webhook.mjs` (the single-fetch-then-report shape, and the "missing vars → fatal" resolution pattern) + - `bot/test/harness.mjs` + `bot/test/runner.mjs` (the suite style the new suite copies) + - `.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md` (§"The corpus runner" code example and the §"Common Pitfalls" #7 tautology law) + + +**`search/test/fixtures/golden-corpus.json`** — the pinned corpus, EXACTLY per the `` block: the header object (`version`, `note`, `threshold.top3_hit_rate: 0.9`) and at least 60 `cases` with the per-category floors (landmark >= 12, abbreviation >= 15, glued >= 6, typo >= 12, alias >= 12, negative >= 6). + +Every `expectId` MUST be a real id in `search/src/data/kl-gazetteer.json`. The corpus's negatives MUST be genuinely out-of-area ("Kuching", "Penang", "Johor Bahru", "Kota Kinabalu", "Singapore", "London") so the refusal path is exercised. + +**`search/scripts/gazetteer-corpus.mjs`** — the runner, EXACTLY per the `` block: shebang + header contract block, the report format (overall line, PER-CATEGORY breakdown, negatives-refused line, `MISSED:` block, `LEAKED (must refuse):` block, threshold line), the fixed escalation ladder printed on failure, and the exit-code contract (`0` only when top-3 >= threshold AND zero leaks). + +Add to `search/package.json`: + +```json +"corpus": "node scripts/gazetteer-corpus.mjs" +``` + +and to the root `package.json`: + +```json +"search:corpus": "node search/scripts/gazetteer-corpus.mjs", +``` + +**`search/test/corpus.test.mjs`** — the Nyquist gate (RED-first). Cases: +- `CO1 top-3 hit rate meets the D-14 threshold` — `top3 / positives.length >= corpus.threshold.top3_hit_rate`; the failure message names the D-14 threshold and the escalation ladder's first lever. +- `CO2 every negative is refused` — zero leaks; the message names the leaking query. +- `CO3 the corpus is not tautological` — at least 15 non-negative cases whose normalized query is NOT string-equal to any alias or name in the gazetteer (RESEARCH Pitfall 7). +- `CO4 every positive's expectId exists in the gazetteer` — no dangling ids. +- `CO5 every expectId matches the start_param-safe charset` — `/^[A-Za-z0-9_-]{1,64}$/`. +- `CO6 the corpus meets its size and category floors` — >= 60 cases; landmark >= 12, abbreviation >= 15, glued >= 6, typo >= 12, alias >= 12, negative >= 6. +- `CO7 every category's top-3 rate is reported` — the suite prints (not asserts) the per-category rates so the runner output and the suite output agree. + +The suite imports the shared module and the corpus directly (in-process), so `npm run search:test` fails on a corpus regression. + +**Acceptance:** `npm run search:corpus` exits 0; `npm run search:test` green with `PASS CO1`…`PASS CO7`. + + + cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -30 ; npm run search:test 2>&1 | grep -E 'PASS CO|runner' ; node -e "const c=require('./search/test/fixtures/golden-corpus.json'); if(c.cases.length<60) throw new Error('corpus too small: '+c.cases.length); console.log('corpus OK '+c.cases.length+' cases')" + + + - `npm run search:corpus` exits 0 + - `npm run search:corpus` output contains a `top-3` line with a percentage >= 90.0% and a `negatives refused` line showing all negatives refused + - `npm run search:corpus` output contains one `PASS`/`FAIL` threshold line naming the 90.0% gate + - `npm run search:test` exits 0 and the output contains `PASS CO1` through `PASS CO7` + - `node -e "const c=require('./search/test/fixtures/golden-corpus.json'); const g=require('./search/src/data/kl-gazetteer.json'); const ids=new Set(g.entries.map(e=>e.id)); const kinds={}; let bad=0; for(const x of c.cases){ kinds[x.kind]=(kinds[x.kind]||0)+1; if(x.expectId!==null && !ids.has(x.expectId)) bad++; if(x.expectId!==null && !/^[A-Za-z0-9_-]{1,64}$/.test(x.expectId)) bad++; } if(c.cases.length<60) throw new Error('size'); if(bad) throw new Error('dangling ids'); console.log(JSON.stringify(kinds))"` exits 0 and prints all six `kind` keys with landmark>=12, abbreviation>=15, glued>=6, typo>=12, alias>=12, negative>=6 + - `grep -q '"search:corpus": "node search/scripts/gazetteer-corpus.mjs"' package.json` exits 0 AND `grep -q '"corpus": "node scripts/gazetteer-corpus.mjs"' search/package.json` exits 0 + - `grep -q 'D-14 threshold not met' search/scripts/gazetteer-corpus.mjs` exits 0 AND `grep -q 'backend #2' search/scripts/gazetteer-corpus.mjs` exits 0 AND `grep -q 'Photon-on-VPS' search/scripts/gazetteer-corpus.mjs` exits 0 (the fixed escalation ladder is printed verbatim on failure) + - `grep -c 'expectId' search/test/fixtures/golden-corpus.json | head -1` returns >= 60 (each case pins an expected id or null) + + The golden corpus is pinned at 60+ hand-written cases across 6 categories with real negatives, the runner mechanically enforces D-14's top-3 >= 90% + zero-leak threshold, and `npm run search:test` fails whenever the corpus regresses. + + + + task 2: the Overpass top-up tool (D-06) + the JSON→SQLite converter (D-05) + search/scripts/overpass-topup.mjs, search/test/fixtures/overpass-response.json, search/test/topup.test.mjs, search/scripts/json-to-sqlite.mjs, search/test/converter.test.mjs + + - `bot/scripts/set-webhook.mjs` (the script conventions: shebang, header contract block, env/flag resolution, one fetch + a category-only failure print, the exit-code contract) + - `bot/scripts/register-commands.mjs` (the `.dev.vars` resolution pattern the `--endpoint` / `OVERPASS_ENDPOINT` override mirrors) + - `search/src/data/kl-gazetteer.json` (the dataset the converter reads and the ids it must preserve) + - `bot/test/privacy.test.mjs` (the source-introspection gate style the TP1/TP7 greps copy) + - `search/test/harness.mjs` (the suite style) + - `.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md` (§"Common Pitfalls" #6 and §"Verification ledger" A4 — the soft-exit posture) + + +**`search/scripts/overpass-topup.mjs`** — the D-06 tool, EXACTLY per the `` block's six laws: repo tool (never imported by `bot/src/` or `web/src/`), ONE request per run, the mandatory identifying User-Agent, a busy instance is a NORMAL outcome that EXITS 0, it NEVER writes the curated file, and the endpoint is overridable via `--endpoint=` or `OVERPASS_ENDPOINT`. + +Header contract block (copy `set-webhook.mjs`'s shape — purpose, inputs, what is echoed/never echoed, exit-code contract): + +```javascript +#!/usr/bin/env node +/* + * overpass-topup.mjs — propose KL gazetteer top-up candidates from OpenStreetMap (D-06). + * ------------------------------------------------------------------------------------- + * A MAINTAINER-TIME REPO TOOL. It is NOT a runtime dependency: nothing in + * bot/src/ or web/src/ imports it, and no deploy, test, or corpus gate depends on + * it succeeding. The Worker only ever reads the COMMITTED curated JSON. + * + * Output is a PROPOSAL — a printed candidate list (or a file at an explicit --out + * path) that a human diffs and hand-merges into search/src/data/kl-gazetteer.json. + * THIS SCRIPT NEVER WRITES THE CURATED FILE. + * + * Exit codes: 0 on success AND on a busy instance (Overpass capacity saturation + * is intermittent — 502/429/timeout is a NORMAL, retryable outcome). Non-zero + * only on a usage error (no bbox resolvable / unreadable fixture). + * + * Inputs: --source=overpass|nominatim|fixture (default overpass), --endpoint= + * (or OVERPASS_ENDPOINT), --bbox=s,w,n,e, --q= (nominatim source), + * --tag-family==, --fixture= (offline input), + * --out= (write the proposal here — never the curated file). + */ +``` + +The request carries EXACTLY this header set: + +```javascript +const headers = { + "User-Agent": "araharah-gazetteer-topup/1.0 (+https://github.com/araharah)", + "Content-Type": "application/x-www-form-urlencoded", +}; +``` + +The Overpass QL query is single-family and bbox-bounded, e.g.: + +``` +[out:json][timeout:25]; +nwr["tourism"="attraction"](3.05,101.55,3.25,101.80); +out center 40; +``` + +On a non-200 status, a network error, or an unparseable response, print exactly: + +``` +Overpass instance busy — retry later +``` + +and `process.exit(0)`. + +**`search/test/fixtures/overpass-response.json`** — a committed, realistic Overpass JSON response with 6 elements: a mix of `node` elements (with `lat`/`lon`) and `way`/`relation` elements (with `center: {lat, lon}`), each carrying `tags` with `name` and `name:en` plus a category-ish tag (`tourism`, `amenity`, `railway`, `office`, `leisure`). The elements must be real KL places (e.g. "Merdeka 118", "National Monument", "Masjid Jamek", "KL Tower", "Central Market", "Petaling Street") with plausible coordinates inside the KL bbox. + +**`search/test/topup.test.mjs`** — the seven cases `TP1`…`TP7` from the `` block. `TP4` and `TP5` RUN the script as a child process (`node scripts/overpass-topup.mjs ...`) from the `search/` workspace directory and assert the exit code and the stdout shape. `TP4` uses a temp fixture that is NOT an Overpass response (e.g. `{"error":"busy"}`) to exercise the soft-exit path. + +**`search/scripts/json-to-sqlite.mjs`** — the D-05 converter. It reads `search/src/data/kl-gazetteer.json` and emits a dataset with the SAME ids so backend #2's format is reachable from day one (D-05). The header block states that it is a repo tool, never a runtime dependency, and that the ODbL notice MUST travel with the output (ODbL §4.4 share-alike on a derived database). + +**Node's SQLite availability:** Node 24.18.0 ships `node:sqlite` (stable-ish behind no flag in Node 22.5+/24). PROBE IT FIRST in this task with `node -e "const {DatabaseSync}=require('node:sqlite'); console.log('sqlite OK')"`. If it works, emit a real `.sqlite` file with a `places` table (`id TEXT PRIMARY KEY, name TEXT, aliases TEXT, lat REAL, lng REAL, category TEXT, address TEXT`). If it does NOT work, emit a deterministic `.sql` dump file plus a JSON sidecar and say so plainly in the header — do NOT add a dependency. + +**`search/test/converter.test.mjs`** — the three cases `CV1`…`CV3` from the `` block. `CV1` runs the converter into a temp directory and asserts the emitted id set is EXACTLY the gazetteer's id set (same members, same count) — reading the ids back out of whatever format was emitted (SQLite via `node:sqlite`, or the `.sql` dump's INSERT lines). + +**Acceptance:** `npm run search:test` green with `PASS TP1`…`PASS TP7` and `PASS CV1`…`PASS CV3`; the top-up script exits 0 on the fixture AND on a busy-shaped input. + + + cd /home/C-Fu/araharah && npm run search:test 2>&1 | grep -E 'PASS TP|PASS CV|runner' ; cd search && node scripts/overpass-topup.mjs --fixture=test/fixtures/overpass-response.json 2>&1 | head -12 ; echo "exit=$?" ; node -e "const {DatabaseSync}=require('node:sqlite'); console.log('node:sqlite available')" 2>&1 | tail -1 + + + - `npm run search:test` exits 0 and the output contains `PASS TP1` through `PASS TP7` and `PASS CV1` through `PASS CV3` + - `cd search && node scripts/overpass-topup.mjs --fixture=test/fixtures/overpass-response.json` exits 0 and its stdout contains `PROPOSAL`, at least one `name=` line, and the literal text `NEVER writes` + - `cd search && node scripts/overpass-topup.mjs --fixture=` exits 0 and its stdout contains exactly the line `Overpass instance busy — retry later` + - `grep -q 'User-Agent: araharah-gazetteer-topup/1.0' search/scripts/overpass-topup.mjs` exits 0 (HTTP 406 without it — live-verified) + - `grep -q '\-\-source' search/scripts/overpass-topup.mjs` exits 0 (the source flag exists: overpass | nominatim | fixture — the resolved-question posture) + - `grep -c 'await fetch(\|fetch(' search/scripts/overpass-topup.mjs | head -1` returns 1 (EXACTLY ONE request per run) + - `grep -c 'kl-gazetteer.json' search/scripts/overpass-topup.mjs | head -1` returns 0 (the script never names the curated file — it cannot write it) + - `grep -rl 'overpass' bot/src/ web/src/ 2>/dev/null | wc -l` returns 0 (the script is never a runtime dependency — RESEARCH Pitfall 6) + - `grep -rl 'sqlite\|json-to-sqlite' bot/src/ web/src/ 2>/dev/null | wc -l` returns 0 + - `node -e "const g=require('./search/src/data/kl-gazetteer.json'); const {execFileSync}=require('node:child_process'); const fs=require('node:fs'); const os=require('node:os'); const path=require('node:path'); const out=fs.mkdtempSync(path.join(os.tmpdir(),'araharah-conv-')); execFileSync('node',['search/scripts/json-to-sqlite.mjs','--out',out],{cwd:process.cwd()}); const files=fs.readdirSync(out); if(!files.length) throw new Error('converter emitted nothing'); const blob=files.map(f=>fs.readFileSync(path.join(out,f),'utf8')).join('\n'); const ids=g.entries.map(e=>e.id); const missing=ids.filter(id=>!blob.includes(id)); if(missing.length) throw new Error('id set not preserved: '+missing.join(',')); console.log('converter OK -> '+files.join(','));"` exits 0 + + Both maintainer-time repo tools ship and are proven offline: the Overpass top-up makes exactly one identified request, prints a proposal, never writes the curated file, and exits 0 on a busy instance; the JSON→SQLite converter preserves the gazetteer's id set exactly, so backend #2's dataset format is reachable today. + + + + task 3: run the bake-off, close it, and record the decision + docs/SEARCH-QUALITY.md, search/test/fixtures/golden-corpus.json, search/src/data/kl-gazetteer.json, search/src/rules/abbreviations.ts + + - `search/scripts/gazetteer-corpus.mjs` (task 1's output — the instrument that measures the bake-off) + - `search/test/fixtures/golden-corpus.json` (task 1's output — the pinned corpus being measured) + - `search/src/data/kl-gazetteer.json` (plan 04-01's output — the curated dataset under measurement) + - `search/src/rules/abbreviations.ts` (plan 04-01's output — the lever-1 table the escalation edits target) + - `.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md` (D-13 and D-14 — the exact threshold and the exact escalation order) + + +**Run the bake-off.** Execute `npm run search:corpus` and record the real output — top-1, top-3, the per-category breakdown, the negatives-refused count, and every MISSED and LEAKED line. + +**If the threshold is met** (top-3 >= 90% on positives AND zero negative leaks): the escalation lever is **"None"**. Record that in `docs/SEARCH-QUALITY.md` and make NO corpus/table/data edits to force a pass. + +**If the threshold is NOT met:** pull the escalation ladder in the FIXED order (D-14) and re-run after each pull: +1. **Lever 1 — alias/normalizer table edits.** For each MISSED case, decide whether the miss is an ALIAS gap (add the local shorthand to the entry's `aliases` in `search/src/data/kl-gazetteer.json`) or an ABBREVIATION gap (add a row to `search/src/rules/abbreviations.ts` — a table edit, never a code change). Add a per-rule test vector to `search/test/normalizer.test.mjs` for every new abbreviation row (D-09: "Per-rule test vectors; extends by table edit, not code change"). Re-run `npm run search:corpus` and `npm run search:test`. +2. **Lever 2 — SearchSource backend #2 (the Malaysia-only WASM geocoder).** This backend does NOT ship in v1 (D-05) and MUST NOT be implemented here. If lever 1 cannot reach the threshold, RECORD that fact in `docs/SEARCH-QUALITY.md` as the documented outcome and leave the threshold unmet with the lever named — the phase's experiment ends with a documented decision (D-14), not with an open question and not with an unbuilt backend smuggled in. +3. **Lever 3 — Photon-on-VPS.** Same handling as lever 2 — record, do not build (D-03 defers VPS provisioning to Phase 5). + +**NEVER edit the corpus to make it pass.** The corpus is PINNED (D-13). A missed case is fixed by improving the DATA or the RULES, never by changing the expectation. If a corpus case is simply WRONG (its `expectId` points at an entry that is not the right landmark), fix the corpus's `expectId` and say so explicitly in `docs/SEARCH-QUALITY.md` under "Corpus corrections" — that is a corpus bug, and it must be visible. + +**Do NOT weaken the admission rule** to make positives pass. The tiered admission (non-fuzzy tiers always; fuzzy only at similarity >= `FUZZY_ADMIT_SIMILARITY` 0.72 — plan 04-01's contract) is what makes the negatives honest (D-13/D-14). If a positive can only be found by lowering that number, the correct fix is an alias or a table row. + +**Coordinate verification (the resolved-question offline posture).** Spot-verify at least 8 of the curated seed entries' coordinates against Nominatim (reachable per RESEARCH §Environment Availability): for each sampled entry, `GET https://nominatim.openstreetmap.org/search?format=jsonv2&q=, Kuala Lumpur&limit=1` with an identifying User-Agent (the same discipline as the Overpass script — a missing UA is the 406 trap), and confirm the returned lat/lon lands within ~0.01° of the entry's committed coordinates. Record the sample size, the method, and the outcome in `docs/SEARCH-QUALITY.md` under a "Coordinate verification" section (N of M sampled within tolerance — or, if Nominatim is unreachable at run time, record that honestly and name the committed-bbox gate G5 as the fallback invariant). This is a one-shot maintainer-time check recorded in the doc — NOT a test-suite dependency and never a runtime path. + +**Write `docs/SEARCH-QUALITY.md`** — EXACTLY per the `` block's structure: the header (corpus version + case count, gazetteer version + entry count, the measurement date), the metric table (top-1, top-3 vs the 90% threshold, negatives refused vs the zero-leak requirement), the per-category table, the "Escalation lever pulled" section naming exactly one lever (or "None"), the "Corpus corrections" section (empty if none), the "Coordinate verification" section, and the "Known limitations" section (the hand-written corpus measures the documented local shorthand, not every phrasing; out-of-KL queries are refused, not routed — there is no backend #2 or #3 in v1). + +**Acceptance:** `npm run search:corpus` exits 0; `docs/SEARCH-QUALITY.md` records the real measured numbers; the lever section names one lever honestly. + + + cd /home/C-Fu/araharah && npm run search:corpus 2>&1 | tail -25 ; npm run search:test 2>&1 | tail -3 ; test -f docs/SEARCH-QUALITY.md && grep -c 'top-3' docs/SEARCH-QUALITY.md && grep -c 'Escalation lever pulled' docs/SEARCH-QUALITY.md && grep -c 'Known limitations' docs/SEARCH-QUALITY.md + + + - `npm run search:corpus` exits 0 + - `npm run search:test` exits 0 (any lever-1 edit kept every other suite green) + - `test -f docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'top-3' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'negatives refused' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'Escalation lever pulled' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'Coordinate verification' docs/SEARCH-QUALITY.md` exits 0 AND `grep -q 'Known limitations' docs/SEARCH-QUALITY.md` exits 0 + - `grep -q 'Lever 1\|Lever 2\|Lever 3\|None — the threshold was met' docs/SEARCH-QUALITY.md` exits 0 (exactly one lever is named) + - `grep -c 'search:corpus' docs/SEARCH-QUALITY.md | head -1` returns >= 1 (the record names how to reproduce the measurement) + - `git diff --stat search/test/fixtures/golden-corpus.json` shows NO expectation edits that lowered a threshold — the corpus's `threshold.top3_hit_rate` is still `0.9`: `grep -q '"top3_hit_rate": 0.9' search/test/fixtures/golden-corpus.json` exits 0 + - `grep -q 'search/src/rules/abbreviations.ts' docs/SEARCH-QUALITY.md` exits 0 when lever 1 was pulled, OR the doc names lever 2/3 instead (the lever section is specific) + + The bake-off is closed with a measured, recorded, honest outcome: top-3 >= 90% on positives, every negative refused, and exactly one named escalation lever (or "None") recorded in `docs/SEARCH-QUALITY.md` with the reproduction command. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Committed corpus file → the measured quality claim | The corpus is the instrument that decides whether the phase ships. A corpus edited to pass is a falsified instrument. | +| Live Overpass endpoint → the maintainer's terminal | A third-party service with no SLA, intermittent capacity, and an egress filter that 406s requests without a User-Agent. | +| Overpass response → the curated dataset | A candidate list is UNTRUSTED third-party data. It must never be written straight into the curated file. | +| Corpus runner (local tool) → stdout | The runner MAY print query text (it is a local dev tool over committed files); the Worker never may. The runner's output must not be shipped anywhere. | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-04-12 | Tampering | The corpus being edited so the gate always passes (RESEARCH Pitfall 7 — "the corpus becomes a generated artifact") | mitigate | The corpus is pinned and hand-written; the tautology gate (CO3) requires >= 15 non-negative cases whose normalized query is NOT string-equal to any gazetteer alias — a regenerated corpus fails it. `threshold.top3_hit_rate` stays `0.9` (asserted in the acceptance criteria). The plan explicitly forbids editing a case's `expectId` to make it pass, and any legitimate corpus correction must be listed under "Corpus corrections" in `docs/SEARCH-QUALITY.md` so it is visible. | +| T-04-13 | Repudiation | The phase closing with an unmeasured or unfalsified quality claim (D-14: "the experiment ends with a documented decision, not an open question") | mitigate | `npm run search:corpus` is a single command with a mechanical exit code; `docs/SEARCH-QUALITY.md` must carry the measured top-1/top-3/refusal numbers, the per-category table, exactly one named lever, and the reproduction command. The "Escalation lever pulled" section is mandatory content, not an optional note. | +| T-04-14 | Denial of Service | The Overpass script becoming a runtime dependency (SSRF-ish: the Worker fetching a third-party service on every cold start) | mitigate | LAW 1 plus the TP1 grep gate: `bot/src/` and `web/src/` contain zero occurrences of `overpass`. The script lives under `search/scripts/`, is never imported by product code, and the curated JSON is the only thing the Worker reads. | +| T-04-15 | Denial of Service | The Overpass script blocking a build, a test, or a deploy because the endpoint is down or busy | mitigate | LAW 4: 502 / 429 / timeout / non-200 / unparseable → print `Overpass instance busy — retry later` and exit 0. The TP4 case asserts the exit code is 0 on a busy-shaped input. No repo script depends on this script's success (RESEARCH §Verification A4 item 4). | +| T-04-16 | Denial of Service | An Overpass query without a bbox causing a global scan that times out and gets the maintainer's IP rate-limited | mitigate | LAW: the query is always bbox-bounded to the KL box (`(3.05,101.55,3.25,101.80)`), and TP3 asserts the bbox tuple is present in the source. The query is single-family and narrow by default, which is what fits a free worker slot. | +| T-04-17 | Spoofing | The Overpass endpoint serving a response that makes the maintainer merge hostile data into the curated dataset | mitigate | The script OUTPUT is a printed proposal only — LAW 5 plus the TP6 assertion that the script's source never names `kl-gazetteer.json` at all, so it CANNOT write it. A human diffs and hand-merges. The curated dataset's own invariants (unique ids, slug charset, KL-bbox coordinates, non-empty aliases) are asserted by `search/test/gazetteer.test.mjs` (plan 04-01), so hostile coordinates or a duplicate id would fail the suite even if a human merged them by mistake. | +| T-04-18 | Information Disclosure | The corpus runner's query-printing leaking into a shipped log | mitigate | The runner is a local dev tool under `search/scripts/`; it is never imported by `bot/src/` or `web/src/` and no product code calls it. The Worker-side logging law (category-only) is enforced separately in plan 04-03's source-introspection gate. The runner's output goes to a maintainer's terminal, not to a log aggregator. | +| T-04-19 | Tampering | The D-14 threshold being lowered (e.g. `0.9` → `0.8`) to make a failing corpus pass | mitigate | The threshold lives in the corpus file as a single literal `0.9`, and the acceptance criteria assert `grep -q '"top3_hit_rate": 0.9'` still holds after the bake-off. D-14 fixes the number as 90%; changing it is a decision change, not a tuning knob. | +| T-04-20 | Information Disclosure | The JSON→SQLite converter emitting the dataset WITHOUT the ODbL notice, producing an unattributed derivative database (ODbL §4.4 share-alike) | mitigate | The converter's header block states that the ODbL notice MUST travel with the output, and the emitted artifact carries the `_attribution` / `_license` / `_notice` fields in a metadata table (or as a header comment in the `.sql` dump). The converter is a repo tool — the notice travels with the data by construction. | + + + +- `npm run search:corpus` exits 0 with top-3 >= 90% and zero negative leaks, and prints the per-category breakdown +- `npm run search:test` green (the corpus gate is inside the suite — a regression fails `npm run search:test`, not just the standalone runner) +- `npm run search:typecheck` green +- `grep -rl 'overpass' bot/src/ web/src/` returns NOTHING — the top-up tool is provably not a runtime dependency +- `grep -rl 'sqlite' bot/src/ web/src/` returns NOTHING — the backend-#2 converter is provably not a runtime dependency +- `docs/SEARCH-QUALITY.md` exists and carries the measured numbers, the per-category table, exactly one named lever, and the reproduction command +- `npm test -w bot` and `npm test -w web` still green (this plan touches no bot/web source) + + + +- D-13: a pinned golden corpus of 60+ hand-written cases exists at `search/test/fixtures/golden-corpus.json`, with 6 categories and at least 15 non-tautological cases, each positive pinning its expected entry id +- D-14: the bake-off CLOSES on a mechanical threshold — `npm run search:corpus` exits 0 only when top-3 >= 90% on positives AND every negative is refused — and the measured outcome plus the lever pulled are recorded in `docs/SEARCH-QUALITY.md` +- D-06: the Overpass top-up is a repo tool that proposes candidates for human approval, never writes the curated file, sends an identifying User-Agent, makes exactly one request, and exits 0 on a busy instance — proven offline against a committed fixture +- D-05: the JSON→SQLite converter exists, reads the same committed dataset, and preserves the id set exactly — backend #2's dataset format is reachable today +- SRCH-01/SRCH-02 are MEASURED: the golden landmarks and the abbreviation/typo/glued/alias buckets each have a measured top-3 rate, so a later regression is diagnosable per category rather than as one diffuse number + + + +After completion, create `.planning/phases/04-destination-search-kl-gazetteer/04-02-SUMMARY.md` + diff --git a/.planning/phases/04-destination-search-kl-gazetteer/04-03-PLAN.md b/.planning/phases/04-destination-search-kl-gazetteer/04-03-PLAN.md new file mode 100644 index 0000000..918d0e8 --- /dev/null +++ b/.planning/phases/04-destination-search-kl-gazetteer/04-03-PLAN.md @@ -0,0 +1,665 @@ +--- +phase: 04-destination-search-kl-gazetteer +plan: 03 +type: execute +wave: 2 +depends_on: [04-01] +files_modified: + - bot/src/bot/commands/cari.ts + - bot/src/bot/handlers/search-alternate.ts + - bot/src/bot/bot.ts + - bot/src/i18n/strings.ts + - bot/src/config.ts + - bot/scripts/register-commands.mjs + - bot/test/commands.test.mjs + - bot/test/cari.test.mjs + - bot/package.json + - bot/.dev.vars.example + - bot/wrangler.jsonc + - docs/CONFIGURATION.md +autonomous: true +requirements: [SRCH-03, ONBD-02] +must_haves: + decisions_implemented: + - "D-01: search entry is the `/cari` command in chat; `/pergi_ke` stays RESERVED for Phase 5 and must NOT be registered; no free-text-to-bot, no inline mode, no in-webview search UI this phase" + - "D-02: the best result ships immediately as the venue card with its OPEN button; 2-4 alternates ride as inline buttons on the SAME message; one tap to proceed, one tap to switch" + - "D-03: the search backend is the pluggable `SearchSource` interface; the in-Worker gazetteer is backend #1 — no Photon, no VPS work in this phase" + - "D-10: the venue card is Bot API `sendVenue` (latitude, longitude, title, address) carrying inline buttons" + - "D-11 + RESEARCH Pitfall 1 (MECHANISM CORRECTED): the deep link carries a PLAIN gazetteer id with NO crypto. The `web_app` inline button's `WebAppInfo` has EXACTLY ONE FIELD — `url` — so the id travels as a URL query param `?dest=` read from `location.search` in the webview. `initData.start_param` is the SECONDARY path (the documented direct-link / home-screen route). The locked intent — venue card -> one tap -> Mini App preloaded with a plain id, no crypto — is UNCHANGED; only the carrier field is corrected" + - "D-15: search queries are user payload — category-only logging; the raw query never appears in a log line" + truths: + - "Typing '/cari KLCC' in Telegram produces ONE message: a native venue card for Suria KLCC (pin, name, address) carrying an OPEN IN ARAHARAH button plus 2-4 inline alternates" + - "The OPEN button's URL is the configured deployed origin plus '?dest=' — the id is a plain slug, never a signed token, and the origin never comes from user input" + - "Tapping an alternate swaps the venue card to that entry, in the same chat, without a second user round" + - "Typing '/cari Kuching' produces an honest 'not in v1 area' reply with NO venue card, NO inline keyboard, and NO fabricated pin" + - "An over-long query (over 128 characters) is rejected before any search work, with no venue card" + - "No log line anywhere in the bot source can contain the user's raw query text" + - "The alternates' callback_data carries only 'alt:' — never a name, a query, or a coordinate — and stays within Telegram's 64-byte limit" + - "Every callback query is answered within Telegram's 30-second window, including unknown and malformed ones" + - "The '/cari' line appears in the bot's command menu and in /help in BOTH locales" + - "The Mini App origin is a documented, configured value — `docs/CONFIGURATION.md` names MINI_APP_URL and says what it is for" + artifacts: + - path: "bot/src/bot/commands/cari.ts" + provides: "the /cari command handler: query extraction, the SearchSource call, the venue-card send, the refusal copy (D-01, D-02, D-10)" + contains: "registerCari" + - path: "bot/src/bot/handlers/search-alternate.ts" + provides: "the alternates callback_query handler + the catch-all that always answers (D-02)" + contains: "registerSearchAlternate" + - path: "bot/src/i18n/strings.ts" + provides: "the search copy in en + ms: the OPEN label, the honest refusal, the usage line, the over-cap line, the OSM attribution, the /cari helpMenu line (D-05 copy law)" + - path: "bot/test/cari.test.mjs" + provides: "the payload-shape gates: venue fields, keyboard contents, callback_data format/length/no-leakage, answerCallbackQuery on every path, the no-match refusal, the no-crypto structural gate" + - path: "bot/test/commands.test.mjs" + provides: "the EXTENDED command-surface gates (CS4 carve for the inline keyboard, CS5's three-command menu, CS6's alias set, the narrowed future-stub ban)" + - path: "bot/src/config.ts" + provides: "the Env contract gaining MINI_APP_URL — the deployed Mini App origin used to build the deep link" + - path: "bot/scripts/register-commands.mjs" + provides: "the setMyCommands script deriving the /cari scope from helpMenu (the hardcoded command list gains 'cari')" + - path: "docs/CONFIGURATION.md" + provides: "the MINI_APP_URL row in the env-var table, with the 'public origin, not a secret' note" + key_links: + - from: "bot/src/bot/commands/cari.ts" + to: "@araharah/search" + via: "the handler calls the shared SearchSource — the same module the corpus measured" + pattern: "SearchSource" + - from: "bot/src/bot/commands/cari.ts" + to: "bot/src/i18n/strings.ts" + via: "every user-facing string is imported from the strings module; nothing inline" + pattern: "strings\\[locale\\]" + - from: "bot/src/bot/commands/cari.ts" + to: "bot/src/config.ts" + via: "the OPEN button's URL origin comes from env.MINI_APP_URL — never from user input" + pattern: "MINI_APP_URL" + - from: "bot/src/bot/handlers/search-alternate.ts" + to: "@araharah/search" + via: "the callback handler resolves 'alt:' through the source's byId() — never by echoing the id into a reply" + pattern: "byId" + - from: "bot/test/commands.test.mjs" + to: "bot/src/bot/commands/cari.ts" + via: "the CS4 gate's documented carve for the venue-card surface (the location-fallback precedent)" + pattern: "cari.ts" +--- + + +Deliver the chat-side search surface: the `/cari` command (D-01), the single-message venue card with 2–4 inline alternates (D-02, D-10), the honest out-of-area refusal (D-13/D-14), and the OPEN button whose deep link carries a plain gazetteer id with no crypto (D-11, mechanism corrected per RESEARCH Pitfall 1) — plus the bilingual copy, the command-menu registration, the `MINI_APP_URL` configuration, and the four `commands.test.mjs` landmines that must land in the SAME change. + +Purpose: this is the phase's user-facing surface and the half that ONBD-02 depends on. The research's single highest-value finding is that a `web_app` inline button CANNOT populate `start_param` — `WebAppInfo` has exactly one field, `url` (verified against the installed grammY typings this session). Building the OPEN button as a `web_app` button whose URL carries `?dest=` and reading it from `location.search` in the webview is the carrier that actually works; `initData.start_param` remains the secondary path. The locked D-11 intent — a plain gazetteer id, no crypto, one tap from the venue card into a preloaded Mini App — is preserved exactly. + +Output: `bot/src/bot/commands/cari.ts`, `bot/src/bot/handlers/search-alternate.ts`, the extended `bot/src/i18n/strings.ts`, the Env contract's `MINI_APP_URL` + the `docs/CONFIGURATION.md` row, the updated `register-commands.mjs`, the extended `bot/test/commands.test.mjs`, and a new `bot/test/cari.test.mjs`. + + + +@$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md +@$HOME/.config/opencode/get-shit-done/templates/summary.md + + + +@AGENTS.md +@.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md +@.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md +@.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md +@bot/src/bot/bot.ts +@bot/src/bot/commands/start.ts +@bot/src/bot/handlers/location-fallback.ts +@bot/src/i18n/strings.ts +@bot/src/config.ts +@bot/scripts/register-commands.mjs +@bot/test/commands.test.mjs +@bot/test/webhook.test.mjs +@docs/CONFIGURATION.md +@search/src/index.ts +@search/src/types.ts + + +**RESEARCH Pitfall 1 — the deep-link carrier, corrected. The D-10/D-11 intent is unchanged; only the carrier field is.** + +D-11 as written reads `?startapp=dest_`. The installed grammY typings prove the `web_app` inline button CANNOT carry that field: + +``` +node_modules/@grammyjs/types/markup.d.ts:219-222 +export interface WebAppInfo { + /** An HTTPS URL of a Web App to be opened with additional data ... */ + url: string; +} +``` + +`WebAppInfo` has EXACTLY ONE field — `url`. There is NO `start_param`. Telegram's official `messages.requestWebView` documents `start_param` as populated ONLY from an attachment-menu `startattach` deep link; the sibling `messages.requestAppWebView` populates it from a `?startapp=` DIRECT LINK (a `t.me` link) — a different launch surface from an inline keyboard button. So an OPEN button built as `web_app: { url: "/?startapp=dest_klcc" }` with the webview reading `initDataUnsafe.start_param` would open the Mini App with NO destination, and ONBD-02 would fail silently. + +**The corrected carrier:** + +| Piece | Value | +|---|---| +| Button type | `InlineKeyboard.webApp(label, url)` — a `web_app` button (D-10's shape is preserved: it is still an inline button on the venue card) | +| URL | `${MINI_APP_URL}/?dest=${entryId}` — the id in the URL's OWN query string | +| Primary reader (webview) | `new URLSearchParams(window.location.search).get("dest")` | +| Secondary reader (webview) | `initData.start_param` (the documented direct-link path, and the home-screen shortcut's path) | +| Payload | the PLAIN slug id (`suria-klcc`) — validated against `/^[A-Za-z0-9_-]{1,64}$/` | +| Crypto | NONE (D-11). A structural gate asserts no HMAC/signature/token code exists in the deep-link path. | + +The locked intent — "venue card -> one tap -> Mini App preloaded with a plain gazetteer id, no crypto" — is delivered UNCHANGED. Only the mechanism (which Telegram field carries the id) is corrected. The `miniAppUrlFor(origin, id)` builder stays a single function so the carrier is a one-line swap if a device test fails. + +**`MINI_APP_URL` — where the origin comes from.** The button's URL origin MUST be a CONFIGURED value, never derived from user input (RESEARCH §Security: "Open redirect / unvalidated Mini App URL — Spoofing"). The origin cannot come from `WEBHOOK_URL` reliably (that env var holds the webhook target including the `/telegram/webhook` path, and `docs/CONFIGURATION.md` line 34 records that the Worker never reads it), so a dedicated env var is the right call (RESEARCH Assumption A6 lists this as a cheap decision). It is a PUBLIC origin, not a secret — it lives in `bot/wrangler.jsonc`'s `vars`, never in a secret. + +**Fallback when `MINI_APP_URL` is unset:** the OPEN button is still rendered, but the handler must not crash and must not guess an origin. It sends the venue card with the alternates only, and logs the category-only line `[search] rejected: mini-app-origin-missing`. The card still confirms the destination (SRCH-03) and the alternates still work (D-02). + + + +`bot/src/bot/commands/cari.ts` — the handler. Structure copies `bot/src/bot/commands/start.ts` exactly (pure text builder → handler → register), with two additions: a command ARGUMENT (no existing handler takes one) and a keyboard surface. + +**Query extraction.** grammY's standard route: `ctx.match` is the text after the command name. For `/cari KLCC`, `ctx.match === "KLCC"`. + +**The length cap (V5 input validation — RESEARCH §Security V5 (a)).** The query is bounded BEFORE any processing: + +```typescript +/** V5: the query is untrusted input — bounded before ANY processing. */ +export const MAX_QUERY_CHARS = 128; +``` + +Over-cap queries are rejected with the honest "query too long" copy and NO venue card. This closes the hostile-query DoS surface (T-04-21). + +**The empty-query case.** `/cari` with no argument replies with a short usage line (the locale's own copy) and sends NO venue card. + +**The venue card.** `sendVenue` has NO `text` parameter (RESEARCH Pitfall 2 — confirmed against the Bot API method page) — its parameters are `latitude`, `longitude`, `title`, `address` (all four required) plus optional `reply_markup`. So the confirming line IS the card: `title` = the entry's canonical `name`, `address` = the entry's CURATED address string (verbatim, never the normalized form — RESEARCH Pitfall 3). D-02's "one message" shape holds: exactly ONE `sendVenue` call, no preceding `sendMessage`. + +```typescript +await ctx.replyWithVenue( + entry.lat, + entry.lng, + entry.name, + entry.address ?? "", + { reply_markup: keyboard }, +); +``` + +**The keyboard.** One `InlineKeyboard`: +- Row 0: the OPEN button — `new InlineKeyboard().webApp(strings[locale].searchOpenButton, miniAppUrlFor(origin, entry.id))`. +- Row 1 (or row 2 for 4 alternates): 2-4 alternates, each `.text(alternate.name, "alt:" + alternate.id)`. + +The alternates are the NEXT best hits after the best one, capped at 4. When the search produced only 1 hit, the keyboard carries the OPEN button alone (no fake alternates). When the search produced 2 hits, ONE alternate rides along (there is nothing else to show — never a fabricated row). + +**The OPEN button URL builder** — one function, one place: + +```typescript +/** The Mini App deep link for a gazetteer entry (D-11). + * Carrier corrected per RESEARCH Pitfall 1: a `web_app` inline button's + * WebAppInfo carries ONLY a `url` — there is no `start_param` field, and + * Telegram populates start_param only from an attachment-menu deep link. + * The id therefore travels in the URL's own query string and the webview + * reads it from location.search. start_param remains the secondary path + * (the documented direct-link mechanism, used by the home-screen shortcut). + * No crypto: the payload is a public place id (D-11). */ +export function miniAppUrlFor(origin: string, id: string): string { + return `${origin.replace(/\/+$/, "")}/?dest=${encodeURIComponent(id)}`; +} +``` + +**The refusal.** An empty search result produces NO venue card and NO keyboard. The reply is the locale's honest "not in v1 area" copy — clinical, one line, naming the v1 area (RESEARCH Open Question 2's "not in v1 area" copy is OpenCode's discretion; the copy must never promise a pin it does not have). + +**Logging (D-15).** Category-only. The handler's only permitted log lines: + +```typescript +console.log("[search] rejected: no-match"); // no query text +console.log("[search] rejected: query-too-long"); // no query text +console.log("[search] rejected: empty-query"); // no query text +console.log("[search] rejected: mini-app-origin-missing"); +``` + +NEVER the query, the results, the entry id, the chat id, or a timestamp. A source-introspection gate enforces this. + +**The register function.** + +```typescript +export function registerCari(bot: Bot, source: SearchSource, origin: string): void { + bot.command("cari", async (ctx) => { ... }); +} +``` + +**Alias decision.** D-01 names `/cari` (the Malay "search"). Phase 2's alias scheme (D-04) paired `/start`↔`/mula`, `/privacy`↔`/privasi`. `/cari` is already the Malay word; its English twin would be `/search`. **This plan registers BOTH `/cari` and `/search`** — the bilingual alias precedent, and it keeps the CS6 alias-completeness gate honest. (`/pergi_ke` stays RESERVED for Phase 5 and must NOT be registered — CS5's future-stub ban keeps `/pergi_ke` banned and now permits only `/cari` + `/search`.) + +**CS4 carve — the second deliberate exception to the "plain text only" command rule.** `bot/src/bot/handlers/location-fallback.ts` (lines 15-20) documents the FIRST exception (the GEO-02 reply keyboard). The venue card + InlineKeyboard is the SECOND. Two options exist: (a) put the keyboard surface in `handlers/` (outside CS4's scan list, which scans only `bot.ts` + `commands/*`), or (b) amend CS4 with a documented carve. **This plan takes option (b)** — the venue card IS a command's output, and hiding it from the gate would be the more dangerous choice: the gate is what keeps the keyboard surface honest. The carve is a documented exception in the CS4 test's own header, mirroring the location-fallback precedent, and it must distinguish an INLINE keyboard (allowed here) from a REPLY keyboard (still banned — the reply keyboard is the location-fallback handler's own surface). + +**The `reply_markup` string problem.** CS4 greps `/keyboard/i` over the command sources. `InlineKeyboard` and `reply_markup` both trip it. The amended CS4 must therefore ban the REPLY-keyboard shape specifically, not the word "keyboard". + + + +`bot/src/bot/handlers/search-alternate.ts` — the alternates switching surface. + +**Callback data format (RESEARCH Pitfall 10).** `callback_data` is `"alt:"` — ONLY the entry id. Never a name, never a query, never coordinates. It is 1-64 bytes; the slug charset keeps it far below the limit. The entry id is a PUBLIC place id, so the payload carries no user data (D-15 posture). + +**The shared card builder.** The venue card is built in ONE place so the card the user first sees and the card they get after tapping an alternate are structurally identical. `sendVenueCard` lives in `search-alternate.ts` and is imported by `cari.ts`: + +```typescript +export async function sendVenueCard( + ctx: Context, + entry: GazetteerEntry, + alternates: readonly GazetteerEntry[], + origin: string, + locale: Locale, +): Promise +``` + +`cari.ts` calls it with `hits.slice(1)`; the callback handler calls it with the alternates it resolved for the tapped entry. + +**The handler.** + +```typescript +export function registerSearchAlternate(bot: Bot, source: SearchSource, origin: string): void { + bot.callbackQuery(/^alt:(.+)$/, async (ctx) => { + await ctx.answerCallbackQuery(); // must happen within 30 s + const id = ctx.match[1] ?? ""; + if (!/^[a-z0-9-]{1,64}$/.test(id)) return; // hostile id → answer, then drop + const entry = source.byId(id); + if (entry === null) return; // honest null — never a fabricated card + await sendVenueCard(ctx, entry, alternatesFor(entry, source), origin, locale); + }); + + // Always answer everything else so Telegram doesn't show a 60-second spinner. + bot.on("callback_query:data", async (ctx) => { + const data = ctx.callbackQuery?.data ?? ""; + if (/^alt:/.test(data)) return; // the specific matcher owns this path + await ctx.answerCallbackQuery(); + }); +} +``` + +**The swap semantics.** Tapping an alternate sends a NEW `sendVenue` for that entry (with that entry's own alternates). `editMessageText` is FORBIDDEN on a venue message — it would replace the venue payload and destroy the card (RESEARCH §Anti-Patterns; A8 lists `editMessageReplyMarkup` as the alternative, but a fresh card matches D-02's "one tap to switch" and is the safe path). The OLD card stays in the chat as history — that is Telegram-native and honest. + +**`answerCallbackQuery` on EVERY path** — the 30-second rule (RESEARCH §Security: "`answerCallbackQuery` not called — Denial of Service (UX spinner)"). The catch-all `bot.on("callback_query:data", ...)` is grammY's documented recommendation ("Always answer all other callback queries … Otherwise, some clients may display a loading animation for up to a minute"). + +**The alternates for a swapped card.** `alternatesFor(entry, source)` asks the source for the entry's own neighbourhood — `source.search({ text: entry.name, limit: 5 })` — and filters out the entry itself, capping at 4. If that yields nothing, the card carries the OPEN button alone. + + + +`bot/src/i18n/strings.ts` — the `BotStrings` interface GAINS these fields (both locales, identical key-sets — CS1 asserts this): + +- `searchUsage` — the empty-query usage line: "/cari — search for a destination in the KL metro." / ms equivalent. +- `searchNoMatch` — the honest refusal: names the v1 area and promises nothing. en: "Not in the v1 KL area. araharah only knows Kuala Lumpur and the Klang Valley for now." ms: "Bukan dalam kawasan KL v1. araharah hanya tahu Kuala Lumpur dan Lembah Klang buat masa ini." +- `searchQueryTooLong` — the over-cap refusal: "That query is too long. Send just the place name." / ms equivalent. +- `searchOpenButton` — the OPEN label: "OPEN IN ARAHARAH" (identical in both locales — it is a product affordance, not prose). +- `osmAttribution` — "© OpenStreetMap contributors" (IDENTICAL in both locales — the same string as `web/src/i18n/strings.ts` line 73). + +**The alternates' button labels are NOT copy** — each alternate's label IS the entry's own `name` (public place data from the gazetteer), so no string field is needed for them. + +**`helpMenu` gains the /cari line** in BOTH locales, and the line format is LOAD-BEARING — `commands.test.mjs`'s `menuEntries` parser (lines 102-111) requires each line to start with `/` and carry the " — " marker (space-em-space), and `register-commands.mjs`'s `menuFor` (lines 59-69) finds lines via `` const marker = `/${command} — ` ``. The existing en value is: + +``` +"Commands:\n/help — How araharah works and what it keeps\n/privacy — What araharah stores and never stores" +``` + +It becomes: + +``` +"Commands:\n/help — How araharah works and what it keeps\n/privacy — What araharah stores and never stores\n/cari — Search for a destination in the KL metro" +``` + +and the ms value gains the matching `\n/cari — ` line. The `/cari` line's description must be in the locale's own language. + +**`register-commands.mjs` — the hardcoded command list.** Line 61 is `return ["help", "privacy"].map((command) => {` — it must become `return ["help", "privacy", "cari"].map((command) => {`. The script's `menuFor` THROWS when a `/cari` line is missing from `helpMenu` (lines 64-66), so the strings change and this change MUST land together. The script's TWO-fetch-call count stays at 2 (CS5 asserts exactly two `await fetch(` calls — the scopes, not the commands). + + + +`bot/test/commands.test.mjs` — FOUR edits, all in the same change as the `/cari` landing. The suite is currently GREEN and goes RED the moment `cari.ts` + the strings land. + +**CS4 (lines 78-87) — the keyboard carve.** The current gate greps `/keyboard/i` over `commandSources`. `cari.ts` contains `InlineKeyboard` and `reply_markup`. The amended gate must: +- keep the ban on the REPLY-keyboard shape (a `keyboard:` property inside a `reply_markup` object) — that is the location-fallback handler's own surface and it stays out of `commands/`; +- ALLOW the INLINE-keyboard shape (`InlineKeyboard` + `reply_markup` with `inline_keyboard`) in `commands/cari.ts`; +- document the carve in the suite's own header, in the same style as `location-fallback.ts` lines 15-20. + +Implementation: keep `/keyboard/i` as the ban but apply it to every command source EXCEPT `commands/cari.ts`; in `cari.ts`, ban the reply-keyboard shape specifically with a pattern like `/reply_markup:\s*\{\s*keyboard/` and `/ReplyKeyboardMarkup/`. + +**CS4b (lines 89-97) — the link-preview rule.** The current gate asserts that any file calling `ctx.reply(` carries `link_preview_options: { is_disabled: true }`. `cari.ts` calls `ctx.replyWithVenue(` (not `ctx.reply(`) for the card, and `ctx.reply(` for the refusal lines — those MUST carry `link_preview_options: { is_disabled: true }` so a query echoed into a refusal message is not turned into a link preview. (The refusal copy must NOT echo the query text — it names the area, not the query.) + +**CS5 (lines 113-122) — the menu expectation.** `["help", "privacy"]` becomes `["help", "privacy", "cari"]` — the order is the helpMenu line order, which `menuEntries` preserves. The test name must change accordingly. + +**CS5b (lines 136-145) — the future-stub ban.** The assertion `assert.doesNotMatch(source, /cari|pergi_ke/, "no future-command stubs …")` currently asserts the ABSENCE of `cari`. It must be inverted: `/cari` is now ALLOWED (it is a live command), but `/pergi_ke` must STAY BANNED (D-01 reserves it for Phase 5). The assertion becomes `assert.doesNotMatch(source, /pergi_ke/, "…")`. + +**CS6 (lines 150-163) — the alias set.** `["help", "mula", "privasi", "privacy", "start"]` becomes `["cari", "help", "mula", "privasi", "privacy", "search", "start"]` (sorted). The grep `bot\.command\("([^"]+)"` scans `commandSources` — so `commands/cari.ts` MUST be added to that list (lines 30-32), otherwise CS6 never sees the new commands and the expected set would not match. + +**The `commandSources` list (lines 30-32)** gains `"commands/cari.ts"`. + +**CS1 (lines 36-41)** needs no change — the key-set identity assertion automatically covers the new string fields. + + + +`bot/test/cari.test.mjs` — the NEW payload-shape suite. It uses the repo's two established gate styles: + +**Style 1 — fake-transport payload capture.** `bot/src/bot/bot.ts` lines 29-31 already expose the seam: `createBot(env, { baseFetch })` — grammY's `client.fetch` override. `bot/test/webhook.test.mjs` lines 72-100 show the in-memory harness shape (`makeKV()` + undici `Request`/`Response`). The suite creates a fake `fetch` that records the outgoing request method, URL, and JSON body, then drives the bot with a synthetic Telegram update. + +The suite drives the `/cari` handler by importing `registerCari` and registering it against a `Bot` whose `baseFetch` records the `sendVenue` request. Assert on the RECORDED request body, which is exactly what Telegram would receive. + +**Style 2 — source-introspection structural gates** (the `commands.test.mjs` CS3/CS4 style) for the claims that are about structure, not behaviour. + +Cases: +- `CR1 the venue payload is structurally correct` — the recorded `sendVenue` body carries `latitude` and `longitude` as FINITE numbers, `title` = the entry's `name` (non-empty), `address` = the entry's curated `address` (non-empty), and `reply_markup.inline_keyboard` present. +- `CR2 the OPEN button's URL carries the plain id and the configured origin` — row 0 of `inline_keyboard` is a button whose `web_app.url` is exactly `${origin}/?dest=`; the origin part equals the configured `MINI_APP_URL` (never a query-derived value); the `dest` value matches `/^[A-Za-z0-9_-]{1,64}$/`. +- `CR3 NO CRYPTO IN THE DEEP-LINK PATH (D-11's structural gate)` — read `bot/src/bot/commands/cari.ts` and `bot/src/bot/handlers/search-alternate.ts` (comments stripped) and assert they contain NONE of: `createHmac`, `crypto.subtle`, `sign(`, `hmac`, `signature`, `sha256`, `secret`. +- `CR4 the alternates ride on the SAME message` — the recorded request count for a query with >= 3 hits is exactly 1; the `inline_keyboard` has 2-4 alternate buttons whose `callback_data` all match `/^alt:[a-z0-9-]{1,64}$/`. +- `CR5 the callback_data carries no user data` — every `callback_data` in the recorded keyboard matches `^alt:$`; assert NONE of them contains a space, a query, a latitude, or a longitude, and that every `callback_data` byte length is <= 64. +- `CR6 a no-match query sends NO venue and NO keyboard` — for "Kuching": the recorded requests contain NO `sendVenue` call and NO `reply_markup`; the reply is the locale's `searchNoMatch` text. +- `CR7 an over-long query is rejected before any search` — a 200-character query produces no `sendVenue` and the `searchQueryTooLong` copy. +- `CR8 an empty query produces no venue card` — `/cari` with no argument sends the usage copy and no `sendVenue`. +- `CR9 every callback path answers the callback query` — the alternates handler's source contains `answerCallbackQuery` in the specific matcher AND in the catch-all; assert the catch-all exists by grepping for `bot.on("callback_query:data"` in the source. +- `CR10 the query text never reaches a log line (D-15)` — read `bot/src/bot/commands/cari.ts` (comments stripped) and assert no `console.log` line references a query-shaped variable (`query`, `ctx.match`, `text`, `q`); assert every `console.log` in the file matches `/\[search\] rejected: [a-z-]+/`. +- `CR11 the copy exists in both locales` — `strings.en.searchNoMatch` and `strings.ms.searchNoMatch` are non-empty and DIFFER (not an untranslated copy), and the same for `searchUsage` and `searchQueryTooLong`; `searchOpenButton` and `osmAttribution` are IDENTICAL across locales by design — assert that. +- `CR12 the helpMenu carries the /cari line in both locales` — the `menuEntries` parse yields a `cari` entry with a non-empty description in BOTH locales, and the description in `ms` is not byte-identical to the `en` one. +- `CR13 the registration script derives the /cari scope from helpMenu` — `register-commands.mjs` contains `"cari"` in its command list and still issues exactly TWO `await fetch(` calls. +- `CR14 the alternates swap produces a new venue card for the tapped entry` — the alternates handler, driven with a `callback_data` of `alt:`, sends a `sendVenue` whose `title` is that entry's name. +- `CR15 a hostile callback id is dropped, not echoed` — a `callback_data` of `alt:../../etc/passwd` produces NO venue card (the id fails the slug validation). + + + +`bot/src/config.ts` — the `Env` interface GAINS: + +```typescript + // The deployed Mini App ORIGIN (public, not a secret) — used to build the + // OPEN button's deep link. NOT derived from user input: the deep-link + // origin is a configured value (RESEARCH §Security V4 — an unvalidated + // origin would be an open redirect). + MINI_APP_URL: string; +``` + +`bot/wrangler.jsonc` — add a `vars` block (NOT a secret — it is a public origin): + +```jsonc + "vars": { "MINI_APP_URL": "https://araharah-bot..workers.dev" }, +``` + +The executor MUST replace the placeholder with the actual deployed origin (the same origin @BotFather has registered as the Mini App URL — the Bot API 10.2 origin restriction makes these the same value). If the origin is not yet known, use a clearly-marked placeholder (`"https://example.invalid"`) and note it in the SUMMARY — the OPEN button will then be built from that placeholder, which is exactly what the missing-origin fallback path is for. + +`bot/.dev.vars.example` — add a commented `MINI_APP_URL=` line (the real `.dev.vars` is gitignored). + +`docs/CONFIGURATION.md` — the env-var table (line 25, "## Environment variables") GAINS a row: + +``` +| `MINI_APP_URL` | **Required for the deep link** | none | Public origin of the deployed Mini App (e.g. `https://araharah-bot..workers.dev`) — NOT a secret, so it lives in `bot/wrangler.jsonc`'s `vars`, not in a secret. The `/cari` venue card's OPEN button is built from it: `/?dest=`. It must equal the Mini App URL registered in @BotFather (Bot API 10.2 enforces the origin). The Worker never derives it from user input — an unvalidated origin would be an open redirect. | +``` + +Also update the `## Worker configuration file (bot/wrangler.jsonc)` section (line 129) to mention the `vars` block, and the `## Required vs optional settings` section (line 103) to name `MINI_APP_URL` among the non-secret configuration values. + +`bot/package.json` — add `"@araharah/search": "*"` to `dependencies` (the `web/package.json` line 15 `"@araharah/theme": "*"` precedent) so the bot can import the shared module. + + + + + + + task 1: the search copy + the /cari command + the venue card + the OPEN deep link + bot/src/i18n/strings.ts, bot/src/bot/commands/cari.ts, bot/src/config.ts, bot/wrangler.jsonc, bot/.dev.vars.example, bot/package.json, docs/CONFIGURATION.md + + - `bot/src/i18n/strings.ts` (the whole file — the interface, the en/ms records, the `pickLocale` rule; every new string goes HERE and nowhere else) + - `bot/src/bot/commands/start.ts` (the whole file — the pure-builder -> handler -> register structure the new handler copies) + - `bot/src/bot/handlers/location-fallback.ts` (lines 15-20 — the header-comment style for documenting a deliberate exception to the text-only rule) + - `bot/src/config.ts` (the Env contract) + - `bot/wrangler.jsonc` (the current config — keep name, main, compatibility_date, workers_dev, assets, run_worker_first, kv_namespaces) + - `bot/package.json` (the dependencies block — the `"@araharah/theme": "*"` precedent lives in web/package.json) + - `docs/CONFIGURATION.md` (lines 25-35 the env-var table, line 103 "Required vs optional settings", line 129 the wrangler section) + - `search/src/index.ts` + `search/src/types.ts` (plan 04-01's output — the `SearchSource` the handler consumes) + + +**`bot/src/i18n/strings.ts`.** Add the six new `BotStrings` fields to the interface (each with a one-line doc comment citing its decision ID, matching the file's existing style) and the values to BOTH the `en` and `ms` records, EXACTLY per the `` block. The `helpMenu` value in each locale gains the `\n/cari — ` line in the locale's own language. The `/cari` line's description: +- en: `Search for a destination in the KL metro` +- ms: `Cari destinasi di kawasan KL` + +`osmAttribution` is `© OpenStreetMap contributors` in BOTH locales (identical — it is a legal attribution, not prose). `searchOpenButton` is `OPEN IN ARAHARAH` in BOTH locales (identical — a product affordance). + +**`bot/src/config.ts`.** Add the `MINI_APP_URL` member per the `` block, with its explanatory comment. + +**`bot/wrangler.jsonc`.** Add the `vars` block per the `` block. Keep every existing key byte-identical. The `MINI_APP_URL` value must be the same origin registered in @BotFather as the Mini App URL. + +**`bot/.dev.vars.example`.** Add a commented line documenting `MINI_APP_URL` (public origin — it is a `vars` entry in `wrangler.jsonc`, not a secret, but the example file is where a maintainer looks). + +**`bot/package.json`.** Add `"@araharah/search": "*"` to `dependencies`. + +**`docs/CONFIGURATION.md`.** Add the `MINI_APP_URL` row to the env-var table per the `` block, and update the `## Required vs optional settings` and `## Worker configuration file` sections to name it. + +**`bot/src/bot/commands/cari.ts`** — NEW. Implement EXACTLY per the `` block. The file's header comment must carry the mechanism correction in full (the `` block's content, condensed to the essential paragraph) and the D-02 exception carve (mirroring `location-fallback.ts` lines 15-20's wording). Type imports come from the shared module via the EXPLICIT SUBPATH-WITH-EXTENSION form — `import type { SearchSource, GazetteerEntry } from "@araharah/search/src/index.ts";` — the package has NO `exports` map (the `@araharah/theme` precedent), so a bare `@araharah/search` specifier resolves to nothing. Exports: + +- `export const MAX_QUERY_CHARS = 128;` +- `export function miniAppUrlFor(origin: string, id: string): string` +- `export function searchUsageText(locale: Locale): string` +- `export function searchNoMatchText(locale: Locale): string` +- `export function registerCari(bot: Bot, source: SearchSource, origin: string): void` + +The handler's flow: + +1. `const raw = (ctx.match ?? "").trim();` +2. `if (raw.length === 0)` → reply `searchUsageText(locale)` with link previews disabled; log `[search] rejected: empty-query`; RETURN (no venue card). +3. `if (raw.length > MAX_QUERY_CHARS)` → reply `searchQueryTooLong` with link previews disabled; log `[search] rejected: query-too-long`; RETURN (no venue card, no search performed). +4. `const hits = await source.search({ text: raw, limit: 5 });` +5. `if (hits.length === 0)` → reply `searchNoMatchText(locale)` with link previews disabled; log `[search] rejected: no-match`; RETURN (no venue card, no keyboard — D-13's honest refusal). +6. Otherwise: build the alternates from `hits.slice(1)` (each `hits[i].entry`), then `await sendVenueCard(ctx, hits[0].entry, alternates, origin, locale)` — exactly ONE message (D-02). NO log line on the success path (there is nothing category-worthy to say, and every log line is a privacy surface). + +The OPEN button is `new InlineKeyboard().webApp(strings[locale].searchOpenButton, miniAppUrlFor(origin, entry.id))`. + +When `origin` is empty or blank: skip the OPEN button, send the card with the alternates only, and log `[search] rejected: mini-app-origin-missing`. The card still confirms the destination (SRCH-03 holds). + +**Register BOTH `/cari` and `/search`** (the bilingual alias precedent — Phase 2's D-04 paired `/start`↔`/mula`, `/privacy`↔`/privasi`). `/pergi_ke` is NOT registered (D-01 reserves it for Phase 5). + +**Acceptance:** `npm run bot:typecheck` exits 0; the file compiles against grammY 1.46's `replyWithVenue` and `InlineKeyboard.webApp` signatures. + + + cd /home/C-Fu/araharah && npm install --package-lock-only 2>&1 | tail -3 ; npm run bot:typecheck 2>&1 | tail -10 ; grep -c 'replyWithVenue' bot/src/bot/commands/cari.ts ; grep -c 'MAX_QUERY_CHARS = 128' bot/src/bot/commands/cari.ts ; grep -c 'MINI_APP_URL' bot/src/config.ts bot/wrangler.jsonc docs/CONFIGURATION.md + + + - `npm run bot:typecheck` exits 0 + - `grep -q 'searchNoMatch' bot/src/i18n/strings.ts` exits 0 AND `grep -q 'searchOpenButton' bot/src/i18n/strings.ts` exits 0 AND `grep -q 'osmAttribution' bot/src/i18n/strings.ts` exits 0 AND `grep -q 'searchUsage' bot/src/i18n/strings.ts` exits 0 AND `grep -q 'searchQueryTooLong' bot/src/i18n/strings.ts` exits 0 (all five new string fields exist) + - `grep -c '/cari — ' bot/src/i18n/strings.ts | head -1` returns 2 (the /cari helpMenu line in BOTH locales — the " — " marker is load-bearing for `menuEntries` AND `menuFor`) + - `node --input-type=module -e "const {strings}=await import('./bot/src/i18n/strings.ts'); const en=Object.keys(strings.en).sort(), ms=Object.keys(strings.ms).sort(); if(JSON.stringify(en)!==JSON.stringify(ms)) throw new Error('locale key-sets differ'); if(!strings.ms.searchNoMatch||strings.ms.searchNoMatch===strings.en.searchNoMatch) throw new Error('ms refusal copy untranslated'); if(strings.en.osmAttribution!=='© OpenStreetMap contributors') throw new Error('attribution'); console.log('strings OK')"` exits 0 (CS1's locale parity holds with the new fields) + - `grep -q 'export function registerCari' bot/src/bot/commands/cari.ts` exits 0 AND `grep -q 'bot.command("cari"' bot/src/bot/commands/cari.ts` exits 0 AND `grep -q 'bot.command("search"' bot/src/bot/commands/cari.ts` exits 0 (D-01's /cari plus its bilingual twin) + - `grep -c 'pergi_ke' bot/src/bot/commands/cari.ts | head -1` returns 0 (`/pergi_ke` stays reserved for Phase 5 — D-01) + - `grep -q 'replyWithVenue' bot/src/bot/commands/cari.ts` exits 0 AND `grep -c 'ctx.reply(' bot/src/bot/commands/cari.ts | head -1` is <= 2 (exactly ONE venue send for the card; the other `ctx.reply` calls are the refusal/usage lines) + - `grep -q 'MAX_QUERY_CHARS = 128' bot/src/bot/commands/cari.ts` exits 0 (V5's length cap) + - `grep -q 'MINI_APP_URL' bot/src/config.ts` exits 0 AND `grep -q 'MINI_APP_URL' bot/wrangler.jsonc` exits 0 AND `grep -q 'MINI_APP_URL' bot/.dev.vars.example` exits 0 AND `grep -q 'MINI_APP_URL' docs/CONFIGURATION.md` exits 0 (the public origin is configured, documented in code, in the example file, and in the configuration docs) + - `grep -q '"vars"' bot/wrangler.jsonc` exits 0 (the origin is a var, NOT a secret) + - `grep -q '"@araharah/search": "\*"' bot/package.json` exits 0 (the bot imports the shared module) + - `grep -v '^\s*\*\|^\s*//\|^/\*\|^\s*$' bot/src/bot/commands/cari.ts | grep -c 'console\.log' | head -1` returns >= 1 AND `grep -v '^\s*\*\|^\s*//\|^/\*\|^\s*$' bot/src/bot/commands/cari.ts | grep -o 'console\.log("[^"]*"' | grep -vc '\[search\] rejected: [a-z-]*"'` returns 0 (every log line is a category-only literal — D-15) + + The `/cari` command exists with its bilingual twin, the venue card is a single `replyWithVenue` carrying the OPEN deep link (`/?dest=`, no crypto), the query is capped at 128 chars before any search, every log line is a category-only literal, and the Mini App origin is configured and documented in `docs/CONFIGURATION.md`. + + + + task 2: the alternates callback handler + the bot factory wiring + bot/src/bot/handlers/search-alternate.ts, bot/src/bot/bot.ts + + - `bot/src/bot/handlers/location-fallback.ts` (the whole file — the handlers/ directory's conventions: the header contract, the register function, the honest-null handling) + - `bot/src/bot/bot.ts` (the whole file — the factory and its registration order; the `opts.baseFetch` seam the test suite drives) + - `bot/src/bot/commands/cari.ts` (task 1's output — the keyboard builder the alternates handler must mirror) + - `search/src/types.ts` (plan 04-01's output — `SearchSource.byId`, the synchronous id lookup the handler uses) + + +**`bot/src/bot/handlers/search-alternate.ts`** — NEW. Implement EXACTLY per the `` block. + +Exports: +- `export function registerSearchAlternate(bot: Bot, source: SearchSource, origin: string): void` +- `export async function sendVenueCard(ctx: Context, entry: GazetteerEntry, alternates: readonly GazetteerEntry[], origin: string, locale: Locale): Promise` — the ONE venue-card builder, shared by the `/cari` handler and the callback handler so the first card and the swapped card are structurally identical. +- `export function alternatesFor(entry: GazetteerEntry, source: SearchSource): GazetteerEntry[]` — the entry's own neighbourhood, filtered to exclude the entry itself, capped at 4. + +The card builder: + +```typescript +export async function sendVenueCard( + ctx: Context, + entry: GazetteerEntry, + alternates: readonly GazetteerEntry[], + origin: string, + locale: Locale, +): Promise { + const keyboard = new InlineKeyboard(); + if (origin.trim() !== "") { + keyboard.webApp(strings[locale].searchOpenButton, miniAppUrlFor(origin, entry.id)).row(); + } + for (const alternate of alternates.slice(0, 4)) { + keyboard.text(alternate.name, `alt:${alternate.id}`); + } + await ctx.replyWithVenue(entry.lat, entry.lng, entry.name, entry.address ?? "", { + reply_markup: keyboard, + }); +} +``` + +**The `callback_query` matcher:** `bot.callbackQuery(/^alt:(.+)$/, async (ctx) => { ... })`. + +- `await ctx.answerCallbackQuery();` FIRST (the 30-second rule). +- The id is validated against `/^[a-z0-9-]{1,64}$/` BEFORE any lookup — a hostile id is dropped silently (after the answer). +- `const entry = source.byId(id);` → `null` means RETURN (honest null — never a fabricated card). +- The new card's alternates: `alternatesFor(entry, source)`. +- The locale comes from `pickLocale(ctx.from?.language_code)` — the same rule the other handlers use. + +**The catch-all:** `bot.on("callback_query:data", async (ctx) => { const data = ctx.callbackQuery?.data ?? ""; if (/^alt:/.test(data)) return; await ctx.answerCallbackQuery(); })` — the specific matcher owns the `alt:` path exclusively, so the catch-all answers everything else (grammY's documented recommendation) and never double-answers. + +**`bot/src/bot/bot.ts`** — wire both new surfaces. Create ONE source instance and share it between the two registrations (a single memoized index): + +```typescript +export function createBot(env: Env, opts: { baseFetch?: typeof fetch } = {}): Bot { + const bot = opts.baseFetch + ? new Bot(env.BOT_TOKEN, { client: { fetch: opts.baseFetch } }) + : new Bot(env.BOT_TOKEN); + registerStart(bot); // /start + /mula (D-04) + registerHelp(bot); // /help + registerPrivacy(bot); // /privacy + /privasi (D-04) + registerLocationFallback(bot, env.chat_sessions); // GEO-02 location reply fallback + const source = createGazetteerSource(); // D-03: the in-Worker gazetteer (backend #1) + const origin = env.MINI_APP_URL ?? ""; // the public Mini App origin (D-11) + registerCari(bot, source, origin); // /cari + /search (D-01/D-02/D-10) + registerSearchAlternate(bot, source, origin); // the alternates switcher (D-02) + return bot; +} +``` + +Update the file's header comment to name the two new surfaces and D-01/D-02/D-10/D-11, and to record that the venue card + inline keyboard is the SECOND documented exception to the D-02 plain-text rule (the first being the GEO-02 reply keyboard in `handlers/location-fallback.ts`). + +**Acceptance:** `npm run bot:typecheck` exits 0. `npm test -w bot` will still FAIL on CS4/CS5/CS6 until task 3 updates them — that is expected and is task 3's job. + + + cd /home/C-Fu/araharah && npm run bot:typecheck 2>&1 | tail -10 ; grep -c 'answerCallbackQuery' bot/src/bot/handlers/search-alternate.ts ; grep -c 'registerCari\|registerSearchAlternate' bot/src/bot/bot.ts ; npm test -w bot 2>&1 | tail -6 + + + - `npm run bot:typecheck` exits 0 + - `grep -q 'export function registerSearchAlternate' bot/src/bot/handlers/search-alternate.ts` exits 0 + - `grep -c 'answerCallbackQuery' bot/src/bot/handlers/search-alternate.ts | head -1` returns >= 2 (the specific matcher AND the catch-all both answer — the 30-second rule) + - `grep -q 'bot.on("callback_query:data"' bot/src/bot/handlers/search-alternate.ts` exits 0 (the catch-all that stops Telegram's 60-second spinner) + - `grep -q 'callbackQuery(/\^alt:' bot/src/bot/handlers/search-alternate.ts` exits 0 (the callback matcher is scoped to the `alt:` prefix) + - `grep -q '\^\[a-z0-9-\]{1,64}\$' bot/src/bot/handlers/search-alternate.ts` exits 0 (the id is regex-validated BEFORE the lookup — RESEARCH §Security V5 (b)) + - `grep -q 'source.byId' bot/src/bot/handlers/search-alternate.ts` exits 0 (the id → entry lookup goes through the source, never an echo) + - `grep -q 'export async function sendVenueCard' bot/src/bot/handlers/search-alternate.ts` exits 0 AND `grep -q 'sendVenueCard' bot/src/bot/commands/cari.ts` exits 0 (ONE card builder, two callers — the first card and the swapped card are structurally identical) + - `grep -q 'registerCari' bot/src/bot/bot.ts` exits 0 AND `grep -q 'registerSearchAlternate' bot/src/bot/bot.ts` exits 0 (both surfaces are wired into the factory) + - `grep -c 'createGazetteerSource()' bot/src/bot/bot.ts | head -1` returns 1 (ONE source instance shared by both registrations — a single memoized index) + - `grep -q 'replyWithVenue' bot/src/bot/handlers/search-alternate.ts` exits 0 AND `grep -c 'editMessageText' bot/src/bot/handlers/search-alternate.ts | head -1` returns 0 (the swap sends a NEW card; `editMessageText` would destroy the venue payload) + + The alternates surface is live: a scoped `alt:` callback matcher that answers within 30 s, validates the id before lookup, swaps the card with a new venue message through the ONE shared card builder, and a catch-all that answers every other callback so Telegram never shows a spinner. + + + + task 3: the four commands.test.mjs landmines + the new cari.test.mjs gate suite + bot/test/commands.test.mjs, bot/test/cari.test.mjs, bot/scripts/register-commands.mjs + + - `bot/test/commands.test.mjs` (the WHOLE file — CS1 through CS6, the `commandSources` list, the `menuEntries` parser, the exact assertions that must change) + - `bot/src/bot/handlers/location-fallback.ts` (lines 15-20 — the precedent for documenting a deliberate exception to the text-only rule IN THE SOURCE that breaks it) + - `bot/test/webhook.test.mjs` (lines 72-100 — the in-memory KV + Request/Response harness and the `baseFetch` seam style the new suite uses) + - `bot/test/privacy.test.mjs` (the source-introspection gate style: read the shipped source, strip comments, assert on code text only) + - `bot/src/bot/commands/cari.ts` + `bot/src/bot/handlers/search-alternate.ts` (tasks 1-2's output — the code the new gates pin) + - `bot/scripts/register-commands.mjs` (the hardcoded command list at line 61) + + +**`bot/scripts/register-commands.mjs`.** Line 61's hardcoded list `["help", "privacy"]` becomes `["help", "privacy", "cari"]`. The `menuFor` function is otherwise unchanged (it derives each description from the `helpMenu` line, so the copy stays single-sourced). The TWO-fetch-call count is unchanged. + +**`bot/test/commands.test.mjs`** — the FOUR edits per the `` block: +1. `commandSources` (lines 29-32) gains `"commands/cari.ts"`. +2. **CS4** — the keyboard carve: apply the `/keyboard/i` ban to every command source EXCEPT `commands/cari.ts`; in `cari.ts` ban the REPLY-keyboard shape specifically (`/reply_markup:\s*\{\s*keyboard/` and `/ReplyKeyboardMarkup/`). Update the CS4 test's name and its header documentation to record the carve in the `location-fallback.ts` style ("this is the ONE deliberate exception to the plain-text rule" — and here it is the SECOND documented exception). +3. **CS5** — the expected menu list becomes `["help", "privacy", "cari"]`; the test's name changes accordingly. The `register-commands` sub-case's future-stub ban becomes `assert.doesNotMatch(source, /pergi_ke/, "…")` — `/pergi_ke` stays reserved for Phase 5 (D-01) while `/cari` is now a live command. +4. **CS6** — the expected alias set becomes `["cari", "help", "mula", "privasi", "privacy", "search", "start"]` (sorted). + +Also update the suite's HEADER comment block to describe CS4's new carve and CS5/CS6's new expectations — the header is the suite's contract and it must not lie about what it enforces. + +**`bot/test/cari.test.mjs`** — NEW. Implement all fifteen cases `CR1`…`CR15` from the `` block. Copy `bot/test/webhook.test.mjs`'s harness conventions (the in-memory `makeKV`, the undici Request/Response shapes, the `createBot(env, { baseFetch })` seam). + +The fake-transport capture: build a `fetch` that records `{ method, url, body }` for every outgoing Telegram API call and returns a synthetic `{ ok: true, result: { message_id: 1 } }` response. Drive the bot by registering `registerCari` / `registerSearchAlternate` against a `Bot` constructed with the fake `baseFetch`, then invoking the registered handler with a minimal fake `Context` (`match`, `from`, `replyWithVenue`, `reply`, `answerCallbackQuery`, `callbackQuery`). + +The suite must NOT hit the live Telegram API — the fake `fetch` intercepts every call. + +**The comment-strip helper.** Several cases read a source file and must ignore prose. Copy `web/test/no-storage.test.mjs`'s `stripComments` helper (lines 38-42) into this suite — it is the repo's established comment-stripping implementation: + +```javascript +function stripComments(src) { + return src + .replace(/\/\*[\s\S]*?\*\//g, " ") + .replace(/(^|[^:\\])\/\/[^\n]*/g, "$1"); +} +``` + +**Acceptance:** `npm test -w bot` exits 0 with every suite green (including the four updated CS gates and the fifteen new CR gates); `npm run bot:typecheck` exits 0. + + + cd /home/C-Fu/araharah && npm test -w bot 2>&1 | tail -30 ; npm run bot:typecheck 2>&1 | tail -5 ; npm test -w bot 2>&1 | grep -c 'PASS CR' ; npm test -w bot 2>&1 | grep -c 'PASS CS' + + + - `npm test -w bot` exits 0 and the runner output contains `[runner] N passed, 0 failed` + - `npm test -w bot` output contains `PASS CR1` through `PASS CR15` + - `npm test -w bot` output contains `PASS CS1` through `PASS CS6` + - `grep -q '"commands/cari.ts"' bot/test/commands.test.mjs` exits 0 (the command source is in CS4's/CS6's scan list) + - `grep -q '\["help", "privacy", "cari"\]' bot/test/commands.test.mjs` exits 0 (CS5's three-command menu expectation) + - `grep -q 'pergi_ke' bot/test/commands.test.mjs` exits 0 AND `grep -c 'cari|pergi_ke' bot/test/commands.test.mjs | head -1` returns 0 (the future-stub ban was NARROWED to `/pergi_ke` only — `/cari` is now live) + - `grep -q '"cari"' bot/scripts/register-commands.mjs` exits 0 AND `grep -c 'await fetch(' bot/scripts/register-commands.mjs | head -1` returns 2 (the hardcoded list gained cari; the two-fetch-call count is unchanged) + - `grep -q 'createHmac\|crypto.subtle' bot/test/cari.test.mjs` exits 0 AND `grep -q 'CR3' bot/test/cari.test.mjs` exits 0 (the D-11 NO-CRYPTO structural gate exists and is asserted) + - `grep -q 'stripComments' bot/test/cari.test.mjs` exits 0 (the comment-stripping helper the source-introspection cases use) + - `npm run bot:typecheck` exits 0 + - `npm test -w web` exits 0 (this plan touched no web source — the web suites must be unaffected) + + All four `commands.test.mjs` landmines are defused in the same change that adds `/cari`, and the new `cari.test.mjs` pins the venue payload, the keyboard shape, the no-crypto deep-link law, the honest refusal, the query cap, the callback-data discipline, the always-answer rule, and the category-only logging law. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Telegram chat → `/cari` handler | The raw command argument is untrusted, arbitrary-length user text; it is the ONLY user-controlled input in this plan. | +| Search results → the venue card | Search hits carry entry names, addresses, and coordinates into a Telegram message. They come from a committed, reviewed dataset — but a bad entry would be broadcast to a user's chat. | +| `callback_data` → the alternates handler | A Telegram callback payload is client-visible and can be forged; the id inside it is untrusted until validated. | +| Bot process → Telegram API | Every outgoing request carries the bot token; the deep-link URL is a configured origin that must never be influenced by user input. | +| Bot process → logs | The query text is user payload; the log surface must carry categories only. | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-04-21 | Denial of Service | A hostile `/cari` argument — megabytes of text, or pathological repetition — forcing unbounded normalization/scoring work inside the Worker (whose Free-tier budget is 10 ms CPU/request) | mitigate | `MAX_QUERY_CHARS = 128` bounds the raw argument BEFORE any search work; the over-cap path returns the `searchQueryTooLong` copy and logs a category only. Inside the shared module the normalized query is compared against a pre-built alias index, so per-query work is bounded by the index size, not by the query length (T-04-03/T-04-05 in plan 04-01). | +| T-04-22 | Information Disclosure | The user's raw query reaching a log line (D-15 / V7) | mitigate | Every log statement in `cari.ts` is a category-only literal (`[search] rejected: no-match`, `: query-too-long`, `: empty-query`, `: mini-app-origin-missing`). `CR10` reads the shipped source, strips comments, and asserts every `console.log` matches `/\[search\] rejected: [a-z-]+/` — a log line carrying a query-shaped variable fails the suite. The success path logs NOTHING at all. | +| T-04-23 | Tampering | `callback_data` injection — a forged or hostile payload naming an entry, carrying a name/query/coordinate, or attempting to smuggle a path | mitigate | `callback_data` is ONLY `alt:` (never a name, a query, or coordinates — Pitfall 10). The handler validates the id against `/^[a-z0-9-]{1,64}$/` BEFORE the lookup, resolves it through `source.byId()` (an exact id lookup, never a substring or echo), and drops an unknown id silently. `CR5` asserts the format, the byte length (<= 64), and the absence of user data; `CR15` asserts a traversal-shaped id produces no venue card. | +| T-04-24 | Spoofing | The deep-link origin being influenced by user input — an open redirect carrying a user to a hostile Mini App | mitigate | The OPEN button's URL is built from `env.MINI_APP_URL` — a CONFIGURED public origin declared in `wrangler.jsonc`'s `vars` — never from the query, the entry, or anything else derived from user input. `miniAppUrlFor(origin, id)` takes the origin as an argument, so the origin cannot be computed from the id. `CR2` asserts the URL's origin equals the configured origin exactly. The id is additionally constrained to the slug charset, so it cannot carry a second origin or a path. | +| T-04-25 | Tampering | The deep-link payload being treated as an identity or a credential — a signed/HMAC'd token introduced to "protect" the destination id | mitigate | D-11 says no crypto, and the gate enforces it: `CR3` reads `cari.ts` and `search-alternate.ts` (comments stripped) and asserts NO `createHmac`, `crypto.subtle`, `sign(`, `hmac`, `signature`, `sha256`, or `secret`. The payload is a PUBLIC place id re-resolved against the local gazetteer; its only threat is a user pranking themselves, and the webview treats an unknown id as an honest "destination unavailable" state (plan 04-04), never a fabricated coordinate. | +| T-04-26 | Repudiation | An out-of-area query answered with the nearest KL pin — a fabricated location presented as a real one (D-13/D-14; an honesty failure as much as a correctness one) | mitigate | The empty result produces NO `sendVenue` and NO keyboard — the handler replies with the locale's honest `searchNoMatch` copy naming the v1 area. `CR6` asserts the recorded Telegram request set contains no `sendVenue` call and no `reply_markup` for a "Kuching" query. The shared module's refusal floor (plan 04-01) is the upstream half of the same guarantee. | +| T-04-27 | Denial of Service | A callback query left unanswered — Telegram's client showing a spinner for up to 60 seconds (RESEARCH §Security: "`answerCallbackQuery` not called — Denial of Service (UX spinner)") | mitigate | `answerCallbackQuery()` is the FIRST statement of the `alt:` matcher, and a catch-all `bot.on("callback_query:data", …)` answers every other callback. `CR9` asserts both call sites exist in the shipped source. The catch-all returns early for `alt:` data so the specific matcher owns that path exclusively and never double-answers. | +| T-04-28 | Tampering | `editMessageText` destroying the venue card's payload when an alternate is tapped | mitigate | The swap sends a NEW `sendVenue` — `editMessageText` is never called on a venue message (RESEARCH §Anti-Patterns: it replaces the venue payload and destroys the card). `CR14` asserts the tapped alternate produces a `sendVenue` whose `title` is that entry's name, and the acceptance criteria assert zero `editMessageText` occurrences in the handler. | +| T-04-29 | Information Disclosure | The venue card leaking the user's query or identity into a chat-visible payload | mitigate | The venue card carries only public place data: the entry's name, curated address, coordinates, and the OPEN button's URL (a public origin plus a public slug id). No user id, no chat id, no query text, no session token. The `alt:` callback data is likewise a public place id. Nothing in the card is derived from the user. | +| T-04-30 | Spoofing | A stale or unconfigured `MINI_APP_URL` sending the user to a wrong origin | mitigate | The origin is a single configured value shared by `wrangler.jsonc`'s `vars`, the Worker env, and `docs/CONFIGURATION.md`; there is no fallback that guesses. When it is blank, the OPEN button is OMITTED (rather than pointing somewhere wrong), the card still confirms the destination, and the handler logs the category-only `[search] rejected: mini-app-origin-missing`. The origin must equal the @BotFather Mini App URL — the Bot API 10.2 origin restriction makes a mismatch a hard failure at launch, caught in the phase's device gate. | +| T-04-31 | Elevation of Privilege | The inline `web_app` button being used to reach a privileged Mini App surface | mitigate | The `web_app` button's `WebAppInfo.url` points at the app's own origin with only a `dest` query param; the Mini App's session gate (Phase 2's HMAC validation + `POST /api/session`) is unchanged and still required for the app to function. The destination id grants no capability of its own — it names a public place. | + + + +- `npm test -w bot` green — including the four updated CS gates and the fifteen new CR gates +- `npm run bot:typecheck` green +- `npm test -w web` green (unaffected by this plan) +- `grep -rl 'createHmac\|crypto\.subtle' bot/src/bot/commands/cari.ts bot/src/bot/handlers/search-alternate.ts` returns NOTHING — the deep-link path carries no crypto (D-11) +- `grep -v '^\s*\*\|^\s*//\|^/\*\|^\s*$' bot/src/bot/commands/cari.ts | grep -o 'console\.log("[^"]*"' | grep -v '\[search\] rejected: [a-z-]*"' | wc -l` returns 0 — every log line is a category-only literal (D-15) +- `grep -c 'pergi_ke' bot/src/bot/commands/cari.ts bot/scripts/register-commands.mjs | head -1` returns 0 for both files — the Phase-5 command stays reserved (D-01) +- `grep -q 'MINI_APP_URL' docs/CONFIGURATION.md` exits 0 — the new configuration value is documented where a maintainer looks + + + +- SRCH-03: typing `/cari KLCC` sends ONE native venue card (latitude, longitude, title, curated address) confirming the destination before navigation opens — `sendVenue`'s lack of a `text` field is honoured by letting the card BE the confirmation +- D-02: the best result ships immediately and 2-4 alternates ride as inline buttons on the SAME message; tapping one swaps the card in one tap, with no second user round +- D-10: the card is `sendVenue` with inline buttons; `reply_markup` carries an `inline_keyboard` +- D-11 (mechanism corrected per RESEARCH Pitfall 1): the OPEN button is a `web_app` button whose URL carries the plain id as `?dest=` — the `WebAppInfo` type has exactly one field, `url`, so `start_param` is NOT a valid carrier for an inline button; `initData.start_param` remains the secondary path for the home-screen/direct-link route; the payload is a plain public id with NO crypto, asserted by CR3 +- D-01: search enters through `/cari` (plus its `/search` twin); `/pergi_ke` stays reserved for Phase 5; no free-text-to-bot, no inline mode, no in-webview search UI +- D-03: the handler consumes the pluggable `SearchSource`; no Photon, no VPS work appears in this phase +- D-15: no log line can carry the query; the card carries only public place data +- The four `commands.test.mjs` landmines (CS4's keyboard grep, CS5's menu set + future-stub ban, CS6's alias set, and `register-commands.mjs`'s hardcoded list) are defused in the SAME change that adds `/cari` +- `MINI_APP_URL` is a configured, documented value — the env table in `docs/CONFIGURATION.md` names it and states that it is a public origin, not a secret + + + +After completion, create `.planning/phases/04-destination-search-kl-gazetteer/04-03-SUMMARY.md` + diff --git a/.planning/phases/04-destination-search-kl-gazetteer/04-04-PLAN.md b/.planning/phases/04-destination-search-kl-gazetteer/04-04-PLAN.md new file mode 100644 index 0000000..5c69a88 --- /dev/null +++ b/.planning/phases/04-destination-search-kl-gazetteer/04-04-PLAN.md @@ -0,0 +1,629 @@ +--- +phase: 04-destination-search-kl-gazetteer +plan: 04 +type: execute +wave: 3 +depends_on: [04-01, 04-03] +files_modified: + - web/src/search/resolve-destination.ts + - web/src/search/destination-card.ts + - web/src/telegram/webapp.ts + - web/src/telegram/homescreen.ts + - web/src/mock/telegram-mock.ts + - web/src/i18n/strings.ts + - web/src/main.ts + - web/index.html + - web/package.json + - web/test/destination-preload.test.mjs + - web/test/homescreen.test.mjs + - web/test/no-storage.test.mjs + - web/test/build-artifact.test.mjs + - web/scripts/capture.mjs +autonomous: false +requirements: [ONBD-02, ONBD-03] +must_haves: + decisions_implemented: + - "D-11 + RESEARCH Pitfall 1 (MECHANISM CORRECTED): the webview reads the destination id from `location.search`'s `dest` param FIRST, and from `initData.start_param` SECOND. Both readers apply the SAME validation (`/^[A-Za-z0-9_-]{1,64}$/`). The id is a plain public gazetteer id — NO crypto, NO signature, NO token" + - "D-12 (ONBD-03 scope): minimal PWA hooks only — `addToHomeScreen` / `checkHomeScreenStatus` behind a capability guard, plus a startapp-less boot landing in the search-ready map. NO manifest, NO service worker, NO `beforeinstallprompt`" + - "D-08/D-11: the webview resolves the id against its OWN build-time copy of the shared gazetteer module — the id mapping stays in sync by construction" + - "GEO-03/D-07: the resolved destination lives in MEMORY ONLY — no storage surface, no persistence call" + - "D-12: a startapp-less boot is UNCHANGED — the Phase 3 staged boot still lands in the search-ready map" + truths: + - "Opening the Mini App with '?dest=suria-klcc' resolves the destination to a valid lat/lng from the webview's own gazetteer copy, with no network call" + - "Opening the Mini App with '?dest=suria-klcc' as a start_param (the direct-link path) resolves the SAME destination" + - "An unknown or malformed id produces an honest 'destination unavailable' state — never a blank map and never a fabricated coordinate" + - "The webview and the bot resolve the SAME id set — a build gate proves the shipped web bundle carries the whole gazetteer" + - "The addToHomeScreen affordance is a no-op on a client older than Bot API 8.0 or one without the method" + - "A checkHomeScreenStatus of 'unknown' (always the case on iOS) is treated as 'supported but indeterminate' — never as an error and never as 'not added'" + - "A startapp-less boot lands in the same search-ready map as before — the Phase 3 boot is unchanged" + - "The destination never reaches any storage surface — the runtime probe still finds empty storage after boot" + - "A Playwright capture shows the webview booted WITH a preloaded destination, and the production bundle carries the gazetteer" + artifacts: + - path: "web/src/search/resolve-destination.ts" + provides: "the memory-only id → destination resolution + the two-source payload reader (D-11, mechanism corrected)" + contains: "readDestinationId" + - path: "web/src/search/destination-card.ts" + provides: "the destination surface built from the resolved entry (the webview's visible confirmation)" + - path: "web/src/telegram/homescreen.ts" + provides: "the ONBD-03 addToHomeScreen / checkHomeScreenStatus wiring behind the capability guard (D-12)" + contains: "wireHomeScreen" + - path: "web/src/telegram/webapp.ts" + provides: "the narrow TelegramWebAppLike contract gaining startParam, addToHomeScreen?, checkHomeScreenStatus?" + - path: "web/src/mock/telegram-mock.ts" + provides: "the harness mock gaining the startParam option, the homeScreenStatus option, and the home-screen stubs" + - path: "web/test/destination-preload.test.mjs" + provides: "the ONBD-02 gates: both readers, the unknown-id honest state, the id validation, the memory-only source gate" + - path: "web/test/homescreen.test.mjs" + provides: "the ONBD-03 gates: the capability guard, the 'unknown' status handling, the 'added' branch" + - path: "web/scripts/capture.mjs" + provides: "the destination-preload capture leg + the ARAHARAH-DEST signal race" + key_links: + - from: "web/src/search/resolve-destination.ts" + to: "@araharah/search" + via: "the id resolves against the webview's own build-time copy of the shared gazetteer — the Worker never re-derives it" + pattern: "getEntryById" + - from: "web/src/search/resolve-destination.ts" + to: "web/src/main.ts" + via: "boot reads the destination once and seeds the in-memory destination holder" + pattern: "resolveDestination" + - from: "web/src/telegram/homescreen.ts" + to: "web/src/telegram/webapp.ts" + via: "the capability guard calls the OPTIONAL members on the narrow contract" + pattern: "addToHomeScreen" + - from: "web/src/main.ts" + to: "web/src/telegram/webapp.ts" + via: "boot wires the home screen affordance only when the client supports it" + pattern: "wireHomeScreen" + - from: "web/scripts/capture.mjs" + to: "web/index.html" + via: "the '?dest=' harness profile drives the capture leg through the real reader path" + pattern: "dest=" + - from: "web/test/build-artifact.test.mjs" + to: "search/src/data/kl-gazetteer.json" + via: "the shipped web bundle must carry the gazetteer's id set (the bot/web drift gate)" + pattern: "suria-klcc" +--- + + +Complete the webview half of the deep-link hand-off: read the destination id from the corrected carrier (`location.search`'s `dest` first, `initData.start_param` second), resolve it against the webview's own build-time copy of the shared gazetteer, render an honest destination surface (never a blank map, never a fabricated coordinate), and land ONBD-03's minimal home-screen hooks — then produce the capture evidence and run the blocking device gate for the deep-link carrier. + +Purpose: ONBD-02 is the requirement whose success depends on undocumented Telegram client behaviour, and the research's highest-value finding is that a `web_app` inline button CANNOT populate `start_param` — so the reader must accept BOTH carriers with the same validation, and the phase must not claim the requirement is met until a real device has demonstrated it. ONBD-03 is deliberately minimal (D-12): the Telegram-native `addToHomeScreen()` / `checkHomeScreenStatus()` behind a capability guard, with the iOS `"unknown"` status treated as "possibly added", never as a failure — and no manifest, no service worker, no `beforeinstallprompt` (the browser API does not exist in the Telegram webview). + +Output: `web/src/search/resolve-destination.ts`, `web/src/search/destination-card.ts`, `web/src/telegram/homescreen.ts`, the extended `web/src/telegram/webapp.ts` + `web/src/mock/telegram-mock.ts` + `web/src/i18n/strings.ts`, the extended `web/src/main.ts` + `web/index.html`, three test suites, the extended capture harness, and the blocking device gate. + + + +@$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md +@$HOME/.config/opencode/get-shit-done/templates/summary.md + + + +@AGENTS.md +@.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md +@.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md +@.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md +@web/src/main.ts +@web/src/telegram/webapp.ts +@web/src/telegram/backbutton.ts +@web/src/session/store.ts +@web/src/mock/telegram-mock.ts +@web/src/i18n/strings.ts +@web/src/ui/cards.ts +@web/src/ui/strip.ts +@web/index.html +@web/scripts/capture.mjs +@web/test/no-storage.test.mjs +@web/test/build-artifact.test.mjs +@search/src/index.ts +@search/src/types.ts + + +The bot's OPEN button (plan 04-03) is a `web_app` inline button whose URL carries `?dest=`. `WebAppInfo` has exactly ONE field — `url` — so `start_param` CANNOT be the carrier for an inline button. The webview therefore reads TWO sources, in this order, with the SAME validation: + +1. **PRIMARY — the URL's own query string:** `new URLSearchParams(window.location.search).get("dest")`. This is the inline-button path. +2. **SECONDARY — `initData.start_param`:** the officially documented direct-link path (`https://t.me/?startapp=`) and the home-screen shortcut's path. Telegram populates `start_param` from a `startattach` attachment-menu link or a `?startapp=` direct link — NOT from an inline `web_app` button. + +Both are validated against `/^[A-Za-z0-9_-]{1,64}$/`. The value may carry a `dest_` prefix (the D-11 wording's literal `dest_` form) — the reader strips a leading `dest_` before validation, so BOTH `?dest=suria-klcc` and `?startapp=dest_suria-klcc` resolve to the same entry. + +`initData` is the SIGNED string (`Telegram.WebApp.initData`), NOT `initDataUnsafe` — the signed string is what the server validates, and the `start_param` field is present in it verbatim. Parsing it is a two-line split on `&` + `decodeURIComponent`; the repo's `bot/src/security/initdata.ts` `parseFields` shows the pattern (and its warning: `+` stays literal, never `URLSearchParams`). + +NO CRYPTO, NO SIGNATURE, NO TOKEN (D-11): the id is a public place id. An unknown id yields an honest "destination unavailable" state — never a fabricated coordinate, never a blank map. + + + +`web/src/search/resolve-destination.ts` — the memory-only resolver. Copy `web/src/session/store.ts`'s header law verbatim-in-spirit (lines 3-10: "WHAT is stored … WHAT is never stored … no persistence call anywhere in this file"). + +**Exports:** + +```typescript +export type DestinationSource = "url" | "start-param" | "none"; + +export interface ResolvedDestination { + entry: GazetteerEntry; + source: DestinationSource; +} + +/** Read the destination id from BOTH carriers (mechanism corrected per RESEARCH Pitfall 1). + * Primary: the URL's own query string (`?dest=`) — the inline-button path. + * Secondary: initData's `start_param` — the documented direct-link / home-screen path. + * Both are validated against the same charset; a `dest_` prefix is stripped. */ +export function readDestinationId( + search: string, + initData: string | null | undefined, +): { id: string | null; source: DestinationSource }; + +/** Resolve an id against the webview's OWN build-time copy of the shared gazetteer. + * Returns null for an unknown id — the honest null (never a fabricated coordinate). */ +export function resolveDestination(id: string | null): GazetteerEntry | null; + +/** The memory-only holder: the resolved destination lives in module scope, nowhere else. */ +export function setDestination(entry: GazetteerEntry | null): void; +export function getDestination(): GazetteerEntry | null; +``` + +**The id validation.** `/^[A-Za-z0-9_-]{1,64}$/` AFTER stripping a leading `dest_`. An id failing the regex is treated as absent (never partially trusted). + +**The unknown-id path.** `resolveDestination` returns `null` — the caller renders the honest "destination unavailable" surface and the boot continues into the normal staged boot. A blank map is FORBIDDEN (the Phase 3 D-09 anti-blank invariant is binding here too). + +**The memory-only law.** The module contains NO storage call. `web/test/no-storage.test.mjs` already walks EVERY `web/src` file, so the new module is covered automatically — but the plan adds an explicit case too. + +**`web/src/search/destination-card.ts`** — the visible confirmation. Copy `web/src/ui/cards.ts`'s construction style (the `.card` / `.card-header` / `.card-ref` / `.btn-primary` composed classes, never restyled locally) and `web/src/ui/strip.ts`'s "every string comes from the strings module" law. It renders the destination's NAME and its curated ADDRESS (the address verbatim, never the normalized form — RESEARCH Pitfall 3), and it uses the existing theme classes so no off-token styling appears. + +The card is NOT a new screen — it is the Phase 3 card stack's existing surface with a different card. The roadmap's Phase 4 `UI hint: no` means no UI-SPEC round: the destination card composes ONLY classes that already exist in the Phase 3 shell (`web/src/shell.css` + the theme's component sheet). + +**The strings.** `web/src/i18n/strings.ts` GAINS: +- `destinationHeadline` — "Destination" / "Destinasi" +- `destinationUnavailableHeadline` — "Destination unavailable" / "Destinasi tidak tersedia" +- `destinationUnavailableBody` — "That place is not in this build. Search again from the bot." / ms equivalent +- `addToHomeScreen` — "ADD TO HOME SCREEN" / "TAMBAH KE SKRIN UTAMA" +- `homeScreenAlreadyAdded` — "Already on your home screen" / "Sudah di skrin utama anda" + +The copy-gate suite (`web/test/copy-gate.test.mjs`) asserts the en/ms key-sets match — the new fields are covered automatically by CS1-style key-set identity. + +**`web/src/main.ts` — the boot wiring.** The destination read slots adjacent to the locale derivation (stage 12, lines 92-95), because both are "derive from the environment, memory only" steps: + +```typescript + // 13 — the destination (ONBD-02). TWO carriers, ONE validation (mechanism + // corrected per RESEARCH Pitfall 1): the URL's own query string first + // (the inline-button path — a web_app button's WebAppInfo carries only + // a `url`), then initData's start_param (the documented direct-link / + // home-screen path). An unknown id yields the honest unavailable state + // and the boot continues into the normal staged boot — never a blank map. + const destPayload = readDestinationId(window.location.search, wa?.initData ?? null); + const destination = resolveDestination(destPayload.id); + setDestination(destination); +``` + +When `destination !== null`, the destination card is mounted into the existing card stack. When the id was present but unknown, the unavailable card is mounted. When there was no id at all, NOTHING changes — the Phase 3 boot is untouched (D-12's "startapp-less boot landing in the search-ready map"). The boot emits ONE capture signal, UNCONDITIONALLY (outside any map guard, the same shape as the existing `ARAHARAH-LIVE` signal — a WebGL2-absent device still resolves the race): + +```typescript + console.log("ARAHARAH-DEST state=" + (destination ? "resolved" : "absent")); +``` + +The signal is a STATE TOKEN, never the id (D-15 hygiene: a console token carries no place data). Because a token alone cannot prove WHICH destination resolved, the capture leg pairs it with a DOM check on the rendered card's text (task 3's waitMode block asserts `Suria KLCC`). + +**`web/src/telegram/webapp.ts` — the contract grows.** The `TelegramWebAppLike` interface gains: + +```typescript + /** The documented direct-link start parameter — NOT populated by an inline + * web_app button (its WebAppInfo carries only a `url`). */ + startParam?: string; + /** Bot API 8.0+ only. Optional — absent on older clients. */ + addToHomeScreen?(): void; + /** Bot API 8.0+ only. On iOS this ALWAYS reports "unknown". */ + checkHomeScreenStatus?(cb: (status: HomeScreenStatus) => void): void; +``` + +plus: + +```typescript +export type HomeScreenStatus = "unsupported" | "unknown" | "added" | "missed"; +``` + +Do NOT add `@types/telegram-web-app` (Phase 3 dropped it deliberately — the hand-rolled contract IS the contract; the official type would force each fake to stub ~50 members). The interface's header comment already states this law and must be extended to name the three new members. + +**`web/src/mock/telegram-mock.ts` — the mock grows in the SAME change.** The interface's header says "every test fake, the D-11 harness mock, and the production clients satisfy exactly these members" — so the mock MUST gain the three members in this change: +- `MockOptions.startParam?: string | null` — appended to the mock initData as `&start_param=` in `buildMockInitData` (the mock's initData is HMAC-unvalidated by design — the runner page stubs `/api/session`). +- `MockOptions.homeScreenStatus?: HomeScreenStatus` — the status the mock's `checkHomeScreenStatus` callback receives. +- The mock's `addToHomeScreen` and `checkHomeScreenStatus` stubs, plus a `__homeScreenAdded` counter the tests can read (the mock already exposes `__events` and `__backButtonClicks` for exactly this purpose). + +**`web/src/telegram/homescreen.ts` — the ONBD-03 wiring.** Copy `web/src/telegram/backbutton.ts` (the whole 20-line file is the template: guard on the optional member, a total-function handler, a header that documents the phase-scoped contract): + +```typescript +export function wireHomeScreen(wa: TelegramWebAppLike, locale: Locale): void +``` + +Behaviour: +- The capability guard is BOTH halves: the method may be absent on older clients, AND the platform documents the API as Bot API 8.0+. `if (typeof wa.addToHomeScreen !== "function" || !wa.isVersionAtLeast("8.0")) return;` — a no-op on an unsupported client. +- `checkHomeScreenStatus` — when it reports `"added"`, the affordance is HIDDEN (the icon is already there; do not nag). When it reports `"unknown"` (always on iOS), the affordance is SHOWN with the neutral prompt — "unknown" means "supported but indeterminate", NEVER an error and NEVER "not added". When it reports `"missed"`, the affordance is shown with the prompt. When it reports `"unsupported"`, the affordance is not shown. +- `addToHomeScreen()` is called when the user taps the affordance. +- `wa.onEvent?.("homeScreenAdded", ...)` — Telegram fires this after the icon is added; the affordance hides then. +- The affordance lives in the EXISTING Phase 3 shell — the status strip's right cluster or the card stack. **No new UI furniture is invented** (the roadmap's Phase 4 `UI hint: no`): the affordance is composed from classes that already exist (`web/src/shell.css` + the theme component sheet). If a genuinely new surface were needed, that would require a UI-SPEC round — it does not. +- **NO `beforeinstallprompt`, NO manifest, NO service worker.** `beforeinstallprompt` does not exist inside the Telegram webview and a second icon the Telegram client does not know about is worse than none. `web/test/no-storage.test.mjs`'s GEO3-2 case already bans `serviceWorker.register` / `navigator.serviceWorker` / `workbox` outright. + +**`web/index.html` — the `?dest=` harness profile.** The profile block (lines 20-35) reads `?mock=` / `?scheme=` / `?locale=` / `?pin=` / `?sessiondelay=` / `?fixdelay=`. It gains `?dest=`: + +```javascript + const dest = q.get("dest") || null; + ... + window.__ARAHARAH_MOCK__ = { profile, scheme, locale, dest }; +``` + +The value must reach the app's reader. Because `page.goto` preserves the query string, `window.location.search` ALREADY carries `?dest=` in the webview — the reader path needs no stub. But the mock's `startParam` option needs the value threaded through `installMockWebApp({ ... dest })` for the SECONDARY carrier's harness leg. + +The whole block stays inside `if (__ARAHARAH_HARNESS__)` — Vite's `define` dead-code-eliminates it from production, and `BA5`/`BA6` scan the emitted HTML for `installMockWebApp` / `__ARAHARAH_MOCK__` / `/api/session` / `/api/location-fallback`. The `?dest=` value must NOT add a new endpoint or a new inline fetch stub to the production build. + +**`web/package.json`** — add `"@araharah/search": "*"` to `dependencies` (the `web/package.json` line 15 `"@araharah/theme": "*"` precedent). + +**`web/scripts/capture.mjs` — the ONBD-02 capture leg.** Copy the chat-pin leg (lines 267-269) and the `expected` array (lines 361-369): + +```javascript + await captureLeg({ name: 'shell-dest-preload.png', query: 'mock=android&dest=suria-klcc', colorScheme: 'light', grantGeo: true, waitMode: 'dest' }); +``` + +Three bookkeeping edits: the header's output list (lines 22-29), the leg itself, and the `expected` array. The wait mechanism is the console-signal race — add `ARAHARAH-DEST` to the `SIGNALS` list (line 165) and the `seen` map (line 189). `web/src/main.ts` emits the STATE-TOKEN signal shown earlier in this contract (next to the boot-wiring paragraph) — `ARAHARAH-DEST state=resolved|absent`, never the id — UNCONDITIONALLY, outside any map guard, so a WebGL2-absent device still resolves the race. + + + +`web/test/build-artifact.test.mjs` — the drift gate. RESEARCH §Runtime State Inventory: "the `web/test/build-artifact.test.mjs` gate may need a new assertion (e.g. the gazetteer JSON *must* be in `web/dist` for the webview copy to work)". + +**BA7 (new)** — the shipped production bundle carries the gazetteer: read every `web/dist/assets/*.js`, concatenate, and assert that EVERY gazetteer id is present (the ids come from the shared dataset — the suite imports `search/src/data/kl-gazetteer.json`). This is the WEB half of the bot/web id-set parity gate: if the web bundle ships without the dataset, the webview's `resolveDestination` can never resolve an id. + +**BA8 (new)** — the production bundle carries NO deep-link crypto: assert the concatenated bundle contains NONE of `createHmac`, `crypto.subtle`, `HMAC(`, `sha256` — the deep-link path is a plain id (D-11). (The Telegram SDK script is loaded from `https://telegram.org/js/telegram-web-app.js` in the HTML — the OFFICIAL script may contain crypto internals. The gate therefore scans the JS ASSETS only, NOT the HTML, and the official script is external (not bundled).) + +**The bot/web id-set parity gate** belongs in the BOT suite (the bot test workspace can read both files); plan 04-03's `cari.test.mjs` is the natural home, but the bot suite's `files_modified` in plan 04-03 does not list it — so this plan adds the parity assertion to the WEB suite's BA7 by comparing the web bundle's ids against the committed dataset (the same committed JSON both sides read, so parity holds by construction). The REMAINING risk — a stale web bundle — is what BA7 catches. + +**The build-artifact suite's existing behaviour is preserved:** BA1-BA6 must keep passing, and the new cases scan `web/dist` ONLY (never `web/dist-harness`). + + + +The blocking human checkpoint at the end of this plan (the phase's exit gate for ONBD-02 and ONBD-03). **A real device is REQUIRED** — RESEARCH §Environment Availability lists it as unavailable in this environment with no fallback, exactly like Phase 3's device matrix. The phase MUST NOT be closed claiming device verification that did not happen. + +**What the device must demonstrate:** + +1. **ONBD-02, Android:** send `/cari KLCC` in the bot's chat → the venue card appears with the OPEN button → tap OPEN → the Mini App opens WITH the destination preloaded (the destination card is visible, showing Suria KLCC). +2. **ONBD-02, iOS:** the same flow on an iPhone. +3. **ONBD-02, Telegram Desktop:** the same flow (Desktop has no geolocation, so the destination card is what is being observed — the location half is Phase 3's separate matrix). +4. **ONBD-03:** add araharah to the phone's home screen (via the app's affordance or Telegram's own menu) → launch from the home-screen icon → the app opens into the SEARCH-READY state (no destination, no crash, no blank map). On iOS, `checkHomeScreenStatus` will report `"unknown"` — that is expected platform behaviour and MUST NOT be reported as a failure. +5. **The unknown-id honesty:** open the Mini App with a deliberately wrong id (e.g. `?dest=not-a-real-place`) → the app shows the honest "destination unavailable" surface, NOT a blank map and NOT a fabricated pin. + +**The record.** Create `.planning/phases/04-destination-search-kl-gazetteer/04-DEVICE-GATE.md` with a row per leg and an explicit `PASS` / `FAIL` / `UNVERIFIED` verdict each. If no device is available, the honest outcome is `UNVERIFIED` — record the blocker in `.planning/STATE.md` under "Blockers/Concerns" and recommend acquiring a device or moving ONBD-02's device verification to Phase 7 with an explicit requirement-level note. NEVER close the phase claiming device verification that did not happen. + + + + + + + task 1: the destination resolver (two carriers, one validation, memory only) + web/src/search/resolve-destination.ts, web/src/search/destination-card.ts, web/src/telegram/webapp.ts, web/src/mock/telegram-mock.ts, web/src/i18n/strings.ts, web/package.json, web/test/destination-preload.test.mjs + + - `web/src/session/store.ts` (the whole file — the memory-only module law the resolver copies) + - `web/src/telegram/webapp.ts` (the whole file — the narrow contract that grows) + - `web/src/mock/telegram-mock.ts` (the whole file — `MockOptions`, `buildMockInitData`, the `__events`/`__backButtonClicks` exposure pattern, and the header law that the mock satisfies the same members as production) + - `web/src/ui/cards.ts` (the card construction style the destination card copies — composed theme classes, strings from the strings module) + - `web/src/i18n/strings.ts` (the interface + en/ms records + `localeFromInitData`) + - `web/test/no-storage.test.mjs` (the comment-strip helper and the banned-token list the resolver must satisfy) + - `web/test/session.test.mjs` (the pure-module + source-gate suite style the new suite copies) + - `search/src/types.ts` (plan 04-01's output — the `GazetteerEntry` shape the resolver returns) + + +**`web/package.json`** — add `"@araharah/search": "*"` to `dependencies` (the `"@araharah/theme": "*"` precedent at line 15). + +**`web/src/telegram/webapp.ts`** — extend the `TelegramWebAppLike` interface with the three members per the `` block, plus the `HomeScreenStatus` type export. Update the header comment to name them and restate the "no `@types/telegram-web-app`" law. + +**`web/src/mock/telegram-mock.ts`** — extend `MockOptions` with `startParam?: string | null` and `homeScreenStatus?: HomeScreenStatus`; append `&start_param=` to `buildMockInitData` when `startParam` is set; add the `addToHomeScreen` / `checkHomeScreenStatus` stubs and a `__homeScreenAdded` counter; add `startParam` to the fake object. Update the header comment to note the new options. + +**`web/src/search/resolve-destination.ts`** — NEW, exactly per the `` block. Its header comment states the mechanism correction, the two carriers, the shared validation, the memory-only law, and the honest-null law. + +The reader: + +```typescript +export function readDestinationId( + search: string, + initData: string | null | undefined, +): { id: string | null; source: DestinationSource } { + const fromUrl = new URLSearchParams(search).get("dest"); + const cleaned = fromUrl === null ? null : cleanId(fromUrl); + if (cleaned !== null) return { id: cleaned, source: "url" }; + const fromStart = startParamFromInitData(initData); + const cleanedStart = fromStart === null ? null : cleanId(fromStart); + if (cleanedStart !== null) return { id: cleanedStart, source: "start-param" }; + return { id: null, source: "none" }; +} +``` + +`cleanId` strips a leading `dest_` and validates against `/^[A-Za-z0-9_-]{1,64}$/`, returning `null` on failure. `startParamFromInitData` parses the SIGNED initData string the way `bot/src/security/initdata.ts`'s `parseFields` does — split on `&`, take the `start_param` field, `decodeURIComponent` it, and treat a decode failure as absent. **Never** use `URLSearchParams` on the initData string (`+` stays literal there) and **never** read `initDataUnsafe`. + +`resolveDestination(id)` calls `getEntryById` from `@araharah/search/src/index.ts` (the EXPLICIT subpath-with-extension form — the package has no exports map, so a bare `@araharah/search` specifier resolves to nothing; the `@araharah/theme` subpath precedent) and returns the entry or `null`. + +**`web/src/search/destination-card.ts`** — NEW. Build the destination surface from the EXISTING Phase 3 theme classes (`.card`, `.card-header`, `.card-ref`, `.text-headline-md`, `.text-body-sm` — the same classes `web/src/ui/cards.ts` uses) and the strings module. Two builders: `buildDestinationCard(locale, entry)` and `buildDestinationUnavailableCard(locale)`. No new CSS; no off-token styling. + +**`web/src/i18n/strings.ts`** — add the five new fields per the `` block to the `ShellStrings` interface and to BOTH the `en` and `ms` records. The attribution string stays as-is. + +**`web/test/destination-preload.test.mjs`** — NEW. Cases: +- `DP1 the URL carrier resolves` — `readDestinationId("?dest=suria-klcc", "")` → `{ id: "suria-klcc", source: "url" }`. +- `DP2 the start_param carrier resolves` — `readDestinationId("", "user=%7B%7D&auth_date=1&start_param=suria-klcc&hash=h")` → `{ id: "suria-klcc", source: "start-param" }`. +- `DP3 the URL carrier wins over start_param` — with both present, the URL wins (the inline-button path is primary). +- `DP4 the dest_ prefix is stripped` — `?dest=dest_suria-klcc` and `start_param=dest_suria-klcc` both → `suria-klcc` (the D-11 wording's literal form still resolves). +- `DP5 no id yields the honest none` — both absent → `{ id: null, source: "none" }`. +- `DP6 a malformed id is rejected` — `?dest=../../etc/passwd`, `?dest=