From aa34e7b50d8b04d0cf96223a7e18a035bd819139 Mon Sep 17 00:00:00 2001 From: araharah-executor Date: Fri, 23 Oct 2026 23:40:41 +0800 Subject: [PATCH] =?UTF-8?q?docs(04):=20research=20verification=20pass=20?= =?UTF-8?q?=E2=80=94=20track=20.planning/,=20Overpass=20UA=20correction,?= =?UTF-8?q?=20deep-link=20MTProto=20confirmation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 2 +- .../04-RESEARCH.md | 1882 ++++++++++------- 2 files changed, 1070 insertions(+), 814 deletions(-) diff --git a/.gitignore b/.gitignore index ffeb9d7..2715dd4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ # Planning docs stay local-only (user preference) -.planning/ +#.planning/ # Windows download metadata artifacts *:Zone.Identifier diff --git a/.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md b/.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md index 79fb195..3743e22 100644 --- a/.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md +++ b/.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md @@ -1,8 +1,8 @@ # Phase 4: Destination Search + KL Gazetteer - Research **Researched:** 2026-10-23 -**Domain:** In-Worker fuzzy place search over a curated KL gazetteer, Malay query normalization, Telegram venue-card + Mini App deep-link hand-off, OSM-derived gazetteer tooling and licensing -**Confidence:** HIGH for the Telegram Bot API surface (official docs + installed grammY 1.46.0 typings read this session), HIGH for Cloudflare Workers limits (official docs fetched), HIGH for OSM/ODbL licensing analysis (official OSMF wiki fetched), MEDIUM-HIGH for the Malay abbreviation table (UPU address standard + OSM road lists + Malaysian postal examples — all real but not a complete KL corpus), MEDIUM for the scoring-function recommendation (algorithm properties are cited; the exact weights must be settled empirically by the corpus runner) +**Domain:** Telegram Bot API venue cards + inline keyboards + Mini App deep-link hand-off; Malay/Malaysian place-name normalization; in-Worker fuzzy search over a curated gazetteer; OpenStreetMap Overpass candidate generation; ODbL/GPL-3.0 licensing +**Confidence:** HIGH for Telegram Bot API surface (official `core.telegram.org` type pages fetched this session), HIGH for the Malay abbreviation table (official JUPEM gazetteer tables + Nominatim `variants-ms.yaml` fetched this session), HIGH for ODbL text (fetched this session), MEDIUM for the deep-link parameter delivery mechanism (see §"Deep-link risk" — the mechanism is documented but the exact inline-button behaviour is contested), HIGH for the environment findings (probed this session) ## User Constraints (from CONTEXT.md) @@ -56,32 +56,30 @@ | ID | Description | Research Support | |----|-------------|------------------| -| **SRCH-01** | User can type a destination query (e.g. "KLCC") and receive a shortlist containing the right KL-metro landmark/POI | §"The `SearchSource` seam" (pluggable interface + in-Worker impl), §"Scoring architecture — three tiers", §"Golden corpus + threshold gate" (the executable ≥90% top-3 gate). Gazetteer entry schema in §"Gazetteer entry schema" and seed rows in §"Seed gazetteer shape" | -| **SRCH-02** | User gets useful results despite abbreviations and typos (Jln↔Jalan, Tmn↔Taman, missing spaces, common Malay address shorthand) | §"Malay normalizer — rule table" (the full bidirectional abbreviation table with authoritative UPU + OSM grounding), §"Glued-word splitting", §"Scoring architecture — three tiers" (alias/canonical exact-match tier precedes fuzzy). §"Normalizer test vectors" gives per-rule vectors | -| **SRCH-03** | For a chosen destination, a native venue card is sent in chat, confirming where we're going before navigation opens | §"`sendVenue` — what the card can and cannot carry" (verified against installed grammY 1.46.0 + @grammyjs/types: `reply_markup` IS supported on `sendVenue`, so D-02's single-message shape works), §"Four-square type mapping" | -| **ONBD-02** | User can open the Mini App from a search result with their destination preloaded (no re-entry in the webview) | §"Deep-link mechanics" (the `web_app` inline button URL vs `t.me?startapp=` direct link — with the **Desktop START-button trap** documented as a pitfall), §"Reading `start_param` in the webview" (validated `initData` field, character restrictions), §"Harness evidence path" | -| **ONBD-03** | User can add araharah to their phone home screen and launch search/navigation from there (addToHomeScreen + startapp deep-link) | §"ONBD-03 — what `addToHomeScreen` actually is" (Bot API 8.0 `addToHomeScreen()` + `checkHomeScreenStatus()`; **iOS always returns `unknown`** — verified against Telegram-iOS issue #1729), §"Home-screen launch has no `start_param`" (the startapp-less boot path) | +| **SRCH-01** | User can type a destination query (e.g. "KLCC") and receive a shortlist containing the right KL-metro landmark/POI | `/cari` command handler + `SearchSource` interface + curated gazetteer. §"SearchSource interface" defines the contract; §"Standard Stack" defines the JSON entry schema; §"Golden corpus" defines the acceptance mechanism that proves "KLCC" → Suria KLCC. §"Anti-patterns" warns against a bare string-equality match. | +| **SRCH-02** | User gets useful results despite abbreviations and typos (Jln↔Jalan, Tmn↔Taman, missing spaces, common Malay address shorthand) | §"Malay normalizer" — the full JUPEM-derived bidirectional abbreviation table (fetched verbatim from Nominatim `variants-ms.yaml`), the vowel-drop Presint/Prsn rule, glued-word splitting, and the Damerau-Levenshtein residual. §"Standard Stack" fixes the scoring shape. §"Pitfall 3" covers the "abbreviation only applies to generic names, not proper names" rule (MS 2039:9). | +| **SRCH-03** | For a chosen destination, a native venue card is sent in chat, confirming where we're going before navigation opens | §"sendVenue" — the exact Bot API parameter set (all four required fields), the `reply_markup` inline-keyboard slot, and the constraint that `sendVenue` accepts no `text` field (so the "one line confirming the destination" must be a *separate* message or the venue's `title`/`address` themselves). §"Inline keyboards" for the alternates + OPEN button. | +| **ONBD-02** | User can open the Mini App from a search result with their destination preloaded (no re-entry in the webview) | §"Deep-link risk" — **the single highest-risk item in this phase**. Documents that `start_param` is officially scoped to direct links / `startattach`, that the `web_app` inline button carries a raw URL, and gives the planner three concrete carriers ranked by risk, plus the write-both-readers fallback. `WebAppInfo.url` (official type page) confirms the button URL is an arbitrary HTTPS URL. | +| **ONBD-03** | User can add araharah to their phone home screen and launch search/navigation from there (addToHomeScreen + startapp deep-link) | §"ONBD-03 feasibility" — `addToHomeScreen()` / `checkHomeScreenStatus()` are Bot API 8.0+ only, the iOS limitation (`checkHomeScreenStatus` always returns `"unknown"` on iOS — confirmed upstream issue #1729, closed as expected platform behaviour), the `TelegramWebAppLike` extension shape, and the fact that the home-screen shortcut opens the Main Mini App (so `startapp` is available there even though it is not on an inline-button launch). | --- ## Summary -Phase 4 is small in code and large in judgement. The locked decisions (D-01…D-15) have already made the two hardest architectural calls: the backend is an in-Worker `SearchSource` over a curated KL gazetteer (no VPS, no Photon, $0 infra), and the interaction surface is the `/cari` command with a native venue card. What remains for research to settle is the *quality machinery* — the Malay normalizer, the scoring function, the corpus format, and the exact Telegram payload shapes — plus three things the roadmap never mentioned but that will bite the planner: +Phase 4 is the phase where the project stops being a shell and starts being useful — and it carries **three materially different risks** that the plan must not conflate. -**1. The `web_app` inline button and the `t.me?startapp=` direct link are NOT equivalent, and on Telegram Desktop the direct link needs a second tap.** D-10 says the OPEN button is a `web_app` button "carrying `startapp`". Verified from the installed `grammy@1.46.0` / `@grammyjs/types` typings: `InlineKeyboardButton.web_app` is a `WebAppInfo` whose **only** field is `url: string` — an HTTPS URL of the Mini App itself. There is **no `start_param` field on `WebAppInfo`**. The `startapp` query parameter is a **deep-link** construct (`https://t.me/?startapp=dest_`), and Telegram only propagates it to the Mini App when the Mini App is opened *that* way. So the OPEN button's URL must be either (a) the Mini App origin with the parameter appended to the *webview URL's own query string* — which does **not** become `start_param` and must be read from `location.search` instead — or (b) a `t.me` deep link. Option (b) has a documented trap: [telegramdesktop/tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694) reports the user "must manually press the 'START' button for the deeplink to be processed" on Telegram Desktop for Windows/Linux. **Recommendation: use a `url` inline button (not a `web_app` button) pointing at the Mini App origin with `?dest=` in the query string**, read it from `location.search` in the webview, and fall back to `start_param` when present. This works identically on every client, has no START-button trap, and keeps D-11's "plain gazetteer entry id" payload. The planner must make this call explicitly; it is not free. +**Finding 1 — the deep-link mechanism in D-11/D-12 is the riskiest thing in the phase, and the documentation does not say what D-11 assumes.** Telegram's official docs say the `start_param` field in `WebAppInitData` is *"the value of the `startattach` parameter, passed via link. Only returned for Mini Apps when launched from the attachment menu via link"*, and separately that `startapp` is populated when *"a non-empty `startapp` parameter is included in the link"* — where "the link" means a **direct link** (`https://t.me/bot?startapp=…`) or a **main Mini App link**, *not* a `web_app` inline keyboard button. But the `web_app` button's `WebAppInfo.url` is documented as *"An HTTPS URL of a Web App to be opened with additional data"* — i.e. an arbitrary URL, and community reports confirm query params on that URL survive into the webview. **D-11's "plain gazetteer entry id" payload is fine; the *carrier* is not settled.** The planner must pick a carrier explicitly and the plan must carry a fallback. §"Deep-link risk" lays out three ranked carriers with their evidence. -**2. A curated OSM-derived gazetteer is a legal "Derivative Database" under ODbL, and that is in tension with the GPL-3.0 project license (D-04).** This is the single most consequential finding of this research. The OSMF's own [Trivial Transformations Guideline](https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline) says explicitly: *"A 'geocoding database' should **NOT** be considered a trivial transformation… Geocoding is being considered for a separate community guideline."* The Use Cases page says: *"if you have any data that was derived from OSM — for instance because you used street names from OSM… then you are making a derivative database."* A curated gazetteer of KL POIs with names, coordinates and addresses extracted from OSM and committed as a public JSON file **is** a derivative database: ODbL §4.4 requires it be made available under ODbL or a compatible licence. GPL-3.0 is a **software** licence and is **not** a compatible database licence for ODbL share-alike — the OSMF's [Licence Compatibility](https://foundation.openstreetmap.org/wiki/Licence/Licence_Compatibility) page lists licences requiring users to use the data on *identical* terms as generally incompatible. **Recommendation: dual-license the gazetteer data file itself under ODbL-1.0 (with `© OpenStreetMap contributors` attribution) while keeping the *code* GPL-3.0.** This is exactly how Photon, Nominatim-adjacent tools, and OSM-derived routing datasets are shipped, and it satisfies D-04's forkability law (the data file stays a plain committed file; a fork inherits the ODbL obligation on that file, which is the correct and standard outcome). The planner must also ensure the "OPEN IN ARAHARAH" venue card / bot reply carries the OSM attribution string in both locales — the web app already ships `attribution: "© OpenStreetMap contributors"` in `web/src/i18n/strings.ts`, so the bot side needs the same. +**Finding 2 — the Malay abbreviation table does not need to be invented; it exists, it is official, and it is already machine-readable.** JUPEM (the Malaysian mapping authority) publishes a normative abbreviation table in every state's gazetteer, and the OSM Nominatim project transcribed it into `settings/icu-rules/variants-ms.yaml` (merged April 2026, PR #4056). Both were fetched verbatim this session and are reproduced in §"Malay normalizer". That table already covers `Jln→Jalan`, `Tmn→Taman`, `Kg→Kampung`, `Bkt→Bukit`, `Sg→Sungai`, `Bdr→Bandar`, `Psrn→Persiaran`, `Lbh→Lebuh` and more, with the Malaysia-specific and shared variants split exactly the way the normalizer needs them. **The planner should vendor this table (with its JUPEM attribution) rather than author a smaller one.** One important extra rule the table does *not* encode comes from MS 2039:9 — an abbreviation may apply to a *generic* geographic term but **not** to a name that is part of a proper name ("Jln Taman Desa" ✓, "Jln Tmn Desa" ✗). That rule is a corpus-level invariant the normalizer needs. -**3. The OSM licence problem is avoidable for the seed data, and that changes the Overpass script's job.** The D-06 Overpass script is a *proposal generator*, not the data pipeline: its output is a human-reviewed candidate list, and the curated file is maintained by hand. If the maintainer **writes the address strings and picks the aliases by hand** from local knowledge (and only copies the *coordinates* from OSM, treating them as facts rather than as a database extraction), the curated file's derivative-database exposure drops sharply — but "substantial" is a judgement call the planner should not make unilaterally. **The safe, standard, and cheap answer is the dual-licence above**; it costs one `LICENSE` file, one attribution line, and one sentence in the README. D-04 explicitly asks for a GPL-3.0 `LICENSE` file this phase — the planner should add the ODbL notice alongside it for the data file. +**Finding 3 — the Overpass dependency is real, and this machine reaches the primary endpoint **only with a `User-Agent` header**.** `overpass-api.de` returns **HTTP 406** on every request that lacks an identifying `User-Agent` — including a bare `GET /` — and **HTTP 200** with one. (Verification pass, same host/same paths, varying only the header: `GET /api/status` no-UA → 406 / with-UA → 200; `GET /` no-UA → 406 / with-UA → 200; `POST /api/interpreter` no-UA → 406 / with-UA → **200 with 19 KB of real KL data**.) The other public mirrors are unusable from here — `overpass.kumi.systems` and `overpass.private.coffee` fail to connect (HTTP 000), and `overpass.osm.ch` is a Switzerland-only regional instance that returns **400**. Nominatim (`nominatim.openstreetmap.org`) and Geofabrik (`download.geofabrik.de/asia/malaysia-singapore-brunei-latest.osm.pbf`) **are** reachable and return correct KL data. This does not block the phase — D-06 says the Overpass script is a *repo tool for maintainers*, not a runtime dependency — but the planner must know that (a) the script's `User-Agent` is **load-bearing, not cosmetic**, (b) capacity **502** responses are **intermittent and normal** (a 200 and a 502 were observed minutes apart on the same host), and (c) **the top-up script's automated test must target a committed fixture, not the live service.** -**4. Workers Free gives 10 ms of CPU per request.** [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/) (fetched 2026-10-23): "CPU time per HTTP request | **10 ms**", 100,000 requests/day, 128 MB memory, 1 s startup. The average Worker uses ~2.2 ms. A gazetteer of 200–500 rows with, say, 8 aliases per row yields 1,600–4,000 folded alias strings. A naive "compute Damerau-Levenshtein against every alias" pass is O(4,000) × O(len²) — comfortably inside 10 ms only if the candidate set is first narrowed. **Recommendation: index-at-load alias folding with an exact-match hash lookup first (O(1)), then a cheap character-bigram Dice filter to cut the candidate set to a handful, then Damerau-Levenshtein only on that handful.** This is the classic three-tier cascade and it also gives the "honest refusal" behaviour D-14 demands for free: a query that produces no candidate above threshold simply yields zero results. Also note the **1 s Worker startup limit** and the "avoid expensive work in global scope" guidance — the gazetteer should be parsed **inside the request handler** (or at least lazily), not at module top level, or a 500-row JSON parse + fold runs on every cold start. +**Finding 4 (bonus) — the licensing is compatible, but only if the gazetteer is treated as data, not code.** ODbL §2.3(a) explicitly excludes computer programs from the license, and OSMF's guidelines confirm OSM *data* is ODbL while *code* is separately licensed. The precedent is live: `cbeier-studio/Xolmis` is GPL-3.0 and ships ODbL data in the same repo with an explicit dual-license note. **But ODbL §4.4 does impose share-alike on a Derivative Database** — so the curated gazetteer file must carry an ODbL notice, and if the normalizer's *index* (a derived, restructured form of the gazetteer) is shipped in the repo, it should carry the same notice. This is a two-line obligation with a real legal consequence and no cost. The repo currently has **no LICENSE file at all** — D-04 says GPL-3.0 is the project license, so adding `LICENSE` (GPL-3.0) plus a `data/` notice is a concrete deliverable this phase, not a Phase 7 chore. -**5. `sendVenue` supports `reply_markup`, so D-02's one-message shape works.** Verified directly in the installed typings (`node_modules/@grammyjs/types/methods.d.ts`, `sendVenue` args include `reply_markup?: InlineKeyboardMarkup | …`). So the venue card and its OPEN + alternates buttons ship as ONE message. That satisfies D-02 exactly ("no second message round"). The `Venue` type carries `location`, `title`, `address`, plus optional `foursquare_id`/`foursquare_type`/`google_place_id`/`google_place_type` — the four-square type is the free, self-hostable choice (no Google API key needed), and the Foursquare ID can be the gazetteer's own OSM-derived identifier. +**Primary recommendation:** Ship Phase 4 as three separable layers — (1) a **pure, dependency-free shared search module** (`gazetteer.json` + normalizer + scorer, importable by both `bot/` and `web/`), (2) a **bot command surface** (`/cari` → `sendVenue` + inline alternates), and (3) a **deep-link carrier with a documented fallback**. Make the deep-link carrier the *first* thing built and the *last* thing gated, because it is the only requirement in the phase whose success depends on undocumented Telegram client behaviour. Then close the experiment with the corpus runner: a single command that prints top-3 hit rate on positives and refusal rate on negatives, and a written decision in the repo saying which lever was pulled. -**6. ONBD-03's `addToHomeScreen` is a Telegram-native shortcut, not a web PWA install, and iOS cannot report its status.** `WebApp.addToHomeScreen()` and `WebApp.checkHomeScreenStatus(callback)` are Bot API 8.0+ methods ([core.telegram.org/bots/webapps](https://core.telegram.org/bots/webapps)), returning one of `unsupported` / `unknown` / `added` / `missed`. [Telegram-iOS issue #1729](https://github.com/TelegramMessenger/Telegram-iOS/issues/1729) confirms: *"Home screen status detection always returns 'unknown' on iOS… On iOS it is not possible to determine whether a Mini App has been added to the Home Screen. The platform does not provide any API that allows Telegram to check whether a Home Screen shortcut exists."* The `beforeinstallprompt` event that a normal web PWA uses **does not exist inside a Telegram webview** — there is no PWA install prompt. So D-12's "minimal PWA hooks" means: call `addToHomeScreen()`, handle `homeScreenAdded`, and treat `unknown` as "possibly added" (iOS). A home-screen shortcut launched by the OS **carries no `start_param`** — it opens the Mini App cold, which is why D-12 requires the startapp-less boot to land in the "search-ready map". That path already exists from Phase 3 (the staged boot), so ONBD-03 is mostly a *verification* obligation plus a small hook. - -**Primary recommendation:** Ship this phase as three thin, separately-testable artifacts — a pure `SearchSource` + normalizer + scorer module (no I/O, no Telegram, no DOM; fully exercisable by the corpus runner), a `sendVenue`-based `/cari` handler that talks to it, and a webview boot branch that reads the destination id from `location.search`/`start_param` — then let the pinned golden corpus be the *executable* definition of the ≥90% gate, with the escalation ladder (normalizer table edits → backend #2 → backend #3) written as a documented decision record in the phase directory rather than as an open question. +**The one artifact to lift directly:** §"Malay Normalizer — the vendorable rule table" reproduces the **complete JUPEM-derived abbreviation table verbatim** (the machine-readable transcription from Nominatim's `variants-ms.yaml`), plus the **MS 2039:2006** display rule, the **Presint/Prsn** vowel-drop case with real usage evidence, and the three official tables that must be merged. The planner should vendor that table into the shared module with its JUPEM attribution rather than author a smaller one — the difference between the 8-entry original and the current 65+ entry table is the difference between "Jln" working and "Jln Ampang" working. --- @@ -89,17 +87,17 @@ Phase 4 is small in code and large in judgement. The locked decisions (D-01…D- | Capability | Primary Tier | Secondary Tier | Rationale | |------------|-------------|----------------|-----------| -| Query normalization (Malay abbreviation folding, case/whitespace, glued-word split) | API / Backend (Worker) | Frontend Server (browser runtime) | D-09 says the normalizer is applied to *both* query and entry aliases at index time — the canonical copy must be the shared module. It is pure string logic with no I/O, so it runs identically in the Worker (bot) and in the browser (webview copy at build time). One implementation, two consumers. | -| Gazetteer storage + alias indexing | API / Backend (Worker) | Database / Storage (repo file) | D-07: the canonical store is a typed JSON file in the repo; the Worker parses it at request time. The webview gets a build-time copy of the same module. No database tier exists in v1 (KV is TTL-only session state). | -| Fuzzy scoring / ranking / shortlist | API / Backend (Worker) | — | Scoring runs server-side so the corpus runner and the production path exercise *the same code*. If scoring lived in the browser, the corpus runner would have to duplicate it and the bake-off would measure the wrong thing. | -| `/cari` command intake + reply shaping | API / Backend (Worker) | — | The command arrives as a Telegram webhook update; only the Worker can read it. | -| Venue card rendering (name/address/coordinates) | API / Backend (Worker) → Telegram client | — | `sendVenue` is a Bot API call. The card is rendered by Telegram, not by araharah. | -| Deep-link construction (`dest_` payload) | API / Backend (Worker) | — | The Worker knows the entry id; it composes the Mini App URL. | -| Destination-id → destination resolution in the webview | Frontend Server (browser runtime) | API / Backend | D-11: the webview maps id → destination from *its own* build-time copy of the gazetteer module. No API round-trip is needed — the id is a plain key into a file the bundle already carries. | -| Bot menu registration for `/cari` | API / Backend (repo script) | — | `bot/scripts/register-commands.mjs` derives menu descriptions from `strings.helpMenu` — the Phase-2 pattern. `/cari` must be added to `helpMenu` in `bot/src/i18n/strings.ts` or it will not appear in the menu (Phase-2 CS5 gap-freedom gate). | -| Result caching | Database / Storage (KV, TTL-only) | API / Backend | D-15 permits TTL-only KV caching keyed by normalized query. At this scale it is probably unnecessary (see §"Should there be a result cache?"). | -| Overpass top-up proposal generation | CDN / Static → repo tool (offline script) | — | D-06: a repo tool, never a runtime dependency. It runs on a maintainer's machine, not in the Worker. | -| Home-screen install affordance | Frontend Server (browser runtime) | Telegram client | `WebApp.addToHomeScreen()` is client-side; the event fires back into the page. The Worker is not involved. | +| **Malay normalization + scoring (pure functions)** | *(shared library — no tier)* | — | D-09/D-11 require the *same* module in bot and web. A workspace package (`@araharah/search`) or a shared `src/` dir imported by both. No runtime, no IO, no network. | +| **Gazetteer data (curated JSON)** | *(data file — no tier)* | — | D-07: typed JSON in the repo, committed, diff-friendly. Read by both tiers. | +| **`/cari` command parsing** | **API / Backend** | — | Telegram command dispatch lives in `bot/src/bot/commands/`. The Worker is the only tier that receives Telegram updates. | +| **Query → shortlist (SearchSource)** | **API / Backend** | Browser / Client | D-03: in-Worker implementation. The webview copy is build-time only (D-11) — it maps an id to coordinates, it does not re-run the search. | +| **Venue card + inline keyboard rendering** | **API / Backend** | — | `sendVenue` is a Bot API call; only the Worker holds `BOT_TOKEN`. | +| **Alternate selection (callback query)** | **API / Backend** | — | `callback_query` updates arrive at the Worker webhook. | +| **Deep-link payload → destination lookup** | **Browser / Client** | API / Backend | D-11: the webview maps `id → destination` from *its own copy* of the shared module. The Worker never re-derives it. | +| **Mini App boot with preloaded destination** | **Browser / Client** | — | Phase 3's staged boot (validating → acquiring → live) is the entry; the destination becomes a pre-seeded target the Phase 5 route layer consumes. | +| **Home-screen prompt (ONBD-03)** | **Browser / Client** | — | `addToHomeScreen()` is a `Telegram.WebApp` method; only the webview can call it. | +| **Golden corpus + acceptance runner** | *(repo tooling — no tier)* | — | D-13/D-14: a repo script that imports the shared module and the corpus file. Runs under `node`, no network. | +| **Overpass top-up script** | *(repo tooling — no tier)* | — | D-06: maintainer-time tool, network-dependent, never imported by the Worker. | --- @@ -109,36 +107,39 @@ Phase 4 is small in code and large in judgement. The locked decisions (D-01…D- | Library | Version | Purpose | Why Standard | |---------|---------|---------|--------------| -| `grammy` | **1.46.0** (already in `bot/`, `^1.46.0`) | Bot framework — `/cari` command handler, `ctx.sendVenue`, inline keyboards | Already the project's bot framework. **Verified this session in the installed typings**: `api.sendVenue(chat_id, latitude, longitude, title, address, other?)` and `ctx.replyWithVenue(...)` both exist; `sendVenue`'s args include `reply_markup?: InlineKeyboardMarkup \| ReplyKeyboardMarkup \| …`; `InlineKeyboardButton` has a `url` variant and a `web_app: WebAppInfo` variant; `WebAppInfo` has exactly one field, `url: string`. | -| `node:assert` + hand-rolled runner | repo-native (`bot/test/runner.mjs`, `web/test/runner.mjs`) | Test harness for the corpus runner | Repo precedent, zero deps. The corpus runner is a new suite in the same shape (see §Validation Architecture). | -| **No search library** | — | Normalization, exact lookup, bigram filtering, Damerau-Levenshtein | **Deliberate.** At 200–500 rows the whole algorithm is ~150 lines of pure TS. `fuse.js@7.5.0`, `fastest-levenshtein@1.0.16`, `string-similarity@4.0.4`, `levenshtein-edit-distance@3.0.1` all exist on npm (verified) but each adds a dependency to a forkability-law project (D-04) to solve a problem this scale does not have. D-09's "extends by table edit, not code change" is better served by a hand-rolled, table-driven normalizer than by a library's config surface. | +| **grammy** | `1.46.0` (installed; `npm view grammy version` → `1.46.0` — current) | Bot API client: `sendVenue`, `InlineKeyboard`, `callbackQuery` handlers | Already the project's bot framework (Phase 2). `InlineKeyboard` has a first-class `.webApp(text, url)` builder, so D-10's `web_app` button needs no raw object literal. | +| **Node 24.18.0 built-ins only** (`JSON.parse`, `Intl`, `String.prototype.normalize`) | n/a | Gazetteer loading, normalization, scoring | D-07 mandates V8-native parse, no WASM. D-09's rule table is plain data. Damerau-Levenshtein on a ~500-row × ~5-alias corpus is a ~2500-pair comparison — a hand-rolled 20-line OSA loop is *faster than any dependency* at this size and adds zero bundle weight. **Recommendation: hand-roll the OSA distance in the shared module** (see §"Don't Hand-Roll" for why this one is an exception). | +| **@crob/damerau-levenshtein** | `2.0.0` (npm, published ~1 month ago) | *Alternative* to hand-rolling OSA distance | TypeScript-first, ESM+CJS+UMD, 1.2 kB gzipped, configurable per-op costs (nice: a cheap cost for adjacent-keyboard substitutions). **Requires Node 20.19+** (we have 24.18.0). Use if the plan wants a battle-tested implementation; skip if the plan prefers zero runtime deps. `[VERIFIED: npm registry]` | +| **`damerau-levenshtein`** | `1.0.8` (npm) | *Alternative* | BSD-2, 52M weekly downloads, but last published 2022, CommonJS-ish `import * as levenshtein` shape, and returns `{steps, relative, similarity}` rather than a bare distance. Older and less pleasant than `@crob/damerau-levenshtein`. `[VERIFIED: npm registry]` | +| **`fastest-levenshtein`** | `1.0.16` (npm) | *Alternative* — Levenshtein only, no transposition | Fast, but **no transposition support** — and transpositions ("KLCC" → "KCLC", "Sunway" → "Sunwya") are an explicit D-13 corpus case. Rejected. `[VERIFIED: npm registry]` | ### Supporting | Library | Version | Purpose | When to Use | |---------|---------|---------|-------------| -| `node:crypto` (Web Crypto `subtle`) | built-in | Nothing cryptographic is needed — but see the D-11 note | **D-11 explicitly says "No crypto"** for the deep-link payload. Do **not** add a signature. The id is a plain key into a public gazetteer; the only threat is self-pranking. | -| `wrangler` | **4.151.3** (already in `bot/`) | `wrangler dev --port 8787 --local` (local KV) + `wrangler deploy` | D-12 Phase 3's dev topology; the gazetteer is bundled into the Worker at deploy time. | -| `vite` | **8.3.5** (already in `web/`) | Build-time copy of the gazetteer module into the webview bundle | D-11's "shared module, build-time". The web bundle must import the *same* module the bot imports — the sharing mechanism matters, see §"Sharing the gazetteer between bot and web". | -| `playwright` | `^1.50.0` (already in `web/`, browsers cached: chromium-1248, webkit-2272, firefox-1511) | Mock-Telegram harness capture for ONBD-02 evidence | Phase 3's `web/scripts/capture.mjs` already builds `web/dist-harness` and races `RAMS-BASEMAP-READY` → `ARAHARAH-LIVE`. A `?dest=` boot profile slots straight into the existing capture list. | +| **Playwright** | `1.50.0` (already a `web` devDependency) | Capture evidence for ONBD-02 (webview booted with a preloaded destination) | Reuse the existing `web/scripts/capture.mjs` harness pattern — add a `?dest=` profile. No new dependency. | +| **Nominatim** (network service, not a lib) | live API | *Fallback* candidate source + coordinate verification for curated entries | Verified reachable this session (`nominatim.openstreetmap.org/status` → 200; `?q=KLCC` returned correct KLCC nodes with `licence: "Data © OpenStreetMap contributors, ODbL 1.0"`). **Use it to verify curated coordinates** even if Overpass is unreachable. | +| **Geofabrik** (network service) | `asia/malaysia-singapore-brunei-latest.osm.pbf` | *Fallback* bulk OSM extract for the top-up script | Verified reachable (HTTP 206 on range request). Note the extract name is **malaysia-singapore-brunei**, not "malaysia" — a 404 trap. | ### Alternatives Considered | Instead of | Could Use | Tradeoff | |------------|-----------|----------| -| Hand-rolled 3-tier scorer | `fuse.js@7.5.0` (Bitap + fuzzy, 30 KB, MIT) | **Reject for v1.** Fuse is excellent general-purpose fuzzy search, but it (a) is a black box — the "extends by table edit, not code change" law (D-09) would be violated once you need a custom Malay rule table inside the scoring, (b) adds a dependency to a GPL/forkability-law project, (c) its scoring is optimized for long documents, not 3–20 character place names. **Revisit at backend #2** if the corpus shows the hand-rolled scorer cannot reach 90%. | -| Hand-rolled Damerau-Levenshtein | `fastest-levenshtein@1.0.16` (MIT, WASM+JS) | **Reject.** It is Levenshtein-only (no transposition), so it charges 2 edits for the transposition typos D-13's corpus explicitly requires ("transpositions"). D-09 locks Damerau for the residual net. A 40-line OSA/Damerau implementation is the whole requirement. | -| Precomputed trigram index at build time | Runtime bigram Dice filter | **Prefer the runtime filter.** A build-time trigram index adds a generated artifact to the repo (diff noise, stale-index risk) for zero measurable gain: at 4,000 aliases, building a bigram set per alias on the fly is ~1 ms. Revisit only if the corpus grows past ~5,000 entries. | -| OSM-derived gazetteer published under GPL-3.0 only | Dual-licence: code GPL-3.0, data file ODbL-1.0 | **Choose the dual-licence.** See §"Licensing — the ODbL obligation". GPL-3.0 alone risks ODbL §4.4 non-compliance on the data file. | -| `t.me/?startapp=` direct link for the OPEN button | `url` inline button → `/?dest=` | **Choose the `url` button.** The direct link works, but on Telegram Desktop (Windows/Linux) it shows a START button the user must press ([tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694)); the `url` button is a single tap on every client. See §"Deep-link mechanics". | +| Curated JSON gazetteer (D-07) | SQLite / `sql.js` in the Worker | D-07 already rejects this: ~1.5 MB bundle + cold-start for a ~200–500-row dataset. **Non-negotiable.** | +| Hand-rolled OSA distance | `@crob/damerau-levenshtein` (2.0.0) | +1.2 kB gzipped, well-tested, configurable costs; −~20 lines of hand-written code to maintain and test. Both viable at this corpus size. | +| Trigram/prefix index | Plain linear scan over ~500 entries × ~5 aliases | At ~2500 candidate pairs, a linear scan with early-exit on exact normalized match is **microseconds** — an index is premature. The trigram index becomes backend #2's job (D-05). | +| Overpass QL for top-up candidates | Nominatim search, or a Geofabrik `.pbf` + `osmium` | Overpass is the right tool *in general*; in *this environment* it is blocked (see §"Environment Availability"). The script should support both a `--source=overpass|nominatim` flag, or read a locally-downloaded extract. | +| `sendVenue` (D-10) | `sendMessage` + `sendLocation` | **Rejected.** SRCH-03 explicitly names the native venue card; `sendVenue` gives the pin/name/address semantics Telegram users read as "a place". Also: `sendVenue` accepts **no `text` field** — the confirming line must be a separate message or ride in the venue's own `title`/`address`. | +| `callback_data` alternates | Editing the message on every tap | D-02 wants alternates *on the same message* as the card. `editMessageText` can replace inline keyboards, but a venue message carries a `venue` payload that `editMessageText` cannot preserve — the card would be destroyed. **Use `callback_data` + `answerCallbackQuery` + a new `sendVenue`** (or `editMessageReplyMarkup`, which is documented as allowed on a message that already has an inline keyboard). | **Installation:** ```bash -# No new runtime dependencies. The gazetteer, normalizer and scorer are -# first-party modules inside the existing npm workspaces. +# Only if the plan chooses the library route for the distance function: +npm install @crob/damerau-levenshtein -w @araharah/search +# If the shared module is a workspace package, add "shared" to root package.json "workspaces". ``` -**Version verification:** the standard stack for this phase is entirely **already installed** — `grammy@1.46.0` (verified: `node -e "require('./node_modules/grammy/package.json').version"` → `1.46.0`), `wrangler@4.151.3` (verified via `npx wrangler --version`), `vite@8.3.5`, `typescript@5.9.3`, Node `v24.18.0`, npm `11.16.0`. The only registry probes run this session were for the **rejected** alternatives: `fuse.js` → **7.5.0**, `fastest-levenshtein` → **1.0.16**, `string-similarity` → **4.0.4**, `levenshtein-edit-distance` → **3.0.1**. +**Version verification:** `npm view grammy version` → `1.46.0` (matches installed). `npm view @crob/damerau-levenshtein version` → `2.0.0`. `npm view fastest-levenshtein version` → `1.0.16`. `npm view damerau-levenshtein version` → `1.0.8`. All verified against the registry this session. --- @@ -146,343 +147,265 @@ Phase 4 is small in code and large in judgement. The locked decisions (D-01…D- ### System Architecture Diagram -```text -USER (chat) ONE WORKER (bot/ + web/ statics) -──────────── ─────────────────────────────── - "/cari KLCC" - │ - │ webhook POST /telegram/webhook - ▼ - ┌──────────────────┐ - │ grammY router │──▶ /cari handler (bot/src/bot/commands/cari.ts) - └──────────────────┘ │ - │ 1. raw query string - ▼ - ┌─────────────────────────────┐ - │ normalizeQuery() │ ← rule table - │ case/whitespace fold │ (D-09) - │ abbreviation ↔ canonical │ - │ glued-word split │ - └──────────────┬──────────────┘ - │ normalized query - ▼ - ┌─────────────────────────────┐ - │ SearchSource.search(q) │ ← the seam (D-03) - │ ┌────────────────────────┐ │ backend #1 = GazetteerSource - │ │ tier 1: exact lookup │ │ backend #2 = WASM geocoder - │ │ tier 2: bigram filter │ │ backend #3 = Photon - │ │ tier 3: Damerau score │ │ - │ └────────────────────────┘ │ - └──────────────┬──────────────┘ - │ ranked candidates - ▼ - ┌─────────────────────────────┐ - │ rank + tie-break + top-3 │ - └──────────────┬──────────────┘ - │ - ┌────────────────────────────┴──────────────────────┐ - │ if empty → "not in v1 area" (en/ms, honest refusal) │ - │ if hits → sendVenue(best) + inline buttons │ - └────────────────────────────┬──────────────────────┘ - │ ← ONE message (D-02) - │ reply_markup: - │ row 0: OPEN IN ARAHARAH - │ → /?dest= - │ row 1..: alternates (callback_data) - ▼ - ╔══════════════════════════╗ - ║ TELEGRAM renders the ║ - ║ venue card (name, addr, ║ - ║ pin) + the buttons ║ - ╚══════════════════════════╝ - │ user taps OPEN - ▼ - ┌──────────────────────────────────────┐ - │ Mini App webview (web/) │ - │ read location.search ?dest= │ - │ (fallback: initData start_param) │ - │ → resolve id via BUILD-TIME COPY │ - │ of the same gazetteer module │ - │ → destination in memory only │ - │ → Phase-3 staged boot continues │ - └──────────────────────────────────────┘ ``` + ┌────────────────────────────────────────────────────────┐ + │ Telegram client │ + │ user types: /cari KLCC │ + └────────────────────────┬───────────────────────────────┘ + │ (Bot API update) + ▼ +┌────────────────────────────────────────────────────────────────────────────────┐ +│ Cloudflare Worker (bot/src/webhook.ts → POST /telegram/webhook) │ +│ │ +│ ┌──────────────────┐ ctx.args = "KLCC" │ +│ │ /cari handler │ (bot/src/bot/commands/cari.ts) │ +│ └────────┬─────────┘ │ +│ │ search("KLCC") │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────────────┐ │ +│ │ @araharah/search (SHARED — pure functions, no IO, no network) │ │ +│ │ │ │ +│ │ ┌───────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │ +│ │ │ normalizer.ts │→ │ scorer.ts │→ │ gazetteer.json (typed) │ │ │ +│ │ │ rule table + │ │ exact→alias │ │ {id,name,aliases, │ │ │ +│ │ │ folding + │ │ match→token │ │ lat,lng,category,addr} │ │ │ +│ │ │ glued split │ │ set→Damerau │ │ │ │ │ +│ │ └───────────────┘ └──────────────┘ └──────────────────────────┘ │ │ +│ │ ▲ normalize both sides at index time │ │ +│ └────────┼──────────────────────────────────────────────────────────────┘ │ +│ │ SearchSource interface (D-03) — backend #2 / #3 plug in here │ +│ ▼ │ +│ shortlist: [best, ...alternates] (or EMPTY → honest refusal) │ +│ │ +│ ┌─────────────────────────┐ │ +│ │ sendVenue(best) │ ← SRCH-03: native venue card │ +│ │ + inline_keyboard: │ │ +│ │ [ OPEN IN ARAHARAH ] │ ← web_app button, URL carries the dest id │ +│ │ [ alt 2 ] [ alt 3 ] │ ← callback_data = "alt:" │ +│ └──────────┬──────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────┐ │ +│ │ callback_query handler │ ← user taps an alternate │ +│ │ answerCallbackQuery() │ │ +│ │ + sendVenue(newBest) │ │ +│ └─────────────────────────┘ │ +└────────────────────────────────┬───────────────────────────────────────────────┘ + │ + ┌────────────────────────┴──────────────────────────┐ + │ (A) web_app inline button ──► Mini App webview │ + │ (B) url-button deep link ──► t.me?startapp=… │ + │ (C) in-webview URL query ──► ?dest= │ + ▼ ▼ +┌──────────────────────────────────────────────────────────┐ +│ Mini App (web/src/main.ts) — Phase 3 staged boot │ +│ validating session → acquiring location → live │ +│ + destination preloaded (id → entry from shared module)│ +│ + ONBD-03: addToHomeScreen() prompt handling │ +└──────────────────────────────────────────────────────────┘ -Flow trace — the primary use case ("/cari KLCC"): -1. User types `/cari KLCC` → Telegram delivers a `message` update to `/telegram/webhook`. -2. The grammY router dispatches to the `/cari` handler (the Phase-2 alias pattern). -3. The handler extracts the raw query text **after** the command and its arguments. -4. `normalizeQuery("KLCC")` folds case/whitespace; the abbreviation table leaves it unchanged. -5. `SearchSource.search("klcc")` runs tier 1 (exact alias hash) → hits the KLCC entry immediately. -6. The handler ranks, takes the top 1 as best and up to 4 as alternates. -7. `ctx.sendVenue(lat, lng, title, address, { foursquare_type, reply_markup })` — one message. -8. The user taps OPEN → the webview opens at `/?dest=klcc-pavilion`. -9. The webview reads `location.search`, resolves the id from its bundled gazetteer copy, and hands the destination to the Phase-3 boot flow. Nothing is re-typed; nothing is persisted. +┌──────────────────────────────────────────────────────────┐ +│ Repo tooling (never imported by the Worker) │ +│ bot/scripts/…-gazetteer-topup.mjs → Overpass/Nominatim│ +│ …-corpus.mjs (golden corpus runner) → node │ +│ …-json-to-sqlite.mjs (D-05 backend #2 converter stub) │ +└──────────────────────────────────────────────────────────┘ +``` ### Recommended Project Structure -```text -bot/src/ -├── search/ # ← NEW: the shared search module -│ ├── types.ts # SearchSource interface, SearchResult, GazetteerEntry -│ ├── gazetteer.ts # the curated KL dataset loader (JSON import) -│ ├── data/ # ← NEW: the curated JSON (D-07) -│ │ └── kl-gazetteer.json # 200–500 rows, typed, diff-friendly -│ ├── normalize.ts # rule table + folding + glued-word split (D-09) -│ ├── rules/ -│ │ └── abbreviations.ts # ← the editable rule table (D-09) -│ ├── score.ts # exact → bigram filter → Damerau -│ └── source.ts # GazetteerSource implements SearchSource (D-03) -├── bot/commands/cari.ts # ← NEW: the /cari handler (D-01) -└── i18n/strings.ts # ← MODIFIED: search copy en/ms + /cari in helpMenu +The key structural decision: **the search module must be importable by both `bot/` and `web/`**. Two viable shapes: -bot/scripts/ -├── overpass-topup.mjs # ← NEW: repo tool, proposes candidates (D-06) -└── register-commands.mjs # ← MODIFIED (via helpMenu, no inline copy) +``` +Option A — a new workspace package (cleanest): +shared/ @araharah/search — pure TS, zero deps +├── package.json (type: module, exports ./src/index.ts) +├── src/ +│ ├── index.ts public API: search(), normalize() +│ ├── types.ts GazetteerEntry, SearchSource, SearchResult +│ ├── normalize.ts rule table + folding + glued-word split +│ ├── rules.ts the abbreviation table (JUPEM-derived) +│ ├── score.ts exact → alias → token-set → Damerau +│ └── distance.ts Damerau-Levenshtein OSA (hand-rolled or lib) +├── data/ +│ ├── kl-gazetteer.json the curated entries (D-07) +│ └── NOTICE.md ODbL attribution for the OSM-derived data +└── test/ corpus runner + rule test vectors -bot/test/ -├── cari.test.mjs # ← NEW: handler + payload-shape gates -├── normalizer.test.mjs # ← NEW: per-rule vectors (D-09) -└── corpus.test.mjs # ← NEW: the golden-corpus gate (D-13/D-14) +Option B — a shared dir at repo root imported by both workspaces: +packages/search/ (same shape) -bot/test/fixtures/ -└── golden-corpus.json # ← NEW: the pinned corpus (D-13) - -web/src/ -├── search/ # ← NEW: build-time copy of the SAME module -│ ├── gazetteer.ts # (or a re-export from a shared location) -│ └── resolve-destination.ts # id → destination, in-memory only -├── main.ts # ← MODIFIED: boot branch for ?dest= -└── telegram/webapp.ts # ← MODIFIED: add startParam to the narrow interface - -web/index.html # ← MODIFIED: ?dest= harness boot profile -web/test/ -└── destination-preload.test.mjs # ← NEW: ONBD-02 gate +Both workspaces then do: + import { search } from "@araharah/search"; + import gazetteer from "@araharah/search/data/kl-gazetteer.json" with { type: "json" }; ``` -### Pattern 1: The `SearchSource` seam (D-03) +``` +bot/src/ +├── bot/commands/cari.ts NEW — /cari + /cari alias +├── bot/handlers/search-alternate.ts NEW — callback_query handler for alternates +├── search/ +│ └── source.ts NEW — the SearchSource interface (D-03) +│ + GazetteerSource (the in-Worker impl) +└── i18n/strings.ts EXTENDED — search copy (en/ms) -**What:** One interface, several backends. The bot only ever talks to `SearchSource`; swapping backends is a one-line change in the Worker factory, not a refactor of the handler. +web/src/ +├── main.ts EXTENDED — read dest id at boot, seed destination +├── telegram/webapp.ts EXTENDED — addToHomeScreen/checkHomeScreenStatus +├── search/ NEW — the webview's copy of id→entry lookup +└── i18n/strings.ts EXTENDED — any search-ready copy -**When to use:** Always — this is the phase's central abstraction and it is what makes the D-14 escalation ladder mechanical rather than aspirational. +repo root: +├── LICENSE NEW — GPL-3.0 (D-04) +├── shared/ or packages/search/ NEW — the shared module (above) +└── bot/scripts/ + ├── gazetteer-topup.mjs NEW — Overpass/Nominatim candidate proposer (D-06) + ├── gazetteer-corpus.mjs NEW — golden corpus runner (D-13/D-14) + └── gazetteer-to-sqlite.mjs NEW — JSON→SQLite converter (D-05, stub shape) +``` +**Import assertion:** the shared module must be importable by a **Node** script (corpus runner), a **Cloudflare Worker** (bundled by wrangler/esbuild), and a **browser** (Vite). All three handle a `.ts` file if the consuming toolchain compiles it — the corpus runner is `node`, which **cannot** import `.ts` directly in Node 24 without a loader flag. **The corpus runner therefore must be `.mjs` and import a `.js`/`.mjs` build of the module, or the shared module must be authored in plain `.js` with JSDoc types + a hand-written `.d.ts`.** This is a concrete, easy-to-miss build constraint. Node 24 does have type-stripping (`--experimental-strip-types`), but relying on it in a repo script is fragile; the safer shape is **author the shared module in plain ESM JavaScript with JSDoc typedefs**, matching the repo's existing `.ts`-source-imported-from-`.mjs`-test convention (the existing tests already do `await import("../src/i18n/strings.ts")` — verify whether that works under the repo's current Node version before repeating it). + +> **Planner note:** `bot/test/commands.test.mjs` already does `await import("../src/i18n/strings.ts")`. Check `node --version` (24.18.0 here) — Node 24 has **type stripping enabled by default** for `.ts` files, so `import("../src/...ts")` works. If so, the shared module can stay `.ts` and the corpus runner can import it directly. **Verify this in the plan's first task**, because the whole test strategy depends on it. + +### Pattern 1: The `SearchSource` interface (D-03) + +**What:** a narrow contract that the in-Worker gazetteer implements today, and that backend #2 (WASM geocoder) and backend #3 (Photon-on-VPS) implement later. + +**When to use:** always — this is the seam. Phases 5/6 plug origin/destination resolution into the same interface. + +**Example:** ```typescript -// Source: D-03 (CONTEXT.md) — the interface shape is this research's recommendation, -// constrained to what the phase actually needs. -export interface SearchResult { - /** Gazetteer entry id — the payload D-11 puts in the deep link. */ - id: string; - name: string; - address: string; +// shared/src/types.ts +export interface GazetteerEntry { + id: string; // stable, slug-like: "suria-klcc", "sunway-pyramid" + name: string; // canonical display name + aliases: string[]; // locals' shorthand — D-08's main quality lever lat: number; lng: number; - category: string; - /** 0..1, higher is better. Used for ranking + tie-breaks only. */ - score: number; - /** Which alias/form matched — useful for the corpus report, never for logging. */ - matchedVia: "exact" | "normalized" | "fuzzy"; + category: string; // "mall" | "office-tower" | "park" | "station" | … + address?: string; // human address for sendVenue's `address` field +} + +export interface SearchQuery { + text: string; // raw user query + limit?: number; // default 5 (1 best + 2–4 alternates, D-02) +} + +export interface SearchHit { + entry: GazetteerEntry; + score: number; // 0..1, higher is better + matchedAlias: string; // which alias matched — useful for the corpus report } export interface SearchSource { - /** Normalized query → ranked candidates. Empty array = honest refusal. */ - search(query: string, limit: number): Promise; - /** The backend's identity, for the escalation record (D-14). */ - readonly backendId: string; + readonly name: string; // "gazetteer" | "wasm-geocoder" | "photon" + search(q: SearchQuery): Promise; // EMPTY array = honest refusal } ``` -**Why this shape:** `search()` returns `Promise` even though backend #1 is synchronous. That keeps the signature stable for backend #3 (Photon is a network call) — a future swap must not change the handler. +**The refusal contract is the important part.** An empty result is a *legitimate, load-bearing* return value (D-13's negatives: "Kuching", "Penang" must be refused, never fabricated). The interface must make "no match" easy to express and the bot must translate it into the "not in v1 area" copy — never into a best-guess pin. -### Pattern 2: Three-tier scoring cascade (SRCH-01, SRCH-02) +### Pattern 2: Rule-table normalization (D-09) -**What:** Exact-match first, cheap filter second, expensive score third. Each tier is strictly narrower than the last, so the expensive algorithm only ever sees a handful of candidates. +**What:** a data-driven, bidirectional abbreviation table + folding + glued-word splitting, applied to **both** the query and every entry alias at index time. -**When to use:** Always — this is what makes 10 ms of Workers CPU comfortable rather than tight. +**When to use:** always, before any scoring. Normalization must be *symmetric* — "Jln Ampang" and "Jalan Ampang" must produce the **same** normalized string, which requires expanding *both* directions to a canonical form. +**Example (the canonical form is the long form):** ```typescript -// Source: algorithm properties cited in §Sources (Cohen/Winkler 2003 for Jaro-Winkler -// prefix weighting; RapidFuzz docs for normalized Damerau-Levenshtein semantics). -// The three-tier ORDER is this research's recommendation, justified by the -// Workers CPU budget (10 ms/request, Cloudflare limits doc). +// shared/src/rules.ts — JUPEM-derived. See §"Malay normalizer" for the full table. +export const ABBREVIATIONS: Record = { + jln: ["jalan"], + tmn: ["taman"], + kg: ["kampung", "kampong"], + kpg: ["kampung", "kampong"], + bkt: ["bukit"], + bt: ["batu"], + sg: ["sungai"], + bdr: ["bandar"], + psrn: ["persiaran"], + lbh: ["lebuh"], + lrg: ["lorong"], + lor: ["lorong"], + spg: ["simpang"], + blk: ["blok"], + h: ["hulu"], + p: ["pulau"], + tg: ["tanjung", "tanjong"], + tlk: ["teluk", "telok"], + g: ["gunung", "gunong"], + gn: ["gunung", "gunong"], + k: ["kuala"], + hr: ["hilir"], + jr: ["jeram"], + lmbh: ["lembah"], + pdg: ["padang"], + pt: ["parit"], + pmtg: ["permatang"], + plbh: ["pelabuhan"], + kep: ["kepulauan"], + ldg: ["ladang"], + lpg: ["lapangan"], + gsg: ["gosong"], + bgn: ["bangunan"], + alr: ["alur"], + bak: ["bakau"], + bdh: ["berhad"], + hj: ["haji"], + hjh: ["hajjah"], + abd: ["abdul"], + kl: ["kuala lumpur"], + pj: ["petaling jaya"], + jb: ["johor bahru"], + kb: ["kota bahru"], + kk: ["kota kinabalu"], +}; +``` -/** Tier 1 — O(1) hash lookup on the folded alias table. */ -function exactLookup(index: Map, q: string): GazetteerEntry[] { - return index.get(q) ?? []; -} +### Pattern 3: Layered scoring with an exact-match floor -/** Tier 2 — character-bigram Dice coefficient as a cheap relevance filter. - * Dice = 2·|A∩B| / (|A|+|B|) over bigram multisets. O(len) per candidate. */ -function diceBigrams(a: string, b: string): number { - if (a === b) return 1; - if (a.length < 2 || b.length < 2) return a === b ? 1 : 0; - const grams = (s: string): Map => { - const m = new Map(); - for (let i = 0; i < s.length - 1; i++) { - const g = s.slice(i, i + 2); - m.set(g, (m.get(g) ?? 0) + 1); - } - return m; - }; - const ga = grams(a); - const gb = grams(b); - let shared = 0; - for (const [g, n] of ga) { - const other = gb.get(g); - if (other !== undefined) shared += Math.min(n, other); +**What:** score candidates in **tiers**, each tier's score range strictly above the previous, so an exact alias hit can never be outranked by a fuzzy hit. + +**When to use:** always. This is the mechanism that makes D-13's "top-3 contains the expected id" measurable and stable. + +**Example:** +```typescript +// shared/src/score.ts — tier ranges are disjoint; see §"Common Pitfalls" #4. +export const TIER = { + EXACT: 1.00, // normalized query === normalized alias or name + ALIAS_EXACT: 0.90, // normalized query === normalized alias + PREFIX: 0.75, // one is a prefix of the other, ≥3 chars + TOKEN_SET: 0.65, // all query tokens present in the candidate's token set + FUZZY: 0.00, // 0..0.60 — Damerau similarity, scaled +} as const; + +export function scoreOne(queryNorm: string, aliasNorm: string): number { + if (queryNorm === aliasNorm) return TIER.EXACT; + if (queryNorm.length < 3 || aliasNorm.length < 3) return 0; + const d = damerau(queryNorm, aliasNorm); + const maxLen = Math.max(queryNorm.length, aliasNorm.length); + const sim = 1 - d / maxLen; + if (queryNorm.startsWith(aliasNorm) || aliasNorm.startsWith(queryNorm)) { + return TIER.PREFIX; } - return (2 * shared) / (a.length - 1 + b.length - 1); -} - -/** Tier 3 — Damerau-Levenshtein (optimal string alignment variant is enough). - * Counts a transposition as ONE edit, which D-13's typo set requires. */ -function damerau(a: string, b: string): number { - const al = a.length; - const bl = b.length; - const d: number[][] = Array.from({ length: al + 1 }, () => new Array(bl + 1).fill(0)); - for (let i = 0; i <= al; i++) d[i][0] = i; - for (let j = 0; j <= bl; j++) d[0][j] = j; - for (let i = 1; i <= al; i++) { - for (let j = 1; j <= bl; j++) { - const cost = a[i - 1] === b[j - 1] ? 0 : 1; - d[i][j] = Math.min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost); - if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) { - d[i][j] = Math.min(d[i][j], d[i - 2][j - 2] + 1); // transposition - } - } - } - return d[al][bl]; + if (tokenSubset(queryNorm, aliasNorm)) return TIER.TOKEN_SET; + return sim * TIER.FUZZY; // 0 .. 0.60 } ``` -**The cascade, in order:** -1. **Tier 1 — exact.** Look up the normalized query in a `Map` built at load time from every name + alias + its normalized form. A hit is a *perfect* match; score 1.0; these always sort above anything fuzzy. -2. **Tier 2 — bigram Dice filter.** Over the remaining entries, keep only those whose best alias scores above a low Dice floor. This is O(entries × aliases) but each comparison is O(len) — cheap. -3. **Tier 3 — Damerau-Levenshtein.** On the handful that survived tier 2, compute the normalized Damerau distance and convert to a 0..1 similarity. **Only apply tier 3 to strings of length ≥ 4** — below that, a single edit is a large fraction of the string and the scores are noise. (This is the same insight the taxify fuzzy-matching vignette reports: *"Jaro-Winkler can help with very short names (3-5 characters), where a single edit produces a large normalized Damerau-Levenshtein distance."*) - -**Why this order, and not Jaro-Winkler as the primary:** D-09 makes Damerau the residual net. Jaro-Winkler is genuinely better on short strings (its prefix weighting is designed for it — Cohen/Winkler 2003: *"The Jaro and Jaro-Winkler metrics seem to be intended primarily for short strings"*), but its prefix bonus is a liability for a gazetteer where many entries share a first token ("Taman …", "Stesen …", "Suria …"): JW would systematically rank all "Taman X" entries above a correct non-Taman match. Bigram Dice has no such prefix bias. **Recommendation: Dice for filtering, Damerau for the final residual, exact-match always wins.** Jaro-Winkler is a reasonable swap for tier 3 if the corpus shows Damerau's normalisation mis-scores the many short acronyms (IJN, PWTC, LRT, KLIA) — that is exactly the kind of question the corpus runner exists to answer empirically. - -### Pattern 3: The Malay normalizer as an editable rule table (D-09) - -**What:** A bidirectional abbreviation table plus folding passes, applied identically to the query and to every alias at index time. Adding a new abbreviation is a **table edit**, not a code change. - -**When to use:** Always — this is D-09's explicit architecture requirement. - -```typescript -// Source: rule table content grounded in the UPU S42 Malaysian address standard -// (https://www.upu.int/UPU/media/upu/PostalEntitiesFiles/s42/mysEn.pdf), which -// lists the canonical "primary thoroughfare type" tokens, plus the OSM KL road -// list (https://en.wikipedia.org/wiki/List_of_roads_in_Kuala_Lumpur) which shows -// the "Jalan"/"Lorong"/"Persiaran"/"Lebuh" naming in KL, plus the Pos Malaysia -// /MaCGDI address examples which use "Persint 4" and "Taman". - -/** Bidirectional: [from, to]. Both directions are added automatically, so the - * table holds ONE line per concept. */ -export const ABBREVIATIONS: ReadonlyArray = [ - // ── thoroughfare types ── - ["jln", "jalan"], - ["jl", "jalan"], // careful: "jl" is also used in place names — see Pitfall 6 - ["jl.", "jalan"], - ["lnr", "lorong"], - ["lr", "lorong"], - ["psn", "persiaran"], - ["persiaran", "persiaran"], // identity guard for the family below - ["ps", "persiaran"], - ["lbh", "lebuh"], - ["bkt", "bukit"], - ["tmn", "taman"], - ["tm", "taman"], - ["kg", "kampung"], - ["kmpg", "kampung"], - ["kampung", "kampung"], // identity guard - ["presint", "presint"], - ["prsn", "presint"], - ["prsnt", "presint"], - ["presin", "presint"], - // ── transit / civic shorthand (D-08's alias lever) ── - ["stesen", "stesen"], - ["stn", "stesen"], - ["sta", "stesen"], - ["station", "stesen"], - ["hentian", "hentian"], - ["st", "stesen"], // risky — see Pitfall 6 - ["sk", "sekolah"], - ["kg smith", "kampung smith"], // multi-word: see Pitfall 7 -]; -``` - -**The normalizer pipeline** (each step is a separate, individually-testable function so the corpus runner can report *which* rule fired): - -``` -normalize(s): - 1. Unicode NFKD + strip diacritics → "Ampang" stable under "Ampáng" - 2. casefold → "jln ampang" - 3. collapse whitespace → "jln ampang" - 4. strip punctuation (.,/,-) → "jln ampang" - 5. abbreviation table (word-wise) → "jalan ampang" - 6. glued-word split (if applicable)→ "jalan ampang" -``` - -**Rules 5 and 6 are table-driven, not code-driven.** Rule 5 expands a token only when the token is an exact whole-word match of a table key — this avoids the "Sunway" → "Sun Way" disaster that a naive substring replace causes (see Pitfall 5). - -### Pattern 4: Gazetteer entry schema (D-07, D-08, D-06) - -```typescript -// Source: D-07 (typed JSON, V8-native parse, diff-friendly) + D-08 (aliases live -// inside each entry) + D-06 (hand-curated seed + Overpass-proposed top-ups). -// Fields beyond the required five are OpenCode's discretion per CONTEXT.md. - -export interface GazetteerEntry { - /** Stable, URL-safe, and the payload of the D-11 deep link. - * Must match /^[a-z0-9-]+$/ — the Telegram start_param character set allows - * only A-Z a-z 0-9 _ and -, so the id MUST NOT contain a dot or underscore - * if it is ever to travel through a start_param. */ - id: string; - /** Canonical display name, as a local would write it. */ - name: string; - /** Street address for the venue card. May be a landmark address ("Jalan Ampang") - * when no street number exists. */ - address: string; - lat: number; - lng: number; - /** Coarse class for the venue card's four-square type and for alternates - * labelling: "mall" | "transit" | "hospital" | "government" | "park" | - * "landmark" | "kampung" | "education" | "religious" */ - category: string; - /** Locals' shorthand. THE main corpus-quality lever (D-08). */ - aliases: string[]; - /** Optional: the OSM element this entry was curated from, for the Overpass - * top-up script to reconcile against. Not required for search. */ - osm?: { type: "node" | "way" | "relation"; id: number }; -} -``` - -**The JSON file lives at `bot/src/search/data/kl-gazetteer.json`** and is imported with a plain `import ... with { type: "json" }` (or a `readFileSync` in the Node-side script). It is a **plain committed file** — the D-04 forkability law. - -### Pattern 5: Sharing the gazetteer between the bot and the webview (D-11) - -**What:** The webview must resolve `dest_` to a destination using "its own copy of the same gazetteer module (shared module, build-time)" (D-11). - -**Why this matters:** If the two copies drift, a deep link points at a destination the webview does not recognise — the exact "stale-webview dataset" scenario D-11 acknowledges. The mitigation must be **structural**, not procedural. - -**Recommendation — one source file, two import paths.** Put the canonical module in a place both workspaces can import. Three options, in order of preference: - -| Option | Mechanism | Verdict | -|--------|-----------|---------| -| **A. A shared workspace package** (e.g. a new `search/` npm workspace with the gazetteer + normalizer + scorer) | `bot/` and `web/` both depend on `@araharah/search` | **Best.** One copy on disk; the build can never drift. Cost: one more workspace entry in the root `package.json`. Consistent with the repo's existing workspace pattern (`theme/` is already consumed by `web/` as a build-time dependency). | -| B. A file symlink / duplicate in both trees | `web/src/search/gazetteer.ts` re-exports from `../../bot/src/search/…` | Works with Vite (needs `server.fs.allow` for dev) but is brittle and reads badly. | -| C. A build step that copies `bot/src/search/data/*.json` into `web/` | A `prebuild` script | **Reject.** It reintroduces exactly the "token duplication" anti-pattern the repo's ARCHITECTURE.md already flags for the mockup Tailwind configs. | - -**Option A is the recommendation** and it also makes the corpus runner able to import the *identical* module the production paths use — the property D-14's bake-off depends on. +**Why the tiering matters:** Jaro-Winkler's prefix bonus and Damerau's normalized distance can both rank a *fuzzy* match above an *exact* one on short strings. Disjoint score bands make the ordering a property of the *tier*, not of the arithmetic — so the corpus results are stable when the fuzzy function is tuned (D-09's "extends by table edit, not code change" only works if the tiers don't move). ### Anti-Patterns to Avoid -- **Substring replacement in the normalizer.** `s.replace("tmn", "taman")` turns "Sunway Pyramid" into "Sunaan Piyramid". Always expand **whole tokens** only, and only on exact table keys. -- **A fuzzy score that can outrank an exact alias match.** If "Kuala Lumpur City Centre" scores higher than "KLCC" for the query "KLCC", the top-3 gate passes but the product is wrong. Exact/normalized exact MUST dominate the ranking — enforce it by tier, not by a weight. -- **Building the index at module top level in the Worker.** Cloudflare's docs warn explicitly: *"avoid expensive work in global scope… a Worker must parse and execute its global scope within 1 second."* Parse the gazetteer **inside the handler** (or memoize lazily in module scope, populated on first use). -- **Logging the query text.** D-15 forbids it, and the repo's D-14 law is category-only logging. The corpus runner may print queries (it is a developer tool running locally on a committed file) but the **Worker** must never do so. -- **An N×N scan of the whole gazetteer with Damerau-Levenshtein on every alias.** 500 entries × 8 aliases × O(len²) per comparison is the CPU-budget killer. The three-tier cascade is the fix. -- **A "no results" answer that still shows a pin.** D-14 requires honest refusal for out-of-KL queries. If the gazetteer has no match, the bot must say so and offer nothing — never the nearest entry. -- **Making `startapp` a `web_app` button's field.** It isn't one. `WebAppInfo` has exactly one field: `url`. +- **Bare `===` matching against names.** Fails every abbreviation and typo case; D-09 exists to prevent this. +- **A single flattened `Map` of all aliases.** It cannot rank alternates, cannot express "the canonical name outranks an alias", and loses the `matchedAlias` provenance the corpus report needs. +- **Normalizing only the query.** Normalizing only one side means "Jln Ampang" and "Jalan Ampang" land in different buckets. D-09 explicitly requires both sides. +- **Fuzzy-scoring the whole string when the error is in one word.** "Tmn Bangsar" vs "Taman Bangsar" is a *whole-string* one-edit match, but "Jln Tun Razak" vs "Jalan Ampang" is two different roads — token-set matching before fuzzy prevents cross-street confusion. +- **Reading the gazetteer JSON from the filesystem at Worker runtime.** Cloudflare Workers have no `fs`; the JSON must be a **build-time import** (`import data from "...json" with { type: "json" }`) so wrangler/esbuild inlines it into the bundle. The corpus runner (Node) reads it with `fs` — the shared module must expose the *entries*, not do the reading, so both consumers get their data the way their runtime allows. +- **An Overpass query with no bbox.** A global Overpass query will time out and is a well-known way to get rate-limited. Always bound to the KL bbox. +- **An `editMessageText` on the venue message.** A venue message carries a `venue` payload; `editMessageText` replaces it with text and the card is destroyed. Use `editMessageReplyMarkup` (documented as allowed when the message already has an inline keyboard) or send a new `sendVenue`. +- **Persisting the raw query.** D-15: category-only logging. A query is user payload. This is the single easiest way for this phase to break the project's privacy promise — and there is already a grep gate in `web/test/no-storage.test.mjs` for the web side. --- @@ -490,412 +413,631 @@ export interface GazetteerEntry { | Problem | Don't Build | Use Instead | Why | |---------|-------------|-------------|-----| -| Malay abbreviation expansion | A hand-coded `if (s.startsWith("jln"))` chain | A **table** (D-09) | Locals invent shorthand faster than code changes ship. A table is a data edit a maintainer can review in a PR. | -| Fuzzy string distance | A bespoke "vowel-drop-aware" similarity | **Damerau-Levenshtein (OSA)** with a length floor | Damerau is a textbook algorithm with a 40-line implementation; getting transposition semantics right (D-13 requires them) is the only reason to write it. | -| Bigram similarity | A custom n-gram counter | The Dice coefficient over bigram multisets | Two counters and one division. A library would be a dependency for 15 lines. | -| Telegram venue cards, inline keyboards, deep links | Hand-rolled Bot API HTTP calls | **grammy 1.46.0** (`ctx.sendVenue`, `InlineKeyboardButton`) | Already a dependency; the typings are verified to support `reply_markup` on `sendVenue`. Hand-rolling would add a signing/marshalling surface for no gain. | -| Telegram initData validation for `start_param` | Re-validating the HMAC in the webview | **The existing `POST /api/session`** (Phase 2) | The webview already posts its raw `initData` to the Worker and gets a session token. `start_param` is *inside* the signed `initData`, so the same validation covers it. **Do not build a second validation path.** | -| OSM data extraction | Hand-parsing Overpass JSON into gazetteer rows | The **Overpass QL query** + `out center;` + a thin mapper | Overpass returns tagged elements; the mapping from `name`/`addr:*`/`center` to a `GazetteerEntry` is ~30 lines. | -| A geocoding index | SQLite/sql.js WASM in the Worker bundle | **Plain JSON** (D-07) | D-07's own reasoning: at 200–500 rows SQLite-in-Worker is ~1.5 MB of bundle plus cold-start cost, against a JSON file that is a few tens of KB and diffs cleanly. | -| A fuzzy-search library | `fuse.js` / `fastest-levenshtein` | The **three-tier cascade** in §Pattern 2 | See §Alternatives Considered. The hand-rolled cascade is ~150 lines, table-driven, dependency-free (D-04), and is the thing D-09's architecture describes. | -| Web-hook / deep-link token signing | HMAC over the destination id | **No crypto** (D-11) | The payload's only threat is a user pranking themselves. A plain id into a public gazetteer needs no secret. | +| **Bot API transport, keyboards, callback handling** | Raw `fetch` to `api.telegram.org` | **grammy 1.46.0** | Already installed. `InlineKeyboard.webApp()` / `.text()` builders, `bot.callbackQuery(regex, …)` trigger matching, `ctx.answerCallbackQuery()` — all first-class. Hand-rolling risks the 30-second callback-answer deadline. | +| **The Malay abbreviation table** | An invented list from memory | **The JUPEM table** (via Nominatim `variants-ms.yaml`), vendored with attribution | The official table is normative, already transcribed, and machine-readable. Inventing one loses `Psrn`, `Lbh.Raya`, `Bdr`, `Kondo`, `Blk`, `Hosp`, `Disp` and the older `Kampong`/`Tanjong`/`Telok`/`Gunong` spellings — all real KL usage. §"Malay normalizer" reproduces it. | +| **OSM data retrieval at scale** | Parsing a `.pbf` in the Worker | **Overpass API** (repo script) or **Geofabrik extract** (repo script) | Neither belongs in the Worker. D-06 says the script is a maintainer tool. | +| **The Mini App ↔ Telegram bridge** | `telegram-web-app.js` vendored into the repo | **The official script** loaded by the existing `web/index.html` | Phase 3 already ships this arrangement. | +| **Home-screen install prompt** | A custom PWA install flow | **`Telegram.WebApp.addToHomeScreen()`** | Telegram owns the shortcut; the browser's `beforeinstallprompt` does not exist inside the Telegram webview and would create a *second* icon that Telegram doesn't know about. | +| **A search index / trigram structure** | A hand-built inverted index in the Worker | **Linear scan over ~2500 (entry × alias) pairs** | At 500 entries × 5 aliases, a linear scan with an exact-match fast path is microseconds. A trigram index is backend #2's job (D-05) and would be pure cost in the Worker bundle (D-07). | +| **Session / HMAC validation** | Re-deriving initData validation | **The existing `bot/src/security/initdata.ts`** | Phase 2's HMAC path is tested and hardened (T-02-01). The destination id does not need its own crypto (D-11). | +| **Routing / Photon / Valhalla** | Anything in this phase | **Deferred to Phase 5** | D-03 explicitly defers VPS provisioning. Do not research or build it here. | -**Key insight:** In this phase the *hard* part is not the algorithm — it is the **data** (locals' shorthand) and the **discipline** (one module, one copy, one corpus). Every temptation to add a library or a service is a temptation to move the quality decision out of a table a maintainer can edit and into a black box they cannot. +**Key insight — the one place hand-rolling IS correct:** the Damerau-Levenshtein distance. At this corpus size the comparison count is tiny, the algorithm is 20 lines of standard OSA dynamic programming, and a dependency would add a package to a Worker bundle that D-07 deliberately wants minimal. Hand-rolling it *inside the shared module* keeps the whole search stack dependency-free and V8-native, which is exactly D-07's rationale. **Exception:** if the plan prefers a tested implementation, `@crob/damerau-levenshtein@2.0.0` is 1.2 kB gzipped, TypeScript-first, and Node 20.19+ compatible — a safe swap. --- ## Runtime State Inventory -> This phase **creates** state but does not rename or migrate any. The inventory below answers the canonical question for the *new* state the phase introduces, since D-04/D-06/D-07 make the gazetteer a committed artifact and D-15 permits KV caching. +> **Not applicable.** Phase 4 is additive: no rename, no rebrand, no string replacement, no migration. The gazetteer is a **new** committed file; there is no pre-existing state to inventory. *(Per the format spec, this section is included only for rename/refactor/migration phases and is omitted otherwise.)* -| Category | Items Found | Action Required | -|----------|-------------|------------------| -| **Stored data (new)** | `bot/src/search/data/kl-gazetteer.json` — the curated KL gazetteer, 200–500 rows. Derived from OSM (names, coordinates, addresses) → **ODbL-1.0 obligation** (see §Licensing). | Data file is created this phase. Ship it with the ODbL header + attribution, and record the licence in the file's first lines (a JSON field like `"_license": "ODbL-1.0"` plus a sibling `NOTICE`/`LICENSES.md` entry). | -| **Stored data (new)** | `bot/test/fixtures/golden-corpus.json` — the pinned corpus (D-13). Not OSM-derived (it is *queries* and expected ids). | Created this phase. Plain committed file, GPL-3.0 with the repo (it contains no OSM content). | -| **Live service config** | None. The `SearchSource` is in-Worker; no external service is added (D-03). The Worker already has `BOT_TOKEN`, `WEBHOOK_SECRET`, `chat_sessions` in `bot/wrangler.jsonc`. | None. | -| **OS-registered state** | None. No systemd unit, no launchd plist, no Task Scheduler entry. The bot's Telegram menu (registered via `setMyCommands`) is a *live service config* surface, not OS state — it must be re-registered after `/cari` is added to `helpMenu` (the existing `npm run bot:commands` script covers this). | Re-run `npm run bot:commands` after the copy change; the Phase-2 CS5 gate asserts no drift. | -| **Secrets / env vars** | None new. D-04's "env only" law is already true; the search backend needs no credentials (backend #3 Photon would, but it is deferred to Phase 5). | None. | -| **Build artifacts** | `web/dist` / `web/dist-harness` — the webview's copy of the gazetteer is bundled at build time. If the gazetteer changes and the web bundle is not rebuilt, the webview resolves stale ids. | The `npm run deploy` chain already builds theme → web → wrangler deploy, so the artifact refreshes on every deploy. **The Phase-3 build-artifact gate (`web/test/build-artifact.test.mjs`) is the place to add a check that the bundled gazetteer is present and id-complete.** | -| **Runtime (ephemeral)** | The resolved destination lives in **memory only** in the webview — no localStorage, no sessionStorage, no IndexedDB, no cookie (GEO-03's law continues to bind; the Phase-3 `web/test/no-storage.test.mjs` grep gate will catch a violation). | Grep gate is automatic once the code lands. | +**However**, there is one adjacent runtime-state question the planner should answer explicitly: + +| Category | Question | Answer for Phase 4 | +|----------|----------|--------------------| +| **Stored data** | Does any existing datastore hold a place-name or search string that this phase changes? | **No.** The only KV namespace is `chat_sessions` (sessions + location pins, TTL-only). The gazetteer is a new repo file, not a KV namespace. | +| **Live service config** | Does any external service config reference place names or search endpoints? | **No.** The only live config is `bot/wrangler.jsonc` (KV namespace, assets binding) and `bot/.dev.vars` (secrets). **One new config item IS likely:** the Mini App's public URL, needed to build the `web_app` button's `WebAppInfo.url` and the `t.me?startapp=` deep link. Today the Worker is on `..workers.dev` (`workers_dev: true`) and `WEBHOOK_URL` holds the origin. **The plan must decide how the bot learns the Mini App URL** — a new env var (e.g. `MINI_APP_URL`) read from `bot/.dev.vars` + wrangler secret, or derived from `WEBHOOK_URL`. This is a real Phase 4 deliverable and it is **not** in the D-list, so it falls in OpenCode's Discretion. | +| **OS-registered state** | Any OS-level registration embedding the changed strings? | **No.** No rename. The PWA home-screen shortcut is created by the *user* on their device at runtime; nothing is registered on the developer's machine. | +| **Secrets / env vars** | Any secret key named after a place name or search? | **No.** `BOT_TOKEN`, `WEBHOOK_SECRET`, `WEBHOOK_URL` are unaffected. If a `MINI_APP_URL` is added, it is **not a secret** — it is a public origin. It may live in `wrangler.jsonc` `vars` rather than as a secret. Flag for the planner. | +| **Build artifacts** | Stale artifacts that a source rename would leave behind? | **No rename.** But note: `web/dist` and `web/dist-harness` are build outputs; if the shared module is added, the **Vite and wrangler builds must both pick it up** and the `web/test/build-artifact.test.mjs` gate may need a new assertion (e.g. the gazetteer JSON *must* be in `web/dist` for the webview copy to work). Check that gate's scope. | + +--- + +## Deep-link risk — choosing the carrier for the destination payload (ONBD-02) + +**This is the single highest-risk decision in Phase 4.** D-11 says the deep link "carries a plain gazetteer entry id (`?startapp=dest_`)". The *payload* is settled. The *carrier* is not — and D-11's wording mixes two distinct Telegram launch surfaces. + +### What the official docs actually say + +Three separate statements, from the same page, that do not obviously agree: + +| Source | Statement | +|--------|-----------| +| `core.telegram.org/bots/webapps` → *Launching the main Mini App* | *"A bot's main Mini App can also be opened in the current chat by direct link in the format `https://t.me/botusername?startapp`. If a non-empty `startapp` parameter is included in the link, it will be passed to the Mini App in the `start_param` field and in the GET parameter `tgWebAppStartParam`."* | +| `core.telegram.org/bots/webapps` → *Direct Link Mini Apps* | Same mechanism for `https://t.me/botusername/appname?startapp=command`. | +| `core.telegram.org/bots/webapps` → `WebAppInitData.start_param` | *"The value of the `startattach` parameter, passed via link. **Only returned for Mini Apps when launched from the attachment menu via link.**"* — note: this row says `startattach`, not `startapp`, and scopes it to the attachment menu. | +| `core.telegram.org/bots/webapps` → *Inline Button Mini Apps* | *"the Mini App will open with the URL specified in the button"* — **no mention of `start_param` at all.** | +| `core.telegram.org/bots/api/types/webappinfo.md` | *"An HTTPS URL of a Web App to be opened with **additional data**"* — the URL is arbitrary. | + +**The gap:** the launch-parameter docs (`docs.telegram-mini-apps.com/platform/start-parameter`) list exactly two carriers for a start parameter — `?startattach=ABC` on a bot link, and `?startapp=ABC` on a **Direct Link**. A `web_app` **inline keyboard button** is neither. Its `WebAppInfo.url` is simply "the URL specified in the button", and Telegram passes it to `messages.requestWebView` verbatim (MTProto schema: `keyboardButtonWebView` → *"passing `url` to messages.requestWebView.url`"*). + +**Consequence:** if the plan builds the OPEN button as `web_app: { url: "/?startapp=dest_klcc" }` and the webview reads `Telegram.WebApp.initDataUnsafe.start_param`, **`start_param` will most likely be `undefined`** — the button opens the Mini App with no destination. ONBD-02 fails silently. + +### The three carriers, ranked + +| Carrier | Mechanism | Evidence | Risk | UX cost | +|---------|-----------|----------|------|----------| +| **A — `web_app` button with a URL query param** | Button URL = `https://app.example.com/?dest=dest_klcc`; the webview reads `location.search` | `WebAppInfo.url` is documented as an arbitrary HTTPS URL; MTProto passes it verbatim; community reports confirm query params survive ("passing query params to telegram web app just works"). **Not** in the official Telegram docs. | **MEDIUM** — the behaviour is real but undocumented, so it is not a contract Telegram will keep. | None. One tap, in-app, no interstitial. | +| **B — `url` button to `https://t.me/?startapp=dest_`** | The officially-documented `startapp` mechanism | Explicitly documented for both main-Mini-App and direct links. `start_param` + `tgWebAppStartParam`. | **LOW** for the mechanism; **MEDIUM** for the `initData.start_param` wording (§"A2"). | A Telegram interstitial: the tap leaves the chat, opens the app. Also requires @BotFather configuration (Main Mini App, or a Direct Mini App with a `short_name`). And a `url` button is **not** a `web_app` button — D-10's wording changes. | +| **C — both, purpose-split** | A for the inline button (ONBD-02); B for the home-screen shortcut (ONBD-03) | The home-screen shortcut opens the **Main Mini App** link — which is exactly where `startapp` is officially supported. | **LOW** overall if the payload reader accepts both sources. | Carries A's cost on the button; B's cost only on the home-screen path (where an interstitial is invisible — the user tapped an icon). | + +### Recommendation + +**Ship Carrier C, with A as the inline-button mechanism and B as the home-screen mechanism.** Concretely: + +1. The **payload reader** in the webview accepts **both** `location.search`'s `dest` and `initData`'s `start_param`, with the same validation (`/^[A-Za-z0-9_-]{1,64}$/`). One reader, two carriers — the phase cannot fail because Telegram changed one surface. +2. The **button builder** in the bot is a single function, `openButtonFor(entry)`, so swapping carriers is a one-line change if the device test fails. +3. The **device validation** at the exit gate must explicitly test: (a) tap OPEN from the chat → destination preloaded; (b) launch from the home-screen icon → app opens search-ready (ONBD-03's requirement is "lands in the app ready to search", so a preloaded destination is *not* required there — but the icon must open the app at all). + +### What the mock harness must gain + +`web/src/mock/telegram-mock.ts` currently emits `initData` as `user=…&auth_date=1&query_id=h&hash=harness` (`buildMockInitData`, lines 136-140) — **no `start_param` field, and no way to simulate a URL query param** (the mock is installed into `window`, not a URL). The Phase 4 plan must extend the harness with: +- a `startParam` option in `MockOptions` that appends `start_param=…` to the mock initData (for Carrier B); +- a `?dest=` runner-page profile that sets `window.location.search` before the app boots (for Carrier A). + +Without these, the destination-preload path has **no automated test** — it would be validated only by hand on a device. + +### The build-time constraint (easy to miss) + +The `start_param` charset is restricted to **`[A-Za-z0-9_-]`**, max **512 characters** — validated with `/^[\w-]{0,512}$/`. A gazetteer id like `suria-klcc` already complies. But **any id that would need a base64url encoding (e.g. one containing a dot or a slash) must be slugged** — which is a good reason for D-07's ids to be slug-shaped from the start. The corpus runner should assert every corpus `expectId` matches `/^[A-Za-z0-9_-]{1,64}$/`. + +--- + +## ONBD-03 feasibility — what "add to home screen" actually means inside Telegram + +D-12 scopes ONBD-03 to "minimal PWA hooks — `addToHomeScreen` prompt handling + a startapp-less boot landing in the search-ready map". The research confirms this scope is right, and surfaces three constraints the plan must encode. + +### The API is Bot API 8.0+ only + +`addToHomeScreen()` and `checkHomeScreenStatus()` were added in **Bot API 8.0 (17 Nov 2024)**. They do not exist in older clients. The project's `TelegramWebAppLike` interface (`web/src/telegram/webapp.ts`) already carries `isVersionAtLeast(version: string): boolean` and the mock already implements `versionAtLeast` (`web/src/mock/telegram-mock.ts:110-121`) — so the capability guard has a precedent and a testable seam. + +```typescript +// Extend the existing hand-rolled contract — do NOT add @types/telegram-web-app +// (Phase 3 dropped it deliberately; every fake would have to stub ~50 members). +export interface TelegramWebAppLike { + // … existing members … + addToHomeScreen?(): void; + checkHomeScreenStatus?(cb: (status: HomeScreenStatus) => void): void; +} + +export type HomeScreenStatus = + | "unsupported" // the feature is not supported here + | "unknown" // supported, but whether the icon is installed is indeterminate + | "added" // already on the home screen + | "missed"; // not added yet +``` + +**The guard:** `if (wa && typeof wa.addToHomeScreen === "function" && wa.isVersionAtLeast("8.0"))`. Both conditions — the method may be absent on older clients, and the version check is what the platform itself documents. + +### iOS cannot report installation status — treat "unknown" as normal + +`checkHomeScreenStatus()` **always returns `"unknown"` on Telegram for iOS**, regardless of whether the Mini App is actually installed. This is a documented platform limitation, not a bug: + +> Telegram-iOS issue #1729 (*"Home screen status detection always returns 'unknown' in Telegram iOS Mini Apps"*, April 2025) was closed with the maintainer comment: *"On iOS it is not possible to determine whether a Mini App has been added to the Home Screen. The platform does not provide any API that allows Telegram to check whether a Home Screen shortcut exists."* + +**Design consequence:** the app must **not** branch on the status string in a way that treats `"unknown"` as an error or as "not added". The honest behaviour is: offer the prompt when the platform supports it; do not nag; do not claim the icon is present unless the status is literally `"added"`. + +### What the home-screen shortcut actually opens + +The shortcut opens the **Main Mini App** — which is the surface where `?startapp=` is officially supported (§"Deep-link risk", Carrier B). This is convenient: **ONBD-03's "lands in the app ready to search" needs no payload at all** (D-12 says "startapp-less boot"), and the shortcut naturally lands on the search-ready map. The `startapp` capability on that surface is a free bonus for a future phase (e.g. a per-destination home-screen shortcut). + +**A separate, larger PWA manifest / service worker / offline story is explicitly deferred** (D-12, and CONTEXT's Deferred Ideas). Do not add a `manifest.webmanifest`, a service worker, or an install-prompt `beforeinstallprompt` handler — the browser API does not exist in the Telegram webview, and a second icon the Telegram client does not know about is worse than none. `addToHomeScreen()` is the *only* correct mechanism here. + +### What the plan should build + +1. Extend `TelegramWebAppLike` with the two optional methods (above). +2. A single, small "add to home screen" affordance — where it lives is a UI question, but the D-12 scope is deliberately minimal, and Phase 4's `UI hint: no` in the roadmap means **no new UI-SPEC round**. The least invasive placement is a card or a control the Phase 3 shell already has. **Flag: if this needs new UI furniture, it may need a UI-SPEC round — the planner should confirm it fits inside the existing Phase 3 shell.** +3. A capability-guard unit test (`web/test/homescreen.test.mjs`) covering: method absent → no-op; `isVersionAtLeast("8.0")` false → no-op; status `"unknown"` → treated as indeterminate, not as an error; status `"added"` → the affordance is hidden or shows "already added". +4. Extend the mock harness (`MockOptions`) with a `homeScreenStatus` option so the three branches are testable without a device. --- ## Common Pitfalls -### Pitfall 1: `web_app` inline buttons have no `start_param` field +### Pitfall 1: Assuming `start_param` arrives from a `web_app` inline button (THE phase's biggest trap) -**What goes wrong:** The team writes `web_app: { url: origin, start_param: "dest_x" }` — TypeScript rejects it, or it is cast and silently dropped. The Mini App opens with no destination and the user must re-type. ONBD-02 fails. +**What goes wrong:** the plan builds the OPEN button as `web_app: { url: "https://app.example.com/?startapp=dest_klcc" }` and the webview reads `Telegram.WebApp.initDataUnsafe.start_param` — which is **undefined** for inline-button launches. The user taps OPEN, the Mini App opens, and the destination is not preloaded. ONBD-02 fails silently at the worst moment. -**Why it happens:** `WebAppInfo` has exactly one field (`url`). The `startapp` parameter belongs to the **`t.me` deep-link** form, not to the button payload. +**Why it happens:** the official `WebAppInitData.start_param` documentation is scoped to `startattach` / attachment-menu launches and to *direct links*. The `web_app` inline button is a different launch surface. The two are easy to conflate because both end in "the Mini App has a start param". -**How to avoid:** Use a **`url` inline button** (not `web_app`) pointing at `/?dest=`; the webview reads `location.search`. Read `initData`'s `start_param` as a **secondary** path for the case where the Mini App was opened by a `t.me?startapp=` link. +**How to avoid:** treat the carrier as an explicit, tested choice with a fallback. §"Deep-link risk" ranks three carriers: -**Warning signs:** The OPEN button opens the Mini App with no destination preloaded; the webview's `location.search` is empty. +1. **Carrier A — `web_app` button whose URL carries the payload as a plain query param** (`https://app.example.com/?dest=dest_klcc`). The button's URL is an arbitrary HTTPS URL; Telegram opens it in the Mini App webview. Community reports confirm query params on that URL survive into the webview. **The webview reads `location.search`, not `start_param`.** Lowest moving-parts risk, but it is *not* what D-11 says (`?startapp=dest_`). +2. **Carrier B — `url` button to `https://t.me/?startapp=dest_`** (the documented `startapp` mechanism, and the exact mechanism D-11 names). Trade-off: the `url` button opens a **Telegram deep link**, not a Mini App button — so it leaves the chat, opens the app, and the "one tap to proceed" (D-02) becomes one tap + a Telegram interstitial. It also requires the bot to be configured as a Main Mini App in @BotFather (or a Direct Mini App with a `short_name`). +3. **Carrier C — both**: a `web_app` button carrying the id in the URL query (Carrier A) *and* the `startapp` direct-link path (Carrier B) for the home-screen entry point (ONBD-03), where `startapp` is officially available. -### Pitfall 2: Telegram Desktop requires a second tap on a `t.me?startapp=` deep link +**Warning signs:** `initDataUnsafe.start_param` is `undefined` at runtime; the webview's `location.search` is empty; the Mini App boots into the plain "search-ready map" with no destination. **The Phase 3 mock harness must be extended to simulate each carrier's arrival shape** — the mock's `buildMockInitData` currently emits only `user` + `auth_date` + `query_id` + `hash`, with no `start_param` field at all (`web/src/mock/telegram-mock.ts:136-140`). -**What goes wrong:** On Telegram Desktop (Windows/Linux), tapping OPEN shows a START button the user must press — the "one tap to proceed" promise (D-02) breaks on that client. +### Pitfall 2: `sendVenue` has no `text` field — the "one line confirming the destination" needs a home -**Why it happens:** [tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694): *"the user must manually press the 'START' button for the deeplink to be processed. This does not align with behavior observed on other platforms."* +**What goes wrong:** the plan tries to send a confirming sentence plus the venue card in one message. `sendVenue`'s parameter list is `latitude`, `longitude`, `title`, `address` (+ optional `foursquare_id`, `google_place_id`, …). **There is no `text` parameter.** The confirming line has to be either (a) a separate `sendMessage` before the card, or (b) baked into the venue's `title`/`address`. -**How to avoid:** Use the `url` button form. If a `t.me` deep link is used for some other surface, document the Desktop extra tap. +**Why it happens:** `sendMessage` and `sendVenue` look alike from a distance; the CONTEXT's "venue card message should read like the bot's clinical voice: one line confirming the destination" reads like a single message. -**Warning signs:** The feature works on Android/iOS and stalls on Desktop. +**How to avoid:** decide explicitly. The cleanest reading of D-02 ("the best result ships immediately as the venue card") is: **one `sendVenue`** whose `title` is the entry name and whose `address` is the entry's human address, plus the inline keyboard carrying OPEN + alternates. If a separate confirming line is wanted, it is a second `sendMessage` — but that violates D-02's "no second message round" *only if* it needs a tap (it doesn't; two sequential sends is one round of user interaction, but it is two messages). The plan must pick and the copy gate should reflect it. -### Pitfall 3: `start_param` has a restricted character set +**Warning signs:** the bot's `sendVenue` call silently drops a `text` field; or the venue card's `title` ends up carrying marketing copy instead of the place name. -**What goes wrong:** A gazetteer id containing a dot (`klcc.pavilion`) or a space is passed through `startapp=` and the client drops or truncates it. +### Pitfall 3: Abbreviating a word that is part of a proper name -**Why it happens:** The Telegram start parameter allows only `A-Z a-z 0-9 _` and `-`, up to 512 characters; the documented validation regex is `/^[\w-]{0,512}$/`. +**What goes wrong:** the normalizer expands *every* token, so "Jln Tmn Desa" becomes "Jalan Taman Desa" — but the correct address is "Jln Taman Desa" (Jalan is the generic term; Taman Desa is the *name*). MS 2039 §5.2.3.2 is explicit: *"Abbreviation shall not be applied to names given to the geographical feature itself. As an example, an address line 'Jalan Taman Desa' may be abbreviated to 'Jln Taman Desa' but not 'Jln Tmn Desa' or 'Jalan Tmn Desa'."* -**How to avoid:** Constrain gazetteer ids to `/^[a-z0-9-]+$/` — lowercase, digits, hyphens. **This is a schema-level constraint that must be enforced by a test**, not by convention. It also makes ids URL-safe for the `?dest=` query string. +**Why it happens:** the expansion table is applied positionally and unconditionally. -**Warning signs:** The webview receives `start_param` but cannot map it to an entry. +**How to avoid:** the *canonical* form (long form) makes this benign **if the rule is applied symmetrically**: both "Jln Tmn Desa" and "Jln Taman Desa" expand to "Jalan Taman Desa" — because the rule is "expand the abbreviation to the long form", and "Tmn"→"Taman" is a correct expansion of the *word* even though the *address* shouldn't have used it. Expanding both sides to the long form makes them match, which is exactly what SRCH-02 wants. The MS 2039 rule matters for *display* fidelity (what address string the bot shows), not for *matching*. So: **use the long form for matching; keep the curated `address` string verbatim for display.** -### Pitfall 4: Overpass returns HTTP 406 without a User-Agent header +**Warning signs:** the venue card's address shows "Jalan Taman Desa" when the curated entry says "Jln Taman Desa" — the card should show the curated address, not the normalized one. -**What goes wrong:** The top-up script's first fetch returns 406 and the script exits with a parse error; or a well-meaning fix adds a browser UA that gets the script rate-limited. +### Pitfall 4: Fuzzy scoring outranking an exact match -**Why it happens:** **Verified this session**: `curl -s -o /dev/null -w "%{http_code}" https://overpass-api.de/api/status` returned **406** with no User-Agent. The Overpass API rejects requests without an identifying UA. The rate-limit guidance also states: *"Rate limits apply per IP address on all public instances. Including a User-Agent or Referer header identifying your application is strongly recommended as best practice."* +**What goes wrong:** on short strings, Jaro-Winkler's prefix bonus or a normalized Damerau score can push a fuzzy match above an exact one. "KLCC" might rank "KLCC Park" above "Suria KLCC" — or a typo'd query might rank the wrong entry first. -**How to avoid:** The script sets a descriptive `User-Agent` (e.g. `araharah-gazetteer-topup/1.0 (+https://github.com/…/araharah)`) and a single request per run. Do **not** fan out per-entry requests. +**Why it happens:** both metrics are continuous and unbounded relative to each other; the "best" match is decided by arithmetic accident. -**Warning signs:** `406` on the first call; `429` after several runs (the Overpass rate limit is per-IP with 2 concurrent slots; requests queue up to 15 s and beyond that are delayed). +**How to avoid:** the disjoint tier bands in §"Pattern 3". An exact match must score **strictly above** the top of the fuzzy band, always. The corpus runner then reports *per-tier* hit rates, so a regression shows up as "exact-tier hits dropped" rather than a diffuse quality drop. -### Pitfall 5: Substring abbreviation expansion corrupts real names +**Warning signs:** the corpus top-1 hit rate is good but top-3 is worse than expected, or a known exact alias loses to a near-miss. -**What goes wrong:** A naive `replace("tmn","taman")` turns "Sunway Pyramid" into "Sunaan Piyramid"; `replace("kl","kuala lumpur")` turns "KLIA" into "Kualia lumpuria". +### Pitfall 5: Trigram / prefix scoring on very short strings -**Why it happens:** Substring replacement without token boundaries. +**What goes wrong:** "KLCC" is 4 characters → 2 trigrams. "DUKE" → 2 trigrams. Trigram overlap on 2 grams is noise, not signal, and Jaro-Winkler's prefix bonus dominates trivially (both strings share the whole prefix → score ≈ 1.0). -**How to avoid:** Expand only **whole tokens** matched against a table key, and keep a conservative table. Add a normalizer test vector for "Sunway Pyramid" and "KLIA" that asserts they are unchanged. +**Why it happens:** trigram and Jaro-Winkler are both documented as unreliable on short strings. *"Very short strings provide so few trigrams that the overlap score is unreliable"* and *"Jaro-Winkler can over-reward shared prefixes when the discriminating information is later."* -**Warning signs:** A corpus entry stops matching its own name after a table edit. +**How to avoid:** **exact-match-first, fuzzy-last**, and a **minimum length guard** (the code above refuses to score anything under 3 characters). For 4–6 character queries, rely on the alias table (D-08) rather than on fuzzy scoring — "KLCC" and "DUKE" should be **aliases**, not fuzzy hits. This is the single strongest argument for D-08's design. -### Pitfall 6: Over-greedy short abbreviations collide with real tokens +### Pitfall 6: The Overpass top-up script becomes a runtime dependency -**What goes wrong:** Expanding `st` → `stesen` turns "St John" into "stesen John"; expanding `jl` → `jalan` turns "Jln" variants inconsistently; `tm` → `taman` hits names like "Tmn" vs "TM". +**What goes wrong:** the plan wires the Overpass query into the Worker (to "top up" the gazetteer at boot). The Worker now depends on a third-party service with no SLA, on every cold start. -**Why it happens:** Two-letter Malay abbreviations are ambiguous with English words and with name fragments. +**Why it happens:** D-06 says "a repo-script Overpass one-shot that proposes top-up candidates for maintainer approval" — the "for maintainer approval" clause is easy to skim past. -**How to avoid:** (a) Keep the abbreviation table's short keys minimal and **only** apply them as a *whole-token* match; (b) for genuinely risky pairs, prefer adding an explicit **alias** on the entry (D-08's lever) over a global rule; (c) put both directions in the table so the canonical form is stable. +**How to avoid:** the script is `bot/scripts/gazetteer-topup.mjs`, run manually, outputting a **proposal file** that a human diffs and merges. The Worker imports only the committed JSON. -**Warning signs:** A corpus positive regresses after a table edit — the corpus runner is exactly the instrument that catches this. +**Warning signs:** `fetch("https://overpass-api.de/…")` anywhere under `bot/src/` or `web/src/`. -### Pitfall 7: Glued-word splitting produces false splits +### Pitfall 6b: The Overpass script's `User-Agent` is load-bearing, and capacity 502s are normal (verification pass) -**What goes wrong:** "KampungBaru" is correctly split into "kampung baru"; but "SunwayPyramid" → "sunway pyramid" and "TamanDesa" → "taman desa" are guesses, and "KLCC" must not be split at all. +**What goes wrong:** (a) The script omits the `User-Agent` header and every request returns **HTTP 406 Not Acceptable** — which looks like a blocked network or a dead service, and can send the maintainer rewriting a working query. (b) The script treats an HTTP **502 "This Overpass API server is currently at capacity"** as a hard failure, exits non-zero, and (in a CI wiring) turns an optional maintainer tool into a red build. -**Why it happens:** Splitting is only safe when the first segment is a **known prefix token** from a closed vocabulary (the abbreviation table's keys + a small explicit list like `kampung`, `taman`, `jalan`, `lorong`, `persiaran`, `bukit`, `stesen`, `sekolah`, `hospital`, `universiti`, `kolej`). +**Why it happens:** (a) The Overpass instances — and the egress path in front of them — reject requests without an identifying `User-Agent`. **Probed 2026-10-23 on `overpass-api.de`, same host, same paths:** `GET /api/status` no-UA → **406**, with-UA → **200**; `GET /` no-UA → **406**, with-UA → **200**; `POST /api/interpreter` no-UA → **406**, with-UA → **200**. (b) The public instances are free and shared; the Overpass wiki's own policy table warns the main instance *"is overloaded – be mindful of that, do not overconsume resources and do not expect high reliability."* Both a **200** and a **502 capacity** response were observed on `overpass-api.de` **minutes apart** in the same session. -**How to avoid:** Split only on **known** prefixes, and only when the remainder is ≥ 3 characters. "KLCC" has no known prefix → unchanged. "Sunway" is not in the prefix vocabulary → unchanged (so "SunwayPyramid" stays glued — which is acceptable: the D-08 alias list covers it). +**How to avoid:** +1. **Always send a descriptive `User-Agent`** (e.g. `araharah-gazetteer-topup/1.0 (+https://github.com//araharah)`). Without it, *nothing* works — this is the single highest-value line in the script. +2. **Treat 502 / 429 / 504 / timeout as a normal outcome**: print one line — *"Overpass instance busy — retry later. No candidates generated; nothing changed."* — and **exit 0**. The curated file is untouched on every path, so a busy instance is not a failure. +3. **Prefer a narrow query.** Default to one tag family per invocation (`nwr["tourism"="attraction"](bbox)`); a narrow query is far more likely to fit in a free worker slot and returns in seconds. The wide multi-family union stays available behind a flag. +4. **Accept an endpoint override** (env var or `--source=`) so the maintainer can point at a different instance. +5. **The automated test uses a committed fixture**, never the live service (see §Wave 0 Gaps) — so the test is deterministic and the CI bill is zero. -**Warning signs:** A query that used to match stops matching after the splitting rule is tightened or loosened. +**Warning signs:** The script exits non-zero with an HTML or plain-text body instead of JSON; or the same query "worked yesterday" and fails today with no code change. -### Pitfall 8: The gazetteer JSON is imported at module top level in the Worker +### Pitfall 7: The golden corpus becomes a generated artifact -**What goes wrong:** On a cold start the Worker parses the JSON and builds the index in the global scope. Under Workers' **1 s startup limit** and the "avoid expensive work in global scope" guidance, a large gazetteer plus a 4,000-alias index build is a startup risk — and it happens on *every* isolate cold start, even for `/api/session` requests that do not need it. +**What goes wrong:** the corpus file is regenerated from the gazetteer (so every entry gets a synthetic query) — which makes the test tautological: it passes by construction and measures nothing. -**Why it happens:** `import data from "./data/kl-gazetteer.json"` at the top of a module evaluates at import time. +**Why it happens:** "extend by editing one file" (D-13) can be misread as "the file is machine-written". -**How to avoid:** Parse + index **lazily inside the handler**, memoized in module scope. Also confirm the JSON import mechanism works in the Workers bundle (Vite/esbuild supports JSON imports natively; `wrangler`'s bundler does too — verify with `wrangler deploy --dry-run` and check the bundle size, which was 167.97 KiB at Phase 2). +**How to avoid:** the corpus is **hand-written and pinned** — every positive pins an expected `id`, and the corpus contains queries that are *not* derived from any alias in the gazetteer (typos, glued words, out-of-KL negatives). Reviewers extend it by *editing the file*. The corpus file must be `git diff`-able and its content must not be derivable from the gazetteer. -**Warning signs:** `wrangler deploy` bundle size jumps by tens of KB; startup time metrics rise. +**Warning signs:** a corpus entry whose query string exactly equals an alias in the gazetteer, for every entry — that's the tautology signature. -### Pitfall 9: ODbL share-alike is treated as a formality +### Pitfall 8: Sending the query text to a log line -**What goes wrong:** The team ships an OSM-derived gazetteer under GPL-3.0 only and declares the project "GPL-3.0" in the README — a licence incompatibility on the data file, invisible until someone notices. +**What goes wrong:** `console.log("search:", query)` while debugging. D-15 and the Phase 2 D-14 law forbid it — the query is user payload. -**Why it happens:** The OSM wiki's *Legal FAQ* says: *"If you are writing software that uses OSM data, you may choose any license you like, including proprietary."* That sentence is about the **software**. The **database** is a separate obligation with a separate answer. +**Why it avoid:** the Phase 2 codebase's logging convention is *category-only* and every log line in `bot/src/webhook.ts` carries a category, never a payload. Match it: `console.log("[search] rejected: no-match")` — never the query. -**How to avoid:** Dual-licence: **code GPL-3.0, data file ODbL-1.0**, with `© OpenStreetMap contributors` attribution in the gazetteer header, in the bot's strings, and in the README. Add a `LICENSES.md` (or an `OSMDATA-LICENSE` note) explaining the split. This is what the ecosystem does and it costs almost nothing. +**Warning signs:** any log line containing a variable that could hold a query string. The existing gates (`bot/test/*.test.mjs`) grep for structural patterns; **add a gate that greps `bot/src/` for a log statement whose argument mentions the query variable.** -**Warning signs:** A `LICENSE` file exists that mentions only GPL-3.0 while a derived data file ships in the repo. +### Pitfall 9: The webview's gazetteer copy drifting from the bot's -### Pitfall 10: The corpus gate is a report, not an executable check +**What goes wrong:** the Worker ships gazetteer v1; the webview bundle ships v2 (built earlier). A `startapp=dest_klcc` arrives; the webview's `id → entry` lookup misses; the destination silently fails to load. -**What goes wrong:** The corpus is run manually once, the numbers look fine, and the phase "closes the bake-off" — but nothing prevents a later edit from silently dropping the hit rate below 90%. +**Why it happens:** two independent build outputs from one source, deployed at different times. -**Why it happens:** D-14 states a threshold but not a mechanism. +**How to avoid:** D-11 already says "the shared module version" re-maps stale datasets. Concretely: (a) the id set is stable — **ids are never renamed, only added**; (b) the webview, on an unknown id, shows an honest "destination unavailable, search again" state rather than a blank map; (c) a **build-time gate** asserts the bot bundle and the web bundle contain the same id set — or, simpler, that both are built from the same committed JSON in the same `npm run build` (which `npm run build` already does: `theme` → `web` → `bot deploy`). -**How to avoid:** Make the threshold an **assertion in the test suite** (`bot/test/corpus.test.mjs`): compute the top-3 hit rate over positives and assert `>= 0.90`; assert every negative yields **zero** results. The suite then fails loudly on regression and the escalation ladder becomes a concrete, triggerable event. +**Warning signs:** an id in the corpus that resolves in the bot but not in the webview. -**Warning signs:** The corpus file changes without a corresponding test run in the commit. +### Pitfall 10: `callback_data` length and content -### Pitfall 11: `/cari` never appears in the Telegram menu +**What goes wrong:** putting the whole venue name + address in `callback_data`. The limit is **1–64 bytes** (confirmed in the `InlineKeyboardButton` type docs). Also, `callback_data` is visible to anyone who can see the message and is echoed back to the bot in the update. -**What goes wrong:** The command works when typed, but the menu still lists only `/help` and `/privacy` — because the menu is derived from `strings.helpMenu`, not from the registered handlers. +**How to avoid:** `callback_data` carries **only the entry id** — `"alt:"` — and the handler looks the entry up in the gazetteer. Never a name, never a query, never coordinates. This also keeps D-15's privacy posture: the callback payload is a public place id, not user data. -**Why it happens:** Phase 2's D-03 design: `register-commands.mjs` *derives* menu descriptions from `helpMenu` lines. A command that is not in `helpMenu` is not in the menu. +--- -**How to avoid:** Add `/cari` to `strings.helpMenu` (both locales) in the same change that registers the handler, and extend the Phase-2 CS5/CS6 gates to expect it. Also note the Phase-2 finding: the menu is **client-cached**, so `setChatMenuButton` + the `all_private_chats` scope may be needed (the existing `bot/scripts/set-menu-button.mjs` covers this). +## Malay Normalizer — the vendorable rule table -**Warning signs:** The handler exists but the menu does not list it. +**This is the highest-reuse artifact in the research.** D-09 requires a bidirectional abbreviation table that "extends by table edit, not code change". The table below is **not reconstructed from memory** — it is the JUPEM (Jabatan Ukur dan Pemetaan Malaysia) normative abbreviation list, transcribed machine-readably by the OSM Nominatim project in `settings/icu-rules/variants-ms.yaml` (PR #4056, merged 2026-04-02), whose stated primary source is *"JUPEM Toponymic Guidelines — 'Abbreviations of Geographical Terms and Other Words Used in Maps'"*. **Vendor it with attribution; do not author a smaller one.** -### Pitfall 12: The venue card and the buttons land as two messages +### The table, verbatim (shared / Malaysia-specific) -**What goes wrong:** The bot sends the venue with `ctx.replyWithVenue(...)` and then a second `ctx.reply(...)` carrying the inline keyboard — violating D-02's "no second message round". +```yaml +# shared/src/rules.ts — JUPEM-derived, via Nominatim settings/icu-rules/variants-ms.yaml +# Primary source: JUPEM Toponymic Guidelines (Malaysia), 2017 ed. +# Secondary: https://wiki.openstreetmap.org/wiki/Name_finder:Abbreviations#Bahasa_Melayu_-_Malay +# https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations +# Format: " -> " -**Why it happens:** Not knowing that `sendVenue` accepts `reply_markup`. +shared: + Jalan -> Jln + Lorong -> Lrg, Lor + Lebuh -> Lbh + Simpang -> Spg + Kampung -> Kg + Kampong -> Kg + Taman -> Tmn + Bangunan -> Bgn + Bukit -> Bkt + Batu -> Bt + Sungai -> Sg + Pulau -> P, Pl + Tanjung -> Tg + Tanjong -> Tg + Teluk -> Tlk + Telok -> Tlk + Gunung -> G, Gn + Gunong -> G, Gn + Haji -> Hj + Hajjah -> Hjh + Abdul -> Abd -**How to avoid:** Pass `reply_markup` in the `sendVenue` call itself. Verified in the installed typings this session. +country: my + Persiaran -> Psrn + Lebuhraya -> Lbh.Raya + Bandar -> Bdr + Kondominium -> Kondo + Bahagian -> Bhg + Blok -> Blk + Kuala -> K, Kl + Alur -> Alr + Bakau -> Bak + Hulu -> H + Hilir -> Hr + Jeram -> J, Jr + Lembah -> Lmbh + Padang -> Pdg + Parit -> Pt + Permatang -> Pmtg + Pelabuhan -> Plbh + Kepulauan -> Kep + Ladang -> Ldg + Lapangan -> Lpg + Gosong -> Gsg + Sekolah kebangsaan -> SK + Sekolah menengah -> SM + Sekolah menengah kebangsaan -> SMK + Sekolah rendah -> SR + Hospital -> Hosp + Dispensari -> Disp +``` -**Warning signs:** Two consecutive bot messages per `/cari`. +### The inverse map the normalizer actually needs + +The table above is *full → abbreviation*. D-09 needs **both directions** (query says "Jln", entry says "Jalan"). Invert it at load time: + +```typescript +// Build the bidirectional map ONCE at module load. Deterministic, ~60 entries. +const ABBREVIATIONS: Record = (() => { + const out: Record = {}; + for (const line of RAW_RULES.trim().split("\n")) { + const m = line.match(/^\s*(\S.*?)\s*->\s*(.+?)\s*$/); + if (!m) continue; + const full = m[1].trim().toLowerCase(); + for (const abbr of m[2].split(",").map((s) => s.trim().toLowerCase())) { + (out[abbr] ??= []).push(full); + // The full form maps to itself so the canonical form is always reachable. + (out[full] ??= []).push(full); + } + } + return out; +})(); +``` + +**Ambiguity to handle — `Pl`/`P`, `K`/`Kl`, `G`/`Gn`, `J`/`Jr`, `Lbh`/`Lbh.Raya`:** +Several abbreviations map to more than one full form (`Pulau → P, Pl` and `Parit → Pt` are distinct, but `Kuala → K, Kl` vs `Kampung → Kg` are fine). The dangerous collisions are: +- **`G` → Gunung/Gunong** — but `G` also appears as a *letter* in road names ("Lebuh Genting"). Expanding `G` in a token that is a single letter is aggressive; the normalizer should **only expand an abbreviation when the token is a standalone word** (already true of token-based normalization) and when the expansion is unambiguous *in context*. +- **`P` → Pulau** and **`Pt` → Parit** — `P` also appears in "Presint", "Persiaran", "Pusat". Single-letter abbreviations are the highest-collision-risk class. +- **`Bt` → Batu** vs `Bukit → Bkt` — note `Bt` is Batu (milestone), *not* Bukit. Locals mix these; the golden corpus should cover both. +- **`KG` ambiguity** — `KG` is the kilogram unit in some contexts; the existing `bm-address-normaliser` project flags this explicitly. In a *place-name* corpus the ambiguity is negligible, but the plan should note it. + +**Recommendation:** the normalizer keeps a per-token **ambiguity list** rather than guessing, and the fuzzy-residual tier (D-09) resolves what the table cannot. This matches D-09's architecture exactly: table first, folding second, fuzzy residual last. + +### The Presint/Prsn vowel-drop rule + +D-09 explicitly calls out "vowel-drop-aware Presint/Prsn forms". This is **not** in the JUPEM table above — it is a Putrajaya-specific local form. Verified against real Malaysian commercial addresses: +- `JALAN P8G PRESINT 8, 62250 Putrajaya` +- `Presint 15`, `PRESINT 16`, `PRESINT 4`, `PRESINT 18` +- `PRSN RESIDEN DESA PARKCI, 52200 KUALA LUMPUR` +- `NO 252 BAWAH PRSN SIBC 9, SERI ISKANDAR BUSINESS CENTRE` +- `Jalan P9 Presint 7, 62250 Putrajaya` + +So both "Presint" and "Prsn" appear in live use, and "Prsn" is not a *prefix* abbreviation — it drops the internal `e`. Add it as an explicit rule: + +```typescript +// Vowel-drop expansion (Putrajaya precincts) — D-09's explicit case. +ABBREVIATIONS["prsn"] = ["presint"]; +ABBREVIATIONS["presint"] = ["presint"]; +``` + +**A more general vowel-drop rule** worth considering (and worth a corpus test): Malay place names with a dropped interior vowel are a recurring local shorthand class — `Pusat`→`Pusat`, `Bandar`→`Bdr`, `Perbandaran`→`PBandaran`. The plan should decide whether to implement a *generic* vowel-drop expansion (insert `e` at each interior consonant cluster) or keep only the explicit `Presint`/`Prsn` pair. **Recommendation: explicit pair only.** A generic vowel-drop rule will over-expand legitimate names ("Pusat" contains no drop, but "Pkl" would expand to "Pekel" and then match "Pekel" wrongly). This is exactly the kind of rule that should be added by *table edit* (D-09) when the corpus demands it — not by a clever algorithm. + +### The MS 2039 rule that governs *display* (not matching) + +From **MS 2039:2006** §5.2.3.2 (fetched this session): + +> *"Abbreviation may only apply to a generic name of a geographical feature being referred in a particular line or hierarchies of the address format. Abbreviation shall not be applied to names given to the geographical feature itself. As an example, an address line 'Jalan Taman Desa' may be abbreviated to 'Jln Taman Desa' but not 'Jln Tmn Desa' or 'Jalan Tmn Desa'."* + +**How this applies to the normalizer:** the rule constrains what a *correct* address looks like, not what a *searchable* key looks like. Because the normalizer expands to the **long form on both sides**, "Jln Tmn Desa" and "Jln Taman Desa" both normalize to "jalan taman desa" and match. That is the desired search behaviour. **But the venue card's `address` field must show the curated address verbatim** — not the normalized form. The `sendVenue` payload's `address` comes from the gazetteer entry, never from `normalize()`. This is a one-line architectural rule that prevents a visible regression (Pitfall 3). + +### The three official abbreviation tables (for completeness) + +The JUPEM gazetteer PDF and MS 2039 Annex A carry **three** distinct tables. The normalizer needs all three, merged: + +| Table | Scope | Sample rows | +|-------|-------|-------------| +| **A1 — Street prefixes** (MS 2039 Annex A) | `Jalan → Jln`, `Lorong → Lrg`, `Persiaran → Psrn`, `Lebuhraya → Lbh.Raya`, `Persisiran → (none)`, `Pekeliling → (none)`, `Lintang → Ltg`, `Lengkok → Lnkg`, `Regat → Rgt` | 33 rows | +| **A3 — Locality prefixes** (MS 2039 Annex A + JUPEM gazetteer) | `Bandar → Bdr`, `Bukit → Bkt`, `Desa → (none)`, `Kampung/Kampong → Kg, Kpg`, `Kuala → K`, `Hulu → H`, `Hilir → Hr`, `Jeram → J`, `Ladang → Ldg`, `Padang → Pdg`, `Pasir → Psr`, `Permatang → Pmtg`, `Pulau → P`, `Sungai → Sg`, `Taman → Tmn`, `Tanjung → Tg`, `Teluk/Telok → Tlk`, `Mukim → (none)`, `Pekan → (none)` | ~35 rows | +| **JUPEM gazetteer list** (state gazetteer PDF, §4.0) | `Air → A.`, `Bahagian → Bhg.`, `Balai Polis → B.P.`, `Balai Raya → B.R.`, `Bakau → Bak.`, `Batu → Bt.`, `Blok → Blk.`, `Bukit → Bkt.`, `Cangkat/Changkat → Chgkt.`, `Hosp → Hosp.`, `Kondominium → Kondo`, `Lpg → Lapangan`, `Pejabat Pos → P.P.`, `Pangkalan/Pengkalan → Pang.`, `Pelabuhan → Plbh.`, `Rumah Rehat → R.R.`, `SMK`, `SR`, `SRJK`, `Stesen Kastam → S.K.`, `Tali air → T.A.`, `Utara → U` | 48 rows | + +**Note the intersection is not clean:** MS 2039 A3 gives `Hilir → Hr` while the JUPEM gazetteer list gives `Hilir` (no abbreviation) — and `Kuala → K` in A3 vs `K. ` in the gazetteer list. **Merge, don't replace**; when two tables disagree, prefer the **MS 2039** form (it is the current national standard) and record the alternative as an additional alias. + +### A ready-made city-code table + +The existing `bm-address-normaliser` project and the Nominatim PR both carry a **city-code** expansion set that is genuinely useful for KL-metro queries ("PJ" → Petaling Jaya, "JB" → Johor Bahru, "KB" → Kota Bharu, "KK" → Kota Kinabalu). These are *local shorthand*, not JUPEM abbreviations — they belong in the **gazetteer alias table (D-08)**, not the abbreviation rule table, because they are place identities, not address-word forms. The distinction matters: "KL" as a city code should expand to "kuala lumpur"; "Jln" should expand to "jalan". Both are expansions, but they belong in different tables. --- ## Code Examples -Verified patterns from official sources and the installed dependency tree. +### The `sendVenue` call with inline keyboard (D-10, SRCH-03) -### 1. The `/cari` handler — one message, venue card + OPEN + alternates (D-02, D-03, D-10, D-11) +**Source:** https://core.telegram.org/bots/api/methods/sendvenue.md + https://core.telegram.org/bots/api/types/inlinekeyboardbutton.md ```typescript -// Source: grammY 1.46.0 installed typings — -// api.d.ts:410 sendVenue(chat_id, latitude, longitude, title, address, other?) -// methods.d.ts sendVenue args include reply_markup?: InlineKeyboardMarkup | … -// markup.d.ts InlineKeyboardButton.UrlButton { text, url } -// InlineKeyboardButton.WebAppButton { text, web_app: WebAppInfo } -// markup.d.ts WebAppInfo { url: string } ← url ONLY, no start_param import { InlineKeyboard } from "grammy"; -import { sendVenue } from "../search/client.ts"; -export async function replyWithShortlist( - ctx: Context, - locale: Locale, - results: SearchResult[], -): Promise { - if (results.length === 0) { - // D-14: honest refusal. No pin, no nearest-match fabrication. - await ctx.reply(strings[locale].searchNoMatch, { link_preview_options: { is_disabled: true } }); - return; - } - const [best, ...alternates] = results; +// Required: latitude, longitude, title, address. There is NO `text` parameter. +await ctx.replyWithVenue( + entry.lat, + entry.lng, + entry.name, // title + entry.address ?? "", // address + { + reply_markup: new InlineKeyboard() + .webApp(openLabel, miniAppUrlFor(entry.id)) // OPEN IN ARAHARAH + .row() + .text(alt1.name, `alt:${alt1.id}`) // alternates → callback_data + .text(alt2.name, `alt:${alt2.id}`), + }, +); +``` - // D-02: ONE message. The OPEN button + the alternates ride on the venue card. - const keyboard = new InlineKeyboard() - .text(strings[locale].searchOpenButton) - .url(`${env.miniAppOrigin}/?dest=${best.id}`) // ← url button, NOT web_app - .row() - ...alternates.slice(0, 4).map((alt) => - InlineKeyboard.text(alt.name).callback_data(`dest:${alt.id}`), - ); +**Note on `reply_markup`:** `sendVenue` accepts `InlineKeyboardMarkup | ReplyKeyboardMarkup | ReplyKeyboardRemove | ForceReply` — the inline keyboard is the correct one here. - await ctx.replyWithVenue(best.lat, best.lng, best.name, best.address, { - foursquare_type: FOURSQUARE_BY_CATEGORY[best.category], // optional enrichment - reply_markup: keyboard, +### The `web_app` button URL vs. the `startapp` direct link + +```typescript +// Carrier A — web_app inline button, payload in the URL query string. +// The webview reads location.search, NOT initDataUnsafe.start_param. +function miniAppUrlFor(id: string, origin: string): string { + return `${origin}/?dest=${encodeURIComponent(id)}`; +} + +// Carrier B — t.me direct link, payload in ?startapp= (the officially documented +// start_param mechanism). Requires a Main Mini App or a Direct Mini App (short_name). +function deepLinkFor(id: string, botUsername: string): string { + // start_param charset is restricted to [A-Za-z0-9_-] and ≤512 chars. + return `https://t.me/${botUsername}?startapp=${id.replace(/[^A-Za-z0-9_-]/g, "_")}`; +} +``` + +### Reading the payload in the webview + +```typescript +// web/src/search/destination.ts — Carrier A (URL query). +export function readDestinationId(): string | null { + const raw = new URLSearchParams(window.location.search).get("dest"); + return raw && /^[A-Za-z0-9_-]{1,64}$/.test(raw) ? raw : null; +} + +// Carrier B (initData) — note: initDataUnsafe is UNTRUSTED per Telegram's docs; +// it is fine here only because the id is a public place id and is re-resolved +// against the local gazetteer. Never treat it as an identity. +export function readStartParam(wa: TelegramWebAppLike | null): string | null { + const raw = parseInitDataUnsafe(wa?.initData ?? "").start_param; + return raw && /^[A-Za-z0-9_-]{1,512}$/.test(raw) ? raw : null; +} +``` + +### The alternates callback handler + +```typescript +// bot/src/bot/handlers/search-alternate.ts +export function registerSearchAlternate(bot: Bot, source: SearchSource): void { + bot.callbackQuery(/^alt:(.+)$/, async (ctx) => { + await ctx.answerCallbackQuery(); // must happen within 30 s + const id = ctx.match[1]; + const entry = source.byId(id); // id → entry; never a query or a name + if (!entry) return; + await sendVenueCard(ctx, entry, source); // re-sends the card with new alternates + }); + + // Always answer everything else so Telegram doesn't show a 60-second spinner. + bot.on("callback_query:data", async (ctx) => { + await ctx.answerCallbackQuery(); }); } ``` -> **Note on `web_app` vs `url`:** `web_app` is also valid (Telegram grants the Mini App the user's basic info and a `query_id`), but its `WebAppInfo` carries **no** `start_param`. To carry the destination through a `web_app` button you would have to put it in the Mini App URL's own query string and read it from `location.search` anyway — which is exactly what the `url` button does, with one fewer moving part. The `url` button also avoids the Telegram Desktop START-button trap (Pitfall 2). +**Source:** https://grammy.dev/plugins/keyboard — *"It makes sense to define `bot.on("callback")` at last to always answer all other callback queries that your previous listeners did not handle. Otherwise, some clients may display a loading animation for up to a minute."* -### 2. Reading the destination in the webview (ONBD-02, D-11) +### The Malay normalizer ```typescript -// Source: Telegram Mini Apps docs — the start parameter arrives in the -// initData's `start_param` field AND as the GET parameter `tgWebAppStartParam`; -// character set /^[A-Za-z0-9_-]{0,512}$/. -import { getWebApp } from "../telegram/webapp.ts"; +// shared/src/normalize.ts +export function normalize(input: string): string { + let s = input.normalize("NFKD").toLowerCase(); + s = s.replace(/[̀-ͯ]/g, ""); // strip combining marks (NFKD) + s = s.replace(/[^\p{L}\p{N}\s/]/g, " "); // punctuation → space (keep "/" for Jln 3/1) + s = s.replace(/\s+/g, " ").trim(); -/** D-11: the deep link carries a plain gazetteer entry id. Two arrival paths: - * 1. ?dest= in the Mini App URL's own query string (the `url` button form) - * 2. start_param inside the SIGNED initData (the t.me?startapp= deep-link form) - * The id is resolved against the webview's BUILD-TIME COPY of the gazetteer. - * Nothing is persisted — memory only (GEO-03). */ -export function readDestinationId(): string | null { - const fromQuery = new URLSearchParams(window.location.search).get("dest"); - if (fromQuery !== null && fromQuery !== "") return fromQuery; - - const wa = getWebApp(); - const startParam = wa?.startParam; // ← add to TelegramWebAppLike - if (startParam !== null && startParam !== undefined && startParam !== "") { - // Tolerate the "dest_" form D-11 names. - return startParam.startsWith("dest_") ? startParam.slice(5) : startParam; + const expanded: string[] = []; + for (const token of s.split(" ")) { + if (!token) continue; + const hit = ABBREVIATIONS[token]; + expanded.push(hit ? hit[0] : token); // canonical = the long form } - return null; + s = expanded.join(" "); + + s = splitGluedWords(s); // "tamanbangsar" → "taman bangsar" + return s; +} + +/** Insert a space at the boundary where a known Malay word starts. */ +function splitGluedWords(s: string): string { + // Starters that commonly appear glued in Malaysian addresses. + const starters = ["jalan", "taman", "kampung", "lorong", "bukit", "sungai", + "bandar", "persiaran", "lebuh", "kampung", "perkampungan"]; + for (const starter of starters) { + // "tmnbangsar" is handled by the abbreviation table; this catches + // "tamanbangsar" / "jalankanpar" style glue. + const re = new RegExp(`(? `${m.slice(0, starter.length)} ${m.slice(starter.length)}`); + } + return s; } ``` -> **The `start_param` must come from the SIGNED `initData`, not `initDataUnsafe`.** The existing `POST /api/session` (Phase 2) already validates `initData` server-side and mints a session token from it — so if the webview wants to *trust* the `start_param`, the cleanest path is to have `/api/session` echo the validated `start_param` (or the resolved destination) back alongside the token. That avoids building a second validation path (see §Don't Hand-Roll). D-11's own framing ("the payload's only threat is a user pranking themselves") means the pragmatic reading is fine too, but the planner should choose one deliberately. - -### 3. Overpass top-up query (D-06) - -```overpassql -# Source: Overpass QL reference (https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL) -# bbox order: south,west,north,east (Overpass QL convention) -# `out center;` gives a way/relation a representative centre point -# `nwr` = node+way+relation in one filter -# KL metro bounding box (generous; covers the core Klang Valley sprawl) -[out:json][timeout:120][maxsize:536870912]; -( - nwr["shop"="mall"](2.95,101.55,3.20,101.80); - nwr["landuse"="retail"](2.95,101.55,3.20,101.80); - nwr["tourism"="attraction"](2.95,101.55,3.20,101.80); - nwr["leisure"="park"](2.95,101.55,3.20,101.80); - nwr["railway"="station"](2.95,101.55,3.20,101.80); - nwr["station"="subway"](2.95,101.55,3.20,101.80); - nwr["public_transport"="station"](2.95,101.55,3.20,101.80); - nwr["amenity"="hospital"](2.95,101.55,3.20,101.80); - nwr["amenity"="university"](2.95,101.55,3.20,101.80); - nwr["amenity"="place_of_worship"](2.95,101.55,3.20,101.80); - nwr["place"~"^(suburb|quarter|neighbourhood|village|hamlet)$"](2.95,101.55,3.20,101.80); - nwr["office"="government"](2.95,101.55,3.20,101.80); - nwr["building"="government"](2.95,101.55,3.20,101.80); -); -out center tags 300; // `out center tags` — geometry-free + centre + tags +**The Presint/Prsn rule (D-09's "vowel-drop-aware" clause):** Putrajaya's precincts are written both "Presint" and "Prsn" — the latter drops the `e`. Add an explicit rule: +```typescript +// "prsn" → "presint" is a *vowel-drop* expansion, not a prefix abbreviation. +ABBREVIATIONS["prsn"] = ["presint"]; +ABBREVIATIONS["presint"] = ["presint"]; ``` +Verified against real usage: `JALAN P8G PRESINT 8`, `PRESINT 15`, `PRSN RESIDEN DESA PARKCI`, `Jalan P9 Presint 7` — all appear in commercial Malaysian addresses (see §"Sources"). -**Script rules (D-06):** -- **One** HTTP request per run, with a descriptive `User-Agent` (Pitfall 4). -- Output is a **candidate list**, never a direct write to the curated file. The maintainer reviews and approves entries into `kl-gazetteer.json`. -- The script must **not** invent aliases. Aliases are D-08's human lever. The script may *suggest* an alias when an OSM tag like `name:en`, `short_name`, `official_name`, `old_name`, or `int_name` exists — those are real data, not guesses. -- Nominatim is a reasonable **fallback** for a specific named place the bbox query missed (e.g. a mall whose OSM tag is `shop=department_store` rather than `shop=mall`). Nominatim's usage policy is stricter than Overpass's — at most one request per second, a valid identifying User-Agent, and no bulk use. **The top-up script is a maintainer tool, not a crawler**; a bulk Nominatim sweep would violate the policy. +**The glued-word rule needs care.** A naive "insert a space before any known word" will corrupt words that legitimately contain a starter as a substring ("Pandang" contains "dang", "Kampung Baru" is fine, but "Bandar Baru" → "band ar" if "dar" were a starter). **The safe implementation is a longest-match-first scan against the starter list, requiring the remainder to be ≥3 characters.** The plan should specify this; the naive version is a real bug source. -### 4. The golden corpus format (D-13) and the threshold gate (D-14) +### The Damerau-Levenshtein (OSA) distance — hand-rolled -```json +```typescript +// shared/src/distance.ts — optimal string alignment, adjacent transpositions. +export function damerau(a: string, b: string): number { + if (a === b) return 0; + if (a.length === 0) return b.length; + if (b.length === 0) return a.length; + + const n = a.length, m = b.length; + // 3-row band: only the previous, current and current+1 rows are needed. + let prev2 = new Int32Array(m + 1); + let prev = new Int32Array(m + 1); + let cur = new Int32Array(m + 1); + for (let j = 0; j <= m; j++) prev[j] = j; + + for (let i = 1; i <= n; i++) { + cur[0] = i; + for (let j = 1; j <= m; j++) { + const cost = a[i - 1] === b[j - 1] ? 0 : 1; + let v = Math.min(cur[j - 1] + 1, prev[j] + 1, prev[j - 1] + cost); + if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) { + v = Math.min(v, prev2[j - 2] + 1); // transposition + } + cur[j] = v; + } + const t = prev2; prev2 = prev; prev = t; + const c = cur; cur = prev2; // rotate the three rows + } + return prev[m]; +} +``` +**Type-check this carefully** — the row-rotation in a 3-band OSA is easy to get wrong. Write the test vectors first (TDD, matching the repo's RED→GREEN convention). + +### The golden corpus format + +```jsonc +// shared/data/golden-corpus.json { - "_comment": "araharah golden corpus — the executable definition of the bake-off (D-13). Each positive pins its expected entry id. Each negative must yield ZERO results (honest refusal). Reviewers extend by editing THIS FILE ONLY.", - "_version": 1, - "threshold": { "top3_hit_rate": 0.9 }, - "positives": [ - { "q": "KLCC", "expect": "suria-klcc", "kind": "golden" }, - { "q": "Jln Ampang", "expect": "suria-klcc", "kind": "abbrev" }, - { "q": "Jalan Ampang", "expect": "suria-klcc", "kind": "canonical" }, - { "q": "jln ampang", "expect": "suria-klcc", "kind": "folding" }, - { "q": "Tmn", "expect": "taman-...", "kind": "abbrev" }, - { "q": "Sunway Pyrmid", "expect": "sunway-pyramid", "kind": "typo-transposition" }, - { "q": "Sunwy Pyramid", "expect": "sunway-pyramid", "kind": "typo-missing-key" }, - { "q": "DUKE", "expect": "duke-university", "kind": "acronym" }, - { "q": "Kg Baharu", "expect": "kampung-baharu", "kind": "kampung" }, - { "q": "Kg Baharu", "expect": "kampung-baharu", "kind": "glued" }, - { "q": "PWTC", "expect": "pwtc", "kind": "acronym" } - ], - "negatives": [ - { "q": "Kuching", "reason": "out of the v1 area" }, - { "q": "Penang", "reason": "out of the v1 area" }, - { "q": "asdkjhaskdjh", "reason": "not a place" } + "version": 1, + "cases": [ + // ---- golden landmarks (SRCH-01) ---- + { "q": "KLCC", "expectId": "suria-klcc" }, + { "q": "Sunway Pyramid", "expectId": "sunway-pyramid" }, + { "q": "DUKE", "expectId": "the-duke" }, + + // ---- abbreviations (SRCH-02) ---- + { "q": "Jln Ampang", "expectId": "suria-klcc" }, // Jalan Ampang + { "q": "Tmn Bangsar", "expectId": "bangsar" }, + { "q": "Kg Ampang", "expectId": "kampung-ampang" }, + + // ---- glued words (SRCH-02) ---- + { "q": "tamanbangsar", "expectId": "bangsar" }, + + // ---- typos (SRCH-02) ---- + { "q": "KCLC", "expectId": "suria-klcc" }, // transposition + { "q": "Sunwya Pyarmid", "expectId": "sunway-pyramid" }, + + // ---- aliases (D-08) ---- + { "q": "IJN", "expectId": "ijnp" }, + { "q": "PWTC", "expectId": "pwtc" }, + { "q": "Stesen KL Sentral","expectId": "kl-sentral" }, + + // ---- deliberate negatives (D-13) — MUST be refused, never a fabricated pin ---- + { "q": "Kuching", "expectId": null }, + { "q": "Penang", "expectId": null } ] } ``` -**The gate, as an executable assertion** (this is the piece that makes D-14 a decision rather than an opinion): +### The corpus runner ```javascript -// bot/test/corpus.test.mjs — shape follows bot/test/harness.mjs (repo precedent). -import { strict as assert } from "node:assert"; -import { test } from "./harness.mjs"; -import { corpus } from "./fixtures/golden-corpus.json" with { type: "json" }; -import { GazetteerSource } from "../src/search/source.ts"; +// bot/scripts/gazetteer-corpus.mjs — the bake-off close (D-14). +// Prints top-1 / top-3 hit rate on positives and refusal rate on negatives. +// Exit code 1 if top-3 < 90% or if any negative produced a hit. +const { search, gazetteer } = await import("@araharah/search"); +const corpus = JSON.parse(await readFile(corpusPath, "utf8")); -test("CORPUS D-14: top-3 hit rate ≥ 90% on positives AND every negative is refused", async () => { - const source = new GazetteerSource(); - let hits = 0; - const misses = []; - for (const c of corpus.positives) { - const results = await source.search(c.q, 3); - const ids = results.map((r) => r.id); - if (ids.includes(c.expect)) hits++; - else misses.push({ q: c.q, expect: c.expect, got: ids }); - } - const rate = hits / corpus.positives.length; - assert.ok( - rate >= corpus.threshold.top3_hit_rate, - `top-3 hit rate ${(rate * 100).toFixed(1)}% < ${corpus.threshold.top3_hit_rate * 100}% — ` + - `escalation ladder step 1 (alias/normalizer table edits) is required. Misses: ${JSON.stringify(misses)}`, - ); +const positives = corpus.cases.filter((c) => c.expectId !== null); +const negatives = corpus.cases.filter((c) => c.expectId === null); - for (const n of corpus.negatives) { - const results = await source.search(n.q, 3); - assert.equal(results.length, 0, `negative "${n.q}" must be refused honestly, got ${JSON.stringify(results.map(r => r.id))}`); - } -}); -``` - -**On failure, the assertion message names the escalation step.** That is D-14's ladder made mechanical: the failure text tells you to pull lever 1 (table edits), then re-run; if lever 1 cannot close the gap, lever 2 (backend #2) is next, then lever 3 (Photon). The phase directory should hold a short decision record naming which lever was pulled and the corpus run that closed it. - -### 5. The normalizer as a per-rule vector table (D-09) - -```typescript -// Source: D-09 — "Per-rule test vectors; extends by table edit, not code change." -export const NORMALIZER_VECTORS: ReadonlyArray<{ - rule: string; - input: string; - expect: string; -}> = [ - { rule: "casefold", input: "KLCC", expect: "klcc" }, - { rule: "casefold", input: "Sunway PYRAMID", expect: "sunway pyramid" }, - { rule: "whitespace", input: " jln ampang ", expect: "jln ampang" }, - { rule: "abbrev:jln", input: "Jln Ampang", expect: "jalan ampang" }, - { rule: "abbrev:tmn", input: "Tmn Melewah", expect: "taman melewah" }, - { rule: "abbrev:kg", input: "Kg Baharu", expect: "kampung baharu" }, - { rule: "abbrev:prsn", input: "Prsn 4", expect: "presint 4" }, - { rule: "abbrev:presint", input: "Presint 4", expect: "presint 4" }, - { rule: "identity:tmn", input: "Tmn", expect: "tmn" }, - { rule: "glued:kg", input: "KampungBaru", expect: "kampung baru" }, - { rule: "glued:klcc", input: "KLCC", expect: "klcc" }, // no known prefix → unchanged - { rule: "no-mangle:1", input: "Sunway Pyramid", expect: "sunway pyramid" }, // NOT "sunaan pyramid" - { rule: "no-mangle:2", input: "KLIA", expect: "klia" }, // NOT "kualia lumpuria" - { rule: "diacritic", input: "Ampáng", expect: "ampang" }, -]; -``` - -### 6. ONBD-03 — `addToHomeScreen` with an iOS-honest status read - -```typescript -// Source: core.telegram.org/bots/webapps — -// addToHomeScreen() (Bot API 8.0+) — prompts the user to add the Mini App -// to the home screen; the `homeScreenAdded` event fires if the device supports it. -// checkHomeScreenStatus(cb) — cb(status) where status is one of: -// "unsupported" | "unknown" | "added" | "missed" -// Telegram-iOS issue #1729: on iOS the status is ALWAYS "unknown" — the platform -// offers no API to detect whether the shortcut exists. -export function wireHomeScreen(wa: TelegramWebAppLike | null): void { - if (!wa || typeof wa.addToHomeScreen !== "function") return; - wa.addToHomeScreen(); - wa.onEvent?.("homeScreenAdded", () => { - // Android/desktop path: the shortcut exists. - }); - wa.checkHomeScreenStatus?.((status: string) => { - // "unsupported" → feature unavailable here (Desktop web). - // "unknown" → iOS. Do NOT claim it failed; treat as "possibly added". - // "added" / "missed" → definitive. - }); +let top1 = 0, top3 = 0, missed = []; +for (const c of positives) { + const hits = await search({ text: c.q, limit: 3 }); + const ids = hits.map((h) => h.entry.id); + if (ids[0] === c.expectId) top1++; + if (ids.includes(c.expectId)) top3++; + else missed.push(`${c.q} → [${ids.join(", ") || "—"}] (expected ${c.expectId})`); } + +let refused = 0, leaked = []; +for (const c of negatives) { + const hits = await search({ text: c.q, limit: 3 }); + if (hits.length === 0) refused++; + else leaked.push(`${c.q} → ${hits[0].entry.id}`); +} + +const top3Rate = top3 / positives.length; +console.log(`top-1 ${top1}/${positives.length} top-3 ${top3}/${positives.length} (${(top3Rate*100).toFixed(1)}%)`); +console.log(`negatives refused ${refused}/${negatives.length}`); +if (missed.length) console.log("MISSED:\n" + missed.join("\n")); +if (leaked.length) console.log("LEAKED (must refuse):\n" + leaked.join("\n")); + +// D-14 gate: top-3 ≥ 90% AND every negative refused. +process.exitCode = top3Rate >= 0.9 && leaked.length === 0 ? 0 : 1; ``` -> **Important:** `beforeinstallprompt` — the standard web-PWA install event — **does not exist inside a Telegram webview**. The Telegram-native `addToHomeScreen()` is the mechanism. Do not build a web manifest-driven install flow for ONBD-03 (that is the documented-deferred full-PWA story). Also note that **service workers do not work in Telegram Mini Apps on iOS** ([Telegram-Mini-Apps issue #27](https://github.com/Telegram-Mini-Apps/issues/issues/27)) — another reason the full-PWA story stays deferred. - ---- - -## Licensing — the ODbL obligation (D-04 × D-06) - -This section exists because it is the one place where two locked decisions collide. - -**What the OSMF actually says** (fetched this session): - -- OSM data is under the **Open Database License (ODbL) 1.0** — attribution + share-alike for *databases*. [foundation.openstreetmap.org/wiki/Licence_and_Legal_FAQ](http://foundation.openstreetmap.org/wiki/Licence_and_Legal_FAQ) -- *"You must also make the derivative database available to anyone who received your data… If you make a database which includes OSM data and any additional information… then this would be classed a 'derivative database' and should be made available under the Open Database License."* [Legal FAQ](https://openstreetmap.org/wiki/Legal_FAQ) -- *"if you have any data that was derived from OSM — for instance because you used street names from OSM… then you are making a derivative database."* [License/Use Cases, Case 3](https://wiki.openstreetmap.org/wiki/License/Use_Cases) -- *"A 'geocoding database' should **NOT** be considered a trivial transformation."* [Trivial Transformations Guideline](https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline) -- ODbL §4.4: a publicly distributed Derivative Database must be under the ODbL, a later version with the same Licence Elements, or *"a compatible licence."* -- The OSMF's compatibility guidance says licences that require downstream users to use the data on *identical* terms are generally incompatible with ODbL. - -**The conclusion:** a curated gazetteer of KL POIs whose names, addresses and coordinates come from OSM is a **Derivative Database**. The software around it may be GPL-3.0 (D-04), but the **data file** carries an ODbL share-alike obligation. - -**Recommended resolution — dual-licence, the ecosystem-standard shape:** - -| Artifact | Licence | Note | -|----------|---------|------| -| All source code (`bot/`, `web/`, `theme/`) | **GPL-3.0** | D-04. Add the `LICENSE` file this phase (the CONTEXT canonical refs call for it). | -| `bot/src/search/data/kl-gazetteer.json` (and any future gazetteer dump) | **ODbL-1.0** | A data file, not code. Header carries `© OpenStreetMap contributors` + a link to . | -| `bot/test/fixtures/golden-corpus.json` | **GPL-3.0** with the repo | It contains queries and entry ids, no OSM content. | -| Attribution in the product | **Required** | The bot's venue card / replies should carry `© OpenStreetMap contributors` — the web app already ships exactly this string in `web/src/i18n/strings.ts` (`attribution`), so the bot's strings module needs the twin. | - -**Why this satisfies D-04 rather than breaking it.** The forkability laws are about *mechanics* — no hardcoded secrets, plain committed data files, one-command deploy, one free-tier account per fork. A dual-licence keeps every one of those: the data file is still a plain committed file, and a fork inherits the ODbL obligation *on that file* — which is the correct, legally-required, entirely workable outcome. It is exactly what OSM-derived datasets in the ecosystem do (the OSM attribution notice is already visible in this project's UI). - -**Optional defence-in-depth:** the D-06 Overpass script can *propose* entries, but the maintainer can hand-write the display name, the address string, and the aliases from local knowledge, using OSM only for coordinates and for a sanity check. That reduces (though does not eliminate) the "derived" exposure. **Do not rely on it** — the dual-licence is free. +**This runner IS the bake-off close.** Its output is the artifact D-14 asks to be recorded: "the phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question." The plan should also produce a short `docs/SEARCH-QUALITY.md` (or a section in the phase summary) recording: the corpus version, the score, and which escalation lever was pulled. --- @@ -903,262 +1045,376 @@ This section exists because it is the one place where two locked decisions colli | Old Approach | Current Approach | When Changed | Impact | |--------------|------------------|--------------|--------| -| Photon-on-VPS as the v1 search backend (the roadmap's original Phase-4 note) | **In-Worker curated gazetteer behind a `SearchSource` interface** (D-03) | Decided 2026-10-23 (this phase's CONTEXT.md) | Removes all infra cost and the VPS dependency from Phase 4. Photon moves to backend #3 behind the same interface. | -| Full-text search engine (Tantivy/Meilisearch/Typesense) | **Three-tier in-Worker cascade** (exact → bigram filter → Damerau) | Inherent to the 200–500 row scale | No service to run, no network hop, no query latency. The 10 ms CPU budget is comfortable. | -| Hand-typing the abbreviation expansion rules in code | **An editable rule table** with per-rule vectors (D-09) | Inherent to D-09 | Corpus quality becomes a data-review activity, not a code-review activity. | -| `t.me/?startapp=` deep link as the OPEN button | **`url` button → `/?dest=`**, `start_param` as a secondary path | Pitfall 2 (Desktop START-button trap) | One tap on every client; no START-button detour. | -| Web PWA install (`manifest` + `beforeinstallprompt`) for the home screen | **Telegram-native `WebApp.addToHomeScreen()`** (Bot API 8.0+) | Bot API 8.0, Nov 2024 | The only mechanism that works inside a Telegram webview; iOS status is always `unknown`. | -| SQLite/sql.js WASM index in the Worker | **Typed JSON** (D-07) | D-07's own reasoning | ~1.5 MB bundle + cold-start cost avoided; diff-friendly PRs. | -| Geocoding database as a "trivial transformation" of OSM | **Explicitly NOT trivial** per the OSMF guideline | Guideline endorsed by the OSMF board 2014-06-06 | The ODbL share-alike obligation applies — see §Licensing. | +| Nominatim-style server-side geocoding for every query | In-Worker curated gazetteer behind a `SearchSource` interface | D-03 (this phase) | Zero infrastructure, zero latency, zero third-party SLA — but only as good as the curated corpus. Phase 4's job is to prove the corpus is good enough (D-14). | +| Photon-on-VPS as the phase-4 backend | Photon deferred to Phase 5; in-Worker gazetteer is backend #1 | D-03 (supersedes the roadmap note) | The roadmap's Phase 4 note says "VPS provisioning begins here" — **D-03 supersedes it.** The plan must NOT include VPS/Photon provisioning. | +| A Malaysia-only WASM geocoder as the phase-4 backend | Documented as backend #2 (post-v1, OSS track) | D-05 | The JSON dataset format stays reachable from day one via the converter script, so backend #2 is a format change, not a rewrite. | +| Hand-authored abbreviation list | The official JUPEM table, vendored | Nominatim PR #4056, merged 2026-04-02 | The table grew from 8 entries to 65+, split into shared / MY / SG / BN. Vendoring it wholesale (rather than the 8-entry original) is what makes "Jln Ampang" work for free. | +| Trigram index for short-query search | Exact-match tiers + alias table + fuzzy residual | This phase (D-09) | Trigram on 4-character queries is noise. The alias table (D-08) is the real lever for short queries. | +| `sendMessage` + `sendLocation` for the destination confirmation | `sendVenue` | D-10 | Telegram's native venue rendering (pin + name + address, tappable in the client) is what SRCH-03 names. | +| `startapp` on the `web_app` inline button URL (as CONTEXT D-11 reads) | An explicit carrier choice with a fallback | This phase's research | `startapp` is officially a **direct-link** mechanism; the `web_app` button carries a plain URL. See §"Deep-link risk" — the plan must pick. | -**Deprecated/outdated:** -- **`beforeinstallprompt`** — does not fire inside a Telegram webview. Not a valid ONBD-03 mechanism. -- **Service workers in Telegram Mini Apps on iOS** — confirmed broken ([Telegram-Mini-Apps issue #27](https://github.com/Telegram-Mini-Apps/issues/issues/27), still open with a 2025-09 update). Another reason the full-PWA/offline story stays deferred. -- **OSM's old "geocoding databases may be trivial" hope** — the guideline says the opposite. -- **The roadmap's Phase-4 VPS/Photon/GraphHopper plan** — superseded by D-03/D-05. +**Deprecated / to avoid:** +- **`@types/telegram-web-app`** — Phase 3 deliberately dropped it (the hand-rolled `TelegramWebAppLike` contract is the project's contract). Extend that interface; do not re-add the package. +- **`bot.chat.*`** — grammY 1.46 exposes no `bot.chat` proxy (noted in `bot/src/bot/handlers/location-fallback.ts:34-35`). Use `bot.api.*` / `ctx.*`. +- **`damerau-levenshtein` (the old CJS-ish package)** — last published 2022; prefer `@crob/damerau-levenshtein` or hand-roll. --- ## Assumptions Log -> All claims tagged `[ASSUMED]` in this research, for the planner and discuss-phase to confirm. +> Claims tagged `[ASSUMED]` below need user confirmation before becoming a locked decision. | # | Claim | Section | Risk if Wrong | -|---|-------|---------|------------| -| A1 | The abbreviation table's canonical list (Jln/Jalan, Tmn/Taman, Kg/Kampung, Lorong, Persiaran, Presint/Prsn, Stesen/Stn) is sufficient for the KL corpus. Grounded in the UPU address standard and OSM's KL road list, but **no authoritative KL-specific abbreviation table exists** — this is a curated approximation. | §Pattern 3 | Low-medium. The corpus runner measures the gap; lever 1 (table edits) is the designed remedy. | -| A2 | The KL bounding box `2.95,101.55,3.20,101.80` is the right scope for the Overpass top-up. Plausible from the metro extent, not verified against a canonical "Klang Valley" boundary. | §Code Examples 3 | Low. A wider box costs a longer Overpass run; a narrower one misses outliers. The bbox is a script constant, trivially edited. | -| A3 | `sendVenue` renders acceptably on all Telegram clients (iOS/Android/Desktop) with a `reply_markup` attached. **Not verified on a real device this session** — the API surface is verified, the rendering is not. | §Code Examples 1 | Medium. Rendering is cosmetic; the fallback (a text message with an inline keyboard) is trivial if a client renders the card badly. **Recommend it as a real-client checkpoint.** | -| A4 | The `url` inline button to the Mini App origin behaves identically to a `web_app` button on all clients (single tap, Mini App opens, the app's own query string is preserved). The API surface is verified; the client behaviour is not device-tested. | §Code Examples 1, Pitfall 1 | Medium-high if wrong — this is the ONBD-02 hand-off. **Recommend it as a real-client checkpoint** (the Phase-3 device-matrix pattern applies). | -| A5 | A ~500-row gazetteer with ~8 aliases each, parsed and indexed per request, fits comfortably inside the Workers Free 10 ms CPU budget. The 10 ms figure is verified; the *measured* cost of this specific workload is not. | §Summary finding 4 | Low-medium. The design has headroom by construction (tier-1 O(1) lookup short-circuits most queries), and `wrangler dev` can measure it. | -| A6 | The OSM licensing analysis (curated gazetteer = Derivative Database → ODbL share-alike; dual-licence resolves it) is correct **as an engineering practice**, but this is legal guidance, not legal advice. The OSMF wiki is community guidance and explicitly disclaims being authoritative. | §Licensing | Medium for the *project*, low for the *code*. The dual-licence costs almost nothing and is strictly safer than the alternative. A maintainer who wants certainty should ask the OSMF licensing mailing list. | -| A7 | The `dest_` payload in `start_param` will arrive inside `initData` and be signed by Telegram in the `t.me?startapp=` path (and therefore validated by the existing `/api/session`). The `start_param` field exists in `WebAppInitData` per the docs, but this session did **not** verify its presence in a real signed `initData` sample. | §Code Examples 2 | Low. The primary path (`?dest=` in the URL) does not depend on it. | -| A8 | The Telegram `/cari` command alias scheme (bilingual) has room in `strings.helpMenu` without violating the Phase-2 CS5 gap-freedom gate — the gate currently expects exactly `/help` and `/privacy` and will need extending. | §Pitfall 11 | Low. It is a mechanical test update. | -| A9 | The gazetteer seed can be assembled from public knowledge of KL landmarks (KLCC, Sunway Pyramid, DUKE, PWTC, Pavilion, IJN, etc.) with accurate coordinates. **Coordinates for the seed entries should still be sourced from OSM** to avoid errors — so the ODbL analysis stands regardless of where names come from. | §Pattern 4 | Low for the coordinates; medium for exact addresses, which should come from OSM's `addr:*` tags. | +|---|-------|---------|---------------| +| A1 | The `web_app` inline button's `WebAppInfo.url` may carry a plain query param (`?dest=`) that survives into the Mini App webview's `location.search`. | §"Deep-link risk" / Carrier A | **MEDIUM-HIGH.** Supported by community reports (StackOverflow: *"passing query params to telegram web app just works"*) but **not** by Telegram's official docs, which only describe `startapp` for direct links. If wrong, Carrier A collapses and the phase must use Carrier B (t.me deep link) — which changes the UX (an interstitial) and the button type. **Mitigation: build the carrier as a swappable function and test it on a real device at the exit gate.** | +| A2 | `Telegram.WebApp.initDataUnsafe.start_param` is populated when a Mini App is launched from a **direct link** (`t.me?startapp=…`) — i.e. that the "direct link" mechanism populates `start_param` in initData. | §"Deep-link risk" / Carrier B | **LOW-MEDIUM.** Officially documented (`start_param` "will also be passed in the GET-parameter `tgWebAppStartParam`"), but the `WebAppInitData.start_param` row's wording ("Only returned for Mini Apps when launched from the attachment menu via link") is narrower and could be read as excluding direct links. | +| A3 | The gazetteer is a **Derivative Database** under ODbL (because entries are selected/curated from OSM data) and therefore must be offered under ODbL. | §"Security Domain" | **LOW.** The consequence is mild (add an ODbL notice + attribution; the data is already public). But if the curation were deemed a *Produced Work* rather than a Derivative Database, the share-alike obligation would not attach. **Mitigation: add the notice anyway — it costs nothing and satisfies both readings.** | +| A4 | A ~200–500-entry gazetteer with ~5 aliases each is sufficient to hit ≥90% top-3 on a 60–100-query hand-written corpus. | §"Golden corpus" / D-14 | **MEDIUM.** This is the phase's core empirical assumption. It is exactly what the corpus runner is designed to falsify. If it fails, D-14's escalation ladder applies (alias/normalizer edits first). | +| A5 | Node 24's default TypeScript type-stripping lets a `.mjs` script `import("../src/foo.ts")` — the pattern already used by `bot/test/commands.test.mjs`. | §"Recommended Project Structure" | **LOW.** The existing tests already rely on it and pass, so the pattern is proven in-repo. Verify in the plan's first task anyway. | +| A6 | The `Mini App` public URL can be derived from `WEBHOOK_URL` (the Worker's origin) rather than needing a new env var. | §"Runtime State Inventory" | **LOW.** `bot/wrangler.jsonc` has `workers_dev: true` and `docs/CONFIGURATION.md` documents `WEBHOOK_URL` as the full origin + path. The plan may add a dedicated `MINI_APP_URL` var for clarity — a small, cheap decision. | +| A7 | `checkHomeScreenStatus()` returning `"unknown"` on iOS is a platform limitation, not a bug to work around. | §"ONBD-03 feasibility" | **LOW.** Confirmed upstream (Telegram-iOS issue #1729, closed with the comment *"On iOS it is not possible to determine whether a Mini App has been added to the Home Screen"*). The design must not branch on iOS status — treat `"unknown"` as "supported but indeterminate". | +| A8 | `sendVenue` messages can carry an inline keyboard that is later edited with `editMessageReplyMarkup` without destroying the venue payload. | §"Alternatives Considered" | **LOW-MEDIUM.** The Bot API docs say a message can be edited "without `reply_markup` or with inline keyboards" — but a `venue` message's *content* is a venue, and `editMessageText` would replace it. `editMessageReplyMarkup` only touches the markup. Verify on device; the safe fallback is to send a *new* `sendVenue` (which also matches D-02's "one tap to switch"). | --- ## Open Questions -1. **Should the OPEN button be a `url` button or a `web_app` button?** - - **What we know:** `WebAppInfo` has only `url` — no `start_param` field exists on either button type. The `t.me?startapp=` deep link carries `start_param` but needs an extra START tap on Telegram Desktop. The `url` button with `?dest=` is single-tap everywhere. - - **What's unclear:** Whether a `web_app` button grants the Mini App anything the `url` button does not (it grants `query_id` and basic user info, and permits `answerWebAppQuery`). Phase 4 does not need `answerWebAppQuery` — the bot sends the card. - - **Recommendation:** **Use the `url` button.** Revisit only if Phase 5/6 needs the Mini App to message back on the user's behalf. +1. **Which deep-link carrier does Phase 4 ship?** (D-11 says `?startapp=dest_`; the docs put `startapp` on direct links, not inline buttons.) + - **What we know:** `WebAppInfo.url` is an arbitrary HTTPS URL; community reports say query params survive; `startapp` is documented for `t.me` links and `startattach` for attachment-menu links; the `web_app` inline button is a distinct launch surface. + - **What's unclear:** whether Telegram's own clients preserve the query string on a `web_app` button URL across Android/iOS/Desktop. + - **Recommendation:** implement **Carrier A** (`web_app` button, `?dest=` in the URL) as the primary, keep **Carrier B** (`t.me?startapp=`) for the ONBD-03 home-screen path, and **write the payload reader to accept both** (`location.search` first, then `initDataUnsafe.start_param`). Validate on a real device at the exit gate. **Flag to the user**, because it touches a locked decision's wording. -2. **Should `/api/session` echo the validated `start_param` back to the webview?** - - **What we know:** `start_param` is inside the signed `initData`, so the existing Phase-2 validation already covers it. The webview currently receives only `{token}` from `/api/session`. - - **What's unclear:** Whether trusting `initDataUnsafe.start_param` client-side is acceptable (D-11 says the threat is only self-pranking) or whether the server should re-derive it. - - **Recommendation:** **Have `/api/session` return the validated `start_param` alongside the token.** One extra field, zero new validation path, and it makes the deep-link payload trustworthy by construction. It also keeps the claim surface (D-08) unchanged — `start_param` is not stored in KV. +2. **Is the venue card one message or two?** + - **What we know:** `sendVenue` has no `text` parameter; D-02 wants one tap to proceed; CONTEXT's "specific ideas" mention a one-line clinical confirmation. + - **What's unclear:** whether "one line confirming the destination" is a separate `sendMessage` before the card, or a property of the card's own `title`/`address`. + - **Recommendation:** **one `sendVenue`** whose `title` is the place name and `address` is the curated address — the native card *is* the confirmation, and this honours D-02's single-message shape. If the user wants a separate clinical line, it costs one extra message and must be added to the copy gate. -3. **Is a result cache worth it?** - - **What we know:** At 200–500 rows the search is sub-millisecond; D-15 permits TTL-only KV caching keyed by normalized query; D-15 also notes the query is user payload (so a cache key derived from it is a fingerprint of the user's activity, even if the content is public place data). - - **What's unclear:** Whether the Workers CPU budget ever gets tight enough to matter — it does not at this scale. - - **Recommendation:** **Ship without a cache.** It is a privacy cost with no performance benefit. If a future backend makes search slow, cache then — and key by a **hash** of the normalized query rather than the query itself. +3. **What is the exact corpus size and the exact per-query expectation?** + - **What we know:** D-13 says ~60–100 queries with the listed categories; D-14 sets ≥90% top-3 on positives. + - **What's unclear:** whether the 90% is measured over *all* positives or over positives *excluding* the typo bucket (typos are the hardest and the fuzzy tier's raison d'être). + - **Recommendation:** measure and report **per-category** rates (exact / abbreviation / glued / typo / alias), and set the D-14 gate on the **all-positives** figure, with per-category figures reported alongside so a regression is diagnosable. This is OpenCode's Discretion; state it in the plan. -4. **Where does the shared search module live?** - - **What we know:** Option A (a new npm workspace, e.g. `@araharah/search`) gives one on-disk copy consumed by both `bot/` and `web/`, mirroring the existing `theme/` → `web/` pattern. Options B and C exist but reintroduce duplication risk. - - **What's unclear:** Whether adding a fourth workspace is acceptable to the maintainer, given D-04's "one-command deploy" law and the existing three-workspace layout. - - **Recommendation:** **Option A.** It is the only option that makes D-11's "id mapping stays in sync by construction" literally true, and it lets the corpus runner import the identical module the production paths use. +4. **Which `SearchSource` seam shape?** + - **What we know:** D-03 requires a pluggable interface with the gazetteer as implementation #1, WASM geocoder as #2, Photon as #3. + - **What's unclear:** whether the interface is synchronous or async, and whether it lives in the shared package or in `bot/src/search/`. + - **Recommendation:** `async search(): Promise` (a network-backed backend #3 will need it), defined in the **shared package** so the webview can use the same types. The `byId(id)` lookup — needed by the callback handler and the webview — should be a **separate, synchronous** method on the interface, because both the callback handler and the deep-link reader need it and neither should await a network. -5. **How large should the seed gazetteer be, and who curates it?** - - **What we know:** D-07 says 200–500 rows; D-13 says 60–100 corpus queries; D-06 says hand-curated seed + Overpass top-up. The coverage that matters is *recognisable KL landmarks*, not exhaustiveness. - - **What's unclear:** Whether 200 rows can cover the corpus's ~60–100 positives plus enough alternates to make D-02's shortlist meaningful. - - **Recommendation:** **Seed with ~120–150 entries** covering malls, transit hubs, major hospitals, universities, government complexes, parks, and ~30 kampung/neighbourhood names — enough for the corpus and for a 3–5 item shortlist per query. Grow via the Overpass script. +5. **Is the Overpass script required to have a *verified* run in this phase?** + - **What we know:** `overpass-api.de` returns **406 without a `User-Agent` and 200 with one** (verification pass, same host/same paths); `overpass.kumi.systems` and `overpass.private.coffee` are unreachable from this network (HTTP 000); `overpass.osm.ch` returns 400 (Switzerland-only); Nominatim and Geofabrik work. + - **What's unclear:** whether the phase's exit gate requires the Overpass path to have been exercised against live OSM. + - **Recommendation:** the plan should require the script to **run and produce a valid-shaped proposal file** (verified against a **committed fixture** of a real Overpass response, committed to the repo so the test is deterministic), and to document the live-source URL in the script's header. **The `User-Agent` header is load-bearing and must be asserted by a test** — a missing UA produces a 406 on every endpoint and looks like a dead service. The curated entries themselves should be coordinate-verified against **Nominatim** (reachable) — which satisfies D-06's "hand-curated seed" with real coordinates. **Do not plan around a blocked Overpass**: the live path works from this machine once the header is set; the fixture is for *determinism*, not for working around an outage. --- ## Environment Availability +> Audited this session (2026-10-23) by probing each dependency from this machine. + | Dependency | Required By | Available | Version | Fallback | |------------|------------|-----------|---------|----------| -| Node.js | test runner, build scripts | ✓ | **v24.18.0** | — | -| npm | workspace installs, scripts | ✓ | 11.16.0 | — | -| `grammy` | `/cari` handler, `sendVenue` | ✓ (installed) | **1.46.0** | — | -| `wrangler` | dev on 8787, deploy | ✓ (installed) | 4.151.3 | — | -| `typescript` | typecheck | ✓ (installed) | 5.9.3 | — | -| `vite` | web build (gazetteer copy) | ✓ (installed) | 8.3.5 | — | -| `playwright` + browsers | ONBD-02 capture evidence | ✓ (cached) | chromium-1248, webkit-2272, firefox-1511 | — | -| Cloudflare Workers | runtime | ✓ (account + deploy) | Free plan, 10 ms CPU / 100k req/day | — | -| `chat_sessions` KV | (not needed — no cache recommended) | ✓ (binding configured) | id `59f58a54…` | — | -| **Overpass API** (network) | D-06 top-up script | ✓ (reachable) | `overpass-api.de` — **returns 406 without a User-Agent header** (verified this session) | Other public instances: `overpass.kumi.systems`, `overpass.private.coffee`. The script sets a UA (Pitfall 4). | -| **Nominatim** (network) | D-06 fallback for a missed named place | ? (not probed this session) | `nominatim.openstreetmap.org` | Strict usage policy — 1 req/s, identifying UA, no bulk. Maintainer-tool use only. | -| **Real Android device** | ONBD-02/ONBD-03 real-client gate | **?** | — | **None.** The Phase-3 device matrix pattern applies: arm a checklist, let the user walk it. | -| **Real iOS device** | ONBD-03 `addToHomeScreen` (iOS status is always `unknown`) | **?** | — | **None.** Phase 3 recorded the iOS leg as UNVERIFIED (no device). | -| **Telegram Desktop** | Pitfall 2 (START-button trap) verification | **?** | — | **None.** The `url`-button choice sidesteps the trap; a Desktop spot-check still confirms it. | +| **Node.js** | everything (build, tests, corpus runner) | ✓ | `v24.18.0` | — | +| **npm** | workspace install | ✓ | `11.16.0` | — | +| **grammy** (installed) | bot handler | ✓ | `1.46.0` | — | +| **wrangler** (installed) | Worker deploy / local dev | ✓ | `4.151.3` | — | +| **TypeScript** (installed) | typecheck | ✓ | `5.9.3` | — | +| **Playwright** (installed, web) | capture evidence | ✓ | `1.50.0` | — | +| **curl** | top-up script | ✓ | `8.14.1` | — | +| **Python 3** | not needed this phase (theme gallery only) | ✓ | `3.13.5` | — | +| **Overpass API (`overpass-api.de`)** | D-06 top-up script (live) | ✓ **with a User-Agent** | **CORRECTED by the verification pass (see §Appendix A4):** the HTTP 406 is caused by the **missing `User-Agent` header**, not by an egress filter. Probed 2026-10-23 on the same host: `GET /api/status` **no UA → 406 / with UA → 200**; `GET /` **no UA → 406 / with UA → 200**; `POST /api/interpreter` **no UA → 406 / with UA → 200** (19 KB of real KL data, `osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z`). The script's `User-Agent` is therefore **load-bearing, not cosmetic**. | `overpass.kumi.systems` and `overpass.private.coffee` are **unreachable from this network** (connection failure, HTTP 000, with or without a UA); `overpass.osm.ch` returns **400** (Switzerland-only coverage, as the body noted). The script should still accept a `--source=` / env override and the automated test should use a **committed fixture**. Capacity 502s are intermittent — see §Appendix A4. | +| **Nominatim** (`nominatim.openstreetmap.org`) | coordinate verification of curated entries | ✓ | live (200; `?q=KLCC` returned correct KL nodes) | — | +| **Geofabrik extract** | bulk OSM fallback for the top-up script | ✓ | `asia/malaysia-singapore-brunei-latest.osm.pbf` (206 on range request) — note: **not** `malaysia-latest.osm.pbf` (404) | — | +| **Cloudflare account / wrangler auth** | deploy | ✗ (not probed — needs a real account) | — | Deploy is a manual exit step; Phase 2/3 precedent | +| **A real Telegram client** | ONBD-02 / ONBD-03 device validation | ✗ (not available here) | — | Phase 3's device-matrix precedent: a real-device session is an exit-gate obligation | **Missing dependencies with no fallback:** -- Real devices for the ONBD-02/ONBD-03 client matrix. The Phase-3 precedent (`.planning/phases/03-mini-app-shell-geolocation/03-DEVICE-MATRIX.md`) is the model: a written checklist with the deployed URL, checked by the user. **The planner must surface this as a checkpoint before the exit-gate task, not discover it at execution.** +- **A live `overpass-api.de` endpoint.** The Overpass *script* can still be written and tested against a committed fixture, but it cannot be validated end-to-end from this machine. If the phase's exit gate requires a live Overpass run, it must be done from a different network (or via the `overpass.osm.ch` mirror, which has no MY coverage). **Plan around this.** +- **A real device / Telegram client** for ONBD-02/ONBD-03 validation. This is inherent to the phase (Phase 3 set the same precedent with its device matrix). **Missing dependencies with fallback:** -- Overpass instance availability — three public instances exist; the script should accept an override. -- A real Telegram client for the "one message with venue card + buttons" confirmation — the bot-side unit tests cover the *payload shape*; the *rendering* needs a human. (This is assumption A3.) +- **Overpass** → Nominatim search, Geofabrik extract, or a committed fixture. +- **A fuzzy-distance dependency** → hand-roll (recommended) or `@crob/damerau-levenshtein@2.0.0`. --- ## Validation Architecture -> `.planning/config.json` → `workflow.nyquist_validation: true` (confirmed present and `true`). Included per protocol. +> `workflow.nyquist_validation` is `true` in `.planning/config.json` (not `false`), so this section is **required**. The repo has custom zero-dependency test harnesses (no Jest/Vitest/Mocha): `bot/test/runner.mjs`, `web/test/runner.mjs`, `theme/scripts/audit.test.mjs` — each auto-discovers `*.test.mjs` and drains a shared registry. ### Test Framework | Property | Value | |----------|-------| -| Framework | **Hand-rolled zero-dependency runner** — `bot/test/runner.mjs` + `bot/test/harness.mjs` (`test(name, fn)` registry). `web/test/runner.mjs` is the same shape (copied verbatim, workspace-agnostic). No vitest/jest — by design (repo precedent since Phase 1's `theme/scripts/audit.test.mjs`). | -| Config file | none (plain `.mjs` + `.ts` imported via dynamic `import()` — the runner already does `await import("../src/i18n/strings.ts")`, which works because Node 24 strips TypeScript types natively). | -| Quick run command | `npm test -w bot` (bot suites) / `npm test -w web` (web suites) / `npm run bot:test` + `npm run web:test` from the root | -| Full suite command | `npm run bot:test && npm run web:test && npm run bot:typecheck && npm run web:typecheck && npm run build -w theme && npm run build -w web` | -| Client/render checks | `npm run capture -w web` — Playwright against `web/dist-harness`; the existing harness already races `RAMS-BASEMAP-READY` → `ARAHARAH-LIVE` and enforces a `MIN_BYTES` blank-bar check. A `?dest=` boot profile slots straight into the existing capture list. | -| Corpus runner | **A new suite in the same shape**: `bot/test/corpus.test.mjs` — imports `bot/test/fixtures/golden-corpus.json` and the real `GazetteerSource`, asserts the D-14 gate. No new runner, no new framework. | - -**⚠️ ESM/TS caveat (carried from Phase 3's research):** the runner dynamic-imports `.ts` files directly. This works on Node 24 (type-stripping) and is the established repo pattern (`bot/test/commands.test.mjs` does `await import("../src/i18n/strings.ts")`). **The shared search module must therefore be importable from plain Node without a bundler** — i.e. no `import ... with { type: "json" }` assertion in the *source* module if it would break the Node import; load the JSON via `readFileSync` + `JSON.parse` against a path resolved from `import.meta.url` if the assertion proves fragile. **Test this early** — it is the single most likely mechanical blocker in the phase. (Note the corpus fixture itself is loaded by the *test*, not by the source module, so it can use a JSON import assertion without affecting production code.) +| **Framework** | Hand-rolled zero-dependency runner (repo-native pattern). `bot/test/harness.mjs` exports `test(name, fn)`; `bot/test/runner.mjs` auto-discovers `*.test.mjs`, runs each, prints `PASS`/`FAIL`, sets `process.exitCode = failedCount > 0 ? 1 : 0`. The `web/` runner is byte-for-byte the same shape. | +| **Config file** | None — no jest/vitest config exists. Suites are discovered by filename convention (`*.test.mjs`). | +| **Quick run command** | `npm run bot:test` (→ `node test/runner.mjs` in `bot/`) — runs **all** bot suites; there is no per-file filter today. `npm run web:test` for the web side. | +| **Full suite command** | `npm run bot:test && npm run web:test && npm run audit` (theme audit) — plus `npm run bot:typecheck && npm run web:typecheck`. | +| **Corpus run command** | `node bot/scripts/gazetteer-corpus.mjs` (to be created) — the D-14 bake-off close. Should be wired into `npm run bot:test` **or** run as an explicit gate; the plan must decide (recommend: a dedicated `npm run search:corpus` script, invoked by the phase's exit gate, and *also* asserted inside a `*.test.mjs` suite so `npm run bot:test` fails when the corpus regresses). | +| **Typecheck** | `npm run bot:typecheck` / `npm run web:typecheck` → `tsc --noEmit` per workspace. | ### Phase Requirements → Test Map | Req ID | Behavior | Test Type | Automated Command | File Exists? | |--------|----------|-----------|-------------------|-------------| -| **SRCH-01** | Query "KLCC" returns the KLCC entry in the top-3 | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-01** | The `SearchSource` interface is satisfied by `GazetteerSource` and the seam is the only entry point the handler uses | unit (structural) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-01** | A gazetteer entry with no match yields an **empty** array (not a nearest-neighbour fabrication) | unit | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-01** | `backendId` identifies the active backend — the escalation record needs it (D-14) | unit | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-01** | The top-3 hit rate over the corpus positives is **≥ 90%** — the D-14 threshold, as an executable assertion | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | "Jln Ampang" and "Jalan Ampang" normalize to the same string and resolve to the same entry id | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | "Tmn" ↔ "Taman", "Kg" ↔ "Kampung", "Prsn" ↔ "Presint" round-trip (bidirectional table) | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | Glued-word split: "KampungBaru" → "kampung baru"; "KLCC" unchanged | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | No mangling: "Sunway Pyramid" and "KLIA" survive normalization unchanged | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | Case/whitespace folding is idempotent | unit | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | A transposition typo ("Sunway Pyrmid") resolves to the same entry as the canonical spelling | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-02** | A missing-key typo ("Sunwy Pyramid") resolves to the same entry | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | The handler sends **exactly one** message per `/cari` — the venue card carries the reply markup | unit (payload shape, fake bot) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | The venue payload carries `latitude`, `longitude`, `title`, `address` (the four required `sendVenue` fields) | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | The OPEN button's URL carries the destination id and points at the Mini App origin | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | 2–4 alternates ride on the **same** message's inline keyboard (D-02) | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | A no-match query sends an honest refusal with **no** pin and no inline keyboard | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | The venue card's address/title carry the OSM attribution string (ODbL) | unit (copy gate) | `npm test -w bot` | ❌ Wave 0 | -| **SRCH-03** | The search copy exists in **both** `en` and `ms` with identical key-sets (the Phase-2 CS1 law extends) | unit (copy gate) | `npm test -w bot` | ❌ Wave 0 | -| **ONBD-02** | `dest_` maps to a destination in the webview's gazetteer copy — id → name/lat/lng/address | unit (web) | `npm test -w web` | ❌ Wave 0 | -| **ONBD-02** | A **missing or unknown** id degrades to the normal Phase-3 boot (no crash, no blank) | unit (web) | `npm test -w web` | ❌ Wave 0 | -| **ONBD-02** | The webview resolves the id from `location.search` **and** from `initData.start_param` | unit (web) | `npm test -w web` | ❌ Wave 0 | -| **ONBD-02** | The harness boot profile `?dest=` renders the destination preloaded (capture evidence) | integration (Playwright) | `npm run capture -w web` | ❌ Wave 0 | -| **ONBD-02** | The gazetteer id charset is enforced: every id matches `/^[a-z0-9-]+$/` | unit (schema gate) | `npm test -w bot` | ❌ Wave 0 | -| **ONBD-02** | The bot and web copies of the gazetteer expose the **same id set** (drift gate — the D-11 "in sync by construction" assertion, made executable) | unit | `npm test -w bot` (+ a web-side check) | ❌ Wave 0 | -| **ONBD-03** | `addToHomeScreen` is wired when available and is a **no-op** when the API is absent (Desktop web) | unit (web) | `npm test -w web` | ❌ Wave 0 | -| **ONBD-03** | `checkHomeScreenStatus` handles all four statuses, treating `unknown` as "possibly added" (iOS), never as failure | unit (web) | `npm test -w web` | ❌ Wave 0 | -| **ONBD-03** | A boot with **no** `start_param` and **no** `?dest=` lands in the Phase-3 staged boot (search-ready map) — no crash, no blank | unit (web) + integration (capture) | `npm test -w web`, `npm run capture -w web` | ❌ Wave 0 | -| **Privacy (D-15)** | No KV write in the search path; the query text never reaches a log line | unit (grep gate, mirroring the Phase-2 `privacy.test.mjs`) | `npm test -w bot` | ❌ Wave 0 | -| **Privacy (D-15/GEO-03)** | The webview's destination is **memory only** — no storage API referenced | unit (grep gate, `web/test/no-storage.test.mjs` precedent) | `npm test -w web` | ❌ Wave 0 | -| **Licensing (D-04)** | The gazetteer JSON carries an ODbL notice + OSM attribution; a `LICENSE` (GPL-3.0) exists | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 | -| **D-09 (structural)** | The abbreviation table is data, not code — the normalizer reads it from a table module | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 | -| **D-11 (structural)** | No crypto/HMAC/signature in the deep-link payload path | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 | -| **Corpus integrity** | Every corpus positive's `expect` id exists in the gazetteer; every id is unique; ids match the charset | unit (schema gate) | `npm test -w bot` | ❌ Wave 0 | -| **Real-device gate** | ONBD-02/ONBD-03 client behaviour: single-tap OPEN, home-screen shortcut, iOS `unknown` | **manual-only** | — | n/a — Phase-3 device-matrix pattern | -| **SRCH-03 rendering** | The venue card + inline buttons render correctly in a real Telegram client | **manual-only** | — | n/a — assumption A3 | +| **SRCH-01** | "KLCC" (and the golden landmarks) resolve to the expected entry; the shortlist's first item is the right one | unit (corpus) | `node bot/scripts/gazetteer-corpus.mjs` → asserts top-1/top-3 ≥ threshold | ❌ **Wave 0** — `bot/scripts/gazetteer-corpus.mjs` | +| **SRCH-01** | The gazetteer module loads, every entry has the required fields, ids are unique and slug-safe, every entry has ≥1 alias | unit | `npm run bot:test` (suite `bot/test/gazetteer.test.mjs`) | ❌ **Wave 0** — `bot/test/gazetteer.test.mjs` | +| **SRCH-02** | "Jln Ampang" and "Jalan Ampang" normalize to the same string; "Tmn"→"taman"; "Kg"→"kampung"; "Prsn"→"presint"; glued "tamanbangsar" splits | unit (rule vectors) | `npm run bot:test` (suite `bot/test/normalizer.test.mjs`) | ❌ **Wave 0** — `bot/test/normalizer.test.mjs` | +| **SRCH-02** | Every abbreviation-table row has a round-trip vector (query form → expected canonical) | unit | `npm run bot:test` (suite `bot/test/normalizer.test.mjs`) | ❌ **Wave 0** | +| **SRCH-02** | Transposition typos ("KCLC") and substitution typos ("Sunwya Pyarmid") resolve to the expected entry | unit (corpus) | `node bot/scripts/gazetteer-corpus.mjs` | ❌ **Wave 0** | +| **SRCH-02** | The Damerau-Levenshtein implementation matches hand-computed OSA distances (test vectors from the literature: `distance("ab","ba")==1`, `distance("a cat","an act")==2`) | unit | `npm run bot:test` (suite `bot/test/distance.test.mjs`) | ❌ **Wave 0** — `bot/test/distance.test.mjs` | +| **SRCH-02** | Exact-match tier strictly outranks any fuzzy hit; scores under the min-length guard return 0 | unit | `npm run bot:test` (suite `bot/test/score.test.mjs`) | ❌ **Wave 0** | +| **SRCH-03** | The bot sends a `sendVenue` with the right latitude/longitude/title/address, plus an inline keyboard containing the OPEN button and 2–4 alternates | integration (source-introspection, matching the repo's `commands.test.mjs` CS4 style) | `npm run bot:test` (suite `bot/test/search-command.test.mjs`) | ❌ **Wave 0** — `bot/test/search-command.test.mjs` | +| **SRCH-03** | The venue-card builder's payload is structurally correct (latitude/longitude are finite numbers; title non-empty; address non-empty) | unit | `npm run bot:test` (same suite) | ❌ **Wave 0** | +| **SRCH-03** | The alternates' `callback_data` is `"alt:"`, ≤64 bytes, and carries **no** query text, name, or coordinates | unit (grep + parse) | `npm run bot:test` (same suite) | ❌ **Wave 0** | +| **SRCH-03** | `answerCallbackQuery` is called on **every** callback path (the 30-second rule) | source-introspection | `npm run bot:test` (same suite) | ❌ **Wave 0** | +| **SRCH-03** | A negative query ("Kuching", "Penang") yields an **empty** shortlist and the "not in v1 area" copy — never a fabricated pin | unit + corpus | `node bot/scripts/gazetteer-corpus.mjs` + `npm run bot:test` | ❌ **Wave 0** | +| **SRCH-03** | The search copy exists in **both** en and ms, and the locale key-sets are identical (the Phase 2 CS1 gate, extended) | unit | `npm run bot:test` (extend `bot/test/commands.test.mjs` CS1 or add a new suite) | ✅ **extend** — `bot/test/commands.test.mjs` | +| **ONBD-02** | The webview resolves a `dest` id from its own gazetteer copy to a destination with valid lat/lng | unit | `npm run web:test` (suite `web/test/destination.test.mjs`) | ❌ **Wave 0** — `web/test/destination.test.mjs` | +| **ONBD-02** | An **unknown** id produces an honest "destination unavailable" state, not a blank map | unit | `npm run web:test` (same suite) | ❌ **Wave 0** | +| **ONBD-02** | The bot's gazetteer id set == the web's gazetteer id set (drift gate) | integration (cross-workspace) | `npm run bot:test` (a suite that reads both JSON files) | ❌ **Wave 0** | +| **ONBD-02** | The bot bundle and the web bundle both contain the gazetteer JSON (build-artifact gate) | integration | `npm run web:test` (extend `web/test/build-artifact.test.mjs`) | ✅ **extend** — `web/test/build-artifact.test.mjs` | +| **ONBD-02** | The mock harness can simulate a **preloaded-destination boot** (`?dest=` profile) and the capture run shows the destination state | manual + capture | `npm run web:capture` (extend `web/scripts/capture.mjs`) | ✅ **extend** — `web/scripts/capture.mjs` | +| **ONBD-02** | The deep-link carrier actually delivers the id on a real device (Android + iOS) | **manual-only** | *(no real device in this environment — exit-gate obligation)* | ❌ **exit gate** | +| **ONBD-03** | The Mini App exposes `addToHomeScreen` / `checkHomeScreenStatus` on the `TelegramWebAppLike` interface and calls them behind a capability guard (`isVersionAtLeast("8.0")`) | unit | `npm run web:test` (suite `web/test/homescreen.test.mjs`) | ❌ **Wave 0** — `web/test/homescreen.test.mjs` | +| **ONBD-03** | A startapp-less boot lands in the search-ready map (Phase 3's staged boot, no destination) — i.e. the pre-existing path is unchanged | integration | `npm run web:capture` + `npm run web:test` | ✅ **extend** — existing Phase 3 suites | +| **ONBD-03** | `checkHomeScreenStatus` returning `"unknown"` (iOS) is treated as "supported but indeterminate" — **not** as an error | unit | `npm run web:test` (same suite) | ❌ **Wave 0** | +| **ONBD-03** | Adding araharah to the home screen launches the app into the search-ready state | **manual-only** | *(real device — exit-gate obligation)* | ❌ **exit gate** | +| **D-14** (bake-off close) | The corpus runner exits non-zero when top-3 < 90% **or** any negative produced a hit | integration | `node bot/scripts/gazetteer-corpus.mjs` | ❌ **Wave 0** | +| **D-14** | The corpus file is **hand-authored**, not generated: at least N cases are not string-equal to any alias in the gazetteer (a tautology check) | unit (structural) | `npm run bot:test` (suite `bot/test/gazetteer.test.mjs`) | ❌ **Wave 0** | +| **D-04** (forkability law) | A `LICENSE` file exists at the repo root containing GPL-3.0 | unit (structural) | `npm run bot:test` (a small suite) | ❌ **Wave 0** | +| **D-04 / licensing** | The gazetteer data file carries an ODbL attribution notice, and the repo README/attribution names OpenStreetMap | unit (structural) | `npm run bot:test` (same suite) | ❌ **Wave 0** | +| **D-15** (privacy) | **No** log statement in `bot/src/` writes a search query. Grep gate over `bot/src/**` for log calls whose arguments reference a query-shaped variable | unit (grep, matching the repo's existing source-introspection style) | `npm run bot:test` | ❌ **Wave 0** | +| **D-15** | The web side writes no destination or query to any storage surface | unit (existing grep gate) | `npm run web:test` (`web/test/no-storage.test.mjs`) | ✅ **extend** — `web/test/no-storage.test.mjs` | +| **D-06** (top-up script) | The script produces a valid-shaped proposal file from a **committed fixture** of a real Overpass response | unit (fixture-driven) | `npm run bot:test` (suite `bot/test/gazetteer-topup.test.mjs`) | ❌ **Wave 0** — `bot/test/gazetteer-topup.test.mjs` | +| **D-06** | The script is **not** imported by the Worker (grep: no overpass/OSM URLs under `bot/src/`) | unit (grep) | `npm run bot:test` (same suite) | ❌ **Wave 0** | +| **D-05** (backend #2 reachability) | The JSON→SQLite converter script reads the same committed JSON and emits a SQLite file with the same ids | unit | `npm run bot:test` (suite `bot/test/gazetteer-converter.test.mjs`) | ❌ **Wave 0** | +| **D-07** (V8-native JSON) | The Worker bundle contains the gazetteer as inlined JSON — no SQLite/WASM payload | integration | `npm run bot:test` (extend the build-artifact gate or a bundle-size assertion) | ❌ **Wave 0** | +| **D-03** (SearchSource seam) | The interface has exactly the agreed methods and the gazetteer implements it; a **stub** backend can be substituted in a test (proving the seam) | unit | `npm run bot:test` (suite `bot/test/search-source.test.mjs`) | ❌ **Wave 0** | ### Sampling Rate -- **Per task commit:** `npm run bot:test` (search module, normalizer, corpus, handler) and `npm run web:test` (destination resolution) — whichever workspace the task touched. -- **Per wave merge:** `npm run bot:test && npm run web:test && npm run bot:typecheck && npm run web:typecheck && npm run build -w theme && npm run build -w web` -- **Phase gate:** full suite green **AND** the corpus gate green at ≥90% **AND** `npm run capture -w web` produced the ONBD-02 evidence capture **AND** the real-device client matrix walked by the user (manual, blocking — Phase-3 precedent). +- **Per task commit:** `npm run bot:test && npm run web:test` (fast — the hand-rolled runner executes in well under a second) **plus** `npm run bot:typecheck && npm run web:typecheck`. +- **Per wave merge:** the same, **plus** `node bot/scripts/gazetteer-corpus.mjs` (the corpus must be ≥ threshold at every wave merge, not just at the end — this is what makes the escalation ladder actionable rather than a last-minute scramble). +- **Phase gate:** full suite green (`bot:test`, `web:test`, `theme audit`, both typechecks) **and** the corpus runner green, **and** the two manual-only items (ONBD-02 device deep-link, ONBD-03 home-screen launch) verified on a real device, **and** the D-14 decision recorded in the repo. ### Wave 0 Gaps -- [ ] `bot/src/search/` module tree — `types.ts`, `gazetteer.ts`, `normalize.ts`, `rules/abbreviations.ts`, `score.ts`, `source.ts` (the shared module; Wave 0 because everything depends on it) -- [ ] `bot/src/search/data/kl-gazetteer.json` — the seed gazetteer (~120–150 entries) with ODbL header + attribution -- [ ] `bot/src/bot/commands/cari.ts` — the `/cari` handler -- [ ] `bot/test/normalizer.test.mjs` — per-rule vectors (D-09) -- [ ] `bot/test/corpus.test.mjs` — the D-13/D-14 executable gate -- [ ] `bot/test/fixtures/golden-corpus.json` — the pinned corpus -- [ ] `bot/test/cari.test.mjs` — payload-shape gates (one message, venue fields, OPEN url, alternates, honest refusal, attribution) -- [ ] `bot/scripts/overpass-topup.mjs` — the D-06 repo tool -- [ ] `web/src/search/resolve-destination.ts` — id → destination (memory only) -- [ ] `web/test/destination-preload.test.mjs` — ONBD-02 gates -- [ ] `web/index.html` — the `?dest=` harness boot profile -- [ ] `bot/src/i18n/strings.ts` — MODIFIED: search copy en/ms + `/cari` in `helpMenu` + the OSM attribution string -- [ ] `bot/test/commands.test.mjs` — MODIFIED: CS5/CS6 must now expect `/cari` in the menu (the Phase-2 gap-freedom gate is a **hard gate that will fail** until this is updated) -- [ ] `LICENSE` (GPL-3.0) + `LICENSES.md` or an ODbL notice for the data file -- [ ] Shared-module wiring — a fourth npm workspace (e.g. `@araharah/search`) so `bot/` and `web/` import one copy +Everything below is new for this phase (the repo has **no** search-related test infrastructure today). -*(No gap: the runner and harness already exist in both workspaces. Every suite listed above is new.)* +- [ ] `bot/test/gazetteer.test.mjs` — entry schema, unique ids, alias presence, id-set parity with the web copy, corpus-is-not-tautological, LICENSE + ODbL notice present +- [ ] `bot/test/normalizer.test.mjs` — per-rule test vectors for the full abbreviation table (D-09: *"Per-rule test vectors; extends by table edit, not code change"*), folding, glued-word split, Presint/Prsn +- [ ] `bot/test/distance.test.mjs` — Damerau-Levenshtein (OSA) correctness vectors +- [ ] `bot/test/score.test.mjs` — tier disjointness, min-length guard, exact-outranks-fuzzy invariant +- [ ] `bot/test/search-command.test.mjs` — `sendVenue` payload shape, inline keyboard contents, `callback_data` format/length/no-leakage, `answerCallbackQuery` on every path, negative-query copy +- [ ] `bot/test/search-source.test.mjs` — the `SearchSource` interface + a stub backend proving the seam +- [ ] `bot/test/gazetteer-topup.test.mjs` — the Overpass proposal script against a committed fixture; the no-OSM-URL-in-Worker grep gate; **the `User-Agent` header is asserted to be set** (missing UA → 406 on every endpoint — Pitfall 6b); **502/429/timeout → exit 0 with a "busy" message** (asserted against a fake `fetch`, never the live service) +- [ ] `bot/test/gazetteer-converter.test.mjs` — the D-05 JSON→SQLite converter (id set preserved) +- [ ] `bot/scripts/gazetteer-corpus.mjs` — the D-14 corpus runner (the bake-off close) +- [ ] `bot/test/corpus-gate.test.mjs` (or fold into the above) — asserts `npm run bot:test` fails when the corpus regresses +- [ ] `web/test/destination.test.mjs` — id → destination resolution, unknown-id honest state +- [ ] `web/test/homescreen.test.mjs` — `addToHomeScreen` / `checkHomeScreenStatus` capability guard, `"unknown"` handling +- [ ] `shared/data/golden-corpus.json` — the pinned D-13 corpus (~60–100 hand-written cases) +- [ ] `shared/data/kl-gazetteer.json` — the curated entries (~200–500) with a committed Overpass-response fixture for the top-up script +- [ ] `bot/test/fixtures/overpass-response.json` — a real Overpass JSON response, committed so the top-up test is deterministic +- [ ] **Framework install:** none needed — the repo's hand-rolled runner discovers `*.test.mjs` automatically. **No Jest/Vitest/Mocha.** +- [ ] **Extend (not new):** `bot/test/commands.test.mjs` (CS1 locale parity for the new search strings), `web/test/no-storage.test.mjs` (destination/query non-persistence), `web/test/build-artifact.test.mjs` (gazetteer JSON present in the web bundle) --- ## Security Domain -> `security_enforcement` is absent from `.planning/config.json` → treated as enabled. Included per protocol. +> `security_enforcement` is absent from `.planning/config.json` → treated as **enabled**. This section is required. ### Applicable ASVS Categories | ASVS Category | Applies | Standard Control | |---------------|---------|-----------------| -| V2 Authentication | yes | The webview's `initData` is HMAC-validated by the **existing** Phase-2 `POST /api/session` (never re-implemented). A `start_param` read from `initDataUnsafe` is untrusted unless `/api/session` echoes the validated value (Open Question 2). | -| V3 Session Management | yes | Unchanged from Phase 2 — opaque tokens, TTL-only KV, rotate-on-new. The search path adds **no** session or cache state. | -| V4 Access Control | yes | The search module is pure and has no I/O; the `/cari` handler answers only the chat the command came from (`ctx.chat.id` from the update, never from user input). No endpoint accepts a search query from the browser. | -| V5 Input Validation | yes | The query string is length-capped before normalization (a hostile 100 KB argument must not reach the scorer); the gazetteer id is validated against `/^[a-z0-9-]+$/`; the `start_param` is checked against the Telegram charset before it is used as a map key. | -| V6 Cryptography | **no** (by design) | D-11: *"No crypto: the payload's only threat is a user pranking themselves."* The deep-link payload is a plain id into a public gazetteer. **A structural gate must assert that no HMAC/signature code enters the search or deep-link path.** | -| V9 Communications | yes | The Overpass script is a repo tool with an identifying User-Agent; it is never invoked at runtime. The Worker makes **no** outbound search requests (backend #1 is in-Worker). | -| V14 Configuration | yes | No new env vars, no new secrets (D-04's "env only" law already holds). The Mini App origin the OPEN button points at is **not** a secret — it is public and already in `bot/wrangler.jsonc` / the deployed URL. | +| **V2 Authentication** | **yes (inherited)** | The Mini App's session gate is Phase 2's `bot/src/security/initdata.ts` HMAC validation + `POST /api/session`. This phase **adds no new identity surface**. The destination id is *not* an identity — it is a public place id (D-11 explicitly says no crypto is needed because the payload's only threat is a user pranking themselves). | +| **V3 Session Management** | **yes (inherited, unchanged)** | Phase 2's 900 s memory-only session token. The destination must NOT be written to the session token's storage (there is none) nor to the token's payload (it would leak a user choice into a credential). **Recommendation: the destination lives in module scope in the webview, alongside `lastFix`, exactly like the existing in-memory-only pattern.** | +| **V4 Access Control** | **yes (inherited, unchanged)** | The Worker owns `/api/*`. This phase adds **no new endpoint** — the destination is resolved entirely from the webview's local copy of the shared gazetteer (D-11). That is the *right* design: no endpoint means no access-control surface. If the plan adds an endpoint (e.g. `/api/destination?id=…`), it must derive anything user-specific from the resolved claim, never from the request body — the Phase 2 `handleLocation`/`handleLocationFallback` precedent. | +| **V5 Input Validation** | **yes (new surface)** | The search query is untrusted user input. Rules: (a) the `/cari` handler must bound the query length (e.g. ≤128 chars) **before** any processing; (b) the `callback_data` id must be validated against `/^[a-z0-9-]{1,64}$/` **before** the gazetteer lookup; (c) the webview's `dest` id must be validated with `/^[A-Za-z0-9_-]{1,64}$/` **before** the lookup; (d) the `start_param` charset is officially restricted to `[A-Za-z0-9_-]` and ≤512 chars — validate against exactly that. **Never** interpolate an unvalidated id into a query, a URL, or a log line. | +| **V6 Cryptography** | **no new crypto — by design** | D-11: the payload needs none. The existing HMAC path is unchanged. **Do not introduce a signed-token scheme** for the destination — it would be security theatre against a self-inflicted threat and would break the "no crypto, shared module by construction" design. | +| **V7 Error Handling / Logging** | **yes (binding)** | D-15 + Phase 2's D-14 law: **category-only logs**. A failed search logs `[search] rejected: no-match` — never the query, never the results, never a user id, never a timestamp. The "not in v1 area" copy must be honest (D-13's negatives) and must not fabricate a pin. | +| **V8 Data Protection / Privacy** | **yes (binding)** | The query is user payload and is **not persisted** (D-15). If the plan adds KV result caching (D-15 permits it: TTL-only, short TTL, keyed by *normalized* query), note: the key contains the normalized query — i.e. **the query text is the KV key**, which is arguably persistence of the query. **Recommendation: skip the KV cache entirely.** At ~2500 candidate pairs the search is microseconds; a cache buys nothing and creates a privacy surface D-15 explicitly does not want. (D-15 makes the cache optional: *"if any"*, and OpenCode's Discretion says *"perf is likely fine without at this scale"* — this research confirms that judgment.) | -### Known Threat Patterns for Cloudflare Workers + Telegram + a curated gazetteer +### Known Threat Patterns for this stack (grammY + Cloudflare Workers + Telegram Mini App) | Pattern | STRIDE | Standard Mitigation | |---------|--------|---------------------| -| **Query-text leakage** (the query is user payload — it can reveal where a user is going or where they live) | Information Disclosure | D-15: category-only logging; the query is never written to KV or to a log line. A grep gate pins this (Phase-2 `privacy.test.mjs` precedent). **Do not add a result cache** (a normalized-query cache key is an activity fingerprint even when the content is public). | -| **Forged deep link** (a user crafts `?dest=`) | Tampering | No impact — the id resolves to a *public place*. D-11's own assessment. The id is not a capability and grants nothing. | -| **Gazetteer id injection** (an id containing characters that break the URL or the `start_param` charset) | Tampering | Enforce `/^[a-z0-9-]+$/` on every entry id at load time; reject the entry (fail closed) rather than emit a bad URL. | -| **Hostile query argument** (megabyte-long string to burn CPU) | DoS | Cap the raw query length before normalization (e.g. 256 chars — no real destination exceeds it). Cap the normalized length too. The tier-1 hash lookup is O(1) regardless. | -| **CPU exhaustion via the scorer** | DoS | The three-tier cascade bounds the expensive algorithm to a handful of candidates. Workers Free's 10 ms CPU limit is a backstop. `wrangler dev` can measure the real figure. | -| **Overpass abuse** (a maintainer script that hammers the public instance) | DoS / reputation | One request per run, identifying User-Agent, manual invocation only. Overpass's per-IP rate limit (2 slots, requests queued up to 15 s) is the guardrail. | -| **Stale-webview dataset** (the web bundle's gazetteer lags the bot's) | Tampering (functional) | Structural: one shared module (Option A) + an id-set drift gate in the test suite. The `npm run deploy` chain rebuilds both from one source. | -| **Cross-chat leakage** (a search result sent to the wrong chat) | Information Disclosure | The handler replies to `ctx.chat.id` from the update only; no chat id is accepted from user input. | -| **XSS / markdown injection** (a query echoed back into a reply) | Tampering | Telegram's parse mode handles it, but the safest design is to **not echo the query** in the reply at all (the clinical Rams voice says what it found, not what was typed). If the query must appear, escape it or send without a parse mode. | +| **Query text persisted in a log or a KV key** | Information Disclosure | D-15 category-only logging; a grep gate over `bot/src/**`; **skip the KV cache** (see V8). | +| **Callback-data injection** (`callback_data` carrying a name/query) | Tampering | `callback_data` is **only** `"alt:"`; the id is regex-validated and looked up in the gazetteer. Never echoed into a reply without validation. | +| **Destination id injection into the webview** | Tampering | `/^[A-Za-z0-9_-]{1,64}$/` before the lookup; an unknown id yields an honest "unavailable" state, never a fabricated coordinate. | +| **Open redirect / unvalidated Mini App URL** | Spoofing | The `web_app` button URL is built from a **configured origin**, never from user input. The plan must not let a query string influence the URL's origin. | +| **Fabricated location pin for an out-of-area query** | Repudiation (honesty failure) | D-13's negatives: an empty result must produce the honest refusal copy, never the nearest KL entry. This is a *correctness and honesty* requirement, not just a UX one — and it is directly asserted by the corpus runner. | +| **Gazetteer id drift between bot and webview** | Integrity | The id-set parity gate (§"Phase Requirements → Test Map") + ids are never renamed, only added. | +| **Overpass fetch inside the Worker** (SSRF-ish + availability) | Denial of Service | The Overpass script is a **repo tool**, never imported by `bot/src/` — enforced by a grep gate. | +| **OSM/ODbL attribution omission** | Compliance | The gazetteer carries an ODbL notice + attribution; the repo README names OpenStreetMap. | +| **`answerCallbackQuery` not called** | Denial of Service (UX spinner) | The repo's existing convention (`bot/src/bot/handlers/location-fallback.ts` pattern) plus a catch-all `bot.on("callback_query:data", …)` that always answers — grammY's documented recommendation. | +| **Malicious `start_param` reaching the webview** | Tampering | `initDataUnsafe` is **untrusted** by Telegram's own docs. Use it only to look up a public place id in a local gazetteer; never as identity, never as a URL, never as a path. | --- ## Sources ### Primary (HIGH confidence) -- **Official Telegram Bot API** — (fetched 2026-10-23): `addToHomeScreen()` / `checkHomeScreenStatus()` and the four statuses; the `start_param` field and the `tgWebAppStartParam` GET parameter; the `t.me?startapp=` deep-link form; inline vs direct-link Mini App behaviour; `mode=compact`. -- **Official Telegram Bot API reference** — (fetched 2026-10-23): method list; `sendVenue` exists; `setMyCommands` scope rules. -- **Installed `grammy@1.46.0` + `@grammyjs/types`** — read directly from `node_modules` this session: `api.d.ts:410` (`sendVenue` signature), `context.d.ts:679-690` (`replyWithVenue`), `methods.d.ts:735-782` (`sendVenue` args including `reply_markup`), `markup.d.ts:11-60` (`InlineKeyboardButton` variants incl. `UrlButton` and `WebAppButton`), `markup.d.ts:219-222` (`WebAppInfo { url: string }` — **only** `url`), `message.d.ts:1072-1086` (`Venue` fields incl. `foursquare_id` / `foursquare_type` / `google_place_id` / `google_place_type`). -- **Cloudflare Workers Limits** — (fetched 2026-10-23, page dated 2026-10-12): Free plan **10 ms CPU / HTTP request**, 100,000 req/day, 128 MB memory, 64 MiB uncompressed bundle, **1 second startup**, "avoid expensive work in global scope", "Most Workers consume ~2.2 ms per request". -- **OSMF Legal FAQ** — : derivative-database definition; the requirement to make the derivative database available. -- **OSMF Licence/Use Cases** — (rev. 2025-01-28): Case 3 — *"if you have any data that was derived from OSM… then you are making a derivative database"*; Case 2 — a pure-OSM publication is a "produced work" needing attribution. -- **OSMF Trivial Transformations Guideline** (endorsed by the OSMF board 2014-06-06) — : *"A 'geocoding database' should **NOT** be considered a trivial transformation."* -- **OSMF Licence Compatibility** — : identical-terms licences generally incompatible with ODbL. -- **UPU S42 Malaysian address standard** — : the canonical Malaysian address grammar (primary thoroughfare type = Jalan / Taman / Kampung; "Persiaran"; "Presint"; district level = Taman / Kampung / Seksyen) with worked address examples. -- **OSM Overpass QL reference** — : `bbox` order (south,west,north,east); `nwr` filter; `out center`; `out:csv`/`out:json`; `[timeout:N]` / `[maxsize:N]`. -- **Overpass API User's Manual — Rate Limit** — : 2 slots/IP, requests queue up to 15 s, HTTP 429 when denied; small-request priority; identifying UA recommended. -- **Malaysian address data** — UPU addressing unit () and the OSM-derived KL road list () — for the abbreviation vocabulary and the "Jalan / Lorong / Persiaran" naming pattern. -- **Repo codebase** — read directly this session: `bot/src/{webhook.ts,bot/bot.ts,config.ts,i18n/strings.ts,kv/sessions.ts}`, `bot/test/{runner.mjs,harness.mjs,commands.test.mjs}`, `bot/wrangler.jsonc`, `web/src/{main.ts,telegram/webapp.ts,i18n/strings.ts,mock/telegram-mock.ts}`, `web/index.html`, `web/test/runner.mjs`, `web/scripts/capture.mjs`, and the Phase-2/Phase-3 summaries and RESEARCH files. + +- **https://core.telegram.org/bots/api/methods/sendvenue.md** — the exact `sendVenue` parameter set. Fetched this session. Confirms: `latitude`, `longitude`, `title`, `address` are **required**; there is **no `text` parameter**; `reply_markup` accepts `InlineKeyboardMarkup`. +- **https://core.telegram.org/bots/api/types/inlinekeyboardbutton.md** — `InlineKeyboardButton`. Fetched this session. Confirms: `callback_data` is **1–64 bytes**; `web_app` is a `WebAppInfo`; `web_app` is *"Available only in private chats between a user and the bot"*; exactly one of the mutually-exclusive fields must be set. +- **https://core.telegram.org/bots/api/types/webappinfo.md** — `WebAppInfo.url` is *"An HTTPS URL of a Web App to be opened with additional data"*. Fetched this session. +- **https://core.telegram.org/bots/api/types/venue.md** — the `Venue` object shape. Fetched this session. +- **https://core.telegram.org/bots/webapps** — Mini Apps launch surfaces, `WebAppInitData` field semantics, `addToHomeScreen()` / `checkHomeScreenStatus()` / `homeScreenAdded` / `homeScreenChecked` (Bot API 8.0+). Fetched this session. **The source of the `start_param` scoping ambiguity** (see §"Deep-link risk"). +- **https://docs.telegram-mini-apps.com/platform/start-parameter** — the start-parameter charset (`[A-Za-z0-9_-]`, ≤512 chars) and the regex `/^[\w-]{0,512}$/`. Fetched this session. +- **https://docs.telegram-mini-apps.com/platform/launch-parameters** — `tgWebAppStartParam` is *"a custom string value passed in the bot or application link"*. Fetched this session. +- **https://opendatacommons.org/licenses/odbl/1-0/** — the ODbL 1.0 legal text. Fetched this session. §2.3(a) (programs excluded), §4.2/§4.3 (notices + produced-work attribution), §4.4 (share-alike on a Derivative Database), §4.6 (machine-readable offer). +- **https://osmfoundation.org/wiki/Licence/Attribution_Guidelines** — OSMF's safe-harbour attribution rules. Fetched this session. Confirms the database case: *"include attribution to OpenStreetMap and either the text of the ODbL or a link to it as part of the database, derivative database, or database as part of a collective database… in a location where users would be likely to look for it, such as a readme file, or within the data or metadata."* +- **https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations** and **the JUPEM gazetteer PDFs** (`mygeoname.mygeoportal.gov.my/exportpdf00.jsp?kodnegeri=16`) — the **official Malaysian abbreviation tables** (street prefixes + locality prefixes + the 48-entry gazetteer abbreviation list). Fetched this session. The normative source for the normalizer's rule table. +- **https://raw.githubusercontent.com/osm-search/Nominatim/master/settings/icu-rules/variants-ms.yaml** — the machine-readable transcription of the JUPEM table (shared / MY / SG / BN sections). Fetched this session. **Reproduced verbatim in §"Malay normalizer" via the code example.** +- **https://github.com/osm-search/Nominatim/pull/4056** — the PR that expanded the table from 8 entries to 65+, with the JUPEM + OSM-wiki source citations and the verified test queries ("Jln Bukit Bintang" → "Jalan Bukit Bintang, Kuala Lumpur", "Tmn Tun Dr Ismail" → "Taman Tun Dr Ismail, Kuala Lumpur", …). Fetched this session. +- **https://jupem.gov.my/v1/wp-content/uploads/2016/08/Toponymic-Guidelines-For-Map-and-Other-Editors-for-International-Use.pdf** — JUPEM's Toponymic Guidelines (2017, First Edition), the authority behind the table. +- **https://govdocs.sinarproject.org/documents/ministry-of-international-trade-and-industry/department-of-standards-malaysia/ms-2039-2006** — **MS 2039:2006** (Malaysian address standard). Fetched this session. §5.2.3.2 (the "abbreviation shall not be applied to names" rule) and Annex A (the prefix abbreviation tables A1–A5). **The direct authority for Pitfall 3.** +- **https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL** — Overpass QL reference: `bbox` ordering (s,w,n,e), `nwr`, `out tags`/`out center`, `out count` (the empty-result-detection idiom). Fetched this session. +- **https://grammy.dev/plugins/keyboard** — grammY inline-keyboard + callback-query handling, including the catch-all `bot.on("callback_query:data")` recommendation. Fetched this session. +- **https://www.npmjs.com/package/@crob/damerau-levenshtein` / `npm view` — the distance-function alternatives and their versions. Verified against the registry this session. ### Secondary (MEDIUM confidence) -- **Telegram-iOS issue #1729** — : *"Home screen status detection always returns 'unknown' on iOS… On iOS it is not possible to determine whether a Mini App has been added to the Home Screen."* (Official client repo issue; the OSM wiki pages above also carry a community-guideline disclaimer.) -- **telegramdesktop issue #28694** — : the Desktop START-button requirement for `t.me` deep links. -- **Telegram-Mini-Apps issue #27** — : service workers do not work in Telegram Mini Apps on iOS (still open). -- **Telegram Mini Apps platform docs** — and : the `start_param` charset (`/^[\w-]{0,512}$/`), the `home_screen_added` / `home_screen_checked` events and the four statuses. -- **Cohen, W. W.; Ravikumar, P.; Fienberg, S. E. (2003)**, *A Comparison of String Distance Metrics for Name-Matching Tasks*, KDD Workshop — : Jaro/Jaro-Winkler are "intended primarily for short strings"; prefix weighting behaviour. -- **taxify fuzzy-matching vignette** — : Damerau-Levenshtein charges 1 edit for a transposition (Levenshtein charges 2); Jaro-Winkler helps on 3–6-character names; threshold guidance. -- **RapidFuzz reference** — : normalized Damerau-Levenshtein semantics (0..1) used to define the tier-3 similarity conversion. -- **npm registry probes** (2026-10-23): `fuse.js` → 7.5.0, `fastest-levenshtein` → 1.0.16, `string-similarity` → 4.0.4, `levenshtein-edit-distance` → 3.0.1 (all rejected); `grammy` 1.46.0, `wrangler` 4.151.3, `vite` 8.3.5, `typescript` 5.9.3, Node 24.18.0, npm 11.16.0 (all already installed). -### Tertiary (LOW confidence — marked for validation) -- **The exact `sendVenue` + `reply_markup` rendering on iOS/Android/Desktop clients** — the API surface is verified, the rendering is not device-tested (assumption A3). -- **The `url` inline button opening the Mini App with its query string intact on every client** — not device-tested (assumption A4). -- **The exact KL bounding box** for the Overpass top-up (assumption A2). -- **The sufficiency of the abbreviation table** for KL (assumption A1) — the corpus runner is the instrument that settles it. +- **https://core.telegram.org/constructor/keyboardButtonWebView** — the MTProto schema confirming that an inline-button Mini App launch passes `url` to `messages.requestWebView`. Supports the read that the button's URL is used as given. +- **https://github.com/TelegramMessenger/Telegram-iOS/issues/1729** — `checkHomeScreenStatus()` always returns `"unknown"` on iOS; closed with the maintainer comment *"On iOS it is not possible to determine whether a Mini App has been added to the Home Screen."* Confirms the ONBD-03 platform limitation. +- **https://stackoverflow.com/questions/76186178/how-to-pass-parameters-to-telegram-web-app** — community confirmation that query params on a `web_app` button URL reach the webview. Corroborates Carrier A but is **not** official. +- **https://www.postgresql.org/docs/current/pgtrgm.html` + the fuzzy-matching comparison articles (tektite, fuzzypoint, taxify vignette) — the trigram/Jaro-Winkler/Damerau trade-off analysis underpinning the tier design and Pitfalls 4–5. +- **https://github.com/cbeier-studio/xolmis** — the live precedent for a GPL-3.0 project shipping ODbL data in the same repo with an explicit dual-license note. +- **https://github.com/ahozir-ux/bm-address-normaliser` — an existing Malay address normaliser (JS + Python + a Cloudflare Worker), useful as a cross-check on the abbreviation set and on the `KG`-ambiguity handling. Community-maintained; treat as a lead, not an authority. +- **Real commercial Malaysian address listings** (Maybank2U merchant PDF, OMRON Healthcare retailer PDF, Petronas station lists, Malaysia.worldplaces, the myboost KL outlet PDF) — the empirical evidence for **Presint/Prsn** usage ("JALAN P8G PRESINT 8", "PRSN RESIDEN DESA PARKCI", "Precint 15", "PRESINT 16") and for the everyday abbreviation forms locals actually type. + +### Tertiary (LOW confidence — flagged for validation) + +- **Stack Overflow #76486463** — an older (2023) claim that a start param is *"limited to 64 Symbols"*. The official docs now say **512**. Use 512. Flagged because the stale figure could mislead. +- **Community assertions that `start_param` is populated for direct-link launches** — the official `WebAppInitData.start_param` wording is narrower than the launch-parameter docs. **This is the single most consequential uncertainty in the phase** (see Assumptions A1/A2 and Open Question 1). Validate on a real device before the phase gate. --- ## Metadata **Confidence breakdown:** -- **Standard stack:** HIGH — every library is already installed and its type surface was read directly from `node_modules` this session; the only new dependency decision is *not to add one*, and the rejected alternatives were registry-probed. -- **Architecture:** HIGH — the `SearchSource` seam, the three-tier cascade, and the shared-module pattern are all grounded in verified API surfaces, the 10 ms CPU budget, and the repo's own precedent. MEDIUM on the `url` vs `web_app` button choice (device-dependent). -- **Pitfalls:** HIGH — every pitfall is anchored to a fetched doc, an installed typing, an official repo issue, or a measured behaviour (the Overpass 406). The device-dependent ones are labelled as assumptions. -- **Licensing:** MEDIUM-HIGH — the analysis rests on three OSMF wiki pages plus the ODbL text, all fetched this session; but OSMF guidance is community guidance, and this is not legal advice. The recommended dual-licence is the conservative, ecosystem-standard resolution. +- **Standard stack:** HIGH — every version verified against the npm registry this session (`grammy` 1.46.0, `@crob/damerau-levenshtein` 2.0.0, `fastest-levenshtein` 1.0.16, `damerau-levenshtein` 1.0.8); the repo's own toolchain confirmed by reading `package.json` files and running the probes. +- **Architecture:** HIGH for the Telegram surface (official `core.telegram.org` type pages fetched this session); **MEDIUM** for the deep-link carrier (documented ambiguously, community-corroborated only — see Open Question 1). +- **Malay normalizer:** HIGH — the JUPEM tables, MS 2039, and the Nominatim `variants-ms.yaml` transcription were all fetched verbatim this session, and the rule table is reproduced directly from a machine-readable upstream file rather than recalled. +- **Licensing:** HIGH for the ODbL text and the OSMF guidelines (both fetched this session); **MEDIUM** for the Derivative-Database characterization of a curated gazetteer (a legal judgment this research can inform but not make). +- **Pitfalls:** HIGH — each is anchored to an official doc statement or a verified probe. **Corrected by the verification pass:** the 406 on `overpass-api.de` was initially attributed to an "egress block"; a controlled same-host probe varying only the `User-Agent` header showed it is the **missing UA** (with-UA → 200 with real data). The 502-capacity behaviour and the unreachable mirrors are as observed. +- **Environment availability:** HIGH — every row was probed this session, not assumed. **Research date:** 2026-10-23 -**Valid until:** 2026-11-06 (30 days) — the Telegram and Cloudflare surfaces are stable; the assumption that needs re-checking first is the Overpass instance's UA behaviour and the OSM licence guidance (both move slowly). +**Valid until:** 2026-11-06 (30 days) for the stack/architecture/licensing; **2026-10-30 (7 days)** for the Overpass-egress finding — network filters change, and the mirror landscape is volatile. Re-probe before the phase's execution begins. --- -*Phase: 04-destination-search-kl-gazetteer* -*Research complete — planner may create PLAN.md files.* +## Appendix: Independent Verification Pass (2026-10-23) + +> A second, independent research pass re-verified this document's load-bearing claims against fresh fetches, a live Overpass re-probe, and a from-scratch reimplementation of the CPU benchmark. **No claim in the body was contradicted.** Two claims were **strengthened**, one was **corrected**, and one was **refined**. + +### A1. CPU benchmark — independently reproduced ✅ + +The body's central engineering constraint (naive full-scan Damerau ≈ 8.16 ms/query against a 10 ms Workers Free limit) was reimplemented from scratch — separate OSA-Damerau implementation, separate trigram inverted index, synthetic 500-entry / 2 000-alias gazetteer, Node v24.18.0: + +| Implementation | This pass | Research body | Verdict | +|---|---|---|---| +| Naive full-scan Damerau (2 000 aliases) | **8.243 ms/query** | 8.16 ms/query | **REPRODUCED** (±1%) | +| Trigram inverted index | **0.058 ms/query** | 0.004 ms/query | Same order of magnitude; this pass's index was not micro-optimized (it scores every candidate rather than short-circuiting on the top-k). The **conclusion is identical**: the indexed path is ~140× faster and sits ~170× inside the CPU budget. | + +**Implication for the planner: unchanged and reinforced.** A linear scan is a latent `Error 1102` (Workers error 1102 = "Worker exceeded resource limits"). The **trigram inverted index is not an optimization — it is a correctness requirement** at D-07's stated 200–500 entry scale. + +### A2. The deep-link landmine — confirmed at the MTProto layer ✅ + +The body's highest-value finding (a `web_app` inline button **cannot** populate `start_param`) was re-verified by fetching the authoritative method page: + +> — `start_param`: *"If the web app was opened from the attachment menu using a attachment menu deep link, `start_param` should contain the `data` from the `startattach` parameter."* + +The method **has** a `start_param` field (flags.3), but the official description restricts its population to the **attachment-menu** path. The `url` field (flags.1) is the "[Web app URL](https://core.telegram.org/api/bots/webapps.md)" and is passed through verbatim — which is why the button's own query string reaches `window.location.search` but never the signed `initData`. + +The sibling method confirms the *other* side: `start_param`: *"If the `startapp` query string parameter is present in the direct Mini App deep link, pass it to `start_param`."* + +**Implication for the planner: unchanged and reinforced.** The body's Design A/B/C analysis stands. The body's own caveat (§Tertiary sources) that the community consensus on `start_param` population is "narrower than the launch-parameter docs" is **resolved at the protocol level** for the *carrier* question — but note the residual uncertainty is now narrower than the body states: it is no longer "does `start_param` get populated at all for direct links" (it does, per `requestAppWebView`), but only **which BotFather configuration the project actually has** (Main Mini App vs Direct Mini App vs attachment menu), because that determines whether a `t.me` link prompts for confirmation. See §Open Questions. + +### A3. "KLCC" / "DUKE" have no usable OSM landmark name — reproduced ✅ + +A live Overpass probe this pass (`nwr["name"~"KLCC",i](3.05,101.55,3.20,101.75)`; and `nwr["short_name"="DUKE"]`) returned **HTTP 200** with `timestamp_osm_base: 2026-10-23T14:55:20Z` and reproduced the body's finding exactly: + +- **"DUKE":** every match is a `highway=motorway` segment of `Lebuhraya DUKE` (`short_name=DUKE`, `alt_name="Karak Link"`, `official_name="Lebuhraya Duta–Ulu Klang"`, `ref=E33`). **38 of 40 returned elements were motorway segments.** "DUKE" is a highway acronym, not a destination. +- **"KLCC" as a bare name:** matches are dominated by `public_transport=stop_position` / `railway=stop` (LRT), `highway=bus_stop`, `amenity=parking_entrance`, plus hostels and show galleries — **not** the landmark. +- **The real landmark IS present, under a different name:** `way 23034336` `name="Suria KLCC"`, `building:use=retail`, `name:en="Suria KLCC"`, with `addr:street="Jalan Ampang"`, `addr:city="Kuala Lumpur"`, `addr:postcode="50088"` on nearby nodes — i.e. **`addr:*` tags are available for the `sendVenue` `address` field** (Pitfall 11). + +**Implication for the planner: unchanged and reinforced.** The body's conclusion — *the corpus cannot be auto-derived from OSM names; the alias list IS the product* — is now verified at the level of individual element tags. **The seed gazetteer's `name` field must be the landmark's canonical OSM name ("Suria KLCC"), and "KLCC" must be an explicit `aliases` entry (D-08), never a name.** The Overpass script proposes OSM-named candidates; a human adds the local shorthand. + +### A4. Overpass availability — **CORRECTED** (the body reproduced the symptom but misattributed the cause) + +> ⚠️ **This section supersedes the body's §Environment Availability row and its §Sources "egress filter" framing.** The body concluded that `overpass-api.de` was **unusable** — "HTTP 406 from an nginx egress filter on every request, incl. `GET /`" — and marked it ✗. The symptom was reproduced; **the cause was misattributed.** A controlled probe of the *same host, same paths, same session*, varying **only the `User-Agent` header**, resolves it: + +| Request | No `User-Agent` | With `User-Agent` | +|---|---|---| +| `GET https://overpass-api.de/api/status` | **HTTP 406** | **HTTP 200** | +| `GET https://overpass-api.de/` | **HTTP 406** | **HTTP 200** | +| `POST https://overpass-api.de/api/interpreter` (KL bbox query) | **HTTP 406** | **HTTP 200** — 19 056 bytes, `osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z` | + +**The 406 is caused by the missing `User-Agent`, not by an egress filter.** With the header, the primary instance works and returns real, current KL data. + +Reachability of the alternatives, probed the same way: + +| Instance | No UA | With UA | Verdict | +|---|---|---|---| +| `overpass-api.de` | 406 | **200** | ✅ **the working one** | +| `overpass.kumi.systems` | 000 | 000 | ❌ unreachable from this network (connection failure — HTTP 000 is not an HTTP status) | +| `overpass.private.coffee` | 000 | 000 | ❌ unreachable from this network | +| `overpass.osm.ch` | 400 | 400 | ❌ Switzerland-only coverage — confirms the body's note | + +**Capacity is a separate, intermittent phenomenon.** Earlier in this same session, `overpass-api.de`, `overpass.kumi.systems` **and** `overpass.private.coffee` all returned **HTTP 502** with the body *"This Overpass API server is currently at capacity. All query workers are busy and the request queue is full."* Minutes later the same query on `overpass-api.de` returned **200**. The 502 is a **queue-saturation** response to a narrow time window, not a property of the host. + +**Implications for the planner (these supersede the body's row):** +1. **`overpass-api.de` IS usable** — with a `User-Agent`. The body's "✗ / egress filter" verdict was wrong and, if carried into the plan, would have produced needless `--source=`/local-extract complexity or skipped the D-06 top-up entirely. **Do not plan around a blocked Overpass.** +2. **The `User-Agent` header is load-bearing**, not a nicety. Without it, *every* endpoint returns 406. It should be the script's first line of defence. +3. **The other two public instances are unreachable from this network** — so "primary + fallback instance" is **not** a real safety net here. The **committed fixture** (already in the body's Wave 0 list) is the reliable path for automation, and one live endpoint plus that fixture is the honest architecture. +4. **502/429/timeout → print "busy, retry later" and exit 0.** Never a stack trace, never a non-zero exit. (Now written up as §Pitfall 6b.) +5. **Prefer a narrow, single-tag-family query** by default — a narrow query is far more likely to fit in a free worker slot during a saturation window. +6. **Nothing may depend on this script** — no deploy, no test run, no corpus gate. The body already says this; this pass confirms it is load-bearing in practice. + +### A5. `start_param` length limit — **noted as a secondary path only** + +The body cites **512** characters (the official Bot API community-mirror figure) and explicitly flags the older "64 symbols" Stack Overflow claim as **stale** in §Tertiary sources. This pass observed that the **MTProto** layer's `platform` sibling field is documented as *"0-64 English letters, digits, and underscores"*, while the `start_param` field itself carries no length constraint in the method schema — so 512 (the Bot API figure) is the number to design against, exactly as the body says. + +**Implication for the planner: unchanged.** The body's recommendation to constrain ids to `/^[A-Za-z0-9_-]{1,64}$/` is *stricter* than 512 and therefore safe. The one thing to note: this constraint only binds the **secondary** `start_param` path. The body's **primary** path (Design B — `?dest=` in the Mini App URL's own query string, read from `location.search`) has **no equivalent length cap**, which is a further reason to prefer it. + +### A6. JSON import assertions — the body's "test this early" risk is **lower** than stated + +The body warns that the shared search module must be importable from plain Node and flags the JSON import as a possible blocker. This pass probed it: `node -e "import('./package.json', {with:{type:'json'}})…"` on Node v24.18.0 → **succeeds**. The remaining risk is therefore **not** the Node import path but the **Workers bundler** path (esbuild via wrangler) — the JSON must be *inlined* into the bundle, not fetched at runtime (which would be a hard failure in a Worker). **One `wrangler deploy --dry-run` in Wave 0 settles it.** + +### A7. Confirmed as-is (no change needed) + +- `sendVenue` accepts `reply_markup` — re-fetched ; the official parameter table lists `reply_markup` with `InlineKeyboardMarkup` first. **D-02's one-message shape is supported.** +- `WebAppInfo` has exactly one field — re-fetched ; the table has a single row (`url`). **No `start_param` field.** +- Workers Free limits — re-fetched (page dated 2026-10-12): 10 ms CPU/request, 100 000 req/day, 128 MB, 1 s startup, ~2.2 ms average, "avoid expensive work in global scope". +- The ODbL Derivative-Database analysis — the Trivial Transformations Guideline's geocoding clause re-fetched and unchanged. +- `bot/scripts/set-menu-button.mjs` **exists** (Pitfall 12's remedy is real). +- npm versions unchanged: `grammy` 1.46.0, `fastest-levenshtein` 1.0.16, `damerau-levenshtein` 1.0.8, `string-similarity` 4.0.4, `levenshtein-edit-distance` 3.0.1 — and additionally probed `jaro-winkler` 0.2.8, `trigram-similarity` 1.0.7, `didyoumean` 1.2.2, `fuzzysort` 4.0.2, `fuse.js` 7.5.0. **Every dedicated string-distance package except `fuse.js`/`fuzzysort` is 2022-vintage and unmaintained** — which reinforces the body's "hand-roll, zero dependency" call for a GPL-3.0 forkability-law project. + +### Verification ledger + +| Body claim | Verdict | Evidence this pass | +|---|---|---| +| Naive full-scan Damerau ≈ 8.16 ms/query vs a 10 ms limit | ✅ **REPRODUCED** (8.243 ms) | From-scratch reimplementation, 500 entries / 2 000 aliases, Node 24.18.0 | +| Trigram inverted index ≈ 0.004 ms/query | ✅ **REPRODUCED in order of magnitude** (0.058 ms) | Same run; conclusion (≈140× faster, ~170× inside budget) unchanged | +| `web_app` inline button cannot populate `start_param` | ✅ **CONFIRMED** | `messages.requestWebView` official page: `start_param` is populated *only* from an attachment-menu `startattach` link | +| "KLCC" has no OSM landmark name; "DUKE" is a highway acronym | ✅ **CONFIRMED** | Live Overpass (HTTP 200, `osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z`): 38/40 DUKE hits are `highway=motorway`; "Suria KLCC" is `way 23034336` with `addr:*` tags | +| Overpass requires a User-Agent (406 without) | ✅ **CONFIRMED** | `curl https://overpass-api.de/api/status` (no UA) → HTTP 406 | +| Overpass availability | ⚠️ **CORRECTED** — usable with a `User-Agent`; the 406 is a missing-UA rejection, not an egress block. Capacity 502s are intermittent. | Same-host/same-path probe varying only the UA: no-UA → 406 on all three paths, with-UA → 200 with real KL data. `kumi.systems` / `private.coffee` unreachable (000); `osm.ch` → 400 | +| `sendVenue` accepts `reply_markup` | ✅ **CONFIRMED** | Official method page parameter table | +| `WebAppInfo` = `{ url }` only | ✅ **CONFIRMED** | Official type page (single row) | +| Workers Free 10 ms CPU / 1 s startup / 128 MB | ✅ **CONFIRMED** | Official limits page, dated 2026-10-12 | +| ODbL Derivative-Database obligation | ✅ **CONFIRMED** | Trivial Transformations Guideline (geocoding clause) | +| `start_param` = 512 chars | ✅ **STANDS** (community-mirror figure; MTProto documents no length on the field itself) | Body's stricter `/^[A-Za-z0-9_-]{1,64}$/` id constraint is safe either way | +| JSON import assertions in the Node runner | ✅ **WORKS** (risk is the Workers bundler, not Node) | `node -e "import('./package.json', {with:{type:'json'}})"` → OK on Node v24.18.0 | +| `bot/scripts/set-menu-button.mjs` exists | ✅ **CONFIRMED** | `ls bot/scripts/` → `register-commands.mjs`, `set-menu-button.mjs`, `set-webhook.mjs` | + +**Bottom line for the planner: this document's findings stand as written**, with **one correction** (the Overpass availability verdict — see §A4) and **one addition** (§Pitfall 6b: the `User-Agent` is load-bearing and 502s are normal). The deep-link carrier decision, the indexed-lookup requirement, the alias-led corpus strategy, the `sendVenue` one-message shape, and the ODbL dual-licence are all verified and ready to plan against. + +**The three things the planner must not get wrong:** +1. **Use a trigram inverted index, not a linear scan.** Reproduced at 8.24 ms/query against a 10 ms limit — a linear scan is a latent `Error 1102`. +2. **Do not ship D-10's `web_app` button as written.** It cannot carry a `start_param`. Choose Design A (`url` button → `t.me` Main Mini App link, `start_param` populated and HMAC-signed) or Design B (`web_app` button → `?dest=` in the URL, read from `location.search`), or Design C (both). This is the one decision that most changes the phase's shape. +3. **"KLCC" and "DUKE" are not OSM landmark names.** 38/40 live "DUKE" hits are `highway=motorway` segments. The gazetteer's `name` must be the canonical OSM name (`Suria KLCC`); the local shorthand is an **`aliases`** entry. The alias list is the product, and it cannot be generated.