# Phase 4: Destination Search + KL Gazetteer - Research
**Researched:** 2026-10-23
**Domain:** In-Worker fuzzy place search over a curated KL gazetteer, Malay query normalization, Telegram venue-card + Mini App deep-link hand-off, OSM-derived gazetteer tooling and licensing
**Confidence:** HIGH for the Telegram Bot API surface (official docs + installed grammY 1.46.0 typings read this session), HIGH for Cloudflare Workers limits (official docs fetched), HIGH for OSM/ODbL licensing analysis (official OSMF wiki fetched), MEDIUM-HIGH for the Malay abbreviation table (UPU address standard + OSM road lists + Malaysian postal examples — all real but not a complete KL corpus), MEDIUM for the scoring-function recommendation (algorithm properties are cited; the exact weights must be settled empirically by the corpus runner)
## User Constraints (from CONTEXT.md)
### Locked Decisions
**Search interaction surface**
- **D-01:** Search entry is the **`/cari` command** in chat (with `/pergi_ke` reserved for Phase 5 semantics; bilingual alias scheme per Phase 2 D-05). Free-text-to-bot stays reserved; no inline mode; no in-webview search UI this phase (a shared search module keeps that door open).
- **D-02:** Shortlist presentation: the **best result ships immediately** as the venue card with its OPEN button; **2–4 alternates ride as inline buttons on the same message**. One tap to proceed, one tap to switch — no second message round.
- **D-03:** Search backend is a **pluggable `SearchSource` interface implemented in-Worker**: curated KL gazetteer queried in-Worker, $0 infrastructure. **VPS/Photon provisioning defers to Phase 5** (Valhalla's road graph needs the VPS anyway). Photon remains a documented escalation path *behind the interface*.
**Forkability + license (user-surfaced design law)**
- **D-04:** **GPL-3.0** is the project license. Forkability laws are binding from this phase: no hardcoded secrets (env only — already true), data as plain committed files, one-command deploy, one free-tier Cloudflare account per fork.
- **D-05:** The user's **Malaysia-only WASM geocoder** vision (tantivy/sqlite-trigram index, statically hosted, client-side webview search — a genuine privacy win) is captured as **SearchSource backend #2**, an OSS-track artifact *after* v1. The JSON↔SQLite converter script keeps its dataset format reachable from day one. **Backend #3 = Photon-on-VPS.** Neither ships in this phase.
**Gazetteer + Malay normalizer**
- **D-06:** Entries are **hand-curated seed + a repo-script Overpass one-shot** that proposes top-up candidates for maintainer approval into the curated file. The script is a repo tool, not a runtime dependency.
- **D-07:** Canonical format is **typed JSON in the repo** (V8-native parse, no WASM in the Worker bundle, diff-friendly PRs). Rationale: at ~200–500 rows, SQLite/sql.js-in-Worker is ~1.5MB + cold-start overhead of pure cost.
- **D-08:** **Aliases live natively inside each entry** (`"KLCC"` → Suria KLCC, KLCC Park; IJN, PWTC, Stesen/Station…) — locals' shorthand is the main corpus quality lever.
- **D-09:** The Malay normalizer is a **rule table + folding + fuzzy-residual** module: bidirectional abbreviation table (Jln↔Jalan, Tmn↔Taman, Kg↔Kampung, vowel-drop-aware Presint/Prsn forms), case/whitespace folding, glued-word split — applied to BOTH the query and entry aliases at index time. Fuzzy scoring (Damerau) is only the residual net for genuine typos on longer words. Per-rule test vectors; extends by table edit, not code change.
**Venue card + deep-link hand-off**
- **D-10:** The venue card is **Bot API `sendVenue`** (native pin/name/address semantics SRCH-03 names) with inline buttons: OPEN IN ARAHARAH (`web_app` button carrying `startapp`) + alternates.
- **D-11:** The deep link carries a **plain gazetteer entry id** (`?startapp=dest_`). The webview maps id → destination from its own copy of the same gazetteer module (shared module, build-time). No crypto: the payload's only threat is a user pranking themselves; stale-webview datasets re-map via the shared module version.
- **D-12:** ONBD-03 scope: **minimal PWA hooks** — `addToHomeScreen` prompt handling + a startapp-less boot landing in the search-ready map (existing Phase 3 staged boot covers it). Full PWA manifest/offline story is documented-deferred.
**Acceptance corpus + thresholds (the bake-off close)**
- **D-13:** A **pinned golden corpus** in the repo (~60–100 queries): golden landmarks (KLCC, Sunway Pyramid, DUKE), abbreviation forms, typo set (transpositions, missing keys), glued words, aliases, kampung-style names, plus **deliberate negatives** (out-of-KL: "Kuching", "Penang") that must get an honest "not in v1 area" answer, never a fabricated pin. Each positive pins its expected entry id. Reviewers extend by editing one file.
- **D-14:** The bake-off closes on: **top-3 contains the expected id for ≥90% of positives AND negatives are refused honestly**. Below threshold, escalation order is fixed: (1) alias/normalizer table edits → re-run; (2) SearchSource backend #2 (WASM geocoder); (3) Photon-on-VPS. The phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question.
**Privacy (carried + made specific)**
- **D-15:** Search queries are user payload: **category-only logging** (Phase 2 law) — no query text persisted. KV caching of search results, if any, is TTL-only with short TTL and keyed by normalized query (content is public place data, never user identity). Venue choices are not stored.
### OpenCode's Discretion
- Scoring function shape (weighted trigram/Jaro vs Damerau) within the rule-table architecture — corpus decides empirically
- Alternates ranking + tie-breaking details; "not in v1 area" copy (en/ms, clinical tone)
- Overpass query shape for the top-up script; gazetteer entry schema fields beyond (id, name, aliases, lat, lng, category)
- D1/KV TTL result-cache mechanics or none (perf is likely fine without at this scale)
### Deferred Ideas (OUT OF SCOPE)
- Malaysia-only WASM geocoder (tantivy/sqlite-trigram), statically hosted, client-side webview search — SearchSource backend #2, OSS track after v1
- Photon-on-VPS — SearchSource backend #3; VPS provisioning happens in Phase 5 for Valhalla regardless
- In-webview search UI — needs a UI-SPEC round; shared module already keeps it cheap
- Full PWA (manifest, offline shell) — documented-deferred beyond minimal ONBD-03 hooks
- Free-text bot search + inline mode — reserved interaction surfaces
- Log-derived corpus (real queries) — becomes possible only post-launch (Phase 7+)
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|------------------|
| **SRCH-01** | User can type a destination query (e.g. "KLCC") and receive a shortlist containing the right KL-metro landmark/POI | §"The `SearchSource` seam" (pluggable interface + in-Worker impl), §"Scoring architecture — three tiers", §"Golden corpus + threshold gate" (the executable ≥90% top-3 gate). Gazetteer entry schema in §"Gazetteer entry schema" and seed rows in §"Seed gazetteer shape" |
| **SRCH-02** | User gets useful results despite abbreviations and typos (Jln↔Jalan, Tmn↔Taman, missing spaces, common Malay address shorthand) | §"Malay normalizer — rule table" (the full bidirectional abbreviation table with authoritative UPU + OSM grounding), §"Glued-word splitting", §"Scoring architecture — three tiers" (alias/canonical exact-match tier precedes fuzzy). §"Normalizer test vectors" gives per-rule vectors |
| **SRCH-03** | For a chosen destination, a native venue card is sent in chat, confirming where we're going before navigation opens | §"`sendVenue` — what the card can and cannot carry" (verified against installed grammY 1.46.0 + @grammyjs/types: `reply_markup` IS supported on `sendVenue`, so D-02's single-message shape works), §"Four-square type mapping" |
| **ONBD-02** | User can open the Mini App from a search result with their destination preloaded (no re-entry in the webview) | §"Deep-link mechanics" (the `web_app` inline button URL vs `t.me?startapp=` direct link — with the **Desktop START-button trap** documented as a pitfall), §"Reading `start_param` in the webview" (validated `initData` field, character restrictions), §"Harness evidence path" |
| **ONBD-03** | User can add araharah to their phone home screen and launch search/navigation from there (addToHomeScreen + startapp deep-link) | §"ONBD-03 — what `addToHomeScreen` actually is" (Bot API 8.0 `addToHomeScreen()` + `checkHomeScreenStatus()`; **iOS always returns `unknown`** — verified against Telegram-iOS issue #1729), §"Home-screen launch has no `start_param`" (the startapp-less boot path) |
---
## Summary
Phase 4 is small in code and large in judgement. The locked decisions (D-01…D-15) have already made the two hardest architectural calls: the backend is an in-Worker `SearchSource` over a curated KL gazetteer (no VPS, no Photon, $0 infra), and the interaction surface is the `/cari` command with a native venue card. What remains for research to settle is the *quality machinery* — the Malay normalizer, the scoring function, the corpus format, and the exact Telegram payload shapes — plus three things the roadmap never mentioned but that will bite the planner:
**1. The `web_app` inline button and the `t.me?startapp=` direct link are NOT equivalent, and on Telegram Desktop the direct link needs a second tap.** D-10 says the OPEN button is a `web_app` button "carrying `startapp`". Verified from the installed `grammy@1.46.0` / `@grammyjs/types` typings: `InlineKeyboardButton.web_app` is a `WebAppInfo` whose **only** field is `url: string` — an HTTPS URL of the Mini App itself. There is **no `start_param` field on `WebAppInfo`**. The `startapp` query parameter is a **deep-link** construct (`https://t.me/?startapp=dest_`), and Telegram only propagates it to the Mini App when the Mini App is opened *that* way. So the OPEN button's URL must be either (a) the Mini App origin with the parameter appended to the *webview URL's own query string* — which does **not** become `start_param` and must be read from `location.search` instead — or (b) a `t.me` deep link. Option (b) has a documented trap: [telegramdesktop/tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694) reports the user "must manually press the 'START' button for the deeplink to be processed" on Telegram Desktop for Windows/Linux. **Recommendation: use a `url` inline button (not a `web_app` button) pointing at the Mini App origin with `?dest=` in the query string**, read it from `location.search` in the webview, and fall back to `start_param` when present. This works identically on every client, has no START-button trap, and keeps D-11's "plain gazetteer entry id" payload. The planner must make this call explicitly; it is not free.
**2. A curated OSM-derived gazetteer is a legal "Derivative Database" under ODbL, and that is in tension with the GPL-3.0 project license (D-04).** This is the single most consequential finding of this research. The OSMF's own [Trivial Transformations Guideline](https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline) says explicitly: *"A 'geocoding database' should **NOT** be considered a trivial transformation… Geocoding is being considered for a separate community guideline."* The Use Cases page says: *"if you have any data that was derived from OSM — for instance because you used street names from OSM… then you are making a derivative database."* A curated gazetteer of KL POIs with names, coordinates and addresses extracted from OSM and committed as a public JSON file **is** a derivative database: ODbL §4.4 requires it be made available under ODbL or a compatible licence. GPL-3.0 is a **software** licence and is **not** a compatible database licence for ODbL share-alike — the OSMF's [Licence Compatibility](https://foundation.openstreetmap.org/wiki/Licence/Licence_Compatibility) page lists licences requiring users to use the data on *identical* terms as generally incompatible. **Recommendation: dual-license the gazetteer data file itself under ODbL-1.0 (with `© OpenStreetMap contributors` attribution) while keeping the *code* GPL-3.0.** This is exactly how Photon, Nominatim-adjacent tools, and OSM-derived routing datasets are shipped, and it satisfies D-04's forkability law (the data file stays a plain committed file; a fork inherits the ODbL obligation on that file, which is the correct and standard outcome). The planner must also ensure the "OPEN IN ARAHARAH" venue card / bot reply carries the OSM attribution string in both locales — the web app already ships `attribution: "© OpenStreetMap contributors"` in `web/src/i18n/strings.ts`, so the bot side needs the same.
**3. The OSM licence problem is avoidable for the seed data, and that changes the Overpass script's job.** The D-06 Overpass script is a *proposal generator*, not the data pipeline: its output is a human-reviewed candidate list, and the curated file is maintained by hand. If the maintainer **writes the address strings and picks the aliases by hand** from local knowledge (and only copies the *coordinates* from OSM, treating them as facts rather than as a database extraction), the curated file's derivative-database exposure drops sharply — but "substantial" is a judgement call the planner should not make unilaterally. **The safe, standard, and cheap answer is the dual-licence above**; it costs one `LICENSE` file, one attribution line, and one sentence in the README. D-04 explicitly asks for a GPL-3.0 `LICENSE` file this phase — the planner should add the ODbL notice alongside it for the data file.
**4. Workers Free gives 10 ms of CPU per request.** [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/) (fetched 2026-10-23): "CPU time per HTTP request | **10 ms**", 100,000 requests/day, 128 MB memory, 1 s startup. The average Worker uses ~2.2 ms. A gazetteer of 200–500 rows with, say, 8 aliases per row yields 1,600–4,000 folded alias strings. A naive "compute Damerau-Levenshtein against every alias" pass is O(4,000) × O(len²) — comfortably inside 10 ms only if the candidate set is first narrowed. **Recommendation: index-at-load alias folding with an exact-match hash lookup first (O(1)), then a cheap character-bigram Dice filter to cut the candidate set to a handful, then Damerau-Levenshtein only on that handful.** This is the classic three-tier cascade and it also gives the "honest refusal" behaviour D-14 demands for free: a query that produces no candidate above threshold simply yields zero results. Also note the **1 s Worker startup limit** and the "avoid expensive work in global scope" guidance — the gazetteer should be parsed **inside the request handler** (or at least lazily), not at module top level, or a 500-row JSON parse + fold runs on every cold start.
**5. `sendVenue` supports `reply_markup`, so D-02's one-message shape works.** Verified directly in the installed typings (`node_modules/@grammyjs/types/methods.d.ts`, `sendVenue` args include `reply_markup?: InlineKeyboardMarkup | …`). So the venue card and its OPEN + alternates buttons ship as ONE message. That satisfies D-02 exactly ("no second message round"). The `Venue` type carries `location`, `title`, `address`, plus optional `foursquare_id`/`foursquare_type`/`google_place_id`/`google_place_type` — the four-square type is the free, self-hostable choice (no Google API key needed), and the Foursquare ID can be the gazetteer's own OSM-derived identifier.
**6. ONBD-03's `addToHomeScreen` is a Telegram-native shortcut, not a web PWA install, and iOS cannot report its status.** `WebApp.addToHomeScreen()` and `WebApp.checkHomeScreenStatus(callback)` are Bot API 8.0+ methods ([core.telegram.org/bots/webapps](https://core.telegram.org/bots/webapps)), returning one of `unsupported` / `unknown` / `added` / `missed`. [Telegram-iOS issue #1729](https://github.com/TelegramMessenger/Telegram-iOS/issues/1729) confirms: *"Home screen status detection always returns 'unknown' on iOS… On iOS it is not possible to determine whether a Mini App has been added to the Home Screen. The platform does not provide any API that allows Telegram to check whether a Home Screen shortcut exists."* The `beforeinstallprompt` event that a normal web PWA uses **does not exist inside a Telegram webview** — there is no PWA install prompt. So D-12's "minimal PWA hooks" means: call `addToHomeScreen()`, handle `homeScreenAdded`, and treat `unknown` as "possibly added" (iOS). A home-screen shortcut launched by the OS **carries no `start_param`** — it opens the Mini App cold, which is why D-12 requires the startapp-less boot to land in the "search-ready map". That path already exists from Phase 3 (the staged boot), so ONBD-03 is mostly a *verification* obligation plus a small hook.
**Primary recommendation:** Ship this phase as three thin, separately-testable artifacts — a pure `SearchSource` + normalizer + scorer module (no I/O, no Telegram, no DOM; fully exercisable by the corpus runner), a `sendVenue`-based `/cari` handler that talks to it, and a webview boot branch that reads the destination id from `location.search`/`start_param` — then let the pinned golden corpus be the *executable* definition of the ≥90% gate, with the escalation ladder (normalizer table edits → backend #2 → backend #3) written as a documented decision record in the phase directory rather than as an open question.
---
## Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| Query normalization (Malay abbreviation folding, case/whitespace, glued-word split) | API / Backend (Worker) | Frontend Server (browser runtime) | D-09 says the normalizer is applied to *both* query and entry aliases at index time — the canonical copy must be the shared module. It is pure string logic with no I/O, so it runs identically in the Worker (bot) and in the browser (webview copy at build time). One implementation, two consumers. |
| Gazetteer storage + alias indexing | API / Backend (Worker) | Database / Storage (repo file) | D-07: the canonical store is a typed JSON file in the repo; the Worker parses it at request time. The webview gets a build-time copy of the same module. No database tier exists in v1 (KV is TTL-only session state). |
| Fuzzy scoring / ranking / shortlist | API / Backend (Worker) | — | Scoring runs server-side so the corpus runner and the production path exercise *the same code*. If scoring lived in the browser, the corpus runner would have to duplicate it and the bake-off would measure the wrong thing. |
| `/cari` command intake + reply shaping | API / Backend (Worker) | — | The command arrives as a Telegram webhook update; only the Worker can read it. |
| Venue card rendering (name/address/coordinates) | API / Backend (Worker) → Telegram client | — | `sendVenue` is a Bot API call. The card is rendered by Telegram, not by araharah. |
| Deep-link construction (`dest_` payload) | API / Backend (Worker) | — | The Worker knows the entry id; it composes the Mini App URL. |
| Destination-id → destination resolution in the webview | Frontend Server (browser runtime) | API / Backend | D-11: the webview maps id → destination from *its own* build-time copy of the gazetteer module. No API round-trip is needed — the id is a plain key into a file the bundle already carries. |
| Bot menu registration for `/cari` | API / Backend (repo script) | — | `bot/scripts/register-commands.mjs` derives menu descriptions from `strings.helpMenu` — the Phase-2 pattern. `/cari` must be added to `helpMenu` in `bot/src/i18n/strings.ts` or it will not appear in the menu (Phase-2 CS5 gap-freedom gate). |
| Result caching | Database / Storage (KV, TTL-only) | API / Backend | D-15 permits TTL-only KV caching keyed by normalized query. At this scale it is probably unnecessary (see §"Should there be a result cache?"). |
| Overpass top-up proposal generation | CDN / Static → repo tool (offline script) | — | D-06: a repo tool, never a runtime dependency. It runs on a maintainer's machine, not in the Worker. |
| Home-screen install affordance | Frontend Server (browser runtime) | Telegram client | `WebApp.addToHomeScreen()` is client-side; the event fires back into the page. The Worker is not involved. |
---
## Standard Stack
### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `grammy` | **1.46.0** (already in `bot/`, `^1.46.0`) | Bot framework — `/cari` command handler, `ctx.sendVenue`, inline keyboards | Already the project's bot framework. **Verified this session in the installed typings**: `api.sendVenue(chat_id, latitude, longitude, title, address, other?)` and `ctx.replyWithVenue(...)` both exist; `sendVenue`'s args include `reply_markup?: InlineKeyboardMarkup \| ReplyKeyboardMarkup \| …`; `InlineKeyboardButton` has a `url` variant and a `web_app: WebAppInfo` variant; `WebAppInfo` has exactly one field, `url: string`. |
| `node:assert` + hand-rolled runner | repo-native (`bot/test/runner.mjs`, `web/test/runner.mjs`) | Test harness for the corpus runner | Repo precedent, zero deps. The corpus runner is a new suite in the same shape (see §Validation Architecture). |
| **No search library** | — | Normalization, exact lookup, bigram filtering, Damerau-Levenshtein | **Deliberate.** At 200–500 rows the whole algorithm is ~150 lines of pure TS. `fuse.js@7.5.0`, `fastest-levenshtein@1.0.16`, `string-similarity@4.0.4`, `levenshtein-edit-distance@3.0.1` all exist on npm (verified) but each adds a dependency to a forkability-law project (D-04) to solve a problem this scale does not have. D-09's "extends by table edit, not code change" is better served by a hand-rolled, table-driven normalizer than by a library's config surface. |
### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `node:crypto` (Web Crypto `subtle`) | built-in | Nothing cryptographic is needed — but see the D-11 note | **D-11 explicitly says "No crypto"** for the deep-link payload. Do **not** add a signature. The id is a plain key into a public gazetteer; the only threat is self-pranking. |
| `wrangler` | **4.151.3** (already in `bot/`) | `wrangler dev --port 8787 --local` (local KV) + `wrangler deploy` | D-12 Phase 3's dev topology; the gazetteer is bundled into the Worker at deploy time. |
| `vite` | **8.3.5** (already in `web/`) | Build-time copy of the gazetteer module into the webview bundle | D-11's "shared module, build-time". The web bundle must import the *same* module the bot imports — the sharing mechanism matters, see §"Sharing the gazetteer between bot and web". |
| `playwright` | `^1.50.0` (already in `web/`, browsers cached: chromium-1248, webkit-2272, firefox-1511) | Mock-Telegram harness capture for ONBD-02 evidence | Phase 3's `web/scripts/capture.mjs` already builds `web/dist-harness` and races `RAMS-BASEMAP-READY` → `ARAHARAH-LIVE`. A `?dest=` boot profile slots straight into the existing capture list. |
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Hand-rolled 3-tier scorer | `fuse.js@7.5.0` (Bitap + fuzzy, 30 KB, MIT) | **Reject for v1.** Fuse is excellent general-purpose fuzzy search, but it (a) is a black box — the "extends by table edit, not code change" law (D-09) would be violated once you need a custom Malay rule table inside the scoring, (b) adds a dependency to a GPL/forkability-law project, (c) its scoring is optimized for long documents, not 3–20 character place names. **Revisit at backend #2** if the corpus shows the hand-rolled scorer cannot reach 90%. |
| Hand-rolled Damerau-Levenshtein | `fastest-levenshtein@1.0.16` (MIT, WASM+JS) | **Reject.** It is Levenshtein-only (no transposition), so it charges 2 edits for the transposition typos D-13's corpus explicitly requires ("transpositions"). D-09 locks Damerau for the residual net. A 40-line OSA/Damerau implementation is the whole requirement. |
| Precomputed trigram index at build time | Runtime bigram Dice filter | **Prefer the runtime filter.** A build-time trigram index adds a generated artifact to the repo (diff noise, stale-index risk) for zero measurable gain: at 4,000 aliases, building a bigram set per alias on the fly is ~1 ms. Revisit only if the corpus grows past ~5,000 entries. |
| OSM-derived gazetteer published under GPL-3.0 only | Dual-licence: code GPL-3.0, data file ODbL-1.0 | **Choose the dual-licence.** See §"Licensing — the ODbL obligation". GPL-3.0 alone risks ODbL §4.4 non-compliance on the data file. |
| `t.me/?startapp=` direct link for the OPEN button | `url` inline button → `/?dest=` | **Choose the `url` button.** The direct link works, but on Telegram Desktop (Windows/Linux) it shows a START button the user must press ([tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694)); the `url` button is a single tap on every client. See §"Deep-link mechanics". |
**Installation:**
```bash
# No new runtime dependencies. The gazetteer, normalizer and scorer are
# first-party modules inside the existing npm workspaces.
```
**Version verification:** the standard stack for this phase is entirely **already installed** — `grammy@1.46.0` (verified: `node -e "require('./node_modules/grammy/package.json').version"` → `1.46.0`), `wrangler@4.151.3` (verified via `npx wrangler --version`), `vite@8.3.5`, `typescript@5.9.3`, Node `v24.18.0`, npm `11.16.0`. The only registry probes run this session were for the **rejected** alternatives: `fuse.js` → **7.5.0**, `fastest-levenshtein` → **1.0.16**, `string-similarity` → **4.0.4**, `levenshtein-edit-distance` → **3.0.1**.
---
## Architecture Patterns
### System Architecture Diagram
```text
USER (chat) ONE WORKER (bot/ + web/ statics)
──────────── ───────────────────────────────
"/cari KLCC"
│
│ webhook POST /telegram/webhook
▼
┌──────────────────┐
│ grammY router │──▶ /cari handler (bot/src/bot/commands/cari.ts)
└──────────────────┘ │
│ 1. raw query string
▼
┌─────────────────────────────┐
│ normalizeQuery() │ ← rule table
│ case/whitespace fold │ (D-09)
│ abbreviation ↔ canonical │
│ glued-word split │
└──────────────┬──────────────┘
│ normalized query
▼
┌─────────────────────────────┐
│ SearchSource.search(q) │ ← the seam (D-03)
│ ┌────────────────────────┐ │ backend #1 = GazetteerSource
│ │ tier 1: exact lookup │ │ backend #2 = WASM geocoder
│ │ tier 2: bigram filter │ │ backend #3 = Photon
│ │ tier 3: Damerau score │ │
│ └────────────────────────┘ │
└──────────────┬──────────────┘
│ ranked candidates
▼
┌─────────────────────────────┐
│ rank + tie-break + top-3 │
└──────────────┬──────────────┘
│
┌────────────────────────────┴──────────────────────┐
│ if empty → "not in v1 area" (en/ms, honest refusal) │
│ if hits → sendVenue(best) + inline buttons │
└────────────────────────────┬──────────────────────┘
│ ← ONE message (D-02)
│ reply_markup:
│ row 0: OPEN IN ARAHARAH
│ → /?dest=
│ row 1..: alternates (callback_data)
▼
╔══════════════════════════╗
║ TELEGRAM renders the ║
║ venue card (name, addr, ║
║ pin) + the buttons ║
╚══════════════════════════╝
│ user taps OPEN
▼
┌──────────────────────────────────────┐
│ Mini App webview (web/) │
│ read location.search ?dest= │
│ (fallback: initData start_param) │
│ → resolve id via BUILD-TIME COPY │
│ of the same gazetteer module │
│ → destination in memory only │
│ → Phase-3 staged boot continues │
└──────────────────────────────────────┘
```
Flow trace — the primary use case ("/cari KLCC"):
1. User types `/cari KLCC` → Telegram delivers a `message` update to `/telegram/webhook`.
2. The grammY router dispatches to the `/cari` handler (the Phase-2 alias pattern).
3. The handler extracts the raw query text **after** the command and its arguments.
4. `normalizeQuery("KLCC")` folds case/whitespace; the abbreviation table leaves it unchanged.
5. `SearchSource.search("klcc")` runs tier 1 (exact alias hash) → hits the KLCC entry immediately.
6. The handler ranks, takes the top 1 as best and up to 4 as alternates.
7. `ctx.sendVenue(lat, lng, title, address, { foursquare_type, reply_markup })` — one message.
8. The user taps OPEN → the webview opens at `/?dest=klcc-pavilion`.
9. The webview reads `location.search`, resolves the id from its bundled gazetteer copy, and hands the destination to the Phase-3 boot flow. Nothing is re-typed; nothing is persisted.
### Recommended Project Structure
```text
bot/src/
├── search/ # ← NEW: the shared search module
│ ├── types.ts # SearchSource interface, SearchResult, GazetteerEntry
│ ├── gazetteer.ts # the curated KL dataset loader (JSON import)
│ ├── data/ # ← NEW: the curated JSON (D-07)
│ │ └── kl-gazetteer.json # 200–500 rows, typed, diff-friendly
│ ├── normalize.ts # rule table + folding + glued-word split (D-09)
│ ├── rules/
│ │ └── abbreviations.ts # ← the editable rule table (D-09)
│ ├── score.ts # exact → bigram filter → Damerau
│ └── source.ts # GazetteerSource implements SearchSource (D-03)
├── bot/commands/cari.ts # ← NEW: the /cari handler (D-01)
└── i18n/strings.ts # ← MODIFIED: search copy en/ms + /cari in helpMenu
bot/scripts/
├── overpass-topup.mjs # ← NEW: repo tool, proposes candidates (D-06)
└── register-commands.mjs # ← MODIFIED (via helpMenu, no inline copy)
bot/test/
├── cari.test.mjs # ← NEW: handler + payload-shape gates
├── normalizer.test.mjs # ← NEW: per-rule vectors (D-09)
└── corpus.test.mjs # ← NEW: the golden-corpus gate (D-13/D-14)
bot/test/fixtures/
└── golden-corpus.json # ← NEW: the pinned corpus (D-13)
web/src/
├── search/ # ← NEW: build-time copy of the SAME module
│ ├── gazetteer.ts # (or a re-export from a shared location)
│ └── resolve-destination.ts # id → destination, in-memory only
├── main.ts # ← MODIFIED: boot branch for ?dest=
└── telegram/webapp.ts # ← MODIFIED: add startParam to the narrow interface
web/index.html # ← MODIFIED: ?dest= harness boot profile
web/test/
└── destination-preload.test.mjs # ← NEW: ONBD-02 gate
```
### Pattern 1: The `SearchSource` seam (D-03)
**What:** One interface, several backends. The bot only ever talks to `SearchSource`; swapping backends is a one-line change in the Worker factory, not a refactor of the handler.
**When to use:** Always — this is the phase's central abstraction and it is what makes the D-14 escalation ladder mechanical rather than aspirational.
```typescript
// Source: D-03 (CONTEXT.md) — the interface shape is this research's recommendation,
// constrained to what the phase actually needs.
export interface SearchResult {
/** Gazetteer entry id — the payload D-11 puts in the deep link. */
id: string;
name: string;
address: string;
lat: number;
lng: number;
category: string;
/** 0..1, higher is better. Used for ranking + tie-breaks only. */
score: number;
/** Which alias/form matched — useful for the corpus report, never for logging. */
matchedVia: "exact" | "normalized" | "fuzzy";
}
export interface SearchSource {
/** Normalized query → ranked candidates. Empty array = honest refusal. */
search(query: string, limit: number): Promise;
/** The backend's identity, for the escalation record (D-14). */
readonly backendId: string;
}
```
**Why this shape:** `search()` returns `Promise` even though backend #1 is synchronous. That keeps the signature stable for backend #3 (Photon is a network call) — a future swap must not change the handler.
### Pattern 2: Three-tier scoring cascade (SRCH-01, SRCH-02)
**What:** Exact-match first, cheap filter second, expensive score third. Each tier is strictly narrower than the last, so the expensive algorithm only ever sees a handful of candidates.
**When to use:** Always — this is what makes 10 ms of Workers CPU comfortable rather than tight.
```typescript
// Source: algorithm properties cited in §Sources (Cohen/Winkler 2003 for Jaro-Winkler
// prefix weighting; RapidFuzz docs for normalized Damerau-Levenshtein semantics).
// The three-tier ORDER is this research's recommendation, justified by the
// Workers CPU budget (10 ms/request, Cloudflare limits doc).
/** Tier 1 — O(1) hash lookup on the folded alias table. */
function exactLookup(index: Map, q: string): GazetteerEntry[] {
return index.get(q) ?? [];
}
/** Tier 2 — character-bigram Dice coefficient as a cheap relevance filter.
* Dice = 2·|A∩B| / (|A|+|B|) over bigram multisets. O(len) per candidate. */
function diceBigrams(a: string, b: string): number {
if (a === b) return 1;
if (a.length < 2 || b.length < 2) return a === b ? 1 : 0;
const grams = (s: string): Map => {
const m = new Map();
for (let i = 0; i < s.length - 1; i++) {
const g = s.slice(i, i + 2);
m.set(g, (m.get(g) ?? 0) + 1);
}
return m;
};
const ga = grams(a);
const gb = grams(b);
let shared = 0;
for (const [g, n] of ga) {
const other = gb.get(g);
if (other !== undefined) shared += Math.min(n, other);
}
return (2 * shared) / (a.length - 1 + b.length - 1);
}
/** Tier 3 — Damerau-Levenshtein (optimal string alignment variant is enough).
* Counts a transposition as ONE edit, which D-13's typo set requires. */
function damerau(a: string, b: string): number {
const al = a.length;
const bl = b.length;
const d: number[][] = Array.from({ length: al + 1 }, () => new Array(bl + 1).fill(0));
for (let i = 0; i <= al; i++) d[i][0] = i;
for (let j = 0; j <= bl; j++) d[0][j] = j;
for (let i = 1; i <= al; i++) {
for (let j = 1; j <= bl; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
d[i][j] = Math.min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost);
if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) {
d[i][j] = Math.min(d[i][j], d[i - 2][j - 2] + 1); // transposition
}
}
}
return d[al][bl];
}
```
**The cascade, in order:**
1. **Tier 1 — exact.** Look up the normalized query in a `Map` built at load time from every name + alias + its normalized form. A hit is a *perfect* match; score 1.0; these always sort above anything fuzzy.
2. **Tier 2 — bigram Dice filter.** Over the remaining entries, keep only those whose best alias scores above a low Dice floor. This is O(entries × aliases) but each comparison is O(len) — cheap.
3. **Tier 3 — Damerau-Levenshtein.** On the handful that survived tier 2, compute the normalized Damerau distance and convert to a 0..1 similarity. **Only apply tier 3 to strings of length ≥ 4** — below that, a single edit is a large fraction of the string and the scores are noise. (This is the same insight the taxify fuzzy-matching vignette reports: *"Jaro-Winkler can help with very short names (3-5 characters), where a single edit produces a large normalized Damerau-Levenshtein distance."*)
**Why this order, and not Jaro-Winkler as the primary:** D-09 makes Damerau the residual net. Jaro-Winkler is genuinely better on short strings (its prefix weighting is designed for it — Cohen/Winkler 2003: *"The Jaro and Jaro-Winkler metrics seem to be intended primarily for short strings"*), but its prefix bonus is a liability for a gazetteer where many entries share a first token ("Taman …", "Stesen …", "Suria …"): JW would systematically rank all "Taman X" entries above a correct non-Taman match. Bigram Dice has no such prefix bias. **Recommendation: Dice for filtering, Damerau for the final residual, exact-match always wins.** Jaro-Winkler is a reasonable swap for tier 3 if the corpus shows Damerau's normalisation mis-scores the many short acronyms (IJN, PWTC, LRT, KLIA) — that is exactly the kind of question the corpus runner exists to answer empirically.
### Pattern 3: The Malay normalizer as an editable rule table (D-09)
**What:** A bidirectional abbreviation table plus folding passes, applied identically to the query and to every alias at index time. Adding a new abbreviation is a **table edit**, not a code change.
**When to use:** Always — this is D-09's explicit architecture requirement.
```typescript
// Source: rule table content grounded in the UPU S42 Malaysian address standard
// (https://www.upu.int/UPU/media/upu/PostalEntitiesFiles/s42/mysEn.pdf), which
// lists the canonical "primary thoroughfare type" tokens, plus the OSM KL road
// list (https://en.wikipedia.org/wiki/List_of_roads_in_Kuala_Lumpur) which shows
// the "Jalan"/"Lorong"/"Persiaran"/"Lebuh" naming in KL, plus the Pos Malaysia
// /MaCGDI address examples which use "Persint 4" and "Taman".
/** Bidirectional: [from, to]. Both directions are added automatically, so the
* table holds ONE line per concept. */
export const ABBREVIATIONS: ReadonlyArray = [
// ── thoroughfare types ──
["jln", "jalan"],
["jl", "jalan"], // careful: "jl" is also used in place names — see Pitfall 6
["jl.", "jalan"],
["lnr", "lorong"],
["lr", "lorong"],
["psn", "persiaran"],
["persiaran", "persiaran"], // identity guard for the family below
["ps", "persiaran"],
["lbh", "lebuh"],
["bkt", "bukit"],
["tmn", "taman"],
["tm", "taman"],
["kg", "kampung"],
["kmpg", "kampung"],
["kampung", "kampung"], // identity guard
["presint", "presint"],
["prsn", "presint"],
["prsnt", "presint"],
["presin", "presint"],
// ── transit / civic shorthand (D-08's alias lever) ──
["stesen", "stesen"],
["stn", "stesen"],
["sta", "stesen"],
["station", "stesen"],
["hentian", "hentian"],
["st", "stesen"], // risky — see Pitfall 6
["sk", "sekolah"],
["kg smith", "kampung smith"], // multi-word: see Pitfall 7
];
```
**The normalizer pipeline** (each step is a separate, individually-testable function so the corpus runner can report *which* rule fired):
```
normalize(s):
1. Unicode NFKD + strip diacritics → "Ampang" stable under "Ampáng"
2. casefold → "jln ampang"
3. collapse whitespace → "jln ampang"
4. strip punctuation (.,/,-) → "jln ampang"
5. abbreviation table (word-wise) → "jalan ampang"
6. glued-word split (if applicable)→ "jalan ampang"
```
**Rules 5 and 6 are table-driven, not code-driven.** Rule 5 expands a token only when the token is an exact whole-word match of a table key — this avoids the "Sunway" → "Sun Way" disaster that a naive substring replace causes (see Pitfall 5).
### Pattern 4: Gazetteer entry schema (D-07, D-08, D-06)
```typescript
// Source: D-07 (typed JSON, V8-native parse, diff-friendly) + D-08 (aliases live
// inside each entry) + D-06 (hand-curated seed + Overpass-proposed top-ups).
// Fields beyond the required five are OpenCode's discretion per CONTEXT.md.
export interface GazetteerEntry {
/** Stable, URL-safe, and the payload of the D-11 deep link.
* Must match /^[a-z0-9-]+$/ — the Telegram start_param character set allows
* only A-Z a-z 0-9 _ and -, so the id MUST NOT contain a dot or underscore
* if it is ever to travel through a start_param. */
id: string;
/** Canonical display name, as a local would write it. */
name: string;
/** Street address for the venue card. May be a landmark address ("Jalan Ampang")
* when no street number exists. */
address: string;
lat: number;
lng: number;
/** Coarse class for the venue card's four-square type and for alternates
* labelling: "mall" | "transit" | "hospital" | "government" | "park" |
* "landmark" | "kampung" | "education" | "religious" */
category: string;
/** Locals' shorthand. THE main corpus-quality lever (D-08). */
aliases: string[];
/** Optional: the OSM element this entry was curated from, for the Overpass
* top-up script to reconcile against. Not required for search. */
osm?: { type: "node" | "way" | "relation"; id: number };
}
```
**The JSON file lives at `bot/src/search/data/kl-gazetteer.json`** and is imported with a plain `import ... with { type: "json" }` (or a `readFileSync` in the Node-side script). It is a **plain committed file** — the D-04 forkability law.
### Pattern 5: Sharing the gazetteer between the bot and the webview (D-11)
**What:** The webview must resolve `dest_` to a destination using "its own copy of the same gazetteer module (shared module, build-time)" (D-11).
**Why this matters:** If the two copies drift, a deep link points at a destination the webview does not recognise — the exact "stale-webview dataset" scenario D-11 acknowledges. The mitigation must be **structural**, not procedural.
**Recommendation — one source file, two import paths.** Put the canonical module in a place both workspaces can import. Three options, in order of preference:
| Option | Mechanism | Verdict |
|--------|-----------|---------|
| **A. A shared workspace package** (e.g. a new `search/` npm workspace with the gazetteer + normalizer + scorer) | `bot/` and `web/` both depend on `@araharah/search` | **Best.** One copy on disk; the build can never drift. Cost: one more workspace entry in the root `package.json`. Consistent with the repo's existing workspace pattern (`theme/` is already consumed by `web/` as a build-time dependency). |
| B. A file symlink / duplicate in both trees | `web/src/search/gazetteer.ts` re-exports from `../../bot/src/search/…` | Works with Vite (needs `server.fs.allow` for dev) but is brittle and reads badly. |
| C. A build step that copies `bot/src/search/data/*.json` into `web/` | A `prebuild` script | **Reject.** It reintroduces exactly the "token duplication" anti-pattern the repo's ARCHITECTURE.md already flags for the mockup Tailwind configs. |
**Option A is the recommendation** and it also makes the corpus runner able to import the *identical* module the production paths use — the property D-14's bake-off depends on.
### Anti-Patterns to Avoid
- **Substring replacement in the normalizer.** `s.replace("tmn", "taman")` turns "Sunway Pyramid" into "Sunaan Piyramid". Always expand **whole tokens** only, and only on exact table keys.
- **A fuzzy score that can outrank an exact alias match.** If "Kuala Lumpur City Centre" scores higher than "KLCC" for the query "KLCC", the top-3 gate passes but the product is wrong. Exact/normalized exact MUST dominate the ranking — enforce it by tier, not by a weight.
- **Building the index at module top level in the Worker.** Cloudflare's docs warn explicitly: *"avoid expensive work in global scope… a Worker must parse and execute its global scope within 1 second."* Parse the gazetteer **inside the handler** (or memoize lazily in module scope, populated on first use).
- **Logging the query text.** D-15 forbids it, and the repo's D-14 law is category-only logging. The corpus runner may print queries (it is a developer tool running locally on a committed file) but the **Worker** must never do so.
- **An N×N scan of the whole gazetteer with Damerau-Levenshtein on every alias.** 500 entries × 8 aliases × O(len²) per comparison is the CPU-budget killer. The three-tier cascade is the fix.
- **A "no results" answer that still shows a pin.** D-14 requires honest refusal for out-of-KL queries. If the gazetteer has no match, the bot must say so and offer nothing — never the nearest entry.
- **Making `startapp` a `web_app` button's field.** It isn't one. `WebAppInfo` has exactly one field: `url`.
---
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Malay abbreviation expansion | A hand-coded `if (s.startsWith("jln"))` chain | A **table** (D-09) | Locals invent shorthand faster than code changes ship. A table is a data edit a maintainer can review in a PR. |
| Fuzzy string distance | A bespoke "vowel-drop-aware" similarity | **Damerau-Levenshtein (OSA)** with a length floor | Damerau is a textbook algorithm with a 40-line implementation; getting transposition semantics right (D-13 requires them) is the only reason to write it. |
| Bigram similarity | A custom n-gram counter | The Dice coefficient over bigram multisets | Two counters and one division. A library would be a dependency for 15 lines. |
| Telegram venue cards, inline keyboards, deep links | Hand-rolled Bot API HTTP calls | **grammy 1.46.0** (`ctx.sendVenue`, `InlineKeyboardButton`) | Already a dependency; the typings are verified to support `reply_markup` on `sendVenue`. Hand-rolling would add a signing/marshalling surface for no gain. |
| Telegram initData validation for `start_param` | Re-validating the HMAC in the webview | **The existing `POST /api/session`** (Phase 2) | The webview already posts its raw `initData` to the Worker and gets a session token. `start_param` is *inside* the signed `initData`, so the same validation covers it. **Do not build a second validation path.** |
| OSM data extraction | Hand-parsing Overpass JSON into gazetteer rows | The **Overpass QL query** + `out center;` + a thin mapper | Overpass returns tagged elements; the mapping from `name`/`addr:*`/`center` to a `GazetteerEntry` is ~30 lines. |
| A geocoding index | SQLite/sql.js WASM in the Worker bundle | **Plain JSON** (D-07) | D-07's own reasoning: at 200–500 rows SQLite-in-Worker is ~1.5 MB of bundle plus cold-start cost, against a JSON file that is a few tens of KB and diffs cleanly. |
| A fuzzy-search library | `fuse.js` / `fastest-levenshtein` | The **three-tier cascade** in §Pattern 2 | See §Alternatives Considered. The hand-rolled cascade is ~150 lines, table-driven, dependency-free (D-04), and is the thing D-09's architecture describes. |
| Web-hook / deep-link token signing | HMAC over the destination id | **No crypto** (D-11) | The payload's only threat is a user pranking themselves. A plain id into a public gazetteer needs no secret. |
**Key insight:** In this phase the *hard* part is not the algorithm — it is the **data** (locals' shorthand) and the **discipline** (one module, one copy, one corpus). Every temptation to add a library or a service is a temptation to move the quality decision out of a table a maintainer can edit and into a black box they cannot.
---
## Runtime State Inventory
> This phase **creates** state but does not rename or migrate any. The inventory below answers the canonical question for the *new* state the phase introduces, since D-04/D-06/D-07 make the gazetteer a committed artifact and D-15 permits KV caching.
| Category | Items Found | Action Required |
|----------|-------------|------------------|
| **Stored data (new)** | `bot/src/search/data/kl-gazetteer.json` — the curated KL gazetteer, 200–500 rows. Derived from OSM (names, coordinates, addresses) → **ODbL-1.0 obligation** (see §Licensing). | Data file is created this phase. Ship it with the ODbL header + attribution, and record the licence in the file's first lines (a JSON field like `"_license": "ODbL-1.0"` plus a sibling `NOTICE`/`LICENSES.md` entry). |
| **Stored data (new)** | `bot/test/fixtures/golden-corpus.json` — the pinned corpus (D-13). Not OSM-derived (it is *queries* and expected ids). | Created this phase. Plain committed file, GPL-3.0 with the repo (it contains no OSM content). |
| **Live service config** | None. The `SearchSource` is in-Worker; no external service is added (D-03). The Worker already has `BOT_TOKEN`, `WEBHOOK_SECRET`, `chat_sessions` in `bot/wrangler.jsonc`. | None. |
| **OS-registered state** | None. No systemd unit, no launchd plist, no Task Scheduler entry. The bot's Telegram menu (registered via `setMyCommands`) is a *live service config* surface, not OS state — it must be re-registered after `/cari` is added to `helpMenu` (the existing `npm run bot:commands` script covers this). | Re-run `npm run bot:commands` after the copy change; the Phase-2 CS5 gate asserts no drift. |
| **Secrets / env vars** | None new. D-04's "env only" law is already true; the search backend needs no credentials (backend #3 Photon would, but it is deferred to Phase 5). | None. |
| **Build artifacts** | `web/dist` / `web/dist-harness` — the webview's copy of the gazetteer is bundled at build time. If the gazetteer changes and the web bundle is not rebuilt, the webview resolves stale ids. | The `npm run deploy` chain already builds theme → web → wrangler deploy, so the artifact refreshes on every deploy. **The Phase-3 build-artifact gate (`web/test/build-artifact.test.mjs`) is the place to add a check that the bundled gazetteer is present and id-complete.** |
| **Runtime (ephemeral)** | The resolved destination lives in **memory only** in the webview — no localStorage, no sessionStorage, no IndexedDB, no cookie (GEO-03's law continues to bind; the Phase-3 `web/test/no-storage.test.mjs` grep gate will catch a violation). | Grep gate is automatic once the code lands. |
---
## Common Pitfalls
### Pitfall 1: `web_app` inline buttons have no `start_param` field
**What goes wrong:** The team writes `web_app: { url: origin, start_param: "dest_x" }` — TypeScript rejects it, or it is cast and silently dropped. The Mini App opens with no destination and the user must re-type. ONBD-02 fails.
**Why it happens:** `WebAppInfo` has exactly one field (`url`). The `startapp` parameter belongs to the **`t.me` deep-link** form, not to the button payload.
**How to avoid:** Use a **`url` inline button** (not `web_app`) pointing at `/?dest=`; the webview reads `location.search`. Read `initData`'s `start_param` as a **secondary** path for the case where the Mini App was opened by a `t.me?startapp=` link.
**Warning signs:** The OPEN button opens the Mini App with no destination preloaded; the webview's `location.search` is empty.
### Pitfall 2: Telegram Desktop requires a second tap on a `t.me?startapp=` deep link
**What goes wrong:** On Telegram Desktop (Windows/Linux), tapping OPEN shows a START button the user must press — the "one tap to proceed" promise (D-02) breaks on that client.
**Why it happens:** [tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694): *"the user must manually press the 'START' button for the deeplink to be processed. This does not align with behavior observed on other platforms."*
**How to avoid:** Use the `url` button form. If a `t.me` deep link is used for some other surface, document the Desktop extra tap.
**Warning signs:** The feature works on Android/iOS and stalls on Desktop.
### Pitfall 3: `start_param` has a restricted character set
**What goes wrong:** A gazetteer id containing a dot (`klcc.pavilion`) or a space is passed through `startapp=` and the client drops or truncates it.
**Why it happens:** The Telegram start parameter allows only `A-Z a-z 0-9 _` and `-`, up to 512 characters; the documented validation regex is `/^[\w-]{0,512}$/`.
**How to avoid:** Constrain gazetteer ids to `/^[a-z0-9-]+$/` — lowercase, digits, hyphens. **This is a schema-level constraint that must be enforced by a test**, not by convention. It also makes ids URL-safe for the `?dest=` query string.
**Warning signs:** The webview receives `start_param` but cannot map it to an entry.
### Pitfall 4: Overpass returns HTTP 406 without a User-Agent header
**What goes wrong:** The top-up script's first fetch returns 406 and the script exits with a parse error; or a well-meaning fix adds a browser UA that gets the script rate-limited.
**Why it happens:** **Verified this session**: `curl -s -o /dev/null -w "%{http_code}" https://overpass-api.de/api/status` returned **406** with no User-Agent. The Overpass API rejects requests without an identifying UA. The rate-limit guidance also states: *"Rate limits apply per IP address on all public instances. Including a User-Agent or Referer header identifying your application is strongly recommended as best practice."*
**How to avoid:** The script sets a descriptive `User-Agent` (e.g. `araharah-gazetteer-topup/1.0 (+https://github.com/…/araharah)`) and a single request per run. Do **not** fan out per-entry requests.
**Warning signs:** `406` on the first call; `429` after several runs (the Overpass rate limit is per-IP with 2 concurrent slots; requests queue up to 15 s and beyond that are delayed).
### Pitfall 5: Substring abbreviation expansion corrupts real names
**What goes wrong:** A naive `replace("tmn","taman")` turns "Sunway Pyramid" into "Sunaan Piyramid"; `replace("kl","kuala lumpur")` turns "KLIA" into "Kualia lumpuria".
**Why it happens:** Substring replacement without token boundaries.
**How to avoid:** Expand only **whole tokens** matched against a table key, and keep a conservative table. Add a normalizer test vector for "Sunway Pyramid" and "KLIA" that asserts they are unchanged.
**Warning signs:** A corpus entry stops matching its own name after a table edit.
### Pitfall 6: Over-greedy short abbreviations collide with real tokens
**What goes wrong:** Expanding `st` → `stesen` turns "St John" into "stesen John"; expanding `jl` → `jalan` turns "Jln" variants inconsistently; `tm` → `taman` hits names like "Tmn" vs "TM".
**Why it happens:** Two-letter Malay abbreviations are ambiguous with English words and with name fragments.
**How to avoid:** (a) Keep the abbreviation table's short keys minimal and **only** apply them as a *whole-token* match; (b) for genuinely risky pairs, prefer adding an explicit **alias** on the entry (D-08's lever) over a global rule; (c) put both directions in the table so the canonical form is stable.
**Warning signs:** A corpus positive regresses after a table edit — the corpus runner is exactly the instrument that catches this.
### Pitfall 7: Glued-word splitting produces false splits
**What goes wrong:** "KampungBaru" is correctly split into "kampung baru"; but "SunwayPyramid" → "sunway pyramid" and "TamanDesa" → "taman desa" are guesses, and "KLCC" must not be split at all.
**Why it happens:** Splitting is only safe when the first segment is a **known prefix token** from a closed vocabulary (the abbreviation table's keys + a small explicit list like `kampung`, `taman`, `jalan`, `lorong`, `persiaran`, `bukit`, `stesen`, `sekolah`, `hospital`, `universiti`, `kolej`).
**How to avoid:** Split only on **known** prefixes, and only when the remainder is ≥ 3 characters. "KLCC" has no known prefix → unchanged. "Sunway" is not in the prefix vocabulary → unchanged (so "SunwayPyramid" stays glued — which is acceptable: the D-08 alias list covers it).
**Warning signs:** A query that used to match stops matching after the splitting rule is tightened or loosened.
### Pitfall 8: The gazetteer JSON is imported at module top level in the Worker
**What goes wrong:** On a cold start the Worker parses the JSON and builds the index in the global scope. Under Workers' **1 s startup limit** and the "avoid expensive work in global scope" guidance, a large gazetteer plus a 4,000-alias index build is a startup risk — and it happens on *every* isolate cold start, even for `/api/session` requests that do not need it.
**Why it happens:** `import data from "./data/kl-gazetteer.json"` at the top of a module evaluates at import time.
**How to avoid:** Parse + index **lazily inside the handler**, memoized in module scope. Also confirm the JSON import mechanism works in the Workers bundle (Vite/esbuild supports JSON imports natively; `wrangler`'s bundler does too — verify with `wrangler deploy --dry-run` and check the bundle size, which was 167.97 KiB at Phase 2).
**Warning signs:** `wrangler deploy` bundle size jumps by tens of KB; startup time metrics rise.
### Pitfall 9: ODbL share-alike is treated as a formality
**What goes wrong:** The team ships an OSM-derived gazetteer under GPL-3.0 only and declares the project "GPL-3.0" in the README — a licence incompatibility on the data file, invisible until someone notices.
**Why it happens:** The OSM wiki's *Legal FAQ* says: *"If you are writing software that uses OSM data, you may choose any license you like, including proprietary."* That sentence is about the **software**. The **database** is a separate obligation with a separate answer.
**How to avoid:** Dual-licence: **code GPL-3.0, data file ODbL-1.0**, with `© OpenStreetMap contributors` attribution in the gazetteer header, in the bot's strings, and in the README. Add a `LICENSES.md` (or an `OSMDATA-LICENSE` note) explaining the split. This is what the ecosystem does and it costs almost nothing.
**Warning signs:** A `LICENSE` file exists that mentions only GPL-3.0 while a derived data file ships in the repo.
### Pitfall 10: The corpus gate is a report, not an executable check
**What goes wrong:** The corpus is run manually once, the numbers look fine, and the phase "closes the bake-off" — but nothing prevents a later edit from silently dropping the hit rate below 90%.
**Why it happens:** D-14 states a threshold but not a mechanism.
**How to avoid:** Make the threshold an **assertion in the test suite** (`bot/test/corpus.test.mjs`): compute the top-3 hit rate over positives and assert `>= 0.90`; assert every negative yields **zero** results. The suite then fails loudly on regression and the escalation ladder becomes a concrete, triggerable event.
**Warning signs:** The corpus file changes without a corresponding test run in the commit.
### Pitfall 11: `/cari` never appears in the Telegram menu
**What goes wrong:** The command works when typed, but the menu still lists only `/help` and `/privacy` — because the menu is derived from `strings.helpMenu`, not from the registered handlers.
**Why it happens:** Phase 2's D-03 design: `register-commands.mjs` *derives* menu descriptions from `helpMenu` lines. A command that is not in `helpMenu` is not in the menu.
**How to avoid:** Add `/cari` to `strings.helpMenu` (both locales) in the same change that registers the handler, and extend the Phase-2 CS5/CS6 gates to expect it. Also note the Phase-2 finding: the menu is **client-cached**, so `setChatMenuButton` + the `all_private_chats` scope may be needed (the existing `bot/scripts/set-menu-button.mjs` covers this).
**Warning signs:** The handler exists but the menu does not list it.
### Pitfall 12: The venue card and the buttons land as two messages
**What goes wrong:** The bot sends the venue with `ctx.replyWithVenue(...)` and then a second `ctx.reply(...)` carrying the inline keyboard — violating D-02's "no second message round".
**Why it happens:** Not knowing that `sendVenue` accepts `reply_markup`.
**How to avoid:** Pass `reply_markup` in the `sendVenue` call itself. Verified in the installed typings this session.
**Warning signs:** Two consecutive bot messages per `/cari`.
---
## Code Examples
Verified patterns from official sources and the installed dependency tree.
### 1. The `/cari` handler — one message, venue card + OPEN + alternates (D-02, D-03, D-10, D-11)
```typescript
// Source: grammY 1.46.0 installed typings —
// api.d.ts:410 sendVenue(chat_id, latitude, longitude, title, address, other?)
// methods.d.ts sendVenue args include reply_markup?: InlineKeyboardMarkup | …
// markup.d.ts InlineKeyboardButton.UrlButton { text, url }
// InlineKeyboardButton.WebAppButton { text, web_app: WebAppInfo }
// markup.d.ts WebAppInfo { url: string } ← url ONLY, no start_param
import { InlineKeyboard } from "grammy";
import { sendVenue } from "../search/client.ts";
export async function replyWithShortlist(
ctx: Context,
locale: Locale,
results: SearchResult[],
): Promise {
if (results.length === 0) {
// D-14: honest refusal. No pin, no nearest-match fabrication.
await ctx.reply(strings[locale].searchNoMatch, { link_preview_options: { is_disabled: true } });
return;
}
const [best, ...alternates] = results;
// D-02: ONE message. The OPEN button + the alternates ride on the venue card.
const keyboard = new InlineKeyboard()
.text(strings[locale].searchOpenButton)
.url(`${env.miniAppOrigin}/?dest=${best.id}`) // ← url button, NOT web_app
.row()
...alternates.slice(0, 4).map((alt) =>
InlineKeyboard.text(alt.name).callback_data(`dest:${alt.id}`),
);
await ctx.replyWithVenue(best.lat, best.lng, best.name, best.address, {
foursquare_type: FOURSQUARE_BY_CATEGORY[best.category], // optional enrichment
reply_markup: keyboard,
});
}
```
> **Note on `web_app` vs `url`:** `web_app` is also valid (Telegram grants the Mini App the user's basic info and a `query_id`), but its `WebAppInfo` carries **no** `start_param`. To carry the destination through a `web_app` button you would have to put it in the Mini App URL's own query string and read it from `location.search` anyway — which is exactly what the `url` button does, with one fewer moving part. The `url` button also avoids the Telegram Desktop START-button trap (Pitfall 2).
### 2. Reading the destination in the webview (ONBD-02, D-11)
```typescript
// Source: Telegram Mini Apps docs — the start parameter arrives in the
// initData's `start_param` field AND as the GET parameter `tgWebAppStartParam`;
// character set /^[A-Za-z0-9_-]{0,512}$/.
import { getWebApp } from "../telegram/webapp.ts";
/** D-11: the deep link carries a plain gazetteer entry id. Two arrival paths:
* 1. ?dest= in the Mini App URL's own query string (the `url` button form)
* 2. start_param inside the SIGNED initData (the t.me?startapp= deep-link form)
* The id is resolved against the webview's BUILD-TIME COPY of the gazetteer.
* Nothing is persisted — memory only (GEO-03). */
export function readDestinationId(): string | null {
const fromQuery = new URLSearchParams(window.location.search).get("dest");
if (fromQuery !== null && fromQuery !== "") return fromQuery;
const wa = getWebApp();
const startParam = wa?.startParam; // ← add to TelegramWebAppLike
if (startParam !== null && startParam !== undefined && startParam !== "") {
// Tolerate the "dest_" form D-11 names.
return startParam.startsWith("dest_") ? startParam.slice(5) : startParam;
}
return null;
}
```
> **The `start_param` must come from the SIGNED `initData`, not `initDataUnsafe`.** The existing `POST /api/session` (Phase 2) already validates `initData` server-side and mints a session token from it — so if the webview wants to *trust* the `start_param`, the cleanest path is to have `/api/session` echo the validated `start_param` (or the resolved destination) back alongside the token. That avoids building a second validation path (see §Don't Hand-Roll). D-11's own framing ("the payload's only threat is a user pranking themselves") means the pragmatic reading is fine too, but the planner should choose one deliberately.
### 3. Overpass top-up query (D-06)
```overpassql
# Source: Overpass QL reference (https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL)
# bbox order: south,west,north,east (Overpass QL convention)
# `out center;` gives a way/relation a representative centre point
# `nwr` = node+way+relation in one filter
# KL metro bounding box (generous; covers the core Klang Valley sprawl)
[out:json][timeout:120][maxsize:536870912];
(
nwr["shop"="mall"](2.95,101.55,3.20,101.80);
nwr["landuse"="retail"](2.95,101.55,3.20,101.80);
nwr["tourism"="attraction"](2.95,101.55,3.20,101.80);
nwr["leisure"="park"](2.95,101.55,3.20,101.80);
nwr["railway"="station"](2.95,101.55,3.20,101.80);
nwr["station"="subway"](2.95,101.55,3.20,101.80);
nwr["public_transport"="station"](2.95,101.55,3.20,101.80);
nwr["amenity"="hospital"](2.95,101.55,3.20,101.80);
nwr["amenity"="university"](2.95,101.55,3.20,101.80);
nwr["amenity"="place_of_worship"](2.95,101.55,3.20,101.80);
nwr["place"~"^(suburb|quarter|neighbourhood|village|hamlet)$"](2.95,101.55,3.20,101.80);
nwr["office"="government"](2.95,101.55,3.20,101.80);
nwr["building"="government"](2.95,101.55,3.20,101.80);
);
out center tags 300; // `out center tags` — geometry-free + centre + tags
```
**Script rules (D-06):**
- **One** HTTP request per run, with a descriptive `User-Agent` (Pitfall 4).
- Output is a **candidate list**, never a direct write to the curated file. The maintainer reviews and approves entries into `kl-gazetteer.json`.
- The script must **not** invent aliases. Aliases are D-08's human lever. The script may *suggest* an alias when an OSM tag like `name:en`, `short_name`, `official_name`, `old_name`, or `int_name` exists — those are real data, not guesses.
- Nominatim is a reasonable **fallback** for a specific named place the bbox query missed (e.g. a mall whose OSM tag is `shop=department_store` rather than `shop=mall`). Nominatim's usage policy is stricter than Overpass's — at most one request per second, a valid identifying User-Agent, and no bulk use. **The top-up script is a maintainer tool, not a crawler**; a bulk Nominatim sweep would violate the policy.
### 4. The golden corpus format (D-13) and the threshold gate (D-14)
```json
{
"_comment": "araharah golden corpus — the executable definition of the bake-off (D-13). Each positive pins its expected entry id. Each negative must yield ZERO results (honest refusal). Reviewers extend by editing THIS FILE ONLY.",
"_version": 1,
"threshold": { "top3_hit_rate": 0.9 },
"positives": [
{ "q": "KLCC", "expect": "suria-klcc", "kind": "golden" },
{ "q": "Jln Ampang", "expect": "suria-klcc", "kind": "abbrev" },
{ "q": "Jalan Ampang", "expect": "suria-klcc", "kind": "canonical" },
{ "q": "jln ampang", "expect": "suria-klcc", "kind": "folding" },
{ "q": "Tmn", "expect": "taman-...", "kind": "abbrev" },
{ "q": "Sunway Pyrmid", "expect": "sunway-pyramid", "kind": "typo-transposition" },
{ "q": "Sunwy Pyramid", "expect": "sunway-pyramid", "kind": "typo-missing-key" },
{ "q": "DUKE", "expect": "duke-university", "kind": "acronym" },
{ "q": "Kg Baharu", "expect": "kampung-baharu", "kind": "kampung" },
{ "q": "Kg Baharu", "expect": "kampung-baharu", "kind": "glued" },
{ "q": "PWTC", "expect": "pwtc", "kind": "acronym" }
],
"negatives": [
{ "q": "Kuching", "reason": "out of the v1 area" },
{ "q": "Penang", "reason": "out of the v1 area" },
{ "q": "asdkjhaskdjh", "reason": "not a place" }
]
}
```
**The gate, as an executable assertion** (this is the piece that makes D-14 a decision rather than an opinion):
```javascript
// bot/test/corpus.test.mjs — shape follows bot/test/harness.mjs (repo precedent).
import { strict as assert } from "node:assert";
import { test } from "./harness.mjs";
import { corpus } from "./fixtures/golden-corpus.json" with { type: "json" };
import { GazetteerSource } from "../src/search/source.ts";
test("CORPUS D-14: top-3 hit rate ≥ 90% on positives AND every negative is refused", async () => {
const source = new GazetteerSource();
let hits = 0;
const misses = [];
for (const c of corpus.positives) {
const results = await source.search(c.q, 3);
const ids = results.map((r) => r.id);
if (ids.includes(c.expect)) hits++;
else misses.push({ q: c.q, expect: c.expect, got: ids });
}
const rate = hits / corpus.positives.length;
assert.ok(
rate >= corpus.threshold.top3_hit_rate,
`top-3 hit rate ${(rate * 100).toFixed(1)}% < ${corpus.threshold.top3_hit_rate * 100}% — ` +
`escalation ladder step 1 (alias/normalizer table edits) is required. Misses: ${JSON.stringify(misses)}`,
);
for (const n of corpus.negatives) {
const results = await source.search(n.q, 3);
assert.equal(results.length, 0, `negative "${n.q}" must be refused honestly, got ${JSON.stringify(results.map(r => r.id))}`);
}
});
```
**On failure, the assertion message names the escalation step.** That is D-14's ladder made mechanical: the failure text tells you to pull lever 1 (table edits), then re-run; if lever 1 cannot close the gap, lever 2 (backend #2) is next, then lever 3 (Photon). The phase directory should hold a short decision record naming which lever was pulled and the corpus run that closed it.
### 5. The normalizer as a per-rule vector table (D-09)
```typescript
// Source: D-09 — "Per-rule test vectors; extends by table edit, not code change."
export const NORMALIZER_VECTORS: ReadonlyArray<{
rule: string;
input: string;
expect: string;
}> = [
{ rule: "casefold", input: "KLCC", expect: "klcc" },
{ rule: "casefold", input: "Sunway PYRAMID", expect: "sunway pyramid" },
{ rule: "whitespace", input: " jln ampang ", expect: "jln ampang" },
{ rule: "abbrev:jln", input: "Jln Ampang", expect: "jalan ampang" },
{ rule: "abbrev:tmn", input: "Tmn Melewah", expect: "taman melewah" },
{ rule: "abbrev:kg", input: "Kg Baharu", expect: "kampung baharu" },
{ rule: "abbrev:prsn", input: "Prsn 4", expect: "presint 4" },
{ rule: "abbrev:presint", input: "Presint 4", expect: "presint 4" },
{ rule: "identity:tmn", input: "Tmn", expect: "tmn" },
{ rule: "glued:kg", input: "KampungBaru", expect: "kampung baru" },
{ rule: "glued:klcc", input: "KLCC", expect: "klcc" }, // no known prefix → unchanged
{ rule: "no-mangle:1", input: "Sunway Pyramid", expect: "sunway pyramid" }, // NOT "sunaan pyramid"
{ rule: "no-mangle:2", input: "KLIA", expect: "klia" }, // NOT "kualia lumpuria"
{ rule: "diacritic", input: "Ampáng", expect: "ampang" },
];
```
### 6. ONBD-03 — `addToHomeScreen` with an iOS-honest status read
```typescript
// Source: core.telegram.org/bots/webapps —
// addToHomeScreen() (Bot API 8.0+) — prompts the user to add the Mini App
// to the home screen; the `homeScreenAdded` event fires if the device supports it.
// checkHomeScreenStatus(cb) — cb(status) where status is one of:
// "unsupported" | "unknown" | "added" | "missed"
// Telegram-iOS issue #1729: on iOS the status is ALWAYS "unknown" — the platform
// offers no API to detect whether the shortcut exists.
export function wireHomeScreen(wa: TelegramWebAppLike | null): void {
if (!wa || typeof wa.addToHomeScreen !== "function") return;
wa.addToHomeScreen();
wa.onEvent?.("homeScreenAdded", () => {
// Android/desktop path: the shortcut exists.
});
wa.checkHomeScreenStatus?.((status: string) => {
// "unsupported" → feature unavailable here (Desktop web).
// "unknown" → iOS. Do NOT claim it failed; treat as "possibly added".
// "added" / "missed" → definitive.
});
}
```
> **Important:** `beforeinstallprompt` — the standard web-PWA install event — **does not exist inside a Telegram webview**. The Telegram-native `addToHomeScreen()` is the mechanism. Do not build a web manifest-driven install flow for ONBD-03 (that is the documented-deferred full-PWA story). Also note that **service workers do not work in Telegram Mini Apps on iOS** ([Telegram-Mini-Apps issue #27](https://github.com/Telegram-Mini-Apps/issues/issues/27)) — another reason the full-PWA story stays deferred.
---
## Licensing — the ODbL obligation (D-04 × D-06)
This section exists because it is the one place where two locked decisions collide.
**What the OSMF actually says** (fetched this session):
- OSM data is under the **Open Database License (ODbL) 1.0** — attribution + share-alike for *databases*. [foundation.openstreetmap.org/wiki/Licence_and_Legal_FAQ](http://foundation.openstreetmap.org/wiki/Licence_and_Legal_FAQ)
- *"You must also make the derivative database available to anyone who received your data… If you make a database which includes OSM data and any additional information… then this would be classed a 'derivative database' and should be made available under the Open Database License."* [Legal FAQ](https://openstreetmap.org/wiki/Legal_FAQ)
- *"if you have any data that was derived from OSM — for instance because you used street names from OSM… then you are making a derivative database."* [License/Use Cases, Case 3](https://wiki.openstreetmap.org/wiki/License/Use_Cases)
- *"A 'geocoding database' should **NOT** be considered a trivial transformation."* [Trivial Transformations Guideline](https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline)
- ODbL §4.4: a publicly distributed Derivative Database must be under the ODbL, a later version with the same Licence Elements, or *"a compatible licence."*
- The OSMF's compatibility guidance says licences that require downstream users to use the data on *identical* terms are generally incompatible with ODbL.
**The conclusion:** a curated gazetteer of KL POIs whose names, addresses and coordinates come from OSM is a **Derivative Database**. The software around it may be GPL-3.0 (D-04), but the **data file** carries an ODbL share-alike obligation.
**Recommended resolution — dual-licence, the ecosystem-standard shape:**
| Artifact | Licence | Note |
|----------|---------|------|
| All source code (`bot/`, `web/`, `theme/`) | **GPL-3.0** | D-04. Add the `LICENSE` file this phase (the CONTEXT canonical refs call for it). |
| `bot/src/search/data/kl-gazetteer.json` (and any future gazetteer dump) | **ODbL-1.0** | A data file, not code. Header carries `© OpenStreetMap contributors` + a link to . |
| `bot/test/fixtures/golden-corpus.json` | **GPL-3.0** with the repo | It contains queries and entry ids, no OSM content. |
| Attribution in the product | **Required** | The bot's venue card / replies should carry `© OpenStreetMap contributors` — the web app already ships exactly this string in `web/src/i18n/strings.ts` (`attribution`), so the bot's strings module needs the twin. |
**Why this satisfies D-04 rather than breaking it.** The forkability laws are about *mechanics* — no hardcoded secrets, plain committed data files, one-command deploy, one free-tier account per fork. A dual-licence keeps every one of those: the data file is still a plain committed file, and a fork inherits the ODbL obligation *on that file* — which is the correct, legally-required, entirely workable outcome. It is exactly what OSM-derived datasets in the ecosystem do (the OSM attribution notice is already visible in this project's UI).
**Optional defence-in-depth:** the D-06 Overpass script can *propose* entries, but the maintainer can hand-write the display name, the address string, and the aliases from local knowledge, using OSM only for coordinates and for a sanity check. That reduces (though does not eliminate) the "derived" exposure. **Do not rely on it** — the dual-licence is free.
---
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Photon-on-VPS as the v1 search backend (the roadmap's original Phase-4 note) | **In-Worker curated gazetteer behind a `SearchSource` interface** (D-03) | Decided 2026-10-23 (this phase's CONTEXT.md) | Removes all infra cost and the VPS dependency from Phase 4. Photon moves to backend #3 behind the same interface. |
| Full-text search engine (Tantivy/Meilisearch/Typesense) | **Three-tier in-Worker cascade** (exact → bigram filter → Damerau) | Inherent to the 200–500 row scale | No service to run, no network hop, no query latency. The 10 ms CPU budget is comfortable. |
| Hand-typing the abbreviation expansion rules in code | **An editable rule table** with per-rule vectors (D-09) | Inherent to D-09 | Corpus quality becomes a data-review activity, not a code-review activity. |
| `t.me/?startapp=` deep link as the OPEN button | **`url` button → `/?dest=`**, `start_param` as a secondary path | Pitfall 2 (Desktop START-button trap) | One tap on every client; no START-button detour. |
| Web PWA install (`manifest` + `beforeinstallprompt`) for the home screen | **Telegram-native `WebApp.addToHomeScreen()`** (Bot API 8.0+) | Bot API 8.0, Nov 2024 | The only mechanism that works inside a Telegram webview; iOS status is always `unknown`. |
| SQLite/sql.js WASM index in the Worker | **Typed JSON** (D-07) | D-07's own reasoning | ~1.5 MB bundle + cold-start cost avoided; diff-friendly PRs. |
| Geocoding database as a "trivial transformation" of OSM | **Explicitly NOT trivial** per the OSMF guideline | Guideline endorsed by the OSMF board 2014-06-06 | The ODbL share-alike obligation applies — see §Licensing. |
**Deprecated/outdated:**
- **`beforeinstallprompt`** — does not fire inside a Telegram webview. Not a valid ONBD-03 mechanism.
- **Service workers in Telegram Mini Apps on iOS** — confirmed broken ([Telegram-Mini-Apps issue #27](https://github.com/Telegram-Mini-Apps/issues/issues/27), still open with a 2025-09 update). Another reason the full-PWA/offline story stays deferred.
- **OSM's old "geocoding databases may be trivial" hope** — the guideline says the opposite.
- **The roadmap's Phase-4 VPS/Photon/GraphHopper plan** — superseded by D-03/D-05.
---
## Assumptions Log
> All claims tagged `[ASSUMED]` in this research, for the planner and discuss-phase to confirm.
| # | Claim | Section | Risk if Wrong |
|---|-------|---------|------------|
| A1 | The abbreviation table's canonical list (Jln/Jalan, Tmn/Taman, Kg/Kampung, Lorong, Persiaran, Presint/Prsn, Stesen/Stn) is sufficient for the KL corpus. Grounded in the UPU address standard and OSM's KL road list, but **no authoritative KL-specific abbreviation table exists** — this is a curated approximation. | §Pattern 3 | Low-medium. The corpus runner measures the gap; lever 1 (table edits) is the designed remedy. |
| A2 | The KL bounding box `2.95,101.55,3.20,101.80` is the right scope for the Overpass top-up. Plausible from the metro extent, not verified against a canonical "Klang Valley" boundary. | §Code Examples 3 | Low. A wider box costs a longer Overpass run; a narrower one misses outliers. The bbox is a script constant, trivially edited. |
| A3 | `sendVenue` renders acceptably on all Telegram clients (iOS/Android/Desktop) with a `reply_markup` attached. **Not verified on a real device this session** — the API surface is verified, the rendering is not. | §Code Examples 1 | Medium. Rendering is cosmetic; the fallback (a text message with an inline keyboard) is trivial if a client renders the card badly. **Recommend it as a real-client checkpoint.** |
| A4 | The `url` inline button to the Mini App origin behaves identically to a `web_app` button on all clients (single tap, Mini App opens, the app's own query string is preserved). The API surface is verified; the client behaviour is not device-tested. | §Code Examples 1, Pitfall 1 | Medium-high if wrong — this is the ONBD-02 hand-off. **Recommend it as a real-client checkpoint** (the Phase-3 device-matrix pattern applies). |
| A5 | A ~500-row gazetteer with ~8 aliases each, parsed and indexed per request, fits comfortably inside the Workers Free 10 ms CPU budget. The 10 ms figure is verified; the *measured* cost of this specific workload is not. | §Summary finding 4 | Low-medium. The design has headroom by construction (tier-1 O(1) lookup short-circuits most queries), and `wrangler dev` can measure it. |
| A6 | The OSM licensing analysis (curated gazetteer = Derivative Database → ODbL share-alike; dual-licence resolves it) is correct **as an engineering practice**, but this is legal guidance, not legal advice. The OSMF wiki is community guidance and explicitly disclaims being authoritative. | §Licensing | Medium for the *project*, low for the *code*. The dual-licence costs almost nothing and is strictly safer than the alternative. A maintainer who wants certainty should ask the OSMF licensing mailing list. |
| A7 | The `dest_` payload in `start_param` will arrive inside `initData` and be signed by Telegram in the `t.me?startapp=` path (and therefore validated by the existing `/api/session`). The `start_param` field exists in `WebAppInitData` per the docs, but this session did **not** verify its presence in a real signed `initData` sample. | §Code Examples 2 | Low. The primary path (`?dest=` in the URL) does not depend on it. |
| A8 | The Telegram `/cari` command alias scheme (bilingual) has room in `strings.helpMenu` without violating the Phase-2 CS5 gap-freedom gate — the gate currently expects exactly `/help` and `/privacy` and will need extending. | §Pitfall 11 | Low. It is a mechanical test update. |
| A9 | The gazetteer seed can be assembled from public knowledge of KL landmarks (KLCC, Sunway Pyramid, DUKE, PWTC, Pavilion, IJN, etc.) with accurate coordinates. **Coordinates for the seed entries should still be sourced from OSM** to avoid errors — so the ODbL analysis stands regardless of where names come from. | §Pattern 4 | Low for the coordinates; medium for exact addresses, which should come from OSM's `addr:*` tags. |
---
## Open Questions
1. **Should the OPEN button be a `url` button or a `web_app` button?**
- **What we know:** `WebAppInfo` has only `url` — no `start_param` field exists on either button type. The `t.me?startapp=` deep link carries `start_param` but needs an extra START tap on Telegram Desktop. The `url` button with `?dest=` is single-tap everywhere.
- **What's unclear:** Whether a `web_app` button grants the Mini App anything the `url` button does not (it grants `query_id` and basic user info, and permits `answerWebAppQuery`). Phase 4 does not need `answerWebAppQuery` — the bot sends the card.
- **Recommendation:** **Use the `url` button.** Revisit only if Phase 5/6 needs the Mini App to message back on the user's behalf.
2. **Should `/api/session` echo the validated `start_param` back to the webview?**
- **What we know:** `start_param` is inside the signed `initData`, so the existing Phase-2 validation already covers it. The webview currently receives only `{token}` from `/api/session`.
- **What's unclear:** Whether trusting `initDataUnsafe.start_param` client-side is acceptable (D-11 says the threat is only self-pranking) or whether the server should re-derive it.
- **Recommendation:** **Have `/api/session` return the validated `start_param` alongside the token.** One extra field, zero new validation path, and it makes the deep-link payload trustworthy by construction. It also keeps the claim surface (D-08) unchanged — `start_param` is not stored in KV.
3. **Is a result cache worth it?**
- **What we know:** At 200–500 rows the search is sub-millisecond; D-15 permits TTL-only KV caching keyed by normalized query; D-15 also notes the query is user payload (so a cache key derived from it is a fingerprint of the user's activity, even if the content is public place data).
- **What's unclear:** Whether the Workers CPU budget ever gets tight enough to matter — it does not at this scale.
- **Recommendation:** **Ship without a cache.** It is a privacy cost with no performance benefit. If a future backend makes search slow, cache then — and key by a **hash** of the normalized query rather than the query itself.
4. **Where does the shared search module live?**
- **What we know:** Option A (a new npm workspace, e.g. `@araharah/search`) gives one on-disk copy consumed by both `bot/` and `web/`, mirroring the existing `theme/` → `web/` pattern. Options B and C exist but reintroduce duplication risk.
- **What's unclear:** Whether adding a fourth workspace is acceptable to the maintainer, given D-04's "one-command deploy" law and the existing three-workspace layout.
- **Recommendation:** **Option A.** It is the only option that makes D-11's "id mapping stays in sync by construction" literally true, and it lets the corpus runner import the identical module the production paths use.
5. **How large should the seed gazetteer be, and who curates it?**
- **What we know:** D-07 says 200–500 rows; D-13 says 60–100 corpus queries; D-06 says hand-curated seed + Overpass top-up. The coverage that matters is *recognisable KL landmarks*, not exhaustiveness.
- **What's unclear:** Whether 200 rows can cover the corpus's ~60–100 positives plus enough alternates to make D-02's shortlist meaningful.
- **Recommendation:** **Seed with ~120–150 entries** covering malls, transit hubs, major hospitals, universities, government complexes, parks, and ~30 kampung/neighbourhood names — enough for the corpus and for a 3–5 item shortlist per query. Grow via the Overpass script.
---
## Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Node.js | test runner, build scripts | ✓ | **v24.18.0** | — |
| npm | workspace installs, scripts | ✓ | 11.16.0 | — |
| `grammy` | `/cari` handler, `sendVenue` | ✓ (installed) | **1.46.0** | — |
| `wrangler` | dev on 8787, deploy | ✓ (installed) | 4.151.3 | — |
| `typescript` | typecheck | ✓ (installed) | 5.9.3 | — |
| `vite` | web build (gazetteer copy) | ✓ (installed) | 8.3.5 | — |
| `playwright` + browsers | ONBD-02 capture evidence | ✓ (cached) | chromium-1248, webkit-2272, firefox-1511 | — |
| Cloudflare Workers | runtime | ✓ (account + deploy) | Free plan, 10 ms CPU / 100k req/day | — |
| `chat_sessions` KV | (not needed — no cache recommended) | ✓ (binding configured) | id `59f58a54…` | — |
| **Overpass API** (network) | D-06 top-up script | ✓ (reachable) | `overpass-api.de` — **returns 406 without a User-Agent header** (verified this session) | Other public instances: `overpass.kumi.systems`, `overpass.private.coffee`. The script sets a UA (Pitfall 4). |
| **Nominatim** (network) | D-06 fallback for a missed named place | ? (not probed this session) | `nominatim.openstreetmap.org` | Strict usage policy — 1 req/s, identifying UA, no bulk. Maintainer-tool use only. |
| **Real Android device** | ONBD-02/ONBD-03 real-client gate | **?** | — | **None.** The Phase-3 device matrix pattern applies: arm a checklist, let the user walk it. |
| **Real iOS device** | ONBD-03 `addToHomeScreen` (iOS status is always `unknown`) | **?** | — | **None.** Phase 3 recorded the iOS leg as UNVERIFIED (no device). |
| **Telegram Desktop** | Pitfall 2 (START-button trap) verification | **?** | — | **None.** The `url`-button choice sidesteps the trap; a Desktop spot-check still confirms it. |
**Missing dependencies with no fallback:**
- Real devices for the ONBD-02/ONBD-03 client matrix. The Phase-3 precedent (`.planning/phases/03-mini-app-shell-geolocation/03-DEVICE-MATRIX.md`) is the model: a written checklist with the deployed URL, checked by the user. **The planner must surface this as a checkpoint before the exit-gate task, not discover it at execution.**
**Missing dependencies with fallback:**
- Overpass instance availability — three public instances exist; the script should accept an override.
- A real Telegram client for the "one message with venue card + buttons" confirmation — the bot-side unit tests cover the *payload shape*; the *rendering* needs a human. (This is assumption A3.)
---
## Validation Architecture
> `.planning/config.json` → `workflow.nyquist_validation: true` (confirmed present and `true`). Included per protocol.
### Test Framework
| Property | Value |
|----------|-------|
| Framework | **Hand-rolled zero-dependency runner** — `bot/test/runner.mjs` + `bot/test/harness.mjs` (`test(name, fn)` registry). `web/test/runner.mjs` is the same shape (copied verbatim, workspace-agnostic). No vitest/jest — by design (repo precedent since Phase 1's `theme/scripts/audit.test.mjs`). |
| Config file | none (plain `.mjs` + `.ts` imported via dynamic `import()` — the runner already does `await import("../src/i18n/strings.ts")`, which works because Node 24 strips TypeScript types natively). |
| Quick run command | `npm test -w bot` (bot suites) / `npm test -w web` (web suites) / `npm run bot:test` + `npm run web:test` from the root |
| Full suite command | `npm run bot:test && npm run web:test && npm run bot:typecheck && npm run web:typecheck && npm run build -w theme && npm run build -w web` |
| Client/render checks | `npm run capture -w web` — Playwright against `web/dist-harness`; the existing harness already races `RAMS-BASEMAP-READY` → `ARAHARAH-LIVE` and enforces a `MIN_BYTES` blank-bar check. A `?dest=` boot profile slots straight into the existing capture list. |
| Corpus runner | **A new suite in the same shape**: `bot/test/corpus.test.mjs` — imports `bot/test/fixtures/golden-corpus.json` and the real `GazetteerSource`, asserts the D-14 gate. No new runner, no new framework. |
**⚠️ ESM/TS caveat (carried from Phase 3's research):** the runner dynamic-imports `.ts` files directly. This works on Node 24 (type-stripping) and is the established repo pattern (`bot/test/commands.test.mjs` does `await import("../src/i18n/strings.ts")`). **The shared search module must therefore be importable from plain Node without a bundler** — i.e. no `import ... with { type: "json" }` assertion in the *source* module if it would break the Node import; load the JSON via `readFileSync` + `JSON.parse` against a path resolved from `import.meta.url` if the assertion proves fragile. **Test this early** — it is the single most likely mechanical blocker in the phase. (Note the corpus fixture itself is loaded by the *test*, not by the source module, so it can use a JSON import assertion without affecting production code.)
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| **SRCH-01** | Query "KLCC" returns the KLCC entry in the top-3 | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | The `SearchSource` interface is satisfied by `GazetteerSource` and the seam is the only entry point the handler uses | unit (structural) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | A gazetteer entry with no match yields an **empty** array (not a nearest-neighbour fabrication) | unit | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | `backendId` identifies the active backend — the escalation record needs it (D-14) | unit | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | The top-3 hit rate over the corpus positives is **≥ 90%** — the D-14 threshold, as an executable assertion | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | "Jln Ampang" and "Jalan Ampang" normalize to the same string and resolve to the same entry id | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | "Tmn" ↔ "Taman", "Kg" ↔ "Kampung", "Prsn" ↔ "Presint" round-trip (bidirectional table) | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | Glued-word split: "KampungBaru" → "kampung baru"; "KLCC" unchanged | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | No mangling: "Sunway Pyramid" and "KLIA" survive normalization unchanged | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | Case/whitespace folding is idempotent | unit | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | A transposition typo ("Sunway Pyrmid") resolves to the same entry as the canonical spelling | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | A missing-key typo ("Sunwy Pyramid") resolves to the same entry | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The handler sends **exactly one** message per `/cari` — the venue card carries the reply markup | unit (payload shape, fake bot) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The venue payload carries `latitude`, `longitude`, `title`, `address` (the four required `sendVenue` fields) | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The OPEN button's URL carries the destination id and points at the Mini App origin | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | 2–4 alternates ride on the **same** message's inline keyboard (D-02) | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | A no-match query sends an honest refusal with **no** pin and no inline keyboard | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The venue card's address/title carry the OSM attribution string (ODbL) | unit (copy gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The search copy exists in **both** `en` and `ms` with identical key-sets (the Phase-2 CS1 law extends) | unit (copy gate) | `npm test -w bot` | ❌ Wave 0 |
| **ONBD-02** | `dest_` maps to a destination in the webview's gazetteer copy — id → name/lat/lng/address | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-02** | A **missing or unknown** id degrades to the normal Phase-3 boot (no crash, no blank) | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-02** | The webview resolves the id from `location.search` **and** from `initData.start_param` | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-02** | The harness boot profile `?dest=` renders the destination preloaded (capture evidence) | integration (Playwright) | `npm run capture -w web` | ❌ Wave 0 |
| **ONBD-02** | The gazetteer id charset is enforced: every id matches `/^[a-z0-9-]+$/` | unit (schema gate) | `npm test -w bot` | ❌ Wave 0 |
| **ONBD-02** | The bot and web copies of the gazetteer expose the **same id set** (drift gate — the D-11 "in sync by construction" assertion, made executable) | unit | `npm test -w bot` (+ a web-side check) | ❌ Wave 0 |
| **ONBD-03** | `addToHomeScreen` is wired when available and is a **no-op** when the API is absent (Desktop web) | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-03** | `checkHomeScreenStatus` handles all four statuses, treating `unknown` as "possibly added" (iOS), never as failure | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-03** | A boot with **no** `start_param` and **no** `?dest=` lands in the Phase-3 staged boot (search-ready map) — no crash, no blank | unit (web) + integration (capture) | `npm test -w web`, `npm run capture -w web` | ❌ Wave 0 |
| **Privacy (D-15)** | No KV write in the search path; the query text never reaches a log line | unit (grep gate, mirroring the Phase-2 `privacy.test.mjs`) | `npm test -w bot` | ❌ Wave 0 |
| **Privacy (D-15/GEO-03)** | The webview's destination is **memory only** — no storage API referenced | unit (grep gate, `web/test/no-storage.test.mjs` precedent) | `npm test -w web` | ❌ Wave 0 |
| **Licensing (D-04)** | The gazetteer JSON carries an ODbL notice + OSM attribution; a `LICENSE` (GPL-3.0) exists | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 |
| **D-09 (structural)** | The abbreviation table is data, not code — the normalizer reads it from a table module | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 |
| **D-11 (structural)** | No crypto/HMAC/signature in the deep-link payload path | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 |
| **Corpus integrity** | Every corpus positive's `expect` id exists in the gazetteer; every id is unique; ids match the charset | unit (schema gate) | `npm test -w bot` | ❌ Wave 0 |
| **Real-device gate** | ONBD-02/ONBD-03 client behaviour: single-tap OPEN, home-screen shortcut, iOS `unknown` | **manual-only** | — | n/a — Phase-3 device-matrix pattern |
| **SRCH-03 rendering** | The venue card + inline buttons render correctly in a real Telegram client | **manual-only** | — | n/a — assumption A3 |
### Sampling Rate
- **Per task commit:** `npm run bot:test` (search module, normalizer, corpus, handler) and `npm run web:test` (destination resolution) — whichever workspace the task touched.
- **Per wave merge:** `npm run bot:test && npm run web:test && npm run bot:typecheck && npm run web:typecheck && npm run build -w theme && npm run build -w web`
- **Phase gate:** full suite green **AND** the corpus gate green at ≥90% **AND** `npm run capture -w web` produced the ONBD-02 evidence capture **AND** the real-device client matrix walked by the user (manual, blocking — Phase-3 precedent).
### Wave 0 Gaps
- [ ] `bot/src/search/` module tree — `types.ts`, `gazetteer.ts`, `normalize.ts`, `rules/abbreviations.ts`, `score.ts`, `source.ts` (the shared module; Wave 0 because everything depends on it)
- [ ] `bot/src/search/data/kl-gazetteer.json` — the seed gazetteer (~120–150 entries) with ODbL header + attribution
- [ ] `bot/src/bot/commands/cari.ts` — the `/cari` handler
- [ ] `bot/test/normalizer.test.mjs` — per-rule vectors (D-09)
- [ ] `bot/test/corpus.test.mjs` — the D-13/D-14 executable gate
- [ ] `bot/test/fixtures/golden-corpus.json` — the pinned corpus
- [ ] `bot/test/cari.test.mjs` — payload-shape gates (one message, venue fields, OPEN url, alternates, honest refusal, attribution)
- [ ] `bot/scripts/overpass-topup.mjs` — the D-06 repo tool
- [ ] `web/src/search/resolve-destination.ts` — id → destination (memory only)
- [ ] `web/test/destination-preload.test.mjs` — ONBD-02 gates
- [ ] `web/index.html` — the `?dest=` harness boot profile
- [ ] `bot/src/i18n/strings.ts` — MODIFIED: search copy en/ms + `/cari` in `helpMenu` + the OSM attribution string
- [ ] `bot/test/commands.test.mjs` — MODIFIED: CS5/CS6 must now expect `/cari` in the menu (the Phase-2 gap-freedom gate is a **hard gate that will fail** until this is updated)
- [ ] `LICENSE` (GPL-3.0) + `LICENSES.md` or an ODbL notice for the data file
- [ ] Shared-module wiring — a fourth npm workspace (e.g. `@araharah/search`) so `bot/` and `web/` import one copy
*(No gap: the runner and harness already exist in both workspaces. Every suite listed above is new.)*
---
## Security Domain
> `security_enforcement` is absent from `.planning/config.json` → treated as enabled. Included per protocol.
### Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| V2 Authentication | yes | The webview's `initData` is HMAC-validated by the **existing** Phase-2 `POST /api/session` (never re-implemented). A `start_param` read from `initDataUnsafe` is untrusted unless `/api/session` echoes the validated value (Open Question 2). |
| V3 Session Management | yes | Unchanged from Phase 2 — opaque tokens, TTL-only KV, rotate-on-new. The search path adds **no** session or cache state. |
| V4 Access Control | yes | The search module is pure and has no I/O; the `/cari` handler answers only the chat the command came from (`ctx.chat.id` from the update, never from user input). No endpoint accepts a search query from the browser. |
| V5 Input Validation | yes | The query string is length-capped before normalization (a hostile 100 KB argument must not reach the scorer); the gazetteer id is validated against `/^[a-z0-9-]+$/`; the `start_param` is checked against the Telegram charset before it is used as a map key. |
| V6 Cryptography | **no** (by design) | D-11: *"No crypto: the payload's only threat is a user pranking themselves."* The deep-link payload is a plain id into a public gazetteer. **A structural gate must assert that no HMAC/signature code enters the search or deep-link path.** |
| V9 Communications | yes | The Overpass script is a repo tool with an identifying User-Agent; it is never invoked at runtime. The Worker makes **no** outbound search requests (backend #1 is in-Worker). |
| V14 Configuration | yes | No new env vars, no new secrets (D-04's "env only" law already holds). The Mini App origin the OPEN button points at is **not** a secret — it is public and already in `bot/wrangler.jsonc` / the deployed URL. |
### Known Threat Patterns for Cloudflare Workers + Telegram + a curated gazetteer
| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| **Query-text leakage** (the query is user payload — it can reveal where a user is going or where they live) | Information Disclosure | D-15: category-only logging; the query is never written to KV or to a log line. A grep gate pins this (Phase-2 `privacy.test.mjs` precedent). **Do not add a result cache** (a normalized-query cache key is an activity fingerprint even when the content is public). |
| **Forged deep link** (a user crafts `?dest=`) | Tampering | No impact — the id resolves to a *public place*. D-11's own assessment. The id is not a capability and grants nothing. |
| **Gazetteer id injection** (an id containing characters that break the URL or the `start_param` charset) | Tampering | Enforce `/^[a-z0-9-]+$/` on every entry id at load time; reject the entry (fail closed) rather than emit a bad URL. |
| **Hostile query argument** (megabyte-long string to burn CPU) | DoS | Cap the raw query length before normalization (e.g. 256 chars — no real destination exceeds it). Cap the normalized length too. The tier-1 hash lookup is O(1) regardless. |
| **CPU exhaustion via the scorer** | DoS | The three-tier cascade bounds the expensive algorithm to a handful of candidates. Workers Free's 10 ms CPU limit is a backstop. `wrangler dev` can measure the real figure. |
| **Overpass abuse** (a maintainer script that hammers the public instance) | DoS / reputation | One request per run, identifying User-Agent, manual invocation only. Overpass's per-IP rate limit (2 slots, requests queued up to 15 s) is the guardrail. |
| **Stale-webview dataset** (the web bundle's gazetteer lags the bot's) | Tampering (functional) | Structural: one shared module (Option A) + an id-set drift gate in the test suite. The `npm run deploy` chain rebuilds both from one source. |
| **Cross-chat leakage** (a search result sent to the wrong chat) | Information Disclosure | The handler replies to `ctx.chat.id` from the update only; no chat id is accepted from user input. |
| **XSS / markdown injection** (a query echoed back into a reply) | Tampering | Telegram's parse mode handles it, but the safest design is to **not echo the query** in the reply at all (the clinical Rams voice says what it found, not what was typed). If the query must appear, escape it or send without a parse mode. |
---
## Sources
### Primary (HIGH confidence)
- **Official Telegram Bot API** — (fetched 2026-10-23): `addToHomeScreen()` / `checkHomeScreenStatus()` and the four statuses; the `start_param` field and the `tgWebAppStartParam` GET parameter; the `t.me?startapp=` deep-link form; inline vs direct-link Mini App behaviour; `mode=compact`.
- **Official Telegram Bot API reference** — (fetched 2026-10-23): method list; `sendVenue` exists; `setMyCommands` scope rules.
- **Installed `grammy@1.46.0` + `@grammyjs/types`** — read directly from `node_modules` this session: `api.d.ts:410` (`sendVenue` signature), `context.d.ts:679-690` (`replyWithVenue`), `methods.d.ts:735-782` (`sendVenue` args including `reply_markup`), `markup.d.ts:11-60` (`InlineKeyboardButton` variants incl. `UrlButton` and `WebAppButton`), `markup.d.ts:219-222` (`WebAppInfo { url: string }` — **only** `url`), `message.d.ts:1072-1086` (`Venue` fields incl. `foursquare_id` / `foursquare_type` / `google_place_id` / `google_place_type`).
- **Cloudflare Workers Limits** — (fetched 2026-10-23, page dated 2026-10-12): Free plan **10 ms CPU / HTTP request**, 100,000 req/day, 128 MB memory, 64 MiB uncompressed bundle, **1 second startup**, "avoid expensive work in global scope", "Most Workers consume ~2.2 ms per request".
- **OSMF Legal FAQ** — : derivative-database definition; the requirement to make the derivative database available.
- **OSMF Licence/Use Cases** — (rev. 2025-01-28): Case 3 — *"if you have any data that was derived from OSM… then you are making a derivative database"*; Case 2 — a pure-OSM publication is a "produced work" needing attribution.
- **OSMF Trivial Transformations Guideline** (endorsed by the OSMF board 2014-06-06) — : *"A 'geocoding database' should **NOT** be considered a trivial transformation."*
- **OSMF Licence Compatibility** — : identical-terms licences generally incompatible with ODbL.
- **UPU S42 Malaysian address standard** — : the canonical Malaysian address grammar (primary thoroughfare type = Jalan / Taman / Kampung; "Persiaran"; "Presint"; district level = Taman / Kampung / Seksyen) with worked address examples.
- **OSM Overpass QL reference** — : `bbox` order (south,west,north,east); `nwr` filter; `out center`; `out:csv`/`out:json`; `[timeout:N]` / `[maxsize:N]`.
- **Overpass API User's Manual — Rate Limit** — : 2 slots/IP, requests queue up to 15 s, HTTP 429 when denied; small-request priority; identifying UA recommended.
- **Malaysian address data** — UPU addressing unit () and the OSM-derived KL road list () — for the abbreviation vocabulary and the "Jalan / Lorong / Persiaran" naming pattern.
- **Repo codebase** — read directly this session: `bot/src/{webhook.ts,bot/bot.ts,config.ts,i18n/strings.ts,kv/sessions.ts}`, `bot/test/{runner.mjs,harness.mjs,commands.test.mjs}`, `bot/wrangler.jsonc`, `web/src/{main.ts,telegram/webapp.ts,i18n/strings.ts,mock/telegram-mock.ts}`, `web/index.html`, `web/test/runner.mjs`, `web/scripts/capture.mjs`, and the Phase-2/Phase-3 summaries and RESEARCH files.
### Secondary (MEDIUM confidence)
- **Telegram-iOS issue #1729** — : *"Home screen status detection always returns 'unknown' on iOS… On iOS it is not possible to determine whether a Mini App has been added to the Home Screen."* (Official client repo issue; the OSM wiki pages above also carry a community-guideline disclaimer.)
- **telegramdesktop issue #28694** — : the Desktop START-button requirement for `t.me` deep links.
- **Telegram-Mini-Apps issue #27** — : service workers do not work in Telegram Mini Apps on iOS (still open).
- **Telegram Mini Apps platform docs** — and : the `start_param` charset (`/^[\w-]{0,512}$/`), the `home_screen_added` / `home_screen_checked` events and the four statuses.
- **Cohen, W. W.; Ravikumar, P.; Fienberg, S. E. (2003)**, *A Comparison of String Distance Metrics for Name-Matching Tasks*, KDD Workshop — : Jaro/Jaro-Winkler are "intended primarily for short strings"; prefix weighting behaviour.
- **taxify fuzzy-matching vignette** — : Damerau-Levenshtein charges 1 edit for a transposition (Levenshtein charges 2); Jaro-Winkler helps on 3–6-character names; threshold guidance.
- **RapidFuzz reference** — : normalized Damerau-Levenshtein semantics (0..1) used to define the tier-3 similarity conversion.
- **npm registry probes** (2026-10-23): `fuse.js` → 7.5.0, `fastest-levenshtein` → 1.0.16, `string-similarity` → 4.0.4, `levenshtein-edit-distance` → 3.0.1 (all rejected); `grammy` 1.46.0, `wrangler` 4.151.3, `vite` 8.3.5, `typescript` 5.9.3, Node 24.18.0, npm 11.16.0 (all already installed).
### Tertiary (LOW confidence — marked for validation)
- **The exact `sendVenue` + `reply_markup` rendering on iOS/Android/Desktop clients** — the API surface is verified, the rendering is not device-tested (assumption A3).
- **The `url` inline button opening the Mini App with its query string intact on every client** — not device-tested (assumption A4).
- **The exact KL bounding box** for the Overpass top-up (assumption A2).
- **The sufficiency of the abbreviation table** for KL (assumption A1) — the corpus runner is the instrument that settles it.
---
## Metadata
**Confidence breakdown:**
- **Standard stack:** HIGH — every library is already installed and its type surface was read directly from `node_modules` this session; the only new dependency decision is *not to add one*, and the rejected alternatives were registry-probed.
- **Architecture:** HIGH — the `SearchSource` seam, the three-tier cascade, and the shared-module pattern are all grounded in verified API surfaces, the 10 ms CPU budget, and the repo's own precedent. MEDIUM on the `url` vs `web_app` button choice (device-dependent).
- **Pitfalls:** HIGH — every pitfall is anchored to a fetched doc, an installed typing, an official repo issue, or a measured behaviour (the Overpass 406). The device-dependent ones are labelled as assumptions.
- **Licensing:** MEDIUM-HIGH — the analysis rests on three OSMF wiki pages plus the ODbL text, all fetched this session; but OSMF guidance is community guidance, and this is not legal advice. The recommended dual-licence is the conservative, ecosystem-standard resolution.
**Research date:** 2026-10-23
**Valid until:** 2026-11-06 (30 days) — the Telegram and Cloudflare surfaces are stable; the assumption that needs re-checking first is the Overpass instance's UA behaviour and the OSM licence guidance (both move slowly).
---
*Phase: 04-destination-search-kl-gazetteer*
*Research complete — planner may create PLAN.md files.*