# Phase 4: Destination Search + KL Gazetteer - Research
**Researched:** 2026-10-23
**Domain:** Telegram Bot API venue cards + inline keyboards + Mini App deep-link hand-off; Malay/Malaysian place-name normalization; in-Worker fuzzy search over a curated gazetteer; OpenStreetMap Overpass candidate generation; ODbL/GPL-3.0 licensing
**Confidence:** HIGH for Telegram Bot API surface (official `core.telegram.org` type pages fetched this session), HIGH for the Malay abbreviation table (official JUPEM gazetteer tables + Nominatim `variants-ms.yaml` fetched this session), HIGH for ODbL text (fetched this session), MEDIUM for the deep-link parameter delivery mechanism (see §"Deep-link risk" — the mechanism is documented but the exact inline-button behaviour is contested), HIGH for the environment findings (probed this session)
## User Constraints (from CONTEXT.md)
### Locked Decisions
**Search interaction surface**
- **D-01:** Search entry is the **`/cari` command** in chat (with `/pergi_ke` reserved for Phase 5 semantics; bilingual alias scheme per Phase 2 D-05). Free-text-to-bot stays reserved; no inline mode; no in-webview search UI this phase (a shared search module keeps that door open).
- **D-02:** Shortlist presentation: the **best result ships immediately** as the venue card with its OPEN button; **2–4 alternates ride as inline buttons on the same message**. One tap to proceed, one tap to switch — no second message round.
- **D-03:** Search backend is a **pluggable `SearchSource` interface implemented in-Worker**: curated KL gazetteer queried in-Worker, $0 infrastructure. **VPS/Photon provisioning defers to Phase 5** (Valhalla's road graph needs the VPS anyway). Photon remains a documented escalation path *behind the interface*.
**Forkability + license (user-surfaced design law)**
- **D-04:** **GPL-3.0** is the project license. Forkability laws are binding from this phase: no hardcoded secrets (env only — already true), data as plain committed files, one-command deploy, one free-tier Cloudflare account per fork.
- **D-05:** The user's **Malaysia-only WASM geocoder** vision (tantivy/sqlite-trigram index, statically hosted, client-side webview search — a genuine privacy win) is captured as **SearchSource backend #2**, an OSS-track artifact *after* v1. The JSON↔SQLite converter script keeps its dataset format reachable from day one. **Backend #3 = Photon-on-VPS.** Neither ships in this phase.
**Gazetteer + Malay normalizer**
- **D-06:** Entries are **hand-curated seed + a repo-script Overpass one-shot** that proposes top-up candidates for maintainer approval into the curated file. The script is a repo tool, not a runtime dependency.
- **D-07:** Canonical format is **typed JSON in the repo** (V8-native parse, no WASM in the Worker bundle, diff-friendly PRs). Rationale: at ~200–500 rows, SQLite/sql.js-in-Worker is ~1.5MB + cold-start overhead of pure cost.
- **D-08:** **Aliases live natively inside each entry** (`"KLCC"` → Suria KLCC, KLCC Park; IJN, PWTC, Stesen/Station…) — locals' shorthand is the main corpus quality lever.
- **D-09:** The Malay normalizer is a **rule table + folding + fuzzy-residual** module: bidirectional abbreviation table (Jln↔Jalan, Tmn↔Taman, Kg↔Kampung, vowel-drop-aware Presint/Prsn forms), case/whitespace folding, glued-word split — applied to BOTH the query and entry aliases at index time. Fuzzy scoring (Damerau) is only the residual net for genuine typos on longer words. Per-rule test vectors; extends by table edit, not code change.
**Venue card + deep-link hand-off**
- **D-10:** The venue card is **Bot API `sendVenue`** (native pin/name/address semantics SRCH-03 names) with inline buttons: OPEN IN ARAHARAH (`web_app` button carrying `startapp`) + alternates.
- **D-11:** The deep link carries a **plain gazetteer entry id** (`?startapp=dest_`). The webview maps id → destination from its own copy of the same gazetteer module (shared module, build-time). No crypto: the payload's only threat is a user pranking themselves; stale-webview datasets re-map via the shared module version.
- **D-12:** ONBD-03 scope: **minimal PWA hooks** — `addToHomeScreen` prompt handling + a startapp-less boot landing in the search-ready map (existing Phase 3 staged boot covers it). Full PWA manifest/offline story is documented-deferred.
**Acceptance corpus + thresholds (the bake-off close)**
- **D-13:** A **pinned golden corpus** in the repo (~60–100 queries): golden landmarks (KLCC, Sunway Pyramid, DUKE), abbreviation forms, typo set (transpositions, missing keys), glued words, aliases, kampung-style names, plus **deliberate negatives** (out-of-KL: "Kuching", "Penang") that must get an honest "not in v1 area" answer, never a fabricated pin. Each positive pins its expected entry id. Reviewers extend by editing one file.
- **D-14:** The bake-off closes on: **top-3 contains the expected id for ≥90% of positives AND negatives are refused honestly**. Below threshold, escalation order is fixed: (1) alias/normalizer table edits → re-run; (2) SearchSource backend #2 (WASM geocoder); (3) Photon-on-VPS. The phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question.
**Privacy (carried + made specific)**
- **D-15:** Search queries are user payload: **category-only logging** (Phase 2 law) — no query text persisted. KV caching of search results, if any, is TTL-only with short TTL and keyed by normalized query (content is public place data, never user identity). Venue choices are not stored.
### OpenCode's Discretion
- Scoring function shape (weighted trigram/Jaro vs Damerau) within the rule-table architecture — corpus decides empirically
- Alternates ranking + tie-breaking details; "not in v1 area" copy (en/ms, clinical tone)
- Overpass query shape for the top-up script; gazetteer entry schema fields beyond (id, name, aliases, lat, lng, category)
- D1/KV TTL result-cache mechanics or none (perf is likely fine without at this scale)
### Deferred Ideas (OUT OF SCOPE)
- Malaysia-only WASM geocoder (tantivy/sqlite-trigram), statically hosted, client-side webview search — SearchSource backend #2, OSS track after v1
- Photon-on-VPS — SearchSource backend #3; VPS provisioning happens in Phase 5 for Valhalla regardless
- In-webview search UI — needs a UI-SPEC round; shared module already keeps it cheap
- Full PWA (manifest, offline shell) — documented-deferred beyond minimal ONBD-03 hooks
- Free-text bot search + inline mode — reserved interaction surfaces
- Log-derived corpus (real queries) — becomes possible only post-launch (Phase 7+)
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|------------------|
| **SRCH-01** | User can type a destination query (e.g. "KLCC") and receive a shortlist containing the right KL-metro landmark/POI | `/cari` command handler + `SearchSource` interface + curated gazetteer. §"SearchSource interface" defines the contract; §"Standard Stack" defines the JSON entry schema; §"Golden corpus" defines the acceptance mechanism that proves "KLCC" → Suria KLCC. §"Anti-patterns" warns against a bare string-equality match. |
| **SRCH-02** | User gets useful results despite abbreviations and typos (Jln↔Jalan, Tmn↔Taman, missing spaces, common Malay address shorthand) | §"Malay normalizer" — the full JUPEM-derived bidirectional abbreviation table (fetched verbatim from Nominatim `variants-ms.yaml`), the vowel-drop Presint/Prsn rule, glued-word splitting, and the Damerau-Levenshtein residual. §"Standard Stack" fixes the scoring shape. §"Pitfall 3" covers the "abbreviation only applies to generic names, not proper names" rule (MS 2039:9). |
| **SRCH-03** | For a chosen destination, a native venue card is sent in chat, confirming where we're going before navigation opens | §"sendVenue" — the exact Bot API parameter set (all four required fields), the `reply_markup` inline-keyboard slot, and the constraint that `sendVenue` accepts no `text` field (so the "one line confirming the destination" must be a *separate* message or the venue's `title`/`address` themselves). §"Inline keyboards" for the alternates + OPEN button. |
| **ONBD-02** | User can open the Mini App from a search result with their destination preloaded (no re-entry in the webview) | §"Deep-link risk" — **the single highest-risk item in this phase**. Documents that `start_param` is officially scoped to direct links / `startattach`, that the `web_app` inline button carries a raw URL, and gives the planner three concrete carriers ranked by risk, plus the write-both-readers fallback. `WebAppInfo.url` (official type page) confirms the button URL is an arbitrary HTTPS URL. |
| **ONBD-03** | User can add araharah to their phone home screen and launch search/navigation from there (addToHomeScreen + startapp deep-link) | §"ONBD-03 feasibility" — `addToHomeScreen()` / `checkHomeScreenStatus()` are Bot API 8.0+ only, the iOS limitation (`checkHomeScreenStatus` always returns `"unknown"` on iOS — confirmed upstream issue #1729, closed as expected platform behaviour), the `TelegramWebAppLike` extension shape, and the fact that the home-screen shortcut opens the Main Mini App (so `startapp` is available there even though it is not on an inline-button launch). |
---
## Summary
Phase 4 is the phase where the project stops being a shell and starts being useful — and it carries **three materially different risks** that the plan must not conflate.
**Finding 1 — the deep-link mechanism in D-11/D-12 is the riskiest thing in the phase, and the documentation does not say what D-11 assumes.** Telegram's official docs say the `start_param` field in `WebAppInitData` is *"the value of the `startattach` parameter, passed via link. Only returned for Mini Apps when launched from the attachment menu via link"*, and separately that `startapp` is populated when *"a non-empty `startapp` parameter is included in the link"* — where "the link" means a **direct link** (`https://t.me/bot?startapp=…`) or a **main Mini App link**, *not* a `web_app` inline keyboard button. But the `web_app` button's `WebAppInfo.url` is documented as *"An HTTPS URL of a Web App to be opened with additional data"* — i.e. an arbitrary URL, and community reports confirm query params on that URL survive into the webview. **D-11's "plain gazetteer entry id" payload is fine; the *carrier* is not settled.** The planner must pick a carrier explicitly and the plan must carry a fallback. §"Deep-link risk" lays out three ranked carriers with their evidence.
**Finding 2 — the Malay abbreviation table does not need to be invented; it exists, it is official, and it is already machine-readable.** JUPEM (the Malaysian mapping authority) publishes a normative abbreviation table in every state's gazetteer, and the OSM Nominatim project transcribed it into `settings/icu-rules/variants-ms.yaml` (merged April 2026, PR #4056). Both were fetched verbatim this session and are reproduced in §"Malay normalizer". That table already covers `Jln→Jalan`, `Tmn→Taman`, `Kg→Kampung`, `Bkt→Bukit`, `Sg→Sungai`, `Bdr→Bandar`, `Psrn→Persiaran`, `Lbh→Lebuh` and more, with the Malaysia-specific and shared variants split exactly the way the normalizer needs them. **The planner should vendor this table (with its JUPEM attribution) rather than author a smaller one.** One important extra rule the table does *not* encode comes from MS 2039:9 — an abbreviation may apply to a *generic* geographic term but **not** to a name that is part of a proper name ("Jln Taman Desa" ✓, "Jln Tmn Desa" ✗). That rule is a corpus-level invariant the normalizer needs.
**Finding 3 — the Overpass dependency is real, and this machine reaches the primary endpoint **only with a `User-Agent` header**.** `overpass-api.de` returns **HTTP 406** on every request that lacks an identifying `User-Agent` — including a bare `GET /` — and **HTTP 200** with one. (Verification pass, same host/same paths, varying only the header: `GET /api/status` no-UA → 406 / with-UA → 200; `GET /` no-UA → 406 / with-UA → 200; `POST /api/interpreter` no-UA → 406 / with-UA → **200 with 19 KB of real KL data**.) The other public mirrors are unusable from here — `overpass.kumi.systems` and `overpass.private.coffee` fail to connect (HTTP 000), and `overpass.osm.ch` is a Switzerland-only regional instance that returns **400**. Nominatim (`nominatim.openstreetmap.org`) and Geofabrik (`download.geofabrik.de/asia/malaysia-singapore-brunei-latest.osm.pbf`) **are** reachable and return correct KL data. This does not block the phase — D-06 says the Overpass script is a *repo tool for maintainers*, not a runtime dependency — but the planner must know that (a) the script's `User-Agent` is **load-bearing, not cosmetic**, (b) capacity **502** responses are **intermittent and normal** (a 200 and a 502 were observed minutes apart on the same host), and (c) **the top-up script's automated test must target a committed fixture, not the live service.**
**Finding 4 (bonus) — the licensing is compatible, but only if the gazetteer is treated as data, not code.** ODbL §2.3(a) explicitly excludes computer programs from the license, and OSMF's guidelines confirm OSM *data* is ODbL while *code* is separately licensed. The precedent is live: `cbeier-studio/Xolmis` is GPL-3.0 and ships ODbL data in the same repo with an explicit dual-license note. **But ODbL §4.4 does impose share-alike on a Derivative Database** — so the curated gazetteer file must carry an ODbL notice, and if the normalizer's *index* (a derived, restructured form of the gazetteer) is shipped in the repo, it should carry the same notice. This is a two-line obligation with a real legal consequence and no cost. The repo currently has **no LICENSE file at all** — D-04 says GPL-3.0 is the project license, so adding `LICENSE` (GPL-3.0) plus a `data/` notice is a concrete deliverable this phase, not a Phase 7 chore.
**Primary recommendation:** Ship Phase 4 as three separable layers — (1) a **pure, dependency-free shared search module** (`gazetteer.json` + normalizer + scorer, importable by both `bot/` and `web/`), (2) a **bot command surface** (`/cari` → `sendVenue` + inline alternates), and (3) a **deep-link carrier with a documented fallback**. Make the deep-link carrier the *first* thing built and the *last* thing gated, because it is the only requirement in the phase whose success depends on undocumented Telegram client behaviour. Then close the experiment with the corpus runner: a single command that prints top-3 hit rate on positives and refusal rate on negatives, and a written decision in the repo saying which lever was pulled.
**The one artifact to lift directly:** §"Malay Normalizer — the vendorable rule table" reproduces the **complete JUPEM-derived abbreviation table verbatim** (the machine-readable transcription from Nominatim's `variants-ms.yaml`), plus the **MS 2039:2006** display rule, the **Presint/Prsn** vowel-drop case with real usage evidence, and the three official tables that must be merged. The planner should vendor that table into the shared module with its JUPEM attribution rather than author a smaller one — the difference between the 8-entry original and the current 65+ entry table is the difference between "Jln" working and "Jln Ampang" working.
---
## Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| **Malay normalization + scoring (pure functions)** | *(shared library — no tier)* | — | D-09/D-11 require the *same* module in bot and web. A workspace package (`@araharah/search`) or a shared `src/` dir imported by both. No runtime, no IO, no network. |
| **Gazetteer data (curated JSON)** | *(data file — no tier)* | — | D-07: typed JSON in the repo, committed, diff-friendly. Read by both tiers. |
| **`/cari` command parsing** | **API / Backend** | — | Telegram command dispatch lives in `bot/src/bot/commands/`. The Worker is the only tier that receives Telegram updates. |
| **Query → shortlist (SearchSource)** | **API / Backend** | Browser / Client | D-03: in-Worker implementation. The webview copy is build-time only (D-11) — it maps an id to coordinates, it does not re-run the search. |
| **Venue card + inline keyboard rendering** | **API / Backend** | — | `sendVenue` is a Bot API call; only the Worker holds `BOT_TOKEN`. |
| **Alternate selection (callback query)** | **API / Backend** | — | `callback_query` updates arrive at the Worker webhook. |
| **Deep-link payload → destination lookup** | **Browser / Client** | API / Backend | D-11: the webview maps `id → destination` from *its own copy* of the shared module. The Worker never re-derives it. |
| **Mini App boot with preloaded destination** | **Browser / Client** | — | Phase 3's staged boot (validating → acquiring → live) is the entry; the destination becomes a pre-seeded target the Phase 5 route layer consumes. |
| **Home-screen prompt (ONBD-03)** | **Browser / Client** | — | `addToHomeScreen()` is a `Telegram.WebApp` method; only the webview can call it. |
| **Golden corpus + acceptance runner** | *(repo tooling — no tier)* | — | D-13/D-14: a repo script that imports the shared module and the corpus file. Runs under `node`, no network. |
| **Overpass top-up script** | *(repo tooling — no tier)* | — | D-06: maintainer-time tool, network-dependent, never imported by the Worker. |
---
## Standard Stack
### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| **grammy** | `1.46.0` (installed; `npm view grammy version` → `1.46.0` — current) | Bot API client: `sendVenue`, `InlineKeyboard`, `callbackQuery` handlers | Already the project's bot framework (Phase 2). `InlineKeyboard` has a first-class `.webApp(text, url)` builder, so D-10's `web_app` button needs no raw object literal. |
| **Node 24.18.0 built-ins only** (`JSON.parse`, `Intl`, `String.prototype.normalize`) | n/a | Gazetteer loading, normalization, scoring | D-07 mandates V8-native parse, no WASM. D-09's rule table is plain data. Damerau-Levenshtein on a ~500-row × ~5-alias corpus is a ~2500-pair comparison — a hand-rolled 20-line OSA loop is *faster than any dependency* at this size and adds zero bundle weight. **Recommendation: hand-roll the OSA distance in the shared module** (see §"Don't Hand-Roll" for why this one is an exception). |
| **@crob/damerau-levenshtein** | `2.0.0` (npm, published ~1 month ago) | *Alternative* to hand-rolling OSA distance | TypeScript-first, ESM+CJS+UMD, 1.2 kB gzipped, configurable per-op costs (nice: a cheap cost for adjacent-keyboard substitutions). **Requires Node 20.19+** (we have 24.18.0). Use if the plan wants a battle-tested implementation; skip if the plan prefers zero runtime deps. `[VERIFIED: npm registry]` |
| **`damerau-levenshtein`** | `1.0.8` (npm) | *Alternative* | BSD-2, 52M weekly downloads, but last published 2022, CommonJS-ish `import * as levenshtein` shape, and returns `{steps, relative, similarity}` rather than a bare distance. Older and less pleasant than `@crob/damerau-levenshtein`. `[VERIFIED: npm registry]` |
| **`fastest-levenshtein`** | `1.0.16` (npm) | *Alternative* — Levenshtein only, no transposition | Fast, but **no transposition support** — and transpositions ("KLCC" → "KCLC", "Sunway" → "Sunwya") are an explicit D-13 corpus case. Rejected. `[VERIFIED: npm registry]` |
### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| **Playwright** | `1.50.0` (already a `web` devDependency) | Capture evidence for ONBD-02 (webview booted with a preloaded destination) | Reuse the existing `web/scripts/capture.mjs` harness pattern — add a `?dest=` profile. No new dependency. |
| **Nominatim** (network service, not a lib) | live API | *Fallback* candidate source + coordinate verification for curated entries | Verified reachable this session (`nominatim.openstreetmap.org/status` → 200; `?q=KLCC` returned correct KLCC nodes with `licence: "Data © OpenStreetMap contributors, ODbL 1.0"`). **Use it to verify curated coordinates** even if Overpass is unreachable. |
| **Geofabrik** (network service) | `asia/malaysia-singapore-brunei-latest.osm.pbf` | *Fallback* bulk OSM extract for the top-up script | Verified reachable (HTTP 206 on range request). Note the extract name is **malaysia-singapore-brunei**, not "malaysia" — a 404 trap. |
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Curated JSON gazetteer (D-07) | SQLite / `sql.js` in the Worker | D-07 already rejects this: ~1.5 MB bundle + cold-start for a ~200–500-row dataset. **Non-negotiable.** |
| Hand-rolled OSA distance | `@crob/damerau-levenshtein` (2.0.0) | +1.2 kB gzipped, well-tested, configurable costs; −~20 lines of hand-written code to maintain and test. Both viable at this corpus size. |
| Trigram/prefix index | Plain linear scan over ~500 entries × ~5 aliases | At ~2500 candidate pairs, a linear scan with early-exit on exact normalized match is **microseconds** — an index is premature. The trigram index becomes backend #2's job (D-05). |
| Overpass QL for top-up candidates | Nominatim search, or a Geofabrik `.pbf` + `osmium` | Overpass is the right tool *in general*; in *this environment* it is blocked (see §"Environment Availability"). The script should support both a `--source=overpass|nominatim` flag, or read a locally-downloaded extract. |
| `sendVenue` (D-10) | `sendMessage` + `sendLocation` | **Rejected.** SRCH-03 explicitly names the native venue card; `sendVenue` gives the pin/name/address semantics Telegram users read as "a place". Also: `sendVenue` accepts **no `text` field** — the confirming line must be a separate message or ride in the venue's own `title`/`address`. |
| `callback_data` alternates | Editing the message on every tap | D-02 wants alternates *on the same message* as the card. `editMessageText` can replace inline keyboards, but a venue message carries a `venue` payload that `editMessageText` cannot preserve — the card would be destroyed. **Use `callback_data` + `answerCallbackQuery` + a new `sendVenue`** (or `editMessageReplyMarkup`, which is documented as allowed on a message that already has an inline keyboard). |
**Installation:**
```bash
# Only if the plan chooses the library route for the distance function:
npm install @crob/damerau-levenshtein -w @araharah/search
# If the shared module is a workspace package, add "shared" to root package.json "workspaces".
```
**Version verification:** `npm view grammy version` → `1.46.0` (matches installed). `npm view @crob/damerau-levenshtein version` → `2.0.0`. `npm view fastest-levenshtein version` → `1.0.16`. `npm view damerau-levenshtein version` → `1.0.8`. All verified against the registry this session.
---
## Architecture Patterns
### System Architecture Diagram
```
┌────────────────────────────────────────────────────────┐
│ Telegram client │
│ user types: /cari KLCC │
└────────────────────────┬───────────────────────────────┘
│ (Bot API update)
▼
┌────────────────────────────────────────────────────────────────────────────────┐
│ Cloudflare Worker (bot/src/webhook.ts → POST /telegram/webhook) │
│ │
│ ┌──────────────────┐ ctx.args = "KLCC" │
│ │ /cari handler │ (bot/src/bot/commands/cari.ts) │
│ └────────┬─────────┘ │
│ │ search("KLCC") │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ @araharah/search (SHARED — pure functions, no IO, no network) │ │
│ │ │ │
│ │ ┌───────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │
│ │ │ normalizer.ts │→ │ scorer.ts │→ │ gazetteer.json (typed) │ │ │
│ │ │ rule table + │ │ exact→alias │ │ {id,name,aliases, │ │ │
│ │ │ folding + │ │ match→token │ │ lat,lng,category,addr} │ │ │
│ │ │ glued split │ │ set→Damerau │ │ │ │ │
│ │ └───────────────┘ └──────────────┘ └──────────────────────────┘ │ │
│ │ ▲ normalize both sides at index time │ │
│ └────────┼──────────────────────────────────────────────────────────────┘ │
│ │ SearchSource interface (D-03) — backend #2 / #3 plug in here │
│ ▼ │
│ shortlist: [best, ...alternates] (or EMPTY → honest refusal) │
│ │
│ ┌─────────────────────────┐ │
│ │ sendVenue(best) │ ← SRCH-03: native venue card │
│ │ + inline_keyboard: │ │
│ │ [ OPEN IN ARAHARAH ] │ ← web_app button, URL carries the dest id │
│ │ [ alt 2 ] [ alt 3 ] │ ← callback_data = "alt:" │
│ └──────────┬──────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────┐ │
│ │ callback_query handler │ ← user taps an alternate │
│ │ answerCallbackQuery() │ │
│ │ + sendVenue(newBest) │ │
│ └─────────────────────────┘ │
└────────────────────────────────┬───────────────────────────────────────────────┘
│
┌────────────────────────┴──────────────────────────┐
│ (A) web_app inline button ──► Mini App webview │
│ (B) url-button deep link ──► t.me?startapp=… │
│ (C) in-webview URL query ──► ?dest= │
▼ ▼
┌──────────────────────────────────────────────────────────┐
│ Mini App (web/src/main.ts) — Phase 3 staged boot │
│ validating session → acquiring location → live │
│ + destination preloaded (id → entry from shared module)│
│ + ONBD-03: addToHomeScreen() prompt handling │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Repo tooling (never imported by the Worker) │
│ bot/scripts/…-gazetteer-topup.mjs → Overpass/Nominatim│
│ …-corpus.mjs (golden corpus runner) → node │
│ …-json-to-sqlite.mjs (D-05 backend #2 converter stub) │
└──────────────────────────────────────────────────────────┘
```
### Recommended Project Structure
The key structural decision: **the search module must be importable by both `bot/` and `web/`**. Two viable shapes:
```
Option A — a new workspace package (cleanest):
shared/ @araharah/search — pure TS, zero deps
├── package.json (type: module, exports ./src/index.ts)
├── src/
│ ├── index.ts public API: search(), normalize()
│ ├── types.ts GazetteerEntry, SearchSource, SearchResult
│ ├── normalize.ts rule table + folding + glued-word split
│ ├── rules.ts the abbreviation table (JUPEM-derived)
│ ├── score.ts exact → alias → token-set → Damerau
│ └── distance.ts Damerau-Levenshtein OSA (hand-rolled or lib)
├── data/
│ ├── kl-gazetteer.json the curated entries (D-07)
│ └── NOTICE.md ODbL attribution for the OSM-derived data
└── test/ corpus runner + rule test vectors
Option B — a shared dir at repo root imported by both workspaces:
packages/search/ (same shape)
Both workspaces then do:
import { search } from "@araharah/search";
import gazetteer from "@araharah/search/data/kl-gazetteer.json" with { type: "json" };
```
```
bot/src/
├── bot/commands/cari.ts NEW — /cari + /cari alias
├── bot/handlers/search-alternate.ts NEW — callback_query handler for alternates
├── search/
│ └── source.ts NEW — the SearchSource interface (D-03)
│ + GazetteerSource (the in-Worker impl)
└── i18n/strings.ts EXTENDED — search copy (en/ms)
web/src/
├── main.ts EXTENDED — read dest id at boot, seed destination
├── telegram/webapp.ts EXTENDED — addToHomeScreen/checkHomeScreenStatus
├── search/ NEW — the webview's copy of id→entry lookup
└── i18n/strings.ts EXTENDED — any search-ready copy
repo root:
├── LICENSE NEW — GPL-3.0 (D-04)
├── shared/ or packages/search/ NEW — the shared module (above)
└── bot/scripts/
├── gazetteer-topup.mjs NEW — Overpass/Nominatim candidate proposer (D-06)
├── gazetteer-corpus.mjs NEW — golden corpus runner (D-13/D-14)
└── gazetteer-to-sqlite.mjs NEW — JSON→SQLite converter (D-05, stub shape)
```
**Import assertion:** the shared module must be importable by a **Node** script (corpus runner), a **Cloudflare Worker** (bundled by wrangler/esbuild), and a **browser** (Vite). All three handle a `.ts` file if the consuming toolchain compiles it — the corpus runner is `node`, which **cannot** import `.ts` directly in Node 24 without a loader flag. **The corpus runner therefore must be `.mjs` and import a `.js`/`.mjs` build of the module, or the shared module must be authored in plain `.js` with JSDoc types + a hand-written `.d.ts`.** This is a concrete, easy-to-miss build constraint. Node 24 does have type-stripping (`--experimental-strip-types`), but relying on it in a repo script is fragile; the safer shape is **author the shared module in plain ESM JavaScript with JSDoc typedefs**, matching the repo's existing `.ts`-source-imported-from-`.mjs`-test convention (the existing tests already do `await import("../src/i18n/strings.ts")` — verify whether that works under the repo's current Node version before repeating it).
> **Planner note:** `bot/test/commands.test.mjs` already does `await import("../src/i18n/strings.ts")`. Check `node --version` (24.18.0 here) — Node 24 has **type stripping enabled by default** for `.ts` files, so `import("../src/...ts")` works. If so, the shared module can stay `.ts` and the corpus runner can import it directly. **Verify this in the plan's first task**, because the whole test strategy depends on it.
### Pattern 1: The `SearchSource` interface (D-03)
**What:** a narrow contract that the in-Worker gazetteer implements today, and that backend #2 (WASM geocoder) and backend #3 (Photon-on-VPS) implement later.
**When to use:** always — this is the seam. Phases 5/6 plug origin/destination resolution into the same interface.
**Example:**
```typescript
// shared/src/types.ts
export interface GazetteerEntry {
id: string; // stable, slug-like: "suria-klcc", "sunway-pyramid"
name: string; // canonical display name
aliases: string[]; // locals' shorthand — D-08's main quality lever
lat: number;
lng: number;
category: string; // "mall" | "office-tower" | "park" | "station" | …
address?: string; // human address for sendVenue's `address` field
}
export interface SearchQuery {
text: string; // raw user query
limit?: number; // default 5 (1 best + 2–4 alternates, D-02)
}
export interface SearchHit {
entry: GazetteerEntry;
score: number; // 0..1, higher is better
matchedAlias: string; // which alias matched — useful for the corpus report
}
export interface SearchSource {
readonly name: string; // "gazetteer" | "wasm-geocoder" | "photon"
search(q: SearchQuery): Promise; // EMPTY array = honest refusal
}
```
**The refusal contract is the important part.** An empty result is a *legitimate, load-bearing* return value (D-13's negatives: "Kuching", "Penang" must be refused, never fabricated). The interface must make "no match" easy to express and the bot must translate it into the "not in v1 area" copy — never into a best-guess pin.
### Pattern 2: Rule-table normalization (D-09)
**What:** a data-driven, bidirectional abbreviation table + folding + glued-word splitting, applied to **both** the query and every entry alias at index time.
**When to use:** always, before any scoring. Normalization must be *symmetric* — "Jln Ampang" and "Jalan Ampang" must produce the **same** normalized string, which requires expanding *both* directions to a canonical form.
**Example (the canonical form is the long form):**
```typescript
// shared/src/rules.ts — JUPEM-derived. See §"Malay normalizer" for the full table.
export const ABBREVIATIONS: Record = {
jln: ["jalan"],
tmn: ["taman"],
kg: ["kampung", "kampong"],
kpg: ["kampung", "kampong"],
bkt: ["bukit"],
bt: ["batu"],
sg: ["sungai"],
bdr: ["bandar"],
psrn: ["persiaran"],
lbh: ["lebuh"],
lrg: ["lorong"],
lor: ["lorong"],
spg: ["simpang"],
blk: ["blok"],
h: ["hulu"],
p: ["pulau"],
tg: ["tanjung", "tanjong"],
tlk: ["teluk", "telok"],
g: ["gunung", "gunong"],
gn: ["gunung", "gunong"],
k: ["kuala"],
hr: ["hilir"],
jr: ["jeram"],
lmbh: ["lembah"],
pdg: ["padang"],
pt: ["parit"],
pmtg: ["permatang"],
plbh: ["pelabuhan"],
kep: ["kepulauan"],
ldg: ["ladang"],
lpg: ["lapangan"],
gsg: ["gosong"],
bgn: ["bangunan"],
alr: ["alur"],
bak: ["bakau"],
bdh: ["berhad"],
hj: ["haji"],
hjh: ["hajjah"],
abd: ["abdul"],
kl: ["kuala lumpur"],
pj: ["petaling jaya"],
jb: ["johor bahru"],
kb: ["kota bahru"],
kk: ["kota kinabalu"],
};
```
### Pattern 3: Layered scoring with an exact-match floor
**What:** score candidates in **tiers**, each tier's score range strictly above the previous, so an exact alias hit can never be outranked by a fuzzy hit.
**When to use:** always. This is the mechanism that makes D-13's "top-3 contains the expected id" measurable and stable.
**Example:**
```typescript
// shared/src/score.ts — tier ranges are disjoint; see §"Common Pitfalls" #4.
export const TIER = {
EXACT: 1.00, // normalized query === normalized alias or name
ALIAS_EXACT: 0.90, // normalized query === normalized alias
PREFIX: 0.75, // one is a prefix of the other, ≥3 chars
TOKEN_SET: 0.65, // all query tokens present in the candidate's token set
FUZZY: 0.00, // 0..0.60 — Damerau similarity, scaled
} as const;
export function scoreOne(queryNorm: string, aliasNorm: string): number {
if (queryNorm === aliasNorm) return TIER.EXACT;
if (queryNorm.length < 3 || aliasNorm.length < 3) return 0;
const d = damerau(queryNorm, aliasNorm);
const maxLen = Math.max(queryNorm.length, aliasNorm.length);
const sim = 1 - d / maxLen;
if (queryNorm.startsWith(aliasNorm) || aliasNorm.startsWith(queryNorm)) {
return TIER.PREFIX;
}
if (tokenSubset(queryNorm, aliasNorm)) return TIER.TOKEN_SET;
return sim * TIER.FUZZY; // 0 .. 0.60
}
```
**Why the tiering matters:** Jaro-Winkler's prefix bonus and Damerau's normalized distance can both rank a *fuzzy* match above an *exact* one on short strings. Disjoint score bands make the ordering a property of the *tier*, not of the arithmetic — so the corpus results are stable when the fuzzy function is tuned (D-09's "extends by table edit, not code change" only works if the tiers don't move).
### Anti-Patterns to Avoid
- **Bare `===` matching against names.** Fails every abbreviation and typo case; D-09 exists to prevent this.
- **A single flattened `Map` of all aliases.** It cannot rank alternates, cannot express "the canonical name outranks an alias", and loses the `matchedAlias` provenance the corpus report needs.
- **Normalizing only the query.** Normalizing only one side means "Jln Ampang" and "Jalan Ampang" land in different buckets. D-09 explicitly requires both sides.
- **Fuzzy-scoring the whole string when the error is in one word.** "Tmn Bangsar" vs "Taman Bangsar" is a *whole-string* one-edit match, but "Jln Tun Razak" vs "Jalan Ampang" is two different roads — token-set matching before fuzzy prevents cross-street confusion.
- **Reading the gazetteer JSON from the filesystem at Worker runtime.** Cloudflare Workers have no `fs`; the JSON must be a **build-time import** (`import data from "...json" with { type: "json" }`) so wrangler/esbuild inlines it into the bundle. The corpus runner (Node) reads it with `fs` — the shared module must expose the *entries*, not do the reading, so both consumers get their data the way their runtime allows.
- **An Overpass query with no bbox.** A global Overpass query will time out and is a well-known way to get rate-limited. Always bound to the KL bbox.
- **An `editMessageText` on the venue message.** A venue message carries a `venue` payload; `editMessageText` replaces it with text and the card is destroyed. Use `editMessageReplyMarkup` (documented as allowed when the message already has an inline keyboard) or send a new `sendVenue`.
- **Persisting the raw query.** D-15: category-only logging. A query is user payload. This is the single easiest way for this phase to break the project's privacy promise — and there is already a grep gate in `web/test/no-storage.test.mjs` for the web side.
---
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| **Bot API transport, keyboards, callback handling** | Raw `fetch` to `api.telegram.org` | **grammy 1.46.0** | Already installed. `InlineKeyboard.webApp()` / `.text()` builders, `bot.callbackQuery(regex, …)` trigger matching, `ctx.answerCallbackQuery()` — all first-class. Hand-rolling risks the 30-second callback-answer deadline. |
| **The Malay abbreviation table** | An invented list from memory | **The JUPEM table** (via Nominatim `variants-ms.yaml`), vendored with attribution | The official table is normative, already transcribed, and machine-readable. Inventing one loses `Psrn`, `Lbh.Raya`, `Bdr`, `Kondo`, `Blk`, `Hosp`, `Disp` and the older `Kampong`/`Tanjong`/`Telok`/`Gunong` spellings — all real KL usage. §"Malay normalizer" reproduces it. |
| **OSM data retrieval at scale** | Parsing a `.pbf` in the Worker | **Overpass API** (repo script) or **Geofabrik extract** (repo script) | Neither belongs in the Worker. D-06 says the script is a maintainer tool. |
| **The Mini App ↔ Telegram bridge** | `telegram-web-app.js` vendored into the repo | **The official script** loaded by the existing `web/index.html` | Phase 3 already ships this arrangement. |
| **Home-screen install prompt** | A custom PWA install flow | **`Telegram.WebApp.addToHomeScreen()`** | Telegram owns the shortcut; the browser's `beforeinstallprompt` does not exist inside the Telegram webview and would create a *second* icon that Telegram doesn't know about. |
| **A search index / trigram structure** | A hand-built inverted index in the Worker | **Linear scan over ~2500 (entry × alias) pairs** | At 500 entries × 5 aliases, a linear scan with an exact-match fast path is microseconds. A trigram index is backend #2's job (D-05) and would be pure cost in the Worker bundle (D-07). |
| **Session / HMAC validation** | Re-deriving initData validation | **The existing `bot/src/security/initdata.ts`** | Phase 2's HMAC path is tested and hardened (T-02-01). The destination id does not need its own crypto (D-11). |
| **Routing / Photon / Valhalla** | Anything in this phase | **Deferred to Phase 5** | D-03 explicitly defers VPS provisioning. Do not research or build it here. |
**Key insight — the one place hand-rolling IS correct:** the Damerau-Levenshtein distance. At this corpus size the comparison count is tiny, the algorithm is 20 lines of standard OSA dynamic programming, and a dependency would add a package to a Worker bundle that D-07 deliberately wants minimal. Hand-rolling it *inside the shared module* keeps the whole search stack dependency-free and V8-native, which is exactly D-07's rationale. **Exception:** if the plan prefers a tested implementation, `@crob/damerau-levenshtein@2.0.0` is 1.2 kB gzipped, TypeScript-first, and Node 20.19+ compatible — a safe swap.
---
## Runtime State Inventory
> **Not applicable.** Phase 4 is additive: no rename, no rebrand, no string replacement, no migration. The gazetteer is a **new** committed file; there is no pre-existing state to inventory. *(Per the format spec, this section is included only for rename/refactor/migration phases and is omitted otherwise.)*
**However**, there is one adjacent runtime-state question the planner should answer explicitly:
| Category | Question | Answer for Phase 4 |
|----------|----------|--------------------|
| **Stored data** | Does any existing datastore hold a place-name or search string that this phase changes? | **No.** The only KV namespace is `chat_sessions` (sessions + location pins, TTL-only). The gazetteer is a new repo file, not a KV namespace. |
| **Live service config** | Does any external service config reference place names or search endpoints? | **No.** The only live config is `bot/wrangler.jsonc` (KV namespace, assets binding) and `bot/.dev.vars` (secrets). **One new config item IS likely:** the Mini App's public URL, needed to build the `web_app` button's `WebAppInfo.url` and the `t.me?startapp=` deep link. Today the Worker is on `..workers.dev` (`workers_dev: true`) and `WEBHOOK_URL` holds the origin. **The plan must decide how the bot learns the Mini App URL** — a new env var (e.g. `MINI_APP_URL`) read from `bot/.dev.vars` + wrangler secret, or derived from `WEBHOOK_URL`. This is a real Phase 4 deliverable and it is **not** in the D-list, so it falls in OpenCode's Discretion. |
| **OS-registered state** | Any OS-level registration embedding the changed strings? | **No.** No rename. The PWA home-screen shortcut is created by the *user* on their device at runtime; nothing is registered on the developer's machine. |
| **Secrets / env vars** | Any secret key named after a place name or search? | **No.** `BOT_TOKEN`, `WEBHOOK_SECRET`, `WEBHOOK_URL` are unaffected. If a `MINI_APP_URL` is added, it is **not a secret** — it is a public origin. It may live in `wrangler.jsonc` `vars` rather than as a secret. Flag for the planner. |
| **Build artifacts** | Stale artifacts that a source rename would leave behind? | **No rename.** But note: `web/dist` and `web/dist-harness` are build outputs; if the shared module is added, the **Vite and wrangler builds must both pick it up** and the `web/test/build-artifact.test.mjs` gate may need a new assertion (e.g. the gazetteer JSON *must* be in `web/dist` for the webview copy to work). Check that gate's scope. |
---
## Deep-link risk — choosing the carrier for the destination payload (ONBD-02)
**This is the single highest-risk decision in Phase 4.** D-11 says the deep link "carries a plain gazetteer entry id (`?startapp=dest_`)". The *payload* is settled. The *carrier* is not — and D-11's wording mixes two distinct Telegram launch surfaces.
### What the official docs actually say
Three separate statements, from the same page, that do not obviously agree:
| Source | Statement |
|--------|-----------|
| `core.telegram.org/bots/webapps` → *Launching the main Mini App* | *"A bot's main Mini App can also be opened in the current chat by direct link in the format `https://t.me/botusername?startapp`. If a non-empty `startapp` parameter is included in the link, it will be passed to the Mini App in the `start_param` field and in the GET parameter `tgWebAppStartParam`."* |
| `core.telegram.org/bots/webapps` → *Direct Link Mini Apps* | Same mechanism for `https://t.me/botusername/appname?startapp=command`. |
| `core.telegram.org/bots/webapps` → `WebAppInitData.start_param` | *"The value of the `startattach` parameter, passed via link. **Only returned for Mini Apps when launched from the attachment menu via link.**"* — note: this row says `startattach`, not `startapp`, and scopes it to the attachment menu. |
| `core.telegram.org/bots/webapps` → *Inline Button Mini Apps* | *"the Mini App will open with the URL specified in the button"* — **no mention of `start_param` at all.** |
| `core.telegram.org/bots/api/types/webappinfo.md` | *"An HTTPS URL of a Web App to be opened with **additional data**"* — the URL is arbitrary. |
**The gap:** the launch-parameter docs (`docs.telegram-mini-apps.com/platform/start-parameter`) list exactly two carriers for a start parameter — `?startattach=ABC` on a bot link, and `?startapp=ABC` on a **Direct Link**. A `web_app` **inline keyboard button** is neither. Its `WebAppInfo.url` is simply "the URL specified in the button", and Telegram passes it to `messages.requestWebView` verbatim (MTProto schema: `keyboardButtonWebView` → *"passing `url` to messages.requestWebView.url`"*).
**Consequence:** if the plan builds the OPEN button as `web_app: { url: "/?startapp=dest_klcc" }` and the webview reads `Telegram.WebApp.initDataUnsafe.start_param`, **`start_param` will most likely be `undefined`** — the button opens the Mini App with no destination. ONBD-02 fails silently.
### The three carriers, ranked
| Carrier | Mechanism | Evidence | Risk | UX cost |
|---------|-----------|----------|------|----------|
| **A — `web_app` button with a URL query param** | Button URL = `https://app.example.com/?dest=dest_klcc`; the webview reads `location.search` | `WebAppInfo.url` is documented as an arbitrary HTTPS URL; MTProto passes it verbatim; community reports confirm query params survive ("passing query params to telegram web app just works"). **Not** in the official Telegram docs. | **MEDIUM** — the behaviour is real but undocumented, so it is not a contract Telegram will keep. | None. One tap, in-app, no interstitial. |
| **B — `url` button to `https://t.me/?startapp=dest_`** | The officially-documented `startapp` mechanism | Explicitly documented for both main-Mini-App and direct links. `start_param` + `tgWebAppStartParam`. | **LOW** for the mechanism; **MEDIUM** for the `initData.start_param` wording (§"A2"). | A Telegram interstitial: the tap leaves the chat, opens the app. Also requires @BotFather configuration (Main Mini App, or a Direct Mini App with a `short_name`). And a `url` button is **not** a `web_app` button — D-10's wording changes. |
| **C — both, purpose-split** | A for the inline button (ONBD-02); B for the home-screen shortcut (ONBD-03) | The home-screen shortcut opens the **Main Mini App** link — which is exactly where `startapp` is officially supported. | **LOW** overall if the payload reader accepts both sources. | Carries A's cost on the button; B's cost only on the home-screen path (where an interstitial is invisible — the user tapped an icon). |
### Recommendation
**Ship Carrier C, with A as the inline-button mechanism and B as the home-screen mechanism.** Concretely:
1. The **payload reader** in the webview accepts **both** `location.search`'s `dest` and `initData`'s `start_param`, with the same validation (`/^[A-Za-z0-9_-]{1,64}$/`). One reader, two carriers — the phase cannot fail because Telegram changed one surface.
2. The **button builder** in the bot is a single function, `openButtonFor(entry)`, so swapping carriers is a one-line change if the device test fails.
3. The **device validation** at the exit gate must explicitly test: (a) tap OPEN from the chat → destination preloaded; (b) launch from the home-screen icon → app opens search-ready (ONBD-03's requirement is "lands in the app ready to search", so a preloaded destination is *not* required there — but the icon must open the app at all).
### What the mock harness must gain
`web/src/mock/telegram-mock.ts` currently emits `initData` as `user=…&auth_date=1&query_id=h&hash=harness` (`buildMockInitData`, lines 136-140) — **no `start_param` field, and no way to simulate a URL query param** (the mock is installed into `window`, not a URL). The Phase 4 plan must extend the harness with:
- a `startParam` option in `MockOptions` that appends `start_param=…` to the mock initData (for Carrier B);
- a `?dest=` runner-page profile that sets `window.location.search` before the app boots (for Carrier A).
Without these, the destination-preload path has **no automated test** — it would be validated only by hand on a device.
### The build-time constraint (easy to miss)
The `start_param` charset is restricted to **`[A-Za-z0-9_-]`**, max **512 characters** — validated with `/^[\w-]{0,512}$/`. A gazetteer id like `suria-klcc` already complies. But **any id that would need a base64url encoding (e.g. one containing a dot or a slash) must be slugged** — which is a good reason for D-07's ids to be slug-shaped from the start. The corpus runner should assert every corpus `expectId` matches `/^[A-Za-z0-9_-]{1,64}$/`.
---
## ONBD-03 feasibility — what "add to home screen" actually means inside Telegram
D-12 scopes ONBD-03 to "minimal PWA hooks — `addToHomeScreen` prompt handling + a startapp-less boot landing in the search-ready map". The research confirms this scope is right, and surfaces three constraints the plan must encode.
### The API is Bot API 8.0+ only
`addToHomeScreen()` and `checkHomeScreenStatus()` were added in **Bot API 8.0 (17 Nov 2024)**. They do not exist in older clients. The project's `TelegramWebAppLike` interface (`web/src/telegram/webapp.ts`) already carries `isVersionAtLeast(version: string): boolean` and the mock already implements `versionAtLeast` (`web/src/mock/telegram-mock.ts:110-121`) — so the capability guard has a precedent and a testable seam.
```typescript
// Extend the existing hand-rolled contract — do NOT add @types/telegram-web-app
// (Phase 3 dropped it deliberately; every fake would have to stub ~50 members).
export interface TelegramWebAppLike {
// … existing members …
addToHomeScreen?(): void;
checkHomeScreenStatus?(cb: (status: HomeScreenStatus) => void): void;
}
export type HomeScreenStatus =
| "unsupported" // the feature is not supported here
| "unknown" // supported, but whether the icon is installed is indeterminate
| "added" // already on the home screen
| "missed"; // not added yet
```
**The guard:** `if (wa && typeof wa.addToHomeScreen === "function" && wa.isVersionAtLeast("8.0"))`. Both conditions — the method may be absent on older clients, and the version check is what the platform itself documents.
### iOS cannot report installation status — treat "unknown" as normal
`checkHomeScreenStatus()` **always returns `"unknown"` on Telegram for iOS**, regardless of whether the Mini App is actually installed. This is a documented platform limitation, not a bug:
> Telegram-iOS issue #1729 (*"Home screen status detection always returns 'unknown' in Telegram iOS Mini Apps"*, April 2025) was closed with the maintainer comment: *"On iOS it is not possible to determine whether a Mini App has been added to the Home Screen. The platform does not provide any API that allows Telegram to check whether a Home Screen shortcut exists."*
**Design consequence:** the app must **not** branch on the status string in a way that treats `"unknown"` as an error or as "not added". The honest behaviour is: offer the prompt when the platform supports it; do not nag; do not claim the icon is present unless the status is literally `"added"`.
### What the home-screen shortcut actually opens
The shortcut opens the **Main Mini App** — which is the surface where `?startapp=` is officially supported (§"Deep-link risk", Carrier B). This is convenient: **ONBD-03's "lands in the app ready to search" needs no payload at all** (D-12 says "startapp-less boot"), and the shortcut naturally lands on the search-ready map. The `startapp` capability on that surface is a free bonus for a future phase (e.g. a per-destination home-screen shortcut).
**A separate, larger PWA manifest / service worker / offline story is explicitly deferred** (D-12, and CONTEXT's Deferred Ideas). Do not add a `manifest.webmanifest`, a service worker, or an install-prompt `beforeinstallprompt` handler — the browser API does not exist in the Telegram webview, and a second icon the Telegram client does not know about is worse than none. `addToHomeScreen()` is the *only* correct mechanism here.
### What the plan should build
1. Extend `TelegramWebAppLike` with the two optional methods (above).
2. A single, small "add to home screen" affordance — where it lives is a UI question, but the D-12 scope is deliberately minimal, and Phase 4's `UI hint: no` in the roadmap means **no new UI-SPEC round**. The least invasive placement is a card or a control the Phase 3 shell already has. **Flag: if this needs new UI furniture, it may need a UI-SPEC round — the planner should confirm it fits inside the existing Phase 3 shell.**
3. A capability-guard unit test (`web/test/homescreen.test.mjs`) covering: method absent → no-op; `isVersionAtLeast("8.0")` false → no-op; status `"unknown"` → treated as indeterminate, not as an error; status `"added"` → the affordance is hidden or shows "already added".
4. Extend the mock harness (`MockOptions`) with a `homeScreenStatus` option so the three branches are testable without a device.
---
## Common Pitfalls
### Pitfall 1: Assuming `start_param` arrives from a `web_app` inline button (THE phase's biggest trap)
**What goes wrong:** the plan builds the OPEN button as `web_app: { url: "https://app.example.com/?startapp=dest_klcc" }` and the webview reads `Telegram.WebApp.initDataUnsafe.start_param` — which is **undefined** for inline-button launches. The user taps OPEN, the Mini App opens, and the destination is not preloaded. ONBD-02 fails silently at the worst moment.
**Why it happens:** the official `WebAppInitData.start_param` documentation is scoped to `startattach` / attachment-menu launches and to *direct links*. The `web_app` inline button is a different launch surface. The two are easy to conflate because both end in "the Mini App has a start param".
**How to avoid:** treat the carrier as an explicit, tested choice with a fallback. §"Deep-link risk" ranks three carriers:
1. **Carrier A — `web_app` button whose URL carries the payload as a plain query param** (`https://app.example.com/?dest=dest_klcc`). The button's URL is an arbitrary HTTPS URL; Telegram opens it in the Mini App webview. Community reports confirm query params on that URL survive into the webview. **The webview reads `location.search`, not `start_param`.** Lowest moving-parts risk, but it is *not* what D-11 says (`?startapp=dest_`).
2. **Carrier B — `url` button to `https://t.me/?startapp=dest_`** (the documented `startapp` mechanism, and the exact mechanism D-11 names). Trade-off: the `url` button opens a **Telegram deep link**, not a Mini App button — so it leaves the chat, opens the app, and the "one tap to proceed" (D-02) becomes one tap + a Telegram interstitial. It also requires the bot to be configured as a Main Mini App in @BotFather (or a Direct Mini App with a `short_name`).
3. **Carrier C — both**: a `web_app` button carrying the id in the URL query (Carrier A) *and* the `startapp` direct-link path (Carrier B) for the home-screen entry point (ONBD-03), where `startapp` is officially available.
**Warning signs:** `initDataUnsafe.start_param` is `undefined` at runtime; the webview's `location.search` is empty; the Mini App boots into the plain "search-ready map" with no destination. **The Phase 3 mock harness must be extended to simulate each carrier's arrival shape** — the mock's `buildMockInitData` currently emits only `user` + `auth_date` + `query_id` + `hash`, with no `start_param` field at all (`web/src/mock/telegram-mock.ts:136-140`).
### Pitfall 2: `sendVenue` has no `text` field — the "one line confirming the destination" needs a home
**What goes wrong:** the plan tries to send a confirming sentence plus the venue card in one message. `sendVenue`'s parameter list is `latitude`, `longitude`, `title`, `address` (+ optional `foursquare_id`, `google_place_id`, …). **There is no `text` parameter.** The confirming line has to be either (a) a separate `sendMessage` before the card, or (b) baked into the venue's `title`/`address`.
**Why it happens:** `sendMessage` and `sendVenue` look alike from a distance; the CONTEXT's "venue card message should read like the bot's clinical voice: one line confirming the destination" reads like a single message.
**How to avoid:** decide explicitly. The cleanest reading of D-02 ("the best result ships immediately as the venue card") is: **one `sendVenue`** whose `title` is the entry name and whose `address` is the entry's human address, plus the inline keyboard carrying OPEN + alternates. If a separate confirming line is wanted, it is a second `sendMessage` — but that violates D-02's "no second message round" *only if* it needs a tap (it doesn't; two sequential sends is one round of user interaction, but it is two messages). The plan must pick and the copy gate should reflect it.
**Warning signs:** the bot's `sendVenue` call silently drops a `text` field; or the venue card's `title` ends up carrying marketing copy instead of the place name.
### Pitfall 3: Abbreviating a word that is part of a proper name
**What goes wrong:** the normalizer expands *every* token, so "Jln Tmn Desa" becomes "Jalan Taman Desa" — but the correct address is "Jln Taman Desa" (Jalan is the generic term; Taman Desa is the *name*). MS 2039 §5.2.3.2 is explicit: *"Abbreviation shall not be applied to names given to the geographical feature itself. As an example, an address line 'Jalan Taman Desa' may be abbreviated to 'Jln Taman Desa' but not 'Jln Tmn Desa' or 'Jalan Tmn Desa'."*
**Why it happens:** the expansion table is applied positionally and unconditionally.
**How to avoid:** the *canonical* form (long form) makes this benign **if the rule is applied symmetrically**: both "Jln Tmn Desa" and "Jln Taman Desa" expand to "Jalan Taman Desa" — because the rule is "expand the abbreviation to the long form", and "Tmn"→"Taman" is a correct expansion of the *word* even though the *address* shouldn't have used it. Expanding both sides to the long form makes them match, which is exactly what SRCH-02 wants. The MS 2039 rule matters for *display* fidelity (what address string the bot shows), not for *matching*. So: **use the long form for matching; keep the curated `address` string verbatim for display.**
**Warning signs:** the venue card's address shows "Jalan Taman Desa" when the curated entry says "Jln Taman Desa" — the card should show the curated address, not the normalized one.
### Pitfall 4: Fuzzy scoring outranking an exact match
**What goes wrong:** on short strings, Jaro-Winkler's prefix bonus or a normalized Damerau score can push a fuzzy match above an exact one. "KLCC" might rank "KLCC Park" above "Suria KLCC" — or a typo'd query might rank the wrong entry first.
**Why it happens:** both metrics are continuous and unbounded relative to each other; the "best" match is decided by arithmetic accident.
**How to avoid:** the disjoint tier bands in §"Pattern 3". An exact match must score **strictly above** the top of the fuzzy band, always. The corpus runner then reports *per-tier* hit rates, so a regression shows up as "exact-tier hits dropped" rather than a diffuse quality drop.
**Warning signs:** the corpus top-1 hit rate is good but top-3 is worse than expected, or a known exact alias loses to a near-miss.
### Pitfall 5: Trigram / prefix scoring on very short strings
**What goes wrong:** "KLCC" is 4 characters → 2 trigrams. "DUKE" → 2 trigrams. Trigram overlap on 2 grams is noise, not signal, and Jaro-Winkler's prefix bonus dominates trivially (both strings share the whole prefix → score ≈ 1.0).
**Why it happens:** trigram and Jaro-Winkler are both documented as unreliable on short strings. *"Very short strings provide so few trigrams that the overlap score is unreliable"* and *"Jaro-Winkler can over-reward shared prefixes when the discriminating information is later."*
**How to avoid:** **exact-match-first, fuzzy-last**, and a **minimum length guard** (the code above refuses to score anything under 3 characters). For 4–6 character queries, rely on the alias table (D-08) rather than on fuzzy scoring — "KLCC" and "DUKE" should be **aliases**, not fuzzy hits. This is the single strongest argument for D-08's design.
### Pitfall 6: The Overpass top-up script becomes a runtime dependency
**What goes wrong:** the plan wires the Overpass query into the Worker (to "top up" the gazetteer at boot). The Worker now depends on a third-party service with no SLA, on every cold start.
**Why it happens:** D-06 says "a repo-script Overpass one-shot that proposes top-up candidates for maintainer approval" — the "for maintainer approval" clause is easy to skim past.
**How to avoid:** the script is `bot/scripts/gazetteer-topup.mjs`, run manually, outputting a **proposal file** that a human diffs and merges. The Worker imports only the committed JSON.
**Warning signs:** `fetch("https://overpass-api.de/…")` anywhere under `bot/src/` or `web/src/`.
### Pitfall 6b: The Overpass script's `User-Agent` is load-bearing, and capacity 502s are normal (verification pass)
**What goes wrong:** (a) The script omits the `User-Agent` header and every request returns **HTTP 406 Not Acceptable** — which looks like a blocked network or a dead service, and can send the maintainer rewriting a working query. (b) The script treats an HTTP **502 "This Overpass API server is currently at capacity"** as a hard failure, exits non-zero, and (in a CI wiring) turns an optional maintainer tool into a red build.
**Why it happens:** (a) The Overpass instances — and the egress path in front of them — reject requests without an identifying `User-Agent`. **Probed 2026-10-23 on `overpass-api.de`, same host, same paths:** `GET /api/status` no-UA → **406**, with-UA → **200**; `GET /` no-UA → **406**, with-UA → **200**; `POST /api/interpreter` no-UA → **406**, with-UA → **200**. (b) The public instances are free and shared; the Overpass wiki's own policy table warns the main instance *"is overloaded – be mindful of that, do not overconsume resources and do not expect high reliability."* Both a **200** and a **502 capacity** response were observed on `overpass-api.de` **minutes apart** in the same session.
**How to avoid:**
1. **Always send a descriptive `User-Agent`** (e.g. `araharah-gazetteer-topup/1.0 (+https://github.com//araharah)`). Without it, *nothing* works — this is the single highest-value line in the script.
2. **Treat 502 / 429 / 504 / timeout as a normal outcome**: print one line — *"Overpass instance busy — retry later. No candidates generated; nothing changed."* — and **exit 0**. The curated file is untouched on every path, so a busy instance is not a failure.
3. **Prefer a narrow query.** Default to one tag family per invocation (`nwr["tourism"="attraction"](bbox)`); a narrow query is far more likely to fit in a free worker slot and returns in seconds. The wide multi-family union stays available behind a flag.
4. **Accept an endpoint override** (env var or `--source=`) so the maintainer can point at a different instance.
5. **The automated test uses a committed fixture**, never the live service (see §Wave 0 Gaps) — so the test is deterministic and the CI bill is zero.
**Warning signs:** The script exits non-zero with an HTML or plain-text body instead of JSON; or the same query "worked yesterday" and fails today with no code change.
### Pitfall 7: The golden corpus becomes a generated artifact
**What goes wrong:** the corpus file is regenerated from the gazetteer (so every entry gets a synthetic query) — which makes the test tautological: it passes by construction and measures nothing.
**Why it happens:** "extend by editing one file" (D-13) can be misread as "the file is machine-written".
**How to avoid:** the corpus is **hand-written and pinned** — every positive pins an expected `id`, and the corpus contains queries that are *not* derived from any alias in the gazetteer (typos, glued words, out-of-KL negatives). Reviewers extend it by *editing the file*. The corpus file must be `git diff`-able and its content must not be derivable from the gazetteer.
**Warning signs:** a corpus entry whose query string exactly equals an alias in the gazetteer, for every entry — that's the tautology signature.
### Pitfall 8: Sending the query text to a log line
**What goes wrong:** `console.log("search:", query)` while debugging. D-15 and the Phase 2 D-14 law forbid it — the query is user payload.
**Why it avoid:** the Phase 2 codebase's logging convention is *category-only* and every log line in `bot/src/webhook.ts` carries a category, never a payload. Match it: `console.log("[search] rejected: no-match")` — never the query.
**Warning signs:** any log line containing a variable that could hold a query string. The existing gates (`bot/test/*.test.mjs`) grep for structural patterns; **add a gate that greps `bot/src/` for a log statement whose argument mentions the query variable.**
### Pitfall 9: The webview's gazetteer copy drifting from the bot's
**What goes wrong:** the Worker ships gazetteer v1; the webview bundle ships v2 (built earlier). A `startapp=dest_klcc` arrives; the webview's `id → entry` lookup misses; the destination silently fails to load.
**Why it happens:** two independent build outputs from one source, deployed at different times.
**How to avoid:** D-11 already says "the shared module version" re-maps stale datasets. Concretely: (a) the id set is stable — **ids are never renamed, only added**; (b) the webview, on an unknown id, shows an honest "destination unavailable, search again" state rather than a blank map; (c) a **build-time gate** asserts the bot bundle and the web bundle contain the same id set — or, simpler, that both are built from the same committed JSON in the same `npm run build` (which `npm run build` already does: `theme` → `web` → `bot deploy`).
**Warning signs:** an id in the corpus that resolves in the bot but not in the webview.
### Pitfall 10: `callback_data` length and content
**What goes wrong:** putting the whole venue name + address in `callback_data`. The limit is **1–64 bytes** (confirmed in the `InlineKeyboardButton` type docs). Also, `callback_data` is visible to anyone who can see the message and is echoed back to the bot in the update.
**How to avoid:** `callback_data` carries **only the entry id** — `"alt:"` — and the handler looks the entry up in the gazetteer. Never a name, never a query, never coordinates. This also keeps D-15's privacy posture: the callback payload is a public place id, not user data.
---
## Malay Normalizer — the vendorable rule table
**This is the highest-reuse artifact in the research.** D-09 requires a bidirectional abbreviation table that "extends by table edit, not code change". The table below is **not reconstructed from memory** — it is the JUPEM (Jabatan Ukur dan Pemetaan Malaysia) normative abbreviation list, transcribed machine-readably by the OSM Nominatim project in `settings/icu-rules/variants-ms.yaml` (PR #4056, merged 2026-04-02), whose stated primary source is *"JUPEM Toponymic Guidelines — 'Abbreviations of Geographical Terms and Other Words Used in Maps'"*. **Vendor it with attribution; do not author a smaller one.**
### The table, verbatim (shared / Malaysia-specific)
```yaml
# shared/src/rules.ts — JUPEM-derived, via Nominatim settings/icu-rules/variants-ms.yaml
# Primary source: JUPEM Toponymic Guidelines (Malaysia), 2017 ed.
# Secondary: https://wiki.openstreetmap.org/wiki/Name_finder:Abbreviations#Bahasa_Melayu_-_Malay
# https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations
# Format: " -> "
shared:
Jalan -> Jln
Lorong -> Lrg, Lor
Lebuh -> Lbh
Simpang -> Spg
Kampung -> Kg
Kampong -> Kg
Taman -> Tmn
Bangunan -> Bgn
Bukit -> Bkt
Batu -> Bt
Sungai -> Sg
Pulau -> P, Pl
Tanjung -> Tg
Tanjong -> Tg
Teluk -> Tlk
Telok -> Tlk
Gunung -> G, Gn
Gunong -> G, Gn
Haji -> Hj
Hajjah -> Hjh
Abdul -> Abd
country: my
Persiaran -> Psrn
Lebuhraya -> Lbh.Raya
Bandar -> Bdr
Kondominium -> Kondo
Bahagian -> Bhg
Blok -> Blk
Kuala -> K, Kl
Alur -> Alr
Bakau -> Bak
Hulu -> H
Hilir -> Hr
Jeram -> J, Jr
Lembah -> Lmbh
Padang -> Pdg
Parit -> Pt
Permatang -> Pmtg
Pelabuhan -> Plbh
Kepulauan -> Kep
Ladang -> Ldg
Lapangan -> Lpg
Gosong -> Gsg
Sekolah kebangsaan -> SK
Sekolah menengah -> SM
Sekolah menengah kebangsaan -> SMK
Sekolah rendah -> SR
Hospital -> Hosp
Dispensari -> Disp
```
### The inverse map the normalizer actually needs
The table above is *full → abbreviation*. D-09 needs **both directions** (query says "Jln", entry says "Jalan"). Invert it at load time:
```typescript
// Build the bidirectional map ONCE at module load. Deterministic, ~60 entries.
const ABBREVIATIONS: Record = (() => {
const out: Record = {};
for (const line of RAW_RULES.trim().split("\n")) {
const m = line.match(/^\s*(\S.*?)\s*->\s*(.+?)\s*$/);
if (!m) continue;
const full = m[1].trim().toLowerCase();
for (const abbr of m[2].split(",").map((s) => s.trim().toLowerCase())) {
(out[abbr] ??= []).push(full);
// The full form maps to itself so the canonical form is always reachable.
(out[full] ??= []).push(full);
}
}
return out;
})();
```
**Ambiguity to handle — `Pl`/`P`, `K`/`Kl`, `G`/`Gn`, `J`/`Jr`, `Lbh`/`Lbh.Raya`:**
Several abbreviations map to more than one full form (`Pulau → P, Pl` and `Parit → Pt` are distinct, but `Kuala → K, Kl` vs `Kampung → Kg` are fine). The dangerous collisions are:
- **`G` → Gunung/Gunong** — but `G` also appears as a *letter* in road names ("Lebuh Genting"). Expanding `G` in a token that is a single letter is aggressive; the normalizer should **only expand an abbreviation when the token is a standalone word** (already true of token-based normalization) and when the expansion is unambiguous *in context*.
- **`P` → Pulau** and **`Pt` → Parit** — `P` also appears in "Presint", "Persiaran", "Pusat". Single-letter abbreviations are the highest-collision-risk class.
- **`Bt` → Batu** vs `Bukit → Bkt` — note `Bt` is Batu (milestone), *not* Bukit. Locals mix these; the golden corpus should cover both.
- **`KG` ambiguity** — `KG` is the kilogram unit in some contexts; the existing `bm-address-normaliser` project flags this explicitly. In a *place-name* corpus the ambiguity is negligible, but the plan should note it.
**Recommendation:** the normalizer keeps a per-token **ambiguity list** rather than guessing, and the fuzzy-residual tier (D-09) resolves what the table cannot. This matches D-09's architecture exactly: table first, folding second, fuzzy residual last.
### The Presint/Prsn vowel-drop rule
D-09 explicitly calls out "vowel-drop-aware Presint/Prsn forms". This is **not** in the JUPEM table above — it is a Putrajaya-specific local form. Verified against real Malaysian commercial addresses:
- `JALAN P8G PRESINT 8, 62250 Putrajaya`
- `Presint 15`, `PRESINT 16`, `PRESINT 4`, `PRESINT 18`
- `PRSN RESIDEN DESA PARKCI, 52200 KUALA LUMPUR`
- `NO 252 BAWAH PRSN SIBC 9, SERI ISKANDAR BUSINESS CENTRE`
- `Jalan P9 Presint 7, 62250 Putrajaya`
So both "Presint" and "Prsn" appear in live use, and "Prsn" is not a *prefix* abbreviation — it drops the internal `e`. Add it as an explicit rule:
```typescript
// Vowel-drop expansion (Putrajaya precincts) — D-09's explicit case.
ABBREVIATIONS["prsn"] = ["presint"];
ABBREVIATIONS["presint"] = ["presint"];
```
**A more general vowel-drop rule** worth considering (and worth a corpus test): Malay place names with a dropped interior vowel are a recurring local shorthand class — `Pusat`→`Pusat`, `Bandar`→`Bdr`, `Perbandaran`→`PBandaran`. The plan should decide whether to implement a *generic* vowel-drop expansion (insert `e` at each interior consonant cluster) or keep only the explicit `Presint`/`Prsn` pair. **Recommendation: explicit pair only.** A generic vowel-drop rule will over-expand legitimate names ("Pusat" contains no drop, but "Pkl" would expand to "Pekel" and then match "Pekel" wrongly). This is exactly the kind of rule that should be added by *table edit* (D-09) when the corpus demands it — not by a clever algorithm.
### The MS 2039 rule that governs *display* (not matching)
From **MS 2039:2006** §5.2.3.2 (fetched this session):
> *"Abbreviation may only apply to a generic name of a geographical feature being referred in a particular line or hierarchies of the address format. Abbreviation shall not be applied to names given to the geographical feature itself. As an example, an address line 'Jalan Taman Desa' may be abbreviated to 'Jln Taman Desa' but not 'Jln Tmn Desa' or 'Jalan Tmn Desa'."*
**How this applies to the normalizer:** the rule constrains what a *correct* address looks like, not what a *searchable* key looks like. Because the normalizer expands to the **long form on both sides**, "Jln Tmn Desa" and "Jln Taman Desa" both normalize to "jalan taman desa" and match. That is the desired search behaviour. **But the venue card's `address` field must show the curated address verbatim** — not the normalized form. The `sendVenue` payload's `address` comes from the gazetteer entry, never from `normalize()`. This is a one-line architectural rule that prevents a visible regression (Pitfall 3).
### The three official abbreviation tables (for completeness)
The JUPEM gazetteer PDF and MS 2039 Annex A carry **three** distinct tables. The normalizer needs all three, merged:
| Table | Scope | Sample rows |
|-------|-------|-------------|
| **A1 — Street prefixes** (MS 2039 Annex A) | `Jalan → Jln`, `Lorong → Lrg`, `Persiaran → Psrn`, `Lebuhraya → Lbh.Raya`, `Persisiran → (none)`, `Pekeliling → (none)`, `Lintang → Ltg`, `Lengkok → Lnkg`, `Regat → Rgt` | 33 rows |
| **A3 — Locality prefixes** (MS 2039 Annex A + JUPEM gazetteer) | `Bandar → Bdr`, `Bukit → Bkt`, `Desa → (none)`, `Kampung/Kampong → Kg, Kpg`, `Kuala → K`, `Hulu → H`, `Hilir → Hr`, `Jeram → J`, `Ladang → Ldg`, `Padang → Pdg`, `Pasir → Psr`, `Permatang → Pmtg`, `Pulau → P`, `Sungai → Sg`, `Taman → Tmn`, `Tanjung → Tg`, `Teluk/Telok → Tlk`, `Mukim → (none)`, `Pekan → (none)` | ~35 rows |
| **JUPEM gazetteer list** (state gazetteer PDF, §4.0) | `Air → A.`, `Bahagian → Bhg.`, `Balai Polis → B.P.`, `Balai Raya → B.R.`, `Bakau → Bak.`, `Batu → Bt.`, `Blok → Blk.`, `Bukit → Bkt.`, `Cangkat/Changkat → Chgkt.`, `Hosp → Hosp.`, `Kondominium → Kondo`, `Lpg → Lapangan`, `Pejabat Pos → P.P.`, `Pangkalan/Pengkalan → Pang.`, `Pelabuhan → Plbh.`, `Rumah Rehat → R.R.`, `SMK`, `SR`, `SRJK`, `Stesen Kastam → S.K.`, `Tali air → T.A.`, `Utara → U` | 48 rows |
**Note the intersection is not clean:** MS 2039 A3 gives `Hilir → Hr` while the JUPEM gazetteer list gives `Hilir` (no abbreviation) — and `Kuala → K` in A3 vs `K. ` in the gazetteer list. **Merge, don't replace**; when two tables disagree, prefer the **MS 2039** form (it is the current national standard) and record the alternative as an additional alias.
### A ready-made city-code table
The existing `bm-address-normaliser` project and the Nominatim PR both carry a **city-code** expansion set that is genuinely useful for KL-metro queries ("PJ" → Petaling Jaya, "JB" → Johor Bahru, "KB" → Kota Bharu, "KK" → Kota Kinabalu). These are *local shorthand*, not JUPEM abbreviations — they belong in the **gazetteer alias table (D-08)**, not the abbreviation rule table, because they are place identities, not address-word forms. The distinction matters: "KL" as a city code should expand to "kuala lumpur"; "Jln" should expand to "jalan". Both are expansions, but they belong in different tables.
---
## Code Examples
### The `sendVenue` call with inline keyboard (D-10, SRCH-03)
**Source:** https://core.telegram.org/bots/api/methods/sendvenue.md + https://core.telegram.org/bots/api/types/inlinekeyboardbutton.md
```typescript
import { InlineKeyboard } from "grammy";
// Required: latitude, longitude, title, address. There is NO `text` parameter.
await ctx.replyWithVenue(
entry.lat,
entry.lng,
entry.name, // title
entry.address ?? "", // address
{
reply_markup: new InlineKeyboard()
.webApp(openLabel, miniAppUrlFor(entry.id)) // OPEN IN ARAHARAH
.row()
.text(alt1.name, `alt:${alt1.id}`) // alternates → callback_data
.text(alt2.name, `alt:${alt2.id}`),
},
);
```
**Note on `reply_markup`:** `sendVenue` accepts `InlineKeyboardMarkup | ReplyKeyboardMarkup | ReplyKeyboardRemove | ForceReply` — the inline keyboard is the correct one here.
### The `web_app` button URL vs. the `startapp` direct link
```typescript
// Carrier A — web_app inline button, payload in the URL query string.
// The webview reads location.search, NOT initDataUnsafe.start_param.
function miniAppUrlFor(id: string, origin: string): string {
return `${origin}/?dest=${encodeURIComponent(id)}`;
}
// Carrier B — t.me direct link, payload in ?startapp= (the officially documented
// start_param mechanism). Requires a Main Mini App or a Direct Mini App (short_name).
function deepLinkFor(id: string, botUsername: string): string {
// start_param charset is restricted to [A-Za-z0-9_-] and ≤512 chars.
return `https://t.me/${botUsername}?startapp=${id.replace(/[^A-Za-z0-9_-]/g, "_")}`;
}
```
### Reading the payload in the webview
```typescript
// web/src/search/destination.ts — Carrier A (URL query).
export function readDestinationId(): string | null {
const raw = new URLSearchParams(window.location.search).get("dest");
return raw && /^[A-Za-z0-9_-]{1,64}$/.test(raw) ? raw : null;
}
// Carrier B (initData) — note: initDataUnsafe is UNTRUSTED per Telegram's docs;
// it is fine here only because the id is a public place id and is re-resolved
// against the local gazetteer. Never treat it as an identity.
export function readStartParam(wa: TelegramWebAppLike | null): string | null {
const raw = parseInitDataUnsafe(wa?.initData ?? "").start_param;
return raw && /^[A-Za-z0-9_-]{1,512}$/.test(raw) ? raw : null;
}
```
### The alternates callback handler
```typescript
// bot/src/bot/handlers/search-alternate.ts
export function registerSearchAlternate(bot: Bot, source: SearchSource): void {
bot.callbackQuery(/^alt:(.+)$/, async (ctx) => {
await ctx.answerCallbackQuery(); // must happen within 30 s
const id = ctx.match[1];
const entry = source.byId(id); // id → entry; never a query or a name
if (!entry) return;
await sendVenueCard(ctx, entry, source); // re-sends the card with new alternates
});
// Always answer everything else so Telegram doesn't show a 60-second spinner.
bot.on("callback_query:data", async (ctx) => {
await ctx.answerCallbackQuery();
});
}
```
**Source:** https://grammy.dev/plugins/keyboard — *"It makes sense to define `bot.on("callback")` at last to always answer all other callback queries that your previous listeners did not handle. Otherwise, some clients may display a loading animation for up to a minute."*
### The Malay normalizer
```typescript
// shared/src/normalize.ts
export function normalize(input: string): string {
let s = input.normalize("NFKD").toLowerCase();
s = s.replace(/[̀-ͯ]/g, ""); // strip combining marks (NFKD)
s = s.replace(/[^\p{L}\p{N}\s/]/g, " "); // punctuation → space (keep "/" for Jln 3/1)
s = s.replace(/\s+/g, " ").trim();
const expanded: string[] = [];
for (const token of s.split(" ")) {
if (!token) continue;
const hit = ABBREVIATIONS[token];
expanded.push(hit ? hit[0] : token); // canonical = the long form
}
s = expanded.join(" ");
s = splitGluedWords(s); // "tamanbangsar" → "taman bangsar"
return s;
}
/** Insert a space at the boundary where a known Malay word starts. */
function splitGluedWords(s: string): string {
// Starters that commonly appear glued in Malaysian addresses.
const starters = ["jalan", "taman", "kampung", "lorong", "bukit", "sungai",
"bandar", "persiaran", "lebuh", "kampung", "perkampungan"];
for (const starter of starters) {
// "tmnbangsar" is handled by the abbreviation table; this catches
// "tamanbangsar" / "jalankanpar" style glue.
const re = new RegExp(`(? `${m.slice(0, starter.length)} ${m.slice(starter.length)}`);
}
return s;
}
```
**The Presint/Prsn rule (D-09's "vowel-drop-aware" clause):** Putrajaya's precincts are written both "Presint" and "Prsn" — the latter drops the `e`. Add an explicit rule:
```typescript
// "prsn" → "presint" is a *vowel-drop* expansion, not a prefix abbreviation.
ABBREVIATIONS["prsn"] = ["presint"];
ABBREVIATIONS["presint"] = ["presint"];
```
Verified against real usage: `JALAN P8G PRESINT 8`, `PRESINT 15`, `PRSN RESIDEN DESA PARKCI`, `Jalan P9 Presint 7` — all appear in commercial Malaysian addresses (see §"Sources").
**The glued-word rule needs care.** A naive "insert a space before any known word" will corrupt words that legitimately contain a starter as a substring ("Pandang" contains "dang", "Kampung Baru" is fine, but "Bandar Baru" → "band ar" if "dar" were a starter). **The safe implementation is a longest-match-first scan against the starter list, requiring the remainder to be ≥3 characters.** The plan should specify this; the naive version is a real bug source.
### The Damerau-Levenshtein (OSA) distance — hand-rolled
```typescript
// shared/src/distance.ts — optimal string alignment, adjacent transpositions.
export function damerau(a: string, b: string): number {
if (a === b) return 0;
if (a.length === 0) return b.length;
if (b.length === 0) return a.length;
const n = a.length, m = b.length;
// 3-row band: only the previous, current and current+1 rows are needed.
let prev2 = new Int32Array(m + 1);
let prev = new Int32Array(m + 1);
let cur = new Int32Array(m + 1);
for (let j = 0; j <= m; j++) prev[j] = j;
for (let i = 1; i <= n; i++) {
cur[0] = i;
for (let j = 1; j <= m; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
let v = Math.min(cur[j - 1] + 1, prev[j] + 1, prev[j - 1] + cost);
if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) {
v = Math.min(v, prev2[j - 2] + 1); // transposition
}
cur[j] = v;
}
const t = prev2; prev2 = prev; prev = t;
const c = cur; cur = prev2; // rotate the three rows
}
return prev[m];
}
```
**Type-check this carefully** — the row-rotation in a 3-band OSA is easy to get wrong. Write the test vectors first (TDD, matching the repo's RED→GREEN convention).
### The golden corpus format
```jsonc
// shared/data/golden-corpus.json
{
"version": 1,
"cases": [
// ---- golden landmarks (SRCH-01) ----
{ "q": "KLCC", "expectId": "suria-klcc" },
{ "q": "Sunway Pyramid", "expectId": "sunway-pyramid" },
{ "q": "DUKE", "expectId": "the-duke" },
// ---- abbreviations (SRCH-02) ----
{ "q": "Jln Ampang", "expectId": "suria-klcc" }, // Jalan Ampang
{ "q": "Tmn Bangsar", "expectId": "bangsar" },
{ "q": "Kg Ampang", "expectId": "kampung-ampang" },
// ---- glued words (SRCH-02) ----
{ "q": "tamanbangsar", "expectId": "bangsar" },
// ---- typos (SRCH-02) ----
{ "q": "KCLC", "expectId": "suria-klcc" }, // transposition
{ "q": "Sunwya Pyarmid", "expectId": "sunway-pyramid" },
// ---- aliases (D-08) ----
{ "q": "IJN", "expectId": "ijnp" },
{ "q": "PWTC", "expectId": "pwtc" },
{ "q": "Stesen KL Sentral","expectId": "kl-sentral" },
// ---- deliberate negatives (D-13) — MUST be refused, never a fabricated pin ----
{ "q": "Kuching", "expectId": null },
{ "q": "Penang", "expectId": null }
]
}
```
### The corpus runner
```javascript
// bot/scripts/gazetteer-corpus.mjs — the bake-off close (D-14).
// Prints top-1 / top-3 hit rate on positives and refusal rate on negatives.
// Exit code 1 if top-3 < 90% or if any negative produced a hit.
const { search, gazetteer } = await import("@araharah/search");
const corpus = JSON.parse(await readFile(corpusPath, "utf8"));
const positives = corpus.cases.filter((c) => c.expectId !== null);
const negatives = corpus.cases.filter((c) => c.expectId === null);
let top1 = 0, top3 = 0, missed = [];
for (const c of positives) {
const hits = await search({ text: c.q, limit: 3 });
const ids = hits.map((h) => h.entry.id);
if (ids[0] === c.expectId) top1++;
if (ids.includes(c.expectId)) top3++;
else missed.push(`${c.q} → [${ids.join(", ") || "—"}] (expected ${c.expectId})`);
}
let refused = 0, leaked = [];
for (const c of negatives) {
const hits = await search({ text: c.q, limit: 3 });
if (hits.length === 0) refused++;
else leaked.push(`${c.q} → ${hits[0].entry.id}`);
}
const top3Rate = top3 / positives.length;
console.log(`top-1 ${top1}/${positives.length} top-3 ${top3}/${positives.length} (${(top3Rate*100).toFixed(1)}%)`);
console.log(`negatives refused ${refused}/${negatives.length}`);
if (missed.length) console.log("MISSED:\n" + missed.join("\n"));
if (leaked.length) console.log("LEAKED (must refuse):\n" + leaked.join("\n"));
// D-14 gate: top-3 ≥ 90% AND every negative refused.
process.exitCode = top3Rate >= 0.9 && leaked.length === 0 ? 0 : 1;
```
**This runner IS the bake-off close.** Its output is the artifact D-14 asks to be recorded: "the phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question." The plan should also produce a short `docs/SEARCH-QUALITY.md` (or a section in the phase summary) recording: the corpus version, the score, and which escalation lever was pulled.
---
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Nominatim-style server-side geocoding for every query | In-Worker curated gazetteer behind a `SearchSource` interface | D-03 (this phase) | Zero infrastructure, zero latency, zero third-party SLA — but only as good as the curated corpus. Phase 4's job is to prove the corpus is good enough (D-14). |
| Photon-on-VPS as the phase-4 backend | Photon deferred to Phase 5; in-Worker gazetteer is backend #1 | D-03 (supersedes the roadmap note) | The roadmap's Phase 4 note says "VPS provisioning begins here" — **D-03 supersedes it.** The plan must NOT include VPS/Photon provisioning. |
| A Malaysia-only WASM geocoder as the phase-4 backend | Documented as backend #2 (post-v1, OSS track) | D-05 | The JSON dataset format stays reachable from day one via the converter script, so backend #2 is a format change, not a rewrite. |
| Hand-authored abbreviation list | The official JUPEM table, vendored | Nominatim PR #4056, merged 2026-04-02 | The table grew from 8 entries to 65+, split into shared / MY / SG / BN. Vendoring it wholesale (rather than the 8-entry original) is what makes "Jln Ampang" work for free. |
| Trigram index for short-query search | Exact-match tiers + alias table + fuzzy residual | This phase (D-09) | Trigram on 4-character queries is noise. The alias table (D-08) is the real lever for short queries. |
| `sendMessage` + `sendLocation` for the destination confirmation | `sendVenue` | D-10 | Telegram's native venue rendering (pin + name + address, tappable in the client) is what SRCH-03 names. |
| `startapp` on the `web_app` inline button URL (as CONTEXT D-11 reads) | An explicit carrier choice with a fallback | This phase's research | `startapp` is officially a **direct-link** mechanism; the `web_app` button carries a plain URL. See §"Deep-link risk" — the plan must pick. |
**Deprecated / to avoid:**
- **`@types/telegram-web-app`** — Phase 3 deliberately dropped it (the hand-rolled `TelegramWebAppLike` contract is the project's contract). Extend that interface; do not re-add the package.
- **`bot.chat.*`** — grammY 1.46 exposes no `bot.chat` proxy (noted in `bot/src/bot/handlers/location-fallback.ts:34-35`). Use `bot.api.*` / `ctx.*`.
- **`damerau-levenshtein` (the old CJS-ish package)** — last published 2022; prefer `@crob/damerau-levenshtein` or hand-roll.
---
## Assumptions Log
> Claims tagged `[ASSUMED]` below need user confirmation before becoming a locked decision.
| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | The `web_app` inline button's `WebAppInfo.url` may carry a plain query param (`?dest=`) that survives into the Mini App webview's `location.search`. | §"Deep-link risk" / Carrier A | **MEDIUM-HIGH.** Supported by community reports (StackOverflow: *"passing query params to telegram web app just works"*) but **not** by Telegram's official docs, which only describe `startapp` for direct links. If wrong, Carrier A collapses and the phase must use Carrier B (t.me deep link) — which changes the UX (an interstitial) and the button type. **Mitigation: build the carrier as a swappable function and test it on a real device at the exit gate.** |
| A2 | `Telegram.WebApp.initDataUnsafe.start_param` is populated when a Mini App is launched from a **direct link** (`t.me?startapp=…`) — i.e. that the "direct link" mechanism populates `start_param` in initData. | §"Deep-link risk" / Carrier B | **LOW-MEDIUM.** Officially documented (`start_param` "will also be passed in the GET-parameter `tgWebAppStartParam`"), but the `WebAppInitData.start_param` row's wording ("Only returned for Mini Apps when launched from the attachment menu via link") is narrower and could be read as excluding direct links. |
| A3 | The gazetteer is a **Derivative Database** under ODbL (because entries are selected/curated from OSM data) and therefore must be offered under ODbL. | §"Security Domain" | **LOW.** The consequence is mild (add an ODbL notice + attribution; the data is already public). But if the curation were deemed a *Produced Work* rather than a Derivative Database, the share-alike obligation would not attach. **Mitigation: add the notice anyway — it costs nothing and satisfies both readings.** |
| A4 | A ~200–500-entry gazetteer with ~5 aliases each is sufficient to hit ≥90% top-3 on a 60–100-query hand-written corpus. | §"Golden corpus" / D-14 | **MEDIUM.** This is the phase's core empirical assumption. It is exactly what the corpus runner is designed to falsify. If it fails, D-14's escalation ladder applies (alias/normalizer edits first). |
| A5 | Node 24's default TypeScript type-stripping lets a `.mjs` script `import("../src/foo.ts")` — the pattern already used by `bot/test/commands.test.mjs`. | §"Recommended Project Structure" | **LOW.** The existing tests already rely on it and pass, so the pattern is proven in-repo. Verify in the plan's first task anyway. |
| A6 | The `Mini App` public URL can be derived from `WEBHOOK_URL` (the Worker's origin) rather than needing a new env var. | §"Runtime State Inventory" | **LOW.** `bot/wrangler.jsonc` has `workers_dev: true` and `docs/CONFIGURATION.md` documents `WEBHOOK_URL` as the full origin + path. The plan may add a dedicated `MINI_APP_URL` var for clarity — a small, cheap decision. |
| A7 | `checkHomeScreenStatus()` returning `"unknown"` on iOS is a platform limitation, not a bug to work around. | §"ONBD-03 feasibility" | **LOW.** Confirmed upstream (Telegram-iOS issue #1729, closed with the comment *"On iOS it is not possible to determine whether a Mini App has been added to the Home Screen"*). The design must not branch on iOS status — treat `"unknown"` as "supported but indeterminate". |
| A8 | `sendVenue` messages can carry an inline keyboard that is later edited with `editMessageReplyMarkup` without destroying the venue payload. | §"Alternatives Considered" | **LOW-MEDIUM.** The Bot API docs say a message can be edited "without `reply_markup` or with inline keyboards" — but a `venue` message's *content* is a venue, and `editMessageText` would replace it. `editMessageReplyMarkup` only touches the markup. Verify on device; the safe fallback is to send a *new* `sendVenue` (which also matches D-02's "one tap to switch"). |
---
## Open Questions (RESOLVED)
> **STATUS: ALL RESOLVED (2026-10-23, during plan-phase).** Each question below carries its
> `(RESOLVED — …)` decision line; the decisions are binding and are implemented in plans
> 04-01…04-04. Where a plan CORRECTED the original recommendation, the correction is stated.
1. **Which deep-link carrier does Phase 4 ship?** (D-11 says `?startapp=dest_`; the docs put `startapp` on direct links, not inline buttons.)
- **What we know:** `WebAppInfo.url` is an arbitrary HTTPS URL; community reports say query params survive; `startapp` is documented for `t.me` links and `startattach` for attachment-menu links; the `web_app` inline button is a distinct launch surface.
- **What's unclear:** whether Telegram's own clients preserve the query string on a `web_app` button URL across Android/iOS/Desktop.
- **Recommendation:** implement **Carrier A** (`web_app` button, `?dest=` in the URL) as the primary, keep **Carrier B** (`t.me?startapp=`) for the ONBD-03 home-screen path, and **write the payload reader to accept both** (`location.search` first, then `initDataUnsafe.start_param`). Validate on a real device at the exit gate. **Flag to the user**, because it touches a locked decision's wording.
- **(RESOLVED — Carrier A (`?dest=` in the `web_app` button's URL, read from `location.search`) is primary; Carrier B is the signed `initData`'s `start_param` field (the direct-link / home-screen path), NOT `initDataUnsafe` — the recommendation's `initDataUnsafe` reading was corrected: only the SIGNED `initData` is parsed. Both carriers share one validation. Device validation happens at 04-04's blocking gate. Plans 04-03/04-04.)**
2. **Is the venue card one message or two?**
- **What we know:** `sendVenue` has no `text` parameter; D-02 wants one tap to proceed; CONTEXT's "specific ideas" mention a one-line clinical confirmation.
- **What's unclear:** whether "one line confirming the destination" is a separate `sendMessage` before the card, or a property of the card's own `title`/`address`.
- **Recommendation:** **one `sendVenue`** whose `title` is the place name and `address` is the curated address — the native card *is* the confirmation, and this honours D-02's single-message shape. If the user wants a separate clinical line, it costs one extra message and must be added to the copy gate.
- **(RESOLVED — exactly one `sendVenue` per card; `sendVenue` has no `text` param, so the card's own `title` (canonical name) + `address` (curated, verbatim) IS the confirmation. Enforced by CR4's request-count gate. Plan 04-03.)**
3. **What is the exact corpus size and the exact per-query expectation?**
- **What we know:** D-13 says ~60–100 queries with the listed categories; D-14 sets ≥90% top-3 on positives.
- **What's unclear:** whether the 90% is measured over *all* positives or over positives *excluding* the typo bucket (typos are the hardest and the fuzzy tier's raison d'être).
- **Recommendation:** measure and report **per-category** rates (exact / abbreviation / glued / typo / alias), and set the D-14 gate on the **all-positives** figure, with per-category figures reported alongside so a regression is diagnosable. This is OpenCode's Discretion; state it in the plan.
- **(RESOLVED — the D-14 gate measures the ALL-POSITIVES top-3 rate (≥ 90%, threshold `0.9` pinned in the corpus file); per-category rates are REPORTED alongside and carry authoring floors (landmark ≥ 12, abbreviation ≥ 15, glued ≥ 6, typo ≥ 12, alias ≥ 12, negative ≥ 6). Plan 04-02, corpus format + CO7.)**
4. **Which `SearchSource` seam shape?**
- **What we know:** D-03 requires a pluggable interface with the gazetteer as implementation #1, WASM geocoder as #2, Photon as #3.
- **What's unclear:** whether the interface is synchronous or async, and whether it lives in the shared package or in `bot/src/search/`.
- **Recommendation:** `async search(): Promise` (a network-backed backend #3 will need it), defined in the **shared package** so the webview can use the same types. The `byId(id)` lookup — needed by the callback handler and the webview — should be a **separate, synchronous** method on the interface, because both the callback handler and the deep-link reader need it and neither should await a network.
- **(RESOLVED — `SearchSource` lives in the shared `search/` workspace (`search/src/types.ts`): `async search(q): Promise` plus a SEPARATE SYNCHRONOUS `byId(id)`. The package also exports a synchronous `search(text, limit?)` convenience wrapper for the corpus runner. Plans 04-01 (contract), 04-03/04-04 (consumers).)**
5. **Is the Overpass script required to have a *verified* run in this phase?**
- **What we know:** `overpass-api.de` returns **406 without a `User-Agent` and 200 with one** (verification pass, same host/same paths); `overpass.kumi.systems` and `overpass.private.coffee` are unreachable from this network (HTTP 000); `overpass.osm.ch` returns 400 (Switzerland-only); Nominatim and Geofabrik work.
- **What's unclear:** whether the phase's exit gate requires the Overpass path to have been exercised against live OSM.
- **Recommendation:** the plan should require the script to **run and produce a valid-shaped proposal file** (verified against a **committed fixture** of a real Overpass response, committed to the repo so the test is deterministic), and to document the live-source URL in the script's header. **The `User-Agent` header is load-bearing and must be asserted by a test** — a missing UA produces a 406 on every endpoint and looks like a dead service. The curated entries themselves should be coordinate-verified against **Nominatim** (reachable) — which satisfies D-06's "hand-curated seed" with real coordinates. **Do not plan around a blocked Overpass**: the live path works from this machine once the header is set; the fixture is for *determinism*, not for working around an outage.
- **(RESOLVED — no live-Overpass obligation: the script is proven offline against the COMMITTED fixture (`--source=fixture`), with `--source=overpass|nominatim` as the maintainer's live paths; the identifying `User-Agent` is mandatory and TP2-asserted; a busy instance soft-exits 0 (TP4); curated coordinates are spot-verified against Nominatim and recorded in `docs/SEARCH-QUALITY.md`. Plan 04-02.)**
---
## Environment Availability
> Audited this session (2026-10-23) by probing each dependency from this machine.
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| **Node.js** | everything (build, tests, corpus runner) | ✓ | `v24.18.0` | — |
| **npm** | workspace install | ✓ | `11.16.0` | — |
| **grammy** (installed) | bot handler | ✓ | `1.46.0` | — |
| **wrangler** (installed) | Worker deploy / local dev | ✓ | `4.151.3` | — |
| **TypeScript** (installed) | typecheck | ✓ | `5.9.3` | — |
| **Playwright** (installed, web) | capture evidence | ✓ | `1.50.0` | — |
| **curl** | top-up script | ✓ | `8.14.1` | — |
| **Python 3** | not needed this phase (theme gallery only) | ✓ | `3.13.5` | — |
| **Overpass API (`overpass-api.de`)** | D-06 top-up script (live) | ✓ **with a User-Agent** | **CORRECTED by the verification pass (see §Appendix A4):** the HTTP 406 is caused by the **missing `User-Agent` header**, not by an egress filter. Probed 2026-10-23 on the same host: `GET /api/status` **no UA → 406 / with UA → 200**; `GET /` **no UA → 406 / with UA → 200**; `POST /api/interpreter` **no UA → 406 / with UA → 200** (19 KB of real KL data, `osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z`). The script's `User-Agent` is therefore **load-bearing, not cosmetic**. | `overpass.kumi.systems` and `overpass.private.coffee` are **unreachable from this network** (connection failure, HTTP 000, with or without a UA); `overpass.osm.ch` returns **400** (Switzerland-only coverage, as the body noted). The script should still accept a `--source=` / env override and the automated test should use a **committed fixture**. Capacity 502s are intermittent — see §Appendix A4. |
| **Nominatim** (`nominatim.openstreetmap.org`) | coordinate verification of curated entries | ✓ | live (200; `?q=KLCC` returned correct KL nodes) | — |
| **Geofabrik extract** | bulk OSM fallback for the top-up script | ✓ | `asia/malaysia-singapore-brunei-latest.osm.pbf` (206 on range request) — note: **not** `malaysia-latest.osm.pbf` (404) | — |
| **Cloudflare account / wrangler auth** | deploy | ✗ (not probed — needs a real account) | — | Deploy is a manual exit step; Phase 2/3 precedent |
| **A real Telegram client** | ONBD-02 / ONBD-03 device validation | ✗ (not available here) | — | Phase 3's device-matrix precedent: a real-device session is an exit-gate obligation |
**Missing dependencies with no fallback:**
- **A live `overpass-api.de` endpoint.** The Overpass *script* can still be written and tested against a committed fixture, but it cannot be validated end-to-end from this machine. If the phase's exit gate requires a live Overpass run, it must be done from a different network (or via the `overpass.osm.ch` mirror, which has no MY coverage). **Plan around this.**
- **A real device / Telegram client** for ONBD-02/ONBD-03 validation. This is inherent to the phase (Phase 3 set the same precedent with its device matrix).
**Missing dependencies with fallback:**
- **Overpass** → Nominatim search, Geofabrik extract, or a committed fixture.
- **A fuzzy-distance dependency** → hand-roll (recommended) or `@crob/damerau-levenshtein@2.0.0`.
---
## Validation Architecture
> `workflow.nyquist_validation` is `true` in `.planning/config.json` (not `false`), so this section is **required**. The repo has custom zero-dependency test harnesses (no Jest/Vitest/Mocha): `bot/test/runner.mjs`, `web/test/runner.mjs`, `theme/scripts/audit.test.mjs` — each auto-discovers `*.test.mjs` and drains a shared registry.
### Test Framework
| Property | Value |
|----------|-------|
| **Framework** | Hand-rolled zero-dependency runner (repo-native pattern). `bot/test/harness.mjs` exports `test(name, fn)`; `bot/test/runner.mjs` auto-discovers `*.test.mjs`, runs each, prints `PASS`/`FAIL`, sets `process.exitCode = failedCount > 0 ? 1 : 0`. The `web/` runner is byte-for-byte the same shape. |
| **Config file** | None — no jest/vitest config exists. Suites are discovered by filename convention (`*.test.mjs`). |
| **Quick run command** | `npm run bot:test` (→ `node test/runner.mjs` in `bot/`) — runs **all** bot suites; there is no per-file filter today. `npm run web:test` for the web side. |
| **Full suite command** | `npm run bot:test && npm run web:test && npm run audit` (theme audit) — plus `npm run bot:typecheck && npm run web:typecheck`. |
| **Corpus run command** | `node bot/scripts/gazetteer-corpus.mjs` (to be created) — the D-14 bake-off close. Should be wired into `npm run bot:test` **or** run as an explicit gate; the plan must decide (recommend: a dedicated `npm run search:corpus` script, invoked by the phase's exit gate, and *also* asserted inside a `*.test.mjs` suite so `npm run bot:test` fails when the corpus regresses). |
| **Typecheck** | `npm run bot:typecheck` / `npm run web:typecheck` → `tsc --noEmit` per workspace. |
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| **SRCH-01** | "KLCC" (and the golden landmarks) resolve to the expected entry; the shortlist's first item is the right one | unit (corpus) | `node bot/scripts/gazetteer-corpus.mjs` → asserts top-1/top-3 ≥ threshold | ❌ **Wave 0** — `bot/scripts/gazetteer-corpus.mjs` |
| **SRCH-01** | The gazetteer module loads, every entry has the required fields, ids are unique and slug-safe, every entry has ≥1 alias | unit | `npm run bot:test` (suite `bot/test/gazetteer.test.mjs`) | ❌ **Wave 0** — `bot/test/gazetteer.test.mjs` |
| **SRCH-02** | "Jln Ampang" and "Jalan Ampang" normalize to the same string; "Tmn"→"taman"; "Kg"→"kampung"; "Prsn"→"presint"; glued "tamanbangsar" splits | unit (rule vectors) | `npm run bot:test` (suite `bot/test/normalizer.test.mjs`) | ❌ **Wave 0** — `bot/test/normalizer.test.mjs` |
| **SRCH-02** | Every abbreviation-table row has a round-trip vector (query form → expected canonical) | unit | `npm run bot:test` (suite `bot/test/normalizer.test.mjs`) | ❌ **Wave 0** |
| **SRCH-02** | Transposition typos ("KCLC") and substitution typos ("Sunwya Pyarmid") resolve to the expected entry | unit (corpus) | `node bot/scripts/gazetteer-corpus.mjs` | ❌ **Wave 0** |
| **SRCH-02** | The Damerau-Levenshtein implementation matches hand-computed OSA distances (test vectors from the literature: `distance("ab","ba")==1`, `distance("a cat","an act")==2`) | unit | `npm run bot:test` (suite `bot/test/distance.test.mjs`) | ❌ **Wave 0** — `bot/test/distance.test.mjs` |
| **SRCH-02** | Exact-match tier strictly outranks any fuzzy hit; scores under the min-length guard return 0 | unit | `npm run bot:test` (suite `bot/test/score.test.mjs`) | ❌ **Wave 0** |
| **SRCH-03** | The bot sends a `sendVenue` with the right latitude/longitude/title/address, plus an inline keyboard containing the OPEN button and 2–4 alternates | integration (source-introspection, matching the repo's `commands.test.mjs` CS4 style) | `npm run bot:test` (suite `bot/test/search-command.test.mjs`) | ❌ **Wave 0** — `bot/test/search-command.test.mjs` |
| **SRCH-03** | The venue-card builder's payload is structurally correct (latitude/longitude are finite numbers; title non-empty; address non-empty) | unit | `npm run bot:test` (same suite) | ❌ **Wave 0** |
| **SRCH-03** | The alternates' `callback_data` is `"alt:"`, ≤64 bytes, and carries **no** query text, name, or coordinates | unit (grep + parse) | `npm run bot:test` (same suite) | ❌ **Wave 0** |
| **SRCH-03** | `answerCallbackQuery` is called on **every** callback path (the 30-second rule) | source-introspection | `npm run bot:test` (same suite) | ❌ **Wave 0** |
| **SRCH-03** | A negative query ("Kuching", "Penang") yields an **empty** shortlist and the "not in v1 area" copy — never a fabricated pin | unit + corpus | `node bot/scripts/gazetteer-corpus.mjs` + `npm run bot:test` | ❌ **Wave 0** |
| **SRCH-03** | The search copy exists in **both** en and ms, and the locale key-sets are identical (the Phase 2 CS1 gate, extended) | unit | `npm run bot:test` (extend `bot/test/commands.test.mjs` CS1 or add a new suite) | ✅ **extend** — `bot/test/commands.test.mjs` |
| **ONBD-02** | The webview resolves a `dest` id from its own gazetteer copy to a destination with valid lat/lng | unit | `npm run web:test` (suite `web/test/destination.test.mjs`) | ❌ **Wave 0** — `web/test/destination.test.mjs` |
| **ONBD-02** | An **unknown** id produces an honest "destination unavailable" state, not a blank map | unit | `npm run web:test` (same suite) | ❌ **Wave 0** |
| **ONBD-02** | The bot's gazetteer id set == the web's gazetteer id set (drift gate) | integration (cross-workspace) | `npm run bot:test` (a suite that reads both JSON files) | ❌ **Wave 0** |
| **ONBD-02** | The bot bundle and the web bundle both contain the gazetteer JSON (build-artifact gate) | integration | `npm run web:test` (extend `web/test/build-artifact.test.mjs`) | ✅ **extend** — `web/test/build-artifact.test.mjs` |
| **ONBD-02** | The mock harness can simulate a **preloaded-destination boot** (`?dest=` profile) and the capture run shows the destination state | manual + capture | `npm run web:capture` (extend `web/scripts/capture.mjs`) | ✅ **extend** — `web/scripts/capture.mjs` |
| **ONBD-02** | The deep-link carrier actually delivers the id on a real device (Android + iOS) | **manual-only** | *(no real device in this environment — exit-gate obligation)* | ❌ **exit gate** |
| **ONBD-03** | The Mini App exposes `addToHomeScreen` / `checkHomeScreenStatus` on the `TelegramWebAppLike` interface and calls them behind a capability guard (`isVersionAtLeast("8.0")`) | unit | `npm run web:test` (suite `web/test/homescreen.test.mjs`) | ❌ **Wave 0** — `web/test/homescreen.test.mjs` |
| **ONBD-03** | A startapp-less boot lands in the search-ready map (Phase 3's staged boot, no destination) — i.e. the pre-existing path is unchanged | integration | `npm run web:capture` + `npm run web:test` | ✅ **extend** — existing Phase 3 suites |
| **ONBD-03** | `checkHomeScreenStatus` returning `"unknown"` (iOS) is treated as "supported but indeterminate" — **not** as an error | unit | `npm run web:test` (same suite) | ❌ **Wave 0** |
| **ONBD-03** | Adding araharah to the home screen launches the app into the search-ready state | **manual-only** | *(real device — exit-gate obligation)* | ❌ **exit gate** |
| **D-14** (bake-off close) | The corpus runner exits non-zero when top-3 < 90% **or** any negative produced a hit | integration | `node bot/scripts/gazetteer-corpus.mjs` | ❌ **Wave 0** |
| **D-14** | The corpus file is **hand-authored**, not generated: at least N cases are not string-equal to any alias in the gazetteer (a tautology check) | unit (structural) | `npm run bot:test` (suite `bot/test/gazetteer.test.mjs`) | ❌ **Wave 0** |
| **D-04** (forkability law) | A `LICENSE` file exists at the repo root containing GPL-3.0 | unit (structural) | `npm run bot:test` (a small suite) | ❌ **Wave 0** |
| **D-04 / licensing** | The gazetteer data file carries an ODbL attribution notice, and the repo README/attribution names OpenStreetMap | unit (structural) | `npm run bot:test` (same suite) | ❌ **Wave 0** |
| **D-15** (privacy) | **No** log statement in `bot/src/` writes a search query. Grep gate over `bot/src/**` for log calls whose arguments reference a query-shaped variable | unit (grep, matching the repo's existing source-introspection style) | `npm run bot:test` | ❌ **Wave 0** |
| **D-15** | The web side writes no destination or query to any storage surface | unit (existing grep gate) | `npm run web:test` (`web/test/no-storage.test.mjs`) | ✅ **extend** — `web/test/no-storage.test.mjs` |
| **D-06** (top-up script) | The script produces a valid-shaped proposal file from a **committed fixture** of a real Overpass response | unit (fixture-driven) | `npm run bot:test` (suite `bot/test/gazetteer-topup.test.mjs`) | ❌ **Wave 0** — `bot/test/gazetteer-topup.test.mjs` |
| **D-06** | The script is **not** imported by the Worker (grep: no overpass/OSM URLs under `bot/src/`) | unit (grep) | `npm run bot:test` (same suite) | ❌ **Wave 0** |
| **D-05** (backend #2 reachability) | The JSON→SQLite converter script reads the same committed JSON and emits a SQLite file with the same ids | unit | `npm run bot:test` (suite `bot/test/gazetteer-converter.test.mjs`) | ❌ **Wave 0** |
| **D-07** (V8-native JSON) | The Worker bundle contains the gazetteer as inlined JSON — no SQLite/WASM payload | integration | `npm run bot:test` (extend the build-artifact gate or a bundle-size assertion) | ❌ **Wave 0** |
| **D-03** (SearchSource seam) | The interface has exactly the agreed methods and the gazetteer implements it; a **stub** backend can be substituted in a test (proving the seam) | unit | `npm run bot:test` (suite `bot/test/search-source.test.mjs`) | ❌ **Wave 0** |
### Sampling Rate
- **Per task commit:** `npm run bot:test && npm run web:test` (fast — the hand-rolled runner executes in well under a second) **plus** `npm run bot:typecheck && npm run web:typecheck`.
- **Per wave merge:** the same, **plus** `node bot/scripts/gazetteer-corpus.mjs` (the corpus must be ≥ threshold at every wave merge, not just at the end — this is what makes the escalation ladder actionable rather than a last-minute scramble).
- **Phase gate:** full suite green (`bot:test`, `web:test`, `theme audit`, both typechecks) **and** the corpus runner green, **and** the two manual-only items (ONBD-02 device deep-link, ONBD-03 home-screen launch) verified on a real device, **and** the D-14 decision recorded in the repo.
### Wave 0 Gaps
Everything below is new for this phase (the repo has **no** search-related test infrastructure today).
- [ ] `bot/test/gazetteer.test.mjs` — entry schema, unique ids, alias presence, id-set parity with the web copy, corpus-is-not-tautological, LICENSE + ODbL notice present
- [ ] `bot/test/normalizer.test.mjs` — per-rule test vectors for the full abbreviation table (D-09: *"Per-rule test vectors; extends by table edit, not code change"*), folding, glued-word split, Presint/Prsn
- [ ] `bot/test/distance.test.mjs` — Damerau-Levenshtein (OSA) correctness vectors
- [ ] `bot/test/score.test.mjs` — tier disjointness, min-length guard, exact-outranks-fuzzy invariant
- [ ] `bot/test/search-command.test.mjs` — `sendVenue` payload shape, inline keyboard contents, `callback_data` format/length/no-leakage, `answerCallbackQuery` on every path, negative-query copy
- [ ] `bot/test/search-source.test.mjs` — the `SearchSource` interface + a stub backend proving the seam
- [ ] `bot/test/gazetteer-topup.test.mjs` — the Overpass proposal script against a committed fixture; the no-OSM-URL-in-Worker grep gate; **the `User-Agent` header is asserted to be set** (missing UA → 406 on every endpoint — Pitfall 6b); **502/429/timeout → exit 0 with a "busy" message** (asserted against a fake `fetch`, never the live service)
- [ ] `bot/test/gazetteer-converter.test.mjs` — the D-05 JSON→SQLite converter (id set preserved)
- [ ] `bot/scripts/gazetteer-corpus.mjs` — the D-14 corpus runner (the bake-off close)
- [ ] `bot/test/corpus-gate.test.mjs` (or fold into the above) — asserts `npm run bot:test` fails when the corpus regresses
- [ ] `web/test/destination.test.mjs` — id → destination resolution, unknown-id honest state
- [ ] `web/test/homescreen.test.mjs` — `addToHomeScreen` / `checkHomeScreenStatus` capability guard, `"unknown"` handling
- [ ] `shared/data/golden-corpus.json` — the pinned D-13 corpus (~60–100 hand-written cases)
- [ ] `shared/data/kl-gazetteer.json` — the curated entries (~200–500) with a committed Overpass-response fixture for the top-up script
- [ ] `bot/test/fixtures/overpass-response.json` — a real Overpass JSON response, committed so the top-up test is deterministic
- [ ] **Framework install:** none needed — the repo's hand-rolled runner discovers `*.test.mjs` automatically. **No Jest/Vitest/Mocha.**
- [ ] **Extend (not new):** `bot/test/commands.test.mjs` (CS1 locale parity for the new search strings), `web/test/no-storage.test.mjs` (destination/query non-persistence), `web/test/build-artifact.test.mjs` (gazetteer JSON present in the web bundle)
---
## Security Domain
> `security_enforcement` is absent from `.planning/config.json` → treated as **enabled**. This section is required.
### Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| **V2 Authentication** | **yes (inherited)** | The Mini App's session gate is Phase 2's `bot/src/security/initdata.ts` HMAC validation + `POST /api/session`. This phase **adds no new identity surface**. The destination id is *not* an identity — it is a public place id (D-11 explicitly says no crypto is needed because the payload's only threat is a user pranking themselves). |
| **V3 Session Management** | **yes (inherited, unchanged)** | Phase 2's 900 s memory-only session token. The destination must NOT be written to the session token's storage (there is none) nor to the token's payload (it would leak a user choice into a credential). **Recommendation: the destination lives in module scope in the webview, alongside `lastFix`, exactly like the existing in-memory-only pattern.** |
| **V4 Access Control** | **yes (inherited, unchanged)** | The Worker owns `/api/*`. This phase adds **no new endpoint** — the destination is resolved entirely from the webview's local copy of the shared gazetteer (D-11). That is the *right* design: no endpoint means no access-control surface. If the plan adds an endpoint (e.g. `/api/destination?id=…`), it must derive anything user-specific from the resolved claim, never from the request body — the Phase 2 `handleLocation`/`handleLocationFallback` precedent. |
| **V5 Input Validation** | **yes (new surface)** | The search query is untrusted user input. Rules: (a) the `/cari` handler must bound the query length (e.g. ≤128 chars) **before** any processing; (b) the `callback_data` id must be validated against `/^[a-z0-9-]{1,64}$/` **before** the gazetteer lookup; (c) the webview's `dest` id must be validated with `/^[A-Za-z0-9_-]{1,64}$/` **before** the lookup; (d) the `start_param` charset is officially restricted to `[A-Za-z0-9_-]` and ≤512 chars — validate against exactly that. **Never** interpolate an unvalidated id into a query, a URL, or a log line. |
| **V6 Cryptography** | **no new crypto — by design** | D-11: the payload needs none. The existing HMAC path is unchanged. **Do not introduce a signed-token scheme** for the destination — it would be security theatre against a self-inflicted threat and would break the "no crypto, shared module by construction" design. |
| **V7 Error Handling / Logging** | **yes (binding)** | D-15 + Phase 2's D-14 law: **category-only logs**. A failed search logs `[search] rejected: no-match` — never the query, never the results, never a user id, never a timestamp. The "not in v1 area" copy must be honest (D-13's negatives) and must not fabricate a pin. |
| **V8 Data Protection / Privacy** | **yes (binding)** | The query is user payload and is **not persisted** (D-15). If the plan adds KV result caching (D-15 permits it: TTL-only, short TTL, keyed by *normalized* query), note: the key contains the normalized query — i.e. **the query text is the KV key**, which is arguably persistence of the query. **Recommendation: skip the KV cache entirely.** At ~2500 candidate pairs the search is microseconds; a cache buys nothing and creates a privacy surface D-15 explicitly does not want. (D-15 makes the cache optional: *"if any"*, and OpenCode's Discretion says *"perf is likely fine without at this scale"* — this research confirms that judgment.) |
### Known Threat Patterns for this stack (grammY + Cloudflare Workers + Telegram Mini App)
| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| **Query text persisted in a log or a KV key** | Information Disclosure | D-15 category-only logging; a grep gate over `bot/src/**`; **skip the KV cache** (see V8). |
| **Callback-data injection** (`callback_data` carrying a name/query) | Tampering | `callback_data` is **only** `"alt:"`; the id is regex-validated and looked up in the gazetteer. Never echoed into a reply without validation. |
| **Destination id injection into the webview** | Tampering | `/^[A-Za-z0-9_-]{1,64}$/` before the lookup; an unknown id yields an honest "unavailable" state, never a fabricated coordinate. |
| **Open redirect / unvalidated Mini App URL** | Spoofing | The `web_app` button URL is built from a **configured origin**, never from user input. The plan must not let a query string influence the URL's origin. |
| **Fabricated location pin for an out-of-area query** | Repudiation (honesty failure) | D-13's negatives: an empty result must produce the honest refusal copy, never the nearest KL entry. This is a *correctness and honesty* requirement, not just a UX one — and it is directly asserted by the corpus runner. |
| **Gazetteer id drift between bot and webview** | Integrity | The id-set parity gate (§"Phase Requirements → Test Map") + ids are never renamed, only added. |
| **Overpass fetch inside the Worker** (SSRF-ish + availability) | Denial of Service | The Overpass script is a **repo tool**, never imported by `bot/src/` — enforced by a grep gate. |
| **OSM/ODbL attribution omission** | Compliance | The gazetteer carries an ODbL notice + attribution; the repo README names OpenStreetMap. |
| **`answerCallbackQuery` not called** | Denial of Service (UX spinner) | The repo's existing convention (`bot/src/bot/handlers/location-fallback.ts` pattern) plus a catch-all `bot.on("callback_query:data", …)` that always answers — grammY's documented recommendation. |
| **Malicious `start_param` reaching the webview** | Tampering | `initDataUnsafe` is **untrusted** by Telegram's own docs. Use it only to look up a public place id in a local gazetteer; never as identity, never as a URL, never as a path. |
---
## Sources
### Primary (HIGH confidence)
- **https://core.telegram.org/bots/api/methods/sendvenue.md** — the exact `sendVenue` parameter set. Fetched this session. Confirms: `latitude`, `longitude`, `title`, `address` are **required**; there is **no `text` parameter**; `reply_markup` accepts `InlineKeyboardMarkup`.
- **https://core.telegram.org/bots/api/types/inlinekeyboardbutton.md** — `InlineKeyboardButton`. Fetched this session. Confirms: `callback_data` is **1–64 bytes**; `web_app` is a `WebAppInfo`; `web_app` is *"Available only in private chats between a user and the bot"*; exactly one of the mutually-exclusive fields must be set.
- **https://core.telegram.org/bots/api/types/webappinfo.md** — `WebAppInfo.url` is *"An HTTPS URL of a Web App to be opened with additional data"*. Fetched this session.
- **https://core.telegram.org/bots/api/types/venue.md** — the `Venue` object shape. Fetched this session.
- **https://core.telegram.org/bots/webapps** — Mini Apps launch surfaces, `WebAppInitData` field semantics, `addToHomeScreen()` / `checkHomeScreenStatus()` / `homeScreenAdded` / `homeScreenChecked` (Bot API 8.0+). Fetched this session. **The source of the `start_param` scoping ambiguity** (see §"Deep-link risk").
- **https://docs.telegram-mini-apps.com/platform/start-parameter** — the start-parameter charset (`[A-Za-z0-9_-]`, ≤512 chars) and the regex `/^[\w-]{0,512}$/`. Fetched this session.
- **https://docs.telegram-mini-apps.com/platform/launch-parameters** — `tgWebAppStartParam` is *"a custom string value passed in the bot or application link"*. Fetched this session.
- **https://opendatacommons.org/licenses/odbl/1-0/** — the ODbL 1.0 legal text. Fetched this session. §2.3(a) (programs excluded), §4.2/§4.3 (notices + produced-work attribution), §4.4 (share-alike on a Derivative Database), §4.6 (machine-readable offer).
- **https://osmfoundation.org/wiki/Licence/Attribution_Guidelines** — OSMF's safe-harbour attribution rules. Fetched this session. Confirms the database case: *"include attribution to OpenStreetMap and either the text of the ODbL or a link to it as part of the database, derivative database, or database as part of a collective database… in a location where users would be likely to look for it, such as a readme file, or within the data or metadata."*
- **https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations** and **the JUPEM gazetteer PDFs** (`mygeoname.mygeoportal.gov.my/exportpdf00.jsp?kodnegeri=16`) — the **official Malaysian abbreviation tables** (street prefixes + locality prefixes + the 48-entry gazetteer abbreviation list). Fetched this session. The normative source for the normalizer's rule table.
- **https://raw.githubusercontent.com/osm-search/Nominatim/master/settings/icu-rules/variants-ms.yaml** — the machine-readable transcription of the JUPEM table (shared / MY / SG / BN sections). Fetched this session. **Reproduced verbatim in §"Malay normalizer" via the code example.**
- **https://github.com/osm-search/Nominatim/pull/4056** — the PR that expanded the table from 8 entries to 65+, with the JUPEM + OSM-wiki source citations and the verified test queries ("Jln Bukit Bintang" → "Jalan Bukit Bintang, Kuala Lumpur", "Tmn Tun Dr Ismail" → "Taman Tun Dr Ismail, Kuala Lumpur", …). Fetched this session.
- **https://jupem.gov.my/v1/wp-content/uploads/2016/08/Toponymic-Guidelines-For-Map-and-Other-Editors-for-International-Use.pdf** — JUPEM's Toponymic Guidelines (2017, First Edition), the authority behind the table.
- **https://govdocs.sinarproject.org/documents/ministry-of-international-trade-and-industry/department-of-standards-malaysia/ms-2039-2006** — **MS 2039:2006** (Malaysian address standard). Fetched this session. §5.2.3.2 (the "abbreviation shall not be applied to names" rule) and Annex A (the prefix abbreviation tables A1–A5). **The direct authority for Pitfall 3.**
- **https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL** — Overpass QL reference: `bbox` ordering (s,w,n,e), `nwr`, `out tags`/`out center`, `out count` (the empty-result-detection idiom). Fetched this session.
- **https://grammy.dev/plugins/keyboard** — grammY inline-keyboard + callback-query handling, including the catch-all `bot.on("callback_query:data")` recommendation. Fetched this session.
- **https://www.npmjs.com/package/@crob/damerau-levenshtein` / `npm view` — the distance-function alternatives and their versions. Verified against the registry this session.
### Secondary (MEDIUM confidence)
- **https://core.telegram.org/constructor/keyboardButtonWebView** — the MTProto schema confirming that an inline-button Mini App launch passes `url` to `messages.requestWebView`. Supports the read that the button's URL is used as given.
- **https://github.com/TelegramMessenger/Telegram-iOS/issues/1729** — `checkHomeScreenStatus()` always returns `"unknown"` on iOS; closed with the maintainer comment *"On iOS it is not possible to determine whether a Mini App has been added to the Home Screen."* Confirms the ONBD-03 platform limitation.
- **https://stackoverflow.com/questions/76186178/how-to-pass-parameters-to-telegram-web-app** — community confirmation that query params on a `web_app` button URL reach the webview. Corroborates Carrier A but is **not** official.
- **https://www.postgresql.org/docs/current/pgtrgm.html` + the fuzzy-matching comparison articles (tektite, fuzzypoint, taxify vignette) — the trigram/Jaro-Winkler/Damerau trade-off analysis underpinning the tier design and Pitfalls 4–5.
- **https://github.com/cbeier-studio/xolmis** — the live precedent for a GPL-3.0 project shipping ODbL data in the same repo with an explicit dual-license note.
- **https://github.com/ahozir-ux/bm-address-normaliser` — an existing Malay address normaliser (JS + Python + a Cloudflare Worker), useful as a cross-check on the abbreviation set and on the `KG`-ambiguity handling. Community-maintained; treat as a lead, not an authority.
- **Real commercial Malaysian address listings** (Maybank2U merchant PDF, OMRON Healthcare retailer PDF, Petronas station lists, Malaysia.worldplaces, the myboost KL outlet PDF) — the empirical evidence for **Presint/Prsn** usage ("JALAN P8G PRESINT 8", "PRSN RESIDEN DESA PARKCI", "Precint 15", "PRESINT 16") and for the everyday abbreviation forms locals actually type.
### Tertiary (LOW confidence — flagged for validation)
- **Stack Overflow #76486463** — an older (2023) claim that a start param is *"limited to 64 Symbols"*. The official docs now say **512**. Use 512. Flagged because the stale figure could mislead.
- **Community assertions that `start_param` is populated for direct-link launches** — the official `WebAppInitData.start_param` wording is narrower than the launch-parameter docs. **This is the single most consequential uncertainty in the phase** (see Assumptions A1/A2 and Open Question 1). Validate on a real device before the phase gate.
---
## Metadata
**Confidence breakdown:**
- **Standard stack:** HIGH — every version verified against the npm registry this session (`grammy` 1.46.0, `@crob/damerau-levenshtein` 2.0.0, `fastest-levenshtein` 1.0.16, `damerau-levenshtein` 1.0.8); the repo's own toolchain confirmed by reading `package.json` files and running the probes.
- **Architecture:** HIGH for the Telegram surface (official `core.telegram.org` type pages fetched this session); **MEDIUM** for the deep-link carrier (documented ambiguously, community-corroborated only — see Open Question 1).
- **Malay normalizer:** HIGH — the JUPEM tables, MS 2039, and the Nominatim `variants-ms.yaml` transcription were all fetched verbatim this session, and the rule table is reproduced directly from a machine-readable upstream file rather than recalled.
- **Licensing:** HIGH for the ODbL text and the OSMF guidelines (both fetched this session); **MEDIUM** for the Derivative-Database characterization of a curated gazetteer (a legal judgment this research can inform but not make).
- **Pitfalls:** HIGH — each is anchored to an official doc statement or a verified probe. **Corrected by the verification pass:** the 406 on `overpass-api.de` was initially attributed to an "egress block"; a controlled same-host probe varying only the `User-Agent` header showed it is the **missing UA** (with-UA → 200 with real data). The 502-capacity behaviour and the unreachable mirrors are as observed.
- **Environment availability:** HIGH — every row was probed this session, not assumed.
**Research date:** 2026-10-23
**Valid until:** 2026-11-06 (30 days) for the stack/architecture/licensing; **2026-10-30 (7 days)** for the Overpass-egress finding — network filters change, and the mirror landscape is volatile. Re-probe before the phase's execution begins.
---
## Appendix: Independent Verification Pass (2026-10-23)
> A second, independent research pass re-verified this document's load-bearing claims against fresh fetches, a live Overpass re-probe, and a from-scratch reimplementation of the CPU benchmark. **No claim in the body was contradicted.** Two claims were **strengthened**, one was **corrected**, and one was **refined**.
### A1. CPU benchmark — independently reproduced ✅
The body's central engineering constraint (naive full-scan Damerau ≈ 8.16 ms/query against a 10 ms Workers Free limit) was reimplemented from scratch — separate OSA-Damerau implementation, separate trigram inverted index, synthetic 500-entry / 2 000-alias gazetteer, Node v24.18.0:
| Implementation | This pass | Research body | Verdict |
|---|---|---|---|
| Naive full-scan Damerau (2 000 aliases) | **8.243 ms/query** | 8.16 ms/query | **REPRODUCED** (±1%) |
| Trigram inverted index | **0.058 ms/query** | 0.004 ms/query | Same order of magnitude; this pass's index was not micro-optimized (it scores every candidate rather than short-circuiting on the top-k). The **conclusion is identical**: the indexed path is ~140× faster and sits ~170× inside the CPU budget. |
**Implication for the planner: unchanged and reinforced.** A linear scan is a latent `Error 1102` (Workers error 1102 = "Worker exceeded resource limits"). The **trigram inverted index is not an optimization — it is a correctness requirement** at D-07's stated 200–500 entry scale.
### A2. The deep-link landmine — confirmed at the MTProto layer ✅
The body's highest-value finding (a `web_app` inline button **cannot** populate `start_param`) was re-verified by fetching the authoritative method page:
> — `start_param`: *"If the web app was opened from the attachment menu using a attachment menu deep link, `start_param` should contain the `data` from the `startattach` parameter."*
The method **has** a `start_param` field (flags.3), but the official description restricts its population to the **attachment-menu** path. The `url` field (flags.1) is the "[Web app URL](https://core.telegram.org/api/bots/webapps.md)" and is passed through verbatim — which is why the button's own query string reaches `window.location.search` but never the signed `initData`.
The sibling method confirms the *other* side: `start_param`: *"If the `startapp` query string parameter is present in the direct Mini App deep link, pass it to `start_param`."*
**Implication for the planner: unchanged and reinforced.** The body's Design A/B/C analysis stands. The body's own caveat (§Tertiary sources) that the community consensus on `start_param` population is "narrower than the launch-parameter docs" is **resolved at the protocol level** for the *carrier* question — but note the residual uncertainty is now narrower than the body states: it is no longer "does `start_param` get populated at all for direct links" (it does, per `requestAppWebView`), but only **which BotFather configuration the project actually has** (Main Mini App vs Direct Mini App vs attachment menu), because that determines whether a `t.me` link prompts for confirmation. See §Open Questions.
### A3. "KLCC" / "DUKE" have no usable OSM landmark name — reproduced ✅
A live Overpass probe this pass (`nwr["name"~"KLCC",i](3.05,101.55,3.20,101.75)`; and `nwr["short_name"="DUKE"]`) returned **HTTP 200** with `timestamp_osm_base: 2026-10-23T14:55:20Z` and reproduced the body's finding exactly:
- **"DUKE":** every match is a `highway=motorway` segment of `Lebuhraya DUKE` (`short_name=DUKE`, `alt_name="Karak Link"`, `official_name="Lebuhraya Duta–Ulu Klang"`, `ref=E33`). **38 of 40 returned elements were motorway segments.** "DUKE" is a highway acronym, not a destination.
- **"KLCC" as a bare name:** matches are dominated by `public_transport=stop_position` / `railway=stop` (LRT), `highway=bus_stop`, `amenity=parking_entrance`, plus hostels and show galleries — **not** the landmark.
- **The real landmark IS present, under a different name:** `way 23034336` `name="Suria KLCC"`, `building:use=retail`, `name:en="Suria KLCC"`, with `addr:street="Jalan Ampang"`, `addr:city="Kuala Lumpur"`, `addr:postcode="50088"` on nearby nodes — i.e. **`addr:*` tags are available for the `sendVenue` `address` field** (Pitfall 11).
**Implication for the planner: unchanged and reinforced.** The body's conclusion — *the corpus cannot be auto-derived from OSM names; the alias list IS the product* — is now verified at the level of individual element tags. **The seed gazetteer's `name` field must be the landmark's canonical OSM name ("Suria KLCC"), and "KLCC" must be an explicit `aliases` entry (D-08), never a name.** The Overpass script proposes OSM-named candidates; a human adds the local shorthand.
### A4. Overpass availability — **CORRECTED** (the body reproduced the symptom but misattributed the cause)
> ⚠️ **This section supersedes the body's §Environment Availability row and its §Sources "egress filter" framing.** The body concluded that `overpass-api.de` was **unusable** — "HTTP 406 from an nginx egress filter on every request, incl. `GET /`" — and marked it ✗. The symptom was reproduced; **the cause was misattributed.** A controlled probe of the *same host, same paths, same session*, varying **only the `User-Agent` header**, resolves it:
| Request | No `User-Agent` | With `User-Agent` |
|---|---|---|
| `GET https://overpass-api.de/api/status` | **HTTP 406** | **HTTP 200** |
| `GET https://overpass-api.de/` | **HTTP 406** | **HTTP 200** |
| `POST https://overpass-api.de/api/interpreter` (KL bbox query) | **HTTP 406** | **HTTP 200** — 19 056 bytes, `osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z` |
**The 406 is caused by the missing `User-Agent`, not by an egress filter.** With the header, the primary instance works and returns real, current KL data.
Reachability of the alternatives, probed the same way:
| Instance | No UA | With UA | Verdict |
|---|---|---|---|
| `overpass-api.de` | 406 | **200** | ✅ **the working one** |
| `overpass.kumi.systems` | 000 | 000 | ❌ unreachable from this network (connection failure — HTTP 000 is not an HTTP status) |
| `overpass.private.coffee` | 000 | 000 | ❌ unreachable from this network |
| `overpass.osm.ch` | 400 | 400 | ❌ Switzerland-only coverage — confirms the body's note |
**Capacity is a separate, intermittent phenomenon.** Earlier in this same session, `overpass-api.de`, `overpass.kumi.systems` **and** `overpass.private.coffee` all returned **HTTP 502** with the body *"This Overpass API server is currently at capacity. All query workers are busy and the request queue is full."* Minutes later the same query on `overpass-api.de` returned **200**. The 502 is a **queue-saturation** response to a narrow time window, not a property of the host.
**Implications for the planner (these supersede the body's row):**
1. **`overpass-api.de` IS usable** — with a `User-Agent`. The body's "✗ / egress filter" verdict was wrong and, if carried into the plan, would have produced needless `--source=`/local-extract complexity or skipped the D-06 top-up entirely. **Do not plan around a blocked Overpass.**
2. **The `User-Agent` header is load-bearing**, not a nicety. Without it, *every* endpoint returns 406. It should be the script's first line of defence.
3. **The other two public instances are unreachable from this network** — so "primary + fallback instance" is **not** a real safety net here. The **committed fixture** (already in the body's Wave 0 list) is the reliable path for automation, and one live endpoint plus that fixture is the honest architecture.
4. **502/429/timeout → print "busy, retry later" and exit 0.** Never a stack trace, never a non-zero exit. (Now written up as §Pitfall 6b.)
5. **Prefer a narrow, single-tag-family query** by default — a narrow query is far more likely to fit in a free worker slot during a saturation window.
6. **Nothing may depend on this script** — no deploy, no test run, no corpus gate. The body already says this; this pass confirms it is load-bearing in practice.
### A5. `start_param` length limit — **noted as a secondary path only**
The body cites **512** characters (the official Bot API community-mirror figure) and explicitly flags the older "64 symbols" Stack Overflow claim as **stale** in §Tertiary sources. This pass observed that the **MTProto** layer's `platform` sibling field is documented as *"0-64 English letters, digits, and underscores"*, while the `start_param` field itself carries no length constraint in the method schema — so 512 (the Bot API figure) is the number to design against, exactly as the body says.
**Implication for the planner: unchanged.** The body's recommendation to constrain ids to `/^[A-Za-z0-9_-]{1,64}$/` is *stricter* than 512 and therefore safe. The one thing to note: this constraint only binds the **secondary** `start_param` path. The body's **primary** path (Design B — `?dest=` in the Mini App URL's own query string, read from `location.search`) has **no equivalent length cap**, which is a further reason to prefer it.
### A6. JSON import assertions — the body's "test this early" risk is **lower** than stated
The body warns that the shared search module must be importable from plain Node and flags the JSON import as a possible blocker. This pass probed it: `node -e "import('./package.json', {with:{type:'json'}})…"` on Node v24.18.0 → **succeeds**. The remaining risk is therefore **not** the Node import path but the **Workers bundler** path (esbuild via wrangler) — the JSON must be *inlined* into the bundle, not fetched at runtime (which would be a hard failure in a Worker). **One `wrangler deploy --dry-run` in Wave 0 settles it.**
### A7. Confirmed as-is (no change needed)
- `sendVenue` accepts `reply_markup` — re-fetched ; the official parameter table lists `reply_markup` with `InlineKeyboardMarkup` first. **D-02's one-message shape is supported.**
- `WebAppInfo` has exactly one field — re-fetched ; the table has a single row (`url`). **No `start_param` field.**
- Workers Free limits — re-fetched (page dated 2026-10-12): 10 ms CPU/request, 100 000 req/day, 128 MB, 1 s startup, ~2.2 ms average, "avoid expensive work in global scope".
- The ODbL Derivative-Database analysis — the Trivial Transformations Guideline's geocoding clause re-fetched and unchanged.
- `bot/scripts/set-menu-button.mjs` **exists** (Pitfall 12's remedy is real).
- npm versions unchanged: `grammy` 1.46.0, `fastest-levenshtein` 1.0.16, `damerau-levenshtein` 1.0.8, `string-similarity` 4.0.4, `levenshtein-edit-distance` 3.0.1 — and additionally probed `jaro-winkler` 0.2.8, `trigram-similarity` 1.0.7, `didyoumean` 1.2.2, `fuzzysort` 4.0.2, `fuse.js` 7.5.0. **Every dedicated string-distance package except `fuse.js`/`fuzzysort` is 2022-vintage and unmaintained** — which reinforces the body's "hand-roll, zero dependency" call for a GPL-3.0 forkability-law project.
### Verification ledger
| Body claim | Verdict | Evidence this pass |
|---|---|---|
| Naive full-scan Damerau ≈ 8.16 ms/query vs a 10 ms limit | ✅ **REPRODUCED** (8.243 ms) | From-scratch reimplementation, 500 entries / 2 000 aliases, Node 24.18.0 |
| Trigram inverted index ≈ 0.004 ms/query | ✅ **REPRODUCED in order of magnitude** (0.058 ms) | Same run; conclusion (≈140× faster, ~170× inside budget) unchanged |
| `web_app` inline button cannot populate `start_param` | ✅ **CONFIRMED** | `messages.requestWebView` official page: `start_param` is populated *only* from an attachment-menu `startattach` link |
| "KLCC" has no OSM landmark name; "DUKE" is a highway acronym | ✅ **CONFIRMED** | Live Overpass (HTTP 200, `osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z`): 38/40 DUKE hits are `highway=motorway`; "Suria KLCC" is `way 23034336` with `addr:*` tags |
| Overpass requires a User-Agent (406 without) | ✅ **CONFIRMED** | `curl https://overpass-api.de/api/status` (no UA) → HTTP 406 |
| Overpass availability | ⚠️ **CORRECTED** — usable with a `User-Agent`; the 406 is a missing-UA rejection, not an egress block. Capacity 502s are intermittent. | Same-host/same-path probe varying only the UA: no-UA → 406 on all three paths, with-UA → 200 with real KL data. `kumi.systems` / `private.coffee` unreachable (000); `osm.ch` → 400 |
| `sendVenue` accepts `reply_markup` | ✅ **CONFIRMED** | Official method page parameter table |
| `WebAppInfo` = `{ url }` only | ✅ **CONFIRMED** | Official type page (single row) |
| Workers Free 10 ms CPU / 1 s startup / 128 MB | ✅ **CONFIRMED** | Official limits page, dated 2026-10-12 |
| ODbL Derivative-Database obligation | ✅ **CONFIRMED** | Trivial Transformations Guideline (geocoding clause) |
| `start_param` = 512 chars | ✅ **STANDS** (community-mirror figure; MTProto documents no length on the field itself) | Body's stricter `/^[A-Za-z0-9_-]{1,64}$/` id constraint is safe either way |
| JSON import assertions in the Node runner | ✅ **WORKS** (risk is the Workers bundler, not Node) | `node -e "import('./package.json', {with:{type:'json'}})"` → OK on Node v24.18.0 |
| `bot/scripts/set-menu-button.mjs` exists | ✅ **CONFIRMED** | `ls bot/scripts/` → `register-commands.mjs`, `set-menu-button.mjs`, `set-webhook.mjs` |
**Bottom line for the planner: this document's findings stand as written**, with **one correction** (the Overpass availability verdict — see §A4) and **one addition** (§Pitfall 6b: the `User-Agent` is load-bearing and 502s are normal). The deep-link carrier decision, the indexed-lookup requirement, the alias-led corpus strategy, the `sendVenue` one-message shape, and the ODbL dual-licence are all verified and ready to plan against.
**The three things the planner must not get wrong:**
1. **Use a trigram inverted index, not a linear scan.** Reproduced at 8.24 ms/query against a 10 ms limit — a linear scan is a latent `Error 1102`.
2. **Do not ship D-10's `web_app` button as written.** It cannot carry a `start_param`. Choose Design A (`url` button → `t.me` Main Mini App link, `start_param` populated and HMAC-signed) or Design B (`web_app` button → `?dest=` in the URL, read from `location.search`), or Design C (both). This is the one decision that most changes the phase's shape.
3. **"KLCC" and "DUKE" are not OSM landmark names.** 38/40 live "DUKE" hits are `highway=motorway` segments. The gazetteer's `name` must be the canonical OSM name (`Suria KLCC`); the local shorthand is an **`aliases`** entry. The alias list is the product, and it cannot be generated.