c-fu/arah2
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7994c84a40a1f86c9635d91ce721e70a7861b957
arah2/.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md
T
araharah-executor 7994c84a40 docs(04): research phase domain — destination search + KL gazetteer 
Phase 4 research: in-Worker SearchSource over a curated KL gazetteer.

Two critical platform findings:
- Telegram start_param is NEVER delivered to inline web_app buttons
  (direct links + attachment menu only). The destination id must ride
  as a URL query param on the button's URL; D-11's substance preserved.
- beforeinstallprompt never fires in a WebView. ONBD-03 must use
  Telegram's native WebApp.addToHomeScreen(); iOS status is always
  "unknown" by platform design.

Benchmarked the CPU budget: naive full-corpus Damerau is 52ms at
D-07's max scale vs the 10ms Workers Free limit. Exact-hash-first +
token-prefiltered residual is 0.0003ms.

Also: Phase 2 CS3/CS4/CS5 gates need declared amendments; ODbL
attribution obligation for the OSM-derived gazetteer.
2026-10-23 22:50:13 +08:00

1165 lines
112 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Phase 4: Destination Search + KL Gazetteer - Research
**Researched:** 2026-10-23
**Domain:** In-Worker fuzzy place search over a curated KL gazetteer, Malay query normalization, Telegram venue-card + Mini App deep-link hand-off, OSM-derived gazetteer tooling and licensing
**Confidence:** HIGH for the Telegram Bot API surface (official docs + installed grammY 1.46.0 typings read this session), HIGH for Cloudflare Workers limits (official docs fetched), HIGH for OSM/ODbL licensing analysis (official OSMF wiki fetched), MEDIUM-HIGH for the Malay abbreviation table (UPU address standard + OSM road lists + Malaysian postal examples — all real but not a complete KL corpus), MEDIUM for the scoring-function recommendation (algorithm properties are cited; the exact weights must be settled empirically by the corpus runner)
<user_constraints>
## User Constraints (from CONTEXT.md)
### Locked Decisions
**Search interaction surface**
- **D-01:** Search entry is the **`/cari` command** in chat (with `/pergi_ke` reserved for Phase 5 semantics; bilingual alias scheme per Phase 2 D-05). Free-text-to-bot stays reserved; no inline mode; no in-webview search UI this phase (a shared search module keeps that door open).
- **D-02:** Shortlist presentation: the **best result ships immediately** as the venue card with its OPEN button; **2–4 alternates ride as inline buttons on the same message**. One tap to proceed, one tap to switch — no second message round.
- **D-03:** Search backend is a **pluggable `SearchSource` interface implemented in-Worker**: curated KL gazetteer queried in-Worker, $0 infrastructure. **VPS/Photon provisioning defers to Phase 5** (Valhalla's road graph needs the VPS anyway). Photon remains a documented escalation path *behind the interface*.
**Forkability + license (user-surfaced design law)**
- **D-04:** **GPL-3.0** is the project license. Forkability laws are binding from this phase: no hardcoded secrets (env only — already true), data as plain committed files, one-command deploy, one free-tier Cloudflare account per fork.
- **D-05:** The user's **Malaysia-only WASM geocoder** vision (tantivy/sqlite-trigram index, statically hosted, client-side webview search — a genuine privacy win) is captured as **SearchSource backend #2**, an OSS-track artifact *after* v1. The JSON↔SQLite converter script keeps its dataset format reachable from day one. **Backend #3 = Photon-on-VPS.** Neither ships in this phase.
**Gazetteer + Malay normalizer**
- **D-06:** Entries are **hand-curated seed + a repo-script Overpass one-shot** that proposes top-up candidates for maintainer approval into the curated file. The script is a repo tool, not a runtime dependency.
- **D-07:** Canonical format is **typed JSON in the repo** (V8-native parse, no WASM in the Worker bundle, diff-friendly PRs). Rationale: at ~200–500 rows, SQLite/sql.js-in-Worker is ~1.5MB + cold-start overhead of pure cost.
- **D-08:** **Aliases live natively inside each entry** (`"KLCC"` → Suria KLCC, KLCC Park; IJN, PWTC, Stesen/Station…) — locals' shorthand is the main corpus quality lever.
- **D-09:** The Malay normalizer is a **rule table + folding + fuzzy-residual** module: bidirectional abbreviation table (Jln↔Jalan, Tmn↔Taman, Kg↔Kampung, vowel-drop-aware Presint/Prsn forms), case/whitespace folding, glued-word split — applied to BOTH the query and entry aliases at index time. Fuzzy scoring (Damerau) is only the residual net for genuine typos on longer words. Per-rule test vectors; extends by table edit, not code change.
**Venue card + deep-link hand-off**
- **D-10:** The venue card is **Bot API `sendVenue`** (native pin/name/address semantics SRCH-03 names) with inline buttons: OPEN IN ARAHARAH (`web_app` button carrying `startapp`) + alternates.
- **D-11:** The deep link carries a **plain gazetteer entry id** (`?startapp=dest_<id>`). The webview maps id → destination from its own copy of the same gazetteer module (shared module, build-time). No crypto: the payload's only threat is a user pranking themselves; stale-webview datasets re-map via the shared module version.
- **D-12:** ONBD-03 scope: **minimal PWA hooks** — `addToHomeScreen` prompt handling + a startapp-less boot landing in the search-ready map (existing Phase 3 staged boot covers it). Full PWA manifest/offline story is documented-deferred.
**Acceptance corpus + thresholds (the bake-off close)**
- **D-13:** A **pinned golden corpus** in the repo (~60–100 queries): golden landmarks (KLCC, Sunway Pyramid, DUKE), abbreviation forms, typo set (transpositions, missing keys), glued words, aliases, kampung-style names, plus **deliberate negatives** (out-of-KL: "Kuching", "Penang") that must get an honest "not in v1 area" answer, never a fabricated pin. Each positive pins its expected entry id. Reviewers extend by editing one file.
- **D-14:** The bake-off closes on: **top-3 contains the expected id for ≥90% of positives AND negatives are refused honestly**. Below threshold, escalation order is fixed: (1) alias/normalizer table edits → re-run; (2) SearchSource backend #2 (WASM geocoder); (3) Photon-on-VPS. The phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question.
**Privacy (carried + made specific)**
- **D-15:** Search queries are user payload: **category-only logging** (Phase 2 law) — no query text persisted. KV caching of search results, if any, is TTL-only with short TTL and keyed by normalized query (content is public place data, never user identity). Venue choices are not stored.
### OpenCode's Discretion
- Scoring function shape (weighted trigram/Jaro vs Damerau) within the rule-table architecture — corpus decides empirically
- Alternates ranking + tie-breaking details; "not in v1 area" copy (en/ms, clinical tone)
- Overpass query shape for the top-up script; gazetteer entry schema fields beyond (id, name, aliases, lat, lng, category)
- D1/KV TTL result-cache mechanics or none (perf is likely fine without at this scale)
### Deferred Ideas (OUT OF SCOPE)
- Malaysia-only WASM geocoder (tantivy/sqlite-trigram), statically hosted, client-side webview search — SearchSource backend #2, OSS track after v1
- Photon-on-VPS — SearchSource backend #3; VPS provisioning happens in Phase 5 for Valhalla regardless
- In-webview search UI — needs a UI-SPEC round; shared module already keeps it cheap
- Full PWA (manifest, offline shell) — documented-deferred beyond minimal ONBD-03 hooks
- Free-text bot search + inline mode — reserved interaction surfaces
- Log-derived corpus (real queries) — becomes possible only post-launch (Phase 7+)
</user_constraints>
<phase_requirements>
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|------------------|
| **SRCH-01** | User can type a destination query (e.g. "KLCC") and receive a shortlist containing the right KL-metro landmark/POI | §"The `SearchSource` seam" (pluggable interface + in-Worker impl), §"Scoring architecture — three tiers", §"Golden corpus + threshold gate" (the executable ≥90% top-3 gate). Gazetteer entry schema in §"Gazetteer entry schema" and seed rows in §"Seed gazetteer shape" |
| **SRCH-02** | User gets useful results despite abbreviations and typos (Jln↔Jalan, Tmn↔Taman, missing spaces, common Malay address shorthand) | §"Malay normalizer — rule table" (the full bidirectional abbreviation table with authoritative UPU + OSM grounding), §"Glued-word splitting", §"Scoring architecture — three tiers" (alias/canonical exact-match tier precedes fuzzy). §"Normalizer test vectors" gives per-rule vectors |
| **SRCH-03** | For a chosen destination, a native venue card is sent in chat, confirming where we're going before navigation opens | §"`sendVenue` — what the card can and cannot carry" (verified against installed grammY 1.46.0 + @grammyjs/types: `reply_markup` IS supported on `sendVenue`, so D-02's single-message shape works), §"Four-square type mapping" |
| **ONBD-02** | User can open the Mini App from a search result with their destination preloaded (no re-entry in the webview) | §"Deep-link mechanics" (the `web_app` inline button URL vs `t.me?startapp=` direct link — with the **Desktop START-button trap** documented as a pitfall), §"Reading `start_param` in the webview" (validated `initData` field, character restrictions), §"Harness evidence path" |
| **ONBD-03** | User can add araharah to their phone home screen and launch search/navigation from there (addToHomeScreen + startapp deep-link) | §"ONBD-03 — what `addToHomeScreen` actually is" (Bot API 8.0 `addToHomeScreen()` + `checkHomeScreenStatus()`; **iOS always returns `unknown`** — verified against Telegram-iOS issue #1729), §"Home-screen launch has no `start_param`" (the startapp-less boot path) |
</phase_requirements>
---
## Summary
Phase 4 is small in code and large in judgement. The locked decisions (D-01…D-15) have already made the two hardest architectural calls: the backend is an in-Worker `SearchSource` over a curated KL gazetteer (no VPS, no Photon, $0 infra), and the interaction surface is the `/cari` command with a native venue card. What remains for research to settle is the *quality machinery* — the Malay normalizer, the scoring function, the corpus format, and the exact Telegram payload shapes — plus three things the roadmap never mentioned but that will bite the planner:
**1. The `web_app` inline button and the `t.me?startapp=` direct link are NOT equivalent, and on Telegram Desktop the direct link needs a second tap.** D-10 says the OPEN button is a `web_app` button "carrying `startapp`". Verified from the installed `grammy@1.46.0` / `@grammyjs/types` typings: `InlineKeyboardButton.web_app` is a `WebAppInfo` whose **only** field is `url: string` — an HTTPS URL of the Mini App itself. There is **no `start_param` field on `WebAppInfo`**. The `startapp` query parameter is a **deep-link** construct (`https://t.me/<bot>?startapp=dest_<id>`), and Telegram only propagates it to the Mini App when the Mini App is opened *that* way. So the OPEN button's URL must be either (a) the Mini App origin with the parameter appended to the *webview URL's own query string* — which does **not** become `start_param` and must be read from `location.search` instead — or (b) a `t.me` deep link. Option (b) has a documented trap: [telegramdesktop/tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694) reports the user "must manually press the 'START' button for the deeplink to be processed" on Telegram Desktop for Windows/Linux. **Recommendation: use a `url` inline button (not a `web_app` button) pointing at the Mini App origin with `?dest=<id>` in the query string**, read it from `location.search` in the webview, and fall back to `start_param` when present. This works identically on every client, has no START-button trap, and keeps D-11's "plain gazetteer entry id" payload. The planner must make this call explicitly; it is not free.
**2. A curated OSM-derived gazetteer is a legal "Derivative Database" under ODbL, and that is in tension with the GPL-3.0 project license (D-04).** This is the single most consequential finding of this research. The OSMF's own [Trivial Transformations Guideline](https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline) says explicitly: *"A 'geocoding database' should **NOT** be considered a trivial transformation… Geocoding is being considered for a separate community guideline."* The Use Cases page says: *"if you have any data that was derived from OSM — for instance because you used street names from OSM… then you are making a derivative database."* A curated gazetteer of KL POIs with names, coordinates and addresses extracted from OSM and committed as a public JSON file **is** a derivative database: ODbL §4.4 requires it be made available under ODbL or a compatible licence. GPL-3.0 is a **software** licence and is **not** a compatible database licence for ODbL share-alike — the OSMF's [Licence Compatibility](https://foundation.openstreetmap.org/wiki/Licence/Licence_Compatibility) page lists licences requiring users to use the data on *identical* terms as generally incompatible. **Recommendation: dual-license the gazetteer data file itself under ODbL-1.0 (with `© OpenStreetMap contributors` attribution) while keeping the *code* GPL-3.0.** This is exactly how Photon, Nominatim-adjacent tools, and OSM-derived routing datasets are shipped, and it satisfies D-04's forkability law (the data file stays a plain committed file; a fork inherits the ODbL obligation on that file, which is the correct and standard outcome). The planner must also ensure the "OPEN IN ARAHARAH" venue card / bot reply carries the OSM attribution string in both locales — the web app already ships `attribution: "© OpenStreetMap contributors"` in `web/src/i18n/strings.ts`, so the bot side needs the same.
**3. The OSM licence problem is avoidable for the seed data, and that changes the Overpass script's job.** The D-06 Overpass script is a *proposal generator*, not the data pipeline: its output is a human-reviewed candidate list, and the curated file is maintained by hand. If the maintainer **writes the address strings and picks the aliases by hand** from local knowledge (and only copies the *coordinates* from OSM, treating them as facts rather than as a database extraction), the curated file's derivative-database exposure drops sharply — but "substantial" is a judgement call the planner should not make unilaterally. **The safe, standard, and cheap answer is the dual-licence above**; it costs one `LICENSE` file, one attribution line, and one sentence in the README. D-04 explicitly asks for a GPL-3.0 `LICENSE` file this phase — the planner should add the ODbL notice alongside it for the data file.
**4. Workers Free gives 10 ms of CPU per request.** [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/) (fetched 2026-10-23): "CPU time per HTTP request | **10 ms**", 100,000 requests/day, 128 MB memory, 1 s startup. The average Worker uses ~2.2 ms. A gazetteer of 200–500 rows with, say, 8 aliases per row yields 1,600–4,000 folded alias strings. A naive "compute Damerau-Levenshtein against every alias" pass is O(4,000) × O(len²) — comfortably inside 10 ms only if the candidate set is first narrowed. **Recommendation: index-at-load alias folding with an exact-match hash lookup first (O(1)), then a cheap character-bigram Dice filter to cut the candidate set to a handful, then Damerau-Levenshtein only on that handful.** This is the classic three-tier cascade and it also gives the "honest refusal" behaviour D-14 demands for free: a query that produces no candidate above threshold simply yields zero results. Also note the **1 s Worker startup limit** and the "avoid expensive work in global scope" guidance — the gazetteer should be parsed **inside the request handler** (or at least lazily), not at module top level, or a 500-row JSON parse + fold runs on every cold start.
**5. `sendVenue` supports `reply_markup`, so D-02's one-message shape works.** Verified directly in the installed typings (`node_modules/@grammyjs/types/methods.d.ts`, `sendVenue` args include `reply_markup?: InlineKeyboardMarkup | …`). So the venue card and its OPEN + alternates buttons ship as ONE message. That satisfies D-02 exactly ("no second message round"). The `Venue` type carries `location`, `title`, `address`, plus optional `foursquare_id`/`foursquare_type`/`google_place_id`/`google_place_type` — the four-square type is the free, self-hostable choice (no Google API key needed), and the Foursquare ID can be the gazetteer's own OSM-derived identifier.
**6. ONBD-03's `addToHomeScreen` is a Telegram-native shortcut, not a web PWA install, and iOS cannot report its status.** `WebApp.addToHomeScreen()` and `WebApp.checkHomeScreenStatus(callback)` are Bot API 8.0+ methods ([core.telegram.org/bots/webapps](https://core.telegram.org/bots/webapps)), returning one of `unsupported` / `unknown` / `added` / `missed`. [Telegram-iOS issue #1729](https://github.com/TelegramMessenger/Telegram-iOS/issues/1729) confirms: *"Home screen status detection always returns 'unknown' on iOS… On iOS it is not possible to determine whether a Mini App has been added to the Home Screen. The platform does not provide any API that allows Telegram to check whether a Home Screen shortcut exists."* The `beforeinstallprompt` event that a normal web PWA uses **does not exist inside a Telegram webview** — there is no PWA install prompt. So D-12's "minimal PWA hooks" means: call `addToHomeScreen()`, handle `homeScreenAdded`, and treat `unknown` as "possibly added" (iOS). A home-screen shortcut launched by the OS **carries no `start_param`** — it opens the Mini App cold, which is why D-12 requires the startapp-less boot to land in the "search-ready map". That path already exists from Phase 3 (the staged boot), so ONBD-03 is mostly a *verification* obligation plus a small hook.
**Primary recommendation:** Ship this phase as three thin, separately-testable artifacts — a pure `SearchSource` + normalizer + scorer module (no I/O, no Telegram, no DOM; fully exercisable by the corpus runner), a `sendVenue`-based `/cari` handler that talks to it, and a webview boot branch that reads the destination id from `location.search`/`start_param` — then let the pinned golden corpus be the *executable* definition of the ≥90% gate, with the escalation ladder (normalizer table edits → backend #2 → backend #3) written as a documented decision record in the phase directory rather than as an open question.
---
## Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| Query normalization (Malay abbreviation folding, case/whitespace, glued-word split) | API / Backend (Worker) | Frontend Server (browser runtime) | D-09 says the normalizer is applied to *both* query and entry aliases at index time — the canonical copy must be the shared module. It is pure string logic with no I/O, so it runs identically in the Worker (bot) and in the browser (webview copy at build time). One implementation, two consumers. |
| Gazetteer storage + alias indexing | API / Backend (Worker) | Database / Storage (repo file) | D-07: the canonical store is a typed JSON file in the repo; the Worker parses it at request time. The webview gets a build-time copy of the same module. No database tier exists in v1 (KV is TTL-only session state). |
| Fuzzy scoring / ranking / shortlist | API / Backend (Worker) | — | Scoring runs server-side so the corpus runner and the production path exercise *the same code*. If scoring lived in the browser, the corpus runner would have to duplicate it and the bake-off would measure the wrong thing. |
| `/cari` command intake + reply shaping | API / Backend (Worker) | — | The command arrives as a Telegram webhook update; only the Worker can read it. |
| Venue card rendering (name/address/coordinates) | API / Backend (Worker) → Telegram client | — | `sendVenue` is a Bot API call. The card is rendered by Telegram, not by araharah. |
| Deep-link construction (`dest_<id>` payload) | API / Backend (Worker) | — | The Worker knows the entry id; it composes the Mini App URL. |
| Destination-id → destination resolution in the webview | Frontend Server (browser runtime) | API / Backend | D-11: the webview maps id → destination from *its own* build-time copy of the gazetteer module. No API round-trip is needed — the id is a plain key into a file the bundle already carries. |
| Bot menu registration for `/cari` | API / Backend (repo script) | — | `bot/scripts/register-commands.mjs` derives menu descriptions from `strings.helpMenu` — the Phase-2 pattern. `/cari` must be added to `helpMenu` in `bot/src/i18n/strings.ts` or it will not appear in the menu (Phase-2 CS5 gap-freedom gate). |
| Result caching | Database / Storage (KV, TTL-only) | API / Backend | D-15 permits TTL-only KV caching keyed by normalized query. At this scale it is probably unnecessary (see §"Should there be a result cache?"). |
| Overpass top-up proposal generation | CDN / Static → repo tool (offline script) | — | D-06: a repo tool, never a runtime dependency. It runs on a maintainer's machine, not in the Worker. |
| Home-screen install affordance | Frontend Server (browser runtime) | Telegram client | `WebApp.addToHomeScreen()` is client-side; the event fires back into the page. The Worker is not involved. |
---
## Standard Stack
### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `grammy` | **1.46.0** (already in `bot/`, `^1.46.0`) | Bot framework — `/cari` command handler, `ctx.sendVenue`, inline keyboards | Already the project's bot framework. **Verified this session in the installed typings**: `api.sendVenue(chat_id, latitude, longitude, title, address, other?)` and `ctx.replyWithVenue(...)` both exist; `sendVenue`'s args include `reply_markup?: InlineKeyboardMarkup \| ReplyKeyboardMarkup \| …`; `InlineKeyboardButton` has a `url` variant and a `web_app: WebAppInfo` variant; `WebAppInfo` has exactly one field, `url: string`. |
| `node:assert` + hand-rolled runner | repo-native (`bot/test/runner.mjs`, `web/test/runner.mjs`) | Test harness for the corpus runner | Repo precedent, zero deps. The corpus runner is a new suite in the same shape (see §Validation Architecture). |
| **No search library** | — | Normalization, exact lookup, bigram filtering, Damerau-Levenshtein | **Deliberate.** At 200–500 rows the whole algorithm is ~150 lines of pure TS. `fuse.js@7.5.0`, `fastest-levenshtein@1.0.16`, `string-similarity@4.0.4`, `levenshtein-edit-distance@3.0.1` all exist on npm (verified) but each adds a dependency to a forkability-law project (D-04) to solve a problem this scale does not have. D-09's "extends by table edit, not code change" is better served by a hand-rolled, table-driven normalizer than by a library's config surface. |
### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `node:crypto` (Web Crypto `subtle`) | built-in | Nothing cryptographic is needed — but see the D-11 note | **D-11 explicitly says "No crypto"** for the deep-link payload. Do **not** add a signature. The id is a plain key into a public gazetteer; the only threat is self-pranking. |
| `wrangler` | **4.151.3** (already in `bot/`) | `wrangler dev --port 8787 --local` (local KV) + `wrangler deploy` | D-12 Phase 3's dev topology; the gazetteer is bundled into the Worker at deploy time. |
| `vite` | **8.3.5** (already in `web/`) | Build-time copy of the gazetteer module into the webview bundle | D-11's "shared module, build-time". The web bundle must import the *same* module the bot imports — the sharing mechanism matters, see §"Sharing the gazetteer between bot and web". |
| `playwright` | `^1.50.0` (already in `web/`, browsers cached: chromium-1248, webkit-2272, firefox-1511) | Mock-Telegram harness capture for ONBD-02 evidence | Phase 3's `web/scripts/capture.mjs` already builds `web/dist-harness` and races `RAMS-BASEMAP-READY` → `ARAHARAH-LIVE`. A `?dest=<id>` boot profile slots straight into the existing capture list. |
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Hand-rolled 3-tier scorer | `fuse.js@7.5.0` (Bitap + fuzzy, 30 KB, MIT) | **Reject for v1.** Fuse is excellent general-purpose fuzzy search, but it (a) is a black box — the "extends by table edit, not code change" law (D-09) would be violated once you need a custom Malay rule table inside the scoring, (b) adds a dependency to a GPL/forkability-law project, (c) its scoring is optimized for long documents, not 3–20 character place names. **Revisit at backend #2** if the corpus shows the hand-rolled scorer cannot reach 90%. |
| Hand-rolled Damerau-Levenshtein | `fastest-levenshtein@1.0.16` (MIT, WASM+JS) | **Reject.** It is Levenshtein-only (no transposition), so it charges 2 edits for the transposition typos D-13's corpus explicitly requires ("transpositions"). D-09 locks Damerau for the residual net. A 40-line OSA/Damerau implementation is the whole requirement. |
| Precomputed trigram index at build time | Runtime bigram Dice filter | **Prefer the runtime filter.** A build-time trigram index adds a generated artifact to the repo (diff noise, stale-index risk) for zero measurable gain: at 4,000 aliases, building a bigram set per alias on the fly is ~1 ms. Revisit only if the corpus grows past ~5,000 entries. |
| OSM-derived gazetteer published under GPL-3.0 only | Dual-licence: code GPL-3.0, data file ODbL-1.0 | **Choose the dual-licence.** See §"Licensing — the ODbL obligation". GPL-3.0 alone risks ODbL §4.4 non-compliance on the data file. |
| `t.me/<bot>?startapp=<id>` direct link for the OPEN button | `url` inline button → `<origin>/?dest=<id>` | **Choose the `url` button.** The direct link works, but on Telegram Desktop (Windows/Linux) it shows a START button the user must press ([tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694)); the `url` button is a single tap on every client. See §"Deep-link mechanics". |
**Installation:**
```bash
# No new runtime dependencies. The gazetteer, normalizer and scorer are
# first-party modules inside the existing npm workspaces.
```
**Version verification:** the standard stack for this phase is entirely **already installed** — `grammy@1.46.0` (verified: `node -e "require('./node_modules/grammy/package.json').version"` → `1.46.0`), `wrangler@4.151.3` (verified via `npx wrangler --version`), `vite@8.3.5`, `typescript@5.9.3`, Node `v24.18.0`, npm `11.16.0`. The only registry probes run this session were for the **rejected** alternatives: `fuse.js` → **7.5.0**, `fastest-levenshtein` → **1.0.16**, `string-similarity` → **4.0.4**, `levenshtein-edit-distance` → **3.0.1**.
---
## Architecture Patterns
### System Architecture Diagram
```text
USER (chat) ONE WORKER (bot/ + web/ statics)
──────────── ───────────────────────────────
"/cari KLCC"
│
│ webhook POST /telegram/webhook
▼
┌──────────────────┐
│ grammY router │──▶ /cari handler (bot/src/bot/commands/cari.ts)
└──────────────────┘ │
│ 1. raw query string
▼
┌─────────────────────────────┐
│ normalizeQuery() │ ← rule table
│ case/whitespace fold │ (D-09)
│ abbreviation ↔ canonical │
│ glued-word split │
└──────────────┬──────────────┘
│ normalized query
▼
┌─────────────────────────────┐
│ SearchSource.search(q) │ ← the seam (D-03)
│ ┌────────────────────────┐ │ backend #1 = GazetteerSource
│ │ tier 1: exact lookup │ │ backend #2 = WASM geocoder
│ │ tier 2: bigram filter │ │ backend #3 = Photon
│ │ tier 3: Damerau score │ │
│ └────────────────────────┘ │
└──────────────┬──────────────┘
│ ranked candidates
▼
┌─────────────────────────────┐
│ rank + tie-break + top-3 │
└──────────────┬──────────────┘
│
┌────────────────────────────┴──────────────────────┐
│ if empty → "not in v1 area" (en/ms, honest refusal) │
│ if hits → sendVenue(best) + inline buttons │
└────────────────────────────┬──────────────────────┘
│ ← ONE message (D-02)
│ reply_markup:
│ row 0: OPEN IN ARAHARAH
│ → <origin>/?dest=<id>
│ row 1..: alternates (callback_data)
▼
╔══════════════════════════╗
║ TELEGRAM renders the ║
║ venue card (name, addr, ║
║ pin) + the buttons ║
╚══════════════════════════╝
│ user taps OPEN
▼
┌──────────────────────────────────────┐
│ Mini App webview (web/) │
│ read location.search ?dest=<id> │
│ (fallback: initData start_param) │
│ → resolve id via BUILD-TIME COPY │
│ of the same gazetteer module │
│ → destination in memory only │
│ → Phase-3 staged boot continues │
└──────────────────────────────────────┘
```
Flow trace — the primary use case ("/cari KLCC"):
1. User types `/cari KLCC` → Telegram delivers a `message` update to `/telegram/webhook`.
2. The grammY router dispatches to the `/cari` handler (the Phase-2 alias pattern).
3. The handler extracts the raw query text **after** the command and its arguments.
4. `normalizeQuery("KLCC")` folds case/whitespace; the abbreviation table leaves it unchanged.
5. `SearchSource.search("klcc")` runs tier 1 (exact alias hash) → hits the KLCC entry immediately.
6. The handler ranks, takes the top 1 as best and up to 4 as alternates.
7. `ctx.sendVenue(lat, lng, title, address, { foursquare_type, reply_markup })` — one message.
8. The user taps OPEN → the webview opens at `<origin>/?dest=klcc-pavilion`.
9. The webview reads `location.search`, resolves the id from its bundled gazetteer copy, and hands the destination to the Phase-3 boot flow. Nothing is re-typed; nothing is persisted.
### Recommended Project Structure
```text
bot/src/
├── search/ # ← NEW: the shared search module
│ ├── types.ts # SearchSource interface, SearchResult, GazetteerEntry
│ ├── gazetteer.ts # the curated KL dataset loader (JSON import)
│ ├── data/ # ← NEW: the curated JSON (D-07)
│ │ └── kl-gazetteer.json # 200–500 rows, typed, diff-friendly
│ ├── normalize.ts # rule table + folding + glued-word split (D-09)
│ ├── rules/
│ │ └── abbreviations.ts # ← the editable rule table (D-09)
│ ├── score.ts # exact → bigram filter → Damerau
│ └── source.ts # GazetteerSource implements SearchSource (D-03)
├── bot/commands/cari.ts # ← NEW: the /cari handler (D-01)
└── i18n/strings.ts # ← MODIFIED: search copy en/ms + /cari in helpMenu
bot/scripts/
├── overpass-topup.mjs # ← NEW: repo tool, proposes candidates (D-06)
└── register-commands.mjs # ← MODIFIED (via helpMenu, no inline copy)
bot/test/
├── cari.test.mjs # ← NEW: handler + payload-shape gates
├── normalizer.test.mjs # ← NEW: per-rule vectors (D-09)
└── corpus.test.mjs # ← NEW: the golden-corpus gate (D-13/D-14)
bot/test/fixtures/
└── golden-corpus.json # ← NEW: the pinned corpus (D-13)
web/src/
├── search/ # ← NEW: build-time copy of the SAME module
│ ├── gazetteer.ts # (or a re-export from a shared location)
│ └── resolve-destination.ts # id → destination, in-memory only
├── main.ts # ← MODIFIED: boot branch for ?dest=
└── telegram/webapp.ts # ← MODIFIED: add startParam to the narrow interface
web/index.html # ← MODIFIED: ?dest=<id> harness boot profile
web/test/
└── destination-preload.test.mjs # ← NEW: ONBD-02 gate
```
### Pattern 1: The `SearchSource` seam (D-03)
**What:** One interface, several backends. The bot only ever talks to `SearchSource`; swapping backends is a one-line change in the Worker factory, not a refactor of the handler.
**When to use:** Always — this is the phase's central abstraction and it is what makes the D-14 escalation ladder mechanical rather than aspirational.
```typescript
// Source: D-03 (CONTEXT.md) — the interface shape is this research's recommendation,
// constrained to what the phase actually needs.
export interface SearchResult {
/** Gazetteer entry id — the payload D-11 puts in the deep link. */
id: string;
name: string;
address: string;
lat: number;
lng: number;
category: string;
/** 0..1, higher is better. Used for ranking + tie-breaks only. */
score: number;
/** Which alias/form matched — useful for the corpus report, never for logging. */
matchedVia: "exact" | "normalized" | "fuzzy";
}
export interface SearchSource {
/** Normalized query → ranked candidates. Empty array = honest refusal. */
search(query: string, limit: number): Promise<SearchResult[]>;
/** The backend's identity, for the escalation record (D-14). */
readonly backendId: string;
}
```
**Why this shape:** `search()` returns `Promise` even though backend #1 is synchronous. That keeps the signature stable for backend #3 (Photon is a network call) — a future swap must not change the handler.
### Pattern 2: Three-tier scoring cascade (SRCH-01, SRCH-02)
**What:** Exact-match first, cheap filter second, expensive score third. Each tier is strictly narrower than the last, so the expensive algorithm only ever sees a handful of candidates.
**When to use:** Always — this is what makes 10 ms of Workers CPU comfortable rather than tight.
```typescript
// Source: algorithm properties cited in §Sources (Cohen/Winkler 2003 for Jaro-Winkler
// prefix weighting; RapidFuzz docs for normalized Damerau-Levenshtein semantics).
// The three-tier ORDER is this research's recommendation, justified by the
// Workers CPU budget (10 ms/request, Cloudflare limits doc).
/** Tier 1 — O(1) hash lookup on the folded alias table. */
function exactLookup(index: Map<string, GazetteerEntry[]>, q: string): GazetteerEntry[] {
return index.get(q) ?? [];
}
/** Tier 2 — character-bigram Dice coefficient as a cheap relevance filter.
* Dice = 2·|A∩B| / (|A|+|B|) over bigram multisets. O(len) per candidate. */
function diceBigrams(a: string, b: string): number {
if (a === b) return 1;
if (a.length < 2 || b.length < 2) return a === b ? 1 : 0;
const grams = (s: string): Map<string, number> => {
const m = new Map<string, number>();
for (let i = 0; i < s.length - 1; i++) {
const g = s.slice(i, i + 2);
m.set(g, (m.get(g) ?? 0) + 1);
}
return m;
};
const ga = grams(a);
const gb = grams(b);
let shared = 0;
for (const [g, n] of ga) {
const other = gb.get(g);
if (other !== undefined) shared += Math.min(n, other);
}
return (2 * shared) / (a.length - 1 + b.length - 1);
}
/** Tier 3 — Damerau-Levenshtein (optimal string alignment variant is enough).
* Counts a transposition as ONE edit, which D-13's typo set requires. */
function damerau(a: string, b: string): number {
const al = a.length;
const bl = b.length;
const d: number[][] = Array.from({ length: al + 1 }, () => new Array<number>(bl + 1).fill(0));
for (let i = 0; i <= al; i++) d[i][0] = i;
for (let j = 0; j <= bl; j++) d[0][j] = j;
for (let i = 1; i <= al; i++) {
for (let j = 1; j <= bl; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
d[i][j] = Math.min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost);
if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) {
d[i][j] = Math.min(d[i][j], d[i - 2][j - 2] + 1); // transposition
}
}
}
return d[al][bl];
}
```
**The cascade, in order:**
1. **Tier 1 — exact.** Look up the normalized query in a `Map<string, GazetteerEntry[]>` built at load time from every name + alias + its normalized form. A hit is a *perfect* match; score 1.0; these always sort above anything fuzzy.
2. **Tier 2 — bigram Dice filter.** Over the remaining entries, keep only those whose best alias scores above a low Dice floor. This is O(entries × aliases) but each comparison is O(len) — cheap.
3. **Tier 3 — Damerau-Levenshtein.** On the handful that survived tier 2, compute the normalized Damerau distance and convert to a 0..1 similarity. **Only apply tier 3 to strings of length ≥ 4** — below that, a single edit is a large fraction of the string and the scores are noise. (This is the same insight the taxify fuzzy-matching vignette reports: *"Jaro-Winkler can help with very short names (3-5 characters), where a single edit produces a large normalized Damerau-Levenshtein distance."*)
**Why this order, and not Jaro-Winkler as the primary:** D-09 makes Damerau the residual net. Jaro-Winkler is genuinely better on short strings (its prefix weighting is designed for it — Cohen/Winkler 2003: *"The Jaro and Jaro-Winkler metrics seem to be intended primarily for short strings"*), but its prefix bonus is a liability for a gazetteer where many entries share a first token ("Taman …", "Stesen …", "Suria …"): JW would systematically rank all "Taman X" entries above a correct non-Taman match. Bigram Dice has no such prefix bias. **Recommendation: Dice for filtering, Damerau for the final residual, exact-match always wins.** Jaro-Winkler is a reasonable swap for tier 3 if the corpus shows Damerau's normalisation mis-scores the many short acronyms (IJN, PWTC, LRT, KLIA) — that is exactly the kind of question the corpus runner exists to answer empirically.
### Pattern 3: The Malay normalizer as an editable rule table (D-09)
**What:** A bidirectional abbreviation table plus folding passes, applied identically to the query and to every alias at index time. Adding a new abbreviation is a **table edit**, not a code change.
**When to use:** Always — this is D-09's explicit architecture requirement.
```typescript
// Source: rule table content grounded in the UPU S42 Malaysian address standard
// (https://www.upu.int/UPU/media/upu/PostalEntitiesFiles/s42/mysEn.pdf), which
// lists the canonical "primary thoroughfare type" tokens, plus the OSM KL road
// list (https://en.wikipedia.org/wiki/List_of_roads_in_Kuala_Lumpur) which shows
// the "Jalan"/"Lorong"/"Persiaran"/"Lebuh" naming in KL, plus the Pos Malaysia
// /MaCGDI address examples which use "Persint 4" and "Taman".
/** Bidirectional: [from, to]. Both directions are added automatically, so the
* table holds ONE line per concept. */
export const ABBREVIATIONS: ReadonlyArray<readonly [string, string]> = [
// ── thoroughfare types ──
["jln", "jalan"],
["jl", "jalan"], // careful: "jl" is also used in place names — see Pitfall 6
["jl.", "jalan"],
["lnr", "lorong"],
["lr", "lorong"],
["psn", "persiaran"],
["persiaran", "persiaran"], // identity guard for the family below
["ps", "persiaran"],
["lbh", "lebuh"],
["bkt", "bukit"],
["tmn", "taman"],
["tm", "taman"],
["kg", "kampung"],
["kmpg", "kampung"],
["kampung", "kampung"], // identity guard
["presint", "presint"],
["prsn", "presint"],
["prsnt", "presint"],
["presin", "presint"],
// ── transit / civic shorthand (D-08's alias lever) ──
["stesen", "stesen"],
["stn", "stesen"],
["sta", "stesen"],
["station", "stesen"],
["hentian", "hentian"],
["st", "stesen"], // risky — see Pitfall 6
["sk", "sekolah"],
["kg smith", "kampung smith"], // multi-word: see Pitfall 7
];
```
**The normalizer pipeline** (each step is a separate, individually-testable function so the corpus runner can report *which* rule fired):
```
normalize(s):
1. Unicode NFKD + strip diacritics → "Ampang" stable under "Ampáng"
2. casefold → "jln ampang"
3. collapse whitespace → "jln ampang"
4. strip punctuation (.,/,-) → "jln ampang"
5. abbreviation table (word-wise) → "jalan ampang"
6. glued-word split (if applicable)→ "jalan ampang"
```
**Rules 5 and 6 are table-driven, not code-driven.** Rule 5 expands a token only when the token is an exact whole-word match of a table key — this avoids the "Sunway" → "Sun Way" disaster that a naive substring replace causes (see Pitfall 5).
### Pattern 4: Gazetteer entry schema (D-07, D-08, D-06)
```typescript
// Source: D-07 (typed JSON, V8-native parse, diff-friendly) + D-08 (aliases live
// inside each entry) + D-06 (hand-curated seed + Overpass-proposed top-ups).
// Fields beyond the required five are OpenCode's discretion per CONTEXT.md.
export interface GazetteerEntry {
/** Stable, URL-safe, and the payload of the D-11 deep link.
* Must match /^[a-z0-9-]+$/ — the Telegram start_param character set allows
* only A-Z a-z 0-9 _ and -, so the id MUST NOT contain a dot or underscore
* if it is ever to travel through a start_param. */
id: string;
/** Canonical display name, as a local would write it. */
name: string;
/** Street address for the venue card. May be a landmark address ("Jalan Ampang")
* when no street number exists. */
address: string;
lat: number;
lng: number;
/** Coarse class for the venue card's four-square type and for alternates
* labelling: "mall" | "transit" | "hospital" | "government" | "park" |
* "landmark" | "kampung" | "education" | "religious" */
category: string;
/** Locals' shorthand. THE main corpus-quality lever (D-08). */
aliases: string[];
/** Optional: the OSM element this entry was curated from, for the Overpass
* top-up script to reconcile against. Not required for search. */
osm?: { type: "node" | "way" | "relation"; id: number };
}
```
**The JSON file lives at `bot/src/search/data/kl-gazetteer.json`** and is imported with a plain `import ... with { type: "json" }` (or a `readFileSync` in the Node-side script). It is a **plain committed file** — the D-04 forkability law.
### Pattern 5: Sharing the gazetteer between the bot and the webview (D-11)
**What:** The webview must resolve `dest_<id>` to a destination using "its own copy of the same gazetteer module (shared module, build-time)" (D-11).
**Why this matters:** If the two copies drift, a deep link points at a destination the webview does not recognise — the exact "stale-webview dataset" scenario D-11 acknowledges. The mitigation must be **structural**, not procedural.
**Recommendation — one source file, two import paths.** Put the canonical module in a place both workspaces can import. Three options, in order of preference:
| Option | Mechanism | Verdict |
|--------|-----------|---------|
| **A. A shared workspace package** (e.g. a new `search/` npm workspace with the gazetteer + normalizer + scorer) | `bot/` and `web/` both depend on `@araharah/search` | **Best.** One copy on disk; the build can never drift. Cost: one more workspace entry in the root `package.json`. Consistent with the repo's existing workspace pattern (`theme/` is already consumed by `web/` as a build-time dependency). |
| B. A file symlink / duplicate in both trees | `web/src/search/gazetteer.ts` re-exports from `../../bot/src/search/…` | Works with Vite (needs `server.fs.allow` for dev) but is brittle and reads badly. |
| C. A build step that copies `bot/src/search/data/*.json` into `web/` | A `prebuild` script | **Reject.** It reintroduces exactly the "token duplication" anti-pattern the repo's ARCHITECTURE.md already flags for the mockup Tailwind configs. |
**Option A is the recommendation** and it also makes the corpus runner able to import the *identical* module the production paths use — the property D-14's bake-off depends on.
### Anti-Patterns to Avoid
- **Substring replacement in the normalizer.** `s.replace("tmn", "taman")` turns "Sunway Pyramid" into "Sunaan Piyramid". Always expand **whole tokens** only, and only on exact table keys.
- **A fuzzy score that can outrank an exact alias match.** If "Kuala Lumpur City Centre" scores higher than "KLCC" for the query "KLCC", the top-3 gate passes but the product is wrong. Exact/normalized exact MUST dominate the ranking — enforce it by tier, not by a weight.
- **Building the index at module top level in the Worker.** Cloudflare's docs warn explicitly: *"avoid expensive work in global scope… a Worker must parse and execute its global scope within 1 second."* Parse the gazetteer **inside the handler** (or memoize lazily in module scope, populated on first use).
- **Logging the query text.** D-15 forbids it, and the repo's D-14 law is category-only logging. The corpus runner may print queries (it is a developer tool running locally on a committed file) but the **Worker** must never do so.
- **An N×N scan of the whole gazetteer with Damerau-Levenshtein on every alias.** 500 entries × 8 aliases × O(len²) per comparison is the CPU-budget killer. The three-tier cascade is the fix.
- **A "no results" answer that still shows a pin.** D-14 requires honest refusal for out-of-KL queries. If the gazetteer has no match, the bot must say so and offer nothing — never the nearest entry.
- **Making `startapp` a `web_app` button's field.** It isn't one. `WebAppInfo` has exactly one field: `url`.
---
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Malay abbreviation expansion | A hand-coded `if (s.startsWith("jln"))` chain | A **table** (D-09) | Locals invent shorthand faster than code changes ship. A table is a data edit a maintainer can review in a PR. |
| Fuzzy string distance | A bespoke "vowel-drop-aware" similarity | **Damerau-Levenshtein (OSA)** with a length floor | Damerau is a textbook algorithm with a 40-line implementation; getting transposition semantics right (D-13 requires them) is the only reason to write it. |
| Bigram similarity | A custom n-gram counter | The Dice coefficient over bigram multisets | Two counters and one division. A library would be a dependency for 15 lines. |
| Telegram venue cards, inline keyboards, deep links | Hand-rolled Bot API HTTP calls | **grammy 1.46.0** (`ctx.sendVenue`, `InlineKeyboardButton`) | Already a dependency; the typings are verified to support `reply_markup` on `sendVenue`. Hand-rolling would add a signing/marshalling surface for no gain. |
| Telegram initData validation for `start_param` | Re-validating the HMAC in the webview | **The existing `POST /api/session`** (Phase 2) | The webview already posts its raw `initData` to the Worker and gets a session token. `start_param` is *inside* the signed `initData`, so the same validation covers it. **Do not build a second validation path.** |
| OSM data extraction | Hand-parsing Overpass JSON into gazetteer rows | The **Overpass QL query** + `out center;` + a thin mapper | Overpass returns tagged elements; the mapping from `name`/`addr:*`/`center` to a `GazetteerEntry` is ~30 lines. |
| A geocoding index | SQLite/sql.js WASM in the Worker bundle | **Plain JSON** (D-07) | D-07's own reasoning: at 200–500 rows SQLite-in-Worker is ~1.5 MB of bundle plus cold-start cost, against a JSON file that is a few tens of KB and diffs cleanly. |
| A fuzzy-search library | `fuse.js` / `fastest-levenshtein` | The **three-tier cascade** in §Pattern 2 | See §Alternatives Considered. The hand-rolled cascade is ~150 lines, table-driven, dependency-free (D-04), and is the thing D-09's architecture describes. |
| Web-hook / deep-link token signing | HMAC over the destination id | **No crypto** (D-11) | The payload's only threat is a user pranking themselves. A plain id into a public gazetteer needs no secret. |
**Key insight:** In this phase the *hard* part is not the algorithm — it is the **data** (locals' shorthand) and the **discipline** (one module, one copy, one corpus). Every temptation to add a library or a service is a temptation to move the quality decision out of a table a maintainer can edit and into a black box they cannot.
---
## Runtime State Inventory
> This phase **creates** state but does not rename or migrate any. The inventory below answers the canonical question for the *new* state the phase introduces, since D-04/D-06/D-07 make the gazetteer a committed artifact and D-15 permits KV caching.
| Category | Items Found | Action Required |
|----------|-------------|------------------|
| **Stored data (new)** | `bot/src/search/data/kl-gazetteer.json` — the curated KL gazetteer, 200–500 rows. Derived from OSM (names, coordinates, addresses) → **ODbL-1.0 obligation** (see §Licensing). | Data file is created this phase. Ship it with the ODbL header + attribution, and record the licence in the file's first lines (a JSON field like `"_license": "ODbL-1.0"` plus a sibling `NOTICE`/`LICENSES.md` entry). |
| **Stored data (new)** | `bot/test/fixtures/golden-corpus.json` — the pinned corpus (D-13). Not OSM-derived (it is *queries* and expected ids). | Created this phase. Plain committed file, GPL-3.0 with the repo (it contains no OSM content). |
| **Live service config** | None. The `SearchSource` is in-Worker; no external service is added (D-03). The Worker already has `BOT_TOKEN`, `WEBHOOK_SECRET`, `chat_sessions` in `bot/wrangler.jsonc`. | None. |
| **OS-registered state** | None. No systemd unit, no launchd plist, no Task Scheduler entry. The bot's Telegram menu (registered via `setMyCommands`) is a *live service config* surface, not OS state — it must be re-registered after `/cari` is added to `helpMenu` (the existing `npm run bot:commands` script covers this). | Re-run `npm run bot:commands` after the copy change; the Phase-2 CS5 gate asserts no drift. |
| **Secrets / env vars** | None new. D-04's "env only" law is already true; the search backend needs no credentials (backend #3 Photon would, but it is deferred to Phase 5). | None. |
| **Build artifacts** | `web/dist` / `web/dist-harness` — the webview's copy of the gazetteer is bundled at build time. If the gazetteer changes and the web bundle is not rebuilt, the webview resolves stale ids. | The `npm run deploy` chain already builds theme → web → wrangler deploy, so the artifact refreshes on every deploy. **The Phase-3 build-artifact gate (`web/test/build-artifact.test.mjs`) is the place to add a check that the bundled gazetteer is present and id-complete.** |
| **Runtime (ephemeral)** | The resolved destination lives in **memory only** in the webview — no localStorage, no sessionStorage, no IndexedDB, no cookie (GEO-03's law continues to bind; the Phase-3 `web/test/no-storage.test.mjs` grep gate will catch a violation). | Grep gate is automatic once the code lands. |
---
## Common Pitfalls
### Pitfall 1: `web_app` inline buttons have no `start_param` field
**What goes wrong:** The team writes `web_app: { url: origin, start_param: "dest_x" }` — TypeScript rejects it, or it is cast and silently dropped. The Mini App opens with no destination and the user must re-type. ONBD-02 fails.
**Why it happens:** `WebAppInfo` has exactly one field (`url`). The `startapp` parameter belongs to the **`t.me` deep-link** form, not to the button payload.
**How to avoid:** Use a **`url` inline button** (not `web_app`) pointing at `<origin>/?dest=<id>`; the webview reads `location.search`. Read `initData`'s `start_param` as a **secondary** path for the case where the Mini App was opened by a `t.me?startapp=` link.
**Warning signs:** The OPEN button opens the Mini App with no destination preloaded; the webview's `location.search` is empty.
### Pitfall 2: Telegram Desktop requires a second tap on a `t.me?startapp=` deep link
**What goes wrong:** On Telegram Desktop (Windows/Linux), tapping OPEN shows a START button the user must press — the "one tap to proceed" promise (D-02) breaks on that client.
**Why it happens:** [tdesktop#28694](https://github.com/telegramdesktop/tdesktop/issues/28694): *"the user must manually press the 'START' button for the deeplink to be processed. This does not align with behavior observed on other platforms."*
**How to avoid:** Use the `url` button form. If a `t.me` deep link is used for some other surface, document the Desktop extra tap.
**Warning signs:** The feature works on Android/iOS and stalls on Desktop.
### Pitfall 3: `start_param` has a restricted character set
**What goes wrong:** A gazetteer id containing a dot (`klcc.pavilion`) or a space is passed through `startapp=` and the client drops or truncates it.
**Why it happens:** The Telegram start parameter allows only `A-Z a-z 0-9 _` and `-`, up to 512 characters; the documented validation regex is `/^[\w-]{0,512}$/`.
**How to avoid:** Constrain gazetteer ids to `/^[a-z0-9-]+$/` — lowercase, digits, hyphens. **This is a schema-level constraint that must be enforced by a test**, not by convention. It also makes ids URL-safe for the `?dest=` query string.
**Warning signs:** The webview receives `start_param` but cannot map it to an entry.
### Pitfall 4: Overpass returns HTTP 406 without a User-Agent header
**What goes wrong:** The top-up script's first fetch returns 406 and the script exits with a parse error; or a well-meaning fix adds a browser UA that gets the script rate-limited.
**Why it happens:** **Verified this session**: `curl -s -o /dev/null -w "%{http_code}" https://overpass-api.de/api/status` returned **406** with no User-Agent. The Overpass API rejects requests without an identifying UA. The rate-limit guidance also states: *"Rate limits apply per IP address on all public instances. Including a User-Agent or Referer header identifying your application is strongly recommended as best practice."*
**How to avoid:** The script sets a descriptive `User-Agent` (e.g. `araharah-gazetteer-topup/1.0 (+https://github.com/…/araharah)`) and a single request per run. Do **not** fan out per-entry requests.
**Warning signs:** `406` on the first call; `429` after several runs (the Overpass rate limit is per-IP with 2 concurrent slots; requests queue up to 15 s and beyond that are delayed).
### Pitfall 5: Substring abbreviation expansion corrupts real names
**What goes wrong:** A naive `replace("tmn","taman")` turns "Sunway Pyramid" into "Sunaan Piyramid"; `replace("kl","kuala lumpur")` turns "KLIA" into "Kualia lumpuria".
**Why it happens:** Substring replacement without token boundaries.
**How to avoid:** Expand only **whole tokens** matched against a table key, and keep a conservative table. Add a normalizer test vector for "Sunway Pyramid" and "KLIA" that asserts they are unchanged.
**Warning signs:** A corpus entry stops matching its own name after a table edit.
### Pitfall 6: Over-greedy short abbreviations collide with real tokens
**What goes wrong:** Expanding `st` → `stesen` turns "St John" into "stesen John"; expanding `jl` → `jalan` turns "Jln" variants inconsistently; `tm` → `taman` hits names like "Tmn" vs "TM".
**Why it happens:** Two-letter Malay abbreviations are ambiguous with English words and with name fragments.
**How to avoid:** (a) Keep the abbreviation table's short keys minimal and **only** apply them as a *whole-token* match; (b) for genuinely risky pairs, prefer adding an explicit **alias** on the entry (D-08's lever) over a global rule; (c) put both directions in the table so the canonical form is stable.
**Warning signs:** A corpus positive regresses after a table edit — the corpus runner is exactly the instrument that catches this.
### Pitfall 7: Glued-word splitting produces false splits
**What goes wrong:** "KampungBaru" is correctly split into "kampung baru"; but "SunwayPyramid" → "sunway pyramid" and "TamanDesa" → "taman desa" are guesses, and "KLCC" must not be split at all.
**Why it happens:** Splitting is only safe when the first segment is a **known prefix token** from a closed vocabulary (the abbreviation table's keys + a small explicit list like `kampung`, `taman`, `jalan`, `lorong`, `persiaran`, `bukit`, `stesen`, `sekolah`, `hospital`, `universiti`, `kolej`).
**How to avoid:** Split only on **known** prefixes, and only when the remainder is ≥ 3 characters. "KLCC" has no known prefix → unchanged. "Sunway" is not in the prefix vocabulary → unchanged (so "SunwayPyramid" stays glued — which is acceptable: the D-08 alias list covers it).
**Warning signs:** A query that used to match stops matching after the splitting rule is tightened or loosened.
### Pitfall 8: The gazetteer JSON is imported at module top level in the Worker
**What goes wrong:** On a cold start the Worker parses the JSON and builds the index in the global scope. Under Workers' **1 s startup limit** and the "avoid expensive work in global scope" guidance, a large gazetteer plus a 4,000-alias index build is a startup risk — and it happens on *every* isolate cold start, even for `/api/session` requests that do not need it.
**Why it happens:** `import data from "./data/kl-gazetteer.json"` at the top of a module evaluates at import time.
**How to avoid:** Parse + index **lazily inside the handler**, memoized in module scope. Also confirm the JSON import mechanism works in the Workers bundle (Vite/esbuild supports JSON imports natively; `wrangler`'s bundler does too — verify with `wrangler deploy --dry-run` and check the bundle size, which was 167.97 KiB at Phase 2).
**Warning signs:** `wrangler deploy` bundle size jumps by tens of KB; startup time metrics rise.
### Pitfall 9: ODbL share-alike is treated as a formality
**What goes wrong:** The team ships an OSM-derived gazetteer under GPL-3.0 only and declares the project "GPL-3.0" in the README — a licence incompatibility on the data file, invisible until someone notices.
**Why it happens:** The OSM wiki's *Legal FAQ* says: *"If you are writing software that uses OSM data, you may choose any license you like, including proprietary."* That sentence is about the **software**. The **database** is a separate obligation with a separate answer.
**How to avoid:** Dual-licence: **code GPL-3.0, data file ODbL-1.0**, with `© OpenStreetMap contributors` attribution in the gazetteer header, in the bot's strings, and in the README. Add a `LICENSES.md` (or an `OSMDATA-LICENSE` note) explaining the split. This is what the ecosystem does and it costs almost nothing.
**Warning signs:** A `LICENSE` file exists that mentions only GPL-3.0 while a derived data file ships in the repo.
### Pitfall 10: The corpus gate is a report, not an executable check
**What goes wrong:** The corpus is run manually once, the numbers look fine, and the phase "closes the bake-off" — but nothing prevents a later edit from silently dropping the hit rate below 90%.
**Why it happens:** D-14 states a threshold but not a mechanism.
**How to avoid:** Make the threshold an **assertion in the test suite** (`bot/test/corpus.test.mjs`): compute the top-3 hit rate over positives and assert `>= 0.90`; assert every negative yields **zero** results. The suite then fails loudly on regression and the escalation ladder becomes a concrete, triggerable event.
**Warning signs:** The corpus file changes without a corresponding test run in the commit.
### Pitfall 11: `/cari` never appears in the Telegram menu
**What goes wrong:** The command works when typed, but the menu still lists only `/help` and `/privacy` — because the menu is derived from `strings.helpMenu`, not from the registered handlers.
**Why it happens:** Phase 2's D-03 design: `register-commands.mjs` *derives* menu descriptions from `helpMenu` lines. A command that is not in `helpMenu` is not in the menu.
**How to avoid:** Add `/cari` to `strings.helpMenu` (both locales) in the same change that registers the handler, and extend the Phase-2 CS5/CS6 gates to expect it. Also note the Phase-2 finding: the menu is **client-cached**, so `setChatMenuButton` + the `all_private_chats` scope may be needed (the existing `bot/scripts/set-menu-button.mjs` covers this).
**Warning signs:** The handler exists but the menu does not list it.
### Pitfall 12: The venue card and the buttons land as two messages
**What goes wrong:** The bot sends the venue with `ctx.replyWithVenue(...)` and then a second `ctx.reply(...)` carrying the inline keyboard — violating D-02's "no second message round".
**Why it happens:** Not knowing that `sendVenue` accepts `reply_markup`.
**How to avoid:** Pass `reply_markup` in the `sendVenue` call itself. Verified in the installed typings this session.
**Warning signs:** Two consecutive bot messages per `/cari`.
---
## Code Examples
Verified patterns from official sources and the installed dependency tree.
### 1. The `/cari` handler — one message, venue card + OPEN + alternates (D-02, D-03, D-10, D-11)
```typescript
// Source: grammY 1.46.0 installed typings —
// api.d.ts:410 sendVenue(chat_id, latitude, longitude, title, address, other?)
// methods.d.ts sendVenue args include reply_markup?: InlineKeyboardMarkup | …
// markup.d.ts InlineKeyboardButton.UrlButton { text, url }
// InlineKeyboardButton.WebAppButton { text, web_app: WebAppInfo }
// markup.d.ts WebAppInfo { url: string } ← url ONLY, no start_param
import { InlineKeyboard } from "grammy";
import { sendVenue } from "../search/client.ts";
export async function replyWithShortlist(
ctx: Context,
locale: Locale,
results: SearchResult[],
): Promise<void> {
if (results.length === 0) {
// D-14: honest refusal. No pin, no nearest-match fabrication.
await ctx.reply(strings[locale].searchNoMatch, { link_preview_options: { is_disabled: true } });
return;
}
const [best, ...alternates] = results;
// D-02: ONE message. The OPEN button + the alternates ride on the venue card.
const keyboard = new InlineKeyboard()
.text(strings[locale].searchOpenButton)
.url(`${env.miniAppOrigin}/?dest=${best.id}`) // ← url button, NOT web_app
.row()
...alternates.slice(0, 4).map((alt) =>
InlineKeyboard.text(alt.name).callback_data(`dest:${alt.id}`),
);
await ctx.replyWithVenue(best.lat, best.lng, best.name, best.address, {
foursquare_type: FOURSQUARE_BY_CATEGORY[best.category], // optional enrichment
reply_markup: keyboard,
});
}
```
> **Note on `web_app` vs `url`:** `web_app` is also valid (Telegram grants the Mini App the user's basic info and a `query_id`), but its `WebAppInfo` carries **no** `start_param`. To carry the destination through a `web_app` button you would have to put it in the Mini App URL's own query string and read it from `location.search` anyway — which is exactly what the `url` button does, with one fewer moving part. The `url` button also avoids the Telegram Desktop START-button trap (Pitfall 2).
### 2. Reading the destination in the webview (ONBD-02, D-11)
```typescript
// Source: Telegram Mini Apps docs — the start parameter arrives in the
// initData's `start_param` field AND as the GET parameter `tgWebAppStartParam`;
// character set /^[A-Za-z0-9_-]{0,512}$/.
import { getWebApp } from "../telegram/webapp.ts";
/** D-11: the deep link carries a plain gazetteer entry id. Two arrival paths:
* 1. ?dest=<id> in the Mini App URL's own query string (the `url` button form)
* 2. start_param inside the SIGNED initData (the t.me?startapp= deep-link form)
* The id is resolved against the webview's BUILD-TIME COPY of the gazetteer.
* Nothing is persisted — memory only (GEO-03). */
export function readDestinationId(): string | null {
const fromQuery = new URLSearchParams(window.location.search).get("dest");
if (fromQuery !== null && fromQuery !== "") return fromQuery;
const wa = getWebApp();
const startParam = wa?.startParam; // ← add to TelegramWebAppLike
if (startParam !== null && startParam !== undefined && startParam !== "") {
// Tolerate the "dest_<id>" form D-11 names.
return startParam.startsWith("dest_") ? startParam.slice(5) : startParam;
}
return null;
}
```
> **The `start_param` must come from the SIGNED `initData`, not `initDataUnsafe`.** The existing `POST /api/session` (Phase 2) already validates `initData` server-side and mints a session token from it — so if the webview wants to *trust* the `start_param`, the cleanest path is to have `/api/session` echo the validated `start_param` (or the resolved destination) back alongside the token. That avoids building a second validation path (see §Don't Hand-Roll). D-11's own framing ("the payload's only threat is a user pranking themselves") means the pragmatic reading is fine too, but the planner should choose one deliberately.
### 3. Overpass top-up query (D-06)
```overpassql
# Source: Overpass QL reference (https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL)
# bbox order: south,west,north,east (Overpass QL convention)
# `out center;` gives a way/relation a representative centre point
# `nwr` = node+way+relation in one filter
# KL metro bounding box (generous; covers the core Klang Valley sprawl)
[out:json][timeout:120][maxsize:536870912];
(
nwr["shop"="mall"](2.95,101.55,3.20,101.80);
nwr["landuse"="retail"](2.95,101.55,3.20,101.80);
nwr["tourism"="attraction"](2.95,101.55,3.20,101.80);
nwr["leisure"="park"](2.95,101.55,3.20,101.80);
nwr["railway"="station"](2.95,101.55,3.20,101.80);
nwr["station"="subway"](2.95,101.55,3.20,101.80);
nwr["public_transport"="station"](2.95,101.55,3.20,101.80);
nwr["amenity"="hospital"](2.95,101.55,3.20,101.80);
nwr["amenity"="university"](2.95,101.55,3.20,101.80);
nwr["amenity"="place_of_worship"](2.95,101.55,3.20,101.80);
nwr["place"~"^(suburb|quarter|neighbourhood|village|hamlet)$"](2.95,101.55,3.20,101.80);
nwr["office"="government"](2.95,101.55,3.20,101.80);
nwr["building"="government"](2.95,101.55,3.20,101.80);
);
out center tags 300; // `out center tags` — geometry-free + centre + tags
```
**Script rules (D-06):**
- **One** HTTP request per run, with a descriptive `User-Agent` (Pitfall 4).
- Output is a **candidate list**, never a direct write to the curated file. The maintainer reviews and approves entries into `kl-gazetteer.json`.
- The script must **not** invent aliases. Aliases are D-08's human lever. The script may *suggest* an alias when an OSM tag like `name:en`, `short_name`, `official_name`, `old_name`, or `int_name` exists — those are real data, not guesses.
- Nominatim is a reasonable **fallback** for a specific named place the bbox query missed (e.g. a mall whose OSM tag is `shop=department_store` rather than `shop=mall`). Nominatim's usage policy is stricter than Overpass's — at most one request per second, a valid identifying User-Agent, and no bulk use. **The top-up script is a maintainer tool, not a crawler**; a bulk Nominatim sweep would violate the policy.
### 4. The golden corpus format (D-13) and the threshold gate (D-14)
```json
{
"_comment": "araharah golden corpus — the executable definition of the bake-off (D-13). Each positive pins its expected entry id. Each negative must yield ZERO results (honest refusal). Reviewers extend by editing THIS FILE ONLY.",
"_version": 1,
"threshold": { "top3_hit_rate": 0.9 },
"positives": [
{ "q": "KLCC", "expect": "suria-klcc", "kind": "golden" },
{ "q": "Jln Ampang", "expect": "suria-klcc", "kind": "abbrev" },
{ "q": "Jalan Ampang", "expect": "suria-klcc", "kind": "canonical" },
{ "q": "jln ampang", "expect": "suria-klcc", "kind": "folding" },
{ "q": "Tmn", "expect": "taman-...", "kind": "abbrev" },
{ "q": "Sunway Pyrmid", "expect": "sunway-pyramid", "kind": "typo-transposition" },
{ "q": "Sunwy Pyramid", "expect": "sunway-pyramid", "kind": "typo-missing-key" },
{ "q": "DUKE", "expect": "duke-university", "kind": "acronym" },
{ "q": "Kg Baharu", "expect": "kampung-baharu", "kind": "kampung" },
{ "q": "Kg Baharu", "expect": "kampung-baharu", "kind": "glued" },
{ "q": "PWTC", "expect": "pwtc", "kind": "acronym" }
],
"negatives": [
{ "q": "Kuching", "reason": "out of the v1 area" },
{ "q": "Penang", "reason": "out of the v1 area" },
{ "q": "asdkjhaskdjh", "reason": "not a place" }
]
}
```
**The gate, as an executable assertion** (this is the piece that makes D-14 a decision rather than an opinion):
```javascript
// bot/test/corpus.test.mjs — shape follows bot/test/harness.mjs (repo precedent).
import { strict as assert } from "node:assert";
import { test } from "./harness.mjs";
import { corpus } from "./fixtures/golden-corpus.json" with { type: "json" };
import { GazetteerSource } from "../src/search/source.ts";
test("CORPUS D-14: top-3 hit rate ≥ 90% on positives AND every negative is refused", async () => {
const source = new GazetteerSource();
let hits = 0;
const misses = [];
for (const c of corpus.positives) {
const results = await source.search(c.q, 3);
const ids = results.map((r) => r.id);
if (ids.includes(c.expect)) hits++;
else misses.push({ q: c.q, expect: c.expect, got: ids });
}
const rate = hits / corpus.positives.length;
assert.ok(
rate >= corpus.threshold.top3_hit_rate,
`top-3 hit rate ${(rate * 100).toFixed(1)}% < ${corpus.threshold.top3_hit_rate * 100}% — ` +
`escalation ladder step 1 (alias/normalizer table edits) is required. Misses: ${JSON.stringify(misses)}`,
);
for (const n of corpus.negatives) {
const results = await source.search(n.q, 3);
assert.equal(results.length, 0, `negative "${n.q}" must be refused honestly, got ${JSON.stringify(results.map(r => r.id))}`);
}
});
```
**On failure, the assertion message names the escalation step.** That is D-14's ladder made mechanical: the failure text tells you to pull lever 1 (table edits), then re-run; if lever 1 cannot close the gap, lever 2 (backend #2) is next, then lever 3 (Photon). The phase directory should hold a short decision record naming which lever was pulled and the corpus run that closed it.
### 5. The normalizer as a per-rule vector table (D-09)
```typescript
// Source: D-09 — "Per-rule test vectors; extends by table edit, not code change."
export const NORMALIZER_VECTORS: ReadonlyArray<{
rule: string;
input: string;
expect: string;
}> = [
{ rule: "casefold", input: "KLCC", expect: "klcc" },
{ rule: "casefold", input: "Sunway PYRAMID", expect: "sunway pyramid" },
{ rule: "whitespace", input: " jln ampang ", expect: "jln ampang" },
{ rule: "abbrev:jln", input: "Jln Ampang", expect: "jalan ampang" },
{ rule: "abbrev:tmn", input: "Tmn Melewah", expect: "taman melewah" },
{ rule: "abbrev:kg", input: "Kg Baharu", expect: "kampung baharu" },
{ rule: "abbrev:prsn", input: "Prsn 4", expect: "presint 4" },
{ rule: "abbrev:presint", input: "Presint 4", expect: "presint 4" },
{ rule: "identity:tmn", input: "Tmn", expect: "tmn" },
{ rule: "glued:kg", input: "KampungBaru", expect: "kampung baru" },
{ rule: "glued:klcc", input: "KLCC", expect: "klcc" }, // no known prefix → unchanged
{ rule: "no-mangle:1", input: "Sunway Pyramid", expect: "sunway pyramid" }, // NOT "sunaan pyramid"
{ rule: "no-mangle:2", input: "KLIA", expect: "klia" }, // NOT "kualia lumpuria"
{ rule: "diacritic", input: "Ampáng", expect: "ampang" },
];
```
### 6. ONBD-03 — `addToHomeScreen` with an iOS-honest status read
```typescript
// Source: core.telegram.org/bots/webapps —
// addToHomeScreen() (Bot API 8.0+) — prompts the user to add the Mini App
// to the home screen; the `homeScreenAdded` event fires if the device supports it.
// checkHomeScreenStatus(cb) — cb(status) where status is one of:
// "unsupported" | "unknown" | "added" | "missed"
// Telegram-iOS issue #1729: on iOS the status is ALWAYS "unknown" — the platform
// offers no API to detect whether the shortcut exists.
export function wireHomeScreen(wa: TelegramWebAppLike | null): void {
if (!wa || typeof wa.addToHomeScreen !== "function") return;
wa.addToHomeScreen();
wa.onEvent?.("homeScreenAdded", () => {
// Android/desktop path: the shortcut exists.
});
wa.checkHomeScreenStatus?.((status: string) => {
// "unsupported" → feature unavailable here (Desktop web).
// "unknown" → iOS. Do NOT claim it failed; treat as "possibly added".
// "added" / "missed" → definitive.
});
}
```
> **Important:** `beforeinstallprompt` — the standard web-PWA install event — **does not exist inside a Telegram webview**. The Telegram-native `addToHomeScreen()` is the mechanism. Do not build a web manifest-driven install flow for ONBD-03 (that is the documented-deferred full-PWA story). Also note that **service workers do not work in Telegram Mini Apps on iOS** ([Telegram-Mini-Apps issue #27](https://github.com/Telegram-Mini-Apps/issues/issues/27)) — another reason the full-PWA story stays deferred.
---
## Licensing — the ODbL obligation (D-04 × D-06)
This section exists because it is the one place where two locked decisions collide.
**What the OSMF actually says** (fetched this session):
- OSM data is under the **Open Database License (ODbL) 1.0** — attribution + share-alike for *databases*. [foundation.openstreetmap.org/wiki/Licence_and_Legal_FAQ](http://foundation.openstreetmap.org/wiki/Licence_and_Legal_FAQ)
- *"You must also make the derivative database available to anyone who received your data… If you make a database which includes OSM data and any additional information… then this would be classed a 'derivative database' and should be made available under the Open Database License."* [Legal FAQ](https://openstreetmap.org/wiki/Legal_FAQ)
- *"if you have any data that was derived from OSM — for instance because you used street names from OSM… then you are making a derivative database."* [License/Use Cases, Case 3](https://wiki.openstreetmap.org/wiki/License/Use_Cases)
- *"A 'geocoding database' should **NOT** be considered a trivial transformation."* [Trivial Transformations Guideline](https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline)
- ODbL §4.4: a publicly distributed Derivative Database must be under the ODbL, a later version with the same Licence Elements, or *"a compatible licence."*
- The OSMF's compatibility guidance says licences that require downstream users to use the data on *identical* terms are generally incompatible with ODbL.
**The conclusion:** a curated gazetteer of KL POIs whose names, addresses and coordinates come from OSM is a **Derivative Database**. The software around it may be GPL-3.0 (D-04), but the **data file** carries an ODbL share-alike obligation.
**Recommended resolution — dual-licence, the ecosystem-standard shape:**
| Artifact | Licence | Note |
|----------|---------|------|
| All source code (`bot/`, `web/`, `theme/`) | **GPL-3.0** | D-04. Add the `LICENSE` file this phase (the CONTEXT canonical refs call for it). |
| `bot/src/search/data/kl-gazetteer.json` (and any future gazetteer dump) | **ODbL-1.0** | A data file, not code. Header carries `© OpenStreetMap contributors` + a link to <https://www.openstreetmap.org/copyright>. |
| `bot/test/fixtures/golden-corpus.json` | **GPL-3.0** with the repo | It contains queries and entry ids, no OSM content. |
| Attribution in the product | **Required** | The bot's venue card / replies should carry `© OpenStreetMap contributors` — the web app already ships exactly this string in `web/src/i18n/strings.ts` (`attribution`), so the bot's strings module needs the twin. |
**Why this satisfies D-04 rather than breaking it.** The forkability laws are about *mechanics* — no hardcoded secrets, plain committed data files, one-command deploy, one free-tier account per fork. A dual-licence keeps every one of those: the data file is still a plain committed file, and a fork inherits the ODbL obligation *on that file* — which is the correct, legally-required, entirely workable outcome. It is exactly what OSM-derived datasets in the ecosystem do (the OSM attribution notice is already visible in this project's UI).
**Optional defence-in-depth:** the D-06 Overpass script can *propose* entries, but the maintainer can hand-write the display name, the address string, and the aliases from local knowledge, using OSM only for coordinates and for a sanity check. That reduces (though does not eliminate) the "derived" exposure. **Do not rely on it** — the dual-licence is free.
---
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Photon-on-VPS as the v1 search backend (the roadmap's original Phase-4 note) | **In-Worker curated gazetteer behind a `SearchSource` interface** (D-03) | Decided 2026-10-23 (this phase's CONTEXT.md) | Removes all infra cost and the VPS dependency from Phase 4. Photon moves to backend #3 behind the same interface. |
| Full-text search engine (Tantivy/Meilisearch/Typesense) | **Three-tier in-Worker cascade** (exact → bigram filter → Damerau) | Inherent to the 200–500 row scale | No service to run, no network hop, no query latency. The 10 ms CPU budget is comfortable. |
| Hand-typing the abbreviation expansion rules in code | **An editable rule table** with per-rule vectors (D-09) | Inherent to D-09 | Corpus quality becomes a data-review activity, not a code-review activity. |
| `t.me/<bot>?startapp=` deep link as the OPEN button | **`url` button → `<origin>/?dest=<id>`**, `start_param` as a secondary path | Pitfall 2 (Desktop START-button trap) | One tap on every client; no START-button detour. |
| Web PWA install (`manifest` + `beforeinstallprompt`) for the home screen | **Telegram-native `WebApp.addToHomeScreen()`** (Bot API 8.0+) | Bot API 8.0, Nov 2024 | The only mechanism that works inside a Telegram webview; iOS status is always `unknown`. |
| SQLite/sql.js WASM index in the Worker | **Typed JSON** (D-07) | D-07's own reasoning | ~1.5 MB bundle + cold-start cost avoided; diff-friendly PRs. |
| Geocoding database as a "trivial transformation" of OSM | **Explicitly NOT trivial** per the OSMF guideline | Guideline endorsed by the OSMF board 2014-06-06 | The ODbL share-alike obligation applies — see §Licensing. |
**Deprecated/outdated:**
- **`beforeinstallprompt`** — does not fire inside a Telegram webview. Not a valid ONBD-03 mechanism.
- **Service workers in Telegram Mini Apps on iOS** — confirmed broken ([Telegram-Mini-Apps issue #27](https://github.com/Telegram-Mini-Apps/issues/issues/27), still open with a 2025-09 update). Another reason the full-PWA/offline story stays deferred.
- **OSM's old "geocoding databases may be trivial" hope** — the guideline says the opposite.
- **The roadmap's Phase-4 VPS/Photon/GraphHopper plan** — superseded by D-03/D-05.
---
## Assumptions Log
> All claims tagged `[ASSUMED]` in this research, for the planner and discuss-phase to confirm.
| # | Claim | Section | Risk if Wrong |
|---|-------|---------|------------|
| A1 | The abbreviation table's canonical list (Jln/Jalan, Tmn/Taman, Kg/Kampung, Lorong, Persiaran, Presint/Prsn, Stesen/Stn) is sufficient for the KL corpus. Grounded in the UPU address standard and OSM's KL road list, but **no authoritative KL-specific abbreviation table exists** — this is a curated approximation. | §Pattern 3 | Low-medium. The corpus runner measures the gap; lever 1 (table edits) is the designed remedy. |
| A2 | The KL bounding box `2.95,101.55,3.20,101.80` is the right scope for the Overpass top-up. Plausible from the metro extent, not verified against a canonical "Klang Valley" boundary. | §Code Examples 3 | Low. A wider box costs a longer Overpass run; a narrower one misses outliers. The bbox is a script constant, trivially edited. |
| A3 | `sendVenue` renders acceptably on all Telegram clients (iOS/Android/Desktop) with a `reply_markup` attached. **Not verified on a real device this session** — the API surface is verified, the rendering is not. | §Code Examples 1 | Medium. Rendering is cosmetic; the fallback (a text message with an inline keyboard) is trivial if a client renders the card badly. **Recommend it as a real-client checkpoint.** |
| A4 | The `url` inline button to the Mini App origin behaves identically to a `web_app` button on all clients (single tap, Mini App opens, the app's own query string is preserved). The API surface is verified; the client behaviour is not device-tested. | §Code Examples 1, Pitfall 1 | Medium-high if wrong — this is the ONBD-02 hand-off. **Recommend it as a real-client checkpoint** (the Phase-3 device-matrix pattern applies). |
| A5 | A ~500-row gazetteer with ~8 aliases each, parsed and indexed per request, fits comfortably inside the Workers Free 10 ms CPU budget. The 10 ms figure is verified; the *measured* cost of this specific workload is not. | §Summary finding 4 | Low-medium. The design has headroom by construction (tier-1 O(1) lookup short-circuits most queries), and `wrangler dev` can measure it. |
| A6 | The OSM licensing analysis (curated gazetteer = Derivative Database → ODbL share-alike; dual-licence resolves it) is correct **as an engineering practice**, but this is legal guidance, not legal advice. The OSMF wiki is community guidance and explicitly disclaims being authoritative. | §Licensing | Medium for the *project*, low for the *code*. The dual-licence costs almost nothing and is strictly safer than the alternative. A maintainer who wants certainty should ask the OSMF licensing mailing list. |
| A7 | The `dest_<id>` payload in `start_param` will arrive inside `initData` and be signed by Telegram in the `t.me?startapp=` path (and therefore validated by the existing `/api/session`). The `start_param` field exists in `WebAppInitData` per the docs, but this session did **not** verify its presence in a real signed `initData` sample. | §Code Examples 2 | Low. The primary path (`?dest=` in the URL) does not depend on it. |
| A8 | The Telegram `/cari` command alias scheme (bilingual) has room in `strings.helpMenu` without violating the Phase-2 CS5 gap-freedom gate — the gate currently expects exactly `/help` and `/privacy` and will need extending. | §Pitfall 11 | Low. It is a mechanical test update. |
| A9 | The gazetteer seed can be assembled from public knowledge of KL landmarks (KLCC, Sunway Pyramid, DUKE, PWTC, Pavilion, IJN, etc.) with accurate coordinates. **Coordinates for the seed entries should still be sourced from OSM** to avoid errors — so the ODbL analysis stands regardless of where names come from. | §Pattern 4 | Low for the coordinates; medium for exact addresses, which should come from OSM's `addr:*` tags. |
---
## Open Questions
1. **Should the OPEN button be a `url` button or a `web_app` button?**
- **What we know:** `WebAppInfo` has only `url` — no `start_param` field exists on either button type. The `t.me?startapp=` deep link carries `start_param` but needs an extra START tap on Telegram Desktop. The `url` button with `?dest=<id>` is single-tap everywhere.
- **What's unclear:** Whether a `web_app` button grants the Mini App anything the `url` button does not (it grants `query_id` and basic user info, and permits `answerWebAppQuery`). Phase 4 does not need `answerWebAppQuery` — the bot sends the card.
- **Recommendation:** **Use the `url` button.** Revisit only if Phase 5/6 needs the Mini App to message back on the user's behalf.
2. **Should `/api/session` echo the validated `start_param` back to the webview?**
- **What we know:** `start_param` is inside the signed `initData`, so the existing Phase-2 validation already covers it. The webview currently receives only `{token}` from `/api/session`.
- **What's unclear:** Whether trusting `initDataUnsafe.start_param` client-side is acceptable (D-11 says the threat is only self-pranking) or whether the server should re-derive it.
- **Recommendation:** **Have `/api/session` return the validated `start_param` alongside the token.** One extra field, zero new validation path, and it makes the deep-link payload trustworthy by construction. It also keeps the claim surface (D-08) unchanged — `start_param` is not stored in KV.
3. **Is a result cache worth it?**
- **What we know:** At 200–500 rows the search is sub-millisecond; D-15 permits TTL-only KV caching keyed by normalized query; D-15 also notes the query is user payload (so a cache key derived from it is a fingerprint of the user's activity, even if the content is public place data).
- **What's unclear:** Whether the Workers CPU budget ever gets tight enough to matter — it does not at this scale.
- **Recommendation:** **Ship without a cache.** It is a privacy cost with no performance benefit. If a future backend makes search slow, cache then — and key by a **hash** of the normalized query rather than the query itself.
4. **Where does the shared search module live?**
- **What we know:** Option A (a new npm workspace, e.g. `@araharah/search`) gives one on-disk copy consumed by both `bot/` and `web/`, mirroring the existing `theme/` → `web/` pattern. Options B and C exist but reintroduce duplication risk.
- **What's unclear:** Whether adding a fourth workspace is acceptable to the maintainer, given D-04's "one-command deploy" law and the existing three-workspace layout.
- **Recommendation:** **Option A.** It is the only option that makes D-11's "id mapping stays in sync by construction" literally true, and it lets the corpus runner import the identical module the production paths use.
5. **How large should the seed gazetteer be, and who curates it?**
- **What we know:** D-07 says 200–500 rows; D-13 says 60–100 corpus queries; D-06 says hand-curated seed + Overpass top-up. The coverage that matters is *recognisable KL landmarks*, not exhaustiveness.
- **What's unclear:** Whether 200 rows can cover the corpus's ~60–100 positives plus enough alternates to make D-02's shortlist meaningful.
- **Recommendation:** **Seed with ~120–150 entries** covering malls, transit hubs, major hospitals, universities, government complexes, parks, and ~30 kampung/neighbourhood names — enough for the corpus and for a 3–5 item shortlist per query. Grow via the Overpass script.
---
## Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Node.js | test runner, build scripts | ✓ | **v24.18.0** | — |
| npm | workspace installs, scripts | ✓ | 11.16.0 | — |
| `grammy` | `/cari` handler, `sendVenue` | ✓ (installed) | **1.46.0** | — |
| `wrangler` | dev on 8787, deploy | ✓ (installed) | 4.151.3 | — |
| `typescript` | typecheck | ✓ (installed) | 5.9.3 | — |
| `vite` | web build (gazetteer copy) | ✓ (installed) | 8.3.5 | — |
| `playwright` + browsers | ONBD-02 capture evidence | ✓ (cached) | chromium-1248, webkit-2272, firefox-1511 | — |
| Cloudflare Workers | runtime | ✓ (account + deploy) | Free plan, 10 ms CPU / 100k req/day | — |
| `chat_sessions` KV | (not needed — no cache recommended) | ✓ (binding configured) | id `59f58a54…` | — |
| **Overpass API** (network) | D-06 top-up script | ✓ (reachable) | `overpass-api.de` — **returns 406 without a User-Agent header** (verified this session) | Other public instances: `overpass.kumi.systems`, `overpass.private.coffee`. The script sets a UA (Pitfall 4). |
| **Nominatim** (network) | D-06 fallback for a missed named place | ? (not probed this session) | `nominatim.openstreetmap.org` | Strict usage policy — 1 req/s, identifying UA, no bulk. Maintainer-tool use only. |
| **Real Android device** | ONBD-02/ONBD-03 real-client gate | **?** | — | **None.** The Phase-3 device matrix pattern applies: arm a checklist, let the user walk it. |
| **Real iOS device** | ONBD-03 `addToHomeScreen` (iOS status is always `unknown`) | **?** | — | **None.** Phase 3 recorded the iOS leg as UNVERIFIED (no device). |
| **Telegram Desktop** | Pitfall 2 (START-button trap) verification | **?** | — | **None.** The `url`-button choice sidesteps the trap; a Desktop spot-check still confirms it. |
**Missing dependencies with no fallback:**
- Real devices for the ONBD-02/ONBD-03 client matrix. The Phase-3 precedent (`.planning/phases/03-mini-app-shell-geolocation/03-DEVICE-MATRIX.md`) is the model: a written checklist with the deployed URL, checked by the user. **The planner must surface this as a checkpoint before the exit-gate task, not discover it at execution.**
**Missing dependencies with fallback:**
- Overpass instance availability — three public instances exist; the script should accept an override.
- A real Telegram client for the "one message with venue card + buttons" confirmation — the bot-side unit tests cover the *payload shape*; the *rendering* needs a human. (This is assumption A3.)
---
## Validation Architecture
> `.planning/config.json` → `workflow.nyquist_validation: true` (confirmed present and `true`). Included per protocol.
### Test Framework
| Property | Value |
|----------|-------|
| Framework | **Hand-rolled zero-dependency runner** — `bot/test/runner.mjs` + `bot/test/harness.mjs` (`test(name, fn)` registry). `web/test/runner.mjs` is the same shape (copied verbatim, workspace-agnostic). No vitest/jest — by design (repo precedent since Phase 1's `theme/scripts/audit.test.mjs`). |
| Config file | none (plain `.mjs` + `.ts` imported via dynamic `import()` — the runner already does `await import("../src/i18n/strings.ts")`, which works because Node 24 strips TypeScript types natively). |
| Quick run command | `npm test -w bot` (bot suites) / `npm test -w web` (web suites) / `npm run bot:test` + `npm run web:test` from the root |
| Full suite command | `npm run bot:test && npm run web:test && npm run bot:typecheck && npm run web:typecheck && npm run build -w theme && npm run build -w web` |
| Client/render checks | `npm run capture -w web` — Playwright against `web/dist-harness`; the existing harness already races `RAMS-BASEMAP-READY` → `ARAHARAH-LIVE` and enforces a `MIN_BYTES` blank-bar check. A `?dest=<id>` boot profile slots straight into the existing capture list. |
| Corpus runner | **A new suite in the same shape**: `bot/test/corpus.test.mjs` — imports `bot/test/fixtures/golden-corpus.json` and the real `GazetteerSource`, asserts the D-14 gate. No new runner, no new framework. |
**⚠️ ESM/TS caveat (carried from Phase 3's research):** the runner dynamic-imports `.ts` files directly. This works on Node 24 (type-stripping) and is the established repo pattern (`bot/test/commands.test.mjs` does `await import("../src/i18n/strings.ts")`). **The shared search module must therefore be importable from plain Node without a bundler** — i.e. no `import ... with { type: "json" }` assertion in the *source* module if it would break the Node import; load the JSON via `readFileSync` + `JSON.parse` against a path resolved from `import.meta.url` if the assertion proves fragile. **Test this early** — it is the single most likely mechanical blocker in the phase. (Note the corpus fixture itself is loaded by the *test*, not by the source module, so it can use a JSON import assertion without affecting production code.)
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| **SRCH-01** | Query "KLCC" returns the KLCC entry in the top-3 | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | The `SearchSource` interface is satisfied by `GazetteerSource` and the seam is the only entry point the handler uses | unit (structural) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | A gazetteer entry with no match yields an **empty** array (not a nearest-neighbour fabrication) | unit | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | `backendId` identifies the active backend — the escalation record needs it (D-14) | unit | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-01** | The top-3 hit rate over the corpus positives is **≥ 90%** — the D-14 threshold, as an executable assertion | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | "Jln Ampang" and "Jalan Ampang" normalize to the same string and resolve to the same entry id | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | "Tmn" ↔ "Taman", "Kg" ↔ "Kampung", "Prsn" ↔ "Presint" round-trip (bidirectional table) | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | Glued-word split: "KampungBaru" → "kampung baru"; "KLCC" unchanged | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | No mangling: "Sunway Pyramid" and "KLIA" survive normalization unchanged | unit (normalizer vectors) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | Case/whitespace folding is idempotent | unit | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | A transposition typo ("Sunway Pyrmid") resolves to the same entry as the canonical spelling | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-02** | A missing-key typo ("Sunwy Pyramid") resolves to the same entry | unit (corpus gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The handler sends **exactly one** message per `/cari` — the venue card carries the reply markup | unit (payload shape, fake bot) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The venue payload carries `latitude`, `longitude`, `title`, `address` (the four required `sendVenue` fields) | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The OPEN button's URL carries the destination id and points at the Mini App origin | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | 2–4 alternates ride on the **same** message's inline keyboard (D-02) | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | A no-match query sends an honest refusal with **no** pin and no inline keyboard | unit (payload shape) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The venue card's address/title carry the OSM attribution string (ODbL) | unit (copy gate) | `npm test -w bot` | ❌ Wave 0 |
| **SRCH-03** | The search copy exists in **both** `en` and `ms` with identical key-sets (the Phase-2 CS1 law extends) | unit (copy gate) | `npm test -w bot` | ❌ Wave 0 |
| **ONBD-02** | `dest_<id>` maps to a destination in the webview's gazetteer copy — id → name/lat/lng/address | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-02** | A **missing or unknown** id degrades to the normal Phase-3 boot (no crash, no blank) | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-02** | The webview resolves the id from `location.search` **and** from `initData.start_param` | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-02** | The harness boot profile `?dest=<id>` renders the destination preloaded (capture evidence) | integration (Playwright) | `npm run capture -w web` | ❌ Wave 0 |
| **ONBD-02** | The gazetteer id charset is enforced: every id matches `/^[a-z0-9-]+$/` | unit (schema gate) | `npm test -w bot` | ❌ Wave 0 |
| **ONBD-02** | The bot and web copies of the gazetteer expose the **same id set** (drift gate — the D-11 "in sync by construction" assertion, made executable) | unit | `npm test -w bot` (+ a web-side check) | ❌ Wave 0 |
| **ONBD-03** | `addToHomeScreen` is wired when available and is a **no-op** when the API is absent (Desktop web) | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-03** | `checkHomeScreenStatus` handles all four statuses, treating `unknown` as "possibly added" (iOS), never as failure | unit (web) | `npm test -w web` | ❌ Wave 0 |
| **ONBD-03** | A boot with **no** `start_param` and **no** `?dest=` lands in the Phase-3 staged boot (search-ready map) — no crash, no blank | unit (web) + integration (capture) | `npm test -w web`, `npm run capture -w web` | ❌ Wave 0 |
| **Privacy (D-15)** | No KV write in the search path; the query text never reaches a log line | unit (grep gate, mirroring the Phase-2 `privacy.test.mjs`) | `npm test -w bot` | ❌ Wave 0 |
| **Privacy (D-15/GEO-03)** | The webview's destination is **memory only** — no storage API referenced | unit (grep gate, `web/test/no-storage.test.mjs` precedent) | `npm test -w web` | ❌ Wave 0 |
| **Licensing (D-04)** | The gazetteer JSON carries an ODbL notice + OSM attribution; a `LICENSE` (GPL-3.0) exists | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 |
| **D-09 (structural)** | The abbreviation table is data, not code — the normalizer reads it from a table module | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 |
| **D-11 (structural)** | No crypto/HMAC/signature in the deep-link payload path | unit (structural grep) | `npm test -w bot` | ❌ Wave 0 |
| **Corpus integrity** | Every corpus positive's `expect` id exists in the gazetteer; every id is unique; ids match the charset | unit (schema gate) | `npm test -w bot` | ❌ Wave 0 |
| **Real-device gate** | ONBD-02/ONBD-03 client behaviour: single-tap OPEN, home-screen shortcut, iOS `unknown` | **manual-only** | — | n/a — Phase-3 device-matrix pattern |
| **SRCH-03 rendering** | The venue card + inline buttons render correctly in a real Telegram client | **manual-only** | — | n/a — assumption A3 |
### Sampling Rate
- **Per task commit:** `npm run bot:test` (search module, normalizer, corpus, handler) and `npm run web:test` (destination resolution) — whichever workspace the task touched.
- **Per wave merge:** `npm run bot:test && npm run web:test && npm run bot:typecheck && npm run web:typecheck && npm run build -w theme && npm run build -w web`
- **Phase gate:** full suite green **AND** the corpus gate green at ≥90% **AND** `npm run capture -w web` produced the ONBD-02 evidence capture **AND** the real-device client matrix walked by the user (manual, blocking — Phase-3 precedent).
### Wave 0 Gaps
- [ ] `bot/src/search/` module tree — `types.ts`, `gazetteer.ts`, `normalize.ts`, `rules/abbreviations.ts`, `score.ts`, `source.ts` (the shared module; Wave 0 because everything depends on it)
- [ ] `bot/src/search/data/kl-gazetteer.json` — the seed gazetteer (~120–150 entries) with ODbL header + attribution
- [ ] `bot/src/bot/commands/cari.ts` — the `/cari` handler
- [ ] `bot/test/normalizer.test.mjs` — per-rule vectors (D-09)
- [ ] `bot/test/corpus.test.mjs` — the D-13/D-14 executable gate
- [ ] `bot/test/fixtures/golden-corpus.json` — the pinned corpus
- [ ] `bot/test/cari.test.mjs` — payload-shape gates (one message, venue fields, OPEN url, alternates, honest refusal, attribution)
- [ ] `bot/scripts/overpass-topup.mjs` — the D-06 repo tool
- [ ] `web/src/search/resolve-destination.ts` — id → destination (memory only)
- [ ] `web/test/destination-preload.test.mjs` — ONBD-02 gates
- [ ] `web/index.html` — the `?dest=<id>` harness boot profile
- [ ] `bot/src/i18n/strings.ts` — MODIFIED: search copy en/ms + `/cari` in `helpMenu` + the OSM attribution string
- [ ] `bot/test/commands.test.mjs` — MODIFIED: CS5/CS6 must now expect `/cari` in the menu (the Phase-2 gap-freedom gate is a **hard gate that will fail** until this is updated)
- [ ] `LICENSE` (GPL-3.0) + `LICENSES.md` or an ODbL notice for the data file
- [ ] Shared-module wiring — a fourth npm workspace (e.g. `@araharah/search`) so `bot/` and `web/` import one copy
*(No gap: the runner and harness already exist in both workspaces. Every suite listed above is new.)*
---
## Security Domain
> `security_enforcement` is absent from `.planning/config.json` → treated as enabled. Included per protocol.
### Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| V2 Authentication | yes | The webview's `initData` is HMAC-validated by the **existing** Phase-2 `POST /api/session` (never re-implemented). A `start_param` read from `initDataUnsafe` is untrusted unless `/api/session` echoes the validated value (Open Question 2). |
| V3 Session Management | yes | Unchanged from Phase 2 — opaque tokens, TTL-only KV, rotate-on-new. The search path adds **no** session or cache state. |
| V4 Access Control | yes | The search module is pure and has no I/O; the `/cari` handler answers only the chat the command came from (`ctx.chat.id` from the update, never from user input). No endpoint accepts a search query from the browser. |
| V5 Input Validation | yes | The query string is length-capped before normalization (a hostile 100 KB argument must not reach the scorer); the gazetteer id is validated against `/^[a-z0-9-]+$/`; the `start_param` is checked against the Telegram charset before it is used as a map key. |
| V6 Cryptography | **no** (by design) | D-11: *"No crypto: the payload's only threat is a user pranking themselves."* The deep-link payload is a plain id into a public gazetteer. **A structural gate must assert that no HMAC/signature code enters the search or deep-link path.** |
| V9 Communications | yes | The Overpass script is a repo tool with an identifying User-Agent; it is never invoked at runtime. The Worker makes **no** outbound search requests (backend #1 is in-Worker). |
| V14 Configuration | yes | No new env vars, no new secrets (D-04's "env only" law already holds). The Mini App origin the OPEN button points at is **not** a secret — it is public and already in `bot/wrangler.jsonc` / the deployed URL. |
### Known Threat Patterns for Cloudflare Workers + Telegram + a curated gazetteer
| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| **Query-text leakage** (the query is user payload — it can reveal where a user is going or where they live) | Information Disclosure | D-15: category-only logging; the query is never written to KV or to a log line. A grep gate pins this (Phase-2 `privacy.test.mjs` precedent). **Do not add a result cache** (a normalized-query cache key is an activity fingerprint even when the content is public). |
| **Forged deep link** (a user crafts `?dest=<other-id>`) | Tampering | No impact — the id resolves to a *public place*. D-11's own assessment. The id is not a capability and grants nothing. |
| **Gazetteer id injection** (an id containing characters that break the URL or the `start_param` charset) | Tampering | Enforce `/^[a-z0-9-]+$/` on every entry id at load time; reject the entry (fail closed) rather than emit a bad URL. |
| **Hostile query argument** (megabyte-long string to burn CPU) | DoS | Cap the raw query length before normalization (e.g. 256 chars — no real destination exceeds it). Cap the normalized length too. The tier-1 hash lookup is O(1) regardless. |
| **CPU exhaustion via the scorer** | DoS | The three-tier cascade bounds the expensive algorithm to a handful of candidates. Workers Free's 10 ms CPU limit is a backstop. `wrangler dev` can measure the real figure. |
| **Overpass abuse** (a maintainer script that hammers the public instance) | DoS / reputation | One request per run, identifying User-Agent, manual invocation only. Overpass's per-IP rate limit (2 slots, requests queued up to 15 s) is the guardrail. |
| **Stale-webview dataset** (the web bundle's gazetteer lags the bot's) | Tampering (functional) | Structural: one shared module (Option A) + an id-set drift gate in the test suite. The `npm run deploy` chain rebuilds both from one source. |
| **Cross-chat leakage** (a search result sent to the wrong chat) | Information Disclosure | The handler replies to `ctx.chat.id` from the update only; no chat id is accepted from user input. |
| **XSS / markdown injection** (a query echoed back into a reply) | Tampering | Telegram's parse mode handles it, but the safest design is to **not echo the query** in the reply at all (the clinical Rams voice says what it found, not what was typed). If the query must appear, escape it or send without a parse mode. |
---
## Sources
### Primary (HIGH confidence)
- **Official Telegram Bot API** — <https://core.telegram.org/bots/webapps> (fetched 2026-10-23): `addToHomeScreen()` / `checkHomeScreenStatus()` and the four statuses; the `start_param` field and the `tgWebAppStartParam` GET parameter; the `t.me?startapp=` deep-link form; inline vs direct-link Mini App behaviour; `mode=compact`.
- **Official Telegram Bot API reference** — <https://core.telegram.org/bots/api#webappinfo> (fetched 2026-10-23): method list; `sendVenue` exists; `setMyCommands` scope rules.
- **Installed `grammy@1.46.0` + `@grammyjs/types`** — read directly from `node_modules` this session: `api.d.ts:410` (`sendVenue` signature), `context.d.ts:679-690` (`replyWithVenue`), `methods.d.ts:735-782` (`sendVenue` args including `reply_markup`), `markup.d.ts:11-60` (`InlineKeyboardButton` variants incl. `UrlButton` and `WebAppButton`), `markup.d.ts:219-222` (`WebAppInfo { url: string }` — **only** `url`), `message.d.ts:1072-1086` (`Venue` fields incl. `foursquare_id` / `foursquare_type` / `google_place_id` / `google_place_type`).
- **Cloudflare Workers Limits** — <https://developers.cloudflare.com/workers/platform/limits/> (fetched 2026-10-23, page dated 2026-10-12): Free plan **10 ms CPU / HTTP request**, 100,000 req/day, 128 MB memory, 64 MiB uncompressed bundle, **1 second startup**, "avoid expensive work in global scope", "Most Workers consume ~2.2 ms per request".
- **OSMF Legal FAQ** — <https://openstreetmap.org/wiki/Legal_FAQ>: derivative-database definition; the requirement to make the derivative database available.
- **OSMF Licence/Use Cases** — <https://wiki.openstreetmap.org/wiki/License/Use_Cases> (rev. 2025-01-28): Case 3 — *"if you have any data that was derived from OSM… then you are making a derivative database"*; Case 2 — a pure-OSM publication is a "produced work" needing attribution.
- **OSMF Trivial Transformations Guideline** (endorsed by the OSMF board 2014-06-06) — <https://wiki.openstreetmap.org/wiki/Open_Data_License/Trivial_Transformations_-_Guideline>: *"A 'geocoding database' should **NOT** be considered a trivial transformation."*
- **OSMF Licence Compatibility** — <http://foundation.openstreetmap.org/wiki/Licence/Licence_Compatibility>: identical-terms licences generally incompatible with ODbL.
- **UPU S42 Malaysian address standard** — <https://www.upu.int/UPU/media/upu/PostalEntitiesFiles/s42/mysEn.pdf>: the canonical Malaysian address grammar (primary thoroughfare type = Jalan / Taman / Kampung; "Persiaran"; "Presint"; district level = Taman / Kampung / Seksyen) with worked address examples.
- **OSM Overpass QL reference** — <https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL>: `bbox` order (south,west,north,east); `nwr` filter; `out center`; `out:csv`/`out:json`; `[timeout:N]` / `[maxsize:N]`.
- **Overpass API User's Manual — Rate Limit** — <https://dev.overpass-api.de/overpass-doc/en/preface/commons.html>: 2 slots/IP, requests queue up to 15 s, HTTP 429 when denied; small-request priority; identifying UA recommended.
- **Malaysian address data** — UPU addressing unit (<https://www.upu.int/UPU/media/upu/PostalEntitiesFiles/addressingUnit/mysEn.pdf>) and the OSM-derived KL road list (<https://en.wikipedia.org/wiki/List_of_roads_in_Kuala_Lumpur>) — for the abbreviation vocabulary and the "Jalan / Lorong / Persiaran" naming pattern.
- **Repo codebase** — read directly this session: `bot/src/{webhook.ts,bot/bot.ts,config.ts,i18n/strings.ts,kv/sessions.ts}`, `bot/test/{runner.mjs,harness.mjs,commands.test.mjs}`, `bot/wrangler.jsonc`, `web/src/{main.ts,telegram/webapp.ts,i18n/strings.ts,mock/telegram-mock.ts}`, `web/index.html`, `web/test/runner.mjs`, `web/scripts/capture.mjs`, and the Phase-2/Phase-3 summaries and RESEARCH files.
### Secondary (MEDIUM confidence)
- **Telegram-iOS issue #1729** — <https://github.com/TelegramMessenger/Telegram-iOS/issues/1729>: *"Home screen status detection always returns 'unknown' on iOS… On iOS it is not possible to determine whether a Mini App has been added to the Home Screen."* (Official client repo issue; the OSM wiki pages above also carry a community-guideline disclaimer.)
- **telegramdesktop issue #28694** — <https://github.com/telegramdesktop/tdesktop/issues/28694>: the Desktop START-button requirement for `t.me` deep links.
- **Telegram-Mini-Apps issue #27** — <https://github.com/Telegram-Mini-Apps/issues/issues/27>: service workers do not work in Telegram Mini Apps on iOS (still open).
- **Telegram Mini Apps platform docs** — <https://docs.telegram-mini-apps.com/platform/start-parameter> and <https://docs.telegram-mini-apps.com/platform/events>: the `start_param` charset (`/^[\w-]{0,512}$/`), the `home_screen_added` / `home_screen_checked` events and the four statuses.
- **Cohen, W. W.; Ravikumar, P.; Fienberg, S. E. (2003)**, *A Comparison of String Distance Metrics for Name-Matching Tasks*, KDD Workshop — <https://www.cs.cmu.edu/~wcohen/postscript/ijcai-ws-2003.pdf>: Jaro/Jaro-Winkler are "intended primarily for short strings"; prefix weighting behaviour.
- **taxify fuzzy-matching vignette** — <https://cran.r-project.org/web//packages//taxify/vignettes/fuzzy-matching.html>: Damerau-Levenshtein charges 1 edit for a transposition (Levenshtein charges 2); Jaro-Winkler helps on 3–6-character names; threshold guidance.
- **RapidFuzz reference** — <https://github.com/StrategicProjects/RapidFuzz>: normalized Damerau-Levenshtein semantics (0..1) used to define the tier-3 similarity conversion.
- **npm registry probes** (2026-10-23): `fuse.js` → 7.5.0, `fastest-levenshtein` → 1.0.16, `string-similarity` → 4.0.4, `levenshtein-edit-distance` → 3.0.1 (all rejected); `grammy` 1.46.0, `wrangler` 4.151.3, `vite` 8.3.5, `typescript` 5.9.3, Node 24.18.0, npm 11.16.0 (all already installed).
### Tertiary (LOW confidence — marked for validation)
- **The exact `sendVenue` + `reply_markup` rendering on iOS/Android/Desktop clients** — the API surface is verified, the rendering is not device-tested (assumption A3).
- **The `url` inline button opening the Mini App with its query string intact on every client** — not device-tested (assumption A4).
- **The exact KL bounding box** for the Overpass top-up (assumption A2).
- **The sufficiency of the abbreviation table** for KL (assumption A1) — the corpus runner is the instrument that settles it.
---
## Metadata
**Confidence breakdown:**
- **Standard stack:** HIGH — every library is already installed and its type surface was read directly from `node_modules` this session; the only new dependency decision is *not to add one*, and the rejected alternatives were registry-probed.
- **Architecture:** HIGH — the `SearchSource` seam, the three-tier cascade, and the shared-module pattern are all grounded in verified API surfaces, the 10 ms CPU budget, and the repo's own precedent. MEDIUM on the `url` vs `web_app` button choice (device-dependent).
- **Pitfalls:** HIGH — every pitfall is anchored to a fetched doc, an installed typing, an official repo issue, or a measured behaviour (the Overpass 406). The device-dependent ones are labelled as assumptions.
- **Licensing:** MEDIUM-HIGH — the analysis rests on three OSMF wiki pages plus the ODbL text, all fetched this session; but OSMF guidance is community guidance, and this is not legal advice. The recommended dual-licence is the conservative, ecosystem-standard resolution.
**Research date:** 2026-10-23
**Valid until:** 2026-11-06 (30 days) — the Telegram and Cloudflare surfaces are stable; the assumption that needs re-checking first is the Overpass instance's UA behaviour and the OSM licence guidance (both move slowly).
---
*Phase: 04-destination-search-kl-gazetteer*
*Research complete — planner may create PLAN.md files.*
Reference in New Issue View Git Blame Copy Permalink