146 KiB
Phase 4: Destination Search + KL Gazetteer - Research
Researched: 2026-10-23
Domain: Telegram Bot API venue cards + inline keyboards + Mini App deep-link hand-off; Malay/Malaysian place-name normalization; in-Worker fuzzy search over a curated gazetteer; OpenStreetMap Overpass candidate generation; ODbL/GPL-3.0 licensing
Confidence: HIGH for Telegram Bot API surface (official core.telegram.org type pages fetched this session), HIGH for the Malay abbreviation table (official JUPEM gazetteer tables + Nominatim variants-ms.yaml fetched this session), HIGH for ODbL text (fetched this session), MEDIUM for the deep-link parameter delivery mechanism (see §"Deep-link risk" — the mechanism is documented but the exact inline-button behaviour is contested), HIGH for the environment findings (probed this session)
<user_constraints>
User Constraints (from CONTEXT.md)
Locked Decisions
Search interaction surface
- D-01: Search entry is the
/caricommand in chat (with/pergi_kereserved for Phase 5 semantics; bilingual alias scheme per Phase 2 D-05). Free-text-to-bot stays reserved; no inline mode; no in-webview search UI this phase (a shared search module keeps that door open). - D-02: Shortlist presentation: the best result ships immediately as the venue card with its OPEN button; 2–4 alternates ride as inline buttons on the same message. One tap to proceed, one tap to switch — no second message round.
- D-03: Search backend is a pluggable
SearchSourceinterface implemented in-Worker: curated KL gazetteer queried in-Worker, $0 infrastructure. VPS/Photon provisioning defers to Phase 5 (Valhalla's road graph needs the VPS anyway). Photon remains a documented escalation path behind the interface.
Forkability + license (user-surfaced design law)
- D-04: GPL-3.0 is the project license. Forkability laws are binding from this phase: no hardcoded secrets (env only — already true), data as plain committed files, one-command deploy, one free-tier Cloudflare account per fork.
- D-05: The user's Malaysia-only WASM geocoder vision (tantivy/sqlite-trigram index, statically hosted, client-side webview search — a genuine privacy win) is captured as SearchSource backend #2, an OSS-track artifact after v1. The JSON↔SQLite converter script keeps its dataset format reachable from day one. Backend #3 = Photon-on-VPS. Neither ships in this phase.
Gazetteer + Malay normalizer
- D-06: Entries are hand-curated seed + a repo-script Overpass one-shot that proposes top-up candidates for maintainer approval into the curated file. The script is a repo tool, not a runtime dependency.
- D-07: Canonical format is typed JSON in the repo (V8-native parse, no WASM in the Worker bundle, diff-friendly PRs). Rationale: at ~200–500 rows, SQLite/sql.js-in-Worker is ~1.5MB + cold-start overhead of pure cost.
- D-08: Aliases live natively inside each entry (
"KLCC"→ Suria KLCC, KLCC Park; IJN, PWTC, Stesen/Station…) — locals' shorthand is the main corpus quality lever. - D-09: The Malay normalizer is a rule table + folding + fuzzy-residual module: bidirectional abbreviation table (Jln↔Jalan, Tmn↔Taman, Kg↔Kampung, vowel-drop-aware Presint/Prsn forms), case/whitespace folding, glued-word split — applied to BOTH the query and entry aliases at index time. Fuzzy scoring (Damerau) is only the residual net for genuine typos on longer words. Per-rule test vectors; extends by table edit, not code change.
Venue card + deep-link hand-off
- D-10: The venue card is Bot API
sendVenue(native pin/name/address semantics SRCH-03 names) with inline buttons: OPEN IN ARAHARAH (web_appbutton carryingstartapp) + alternates. - D-11: The deep link carries a plain gazetteer entry id (
?startapp=dest_<id>). The webview maps id → destination from its own copy of the same gazetteer module (shared module, build-time). No crypto: the payload's only threat is a user pranking themselves; stale-webview datasets re-map via the shared module version. - D-12: ONBD-03 scope: minimal PWA hooks —
addToHomeScreenprompt handling + a startapp-less boot landing in the search-ready map (existing Phase 3 staged boot covers it). Full PWA manifest/offline story is documented-deferred.
Acceptance corpus + thresholds (the bake-off close)
- D-13: A pinned golden corpus in the repo (~60–100 queries): golden landmarks (KLCC, Sunway Pyramid, DUKE), abbreviation forms, typo set (transpositions, missing keys), glued words, aliases, kampung-style names, plus deliberate negatives (out-of-KL: "Kuching", "Penang") that must get an honest "not in v1 area" answer, never a fabricated pin. Each positive pins its expected entry id. Reviewers extend by editing one file.
- D-14: The bake-off closes on: top-3 contains the expected id for ≥90% of positives AND negatives are refused honestly. Below threshold, escalation order is fixed: (1) alias/normalizer table edits → re-run; (2) SearchSource backend #2 (WASM geocoder); (3) Photon-on-VPS. The phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question.
Privacy (carried + made specific)
- D-15: Search queries are user payload: category-only logging (Phase 2 law) — no query text persisted. KV caching of search results, if any, is TTL-only with short TTL and keyed by normalized query (content is public place data, never user identity). Venue choices are not stored.
OpenCode's Discretion
- Scoring function shape (weighted trigram/Jaro vs Damerau) within the rule-table architecture — corpus decides empirically
- Alternates ranking + tie-breaking details; "not in v1 area" copy (en/ms, clinical tone)
- Overpass query shape for the top-up script; gazetteer entry schema fields beyond (id, name, aliases, lat, lng, category)
- D1/KV TTL result-cache mechanics or none (perf is likely fine without at this scale)
Deferred Ideas (OUT OF SCOPE)
- Malaysia-only WASM geocoder (tantivy/sqlite-trigram), statically hosted, client-side webview search — SearchSource backend #2, OSS track after v1
- Photon-on-VPS — SearchSource backend #3; VPS provisioning happens in Phase 5 for Valhalla regardless
- In-webview search UI — needs a UI-SPEC round; shared module already keeps it cheap
- Full PWA (manifest, offline shell) — documented-deferred beyond minimal ONBD-03 hooks
- Free-text bot search + inline mode — reserved interaction surfaces
- Log-derived corpus (real queries) — becomes possible only post-launch (Phase 7+) </user_constraints>
<phase_requirements>
Phase Requirements
| ID | Description | Research Support |
|---|---|---|
| SRCH-01 | User can type a destination query (e.g. "KLCC") and receive a shortlist containing the right KL-metro landmark/POI | /cari command handler + SearchSource interface + curated gazetteer. §"SearchSource interface" defines the contract; §"Standard Stack" defines the JSON entry schema; §"Golden corpus" defines the acceptance mechanism that proves "KLCC" → Suria KLCC. §"Anti-patterns" warns against a bare string-equality match. |
| SRCH-02 | User gets useful results despite abbreviations and typos (Jln↔Jalan, Tmn↔Taman, missing spaces, common Malay address shorthand) | §"Malay normalizer" — the full JUPEM-derived bidirectional abbreviation table (fetched verbatim from Nominatim variants-ms.yaml), the vowel-drop Presint/Prsn rule, glued-word splitting, and the Damerau-Levenshtein residual. §"Standard Stack" fixes the scoring shape. §"Pitfall 3" covers the "abbreviation only applies to generic names, not proper names" rule (MS 2039:9). |
| SRCH-03 | For a chosen destination, a native venue card is sent in chat, confirming where we're going before navigation opens | §"sendVenue" — the exact Bot API parameter set (all four required fields), the reply_markup inline-keyboard slot, and the constraint that sendVenue accepts no text field (so the "one line confirming the destination" must be a separate message or the venue's title/address themselves). §"Inline keyboards" for the alternates + OPEN button. |
| ONBD-02 | User can open the Mini App from a search result with their destination preloaded (no re-entry in the webview) | §"Deep-link risk" — the single highest-risk item in this phase. Documents that start_param is officially scoped to direct links / startattach, that the web_app inline button carries a raw URL, and gives the planner three concrete carriers ranked by risk, plus the write-both-readers fallback. WebAppInfo.url (official type page) confirms the button URL is an arbitrary HTTPS URL. |
| ONBD-03 | User can add araharah to their phone home screen and launch search/navigation from there (addToHomeScreen + startapp deep-link) | §"ONBD-03 feasibility" — addToHomeScreen() / checkHomeScreenStatus() are Bot API 8.0+ only, the iOS limitation (checkHomeScreenStatus always returns "unknown" on iOS — confirmed upstream issue #1729, closed as expected platform behaviour), the TelegramWebAppLike extension shape, and the fact that the home-screen shortcut opens the Main Mini App (so startapp is available there even though it is not on an inline-button launch). |
| </phase_requirements> |
Summary
Phase 4 is the phase where the project stops being a shell and starts being useful — and it carries three materially different risks that the plan must not conflate.
Finding 1 — the deep-link mechanism in D-11/D-12 is the riskiest thing in the phase, and the documentation does not say what D-11 assumes. Telegram's official docs say the start_param field in WebAppInitData is "the value of the startattach parameter, passed via link. Only returned for Mini Apps when launched from the attachment menu via link", and separately that startapp is populated when "a non-empty startapp parameter is included in the link" — where "the link" means a direct link (https://t.me/bot?startapp=…) or a main Mini App link, not a web_app inline keyboard button. But the web_app button's WebAppInfo.url is documented as "An HTTPS URL of a Web App to be opened with additional data" — i.e. an arbitrary URL, and community reports confirm query params on that URL survive into the webview. D-11's "plain gazetteer entry id" payload is fine; the carrier is not settled. The planner must pick a carrier explicitly and the plan must carry a fallback. §"Deep-link risk" lays out three ranked carriers with their evidence.
Finding 2 — the Malay abbreviation table does not need to be invented; it exists, it is official, and it is already machine-readable. JUPEM (the Malaysian mapping authority) publishes a normative abbreviation table in every state's gazetteer, and the OSM Nominatim project transcribed it into settings/icu-rules/variants-ms.yaml (merged April 2026, PR #4056). Both were fetched verbatim this session and are reproduced in §"Malay normalizer". That table already covers Jln→Jalan, Tmn→Taman, Kg→Kampung, Bkt→Bukit, Sg→Sungai, Bdr→Bandar, Psrn→Persiaran, Lbh→Lebuh and more, with the Malaysia-specific and shared variants split exactly the way the normalizer needs them. The planner should vendor this table (with its JUPEM attribution) rather than author a smaller one. One important extra rule the table does not encode comes from MS 2039:9 — an abbreviation may apply to a generic geographic term but not to a name that is part of a proper name ("Jln Taman Desa" ✓, "Jln Tmn Desa" ✗). That rule is a corpus-level invariant the normalizer needs.
Finding 3 — the Overpass dependency is real, and this machine reaches the primary endpoint only with a User-Agent header. overpass-api.de returns HTTP 406 on every request that lacks an identifying User-Agent — including a bare GET / — and HTTP 200 with one. (Verification pass, same host/same paths, varying only the header: GET /api/status no-UA → 406 / with-UA → 200; GET / no-UA → 406 / with-UA → 200; POST /api/interpreter no-UA → 406 / with-UA → 200 with 19 KB of real KL data.) The other public mirrors are unusable from here — overpass.kumi.systems and overpass.private.coffee fail to connect (HTTP 000), and overpass.osm.ch is a Switzerland-only regional instance that returns 400. Nominatim (nominatim.openstreetmap.org) and Geofabrik (download.geofabrik.de/asia/malaysia-singapore-brunei-latest.osm.pbf) are reachable and return correct KL data. This does not block the phase — D-06 says the Overpass script is a repo tool for maintainers, not a runtime dependency — but the planner must know that (a) the script's User-Agent is load-bearing, not cosmetic, (b) capacity 502 responses are intermittent and normal (a 200 and a 502 were observed minutes apart on the same host), and (c) the top-up script's automated test must target a committed fixture, not the live service.
Finding 4 (bonus) — the licensing is compatible, but only if the gazetteer is treated as data, not code. ODbL §2.3(a) explicitly excludes computer programs from the license, and OSMF's guidelines confirm OSM data is ODbL while code is separately licensed. The precedent is live: cbeier-studio/Xolmis is GPL-3.0 and ships ODbL data in the same repo with an explicit dual-license note. But ODbL §4.4 does impose share-alike on a Derivative Database — so the curated gazetteer file must carry an ODbL notice, and if the normalizer's index (a derived, restructured form of the gazetteer) is shipped in the repo, it should carry the same notice. This is a two-line obligation with a real legal consequence and no cost. The repo currently has no LICENSE file at all — D-04 says GPL-3.0 is the project license, so adding LICENSE (GPL-3.0) plus a data/ notice is a concrete deliverable this phase, not a Phase 7 chore.
Primary recommendation: Ship Phase 4 as three separable layers — (1) a pure, dependency-free shared search module (gazetteer.json + normalizer + scorer, importable by both bot/ and web/), (2) a bot command surface (/cari → sendVenue + inline alternates), and (3) a deep-link carrier with a documented fallback. Make the deep-link carrier the first thing built and the last thing gated, because it is the only requirement in the phase whose success depends on undocumented Telegram client behaviour. Then close the experiment with the corpus runner: a single command that prints top-3 hit rate on positives and refusal rate on negatives, and a written decision in the repo saying which lever was pulled.
The one artifact to lift directly: §"Malay Normalizer — the vendorable rule table" reproduces the complete JUPEM-derived abbreviation table verbatim (the machine-readable transcription from Nominatim's variants-ms.yaml), plus the MS 2039:2006 display rule, the Presint/Prsn vowel-drop case with real usage evidence, and the three official tables that must be merged. The planner should vendor that table into the shared module with its JUPEM attribution rather than author a smaller one — the difference between the 8-entry original and the current 65+ entry table is the difference between "Jln" working and "Jln Ampang" working.
Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|---|---|---|---|
| Malay normalization + scoring (pure functions) | (shared library — no tier) | — | D-09/D-11 require the same module in bot and web. A workspace package (@araharah/search) or a shared src/ dir imported by both. No runtime, no IO, no network. |
| Gazetteer data (curated JSON) | (data file — no tier) | — | D-07: typed JSON in the repo, committed, diff-friendly. Read by both tiers. |
/cari command parsing |
API / Backend | — | Telegram command dispatch lives in bot/src/bot/commands/. The Worker is the only tier that receives Telegram updates. |
| Query → shortlist (SearchSource) | API / Backend | Browser / Client | D-03: in-Worker implementation. The webview copy is build-time only (D-11) — it maps an id to coordinates, it does not re-run the search. |
| Venue card + inline keyboard rendering | API / Backend | — | sendVenue is a Bot API call; only the Worker holds BOT_TOKEN. |
| Alternate selection (callback query) | API / Backend | — | callback_query updates arrive at the Worker webhook. |
| Deep-link payload → destination lookup | Browser / Client | API / Backend | D-11: the webview maps id → destination from its own copy of the shared module. The Worker never re-derives it. |
| Mini App boot with preloaded destination | Browser / Client | — | Phase 3's staged boot (validating → acquiring → live) is the entry; the destination becomes a pre-seeded target the Phase 5 route layer consumes. |
| Home-screen prompt (ONBD-03) | Browser / Client | — | addToHomeScreen() is a Telegram.WebApp method; only the webview can call it. |
| Golden corpus + acceptance runner | (repo tooling — no tier) | — | D-13/D-14: a repo script that imports the shared module and the corpus file. Runs under node, no network. |
| Overpass top-up script | (repo tooling — no tier) | — | D-06: maintainer-time tool, network-dependent, never imported by the Worker. |
Standard Stack
Core
| Library | Version | Purpose | Why Standard |
|---|---|---|---|
| grammy | 1.46.0 (installed; npm view grammy version → 1.46.0 — current) |
Bot API client: sendVenue, InlineKeyboard, callbackQuery handlers |
Already the project's bot framework (Phase 2). InlineKeyboard has a first-class .webApp(text, url) builder, so D-10's web_app button needs no raw object literal. |
Node 24.18.0 built-ins only (JSON.parse, Intl, String.prototype.normalize) |
n/a | Gazetteer loading, normalization, scoring | D-07 mandates V8-native parse, no WASM. D-09's rule table is plain data. Damerau-Levenshtein on a ~500-row × ~5-alias corpus is a ~2500-pair comparison — a hand-rolled 20-line OSA loop is faster than any dependency at this size and adds zero bundle weight. Recommendation: hand-roll the OSA distance in the shared module (see §"Don't Hand-Roll" for why this one is an exception). |
| @crob/damerau-levenshtein | 2.0.0 (npm, published ~1 month ago) |
Alternative to hand-rolling OSA distance | TypeScript-first, ESM+CJS+UMD, 1.2 kB gzipped, configurable per-op costs (nice: a cheap cost for adjacent-keyboard substitutions). Requires Node 20.19+ (we have 24.18.0). Use if the plan wants a battle-tested implementation; skip if the plan prefers zero runtime deps. [VERIFIED: npm registry] |
damerau-levenshtein |
1.0.8 (npm) |
Alternative | BSD-2, 52M weekly downloads, but last published 2022, CommonJS-ish import * as levenshtein shape, and returns {steps, relative, similarity} rather than a bare distance. Older and less pleasant than @crob/damerau-levenshtein. [VERIFIED: npm registry] |
fastest-levenshtein |
1.0.16 (npm) |
Alternative — Levenshtein only, no transposition | Fast, but no transposition support — and transpositions ("KLCC" → "KCLC", "Sunway" → "Sunwya") are an explicit D-13 corpus case. Rejected. [VERIFIED: npm registry] |
Supporting
| Library | Version | Purpose | When to Use |
|---|---|---|---|
| Playwright | 1.50.0 (already a web devDependency) |
Capture evidence for ONBD-02 (webview booted with a preloaded destination) | Reuse the existing web/scripts/capture.mjs harness pattern — add a ?dest=<id> profile. No new dependency. |
| Nominatim (network service, not a lib) | live API | Fallback candidate source + coordinate verification for curated entries | Verified reachable this session (nominatim.openstreetmap.org/status → 200; ?q=KLCC returned correct KLCC nodes with licence: "Data © OpenStreetMap contributors, ODbL 1.0"). Use it to verify curated coordinates even if Overpass is unreachable. |
| Geofabrik (network service) | asia/malaysia-singapore-brunei-latest.osm.pbf |
Fallback bulk OSM extract for the top-up script | Verified reachable (HTTP 206 on range request). Note the extract name is malaysia-singapore-brunei, not "malaysia" — a 404 trap. |
Alternatives Considered
| Instead of | Could Use | Tradeoff |
|---|---|---|
| Curated JSON gazetteer (D-07) | SQLite / sql.js in the Worker |
D-07 already rejects this: ~1.5 MB bundle + cold-start for a ~200–500-row dataset. Non-negotiable. |
| Hand-rolled OSA distance | @crob/damerau-levenshtein (2.0.0) |
+1.2 kB gzipped, well-tested, configurable costs; −~20 lines of hand-written code to maintain and test. Both viable at this corpus size. |
| Trigram/prefix index | Plain linear scan over ~500 entries × ~5 aliases | At ~2500 candidate pairs, a linear scan with early-exit on exact normalized match is microseconds — an index is premature. The trigram index becomes backend #2's job (D-05). |
| Overpass QL for top-up candidates | Nominatim search, or a Geofabrik .pbf + osmium |
Overpass is the right tool in general; in this environment it is blocked (see §"Environment Availability"). The script should support both a `--source=overpass |
sendVenue (D-10) |
sendMessage + sendLocation |
Rejected. SRCH-03 explicitly names the native venue card; sendVenue gives the pin/name/address semantics Telegram users read as "a place". Also: sendVenue accepts no text field — the confirming line must be a separate message or ride in the venue's own title/address. |
callback_data alternates |
Editing the message on every tap | D-02 wants alternates on the same message as the card. editMessageText can replace inline keyboards, but a venue message carries a venue payload that editMessageText cannot preserve — the card would be destroyed. Use callback_data + answerCallbackQuery + a new sendVenue (or editMessageReplyMarkup, which is documented as allowed on a message that already has an inline keyboard). |
Installation:
# Only if the plan chooses the library route for the distance function:
npm install @crob/damerau-levenshtein -w @araharah/search
# If the shared module is a workspace package, add "shared" to root package.json "workspaces".
Version verification: npm view grammy version → 1.46.0 (matches installed). npm view @crob/damerau-levenshtein version → 2.0.0. npm view fastest-levenshtein version → 1.0.16. npm view damerau-levenshtein version → 1.0.8. All verified against the registry this session.
Architecture Patterns
System Architecture Diagram
┌────────────────────────────────────────────────────────┐
│ Telegram client │
│ user types: /cari KLCC │
└────────────────────────┬───────────────────────────────┘
│ (Bot API update)
▼
┌────────────────────────────────────────────────────────────────────────────────┐
│ Cloudflare Worker (bot/src/webhook.ts → POST /telegram/webhook) │
│ │
│ ┌──────────────────┐ ctx.args = "KLCC" │
│ │ /cari handler │ (bot/src/bot/commands/cari.ts) │
│ └────────┬─────────┘ │
│ │ search("KLCC") │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ @araharah/search (SHARED — pure functions, no IO, no network) │ │
│ │ │ │
│ │ ┌───────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │
│ │ │ normalizer.ts │→ │ scorer.ts │→ │ gazetteer.json (typed) │ │ │
│ │ │ rule table + │ │ exact→alias │ │ {id,name,aliases, │ │ │
│ │ │ folding + │ │ match→token │ │ lat,lng,category,addr} │ │ │
│ │ │ glued split │ │ set→Damerau │ │ │ │ │
│ │ └───────────────┘ └──────────────┘ └──────────────────────────┘ │ │
│ │ ▲ normalize both sides at index time │ │
│ └────────┼──────────────────────────────────────────────────────────────┘ │
│ │ SearchSource interface (D-03) — backend #2 / #3 plug in here │
│ ▼ │
│ shortlist: [best, ...alternates] (or EMPTY → honest refusal) │
│ │
│ ┌─────────────────────────┐ │
│ │ sendVenue(best) │ ← SRCH-03: native venue card │
│ │ + inline_keyboard: │ │
│ │ [ OPEN IN ARAHARAH ] │ ← web_app button, URL carries the dest id │
│ │ [ alt 2 ] [ alt 3 ] │ ← callback_data = "alt:<entryId>" │
│ └──────────┬──────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────┐ │
│ │ callback_query handler │ ← user taps an alternate │
│ │ answerCallbackQuery() │ │
│ │ + sendVenue(newBest) │ │
│ └─────────────────────────┘ │
└────────────────────────────────┬───────────────────────────────────────────────┘
│
┌────────────────────────┴──────────────────────────┐
│ (A) web_app inline button ──► Mini App webview │
│ (B) url-button deep link ──► t.me?startapp=… │
│ (C) in-webview URL query ──► ?dest=<id> │
▼ ▼
┌──────────────────────────────────────────────────────────┐
│ Mini App (web/src/main.ts) — Phase 3 staged boot │
│ validating session → acquiring location → live │
│ + destination preloaded (id → entry from shared module)│
│ + ONBD-03: addToHomeScreen() prompt handling │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Repo tooling (never imported by the Worker) │
│ bot/scripts/…-gazetteer-topup.mjs → Overpass/Nominatim│
│ …-corpus.mjs (golden corpus runner) → node │
│ …-json-to-sqlite.mjs (D-05 backend #2 converter stub) │
└──────────────────────────────────────────────────────────┘
Recommended Project Structure
The key structural decision: the search module must be importable by both bot/ and web/. Two viable shapes:
Option A — a new workspace package (cleanest):
shared/ @araharah/search — pure TS, zero deps
├── package.json (type: module, exports ./src/index.ts)
├── src/
│ ├── index.ts public API: search(), normalize()
│ ├── types.ts GazetteerEntry, SearchSource, SearchResult
│ ├── normalize.ts rule table + folding + glued-word split
│ ├── rules.ts the abbreviation table (JUPEM-derived)
│ ├── score.ts exact → alias → token-set → Damerau
│ └── distance.ts Damerau-Levenshtein OSA (hand-rolled or lib)
├── data/
│ ├── kl-gazetteer.json the curated entries (D-07)
│ └── NOTICE.md ODbL attribution for the OSM-derived data
└── test/ corpus runner + rule test vectors
Option B — a shared dir at repo root imported by both workspaces:
packages/search/ (same shape)
Both workspaces then do:
import { search } from "@araharah/search";
import gazetteer from "@araharah/search/data/kl-gazetteer.json" with { type: "json" };
bot/src/
├── bot/commands/cari.ts NEW — /cari + /cari alias
├── bot/handlers/search-alternate.ts NEW — callback_query handler for alternates
├── search/
│ └── source.ts NEW — the SearchSource interface (D-03)
│ + GazetteerSource (the in-Worker impl)
└── i18n/strings.ts EXTENDED — search copy (en/ms)
web/src/
├── main.ts EXTENDED — read dest id at boot, seed destination
├── telegram/webapp.ts EXTENDED — addToHomeScreen/checkHomeScreenStatus
├── search/ NEW — the webview's copy of id→entry lookup
└── i18n/strings.ts EXTENDED — any search-ready copy
repo root:
├── LICENSE NEW — GPL-3.0 (D-04)
├── shared/ or packages/search/ NEW — the shared module (above)
└── bot/scripts/
├── gazetteer-topup.mjs NEW — Overpass/Nominatim candidate proposer (D-06)
├── gazetteer-corpus.mjs NEW — golden corpus runner (D-13/D-14)
└── gazetteer-to-sqlite.mjs NEW — JSON→SQLite converter (D-05, stub shape)
Import assertion: the shared module must be importable by a Node script (corpus runner), a Cloudflare Worker (bundled by wrangler/esbuild), and a browser (Vite). All three handle a .ts file if the consuming toolchain compiles it — the corpus runner is node, which cannot import .ts directly in Node 24 without a loader flag. The corpus runner therefore must be .mjs and import a .js/.mjs build of the module, or the shared module must be authored in plain .js with JSDoc types + a hand-written .d.ts. This is a concrete, easy-to-miss build constraint. Node 24 does have type-stripping (--experimental-strip-types), but relying on it in a repo script is fragile; the safer shape is author the shared module in plain ESM JavaScript with JSDoc typedefs, matching the repo's existing .ts-source-imported-from-.mjs-test convention (the existing tests already do await import("../src/i18n/strings.ts") — verify whether that works under the repo's current Node version before repeating it).
Planner note:
bot/test/commands.test.mjsalready doesawait import("../src/i18n/strings.ts"). Checknode --version(24.18.0 here) — Node 24 has type stripping enabled by default for.tsfiles, soimport("../src/...ts")works. If so, the shared module can stay.tsand the corpus runner can import it directly. Verify this in the plan's first task, because the whole test strategy depends on it.
Pattern 1: The SearchSource interface (D-03)
What: a narrow contract that the in-Worker gazetteer implements today, and that backend #2 (WASM geocoder) and backend #3 (Photon-on-VPS) implement later.
When to use: always — this is the seam. Phases 5/6 plug origin/destination resolution into the same interface.
Example:
// shared/src/types.ts
export interface GazetteerEntry {
id: string; // stable, slug-like: "suria-klcc", "sunway-pyramid"
name: string; // canonical display name
aliases: string[]; // locals' shorthand — D-08's main quality lever
lat: number;
lng: number;
category: string; // "mall" | "office-tower" | "park" | "station" | …
address?: string; // human address for sendVenue's `address` field
}
export interface SearchQuery {
text: string; // raw user query
limit?: number; // default 5 (1 best + 2–4 alternates, D-02)
}
export interface SearchHit {
entry: GazetteerEntry;
score: number; // 0..1, higher is better
matchedAlias: string; // which alias matched — useful for the corpus report
}
export interface SearchSource {
readonly name: string; // "gazetteer" | "wasm-geocoder" | "photon"
search(q: SearchQuery): Promise<SearchHit[]>; // EMPTY array = honest refusal
}
The refusal contract is the important part. An empty result is a legitimate, load-bearing return value (D-13's negatives: "Kuching", "Penang" must be refused, never fabricated). The interface must make "no match" easy to express and the bot must translate it into the "not in v1 area" copy — never into a best-guess pin.
Pattern 2: Rule-table normalization (D-09)
What: a data-driven, bidirectional abbreviation table + folding + glued-word splitting, applied to both the query and every entry alias at index time.
When to use: always, before any scoring. Normalization must be symmetric — "Jln Ampang" and "Jalan Ampang" must produce the same normalized string, which requires expanding both directions to a canonical form.
Example (the canonical form is the long form):
// shared/src/rules.ts — JUPEM-derived. See §"Malay normalizer" for the full table.
export const ABBREVIATIONS: Record<string, string[]> = {
jln: ["jalan"],
tmn: ["taman"],
kg: ["kampung", "kampong"],
kpg: ["kampung", "kampong"],
bkt: ["bukit"],
bt: ["batu"],
sg: ["sungai"],
bdr: ["bandar"],
psrn: ["persiaran"],
lbh: ["lebuh"],
lrg: ["lorong"],
lor: ["lorong"],
spg: ["simpang"],
blk: ["blok"],
h: ["hulu"],
p: ["pulau"],
tg: ["tanjung", "tanjong"],
tlk: ["teluk", "telok"],
g: ["gunung", "gunong"],
gn: ["gunung", "gunong"],
k: ["kuala"],
hr: ["hilir"],
jr: ["jeram"],
lmbh: ["lembah"],
pdg: ["padang"],
pt: ["parit"],
pmtg: ["permatang"],
plbh: ["pelabuhan"],
kep: ["kepulauan"],
ldg: ["ladang"],
lpg: ["lapangan"],
gsg: ["gosong"],
bgn: ["bangunan"],
alr: ["alur"],
bak: ["bakau"],
bdh: ["berhad"],
hj: ["haji"],
hjh: ["hajjah"],
abd: ["abdul"],
kl: ["kuala lumpur"],
pj: ["petaling jaya"],
jb: ["johor bahru"],
kb: ["kota bahru"],
kk: ["kota kinabalu"],
};
Pattern 3: Layered scoring with an exact-match floor
What: score candidates in tiers, each tier's score range strictly above the previous, so an exact alias hit can never be outranked by a fuzzy hit.
When to use: always. This is the mechanism that makes D-13's "top-3 contains the expected id" measurable and stable.
Example:
// shared/src/score.ts — tier ranges are disjoint; see §"Common Pitfalls" #4.
export const TIER = {
EXACT: 1.00, // normalized query === normalized alias or name
ALIAS_EXACT: 0.90, // normalized query === normalized alias
PREFIX: 0.75, // one is a prefix of the other, ≥3 chars
TOKEN_SET: 0.65, // all query tokens present in the candidate's token set
FUZZY: 0.00, // 0..0.60 — Damerau similarity, scaled
} as const;
export function scoreOne(queryNorm: string, aliasNorm: string): number {
if (queryNorm === aliasNorm) return TIER.EXACT;
if (queryNorm.length < 3 || aliasNorm.length < 3) return 0;
const d = damerau(queryNorm, aliasNorm);
const maxLen = Math.max(queryNorm.length, aliasNorm.length);
const sim = 1 - d / maxLen;
if (queryNorm.startsWith(aliasNorm) || aliasNorm.startsWith(queryNorm)) {
return TIER.PREFIX;
}
if (tokenSubset(queryNorm, aliasNorm)) return TIER.TOKEN_SET;
return sim * TIER.FUZZY; // 0 .. 0.60
}
Why the tiering matters: Jaro-Winkler's prefix bonus and Damerau's normalized distance can both rank a fuzzy match above an exact one on short strings. Disjoint score bands make the ordering a property of the tier, not of the arithmetic — so the corpus results are stable when the fuzzy function is tuned (D-09's "extends by table edit, not code change" only works if the tiers don't move).
Anti-Patterns to Avoid
- Bare
===matching against names. Fails every abbreviation and typo case; D-09 exists to prevent this. - A single flattened
Map<string, Entry>of all aliases. It cannot rank alternates, cannot express "the canonical name outranks an alias", and loses thematchedAliasprovenance the corpus report needs. - Normalizing only the query. Normalizing only one side means "Jln Ampang" and "Jalan Ampang" land in different buckets. D-09 explicitly requires both sides.
- Fuzzy-scoring the whole string when the error is in one word. "Tmn Bangsar" vs "Taman Bangsar" is a whole-string one-edit match, but "Jln Tun Razak" vs "Jalan Ampang" is two different roads — token-set matching before fuzzy prevents cross-street confusion.
- Reading the gazetteer JSON from the filesystem at Worker runtime. Cloudflare Workers have no
fs; the JSON must be a build-time import (import data from "...json" with { type: "json" }) so wrangler/esbuild inlines it into the bundle. The corpus runner (Node) reads it withfs— the shared module must expose the entries, not do the reading, so both consumers get their data the way their runtime allows. - An Overpass query with no bbox. A global Overpass query will time out and is a well-known way to get rate-limited. Always bound to the KL bbox.
- An
editMessageTexton the venue message. A venue message carries avenuepayload;editMessageTextreplaces it with text and the card is destroyed. UseeditMessageReplyMarkup(documented as allowed when the message already has an inline keyboard) or send a newsendVenue. - Persisting the raw query. D-15: category-only logging. A query is user payload. This is the single easiest way for this phase to break the project's privacy promise — and there is already a grep gate in
web/test/no-storage.test.mjsfor the web side.
Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---|---|---|---|
| Bot API transport, keyboards, callback handling | Raw fetch to api.telegram.org |
grammy 1.46.0 | Already installed. InlineKeyboard.webApp() / .text() builders, bot.callbackQuery(regex, …) trigger matching, ctx.answerCallbackQuery() — all first-class. Hand-rolling risks the 30-second callback-answer deadline. |
| The Malay abbreviation table | An invented list from memory | The JUPEM table (via Nominatim variants-ms.yaml), vendored with attribution |
The official table is normative, already transcribed, and machine-readable. Inventing one loses Psrn, Lbh.Raya, Bdr, Kondo, Blk, Hosp, Disp and the older Kampong/Tanjong/Telok/Gunong spellings — all real KL usage. §"Malay normalizer" reproduces it. |
| OSM data retrieval at scale | Parsing a .pbf in the Worker |
Overpass API (repo script) or Geofabrik extract (repo script) | Neither belongs in the Worker. D-06 says the script is a maintainer tool. |
| The Mini App ↔ Telegram bridge | telegram-web-app.js vendored into the repo |
The official script loaded by the existing web/index.html |
Phase 3 already ships this arrangement. |
| Home-screen install prompt | A custom PWA install flow | Telegram.WebApp.addToHomeScreen() |
Telegram owns the shortcut; the browser's beforeinstallprompt does not exist inside the Telegram webview and would create a second icon that Telegram doesn't know about. |
| A search index / trigram structure | A hand-built inverted index in the Worker | Linear scan over ~2500 (entry × alias) pairs | At 500 entries × 5 aliases, a linear scan with an exact-match fast path is microseconds. A trigram index is backend #2's job (D-05) and would be pure cost in the Worker bundle (D-07). |
| Session / HMAC validation | Re-deriving initData validation | The existing bot/src/security/initdata.ts |
Phase 2's HMAC path is tested and hardened (T-02-01). The destination id does not need its own crypto (D-11). |
| Routing / Photon / Valhalla | Anything in this phase | Deferred to Phase 5 | D-03 explicitly defers VPS provisioning. Do not research or build it here. |
Key insight — the one place hand-rolling IS correct: the Damerau-Levenshtein distance. At this corpus size the comparison count is tiny, the algorithm is 20 lines of standard OSA dynamic programming, and a dependency would add a package to a Worker bundle that D-07 deliberately wants minimal. Hand-rolling it inside the shared module keeps the whole search stack dependency-free and V8-native, which is exactly D-07's rationale. Exception: if the plan prefers a tested implementation, @crob/damerau-levenshtein@2.0.0 is 1.2 kB gzipped, TypeScript-first, and Node 20.19+ compatible — a safe swap.
Runtime State Inventory
Not applicable. Phase 4 is additive: no rename, no rebrand, no string replacement, no migration. The gazetteer is a new committed file; there is no pre-existing state to inventory. (Per the format spec, this section is included only for rename/refactor/migration phases and is omitted otherwise.)
However, there is one adjacent runtime-state question the planner should answer explicitly:
| Category | Question | Answer for Phase 4 |
|---|---|---|
| Stored data | Does any existing datastore hold a place-name or search string that this phase changes? | No. The only KV namespace is chat_sessions (sessions + location pins, TTL-only). The gazetteer is a new repo file, not a KV namespace. |
| Live service config | Does any external service config reference place names or search endpoints? | No. The only live config is bot/wrangler.jsonc (KV namespace, assets binding) and bot/.dev.vars (secrets). One new config item IS likely: the Mini App's public URL, needed to build the web_app button's WebAppInfo.url and the t.me?startapp= deep link. Today the Worker is on <name>.<account>.workers.dev (workers_dev: true) and WEBHOOK_URL holds the origin. The plan must decide how the bot learns the Mini App URL — a new env var (e.g. MINI_APP_URL) read from bot/.dev.vars + wrangler secret, or derived from WEBHOOK_URL. This is a real Phase 4 deliverable and it is not in the D-list, so it falls in OpenCode's Discretion. |
| OS-registered state | Any OS-level registration embedding the changed strings? | No. No rename. The PWA home-screen shortcut is created by the user on their device at runtime; nothing is registered on the developer's machine. |
| Secrets / env vars | Any secret key named after a place name or search? | No. BOT_TOKEN, WEBHOOK_SECRET, WEBHOOK_URL are unaffected. If a MINI_APP_URL is added, it is not a secret — it is a public origin. It may live in wrangler.jsonc vars rather than as a secret. Flag for the planner. |
| Build artifacts | Stale artifacts that a source rename would leave behind? | No rename. But note: web/dist and web/dist-harness are build outputs; if the shared module is added, the Vite and wrangler builds must both pick it up and the web/test/build-artifact.test.mjs gate may need a new assertion (e.g. the gazetteer JSON must be in web/dist for the webview copy to work). Check that gate's scope. |
Deep-link risk — choosing the carrier for the destination payload (ONBD-02)
This is the single highest-risk decision in Phase 4. D-11 says the deep link "carries a plain gazetteer entry id (?startapp=dest_<id>)". The payload is settled. The carrier is not — and D-11's wording mixes two distinct Telegram launch surfaces.
What the official docs actually say
Three separate statements, from the same page, that do not obviously agree:
| Source | Statement |
|---|---|
core.telegram.org/bots/webapps → Launching the main Mini App |
"A bot's main Mini App can also be opened in the current chat by direct link in the format https://t.me/botusername?startapp. If a non-empty startapp parameter is included in the link, it will be passed to the Mini App in the start_param field and in the GET parameter tgWebAppStartParam." |
core.telegram.org/bots/webapps → Direct Link Mini Apps |
Same mechanism for https://t.me/botusername/appname?startapp=command. |
core.telegram.org/bots/webapps → WebAppInitData.start_param |
"The value of the startattach parameter, passed via link. Only returned for Mini Apps when launched from the attachment menu via link." — note: this row says startattach, not startapp, and scopes it to the attachment menu. |
core.telegram.org/bots/webapps → Inline Button Mini Apps |
"the Mini App will open with the URL specified in the button" — no mention of start_param at all. |
core.telegram.org/bots/api/types/webappinfo.md |
"An HTTPS URL of a Web App to be opened with additional data" — the URL is arbitrary. |
The gap: the launch-parameter docs (docs.telegram-mini-apps.com/platform/start-parameter) list exactly two carriers for a start parameter — ?startattach=ABC on a bot link, and ?startapp=ABC on a Direct Link. A web_app inline keyboard button is neither. Its WebAppInfo.url is simply "the URL specified in the button", and Telegram passes it to messages.requestWebView verbatim (MTProto schema: keyboardButtonWebView → "passing url to messages.requestWebView.url`").
Consequence: if the plan builds the OPEN button as web_app: { url: "<origin>/?startapp=dest_klcc" } and the webview reads Telegram.WebApp.initDataUnsafe.start_param, start_param will most likely be undefined — the button opens the Mini App with no destination. ONBD-02 fails silently.
The three carriers, ranked
| Carrier | Mechanism | Evidence | Risk | UX cost |
|---|---|---|---|---|
A — web_app button with a URL query param |
Button URL = https://app.example.com/?dest=dest_klcc; the webview reads location.search |
WebAppInfo.url is documented as an arbitrary HTTPS URL; MTProto passes it verbatim; community reports confirm query params survive ("passing query params to telegram web app just works"). Not in the official Telegram docs. |
MEDIUM — the behaviour is real but undocumented, so it is not a contract Telegram will keep. | None. One tap, in-app, no interstitial. |
B — url button to https://t.me/<bot>?startapp=dest_<id> |
The officially-documented startapp mechanism |
Explicitly documented for both main-Mini-App and direct links. start_param + tgWebAppStartParam. |
LOW for the mechanism; MEDIUM for the initData.start_param wording (§"A2"). |
A Telegram interstitial: the tap leaves the chat, opens the app. Also requires @BotFather configuration (Main Mini App, or a Direct Mini App with a short_name). And a url button is not a web_app button — D-10's wording changes. |
| C — both, purpose-split | A for the inline button (ONBD-02); B for the home-screen shortcut (ONBD-03) | The home-screen shortcut opens the Main Mini App link — which is exactly where startapp is officially supported. |
LOW overall if the payload reader accepts both sources. | Carries A's cost on the button; B's cost only on the home-screen path (where an interstitial is invisible — the user tapped an icon). |
Recommendation
Ship Carrier C, with A as the inline-button mechanism and B as the home-screen mechanism. Concretely:
- The payload reader in the webview accepts both
location.search'sdestandinitData'sstart_param, with the same validation (/^[A-Za-z0-9_-]{1,64}$/). One reader, two carriers — the phase cannot fail because Telegram changed one surface. - The button builder in the bot is a single function,
openButtonFor(entry), so swapping carriers is a one-line change if the device test fails. - The device validation at the exit gate must explicitly test: (a) tap OPEN from the chat → destination preloaded; (b) launch from the home-screen icon → app opens search-ready (ONBD-03's requirement is "lands in the app ready to search", so a preloaded destination is not required there — but the icon must open the app at all).
What the mock harness must gain
web/src/mock/telegram-mock.ts currently emits initData as user=…&auth_date=1&query_id=h&hash=harness (buildMockInitData, lines 136-140) — no start_param field, and no way to simulate a URL query param (the mock is installed into window, not a URL). The Phase 4 plan must extend the harness with:
- a
startParamoption inMockOptionsthat appendsstart_param=…to the mock initData (for Carrier B); - a
?dest=<id>runner-page profile that setswindow.location.searchbefore the app boots (for Carrier A).
Without these, the destination-preload path has no automated test — it would be validated only by hand on a device.
The build-time constraint (easy to miss)
The start_param charset is restricted to [A-Za-z0-9_-], max 512 characters — validated with /^[\w-]{0,512}$/. A gazetteer id like suria-klcc already complies. But any id that would need a base64url encoding (e.g. one containing a dot or a slash) must be slugged — which is a good reason for D-07's ids to be slug-shaped from the start. The corpus runner should assert every corpus expectId matches /^[A-Za-z0-9_-]{1,64}$/.
ONBD-03 feasibility — what "add to home screen" actually means inside Telegram
D-12 scopes ONBD-03 to "minimal PWA hooks — addToHomeScreen prompt handling + a startapp-less boot landing in the search-ready map". The research confirms this scope is right, and surfaces three constraints the plan must encode.
The API is Bot API 8.0+ only
addToHomeScreen() and checkHomeScreenStatus() were added in Bot API 8.0 (17 Nov 2024). They do not exist in older clients. The project's TelegramWebAppLike interface (web/src/telegram/webapp.ts) already carries isVersionAtLeast(version: string): boolean and the mock already implements versionAtLeast (web/src/mock/telegram-mock.ts:110-121) — so the capability guard has a precedent and a testable seam.
// Extend the existing hand-rolled contract — do NOT add @types/telegram-web-app
// (Phase 3 dropped it deliberately; every fake would have to stub ~50 members).
export interface TelegramWebAppLike {
// … existing members …
addToHomeScreen?(): void;
checkHomeScreenStatus?(cb: (status: HomeScreenStatus) => void): void;
}
export type HomeScreenStatus =
| "unsupported" // the feature is not supported here
| "unknown" // supported, but whether the icon is installed is indeterminate
| "added" // already on the home screen
| "missed"; // not added yet
The guard: if (wa && typeof wa.addToHomeScreen === "function" && wa.isVersionAtLeast("8.0")). Both conditions — the method may be absent on older clients, and the version check is what the platform itself documents.
iOS cannot report installation status — treat "unknown" as normal
checkHomeScreenStatus() always returns "unknown" on Telegram for iOS, regardless of whether the Mini App is actually installed. This is a documented platform limitation, not a bug:
Telegram-iOS issue #1729 ("Home screen status detection always returns 'unknown' in Telegram iOS Mini Apps", April 2025) was closed with the maintainer comment: "On iOS it is not possible to determine whether a Mini App has been added to the Home Screen. The platform does not provide any API that allows Telegram to check whether a Home Screen shortcut exists."
Design consequence: the app must not branch on the status string in a way that treats "unknown" as an error or as "not added". The honest behaviour is: offer the prompt when the platform supports it; do not nag; do not claim the icon is present unless the status is literally "added".
What the home-screen shortcut actually opens
The shortcut opens the Main Mini App — which is the surface where ?startapp= is officially supported (§"Deep-link risk", Carrier B). This is convenient: ONBD-03's "lands in the app ready to search" needs no payload at all (D-12 says "startapp-less boot"), and the shortcut naturally lands on the search-ready map. The startapp capability on that surface is a free bonus for a future phase (e.g. a per-destination home-screen shortcut).
A separate, larger PWA manifest / service worker / offline story is explicitly deferred (D-12, and CONTEXT's Deferred Ideas). Do not add a manifest.webmanifest, a service worker, or an install-prompt beforeinstallprompt handler — the browser API does not exist in the Telegram webview, and a second icon the Telegram client does not know about is worse than none. addToHomeScreen() is the only correct mechanism here.
What the plan should build
- Extend
TelegramWebAppLikewith the two optional methods (above). - A single, small "add to home screen" affordance — where it lives is a UI question, but the D-12 scope is deliberately minimal, and Phase 4's
UI hint: noin the roadmap means no new UI-SPEC round. The least invasive placement is a card or a control the Phase 3 shell already has. Flag: if this needs new UI furniture, it may need a UI-SPEC round — the planner should confirm it fits inside the existing Phase 3 shell. - A capability-guard unit test (
web/test/homescreen.test.mjs) covering: method absent → no-op;isVersionAtLeast("8.0")false → no-op; status"unknown"→ treated as indeterminate, not as an error; status"added"→ the affordance is hidden or shows "already added". - Extend the mock harness (
MockOptions) with ahomeScreenStatusoption so the three branches are testable without a device.
Common Pitfalls
Pitfall 1: Assuming start_param arrives from a web_app inline button (THE phase's biggest trap)
What goes wrong: the plan builds the OPEN button as web_app: { url: "https://app.example.com/?startapp=dest_klcc" } and the webview reads Telegram.WebApp.initDataUnsafe.start_param — which is undefined for inline-button launches. The user taps OPEN, the Mini App opens, and the destination is not preloaded. ONBD-02 fails silently at the worst moment.
Why it happens: the official WebAppInitData.start_param documentation is scoped to startattach / attachment-menu launches and to direct links. The web_app inline button is a different launch surface. The two are easy to conflate because both end in "the Mini App has a start param".
How to avoid: treat the carrier as an explicit, tested choice with a fallback. §"Deep-link risk" ranks three carriers:
- Carrier A —
web_appbutton whose URL carries the payload as a plain query param (https://app.example.com/?dest=dest_klcc). The button's URL is an arbitrary HTTPS URL; Telegram opens it in the Mini App webview. Community reports confirm query params on that URL survive into the webview. The webview readslocation.search, notstart_param. Lowest moving-parts risk, but it is not what D-11 says (?startapp=dest_<id>). - Carrier B —
urlbutton tohttps://t.me/<bot>?startapp=dest_<id>(the documentedstartappmechanism, and the exact mechanism D-11 names). Trade-off: theurlbutton opens a Telegram deep link, not a Mini App button — so it leaves the chat, opens the app, and the "one tap to proceed" (D-02) becomes one tap + a Telegram interstitial. It also requires the bot to be configured as a Main Mini App in @BotFather (or a Direct Mini App with ashort_name). - Carrier C — both: a
web_appbutton carrying the id in the URL query (Carrier A) and thestartappdirect-link path (Carrier B) for the home-screen entry point (ONBD-03), wherestartappis officially available.
Warning signs: initDataUnsafe.start_param is undefined at runtime; the webview's location.search is empty; the Mini App boots into the plain "search-ready map" with no destination. The Phase 3 mock harness must be extended to simulate each carrier's arrival shape — the mock's buildMockInitData currently emits only user + auth_date + query_id + hash, with no start_param field at all (web/src/mock/telegram-mock.ts:136-140).
Pitfall 2: sendVenue has no text field — the "one line confirming the destination" needs a home
What goes wrong: the plan tries to send a confirming sentence plus the venue card in one message. sendVenue's parameter list is latitude, longitude, title, address (+ optional foursquare_id, google_place_id, …). There is no text parameter. The confirming line has to be either (a) a separate sendMessage before the card, or (b) baked into the venue's title/address.
Why it happens: sendMessage and sendVenue look alike from a distance; the CONTEXT's "venue card message should read like the bot's clinical voice: one line confirming the destination" reads like a single message.
How to avoid: decide explicitly. The cleanest reading of D-02 ("the best result ships immediately as the venue card") is: one sendVenue whose title is the entry name and whose address is the entry's human address, plus the inline keyboard carrying OPEN + alternates. If a separate confirming line is wanted, it is a second sendMessage — but that violates D-02's "no second message round" only if it needs a tap (it doesn't; two sequential sends is one round of user interaction, but it is two messages). The plan must pick and the copy gate should reflect it.
Warning signs: the bot's sendVenue call silently drops a text field; or the venue card's title ends up carrying marketing copy instead of the place name.
Pitfall 3: Abbreviating a word that is part of a proper name
What goes wrong: the normalizer expands every token, so "Jln Tmn Desa" becomes "Jalan Taman Desa" — but the correct address is "Jln Taman Desa" (Jalan is the generic term; Taman Desa is the name). MS 2039 §5.2.3.2 is explicit: "Abbreviation shall not be applied to names given to the geographical feature itself. As an example, an address line 'Jalan Taman Desa' may be abbreviated to 'Jln Taman Desa' but not 'Jln Tmn Desa' or 'Jalan Tmn Desa'."
Why it happens: the expansion table is applied positionally and unconditionally.
How to avoid: the canonical form (long form) makes this benign if the rule is applied symmetrically: both "Jln Tmn Desa" and "Jln Taman Desa" expand to "Jalan Taman Desa" — because the rule is "expand the abbreviation to the long form", and "Tmn"→"Taman" is a correct expansion of the word even though the address shouldn't have used it. Expanding both sides to the long form makes them match, which is exactly what SRCH-02 wants. The MS 2039 rule matters for display fidelity (what address string the bot shows), not for matching. So: use the long form for matching; keep the curated address string verbatim for display.
Warning signs: the venue card's address shows "Jalan Taman Desa" when the curated entry says "Jln Taman Desa" — the card should show the curated address, not the normalized one.
Pitfall 4: Fuzzy scoring outranking an exact match
What goes wrong: on short strings, Jaro-Winkler's prefix bonus or a normalized Damerau score can push a fuzzy match above an exact one. "KLCC" might rank "KLCC Park" above "Suria KLCC" — or a typo'd query might rank the wrong entry first.
Why it happens: both metrics are continuous and unbounded relative to each other; the "best" match is decided by arithmetic accident.
How to avoid: the disjoint tier bands in §"Pattern 3". An exact match must score strictly above the top of the fuzzy band, always. The corpus runner then reports per-tier hit rates, so a regression shows up as "exact-tier hits dropped" rather than a diffuse quality drop.
Warning signs: the corpus top-1 hit rate is good but top-3 is worse than expected, or a known exact alias loses to a near-miss.
Pitfall 5: Trigram / prefix scoring on very short strings
What goes wrong: "KLCC" is 4 characters → 2 trigrams. "DUKE" → 2 trigrams. Trigram overlap on 2 grams is noise, not signal, and Jaro-Winkler's prefix bonus dominates trivially (both strings share the whole prefix → score ≈ 1.0).
Why it happens: trigram and Jaro-Winkler are both documented as unreliable on short strings. "Very short strings provide so few trigrams that the overlap score is unreliable" and "Jaro-Winkler can over-reward shared prefixes when the discriminating information is later."
How to avoid: exact-match-first, fuzzy-last, and a minimum length guard (the code above refuses to score anything under 3 characters). For 4–6 character queries, rely on the alias table (D-08) rather than on fuzzy scoring — "KLCC" and "DUKE" should be aliases, not fuzzy hits. This is the single strongest argument for D-08's design.
Pitfall 6: The Overpass top-up script becomes a runtime dependency
What goes wrong: the plan wires the Overpass query into the Worker (to "top up" the gazetteer at boot). The Worker now depends on a third-party service with no SLA, on every cold start.
Why it happens: D-06 says "a repo-script Overpass one-shot that proposes top-up candidates for maintainer approval" — the "for maintainer approval" clause is easy to skim past.
How to avoid: the script is bot/scripts/gazetteer-topup.mjs, run manually, outputting a proposal file that a human diffs and merges. The Worker imports only the committed JSON.
Warning signs: fetch("https://overpass-api.de/…") anywhere under bot/src/ or web/src/.
Pitfall 6b: The Overpass script's User-Agent is load-bearing, and capacity 502s are normal (verification pass)
What goes wrong: (a) The script omits the User-Agent header and every request returns HTTP 406 Not Acceptable — which looks like a blocked network or a dead service, and can send the maintainer rewriting a working query. (b) The script treats an HTTP 502 "This Overpass API server is currently at capacity" as a hard failure, exits non-zero, and (in a CI wiring) turns an optional maintainer tool into a red build.
Why it happens: (a) The Overpass instances — and the egress path in front of them — reject requests without an identifying User-Agent. Probed 2026-10-23 on overpass-api.de, same host, same paths: GET /api/status no-UA → 406, with-UA → 200; GET / no-UA → 406, with-UA → 200; POST /api/interpreter no-UA → 406, with-UA → 200. (b) The public instances are free and shared; the Overpass wiki's own policy table warns the main instance "is overloaded – be mindful of that, do not overconsume resources and do not expect high reliability." Both a 200 and a 502 capacity response were observed on overpass-api.de minutes apart in the same session.
How to avoid:
- Always send a descriptive
User-Agent(e.g.araharah-gazetteer-topup/1.0 (+https://github.com/<org>/araharah)). Without it, nothing works — this is the single highest-value line in the script. - Treat 502 / 429 / 504 / timeout as a normal outcome: print one line — "Overpass instance busy — retry later. No candidates generated; nothing changed." — and exit 0. The curated file is untouched on every path, so a busy instance is not a failure.
- Prefer a narrow query. Default to one tag family per invocation (
nwr["tourism"="attraction"](bbox)); a narrow query is far more likely to fit in a free worker slot and returns in seconds. The wide multi-family union stays available behind a flag. - Accept an endpoint override (env var or
--source=) so the maintainer can point at a different instance. - The automated test uses a committed fixture, never the live service (see §Wave 0 Gaps) — so the test is deterministic and the CI bill is zero.
Warning signs: The script exits non-zero with an HTML or plain-text body instead of JSON; or the same query "worked yesterday" and fails today with no code change.
Pitfall 7: The golden corpus becomes a generated artifact
What goes wrong: the corpus file is regenerated from the gazetteer (so every entry gets a synthetic query) — which makes the test tautological: it passes by construction and measures nothing.
Why it happens: "extend by editing one file" (D-13) can be misread as "the file is machine-written".
How to avoid: the corpus is hand-written and pinned — every positive pins an expected id, and the corpus contains queries that are not derived from any alias in the gazetteer (typos, glued words, out-of-KL negatives). Reviewers extend it by editing the file. The corpus file must be git diff-able and its content must not be derivable from the gazetteer.
Warning signs: a corpus entry whose query string exactly equals an alias in the gazetteer, for every entry — that's the tautology signature.
Pitfall 8: Sending the query text to a log line
What goes wrong: console.log("search:", query) while debugging. D-15 and the Phase 2 D-14 law forbid it — the query is user payload.
Why it avoid: the Phase 2 codebase's logging convention is category-only and every log line in bot/src/webhook.ts carries a category, never a payload. Match it: console.log("[search] rejected: no-match") — never the query.
Warning signs: any log line containing a variable that could hold a query string. The existing gates (bot/test/*.test.mjs) grep for structural patterns; add a gate that greps bot/src/ for a log statement whose argument mentions the query variable.
Pitfall 9: The webview's gazetteer copy drifting from the bot's
What goes wrong: the Worker ships gazetteer v1; the webview bundle ships v2 (built earlier). A startapp=dest_klcc arrives; the webview's id → entry lookup misses; the destination silently fails to load.
Why it happens: two independent build outputs from one source, deployed at different times.
How to avoid: D-11 already says "the shared module version" re-maps stale datasets. Concretely: (a) the id set is stable — ids are never renamed, only added; (b) the webview, on an unknown id, shows an honest "destination unavailable, search again" state rather than a blank map; (c) a build-time gate asserts the bot bundle and the web bundle contain the same id set — or, simpler, that both are built from the same committed JSON in the same npm run build (which npm run build already does: theme → web → bot deploy).
Warning signs: an id in the corpus that resolves in the bot but not in the webview.
Pitfall 10: callback_data length and content
What goes wrong: putting the whole venue name + address in callback_data. The limit is 1–64 bytes (confirmed in the InlineKeyboardButton type docs). Also, callback_data is visible to anyone who can see the message and is echoed back to the bot in the update.
How to avoid: callback_data carries only the entry id — "alt:<entryId>" — and the handler looks the entry up in the gazetteer. Never a name, never a query, never coordinates. This also keeps D-15's privacy posture: the callback payload is a public place id, not user data.
Malay Normalizer — the vendorable rule table
This is the highest-reuse artifact in the research. D-09 requires a bidirectional abbreviation table that "extends by table edit, not code change". The table below is not reconstructed from memory — it is the JUPEM (Jabatan Ukur dan Pemetaan Malaysia) normative abbreviation list, transcribed machine-readably by the OSM Nominatim project in settings/icu-rules/variants-ms.yaml (PR #4056, merged 2026-04-02), whose stated primary source is "JUPEM Toponymic Guidelines — 'Abbreviations of Geographical Terms and Other Words Used in Maps'". Vendor it with attribution; do not author a smaller one.
The table, verbatim (shared / Malaysia-specific)
# shared/src/rules.ts — JUPEM-derived, via Nominatim settings/icu-rules/variants-ms.yaml
# Primary source: JUPEM Toponymic Guidelines (Malaysia), 2017 ed.
# Secondary: https://wiki.openstreetmap.org/wiki/Name_finder:Abbreviations#Bahasa_Melayu_-_Malay
# https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations
# Format: "<full form> -> <comma-separated abbreviations>"
shared:
Jalan -> Jln
Lorong -> Lrg, Lor
Lebuh -> Lbh
Simpang -> Spg
Kampung -> Kg
Kampong -> Kg
Taman -> Tmn
Bangunan -> Bgn
Bukit -> Bkt
Batu -> Bt
Sungai -> Sg
Pulau -> P, Pl
Tanjung -> Tg
Tanjong -> Tg
Teluk -> Tlk
Telok -> Tlk
Gunung -> G, Gn
Gunong -> G, Gn
Haji -> Hj
Hajjah -> Hjh
Abdul -> Abd
country: my
Persiaran -> Psrn
Lebuhraya -> Lbh.Raya
Bandar -> Bdr
Kondominium -> Kondo
Bahagian -> Bhg
Blok -> Blk
Kuala -> K, Kl
Alur -> Alr
Bakau -> Bak
Hulu -> H
Hilir -> Hr
Jeram -> J, Jr
Lembah -> Lmbh
Padang -> Pdg
Parit -> Pt
Permatang -> Pmtg
Pelabuhan -> Plbh
Kepulauan -> Kep
Ladang -> Ldg
Lapangan -> Lpg
Gosong -> Gsg
Sekolah kebangsaan -> SK
Sekolah menengah -> SM
Sekolah menengah kebangsaan -> SMK
Sekolah rendah -> SR
Hospital -> Hosp
Dispensari -> Disp
The inverse map the normalizer actually needs
The table above is full → abbreviation. D-09 needs both directions (query says "Jln", entry says "Jalan"). Invert it at load time:
// Build the bidirectional map ONCE at module load. Deterministic, ~60 entries.
const ABBREVIATIONS: Record<string, string[]> = (() => {
const out: Record<string, string[]> = {};
for (const line of RAW_RULES.trim().split("\n")) {
const m = line.match(/^\s*(\S.*?)\s*->\s*(.+?)\s*$/);
if (!m) continue;
const full = m[1].trim().toLowerCase();
for (const abbr of m[2].split(",").map((s) => s.trim().toLowerCase())) {
(out[abbr] ??= []).push(full);
// The full form maps to itself so the canonical form is always reachable.
(out[full] ??= []).push(full);
}
}
return out;
})();
Ambiguity to handle — Pl/P, K/Kl, G/Gn, J/Jr, Lbh/Lbh.Raya:
Several abbreviations map to more than one full form (Pulau → P, Pl and Parit → Pt are distinct, but Kuala → K, Kl vs Kampung → Kg are fine). The dangerous collisions are:
G→ Gunung/Gunong — butGalso appears as a letter in road names ("Lebuh Genting"). ExpandingGin a token that is a single letter is aggressive; the normalizer should only expand an abbreviation when the token is a standalone word (already true of token-based normalization) and when the expansion is unambiguous in context.P→ Pulau andPt→ Parit —Palso appears in "Presint", "Persiaran", "Pusat". Single-letter abbreviations are the highest-collision-risk class.Bt→ Batu vsBukit → Bkt— noteBtis Batu (milestone), not Bukit. Locals mix these; the golden corpus should cover both.KGambiguity —KGis the kilogram unit in some contexts; the existingbm-address-normaliserproject flags this explicitly. In a place-name corpus the ambiguity is negligible, but the plan should note it.
Recommendation: the normalizer keeps a per-token ambiguity list rather than guessing, and the fuzzy-residual tier (D-09) resolves what the table cannot. This matches D-09's architecture exactly: table first, folding second, fuzzy residual last.
The Presint/Prsn vowel-drop rule
D-09 explicitly calls out "vowel-drop-aware Presint/Prsn forms". This is not in the JUPEM table above — it is a Putrajaya-specific local form. Verified against real Malaysian commercial addresses:
JALAN P8G PRESINT 8, 62250 PutrajayaPresint 15,PRESINT 16,PRESINT 4,PRESINT 18PRSN RESIDEN DESA PARKCI, 52200 KUALA LUMPURNO 252 BAWAH PRSN SIBC 9, SERI ISKANDAR BUSINESS CENTREJalan P9 Presint 7, 62250 Putrajaya
So both "Presint" and "Prsn" appear in live use, and "Prsn" is not a prefix abbreviation — it drops the internal e. Add it as an explicit rule:
// Vowel-drop expansion (Putrajaya precincts) — D-09's explicit case.
ABBREVIATIONS["prsn"] = ["presint"];
ABBREVIATIONS["presint"] = ["presint"];
A more general vowel-drop rule worth considering (and worth a corpus test): Malay place names with a dropped interior vowel are a recurring local shorthand class — Pusat→Pusat, Bandar→Bdr, Perbandaran→PBandaran. The plan should decide whether to implement a generic vowel-drop expansion (insert e at each interior consonant cluster) or keep only the explicit Presint/Prsn pair. Recommendation: explicit pair only. A generic vowel-drop rule will over-expand legitimate names ("Pusat" contains no drop, but "Pkl" would expand to "Pekel" and then match "Pekel" wrongly). This is exactly the kind of rule that should be added by table edit (D-09) when the corpus demands it — not by a clever algorithm.
The MS 2039 rule that governs display (not matching)
From MS 2039:2006 §5.2.3.2 (fetched this session):
"Abbreviation may only apply to a generic name of a geographical feature being referred in a particular line or hierarchies of the address format. Abbreviation shall not be applied to names given to the geographical feature itself. As an example, an address line 'Jalan Taman Desa' may be abbreviated to 'Jln Taman Desa' but not 'Jln Tmn Desa' or 'Jalan Tmn Desa'."
How this applies to the normalizer: the rule constrains what a correct address looks like, not what a searchable key looks like. Because the normalizer expands to the long form on both sides, "Jln Tmn Desa" and "Jln Taman Desa" both normalize to "jalan taman desa" and match. That is the desired search behaviour. But the venue card's address field must show the curated address verbatim — not the normalized form. The sendVenue payload's address comes from the gazetteer entry, never from normalize(). This is a one-line architectural rule that prevents a visible regression (Pitfall 3).
The three official abbreviation tables (for completeness)
The JUPEM gazetteer PDF and MS 2039 Annex A carry three distinct tables. The normalizer needs all three, merged:
| Table | Scope | Sample rows |
|---|---|---|
| A1 — Street prefixes (MS 2039 Annex A) | Jalan → Jln, Lorong → Lrg, Persiaran → Psrn, Lebuhraya → Lbh.Raya, Persisiran → (none), Pekeliling → (none), Lintang → Ltg, Lengkok → Lnkg, Regat → Rgt |
33 rows |
| A3 — Locality prefixes (MS 2039 Annex A + JUPEM gazetteer) | Bandar → Bdr, Bukit → Bkt, Desa → (none), Kampung/Kampong → Kg, Kpg, Kuala → K, Hulu → H, Hilir → Hr, Jeram → J, Ladang → Ldg, Padang → Pdg, Pasir → Psr, Permatang → Pmtg, Pulau → P, Sungai → Sg, Taman → Tmn, Tanjung → Tg, Teluk/Telok → Tlk, Mukim → (none), Pekan → (none) |
~35 rows |
| JUPEM gazetteer list (state gazetteer PDF, §4.0) | Air → A., Bahagian → Bhg., Balai Polis → B.P., Balai Raya → B.R., Bakau → Bak., Batu → Bt., Blok → Blk., Bukit → Bkt., Cangkat/Changkat → Chgkt., Hosp → Hosp., Kondominium → Kondo, Lpg → Lapangan, Pejabat Pos → P.P., Pangkalan/Pengkalan → Pang., Pelabuhan → Plbh., Rumah Rehat → R.R., SMK, SR, SRJK, Stesen Kastam → S.K., Tali air → T.A., Utara → U |
48 rows |
Note the intersection is not clean: MS 2039 A3 gives Hilir → Hr while the JUPEM gazetteer list gives Hilir (no abbreviation) — and Kuala → K in A3 vs K. in the gazetteer list. Merge, don't replace; when two tables disagree, prefer the MS 2039 form (it is the current national standard) and record the alternative as an additional alias.
A ready-made city-code table
The existing bm-address-normaliser project and the Nominatim PR both carry a city-code expansion set that is genuinely useful for KL-metro queries ("PJ" → Petaling Jaya, "JB" → Johor Bahru, "KB" → Kota Bharu, "KK" → Kota Kinabalu). These are local shorthand, not JUPEM abbreviations — they belong in the gazetteer alias table (D-08), not the abbreviation rule table, because they are place identities, not address-word forms. The distinction matters: "KL" as a city code should expand to "kuala lumpur"; "Jln" should expand to "jalan". Both are expansions, but they belong in different tables.
Code Examples
The sendVenue call with inline keyboard (D-10, SRCH-03)
Source: https://core.telegram.org/bots/api/methods/sendvenue.md + https://core.telegram.org/bots/api/types/inlinekeyboardbutton.md
import { InlineKeyboard } from "grammy";
// Required: latitude, longitude, title, address. There is NO `text` parameter.
await ctx.replyWithVenue(
entry.lat,
entry.lng,
entry.name, // title
entry.address ?? "", // address
{
reply_markup: new InlineKeyboard()
.webApp(openLabel, miniAppUrlFor(entry.id)) // OPEN IN ARAHARAH
.row()
.text(alt1.name, `alt:${alt1.id}`) // alternates → callback_data
.text(alt2.name, `alt:${alt2.id}`),
},
);
Note on reply_markup: sendVenue accepts InlineKeyboardMarkup | ReplyKeyboardMarkup | ReplyKeyboardRemove | ForceReply — the inline keyboard is the correct one here.
The web_app button URL vs. the startapp direct link
// Carrier A — web_app inline button, payload in the URL query string.
// The webview reads location.search, NOT initDataUnsafe.start_param.
function miniAppUrlFor(id: string, origin: string): string {
return `${origin}/?dest=${encodeURIComponent(id)}`;
}
// Carrier B — t.me direct link, payload in ?startapp= (the officially documented
// start_param mechanism). Requires a Main Mini App or a Direct Mini App (short_name).
function deepLinkFor(id: string, botUsername: string): string {
// start_param charset is restricted to [A-Za-z0-9_-] and ≤512 chars.
return `https://t.me/${botUsername}?startapp=${id.replace(/[^A-Za-z0-9_-]/g, "_")}`;
}
Reading the payload in the webview
// web/src/search/destination.ts — Carrier A (URL query).
export function readDestinationId(): string | null {
const raw = new URLSearchParams(window.location.search).get("dest");
return raw && /^[A-Za-z0-9_-]{1,64}$/.test(raw) ? raw : null;
}
// Carrier B (initData) — note: initDataUnsafe is UNTRUSTED per Telegram's docs;
// it is fine here only because the id is a public place id and is re-resolved
// against the local gazetteer. Never treat it as an identity.
export function readStartParam(wa: TelegramWebAppLike | null): string | null {
const raw = parseInitDataUnsafe(wa?.initData ?? "").start_param;
return raw && /^[A-Za-z0-9_-]{1,512}$/.test(raw) ? raw : null;
}
The alternates callback handler
// bot/src/bot/handlers/search-alternate.ts
export function registerSearchAlternate(bot: Bot, source: SearchSource): void {
bot.callbackQuery(/^alt:(.+)$/, async (ctx) => {
await ctx.answerCallbackQuery(); // must happen within 30 s
const id = ctx.match[1];
const entry = source.byId(id); // id → entry; never a query or a name
if (!entry) return;
await sendVenueCard(ctx, entry, source); // re-sends the card with new alternates
});
// Always answer everything else so Telegram doesn't show a 60-second spinner.
bot.on("callback_query:data", async (ctx) => {
await ctx.answerCallbackQuery();
});
}
Source: https://grammy.dev/plugins/keyboard — "It makes sense to define bot.on("callback") at last to always answer all other callback queries that your previous listeners did not handle. Otherwise, some clients may display a loading animation for up to a minute."
The Malay normalizer
// shared/src/normalize.ts
export function normalize(input: string): string {
let s = input.normalize("NFKD").toLowerCase();
s = s.replace(/[̀-ͯ]/g, ""); // strip combining marks (NFKD)
s = s.replace(/[^\p{L}\p{N}\s/]/g, " "); // punctuation → space (keep "/" for Jln 3/1)
s = s.replace(/\s+/g, " ").trim();
const expanded: string[] = [];
for (const token of s.split(" ")) {
if (!token) continue;
const hit = ABBREVIATIONS[token];
expanded.push(hit ? hit[0] : token); // canonical = the long form
}
s = expanded.join(" ");
s = splitGluedWords(s); // "tamanbangsar" → "taman bangsar"
return s;
}
/** Insert a space at the boundary where a known Malay word starts. */
function splitGluedWords(s: string): string {
// Starters that commonly appear glued in Malaysian addresses.
const starters = ["jalan", "taman", "kampung", "lorong", "bukit", "sungai",
"bandar", "persiaran", "lebuh", "kampung", "perkampungan"];
for (const starter of starters) {
// "tmnbangsar" is handled by the abbreviation table; this catches
// "tamanbangsar" / "jalankanpar" style glue.
const re = new RegExp(`(?<![\\p{L}])${starter}(?=\\p{L})`, "gu");
s = s.replace(re, (m) => `${m.slice(0, starter.length)} ${m.slice(starter.length)}`);
}
return s;
}
The Presint/Prsn rule (D-09's "vowel-drop-aware" clause): Putrajaya's precincts are written both "Presint" and "Prsn" — the latter drops the e. Add an explicit rule:
// "prsn" → "presint" is a *vowel-drop* expansion, not a prefix abbreviation.
ABBREVIATIONS["prsn"] = ["presint"];
ABBREVIATIONS["presint"] = ["presint"];
Verified against real usage: JALAN P8G PRESINT 8, PRESINT 15, PRSN RESIDEN DESA PARKCI, Jalan P9 Presint 7 — all appear in commercial Malaysian addresses (see §"Sources").
The glued-word rule needs care. A naive "insert a space before any known word" will corrupt words that legitimately contain a starter as a substring ("Pandang" contains "dang", "Kampung Baru" is fine, but "Bandar Baru" → "band ar" if "dar" were a starter). The safe implementation is a longest-match-first scan against the starter list, requiring the remainder to be ≥3 characters. The plan should specify this; the naive version is a real bug source.
The Damerau-Levenshtein (OSA) distance — hand-rolled
// shared/src/distance.ts — optimal string alignment, adjacent transpositions.
export function damerau(a: string, b: string): number {
if (a === b) return 0;
if (a.length === 0) return b.length;
if (b.length === 0) return a.length;
const n = a.length, m = b.length;
// 3-row band: only the previous, current and current+1 rows are needed.
let prev2 = new Int32Array(m + 1);
let prev = new Int32Array(m + 1);
let cur = new Int32Array(m + 1);
for (let j = 0; j <= m; j++) prev[j] = j;
for (let i = 1; i <= n; i++) {
cur[0] = i;
for (let j = 1; j <= m; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
let v = Math.min(cur[j - 1] + 1, prev[j] + 1, prev[j - 1] + cost);
if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) {
v = Math.min(v, prev2[j - 2] + 1); // transposition
}
cur[j] = v;
}
const t = prev2; prev2 = prev; prev = t;
const c = cur; cur = prev2; // rotate the three rows
}
return prev[m];
}
Type-check this carefully — the row-rotation in a 3-band OSA is easy to get wrong. Write the test vectors first (TDD, matching the repo's RED→GREEN convention).
The golden corpus format
// shared/data/golden-corpus.json
{
"version": 1,
"cases": [
// ---- golden landmarks (SRCH-01) ----
{ "q": "KLCC", "expectId": "suria-klcc" },
{ "q": "Sunway Pyramid", "expectId": "sunway-pyramid" },
{ "q": "DUKE", "expectId": "the-duke" },
// ---- abbreviations (SRCH-02) ----
{ "q": "Jln Ampang", "expectId": "suria-klcc" }, // Jalan Ampang
{ "q": "Tmn Bangsar", "expectId": "bangsar" },
{ "q": "Kg Ampang", "expectId": "kampung-ampang" },
// ---- glued words (SRCH-02) ----
{ "q": "tamanbangsar", "expectId": "bangsar" },
// ---- typos (SRCH-02) ----
{ "q": "KCLC", "expectId": "suria-klcc" }, // transposition
{ "q": "Sunwya Pyarmid", "expectId": "sunway-pyramid" },
// ---- aliases (D-08) ----
{ "q": "IJN", "expectId": "ijnp" },
{ "q": "PWTC", "expectId": "pwtc" },
{ "q": "Stesen KL Sentral","expectId": "kl-sentral" },
// ---- deliberate negatives (D-13) — MUST be refused, never a fabricated pin ----
{ "q": "Kuching", "expectId": null },
{ "q": "Penang", "expectId": null }
]
}
The corpus runner
// bot/scripts/gazetteer-corpus.mjs — the bake-off close (D-14).
// Prints top-1 / top-3 hit rate on positives and refusal rate on negatives.
// Exit code 1 if top-3 < 90% or if any negative produced a hit.
const { search, gazetteer } = await import("@araharah/search");
const corpus = JSON.parse(await readFile(corpusPath, "utf8"));
const positives = corpus.cases.filter((c) => c.expectId !== null);
const negatives = corpus.cases.filter((c) => c.expectId === null);
let top1 = 0, top3 = 0, missed = [];
for (const c of positives) {
const hits = await search({ text: c.q, limit: 3 });
const ids = hits.map((h) => h.entry.id);
if (ids[0] === c.expectId) top1++;
if (ids.includes(c.expectId)) top3++;
else missed.push(`${c.q} → [${ids.join(", ") || "—"}] (expected ${c.expectId})`);
}
let refused = 0, leaked = [];
for (const c of negatives) {
const hits = await search({ text: c.q, limit: 3 });
if (hits.length === 0) refused++;
else leaked.push(`${c.q} → ${hits[0].entry.id}`);
}
const top3Rate = top3 / positives.length;
console.log(`top-1 ${top1}/${positives.length} top-3 ${top3}/${positives.length} (${(top3Rate*100).toFixed(1)}%)`);
console.log(`negatives refused ${refused}/${negatives.length}`);
if (missed.length) console.log("MISSED:\n" + missed.join("\n"));
if (leaked.length) console.log("LEAKED (must refuse):\n" + leaked.join("\n"));
// D-14 gate: top-3 ≥ 90% AND every negative refused.
process.exitCode = top3Rate >= 0.9 && leaked.length === 0 ? 0 : 1;
This runner IS the bake-off close. Its output is the artifact D-14 asks to be recorded: "the phase records which lever got pulled and re-runs the corpus — the experiment ends with a documented decision, not an open question." The plan should also produce a short docs/SEARCH-QUALITY.md (or a section in the phase summary) recording: the corpus version, the score, and which escalation lever was pulled.
State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|---|---|---|---|
| Nominatim-style server-side geocoding for every query | In-Worker curated gazetteer behind a SearchSource interface |
D-03 (this phase) | Zero infrastructure, zero latency, zero third-party SLA — but only as good as the curated corpus. Phase 4's job is to prove the corpus is good enough (D-14). |
| Photon-on-VPS as the phase-4 backend | Photon deferred to Phase 5; in-Worker gazetteer is backend #1 | D-03 (supersedes the roadmap note) | The roadmap's Phase 4 note says "VPS provisioning begins here" — D-03 supersedes it. The plan must NOT include VPS/Photon provisioning. |
| A Malaysia-only WASM geocoder as the phase-4 backend | Documented as backend #2 (post-v1, OSS track) | D-05 | The JSON dataset format stays reachable from day one via the converter script, so backend #2 is a format change, not a rewrite. |
| Hand-authored abbreviation list | The official JUPEM table, vendored | Nominatim PR #4056, merged 2026-04-02 | The table grew from 8 entries to 65+, split into shared / MY / SG / BN. Vendoring it wholesale (rather than the 8-entry original) is what makes "Jln Ampang" work for free. |
| Trigram index for short-query search | Exact-match tiers + alias table + fuzzy residual | This phase (D-09) | Trigram on 4-character queries is noise. The alias table (D-08) is the real lever for short queries. |
sendMessage + sendLocation for the destination confirmation |
sendVenue |
D-10 | Telegram's native venue rendering (pin + name + address, tappable in the client) is what SRCH-03 names. |
startapp on the web_app inline button URL (as CONTEXT D-11 reads) |
An explicit carrier choice with a fallback | This phase's research | startapp is officially a direct-link mechanism; the web_app button carries a plain URL. See §"Deep-link risk" — the plan must pick. |
Deprecated / to avoid:
@types/telegram-web-app— Phase 3 deliberately dropped it (the hand-rolledTelegramWebAppLikecontract is the project's contract). Extend that interface; do not re-add the package.bot.chat.*— grammY 1.46 exposes nobot.chatproxy (noted inbot/src/bot/handlers/location-fallback.ts:34-35). Usebot.api.*/ctx.*.damerau-levenshtein(the old CJS-ish package) — last published 2022; prefer@crob/damerau-levenshteinor hand-roll.
Assumptions Log
Claims tagged
[ASSUMED]below need user confirmation before becoming a locked decision.
| # | Claim | Section | Risk if Wrong |
|---|---|---|---|
| A1 | The web_app inline button's WebAppInfo.url may carry a plain query param (?dest=<id>) that survives into the Mini App webview's location.search. |
§"Deep-link risk" / Carrier A | MEDIUM-HIGH. Supported by community reports (StackOverflow: "passing query params to telegram web app just works") but not by Telegram's official docs, which only describe startapp for direct links. If wrong, Carrier A collapses and the phase must use Carrier B (t.me deep link) — which changes the UX (an interstitial) and the button type. Mitigation: build the carrier as a swappable function and test it on a real device at the exit gate. |
| A2 | Telegram.WebApp.initDataUnsafe.start_param is populated when a Mini App is launched from a direct link (t.me?startapp=…) — i.e. that the "direct link" mechanism populates start_param in initData. |
§"Deep-link risk" / Carrier B | LOW-MEDIUM. Officially documented (start_param "will also be passed in the GET-parameter tgWebAppStartParam"), but the WebAppInitData.start_param row's wording ("Only returned for Mini Apps when launched from the attachment menu via link") is narrower and could be read as excluding direct links. |
| A3 | The gazetteer is a Derivative Database under ODbL (because entries are selected/curated from OSM data) and therefore must be offered under ODbL. | §"Security Domain" | LOW. The consequence is mild (add an ODbL notice + attribution; the data is already public). But if the curation were deemed a Produced Work rather than a Derivative Database, the share-alike obligation would not attach. Mitigation: add the notice anyway — it costs nothing and satisfies both readings. |
| A4 | A ~200–500-entry gazetteer with ~5 aliases each is sufficient to hit ≥90% top-3 on a 60–100-query hand-written corpus. | §"Golden corpus" / D-14 | MEDIUM. This is the phase's core empirical assumption. It is exactly what the corpus runner is designed to falsify. If it fails, D-14's escalation ladder applies (alias/normalizer edits first). |
| A5 | Node 24's default TypeScript type-stripping lets a .mjs script import("../src/foo.ts") — the pattern already used by bot/test/commands.test.mjs. |
§"Recommended Project Structure" | LOW. The existing tests already rely on it and pass, so the pattern is proven in-repo. Verify in the plan's first task anyway. |
| A6 | The Mini App public URL can be derived from WEBHOOK_URL (the Worker's origin) rather than needing a new env var. |
§"Runtime State Inventory" | LOW. bot/wrangler.jsonc has workers_dev: true and docs/CONFIGURATION.md documents WEBHOOK_URL as the full origin + path. The plan may add a dedicated MINI_APP_URL var for clarity — a small, cheap decision. |
| A7 | checkHomeScreenStatus() returning "unknown" on iOS is a platform limitation, not a bug to work around. |
§"ONBD-03 feasibility" | LOW. Confirmed upstream (Telegram-iOS issue #1729, closed with the comment "On iOS it is not possible to determine whether a Mini App has been added to the Home Screen"). The design must not branch on iOS status — treat "unknown" as "supported but indeterminate". |
| A8 | sendVenue messages can carry an inline keyboard that is later edited with editMessageReplyMarkup without destroying the venue payload. |
§"Alternatives Considered" | LOW-MEDIUM. The Bot API docs say a message can be edited "without reply_markup or with inline keyboards" — but a venue message's content is a venue, and editMessageText would replace it. editMessageReplyMarkup only touches the markup. Verify on device; the safe fallback is to send a new sendVenue (which also matches D-02's "one tap to switch"). |
Open Questions (RESOLVED)
STATUS: ALL RESOLVED (2026-10-23, during plan-phase). Each question below carries its
(RESOLVED — …)decision line; the decisions are binding and are implemented in plans 04-01…04-04. Where a plan CORRECTED the original recommendation, the correction is stated.
-
Which deep-link carrier does Phase 4 ship? (D-11 says
?startapp=dest_<id>; the docs putstartappon direct links, not inline buttons.)- What we know:
WebAppInfo.urlis an arbitrary HTTPS URL; community reports say query params survive;startappis documented fort.melinks andstartattachfor attachment-menu links; theweb_appinline button is a distinct launch surface. - What's unclear: whether Telegram's own clients preserve the query string on a
web_appbutton URL across Android/iOS/Desktop. - Recommendation: implement Carrier A (
web_appbutton,?dest=<id>in the URL) as the primary, keep Carrier B (t.me?startapp=) for the ONBD-03 home-screen path, and write the payload reader to accept both (location.searchfirst, theninitDataUnsafe.start_param). Validate on a real device at the exit gate. Flag to the user, because it touches a locked decision's wording. - (RESOLVED — Carrier A (
?dest=<id>in theweb_appbutton's URL, read fromlocation.search) is primary; Carrier B is the signedinitData'sstart_paramfield (the direct-link / home-screen path), NOTinitDataUnsafe— the recommendation'sinitDataUnsafereading was corrected: only the SIGNEDinitDatais parsed. Both carriers share one validation. Device validation happens at 04-04's blocking gate. Plans 04-03/04-04.)
- What we know:
-
Is the venue card one message or two?
- What we know:
sendVenuehas notextparameter; D-02 wants one tap to proceed; CONTEXT's "specific ideas" mention a one-line clinical confirmation. - What's unclear: whether "one line confirming the destination" is a separate
sendMessagebefore the card, or a property of the card's owntitle/address. - Recommendation: one
sendVenuewhosetitleis the place name andaddressis the curated address — the native card is the confirmation, and this honours D-02's single-message shape. If the user wants a separate clinical line, it costs one extra message and must be added to the copy gate. - (RESOLVED — exactly one
sendVenueper card;sendVenuehas notextparam, so the card's owntitle(canonical name) +address(curated, verbatim) IS the confirmation. Enforced by CR4's request-count gate. Plan 04-03.)
- What we know:
-
What is the exact corpus size and the exact per-query expectation?
- What we know: D-13 says ~60–100 queries with the listed categories; D-14 sets ≥90% top-3 on positives.
- What's unclear: whether the 90% is measured over all positives or over positives excluding the typo bucket (typos are the hardest and the fuzzy tier's raison d'être).
- Recommendation: measure and report per-category rates (exact / abbreviation / glued / typo / alias), and set the D-14 gate on the all-positives figure, with per-category figures reported alongside so a regression is diagnosable. This is OpenCode's Discretion; state it in the plan.
- (RESOLVED — the D-14 gate measures the ALL-POSITIVES top-3 rate (≥ 90%, threshold
0.9pinned in the corpus file); per-category rates are REPORTED alongside and carry authoring floors (landmark ≥ 12, abbreviation ≥ 15, glued ≥ 6, typo ≥ 12, alias ≥ 12, negative ≥ 6). Plan 04-02, corpus format + CO7.)
-
Which
SearchSourceseam shape?- What we know: D-03 requires a pluggable interface with the gazetteer as implementation #1, WASM geocoder as #2, Photon as #3.
- What's unclear: whether the interface is synchronous or async, and whether it lives in the shared package or in
bot/src/search/. - Recommendation:
async search(): Promise<SearchHit[]>(a network-backed backend #3 will need it), defined in the shared package so the webview can use the same types. ThebyId(id)lookup — needed by the callback handler and the webview — should be a separate, synchronous method on the interface, because both the callback handler and the deep-link reader need it and neither should await a network. - (RESOLVED —
SearchSourcelives in the sharedsearch/workspace (search/src/types.ts):async search(q): Promise<SearchHit[]>plus a SEPARATE SYNCHRONOUSbyId(id). The package also exports a synchronoussearch(text, limit?)convenience wrapper for the corpus runner. Plans 04-01 (contract), 04-03/04-04 (consumers).)
-
Is the Overpass script required to have a verified run in this phase?
- What we know:
overpass-api.dereturns 406 without aUser-Agentand 200 with one (verification pass, same host/same paths);overpass.kumi.systemsandoverpass.private.coffeeare unreachable from this network (HTTP 000);overpass.osm.chreturns 400 (Switzerland-only); Nominatim and Geofabrik work. - What's unclear: whether the phase's exit gate requires the Overpass path to have been exercised against live OSM.
- Recommendation: the plan should require the script to run and produce a valid-shaped proposal file (verified against a committed fixture of a real Overpass response, committed to the repo so the test is deterministic), and to document the live-source URL in the script's header. The
User-Agentheader is load-bearing and must be asserted by a test — a missing UA produces a 406 on every endpoint and looks like a dead service. The curated entries themselves should be coordinate-verified against Nominatim (reachable) — which satisfies D-06's "hand-curated seed" with real coordinates. Do not plan around a blocked Overpass: the live path works from this machine once the header is set; the fixture is for determinism, not for working around an outage. - (RESOLVED — no live-Overpass obligation: the script is proven offline against the COMMITTED fixture (
--source=fixture), with--source=overpass|nominatimas the maintainer's live paths; the identifyingUser-Agentis mandatory and TP2-asserted; a busy instance soft-exits 0 (TP4); curated coordinates are spot-verified against Nominatim and recorded indocs/SEARCH-QUALITY.md. Plan 04-02.)
- What we know:
Environment Availability
Audited this session (2026-10-23) by probing each dependency from this machine.
| Dependency | Required By | Available | Version | Fallback |
|---|---|---|---|---|
| Node.js | everything (build, tests, corpus runner) | ✓ | v24.18.0 |
— |
| npm | workspace install | ✓ | 11.16.0 |
— |
| grammy (installed) | bot handler | ✓ | 1.46.0 |
— |
| wrangler (installed) | Worker deploy / local dev | ✓ | 4.151.3 |
— |
| TypeScript (installed) | typecheck | ✓ | 5.9.3 |
— |
| Playwright (installed, web) | capture evidence | ✓ | 1.50.0 |
— |
| curl | top-up script | ✓ | 8.14.1 |
— |
| Python 3 | not needed this phase (theme gallery only) | ✓ | 3.13.5 |
— |
Overpass API (overpass-api.de) |
D-06 top-up script (live) | ✓ with a User-Agent | CORRECTED by the verification pass (see §Appendix A4): the HTTP 406 is caused by the missing User-Agent header, not by an egress filter. Probed 2026-10-23 on the same host: GET /api/status no UA → 406 / with UA → 200; GET / no UA → 406 / with UA → 200; POST /api/interpreter no UA → 406 / with UA → 200 (19 KB of real KL data, osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z). The script's User-Agent is therefore load-bearing, not cosmetic. |
overpass.kumi.systems and overpass.private.coffee are unreachable from this network (connection failure, HTTP 000, with or without a UA); overpass.osm.ch returns 400 (Switzerland-only coverage, as the body noted). The script should still accept a --source= / env override and the automated test should use a committed fixture. Capacity 502s are intermittent — see §Appendix A4. |
Nominatim (nominatim.openstreetmap.org) |
coordinate verification of curated entries | ✓ | live (200; ?q=KLCC returned correct KL nodes) |
— |
| Geofabrik extract | bulk OSM fallback for the top-up script | ✓ | asia/malaysia-singapore-brunei-latest.osm.pbf (206 on range request) — note: not malaysia-latest.osm.pbf (404) |
— |
| Cloudflare account / wrangler auth | deploy | ✗ (not probed — needs a real account) | — | Deploy is a manual exit step; Phase 2/3 precedent |
| A real Telegram client | ONBD-02 / ONBD-03 device validation | ✗ (not available here) | — | Phase 3's device-matrix precedent: a real-device session is an exit-gate obligation |
Missing dependencies with no fallback:
- A live
overpass-api.deendpoint. The Overpass script can still be written and tested against a committed fixture, but it cannot be validated end-to-end from this machine. If the phase's exit gate requires a live Overpass run, it must be done from a different network (or via theoverpass.osm.chmirror, which has no MY coverage). Plan around this. - A real device / Telegram client for ONBD-02/ONBD-03 validation. This is inherent to the phase (Phase 3 set the same precedent with its device matrix).
Missing dependencies with fallback:
- Overpass → Nominatim search, Geofabrik extract, or a committed fixture.
- A fuzzy-distance dependency → hand-roll (recommended) or
@crob/damerau-levenshtein@2.0.0.
Validation Architecture
workflow.nyquist_validationistruein.planning/config.json(notfalse), so this section is required. The repo has custom zero-dependency test harnesses (no Jest/Vitest/Mocha):bot/test/runner.mjs,web/test/runner.mjs,theme/scripts/audit.test.mjs— each auto-discovers*.test.mjsand drains a shared registry.
Test Framework
| Property | Value |
|---|---|
| Framework | Hand-rolled zero-dependency runner (repo-native pattern). bot/test/harness.mjs exports test(name, fn); bot/test/runner.mjs auto-discovers *.test.mjs, runs each, prints PASS/FAIL, sets process.exitCode = failedCount > 0 ? 1 : 0. The web/ runner is byte-for-byte the same shape. |
| Config file | None — no jest/vitest config exists. Suites are discovered by filename convention (*.test.mjs). |
| Quick run command | npm run bot:test (→ node test/runner.mjs in bot/) — runs all bot suites; there is no per-file filter today. npm run web:test for the web side. |
| Full suite command | npm run bot:test && npm run web:test && npm run audit (theme audit) — plus npm run bot:typecheck && npm run web:typecheck. |
| Corpus run command | node bot/scripts/gazetteer-corpus.mjs (to be created) — the D-14 bake-off close. Should be wired into npm run bot:test or run as an explicit gate; the plan must decide (recommend: a dedicated npm run search:corpus script, invoked by the phase's exit gate, and also asserted inside a *.test.mjs suite so npm run bot:test fails when the corpus regresses). |
| Typecheck | npm run bot:typecheck / npm run web:typecheck → tsc --noEmit per workspace. |
Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|---|---|---|---|---|
| SRCH-01 | "KLCC" (and the golden landmarks) resolve to the expected entry; the shortlist's first item is the right one | unit (corpus) | node bot/scripts/gazetteer-corpus.mjs → asserts top-1/top-3 ≥ threshold |
❌ Wave 0 — bot/scripts/gazetteer-corpus.mjs |
| SRCH-01 | The gazetteer module loads, every entry has the required fields, ids are unique and slug-safe, every entry has ≥1 alias | unit | npm run bot:test (suite bot/test/gazetteer.test.mjs) |
❌ Wave 0 — bot/test/gazetteer.test.mjs |
| SRCH-02 | "Jln Ampang" and "Jalan Ampang" normalize to the same string; "Tmn"→"taman"; "Kg"→"kampung"; "Prsn"→"presint"; glued "tamanbangsar" splits | unit (rule vectors) | npm run bot:test (suite bot/test/normalizer.test.mjs) |
❌ Wave 0 — bot/test/normalizer.test.mjs |
| SRCH-02 | Every abbreviation-table row has a round-trip vector (query form → expected canonical) | unit | npm run bot:test (suite bot/test/normalizer.test.mjs) |
❌ Wave 0 |
| SRCH-02 | Transposition typos ("KCLC") and substitution typos ("Sunwya Pyarmid") resolve to the expected entry | unit (corpus) | node bot/scripts/gazetteer-corpus.mjs |
❌ Wave 0 |
| SRCH-02 | The Damerau-Levenshtein implementation matches hand-computed OSA distances (test vectors from the literature: distance("ab","ba")==1, distance("a cat","an act")==2) |
unit | npm run bot:test (suite bot/test/distance.test.mjs) |
❌ Wave 0 — bot/test/distance.test.mjs |
| SRCH-02 | Exact-match tier strictly outranks any fuzzy hit; scores under the min-length guard return 0 | unit | npm run bot:test (suite bot/test/score.test.mjs) |
❌ Wave 0 |
| SRCH-03 | The bot sends a sendVenue with the right latitude/longitude/title/address, plus an inline keyboard containing the OPEN button and 2–4 alternates |
integration (source-introspection, matching the repo's commands.test.mjs CS4 style) |
npm run bot:test (suite bot/test/search-command.test.mjs) |
❌ Wave 0 — bot/test/search-command.test.mjs |
| SRCH-03 | The venue-card builder's payload is structurally correct (latitude/longitude are finite numbers; title non-empty; address non-empty) | unit | npm run bot:test (same suite) |
❌ Wave 0 |
| SRCH-03 | The alternates' callback_data is "alt:<id>", ≤64 bytes, and carries no query text, name, or coordinates |
unit (grep + parse) | npm run bot:test (same suite) |
❌ Wave 0 |
| SRCH-03 | answerCallbackQuery is called on every callback path (the 30-second rule) |
source-introspection | npm run bot:test (same suite) |
❌ Wave 0 |
| SRCH-03 | A negative query ("Kuching", "Penang") yields an empty shortlist and the "not in v1 area" copy — never a fabricated pin | unit + corpus | node bot/scripts/gazetteer-corpus.mjs + npm run bot:test |
❌ Wave 0 |
| SRCH-03 | The search copy exists in both en and ms, and the locale key-sets are identical (the Phase 2 CS1 gate, extended) | unit | npm run bot:test (extend bot/test/commands.test.mjs CS1 or add a new suite) |
✅ extend — bot/test/commands.test.mjs |
| ONBD-02 | The webview resolves a dest id from its own gazetteer copy to a destination with valid lat/lng |
unit | npm run web:test (suite web/test/destination.test.mjs) |
❌ Wave 0 — web/test/destination.test.mjs |
| ONBD-02 | An unknown id produces an honest "destination unavailable" state, not a blank map | unit | npm run web:test (same suite) |
❌ Wave 0 |
| ONBD-02 | The bot's gazetteer id set == the web's gazetteer id set (drift gate) | integration (cross-workspace) | npm run bot:test (a suite that reads both JSON files) |
❌ Wave 0 |
| ONBD-02 | The bot bundle and the web bundle both contain the gazetteer JSON (build-artifact gate) | integration | npm run web:test (extend web/test/build-artifact.test.mjs) |
✅ extend — web/test/build-artifact.test.mjs |
| ONBD-02 | The mock harness can simulate a preloaded-destination boot (?dest=<id> profile) and the capture run shows the destination state |
manual + capture | npm run web:capture (extend web/scripts/capture.mjs) |
✅ extend — web/scripts/capture.mjs |
| ONBD-02 | The deep-link carrier actually delivers the id on a real device (Android + iOS) | manual-only | (no real device in this environment — exit-gate obligation) | ❌ exit gate |
| ONBD-03 | The Mini App exposes addToHomeScreen / checkHomeScreenStatus on the TelegramWebAppLike interface and calls them behind a capability guard (isVersionAtLeast("8.0")) |
unit | npm run web:test (suite web/test/homescreen.test.mjs) |
❌ Wave 0 — web/test/homescreen.test.mjs |
| ONBD-03 | A startapp-less boot lands in the search-ready map (Phase 3's staged boot, no destination) — i.e. the pre-existing path is unchanged | integration | npm run web:capture + npm run web:test |
✅ extend — existing Phase 3 suites |
| ONBD-03 | checkHomeScreenStatus returning "unknown" (iOS) is treated as "supported but indeterminate" — not as an error |
unit | npm run web:test (same suite) |
❌ Wave 0 |
| ONBD-03 | Adding araharah to the home screen launches the app into the search-ready state | manual-only | (real device — exit-gate obligation) | ❌ exit gate |
| D-14 (bake-off close) | The corpus runner exits non-zero when top-3 < 90% or any negative produced a hit | integration | node bot/scripts/gazetteer-corpus.mjs |
❌ Wave 0 |
| D-14 | The corpus file is hand-authored, not generated: at least N cases are not string-equal to any alias in the gazetteer (a tautology check) | unit (structural) | npm run bot:test (suite bot/test/gazetteer.test.mjs) |
❌ Wave 0 |
| D-04 (forkability law) | A LICENSE file exists at the repo root containing GPL-3.0 |
unit (structural) | npm run bot:test (a small suite) |
❌ Wave 0 |
| D-04 / licensing | The gazetteer data file carries an ODbL attribution notice, and the repo README/attribution names OpenStreetMap | unit (structural) | npm run bot:test (same suite) |
❌ Wave 0 |
| D-15 (privacy) | No log statement in bot/src/ writes a search query. Grep gate over bot/src/** for log calls whose arguments reference a query-shaped variable |
unit (grep, matching the repo's existing source-introspection style) | npm run bot:test |
❌ Wave 0 |
| D-15 | The web side writes no destination or query to any storage surface | unit (existing grep gate) | npm run web:test (web/test/no-storage.test.mjs) |
✅ extend — web/test/no-storage.test.mjs |
| D-06 (top-up script) | The script produces a valid-shaped proposal file from a committed fixture of a real Overpass response | unit (fixture-driven) | npm run bot:test (suite bot/test/gazetteer-topup.test.mjs) |
❌ Wave 0 — bot/test/gazetteer-topup.test.mjs |
| D-06 | The script is not imported by the Worker (grep: no overpass/OSM URLs under bot/src/) |
unit (grep) | npm run bot:test (same suite) |
❌ Wave 0 |
| D-05 (backend #2 reachability) | The JSON→SQLite converter script reads the same committed JSON and emits a SQLite file with the same ids | unit | npm run bot:test (suite bot/test/gazetteer-converter.test.mjs) |
❌ Wave 0 |
| D-07 (V8-native JSON) | The Worker bundle contains the gazetteer as inlined JSON — no SQLite/WASM payload | integration | npm run bot:test (extend the build-artifact gate or a bundle-size assertion) |
❌ Wave 0 |
| D-03 (SearchSource seam) | The interface has exactly the agreed methods and the gazetteer implements it; a stub backend can be substituted in a test (proving the seam) | unit | npm run bot:test (suite bot/test/search-source.test.mjs) |
❌ Wave 0 |
Sampling Rate
- Per task commit:
npm run bot:test && npm run web:test(fast — the hand-rolled runner executes in well under a second) plusnpm run bot:typecheck && npm run web:typecheck. - Per wave merge: the same, plus
node bot/scripts/gazetteer-corpus.mjs(the corpus must be ≥ threshold at every wave merge, not just at the end — this is what makes the escalation ladder actionable rather than a last-minute scramble). - Phase gate: full suite green (
bot:test,web:test,theme audit, both typechecks) and the corpus runner green, and the two manual-only items (ONBD-02 device deep-link, ONBD-03 home-screen launch) verified on a real device, and the D-14 decision recorded in the repo.
Wave 0 Gaps
Everything below is new for this phase (the repo has no search-related test infrastructure today).
bot/test/gazetteer.test.mjs— entry schema, unique ids, alias presence, id-set parity with the web copy, corpus-is-not-tautological, LICENSE + ODbL notice presentbot/test/normalizer.test.mjs— per-rule test vectors for the full abbreviation table (D-09: "Per-rule test vectors; extends by table edit, not code change"), folding, glued-word split, Presint/Prsnbot/test/distance.test.mjs— Damerau-Levenshtein (OSA) correctness vectorsbot/test/score.test.mjs— tier disjointness, min-length guard, exact-outranks-fuzzy invariantbot/test/search-command.test.mjs—sendVenuepayload shape, inline keyboard contents,callback_dataformat/length/no-leakage,answerCallbackQueryon every path, negative-query copybot/test/search-source.test.mjs— theSearchSourceinterface + a stub backend proving the seambot/test/gazetteer-topup.test.mjs— the Overpass proposal script against a committed fixture; the no-OSM-URL-in-Worker grep gate; theUser-Agentheader is asserted to be set (missing UA → 406 on every endpoint — Pitfall 6b); 502/429/timeout → exit 0 with a "busy" message (asserted against a fakefetch, never the live service)bot/test/gazetteer-converter.test.mjs— the D-05 JSON→SQLite converter (id set preserved)bot/scripts/gazetteer-corpus.mjs— the D-14 corpus runner (the bake-off close)bot/test/corpus-gate.test.mjs(or fold into the above) — assertsnpm run bot:testfails when the corpus regressesweb/test/destination.test.mjs— id → destination resolution, unknown-id honest stateweb/test/homescreen.test.mjs—addToHomeScreen/checkHomeScreenStatuscapability guard,"unknown"handlingshared/data/golden-corpus.json— the pinned D-13 corpus (~60–100 hand-written cases)shared/data/kl-gazetteer.json— the curated entries (~200–500) with a committed Overpass-response fixture for the top-up scriptbot/test/fixtures/overpass-response.json— a real Overpass JSON response, committed so the top-up test is deterministic- Framework install: none needed — the repo's hand-rolled runner discovers
*.test.mjsautomatically. No Jest/Vitest/Mocha. - Extend (not new):
bot/test/commands.test.mjs(CS1 locale parity for the new search strings),web/test/no-storage.test.mjs(destination/query non-persistence),web/test/build-artifact.test.mjs(gazetteer JSON present in the web bundle)
Security Domain
security_enforcementis absent from.planning/config.json→ treated as enabled. This section is required.
Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---|---|---|
| V2 Authentication | yes (inherited) | The Mini App's session gate is Phase 2's bot/src/security/initdata.ts HMAC validation + POST /api/session. This phase adds no new identity surface. The destination id is not an identity — it is a public place id (D-11 explicitly says no crypto is needed because the payload's only threat is a user pranking themselves). |
| V3 Session Management | yes (inherited, unchanged) | Phase 2's 900 s memory-only session token. The destination must NOT be written to the session token's storage (there is none) nor to the token's payload (it would leak a user choice into a credential). Recommendation: the destination lives in module scope in the webview, alongside lastFix, exactly like the existing in-memory-only pattern. |
| V4 Access Control | yes (inherited, unchanged) | The Worker owns /api/*. This phase adds no new endpoint — the destination is resolved entirely from the webview's local copy of the shared gazetteer (D-11). That is the right design: no endpoint means no access-control surface. If the plan adds an endpoint (e.g. /api/destination?id=…), it must derive anything user-specific from the resolved claim, never from the request body — the Phase 2 handleLocation/handleLocationFallback precedent. |
| V5 Input Validation | yes (new surface) | The search query is untrusted user input. Rules: (a) the /cari handler must bound the query length (e.g. ≤128 chars) before any processing; (b) the callback_data id must be validated against /^[a-z0-9-]{1,64}$/ before the gazetteer lookup; (c) the webview's dest id must be validated with /^[A-Za-z0-9_-]{1,64}$/ before the lookup; (d) the start_param charset is officially restricted to [A-Za-z0-9_-] and ≤512 chars — validate against exactly that. Never interpolate an unvalidated id into a query, a URL, or a log line. |
| V6 Cryptography | no new crypto — by design | D-11: the payload needs none. The existing HMAC path is unchanged. Do not introduce a signed-token scheme for the destination — it would be security theatre against a self-inflicted threat and would break the "no crypto, shared module by construction" design. |
| V7 Error Handling / Logging | yes (binding) | D-15 + Phase 2's D-14 law: category-only logs. A failed search logs [search] rejected: no-match — never the query, never the results, never a user id, never a timestamp. The "not in v1 area" copy must be honest (D-13's negatives) and must not fabricate a pin. |
| V8 Data Protection / Privacy | yes (binding) | The query is user payload and is not persisted (D-15). If the plan adds KV result caching (D-15 permits it: TTL-only, short TTL, keyed by normalized query), note: the key contains the normalized query — i.e. the query text is the KV key, which is arguably persistence of the query. Recommendation: skip the KV cache entirely. At ~2500 candidate pairs the search is microseconds; a cache buys nothing and creates a privacy surface D-15 explicitly does not want. (D-15 makes the cache optional: "if any", and OpenCode's Discretion says "perf is likely fine without at this scale" — this research confirms that judgment.) |
Known Threat Patterns for this stack (grammY + Cloudflare Workers + Telegram Mini App)
| Pattern | STRIDE | Standard Mitigation |
|---|---|---|
| Query text persisted in a log or a KV key | Information Disclosure | D-15 category-only logging; a grep gate over bot/src/**; skip the KV cache (see V8). |
Callback-data injection (callback_data carrying a name/query) |
Tampering | callback_data is only "alt:<entryId>"; the id is regex-validated and looked up in the gazetteer. Never echoed into a reply without validation. |
| Destination id injection into the webview | Tampering | /^[A-Za-z0-9_-]{1,64}$/ before the lookup; an unknown id yields an honest "unavailable" state, never a fabricated coordinate. |
| Open redirect / unvalidated Mini App URL | Spoofing | The web_app button URL is built from a configured origin, never from user input. The plan must not let a query string influence the URL's origin. |
| Fabricated location pin for an out-of-area query | Repudiation (honesty failure) | D-13's negatives: an empty result must produce the honest refusal copy, never the nearest KL entry. This is a correctness and honesty requirement, not just a UX one — and it is directly asserted by the corpus runner. |
| Gazetteer id drift between bot and webview | Integrity | The id-set parity gate (§"Phase Requirements → Test Map") + ids are never renamed, only added. |
| Overpass fetch inside the Worker (SSRF-ish + availability) | Denial of Service | The Overpass script is a repo tool, never imported by bot/src/ — enforced by a grep gate. |
| OSM/ODbL attribution omission | Compliance | The gazetteer carries an ODbL notice + attribution; the repo README names OpenStreetMap. |
answerCallbackQuery not called |
Denial of Service (UX spinner) | The repo's existing convention (bot/src/bot/handlers/location-fallback.ts pattern) plus a catch-all bot.on("callback_query:data", …) that always answers — grammY's documented recommendation. |
Malicious start_param reaching the webview |
Tampering | initDataUnsafe is untrusted by Telegram's own docs. Use it only to look up a public place id in a local gazetteer; never as identity, never as a URL, never as a path. |
Sources
Primary (HIGH confidence)
- https://core.telegram.org/bots/api/methods/sendvenue.md — the exact
sendVenueparameter set. Fetched this session. Confirms:latitude,longitude,title,addressare required; there is notextparameter;reply_markupacceptsInlineKeyboardMarkup. - https://core.telegram.org/bots/api/types/inlinekeyboardbutton.md —
InlineKeyboardButton. Fetched this session. Confirms:callback_datais 1–64 bytes;web_appis aWebAppInfo;web_appis "Available only in private chats between a user and the bot"; exactly one of the mutually-exclusive fields must be set. - https://core.telegram.org/bots/api/types/webappinfo.md —
WebAppInfo.urlis "An HTTPS URL of a Web App to be opened with additional data". Fetched this session. - https://core.telegram.org/bots/api/types/venue.md — the
Venueobject shape. Fetched this session. - https://core.telegram.org/bots/webapps — Mini Apps launch surfaces,
WebAppInitDatafield semantics,addToHomeScreen()/checkHomeScreenStatus()/homeScreenAdded/homeScreenChecked(Bot API 8.0+). Fetched this session. The source of thestart_paramscoping ambiguity (see §"Deep-link risk"). - https://docs.telegram-mini-apps.com/platform/start-parameter — the start-parameter charset (
[A-Za-z0-9_-], ≤512 chars) and the regex/^[\w-]{0,512}$/. Fetched this session. - https://docs.telegram-mini-apps.com/platform/launch-parameters —
tgWebAppStartParamis "a custom string value passed in the bot or application link". Fetched this session. - https://opendatacommons.org/licenses/odbl/1-0/ — the ODbL 1.0 legal text. Fetched this session. §2.3(a) (programs excluded), §4.2/§4.3 (notices + produced-work attribution), §4.4 (share-alike on a Derivative Database), §4.6 (machine-readable offer).
- https://osmfoundation.org/wiki/Licence/Attribution_Guidelines — OSMF's safe-harbour attribution rules. Fetched this session. Confirms the database case: "include attribution to OpenStreetMap and either the text of the ODbL or a link to it as part of the database, derivative database, or database as part of a collective database… in a location where users would be likely to look for it, such as a readme file, or within the data or metadata."
- https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations and the JUPEM gazetteer PDFs (
mygeoname.mygeoportal.gov.my/exportpdf00.jsp?kodnegeri=16) — the official Malaysian abbreviation tables (street prefixes + locality prefixes + the 48-entry gazetteer abbreviation list). Fetched this session. The normative source for the normalizer's rule table. - https://raw.githubusercontent.com/osm-search/Nominatim/master/settings/icu-rules/variants-ms.yaml — the machine-readable transcription of the JUPEM table (shared / MY / SG / BN sections). Fetched this session. Reproduced verbatim in §"Malay normalizer" via the code example.
- https://github.com/osm-search/Nominatim/pull/4056 — the PR that expanded the table from 8 entries to 65+, with the JUPEM + OSM-wiki source citations and the verified test queries ("Jln Bukit Bintang" → "Jalan Bukit Bintang, Kuala Lumpur", "Tmn Tun Dr Ismail" → "Taman Tun Dr Ismail, Kuala Lumpur", …). Fetched this session.
- https://jupem.gov.my/v1/wp-content/uploads/2016/08/Toponymic-Guidelines-For-Map-and-Other-Editors-for-International-Use.pdf — JUPEM's Toponymic Guidelines (2017, First Edition), the authority behind the table.
- https://govdocs.sinarproject.org/documents/ministry-of-international-trade-and-industry/department-of-standards-malaysia/ms-2039-2006 — MS 2039:2006 (Malaysian address standard). Fetched this session. §5.2.3.2 (the "abbreviation shall not be applied to names" rule) and Annex A (the prefix abbreviation tables A1–A5). The direct authority for Pitfall 3.
- https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL — Overpass QL reference:
bboxordering (s,w,n,e),nwr,out tags/out center,out count(the empty-result-detection idiom). Fetched this session. - https://grammy.dev/plugins/keyboard — grammY inline-keyboard + callback-query handling, including the catch-all
bot.on("callback_query:data")recommendation. Fetched this session. - **https://www.npmjs.com/package/@crob/damerau-levenshtein
/npm view` — the distance-function alternatives and their versions. Verified against the registry this session.
Secondary (MEDIUM confidence)
- https://core.telegram.org/constructor/keyboardButtonWebView — the MTProto schema confirming that an inline-button Mini App launch passes
urltomessages.requestWebView. Supports the read that the button's URL is used as given. - https://github.com/TelegramMessenger/Telegram-iOS/issues/1729 —
checkHomeScreenStatus()always returns"unknown"on iOS; closed with the maintainer comment "On iOS it is not possible to determine whether a Mini App has been added to the Home Screen." Confirms the ONBD-03 platform limitation. - https://stackoverflow.com/questions/76186178/how-to-pass-parameters-to-telegram-web-app — community confirmation that query params on a
web_appbutton URL reach the webview. Corroborates Carrier A but is not official. - **https://www.postgresql.org/docs/current/pgtrgm.html` + the fuzzy-matching comparison articles (tektite, fuzzypoint, taxify vignette) — the trigram/Jaro-Winkler/Damerau trade-off analysis underpinning the tier design and Pitfalls 4–5.
- https://github.com/cbeier-studio/xolmis — the live precedent for a GPL-3.0 project shipping ODbL data in the same repo with an explicit dual-license note.
- **https://github.com/ahozir-ux/bm-address-normaliser
— an existing Malay address normaliser (JS + Python + a Cloudflare Worker), useful as a cross-check on the abbreviation set and on theKG`-ambiguity handling. Community-maintained; treat as a lead, not an authority. - Real commercial Malaysian address listings (Maybank2U merchant PDF, OMRON Healthcare retailer PDF, Petronas station lists, Malaysia.worldplaces, the myboost KL outlet PDF) — the empirical evidence for Presint/Prsn usage ("JALAN P8G PRESINT 8", "PRSN RESIDEN DESA PARKCI", "Precint 15", "PRESINT 16") and for the everyday abbreviation forms locals actually type.
Tertiary (LOW confidence — flagged for validation)
- Stack Overflow #76486463 — an older (2023) claim that a start param is "limited to 64 Symbols". The official docs now say 512. Use 512. Flagged because the stale figure could mislead.
- Community assertions that
start_paramis populated for direct-link launches — the officialWebAppInitData.start_paramwording is narrower than the launch-parameter docs. This is the single most consequential uncertainty in the phase (see Assumptions A1/A2 and Open Question 1). Validate on a real device before the phase gate.
Metadata
Confidence breakdown:
- Standard stack: HIGH — every version verified against the npm registry this session (
grammy1.46.0,@crob/damerau-levenshtein2.0.0,fastest-levenshtein1.0.16,damerau-levenshtein1.0.8); the repo's own toolchain confirmed by readingpackage.jsonfiles and running the probes. - Architecture: HIGH for the Telegram surface (official
core.telegram.orgtype pages fetched this session); MEDIUM for the deep-link carrier (documented ambiguously, community-corroborated only — see Open Question 1). - Malay normalizer: HIGH — the JUPEM tables, MS 2039, and the Nominatim
variants-ms.yamltranscription were all fetched verbatim this session, and the rule table is reproduced directly from a machine-readable upstream file rather than recalled. - Licensing: HIGH for the ODbL text and the OSMF guidelines (both fetched this session); MEDIUM for the Derivative-Database characterization of a curated gazetteer (a legal judgment this research can inform but not make).
- Pitfalls: HIGH — each is anchored to an official doc statement or a verified probe. Corrected by the verification pass: the 406 on
overpass-api.dewas initially attributed to an "egress block"; a controlled same-host probe varying only theUser-Agentheader showed it is the missing UA (with-UA → 200 with real data). The 502-capacity behaviour and the unreachable mirrors are as observed. - Environment availability: HIGH — every row was probed this session, not assumed.
Research date: 2026-10-23 Valid until: 2026-11-06 (30 days) for the stack/architecture/licensing; 2026-10-30 (7 days) for the Overpass-egress finding — network filters change, and the mirror landscape is volatile. Re-probe before the phase's execution begins.
Appendix: Independent Verification Pass (2026-10-23)
A second, independent research pass re-verified this document's load-bearing claims against fresh fetches, a live Overpass re-probe, and a from-scratch reimplementation of the CPU benchmark. No claim in the body was contradicted. Two claims were strengthened, one was corrected, and one was refined.
A1. CPU benchmark — independently reproduced ✅
The body's central engineering constraint (naive full-scan Damerau ≈ 8.16 ms/query against a 10 ms Workers Free limit) was reimplemented from scratch — separate OSA-Damerau implementation, separate trigram inverted index, synthetic 500-entry / 2 000-alias gazetteer, Node v24.18.0:
| Implementation | This pass | Research body | Verdict |
|---|---|---|---|
| Naive full-scan Damerau (2 000 aliases) | 8.243 ms/query | 8.16 ms/query | REPRODUCED (±1%) |
| Trigram inverted index | 0.058 ms/query | 0.004 ms/query | Same order of magnitude; this pass's index was not micro-optimized (it scores every candidate rather than short-circuiting on the top-k). The conclusion is identical: the indexed path is ~140× faster and sits ~170× inside the CPU budget. |
Implication for the planner: unchanged and reinforced. A linear scan is a latent Error 1102 (Workers error 1102 = "Worker exceeded resource limits"). The trigram inverted index is not an optimization — it is a correctness requirement at D-07's stated 200–500 entry scale.
A2. The deep-link landmine — confirmed at the MTProto layer ✅
The body's highest-value finding (a web_app inline button cannot populate start_param) was re-verified by fetching the authoritative method page:
https://core.telegram.org/method/messages.requestWebView —
start_param: "If the web app was opened from the attachment menu using a attachment menu deep link,start_paramshould contain thedatafrom thestartattachparameter."
The method has a start_param field (flags.3), but the official description restricts its population to the attachment-menu path. The url field (flags.1) is the "Web app URL" and is passed through verbatim — which is why the button's own query string reaches window.location.search but never the signed initData.
The sibling method https://core.telegram.org/method/messages.requestAppWebView confirms the other side: start_param: "If the startapp query string parameter is present in the direct Mini App deep link, pass it to start_param."
Implication for the planner: unchanged and reinforced. The body's Design A/B/C analysis stands. The body's own caveat (§Tertiary sources) that the community consensus on start_param population is "narrower than the launch-parameter docs" is resolved at the protocol level for the carrier question — but note the residual uncertainty is now narrower than the body states: it is no longer "does start_param get populated at all for direct links" (it does, per requestAppWebView), but only which BotFather configuration the project actually has (Main Mini App vs Direct Mini App vs attachment menu), because that determines whether a t.me link prompts for confirmation. See §Open Questions.
A3. "KLCC" / "DUKE" have no usable OSM landmark name — reproduced ✅
A live Overpass probe this pass (nwr["name"~"KLCC",i](3.05,101.55,3.20,101.75); and nwr["short_name"="DUKE"]) returned HTTP 200 with timestamp_osm_base: 2026-10-23T14:55:20Z and reproduced the body's finding exactly:
- "DUKE": every match is a
highway=motorwaysegment ofLebuhraya DUKE(short_name=DUKE,alt_name="Karak Link",official_name="Lebuhraya Duta–Ulu Klang",ref=E33). 38 of 40 returned elements were motorway segments. "DUKE" is a highway acronym, not a destination. - "KLCC" as a bare name: matches are dominated by
public_transport=stop_position/railway=stop(LRT),highway=bus_stop,amenity=parking_entrance, plus hostels and show galleries — not the landmark. - The real landmark IS present, under a different name:
way 23034336name="Suria KLCC",building:use=retail,name:en="Suria KLCC", withaddr:street="Jalan Ampang",addr:city="Kuala Lumpur",addr:postcode="50088"on nearby nodes — i.e.addr:*tags are available for thesendVenueaddressfield (Pitfall 11).
Implication for the planner: unchanged and reinforced. The body's conclusion — the corpus cannot be auto-derived from OSM names; the alias list IS the product — is now verified at the level of individual element tags. The seed gazetteer's name field must be the landmark's canonical OSM name ("Suria KLCC"), and "KLCC" must be an explicit aliases entry (D-08), never a name. The Overpass script proposes OSM-named candidates; a human adds the local shorthand.
A4. Overpass availability — CORRECTED (the body reproduced the symptom but misattributed the cause)
⚠️ This section supersedes the body's §Environment Availability row and its §Sources "egress filter" framing. The body concluded that
overpass-api.dewas unusable — "HTTP 406 from an nginx egress filter on every request, incl.GET /" — and marked it ✗. The symptom was reproduced; the cause was misattributed. A controlled probe of the same host, same paths, same session, varying only theUser-Agentheader, resolves it:
| Request | No User-Agent |
With User-Agent |
|---|---|---|
GET https://overpass-api.de/api/status |
HTTP 406 | HTTP 200 |
GET https://overpass-api.de/ |
HTTP 406 | HTTP 200 |
POST https://overpass-api.de/api/interpreter (KL bbox query) |
HTTP 406 | HTTP 200 — 19 056 bytes, osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z |
The 406 is caused by the missing User-Agent, not by an egress filter. With the header, the primary instance works and returns real, current KL data.
Reachability of the alternatives, probed the same way:
| Instance | No UA | With UA | Verdict |
|---|---|---|---|
overpass-api.de |
406 | 200 | ✅ the working one |
overpass.kumi.systems |
000 | 000 | ❌ unreachable from this network (connection failure — HTTP 000 is not an HTTP status) |
overpass.private.coffee |
000 | 000 | ❌ unreachable from this network |
overpass.osm.ch |
400 | 400 | ❌ Switzerland-only coverage — confirms the body's note |
Capacity is a separate, intermittent phenomenon. Earlier in this same session, overpass-api.de, overpass.kumi.systems and overpass.private.coffee all returned HTTP 502 with the body "This Overpass API server is currently at capacity. All query workers are busy and the request queue is full." Minutes later the same query on overpass-api.de returned 200. The 502 is a queue-saturation response to a narrow time window, not a property of the host.
Implications for the planner (these supersede the body's row):
overpass-api.deIS usable — with aUser-Agent. The body's "✗ / egress filter" verdict was wrong and, if carried into the plan, would have produced needless--source=/local-extract complexity or skipped the D-06 top-up entirely. Do not plan around a blocked Overpass.- The
User-Agentheader is load-bearing, not a nicety. Without it, every endpoint returns 406. It should be the script's first line of defence. - The other two public instances are unreachable from this network — so "primary + fallback instance" is not a real safety net here. The committed fixture (already in the body's Wave 0 list) is the reliable path for automation, and one live endpoint plus that fixture is the honest architecture.
- 502/429/timeout → print "busy, retry later" and exit 0. Never a stack trace, never a non-zero exit. (Now written up as §Pitfall 6b.)
- Prefer a narrow, single-tag-family query by default — a narrow query is far more likely to fit in a free worker slot during a saturation window.
- Nothing may depend on this script — no deploy, no test run, no corpus gate. The body already says this; this pass confirms it is load-bearing in practice.
A5. start_param length limit — noted as a secondary path only
The body cites 512 characters (the official Bot API community-mirror figure) and explicitly flags the older "64 symbols" Stack Overflow claim as stale in §Tertiary sources. This pass observed that the MTProto layer's platform sibling field is documented as "0-64 English letters, digits, and underscores", while the start_param field itself carries no length constraint in the method schema — so 512 (the Bot API figure) is the number to design against, exactly as the body says.
Implication for the planner: unchanged. The body's recommendation to constrain ids to /^[A-Za-z0-9_-]{1,64}$/ is stricter than 512 and therefore safe. The one thing to note: this constraint only binds the secondary start_param path. The body's primary path (Design B — ?dest=<id> in the Mini App URL's own query string, read from location.search) has no equivalent length cap, which is a further reason to prefer it.
A6. JSON import assertions — the body's "test this early" risk is lower than stated
The body warns that the shared search module must be importable from plain Node and flags the JSON import as a possible blocker. This pass probed it: node -e "import('./package.json', {with:{type:'json'}})…" on Node v24.18.0 → succeeds. The remaining risk is therefore not the Node import path but the Workers bundler path (esbuild via wrangler) — the JSON must be inlined into the bundle, not fetched at runtime (which would be a hard failure in a Worker). One wrangler deploy --dry-run in Wave 0 settles it.
A7. Confirmed as-is (no change needed)
sendVenueacceptsreply_markup— re-fetched https://core.telegram.org/bots/api/methods/sendvenue.md; the official parameter table listsreply_markupwithInlineKeyboardMarkupfirst. D-02's one-message shape is supported.WebAppInfohas exactly one field — re-fetched https://core.telegram.org/bots/api/types/webappinfo.md; the table has a single row (url). Nostart_paramfield.- Workers Free limits — re-fetched https://developers.cloudflare.com/workers/platform/limits/ (page dated 2026-10-12): 10 ms CPU/request, 100 000 req/day, 128 MB, 1 s startup, ~2.2 ms average, "avoid expensive work in global scope".
- The ODbL Derivative-Database analysis — the Trivial Transformations Guideline's geocoding clause re-fetched and unchanged.
bot/scripts/set-menu-button.mjsexists (Pitfall 12's remedy is real).- npm versions unchanged:
grammy1.46.0,fastest-levenshtein1.0.16,damerau-levenshtein1.0.8,string-similarity4.0.4,levenshtein-edit-distance3.0.1 — and additionally probedjaro-winkler0.2.8,trigram-similarity1.0.7,didyoumean1.2.2,fuzzysort4.0.2,fuse.js7.5.0. Every dedicated string-distance package exceptfuse.js/fuzzysortis 2022-vintage and unmaintained — which reinforces the body's "hand-roll, zero dependency" call for a GPL-3.0 forkability-law project.
Verification ledger
| Body claim | Verdict | Evidence this pass |
|---|---|---|
| Naive full-scan Damerau ≈ 8.16 ms/query vs a 10 ms limit | ✅ REPRODUCED (8.243 ms) | From-scratch reimplementation, 500 entries / 2 000 aliases, Node 24.18.0 |
| Trigram inverted index ≈ 0.004 ms/query | ✅ REPRODUCED in order of magnitude (0.058 ms) | Same run; conclusion (≈140× faster, ~170× inside budget) unchanged |
web_app inline button cannot populate start_param |
✅ CONFIRMED | messages.requestWebView official page: start_param is populated only from an attachment-menu startattach link |
| "KLCC" has no OSM landmark name; "DUKE" is a highway acronym | ✅ CONFIRMED | Live Overpass (HTTP 200, osm3s.timestamp_osm_base: 2026-10-23T14:55:20Z): 38/40 DUKE hits are highway=motorway; "Suria KLCC" is way 23034336 with addr:* tags |
| Overpass requires a User-Agent (406 without) | ✅ CONFIRMED | curl https://overpass-api.de/api/status (no UA) → HTTP 406 |
| Overpass availability | ⚠️ CORRECTED — usable with a User-Agent; the 406 is a missing-UA rejection, not an egress block. Capacity 502s are intermittent. |
Same-host/same-path probe varying only the UA: no-UA → 406 on all three paths, with-UA → 200 with real KL data. kumi.systems / private.coffee unreachable (000); osm.ch → 400 |
sendVenue accepts reply_markup |
✅ CONFIRMED | Official method page parameter table |
WebAppInfo = { url } only |
✅ CONFIRMED | Official type page (single row) |
| Workers Free 10 ms CPU / 1 s startup / 128 MB | ✅ CONFIRMED | Official limits page, dated 2026-10-12 |
| ODbL Derivative-Database obligation | ✅ CONFIRMED | Trivial Transformations Guideline (geocoding clause) |
start_param = 512 chars |
✅ STANDS (community-mirror figure; MTProto documents no length on the field itself) | Body's stricter /^[A-Za-z0-9_-]{1,64}$/ id constraint is safe either way |
| JSON import assertions in the Node runner | ✅ WORKS (risk is the Workers bundler, not Node) | node -e "import('./package.json', {with:{type:'json'}})" → OK on Node v24.18.0 |
bot/scripts/set-menu-button.mjs exists |
✅ CONFIRMED | ls bot/scripts/ → register-commands.mjs, set-menu-button.mjs, set-webhook.mjs |
Bottom line for the planner: this document's findings stand as written, with one correction (the Overpass availability verdict — see §A4) and one addition (§Pitfall 6b: the User-Agent is load-bearing and 502s are normal). The deep-link carrier decision, the indexed-lookup requirement, the alias-led corpus strategy, the sendVenue one-message shape, and the ODbL dual-licence are all verified and ready to plan against.
The three things the planner must not get wrong:
- Use a trigram inverted index, not a linear scan. Reproduced at 8.24 ms/query against a 10 ms limit — a linear scan is a latent
Error 1102. - Do not ship D-10's
web_appbutton as written. It cannot carry astart_param. Choose Design A (urlbutton →t.meMain Mini App link,start_parampopulated and HMAC-signed) or Design B (web_appbutton →?dest=<id>in the URL, read fromlocation.search), or Design C (both). This is the one decision that most changes the phase's shape. - "KLCC" and "DUKE" are not OSM landmark names. 38/40 live "DUKE" hits are
highway=motorwaysegments. The gazetteer'snamemust be the canonical OSM name (Suria KLCC); the local shorthand is analiasesentry. The alias list is the product, and it cannot be generated.