ambek .planning

.planning/research/SUMMARY.md

@@ -0,0 +1,182 @@
+# Project Research Summary
+
+**Project:** araharah
+**Domain:** Telegram-first privacy-respecting GPS navigation (bot + Mini App, OSM data, KL metro)
+**Researched:** 2026-10-22
+**Confidence:** HIGH (stack + architecture core, verified against official docs) / MEDIUM-HIGH (features) / MEDIUM (geolocation client quirks, KL data quality, backgrounded turn-card UX)
+
+## Executive Summary
+
+araharah is a turn-by-turn car navigation assistant delivered entirely inside Telegram: a bot handles search and launch, a Mini App (webview) renders the Rams-designed guidance HUD, and the map/routing data comes from the OSM ecosystem. The research converges on an unusually crisp shape for v1: **grammY on a single Cloudflare Worker** (webhook + `/api/*` + static Mini App assets via Workers static assets — not legacy Pages), **MapLibre GL JS v6 + OpenFreeMap** vector tiles styled from a forked `liberty` style JSON into the Telemetry Rams palette, and **self-hosted Photon + OSRM on one small VPS** fed by a Geofabrik Malaysia extract. Storage is **Cloudflare KV with TTLs only — zero databases, zero queues** in v1. Privacy is not a bolt-on; the client-authoritative guidance architecture (server never tracks position, everything ephemeral) makes the brand promise true by construction. Every public/community OSM service (OSRM demo, photon.komoot.io, public Nominatim) is explicitly non-production and throttles or forbids exactly this traffic pattern — self-hosting a country extract is cheap (tens of GB, ~€5–10/mo) and the only privacy-consistent option.
+
+The recommended build order follows the strict dependency funnel: design tokens first (design fidelity is the user-flagged #1 risk), then bot skeleton, Mini App shell + geolocation, destination search (KL landmark quality is the biggest data experiment), routing, then live guidance, then hardening. Two features define "is this real navigation": **KL-accurate landmark search** ("KLCC" must resolve) and **missed-turn auto-recalculation** — both get dedicated research and test-corpus treatment.
+
+Key risks, in priority order: (1) **design-fidelity death by translation** — the Tailwind-CDN mockups must be distilled into a tokens package with screenshot-diff acceptance against the mockups, never normalized into a generic component library; (2) **Telegram WebView geolocation quirks** — `LocationManager` (Bot API 8.0+) is the primary path, with Android stale-fix bugs coded around and a chat-side `request_location` fallback for Desktop; (3) **turn-by-turn engineering complexity** — snapping jitter, recalculation storms, ETA whiplash, and battery drain all need an explicit guidance state machine with hysteresis and debounce; (4) **KL OSM data quality** — patchy `maxspeed`, weak `turn:lanes`, inconsistent interchange modeling; scope the quality bar honestly and validate against a golden-route corpus; (5) **ops-stack scope creep** — RabbitMQ/PostGIS/Grafana/Prometheus from IDEAS.md are explicitly deferred with trigger conditions.
+
+## Key Findings
+
+### Recommended Stack
+
+The viable 2026 stack is already assembled from cloud-native + OSM parts (full detail in STACK.md). Total infra cost ≈ €5–10/mo VPS + $0 Cloudflare.
+
+**Core technologies:**
+- **Telegram Bot API 10.3 + grammY 1.46** (`@grammyjs/conversations`) — webhook mode on Workers, secret-token validation, KV-backed sessions
+- **Cloudflare Worker (single deployable, static assets)** — bot webhook + `/api/*` (Hono) + Mini App SPA in one `wrangler.jsonc` unit; Pages is legacy per Cloudflare's own docs (this resolves the Pages mention in earlier project docs)
+- **telegram-web-app.js + LocationManager** — initData identity, persistent geolocation grant on mobile, BackButton/theme/haptics/fullscreen platform glue
+- **Vite ^8 + TypeScript + MapLibre GL JS ^6.15 + OpenFreeMap** — no SPA framework; <200 KB gzip budget; custom style JSON = the Rams look; requires explicit `setWorkerUrl` wiring in bundlers
+- **Self-hosted Photon** (GraphHopper Malaysia dump, ~<5 GB) — the only free geocoder with native typeahead + typo tolerance; behind a Worker proxy with KV caching
+- **Self-hosted OSRM v26.10** (GHCR image, not stale Docker Hub) — car routing with `steps=true`, alternatives, single-digit-ms latency on a Malaysia extract
+- **Cloudflare KV** (2 namespaces: `chat_sessions` minutes-TTL, `guidance_sessions` ~2h TTL) — the only storage in v1
+
+**What NOT to use:** OSRM demo server, public Nominatim (1 req/s cap, forbids autocomplete), photon.komoot.io in production, Google/Mapbox/HERE, Neon/PostGIS/RabbitMQ/Prometheus/Grafana/Resend in v1, MapTiler as primary (non-commercial free tier + forced logo), heavy SPA frameworks, MapLibre ≤ v4.
+
+### Expected Features
+
+Full prioritization in FEATURES.md. The funnel has two layers: navigator table stakes + Telegram-native table stakes.
+
+**Must have (v1 / table stakes):**
+- `/start` onboarding + commands; free-text destination search → inline keyboard results → Mini App launch with destination via `startapp`
+- KL-accurate search: curated landmark/alias gazetteer (~200 entries) + Malay abbreviation normalization ("Jln"↔"Jalan") layered over Photon, KL-bounded
+- `sendVenue` destination card; geolocation via LocationManager + chat `request_location` fallback
+- Route draw + 2–3 alternative cards (Rams-quiet); turn-by-turn HUD (next-maneuver banner + collapsible list)
+- **Missed-turn detection + auto recalculation** — the "is this real navigation" bar
+- ETA + remaining distance progression (honest, self-paced — no live traffic claims)
+- Haptic turn cues; Telegram platform hygiene (BackButton, closing confirmation during guidance, `disableVerticalSwipes`, theme integration, safe areas)
+- Visible privacy statement in-product (`/privacy`, "trip ended, deleted" confirmation)
+
+**Should have (v1.x differentiators):**
+- Backgrounded chat turn-cards (the marquee differentiator — guidance that lives in the chat; needs ephemeral server-side guidance session + static map snapshots; framed as best-effort projection, never promised as real-time)
+- Voice guidance (mute-by-default, iOS webview TTS reliability risk), live-trip share via `sendLocation(live_period)`, toll-free route toggle, speed-limit display where OSM `maxspeed` exists
+
+**Defer (v2+):** opt-in anonymized crowd speeds → traffic-weighted ETA (v1 ships only an architecture stub); incident reports (per PROJECT.md scope); gamification; AES camera layer (legal gray area).
+
+**Anti-features (explicitly rejected):** ads/sponsored pins, gamification, re-engagement pushes, accounts/sign-up (initData is all), fat settings menus (max 3 route choices: fastest / toll-free / shorter), offline maps product, AI-assistant chat.
+
+### Architecture Approach
+
+One Cloudflare Worker serves everything; the Mini App owns the guidance state machine; the server stays stateless and ephemeral (full detail in ARCHITECTURE.md).
+
+**Major components:**
+1. **Bot/API Worker** — webhook intake, command router, search orchestration, geocoder/router proxies, chat turn-card pushes, secret custody. Both bot and Mini App call the identical `geo/` module.
+2. **guidance_sessions (KV)** — the only "database": who's driving where, ~2h TTL, auto-purged. Everything longer does not exist in v1.
+3. **Mini App (static SPA)** — map render (MapLibre + Rams style), guidance state machine `idle → routing → guiding ⇄ rerouting → arrived`, off-route detection local (snap to route polyline, >50m for >10s with hysteresis), recalculation is a client decision.
+4. **Geocoder/Router boundaries** — TS interfaces in the Worker; self-hosted Photon/OSRM behind them, demo servers allowed only in local dev via config flag; PostGIS/Valhalla slot in later without rewrites.
+
+**Key patterns:** webhook with secret path; initData → HMAC-validated session token exchange (Pattern 2); `start_param` destination handoff via short KV id; client-authoritative guidance (anti-pattern: server-driven turn-by-turn — webview JS pauses when minimized); Tailwind v4 preset + tokens package generated from mockup YAML, mockups as screenshot acceptance criteria only. Monorepo: `apps/worker`, `apps/web`, `packages/design-tokens`.
+
+### Critical Pitfalls
+
+Top 5 (full 10 + checklists in PITFALLS.md):
+
+1. **Trusting `initDataUnsafe` / client-side validation** — HMAC with key `= HMAC(bot_token, "WebAppData")` (not the raw token), sorted `\n`-joined data-check string, `auth_date` freshness, one-time session-token exchange. Build the validator as the first backend artifact with CI vectors + tampered payloads.
+2. **Silent geolocation death in Telegram WebView** — `navigator.geolocation` can fail with code 1 and no prompt in clients lacking the permission bridge. LocationManager primary, version-gated, `navigator.geolocation` fallback, chat `request_location` universal fallback, denied-screen; tested on a real Android/iOS/Desktop/browser matrix as the location slice's exit criterion. Android LocationManager stale-fix bug → combine both sources via one `getLiveData()` wrapper.
+3. **Turn-by-turn engineering traps** — snapping (project GPS onto route polyline), reroute debounce/cooldown (≥5–8s, one OSRM call per off-route event), ETA smoothing (windowed speed, no visible whiplash), adaptive `watchPosition` sampling + Wake Lock only during active guidance; validate with recorded-drive replays in CI.
+4. **Design-fidelity erosion** (user's #1 risk) — extract tokens from mockup HTML before any screen; no default-styled component libraries; per-UI-phase side-by-side mockup-vs-screenshot approval artifact; reject refactors touching spacing/color/typography without a mockup diff.
+5. **KL search quality + OSM data gaps** — Nominatim-style failures on "KLCC"/"Jln"/"DUKE" queries kill the marquee feature; the curated alias gazetteer is the substance of the search phase, not polish. OSM Malaysia has patchy turn-lane/flyover data → golden-route test corpus (10–20 real KL pairs), honest quality bar, no lane-guidance claims.
+
+Also load-bearing: Cloudflare Worker limits (free plan: 10ms CPU / 50 subrequests — cache-first, one intent = one small invocation, ≤6 subrequests worst-case, measured via `wrangler tail`), and ops-scope-creep discipline (every v1 service maps to a shipped requirement).
+
+## Implications for Roadmap
+
+Based on combined research, a 7-phase skeleton (Pitfalls assumed P1–P7 roles; Architecture's P0 maps to Phase 1).
+
+### Phase 1: Design Tokens + Component Foundation
+**Rationale:** Design fidelity is the flagged #1 risk; tokens are zero-risk work every later phase consumes. Building UI first and tokens later is the single most likely way to lose the Rams look. (Architecture P0; Pitfalls P1.)
+**Delivers:** `packages/design-tokens` (exacted hexes, 8px grid + documented exceptions, fonts self-hosted as woff2 subsets), Tailwind v4 preset, ~15 hand-rolled Rams components, user-approved side-by-side gallery-vs-mockup artifact, map style JSON base (forked OpenFreeMap `liberty` restyled to Rams).
+**Addresses:** visible platform for all FEATURES P1 UI items. **Avoids:** Pitfall 9 (fidelity death by translation).
+
+### Phase 2: Bot Skeleton + initData Security
+**Rationale:** The bot is how users reach the Mini App; the `startapp` handshake needs a real bot to test against. The initData validator must be the first backend artifact. (Architecture P1; Pitfalls P1–P2.)
+**Delivers:** Worker deployed with webhook + secret path, `/start` + command menu, grammY + KV conversations, initData HMAC validator with CI vectors, session-token exchange, `/privacy`.
+**Uses:** grammY, Hono, KV, wrangler. **Avoids:** Pitfall 1 (identity spoofing) — tampered/stale-payload tests gate the phase.
+
+### Phase 3: Mini App Shell + Geolocation
+**Rationale:** Session/auth + geolocation plumbing precedes any geo-service consumption; this is the riskiest platform-integration phase and its unknowns (WebView quirks, permission bridges) must surface early — the exit criterion is reading a live position on a real phone across the client matrix. (Architecture P2; Pitfalls P3–P4.)
+**Delivers:** telegram-web-app.js shell (`ready`/`expand`, viewport handling — never `100vh`, BackButton wired to router, CloudStorage for tokens, theme vars), init/session flow, LocationManager + fallbacks (`getLiveData()` wrapper, denied screen, manual map-origin fallback), map rendering user position on styled basemap, boot with empty `start_param`.
+**Avoids:** Pitfalls 2 (silent geolocation death) and 4 (localStorage/viewport/back-button traps).
+
+### Phase 4: Destination Search (Geocoder + KL Gazetteer)
+**Rationale:** Nothing user-meaningful exists until search resolves; search quality is the data-correctness experiment with the most unknowns — a Photon bake-off belongs here (budgeted explicitly, per Architecture gap #2). (Architecture P3; Pitfalls P5.)
+**Delivers:** VPS provisioned with self-hosted Photon (GraphHopper MY dump) behind Worker proxy + KV cache; free-text bot search → inline keyboard results; curated KL landmark/alias gazetteer (~200 entries) + Malay abbreviation normalizer; `/api/search`; real-KL query acceptance set ("KLCC", "Sunway Pyramid", "Jln", "DUKE").
+**Addresses:** search table stakes. **Avoids:** Pitfall 5 (Nominatim rate walls + landmark weakness).
+
+### Phase 5: Routing Service + Route Display
+**Rationale:** Full search → route → render loop lands here — the "product exists" milestone. Self-hosted OSRM must stand up *before* guidance makes routing latency-sensitive and chatty. Route model must carry step geometry + snapped-progress support from day one (Pitfall 3 pre-work). (Architecture P4; Pitfalls P6–P7, P8.)
+**Delivers:** Self-hosted OSRM (GHCR image, MY extract, preprocess pipeline in Docker Compose); Router interface with demo-server dev-only config; route computation with `steps=true` + alternatives; Mini App route-choice screen (2–3 quiet Rams cards); static map snapshot renderer groundwork; Golden-route corpus (10–20 real KL pairs) + subrequest-budgets instrumentation.
+**Addresses:** route display + alternatives table stakes. **Avoids:** Pitfalls 6 (demo-server dependency), 7 partially (golden routes), 8 (Worker limits).
+
+### Phase 6: Turn-by-Turn Guidance (Live Driving)
+**Rationale:** Builds on every prior piece and carries the most behavioral unknowns (reroute triggers, hysteresis tuning, ETA smoothing) — schedule slack here. Hybrid chat turn-cards may be its own follow-up phase or folded in; backgrounded turn-card framing (best-effort projection) needs a small product decision + live experiment. (Architecture P5; Pitfalls P3.)
+**Delivers:** Client guidance state machine (snapped progress, off-route hysteresis, debounced reroute), maneuver banner + collapsible list, ETA progression, haptics, Wake Lock handling, `heartbeat` TTL endpoint, chat turn-cards (best-effort projection while backgrounded), closing confirmation during guidance.
+**Addresses:** missed-turn recalculation, ETA, haptics, backgrounded cards. **Avoids:** Pitfall 3 (recalc storms/ETA whiplash/battery).
+
+### Phase 7: Hardening + Private Launch
+**Rationale:** The empty/error/permission states ARE the polish surface of a navigation tool; a named hardening phase (Architecture P6, Pitfalls pre-launch P7) enforces the "Looks Done But Isn't" checklist before real drivers.
+**Delivers:** Timeout/off-road/no-geometry states, version-gating fallbacks, budgets audit (subrequests/CPU, cache hit-rate), battery drain QA (<10%/hr), full device matrix pass, privacy verification (no coordinates in logs), golden-route regression pass, known-limitations doc.
+**Avoids:** Pitfalls 7 (data honesty), 8 (limits), plus the "looks done but isn't" checklist.
+
+### Phase Ordering Rationale
+
+- Strict dependency funnel: tokens → bot/auth → shell/location → search → route → guidance → hardening; each phase produces something end-to-end verifiable.
+- Phase order deliberately surfaces unknowns early where they're cheap: WebView geolocation (Phase 3) and KL search quality (Phase 4) are the two experiments with the most uncertainty.
+- v1.x features (voice, live-trip share, toll toggle, speed limits) hang off validated routing+guidance and should not enter the 7-phase spine until the core proves out — plan a v1.x milestone after user validation.
+- Apparent conflict resolved: PROJECT.md/IDEAS.md say "Cloudflare Pages" for hosting; both ARCHITECTURE.md and STACK.md converge on **Workers with static assets** (one deployable, Cloudflare's directed path for all new projects). Recommend recording this as a Key Decision.
+
+### Research Flags
+
+Phases likely needing `/gsd-research-phase` during planning:
+- **Phase 3 (Shell + Geolocation):** client-matrix quirks, Android LocationManager stale-fix workaround, browser-based testing harness — MEDIUM-confidence community findings.
+- **Phase 4 (Search):** Photon-vs-alternatives KL bake-off is a budgeted data experiment; gazetteer sourcing/structure.
+- **Phase 6 (Guidance):** backgrounded turn-card product framing ("best-effort projection"), static map snapshot generation, voice guidance TTS reliability (if Phase 6.x).
+
+Phases with standard patterns (skip research-phase, use established docs):
+- **Phase 1 (Design tokens):** internal asset work; referenced mockups are the spec.
+- **Phase 2 (Bot + auth):** grammY + Cloudflare Workers hosting is first-class documented (official tutorial covers the exact setup verbatim).
+- **Phase 5 (Routing/OSRM):** standardize on documented preprocess pipeline; no exotic APIs.
+
+## Confidence Assessment
+
+| Area | Confidence | Notes |
+|------|------------|-------|
+| Stack | HIGH | Core versions verified against registries/official docs 2026-10-22; explicit "why not" for every rejected alternative; MEDIUM for geolocation workarounds + snapshot path |
+| Features | MEDIUM-HIGH | Telegram platform details HIGH (official docs); competitor/UX findings MEDIUM; KL-local conventions MEDIUM (some single-source, flagged) |
+| Architecture | HIGH | Core verified against official Telegram/grammY docs; two flagged platform-behavior gaps (turn-card UX, Photon KL quality) MEDIUM |
+| Pitfalls | HIGH (security/platform) / MEDIUM (nav-engine UX, KL data quality) | Official sources for security/limits; navigation-engineering practice synthesized, community-corroborated |
+
+**Overall confidence:** HIGH on the shape of the system; MEDIUM on KL-specific data quality and phone-in-hand platform behavior — both have concrete experiments baked into the phase plan.
+
+### Gaps to Address
+
+- **Backgrounded turn-card acceptability:** what "good enough" means for minimized-app hints is a product decision requiring a small live experiment — resolve during Phase 6 planning.
+- **Photon KL search quality:** untested against real KL queries (abbreviation, mall, subdistrict phrasing) — Phase 4 must budget a bake-off with a fallback (self-hosted Nominatim or hand-seeded alias file escalation).
+- **Voice guidance on iOS WKWebView:** `speechSynthesis` reliability LOW confidence, single-source inference — phase-specific verification before planning voice features (v1.x).
+- **OSM maxspeed completeness per KL arterial / AES camera availability:** LOW — defer speed-limit display until data is sampled; AES stays in v2.
+- **Bot API Rich Messages (`InputRichBlockMap`, 10.2):** potential native chat map blocks; flagged watch-only, not a v1 dependency.
+- **Bot API "Serverless" (June 2026):** noted but not recommended; watch-only.
+- **Exact `initData` validation API shape at grammY 1.46:** code-verified HIGH for mechanics, verify exact typings at Phase 2 start.
+
+## Sources
+
+Aggregated from STACK.md, FEATURES.md, ARCHITECTURE.md, PITFALLS.md (full source lists with per-source confidence in each file).
+
+### Primary (HIGH confidence)
+- Telegram Bot API / Mini Apps official docs (core.telegram.org) — Bot API 10.3, LocationManager, initData HMAC recipe, BackButton/viewport/CloudStorage, start_param
+- OpenFreeMap official site + GitHub; MapTiler pricing/terms
+- Cloudflare Workers docs (static assets, limits, migrate-from-Pages); Cloudflare Pages pricing
+- Nominatim usage policy (OSMF); OSRM demo-server policy; OSRM v26.10.0 / Valhalla 3.9.1 releases; Docker Hub staleness (GHCR is current)
+- npm registry version checks (grammy, maplibre-gl, wrangler, hono, vite, zod, turf) 2026-10-22
+
+### Secondary (MEDIUM confidence)
+- Telegram-Mini-Apps / tma.js issue trackers (Android LocationManager stale fix, Desktop geolocation gap, viewport quirks)
+- OSM Malaysia quality studies; OSM wiki addressing/tagging
+- grammY community walkthroughs (Workers + KV/DO patterns); AdsGram TMA mistakes write-up; Photon/Nominatim ecosystem notes
+- Waze/Google Maps 2025–2026 comparative reviews
+- Turn-by-turn engineering practice (snapping, hysteresis, replays) — known-wisdom synthesis
+
+### Tertiary (LOW confidence — validate at phase research)
+- WKWebView `speechSynthesis` reliability; per-arterial maxspeed completeness; AES camera data in OSM
+- Bot API Serverless (June 2026) maturity; exact grammY 1.46 typed API surface
+
+---
+*Research completed: 2026-10-22*
+*Ready for roadmap: yes*