863 lines
75 KiB
Markdown
863 lines
75 KiB
Markdown
---
|
|
phase: 04-destination-search-kl-gazetteer
|
|
plan: 01
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- package.json
|
|
- search/package.json
|
|
- search/tsconfig.json
|
|
- search/src/types.ts
|
|
- search/src/rules/abbreviations.ts
|
|
- search/src/normalize.ts
|
|
- search/src/distance.ts
|
|
- search/src/score.ts
|
|
- search/src/data/kl-gazetteer.json
|
|
- search/src/gazetteer.ts
|
|
- search/src/source.ts
|
|
- search/src/index.ts
|
|
- search/test/runner.mjs
|
|
- search/test/harness.mjs
|
|
- search/test/normalizer.test.mjs
|
|
- search/test/distance.test.mjs
|
|
- search/test/score.test.mjs
|
|
- search/test/gazetteer.test.mjs
|
|
- search/test/loader-env.test.mjs
|
|
- LICENSE
|
|
- LICENSES.md
|
|
- README.md
|
|
autonomous: true
|
|
requirements: [SRCH-01, SRCH-02]
|
|
must_haves:
|
|
decisions_implemented:
|
|
- "D-04: GPL-3.0 LICENSE at the repo root; data as plain committed files; no hardcoded secrets (the search module reads no env at all)"
|
|
- "D-05: the JSON dataset format stays reachable from day one — the curated file is plain JSON, so backend #2 (WASM geocoder) is a format change, not a rewrite"
|
|
- "D-06: the curated gazetteer is hand-curated seed data committed to the repo (the Overpass top-up script is plan 04-02)"
|
|
- "D-07: canonical format is typed JSON in the repo — V8-native parse, no WASM/sqlite in the Worker bundle"
|
|
- "D-08: aliases live natively inside each entry as an `aliases` array — locals' shorthand is the corpus-quality lever"
|
|
- "D-09: the normalizer is a rule table + folding + fuzzy-residual module; the abbreviation table is DATA (a file edit extends it, never a code change); normalization is applied symmetrically to query AND entry aliases"
|
|
- "D-15: the shared module contains no logging and no storage — the query text never leaves the caller"
|
|
truths:
|
|
- "A user query 'KLCC' resolves to the Suria KLCC entry id 'suria-klcc' through the shared module, with no network call and no infrastructure"
|
|
- "'Jln Ampang' and 'Jalan Ampang' normalize to the IDENTICAL string; 'Tmn' expands to 'taman'; 'Kg' to 'kampung'; 'Prsn' to 'presint'"
|
|
- "A transposition typo ('KCLC') still resolves to the expected entry id, via the Damerau-Levenshtein residual tier"
|
|
- "An exact normalized match ALWAYS outranks any fuzzy match — the tier score bands are disjoint and asserted by a test"
|
|
- "The gazetteer JSON loads identically in three environments: the Worker bundle (esbuild/wrangler — inline proven on the search module's own esbuild bundle, dry-run as the build guard), the Vite bundle, and bare Node 24 (the corpus runner)"
|
|
- "The Worker never builds an index at module scope — the index is built lazily on first search and memoized"
|
|
- "A search over the full gazetteer costs well under the Workers Free 10 ms CPU/request budget"
|
|
- "The repo carries a GPL-3.0 LICENSE and the gazetteer carries an ODbL-1.0 + OpenStreetMap attribution notice"
|
|
- "No log line anywhere in the shared module can carry query text — the module has no logging at all"
|
|
artifacts:
|
|
- path: "search/src/types.ts"
|
|
provides: "GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts (D-03, D-07, D-08)"
|
|
contains: "export interface SearchSource"
|
|
- path: "search/src/rules/abbreviations.ts"
|
|
provides: "the vendored JUPEM-derived bidirectional abbreviation table as DATA (D-09)"
|
|
contains: "ABBREVIATION_ROWS"
|
|
- path: "search/src/normalize.ts"
|
|
provides: "pure normalize(): NFKD-fold → whitespace/punctuation collapse → whole-token abbreviation expansion → glued-word split (D-09)"
|
|
- path: "search/src/distance.ts"
|
|
provides: "hand-rolled Damerau-Levenshtein (OSA) distance — zero runtime dependencies"
|
|
- path: "search/src/score.ts"
|
|
provides: "the disjoint-tier scorer: EXACT > ALIAS_EXACT > PREFIX > TOKEN_SET > FUZZY (D-09)"
|
|
contains: "export const TIER"
|
|
- path: "search/src/data/kl-gazetteer.json"
|
|
provides: "the committed curated KL gazetteer with per-entry aliases (D-06, D-07, D-08)"
|
|
contains: "\"entries\""
|
|
- path: "search/src/gazetteer.ts"
|
|
provides: "the lazy loader + index: parse-once, memoized, three-environment safe (D-07)"
|
|
- path: "search/src/index.ts"
|
|
provides: "the public API surface — search(), normalize(), byId(), entries — imported by bot, web, and the corpus runner"
|
|
- path: "LICENSE"
|
|
provides: "the GPL-3.0 license text (D-04)"
|
|
- path: "LICENSES.md"
|
|
provides: "the dual-license note: code GPL-3.0, gazetteer data ODbL-1.0 with OSM attribution (D-04, ODbL §4.4)"
|
|
- path: "search/test/normalizer.test.mjs"
|
|
provides: "per-rule vectors for the whole abbreviation table + idempotence + glued-split vectors (D-09)"
|
|
- path: "search/test/distance.test.mjs"
|
|
provides: "hand-computed OSA distance vectors (SRCH-02)"
|
|
- path: "search/test/score.test.mjs"
|
|
provides: "tier disjointness, min-length guard, exact-outranks-fuzzy invariants (SRCH-02)"
|
|
- path: "search/test/gazetteer.test.mjs"
|
|
provides: "entry schema, unique slug-safe ids, alias presence, ODbL header, LICENSE presence (D-07, D-08)"
|
|
- path: "search/test/loader-env.test.mjs"
|
|
provides: "the three-environment loader proof: bare Node import, the esbuild inline-bundle marker + wrangler dry-run build guard, the Vite static-import assertion (the phase's top mechanical risk)"
|
|
key_links:
|
|
- from: "search/src/normalize.ts"
|
|
to: "search/src/rules/abbreviations.ts"
|
|
via: "the whole-token expansion reads the imported table; a table edit extends behavior without a code change (D-09)"
|
|
pattern: "ABBREVIATION_ROWS"
|
|
- from: "search/src/gazetteer.ts"
|
|
to: "search/src/data/kl-gazetteer.json"
|
|
via: "a static JSON import (build-time inlined), never a runtime fetch — Workers have no fs"
|
|
pattern: "kl-gazetteer.json"
|
|
- from: "search/src/gazetteer.ts"
|
|
to: "search/src/normalize.ts"
|
|
via: "every entry alias is normalized at index-build time, so both sides of a comparison are canonical (D-09 symmetry law)"
|
|
pattern: "normalize"
|
|
- from: "search/src/score.ts"
|
|
to: "search/src/distance.ts"
|
|
via: "the FUZZY tier's similarity is 1 - damerau/maxLen"
|
|
pattern: "damerau"
|
|
- from: "search/src/index.ts"
|
|
to: "search/src/gazetteer.ts"
|
|
via: "the public search()/byId() surface delegates to the lazily-built index"
|
|
pattern: "export"
|
|
---
|
|
|
|
<objective>
|
|
Create the shared, dependency-free search module that the bot, the Mini App, and the corpus runner all import — the pure foundation the whole phase rests on — and land the repo's licensing (GPL-3.0 code + ODbL-1.0 gazetteer data) in the same change.
|
|
|
|
Purpose: D-03 makes the search backend a pluggable `SearchSource` seam, D-07 fixes the dataset format as typed JSON, and D-11 requires the bot and the webview to map the SAME gazetteer id → destination. All three are satisfied by one shared workspace package (`@araharah/search`) imported by both `bot/` and `web/`. This plan builds that package: the JUPEM-derived abbreviation table as editable DATA, the symmetric normalizer, the hand-rolled Damerau distance, the disjoint-tier scorer, the curated KL gazetteer with locals' aliases, and the lazy three-environment loader. It also de-risks the phase's #1 mechanical risk (RESEARCH: "the JSON loader must survive THREE environments — wrangler bundle / Vite bundle / bare Node corpus runner") with an explicit spike.
|
|
|
|
Output: a fourth npm workspace `search/` (package `@araharah/search`) with types, rules, normalize, distance, score, gazetteer loader, the curated `kl-gazetteer.json`, four test suites, the root `LICENSE` (GPL-3.0), `LICENSES.md` (ODbL notice), and a corrected README §License.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.config/opencode/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.config/opencode/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@AGENTS.md
|
|
@.planning/phases/04-destination-search-kl-gazetteer/04-CONTEXT.md
|
|
@.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md
|
|
@.planning/phases/04-destination-search-kl-gazetteer/04-PATTERNS.md
|
|
@bot/src/security/initdata.ts
|
|
@bot/src/i18n/strings.ts
|
|
@bot/src/kv/sessions.ts
|
|
@web/src/location/types.ts
|
|
@web/src/map/map.ts
|
|
@web/tsconfig.json
|
|
@theme/package.json
|
|
@package.json
|
|
@README.md
|
|
|
|
<interfaces>
|
|
Key contracts the executor needs. `GazetteerEntry` / `SearchSource` are given verbatim by RESEARCH §Pattern 1 (04-RESEARCH.md lines 294-319) and must be implemented EXACTLY as shaped here — downstream plans (04-03 bot, 04-04 web) import these names.
|
|
|
|
From `search/src/types.ts` (this plan's output — the contract everything else implements against):
|
|
|
|
```typescript
|
|
export interface GazetteerEntry {
|
|
id: string; // stable, slug-shaped: "suria-klcc", "sunway-pyramid"
|
|
name: string; // canonical display name (the OSM landmark name, e.g. "Suria KLCC")
|
|
aliases: string[]; // locals' shorthand (D-08) — never empty
|
|
lat: number;
|
|
lng: number;
|
|
category: string; // "mall" | "office-tower" | "park" | "station" | ...
|
|
address?: string; // human address for sendVenue's `address` field — shown VERBATIM
|
|
}
|
|
|
|
export interface SearchQuery {
|
|
text: string; // raw user query
|
|
limit?: number; // default 5 (1 best + 2-4 alternates, D-02)
|
|
}
|
|
|
|
export interface SearchHit {
|
|
entry: GazetteerEntry;
|
|
score: number; // 0..1, higher is better
|
|
matchedAlias: string; // which alias matched — the corpus report needs this
|
|
}
|
|
|
|
export interface SearchSource {
|
|
readonly name: string; // "gazetteer" | "wasm-geocoder" | "photon"
|
|
search(q: SearchQuery): Promise<SearchHit[]>; // EMPTY array = honest refusal
|
|
byId(id: string): GazetteerEntry | null; // synchronous — the callback handler + webview both need it
|
|
}
|
|
```
|
|
|
|
`byId` is a SEPARATE SYNCHRONOUS method (not part of `search`): the alternates callback handler and the webview's deep-link reader both need an id → entry lookup and neither should await a network. RESEARCH Open Question 4 recommends exactly this shape.
|
|
|
|
From `bot/src/security/initdata.ts` — the header-contract style every new module copies:
|
|
|
|
```typescript
|
|
/*
|
|
* <file> — <one-line purpose> (<decision/requirement IDs>).
|
|
* ---...
|
|
* <the structural laws this module obeys, citing D-xx/CS-xx>
|
|
*/
|
|
```
|
|
</interfaces>
|
|
|
|
<environment_facts>
|
|
Verified in this session — do not re-derive, do not spend a task re-probing:
|
|
|
|
- Node is **v24.18.0**, npm **11.16.0**. Node 24 strips TypeScript types by DEFAULT, so a `.mjs` script CAN `await import("../src/foo.ts")` — this is the repo's existing pattern (`bot/test/commands.test.mjs` line 27 does exactly this and passes).
|
|
- `import('./package.json', {with:{type:'json'}})` WORKS on this Node (probed: "json import OK").
|
|
- `web/src/map/map.ts` line 20 already does `import ramsStyle from "@araharah/theme/basemap/liberty-rams.json";` — the build-time JSON-inlining precedent for a workspace subpath import. Vite and wrangler/esbuild both inline it.
|
|
- grammY 1.46.0 is installed. `ctx.replyWithVenue(latitude, longitude, title, address, other?)` exists (node_modules/grammy/out/context.d.ts:690). `InlineKeyboard.webApp(text, url)` exists (node_modules/grammy/out/convenience/keyboard.d.ts:594).
|
|
- **`WebAppInfo` has EXACTLY ONE field — `url`** (node_modules/@grammyjs/types/markup.d.ts:219-222). There is NO `start_param` field on a `web_app` inline button. This is the verified basis for the deep-link mechanism correction in plan 04-03.
|
|
- The `search/` workspace must be added to the root `package.json` `"workspaces"` array (currently `["theme","bot","web"]`).
|
|
</environment_facts>
|
|
|
|
<gazetteer_seed_contract>
|
|
The curated gazetteer is the phase's product, not a generated artifact (RESEARCH §Verification A3: "KLCC" has no usable OSM landmark name in OSM — the landmark is `name="Suria KLCC"`, and "DUKE" is a highway acronym; therefore the alias list IS the product, and "KLCC" must be an explicit `aliases` entry, never a `name`).
|
|
|
|
Minimum seed content (hand-authored, coordinates from the landmarks' real KL locations):
|
|
|
|
Required entries (each with `aliases` covering the local shorthand, per D-08):
|
|
- `suria-klcc` — name "Suria KLCC", aliases `["KLCC","KLCC Park","KLCC Mall","Jalan Ampang"]`, category "mall", lat 3.1578, lng 101.7117, address "Jalan Ampang, 50088 Kuala Lumpur" — NOTE: "Jalan Ampang" is the complex's address street and a real local shorthand for it (the corpus's "Jln Ampang"/"Jalan Ampang" cases pin `suria-klcc`); "KL Sentral" must NEVER appear here — it is a different landmark with its own entry, and aliasing it here would be a cross-landmark collision
|
|
- `sunway-pyramid` — name "Sunway Pyramid", aliases `["Pyramid","Sunway","Sunway Pyramid Shopping Mall"]`, category "mall", lat 3.0726, lng 101.6069, address "Jalan PJS 11/15, Bandar Sunway, 47500 Subang Jaya"
|
|
- `the-duke` — name "Lebuhraya DUKE", aliases `["DUKE","Duta-Ulu Klang Expressway","Karak Link"]`, category "highway", lat 3.1357, lng 101.6864, address "Lebuhraya DUKE, 50000 Kuala Lumpur"
|
|
- `pavilion-kl` — name "Pavilion Kuala Lumpur", aliases `["Pavilion","Pavilion KL","KL Pavilion"]`, category "mall", lat 3.1478, lng 101.6950, address "Jalan Bukit Bintang, 55100 Kuala Lumpur"
|
|
- `ijnp` — name "Institut Jantung Negara", aliases `["IJN","IJN Hospital","Institut Jantung Negara Malaysia"]`, category "hospital", lat 3.1636, lng 101.7060, address "Jalan Tun Razak, 50400 Kuala Lumpur"
|
|
- `pwtc` — name "PWTC", aliases `["PWTC","Putra World Trade Centre"]`, category "convention", lat 3.1768, lng 101.6980, address "Jalan Tun Razak, 50400 Kuala Lumpur" — aliases must NOT invent a wrong expansion ("Pusat equated Trade Fair" is not a thing; PWTC = Putra World Trade Centre)
|
|
- `kl-sentral` — name "KL Sentral", aliases `["KL Sentral","Stesen KL Sentral","KL Central","Sentral Station","Stesen Sentral"]`, category "station", lat 3.1343, lng 101.6869, address "Jalan Stesen Sentral 5, 50470 Kuala Lumpur"
|
|
- `bangsar` — name "Taman Bangsar", aliases `["Bangsar","Taman Bangsar","Bangsar Park"]`, category "park", lat 3.1477, lng 101.6875, address "Jalan Maarof, 59000 Kuala Lumpur"
|
|
- `kampung-ampang` — name "Kampung Ampang", aliases `["Kg Ampang","Kampung Ampang","Ampang Village"]`, category "village", lat 3.1456, lng 101.7120, address "Jalan Ampang, 68000 Ampang"
|
|
- `tun-razak` — name "Jalan Tun Razak", aliases `["Tun Razak","Jalan Tun Razak"]`, category "road", lat 3.1560, lng 101.7100, address "Jalan Tun Razak, 50400 Kuala Lumpur"
|
|
|
|
Add at least **20 more** hand-curated KL-metro landmarks to reach the seed floor of 30 entries (10 required + 20 more). Suggested additional KL landmarks with local aliases: `sungei-wang` (aliases "Sungei Wang","Sg Wang","Sungai Wang" — the corpus's "Sg Wang" cases need this entry), `berjaya-times-square` (aliases "Berjaya Times Square","Times Square","BTS"), `mid-valley` (aliases "Mid Valley","Mid Valley Megamall","MV"), `sunway-velocity` (aliases "Velocity","Sunway Velocity Mall"), `ioi-city-mall` (aliases "IOI City Mall","IOI City"), `nu-sentral` (aliases "Nu Sentral","KL Sentral Mall"), `merdeka-118`, `kl-tower` (aliases "KL Tower","Menara Kuala Lumpur"), `petronas-twin-towers` (aliases "Petronas Towers","Twin Towers","KLCC Towers"), `aquaria-klcc`, `central-market` (aliases "Central Market","Pasar Seni"), `petaling-street` (aliases "Petaling Street","Chi Cheong Kai"), `masjid-jamek`, `pudu-raya` (aliases "Pudu Raya","Puduraya"), `bukit-bintang` (aliases "Bukit Bintang","BB Park"), `chinatown`, `little-india-brickfields` (aliases "Brickfields","Little India"), `titiwangsa`, `lake-gardens` (aliases "Lake Gardens","Perdana Botanical Garden"), `national-museum` (aliases "National Museum","Muzium Negara").
|
|
|
|
Every entry MUST have:
|
|
- `id` matching `/^[a-z0-9-]{1,64}$/` (slug-shaped, unique, never renamed once shipped)
|
|
- `name` = the landmark's canonical OSM name (NOT the local shorthand)
|
|
- `aliases` = a NON-EMPTY array including the canonical name itself
|
|
- `lat` / `lng` = the real coordinates (finite numbers, KL metro bbox roughly lat 3.0-3.3, lng 101.5-101.9)
|
|
- `category` from a small closed set: `mall | office-tower | park | station | hospital | road | highway | landmark | convention | village | education | government`
|
|
- `address` = a human address string, shown VERBATIM on the venue card (never the normalized form — RESEARCH Pitfall 3)
|
|
|
|
The JSON file ALSO carries an ODbL attribution header object, so the notice travels with the data (OSMF attribution guidelines require it "within the data or metadata"):
|
|
|
|
```json
|
|
{
|
|
"_license": "ODbL-1.0",
|
|
"_attribution": "© OpenStreetMap contributors",
|
|
"_notice": "Curated from OpenStreetMap data. The curated gazetteer and any derived index are made available under the Open Database License (ODbL) v1.0 — https://opendatacommons.org/licenses/odbl/1-0/",
|
|
"version": 1,
|
|
"entries": [ ... ]
|
|
}
|
|
```
|
|
</gazetteer_seed_contract>
|
|
|
|
<abbreviation_table_contract>
|
|
The table is DATA, vendored from the JUPEM-normative list as transcribed by the OSM Nominatim project in `settings/icu-rules/variants-ms.yaml` (PR #4056, merged 2026-04-02; primary source: "JUPEM Toponymic Guidelines — 'Abbreviations of Geographical Terms and Other Words Used in Maps'"). Vendored wholesale — do NOT author a smaller list (RESEARCH: "the difference between the 8-entry original and the current 65+ entry table is the difference between 'Jln' working and 'Jln Ampang' working").
|
|
|
|
The file `search/src/rules/abbreviations.ts` holds a `ReadonlyArray<readonly [string, string]>` of `[fullForm, commaSeparatedAbbreviations]` pairs, with the header comment citing JUPEM + the Nominatim transcription. Minimum rows (the shared + MY sections of the upstream table):
|
|
|
|
```
|
|
Jalan → Jln
|
|
Lorong → Lrg, Lor
|
|
Lebuh → Lbh
|
|
Simpang → Spg
|
|
Kampung → Kg
|
|
Kampong → Kg
|
|
Kpg → Kampung
|
|
Taman → Tmn
|
|
Bangunan → Bgn
|
|
Bukit → Bkt
|
|
Batu → Bt
|
|
Sungai → Sg
|
|
Pulau → P, Pl
|
|
Tanjung → Tg
|
|
Tanjong → Tg
|
|
Teluk → Tlk
|
|
Telok → Tlk
|
|
Gunung → G, Gn
|
|
Gunong → G, Gn
|
|
Haji → Hj
|
|
Hajjah → Hjh
|
|
Abdul → Abd
|
|
Persiaran → Psrn
|
|
Lebuhraya → Lbh.Raya
|
|
Bandar → Bdr
|
|
Kondominium → Kondo
|
|
Bahagian → Bhg
|
|
Blok → Blk
|
|
Kuala → K, Kl
|
|
Alur → Alr
|
|
Bakau → Bak
|
|
Hulu → H
|
|
Hilir → Hr
|
|
Jeram → J, Jr
|
|
Lembah → Lmbh
|
|
Padang → Pdg
|
|
Parit → Pt
|
|
Permatang → Pmtg
|
|
Pelabuhan → Plbh
|
|
Kepulauan → Kep
|
|
Ladang → Ldg
|
|
Lapangan → Lpg
|
|
Gosong → Gsg
|
|
Sekolah kebangsaan → SK
|
|
Sekolah menengah → SM
|
|
Sekolah menengah kebangsaan → SMK
|
|
Sekolah rendah → SR
|
|
Hospital → Hosp
|
|
Dispensari → Disp
|
|
```
|
|
|
|
Plus the two explicit D-09 vowel-drop rows (RESEARCH: Putrajaya precincts are written both "Presint" and "Prsn" — the latter drops the internal `e`; `Prsn` is NOT a prefix abbreviation):
|
|
|
|
```
|
|
Presint → Prsn
|
|
```
|
|
|
|
"Presint" must map to itself so the canonical long form is always reachable.
|
|
|
|
The module builds the BIDIRECTIONAL map ONCE at module load by inverting the table: every abbreviation maps to its full form(s), and every full form maps to itself. The canonical form is ALWAYS the long form (so "Jln Ampang" and "Jalan Ampang" both normalize to "jalan ampang" — the symmetry law D-09 requires).
|
|
|
|
AMBIGUITY POLICY (RESEARCH §"Ambiguity to handle"): where one abbreviation maps to several full forms (`G` → Gunung/Gunong, `P` → Pulau/Parit…), the normalizer keeps a per-token ambiguity LIST rather than guessing, and the fuzzy-residual tier resolves what the table cannot. In practice: single-letter abbreviations (`G`, `P`, `H`, `J`, `K`, `L`) expand to their FIRST listed full form ONLY when the token is a standalone word AND the expansion is unambiguous in context; the normalizer must NOT expand a single letter that is part of a longer token.
|
|
|
|
Whole-token-only expansion is a hard law (RESEARCH Pitfall 5): "Pdang" is never expanded, only "pdg".
|
|
</abbreviation_table_contract>
|
|
|
|
<normalizer_contract>
|
|
`normalize(input: string): string` — pure, no I/O, no imports beyond the rules table. Pipeline (each step separately testable):
|
|
|
|
1. NFKD normalize + strip combining marks.
|
|
2. Lowercase (casefold).
|
|
3. Punctuation → space, but KEEP `/` (Malaysian address shorthand "Jln 3/1" and "Presint 8/1" need it) and keep `-`? — NO: strip `-` too (it is a separator in "Sunway-Pyramid"). Keep only letters, digits, spaces, and `/`.
|
|
4. Collapse whitespace + trim.
|
|
5. Per-token abbreviation expansion (whole-token only, per the table above).
|
|
6. Glued-word split — LONGEST-MATCH-FIRST against a starter list, requiring the remainder to be >= 3 characters (RESEARCH: "a naive 'insert a space before any known word' will corrupt words that legitimately contain a starter as a substring"). Starters: `jalan, taman, kampung, kampong, lorong, bukit, sungai, bandar, persiaran, lebuh, jalan, perkampungan, presint`. Longest starters first (`kampung` before `kampong` before `kg`-expanded forms are already separate tokens by step 5).
|
|
7. Collapse whitespace again + trim.
|
|
|
|
IDEMPOTENCE LAW: `normalize(normalize(x)) === normalize(x)` for every vector — assert this in the normalizer suite (the repo's T7-style independence self-check, `bot/test/initdata.test.mjs` lines 240-243).
|
|
|
|
The normalizer MUST NOT strip the `/` in "3/1" and MUST NOT expand a token that is a single letter unless it is in the table with an unambiguous mapping.
|
|
|
|
Example vectors that MUST hold (these are the suite's minimum):
|
|
- "Jln Ampang" → "jalan ampang"
|
|
- "Jalan Ampang" → "jalan ampang" (IDENTICAL to the above — the symmetry law)
|
|
- "Jln Tmn Desa" → "jalan taman desa" (both sides expand; RESEARCH Pitfall 3: this is correct for MATCHING even though the DISPLAY form differs)
|
|
- "Tmn Bangsar" → "taman bangsar"
|
|
- "Kg Ampang" → "kampung ampang"
|
|
- "Kpg Ampang" → "kampung ampang"
|
|
- "Prsn 15" → "presint 15"
|
|
- "Presint 15" → "presint 15"
|
|
- "tamanbangsar" → "taman bangsar" (glued split)
|
|
- "Jln 3/1" → "jalan 3/1"
|
|
- "Bdr Baru" → "bandar baru"
|
|
- "Sunway Pyramid" → "sunway pyramid"
|
|
- " SUNWAY PYRAMID " → "sunway pyramid"
|
|
- "Kuala Lumpur" → "kuala lumpur"
|
|
- "PJ" → "pj" (city codes are ALIASES, not abbreviation rules — RESEARCH §"A ready-made city-code table": "PJ" → Petaling Jaya belongs in the gazetteer ALIAS table, not the abbreviation rule table)
|
|
</normalizer_contract>
|
|
|
|
<scorer_contract>
|
|
`score.ts` — the disjoint-tier scorer. The tiers MUST have strictly disjoint, descending score bands so an exact match can never be outranked by a fuzzy hit (RESEARCH Pitfall 4: "the ordering is a property of the TIER, not of the arithmetic").
|
|
|
|
```typescript
|
|
export const TIER = {
|
|
EXACT: 1.00, // normalized query === normalized alias or name
|
|
ALIAS_EXACT: 0.90, // normalized query === normalized alias (entry name is a different string)
|
|
PREFIX: 0.75, // one is a prefix of the other, >= 3 chars
|
|
TOKEN_SET: 0.65, // all query tokens present in the candidate's token set
|
|
FUZZY: 0.60, // the fuzzy band's CEILING — a fuzzy score is sim * 0.60, i.e. always in (0, 0.60)
|
|
} as const;
|
|
|
|
/** A FUZZY hit is ADMITTED only at this similarity or above — this is the
|
|
* mechanism behind D-13's honest refusal, tuned between the corpus's two
|
|
* loads: "KCLC" vs "KLCC" is sim 0.75 (admitted — the typo bucket passes)
|
|
* while "Penang" vs "Padang" is sim 0.667 (refused — no fabricated pin). */
|
|
export const FUZZY_ADMIT_SIMILARITY = 0.72;
|
|
```
|
|
|
|
NOTE the scaling: `FUZZY` is the band's CEILING (0.60), and a fuzzy score is `sim * 0.60` — a naive `TIER.FUZZY: 0.00` multiplied into the similarity would zero the whole tier. The tier values are exact literals; a test asserts them.
|
|
|
|
`scoreOne(queryNorm: string, aliasNorm: string): number`:
|
|
- `queryNorm === aliasNorm` → `TIER.EXACT`
|
|
- either side shorter than 3 characters → `0` (the min-length guard — RESEARCH Pitfall 5: "KLCC" and "DUKE" must be ALIAS hits, not fuzzy hits)
|
|
- prefix (one starts with the other, both >= 3 chars) → `TIER.PREFIX`
|
|
- all query tokens present in the candidate's token set → `TIER.TOKEN_SET`
|
|
- otherwise → the FUZZY residual: `sim = 1 - damerau(a, b) / Math.max(a.length, b.length)`, computed at BOTH the whole-string level AND the token level. The token level compares each query token against each candidate token (both >= 3 chars) and takes the mean of the best per-token similarities, scaled by the fraction of query tokens that found any candidate token. The fuzzy similarity is `max(wholeStringSim, tokenLevelSim)`, and the fuzzy score is `fuzzySim * TIER.FUZZY` — a value in (0, 0.60). The token level is what lets "Sentrl" reach the "KL Sentral" alias (the error is in ONE word — RESEARCH's anti-pattern against whole-string-only fuzzy) while "Jln Tun Razak" still cannot reach "Jalan Ampang" (the non-matching tokens drag the token-level mean down).
|
|
|
|
The FUZZY tier is the RESIDUAL net only (D-09: "fuzzy scoring (Damerau) is only the residual net for genuine typos on longer words"). The TOKEN_SET tier runs BEFORE fuzzy so a one-word error does not cause cross-street confusion ("Jln Tun Razak" must not match "Jalan Ampang").
|
|
|
|
`score.ts` MUST export the tier constants AND a `scoreCandidate` that takes the query's normalized form and an entry's normalized alias set, returning the BEST tier hit plus the `matchedAlias` string (the corpus report needs that provenance).
|
|
</scorer_contract>
|
|
|
|
<distance_contract>
|
|
`distance.ts` — hand-rolled Optimal String Alignment (Damerau-Levenshtein with adjacent transpositions), ~25 lines, ZERO dependencies (RESEARCH §"Don't Hand-Roll": this is the ONE place hand-rolling IS correct — a dependency would add package weight to a Worker bundle D-07 deliberately wants minimal, and every dedicated string-distance npm package is 2022-vintage and unmaintained).
|
|
|
|
```typescript
|
|
export function damerau(a: string, b: string): number {
|
|
if (a === b) return 0;
|
|
if (a.length === 0) return b.length;
|
|
if (b.length === 0) return a.length;
|
|
// 3-row band OSA with the adjacent-transposition term
|
|
// (i > 1 && j > 1 && a[i-1] === b[j-2] && a[i-2] === b[j-1] → prev2[j-2] + 1)
|
|
}
|
|
```
|
|
|
|
Hand-computed vectors the suite MUST assert (from the literature / RESEARCH §Code Examples):
|
|
- `damerau("ab", "ba") === 1` (adjacent transposition)
|
|
- `damerau("a cat", "an act") === 2` (two transpositions)
|
|
- `damerau("", "") === 0`
|
|
- `damerau("", "abc") === 3`
|
|
- `damerau("abc", "") === 3`
|
|
- `damerau("same", "same") === 0`
|
|
- `damerau("klcc", "kclc") === 1`
|
|
- `damerau("sunway", "sunwya") === 1` (adjacent transposition of a real corpus typo)
|
|
- `damerau("kitten", "sitting") === 3` (classic Levenshtein sanity)
|
|
|
|
The row-rotation in a 3-band OSA is easy to get wrong — write the vectors FIRST and make them pass (repo RED→GREEN convention).
|
|
</distance_contract>
|
|
|
|
<gazetteer_loader_contract>
|
|
`search/src/gazetteer.ts` — the dataset loader + index. Two hard laws:
|
|
|
|
**LAW 1 — lazy, memoized, NEVER at module scope (RESEARCH Pitfall 8, Workers 1 s startup limit / "avoid expensive work in global scope").** The pattern is `bot/test/initdata.test.mjs` lines 42-51 (the repo's only lazy-init precedent):
|
|
|
|
```typescript
|
|
let index: Map<string, GazetteerEntry[]> | null = null;
|
|
function loadIndex(): Map<string, GazetteerEntry[]> {
|
|
if (index === null) { /* build once */ }
|
|
return index;
|
|
}
|
|
```
|
|
|
|
**LAW 2 — the loader must survive THREE environments**: the wrangler/esbuild bundle (Worker), the Vite bundle (browser), and bare Node 24 (the corpus runner, which imports the `.ts` source directly via type-stripping). The mechanism is a STATIC JSON IMPORT — `import raw from "./data/kl-gazetteer.json" with { type: "json" }` — because Workers have no `fs` and a runtime fetch would be a hard failure. esbuild and Vite both INLINE a static JSON import at build time. The `with { type: "json" }` attribute is valid on Node 24 (probed) and ignored/harmless under esbuild/Vite (verify with the loader-env suite).
|
|
|
|
The `with { type: "json" }` attribute under **esbuild** is the one thing that must be proven, not assumed: if esbuild rejects the import attribute, fall back to `import raw from "./data/kl-gazetteer.json";` (esbuild inlines JSON by default without the attribute) and let the Node-side import assert via the loader-env suite. The loader-env suite is the arbiter.
|
|
|
|
The module exports:
|
|
- `getEntries(): readonly GazetteerEntry[]` — the parsed entries
|
|
- `getEntryById(id: string): GazetteerEntry | null` — id → entry, null when unknown (the honest-null law, `bot/src/kv/sessions.ts` `getClaimByToken` precedent)
|
|
- `getIndex(): Map<string, GazetteerEntry[]>` — the memoized alias → entries map
|
|
|
|
The index maps each entry's NORMALIZED name AND each NORMALIZED alias → the entry (D-09 symmetry: both sides normalized). Normalization happens ONCE at index-build time, never per query.
|
|
|
|
**LAW 3 — no logging and no storage anywhere in the module** (D-15: the query text is user payload). The module contains NO `console.log` at all, and no storage API call of any kind.
|
|
</gazetteer_loader_contract>
|
|
|
|
<index_contract>
|
|
`search/src/index.ts` — the public API. Named exports ONLY (no default export). The package has NO `exports`/`main` field (the `@araharah/theme` precedent), so consumers MUST import the EXPLICIT SUBPATH WITH EXTENSION — `import { createGazetteerSource } from "@araharah/search/src/index.ts";` (both tsconfigs carry `allowImportingTsExtensions` + bundler resolution; esbuild and Vite compile the `.ts` source; Node 24 type-stripping handles the corpus runner's RELATIVE form `../src/index.ts`). A bare `@araharah/search` specifier resolves to NOTHING (there is no `index.js` for default resolution) — never use it:
|
|
|
|
```typescript
|
|
export type { GazetteerEntry, SearchQuery, SearchHit, SearchSource } from "./types.ts";
|
|
export { normalize } from "./normalize.ts";
|
|
export { TIER, FUZZY_ADMIT_SIMILARITY } from "./score.ts";
|
|
export { damerau } from "./distance.ts";
|
|
export { getEntries, getEntryById, getIndex } from "./gazetteer.ts";
|
|
export { createGazetteerSource, search } from "./source.ts";
|
|
```
|
|
|
|
`search(text, limit?)` is the CONVENIENCE wrapper over `createGazetteerSource()` — a synchronous-search wrapper the corpus runner can call without await (the `SearchSource.search` method stays `async` per RESEARCH Open Question 4, because backend #3 (Photon-on-VPS) will need it).
|
|
|
|
`createGazetteerSource(): SearchSource` returns the D-03 in-Worker implementation: `name: "gazetteer"`, an async `search(q)` that returns hits ranked by tier then score, and a synchronous `byId(id)`.
|
|
|
|
**The refusal contract is load-bearing** (RESEARCH §Pattern 1): an EMPTY array is a legitimate, load-bearing return value. D-13's negatives ("Kuching", "Penang") MUST produce `[]`, never a nearest-match pin. The admission rule is TIERED, not a single floor: the EXACT / ALIAS_EXACT / PREFIX / TOKEN_SET tiers are always admitted; a FUZZY hit is admitted only when its similarity reaches `FUZZY_ADMIT_SIMILARITY` (0.72). The arithmetic: "KCLC" vs "KLCC" is sim 0.75 → admitted (score 0.45); "Sunwya Pyarmid" vs "Sunway Pyramid" is sim 0.857 → admitted (score 0.514); "Penang" vs "Padang" is sim 0.667 → REFUSED. This is what makes the negatives honest WITHOUT starving the corpus's typo bucket — a flat `score >= 0.60` floor would refuse every typo positive and make D-14 unreachable. A query whose best candidate does not clear its tier's admission returns `[]`. Asserted by the corpus runner in plan 04-02 and documented in the module header.
|
|
|
|
Ranking: sort by tier band DESC, then by score DESC, then by entry id ASC (a deterministic tie-break — the corpus results must be reproducible).
|
|
</index_contract>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>task 1: shared workspace scaffold + types + the JUPEM abbreviation table as DATA</name>
|
|
<files>package.json, search/package.json, search/tsconfig.json, search/src/types.ts, search/src/rules/abbreviations.ts, search/src/normalize.ts, search/test/normalizer.test.mjs</files>
|
|
<read_first>
|
|
- `package.json` (root — the `workspaces` array is `["theme","bot","web"]`; add `"search"`; keep every existing script)
|
|
- `theme/package.json` (the manifest shape to copy — `name`, `type: "module"`, no `exports`/`main`, so consumers import sub-paths directly)
|
|
- `web/src/location/types.ts` (the type-module style: type-only module, no runtime values, no imports)
|
|
- `bot/src/i18n/strings.ts` (the data-table-first module style — the `rules/abbreviations.ts` shape: an editable data structure with a header contract naming how to extend it)
|
|
- `bot/src/security/initdata.ts` (the header-comment contract every new module copies)
|
|
- `bot/test/initdata.test.mjs` (the lazy-module-load + T-numbered-vector suite style; the normalizer suite copies this shape)
|
|
</read_first>
|
|
<action>
|
|
**Root `package.json`.** Add `"search"` to `workspaces` (so it becomes `["theme","bot","web","search"]`) and add these delegate scripts, matching the existing `bot:test` / `web:test` naming:
|
|
|
|
```json
|
|
"search:test": "npm test -w search",
|
|
"search:typecheck": "npm run typecheck -w search",
|
|
```
|
|
|
|
Do NOT change `build`, `deploy`, or `dev` — `search/` is buildless (pure `.ts` + `.json` sources imported directly by consumers), exactly like the `@araharah/theme/basemap/liberty-rams.json` precedent, so the existing deploy chain needs no change.
|
|
|
|
**`search/package.json`** — copy `theme/package.json`'s manifest shape:
|
|
|
|
```json
|
|
{
|
|
"name": "@araharah/search",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"test": "node test/runner.mjs",
|
|
"typecheck": "tsc --noEmit"
|
|
},
|
|
"devDependencies": {
|
|
"typescript": "^5.9.3"
|
|
}
|
|
}
|
|
```
|
|
|
|
NO `exports`/`main` field — consumers import sub-paths directly (the `@araharah/theme` precedent). NO runtime dependencies (D-07: zero package weight in the Worker bundle).
|
|
|
|
**`search/tsconfig.json`** — copy `web/tsconfig.json`'s compilerOptions (it has `resolveJsonModule: true`, which this workspace needs for the JSON import) but with `"types": ["node"]` replaced by nothing (no DOM, no Workers types — this module is platform-neutral) and `"include": ["src", "test"]`:
|
|
|
|
```json
|
|
{
|
|
"compilerOptions": {
|
|
"target": "ES2023",
|
|
"lib": ["ES2023"],
|
|
"module": "ESNext",
|
|
"moduleResolution": "bundler",
|
|
"strict": true,
|
|
"noUncheckedIndexedAccess": true,
|
|
"verbatimModuleSyntax": true,
|
|
"allowImportingTsExtensions": true,
|
|
"noEmit": true,
|
|
"skipLibCheck": true,
|
|
"forceConsistentCasingInFileNames": true,
|
|
"resolveJsonModule": true
|
|
},
|
|
"include": ["src", "test"]
|
|
}
|
|
```
|
|
|
|
**`search/test/runner.mjs` + `search/test/harness.mjs`** — copy `bot/test/runner.mjs` and `bot/test/harness.mjs` VERBATIM (both are 47 and 10 lines; the runner auto-discovers `*.test.mjs` and drains the registry). This is the repo-native zero-dependency harness — no Jest, no Vitest, no Mocha (RESEARCH §Validation Architecture: "No Jest/Vitest/Mocha"). The runner must resolve the suite directory from `import.meta.dirname` exactly as `bot/test/runner.mjs` does.
|
|
|
|
**`search/src/types.ts`** — the contract from the `<interfaces>` block, implemented EXACTLY (the names and shapes are load-bearing for plans 04-03 and 04-04). Add the header comment:
|
|
|
|
```typescript
|
|
/*
|
|
* types.ts — the search contracts (D-03, D-07, D-08, D-11).
|
|
* --------------------------------------------
|
|
* GazetteerEntry / SearchQuery / SearchHit / SearchSource are the seam
|
|
* every backend implements: the in-Worker gazetteer (backend #1, this
|
|
* phase), the Malaysia-only WASM geocoder (backend #2, D-05, post-v1),
|
|
* and Photon-on-VPS (backend #3, D-05, Phase 5). Backend #2/#3 do NOT
|
|
* ship in this phase — the interface exists so they can be added without
|
|
* a rewrite.
|
|
*
|
|
* SearchSource.search() returns an EMPTY array as a legitimate,
|
|
* load-bearing value: "no match in the KL gazetteer" (D-13's
|
|
* out-of-area negatives must be refused honestly, never answered with
|
|
* the nearest KL pin).
|
|
*
|
|
* byId() is separate and SYNCHRONOUS: the alternates callback handler
|
|
* and the webview's deep-link reader both need an id -> entry lookup and
|
|
* neither should await a network.
|
|
*
|
|
* Type-only module: no runtime values, no imports, no storage (GEO-03).
|
|
*/
|
|
```
|
|
|
|
**`search/src/rules/abbreviations.ts`** — the vendored table as DATA, exactly per the `<abbreviation_table_contract>` block: a `ReadonlyArray<readonly [string, string]>` named `ABBREVIATION_ROWS` with every row listed there (48 minimum rows, including `Kpg → Kampung` — required by the `"Kpg Ampang" → "kampung ampang"` normalizer vector) plus the explicit `Presint → Prsn` row, and an exported `buildAbbreviationMap(): Map<string, string[]>` that inverts the table once (abbreviation → full forms; full form → itself) — the canonical form is ALWAYS the long form. The header comment MUST cite the source:
|
|
|
|
```typescript
|
|
/*
|
|
* abbreviations.ts — the Malay abbreviation table, as DATA (D-09).
|
|
* --------------------------------------------------------------
|
|
* PRIMARY SOURCE: JUPEM Toponymic Guidelines — "Abbreviations of
|
|
* Geographical Terms and Other Words Used in Maps"
|
|
* (Malaysia, 2017 ed.) — the normative national list.
|
|
* TRANSCRIPTION: https://github.com/osm-search/Nominatim/pull/4056
|
|
* (settings/icu-rules/variants-ms.yaml, merged 2026-04-02)
|
|
* SECONDARY: https://wiki.openstreetmap.org/wiki/Malaysia/Abbreviations
|
|
*
|
|
* D-09's law: this table extends by FILE EDIT, never by code change.
|
|
* Add a row, add a vector to test/normalizer.test.mjs, done.
|
|
*
|
|
* "Presint → Prsn" is an explicit vowel-drop expansion (Putrajaya
|
|
* precincts are written both ways in live addresses: "JALAN P8G
|
|
* PRESINT 8", "PRSN RESIDEN DESA PARKCI"). "Prsn" is NOT a prefix
|
|
* abbreviation — it drops an interior "e". Do NOT generalize this into
|
|
* a generic vowel-drop rule (it over-expands legitimate names).
|
|
*
|
|
* City codes (PJ, JB, KB, KK) are deliberately ABSENT: they are place
|
|
* identities, not address-word forms, so they live in the gazetteer's
|
|
* alias table (D-08), not here.
|
|
*/
|
|
```
|
|
|
|
**`search/test/normalizer.test.mjs`** — RED-first. The suite imports the normalizer LAZILY (the `bot/test/initdata.test.mjs` lines 42-51 pattern) so a missing module reports per-case failures rather than a lost suite:
|
|
|
|
```javascript
|
|
let normalizePromise = null;
|
|
function loadNormalizer() {
|
|
if (!normalizePromise) {
|
|
normalizePromise = import("../src/normalize.ts").then((m) => m.normalize);
|
|
}
|
|
return normalizePromise;
|
|
}
|
|
```
|
|
|
|
Cases (T-numbered, one per law):
|
|
- `T1 rule vectors` — every vector in the `<normalizer_contract>` block, asserted one by one with a named message.
|
|
- `T2 symmetry law` — `normalize("Jln Ampang") === normalize("Jalan Ampang")` and `normalize("Tmn Bangsar") === normalize("Taman Bangsar")`.
|
|
- `T3 whole-token-only expansion` — `"Pdang"` does NOT contain "jalan"/"persiaran" (it stays "pdang"-like); `"pdg"` DOES expand to "padang".
|
|
- `T4 slash preservation` — `normalize("Jln 3/1") === "jalan 3/1"`.
|
|
- `T5 glued-word split` — `normalize("tamanbangsar") === "taman bangsar"`; `normalize("jalankanpar")` contains "jalan" as a separate token.
|
|
- `T6 vowel-drop` — `normalize("Prsn 15") === "presint 15"` and `normalize("Presint 15") === "presint 15"`.
|
|
- `T7 idempotence` — for every vector `v`: `normalize(normalize(v)) === normalize(v)`.
|
|
- `T8 case/whitespace folding` — `normalize(" SUNWAY PYRAMID ") === "sunway pyramid"`.
|
|
- `T9 the table is data` — read `search/src/rules/abbreviations.ts` as TEXT (comments stripped) and assert that the ONLY `function` in the file is `buildAbbreviationMap` — no other function, and no control-flow keywords (`if`, `for`, `=>`) outside it — so the module is data plus one generic builder, never algorithm code; and assert the row count is >= 47.
|
|
|
|
**`search/src/normalize.ts`** — implement EXACTLY per the `<normalizer_contract>` pipeline. The exported signature is `export function normalize(input: string): string`. Import the table via `import { buildAbbreviationMap } from "./rules/abbreviations.ts";` and build the map ONCE at module scope (the map is ~60 entries of plain data — cheap; the D-09 "lazy" law applies to the 500-entry gazetteer INDEX, not to this table).
|
|
|
|
Write `search/src/normalize.ts` NOW as part of this task so the suite in this task can go GREEN — the RED phase is the suite itself.
|
|
|
|
**Acceptance:** `npm run search:test` exits 0 and the output contains `PASS T1` … `PASS T9`; `npm run search:typecheck` exits 0.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/C-Fu/araharah && npm install --package-lock-only --workspaces 2>&1 | tail -3 ; npm run search:test 2>&1 | tail -20 ; npm run search:typecheck 2>&1 | tail -5 ; grep -c '"search"' package.json</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `grep -q '"search"' package.json` exits 0 AND `node -e "const w=require('./package.json').workspaces; if(!w.includes('search')) throw new Error('search workspace missing'); console.log('OK')"` exits 0
|
|
- `grep -q '"search:test": "npm test -w search"' package.json` exits 0
|
|
- `test -f search/package.json && test -f search/tsconfig.json && test -f search/test/runner.mjs && test -f search/test/harness.mjs` exits 0
|
|
- `node -e "const p=require('./search/package.json'); if(p.name!=='@araharah/search') throw new Error(1); if(p.dependencies) throw new Error('search must have zero runtime dependencies'); console.log('OK')"` exits 0 (D-07: zero Worker bundle weight)
|
|
- `npm run search:test` exits 0 and the runner output contains `PASS T1` through `PASS T9`
|
|
- `npm run search:typecheck` exits 0
|
|
- `grep -q 'export interface SearchSource' search/src/types.ts` exits 0 AND `grep -q 'byId(id: string)' search/src/types.ts` exits 0 (the synchronous id lookup is part of the contract — D-03 seam)
|
|
- `grep -q 'ABBREVIATION_ROWS' search/src/rules/abbreviations.ts` exits 0 (the table is DATA)
|
|
- `grep -c 'JUPEM' search/src/rules/abbreviations.ts | head -1` returns >= 1 (the vendored table cites its normative source)
|
|
- `grep -q 'Presint' search/src/rules/abbreviations.ts` exits 0 (the D-09 vowel-drop rule is present)
|
|
- `node --experimental-strip-types -e "1" 2>/dev/null; node -e "import('./search/src/normalize.ts').then(m=>{ if(m.normalize('Jln Ampang')!=='jalan ampang') throw new Error('expansion broken'); if(m.normalize('Jalan Ampang')!=='jalan ampang') throw new Error('symmetry broken'); if(m.normalize('tamanbangsar')!=='taman bangsar') throw new Error('glued split broken'); if(m.normalize('Prsn 15')!=='presint 15') throw new Error('vowel-drop broken'); console.log('normalize OK')})"` exits 0
|
|
</acceptance_criteria>
|
|
<done>The `search/` workspace exists as a buildless fourth npm workspace with the type contract, the JUPEM-derived abbreviation table as editable DATA, and a symmetric normalizer proven by 9 named test cases including the symmetry, whole-token-only, glued-split, vowel-drop, and idempotence laws.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>task 2: Damerau distance + disjoint-tier scorer (the residual net)</name>
|
|
<files>search/src/distance.ts, search/src/score.ts, search/test/distance.test.mjs, search/test/score.test.mjs</files>
|
|
<read_first>
|
|
- `search/src/normalize.ts` (task 1's output — the scorer consumes NORMALIZED strings; do not re-normalize inside the scorer)
|
|
- `search/src/rules/abbreviations.ts` (task 1's output — the rules table is the tier-1 lever, the fuzzy tier is only the residual)
|
|
- `bot/test/initdata.test.mjs` (the RED→GREEN vector-suite style the two new suites copy)
|
|
- `.planning/phases/04-destination-search-kl-gazetteer/04-RESEARCH.md` (the §"Common Pitfalls" #4 and #5 entries — the tier-disjointness and min-length-guard laws)
|
|
</read_first>
|
|
<action>
|
|
**`search/src/distance.ts`** — hand-rolled OSA Damerau-Levenshtein, per the `<distance_contract>` block. ZERO imports, ZERO dependencies. Header comment:
|
|
|
|
```typescript
|
|
/*
|
|
* distance.ts — Damerau-Levenshtein (OSA) distance, hand-rolled.
|
|
* ----------------------------------------------------------
|
|
* The ONE place hand-rolling IS correct (RESEARCH §"Don't Hand-Roll"):
|
|
* at ~500 entries x ~5 aliases the comparison count is tiny, the
|
|
* algorithm is ~25 lines of standard OSA dynamic programming, and a
|
|
* dependency would add package weight to a Worker bundle D-07
|
|
* deliberately wants minimal. Every dedicated string-distance npm
|
|
* package is 2022-vintage and unmaintained (verified 2026-10-23).
|
|
*
|
|
* Adjacent transpositions count as ONE edit — "KCLC" must reach
|
|
* "KLCC" (D-13's transposition typo bucket), and a Levenshtein-only
|
|
* distance would score it 2 and fall out of the FUZZY tier.
|
|
*
|
|
* Pure: no I/O, no imports, no third-party packages.
|
|
*/
|
|
```
|
|
|
|
**`search/src/score.ts`** — the disjoint-tier scorer, per the `<scorer_contract>` block. Exports:
|
|
- `export const TIER = { EXACT: 1.00, ALIAS_EXACT: 0.90, PREFIX: 0.75, TOKEN_SET: 0.65, FUZZY: 0.60 } as const;` — `FUZZY` is the band's CEILING (0.60), NOT a zero multiplier: a fuzzy score is `fuzzySim * TIER.FUZZY`, so the whole tier lands strictly inside `(0, 0.60)` (a similarity of 1.0 would have been caught by the EXACT tier first). Multiplying the similarity by `0.00` would zero the entire residual tier — see the `<scorer_contract>` block's NOTE.
|
|
- `export const FUZZY_ADMIT_SIMILARITY = 0.72;` — the fuzzy admission floor, defined HERE (the `<scorer_contract>` block places it in score.ts) and imported by `source.ts` (task 3) and re-exported through `index.ts` (the `<index_contract>` block) so the corpus runner (plan 04-02's CO8) reads the floor from the module instead of hardcoding it.
|
|
- `export function scoreOne(queryNorm: string, aliasNorm: string): number`
|
|
- `export function scoreCandidate(queryNorm: string, aliasNorms: readonly string[]): { score: number; matchedAlias: string } | null` — returns the BEST tier hit and WHICH alias matched (the corpus report needs that provenance), or `null` when nothing scores.
|
|
|
|
Implementation notes:
|
|
- The min-length guard fires BEFORE the prefix/fuzzy tiers: if `queryNorm.length < 3 || aliasNorm.length < 3` → return `0` (RESEARCH Pitfall 5).
|
|
- The EXACT tier fires on `queryNorm === aliasNorm`.
|
|
- The ALIAS_EXACT tier exists for the case where the query equals an entry's normalized ALIAS but the comparison is being made against the entry NAME (the name-match and alias-match are distinct comparisons). `scoreCandidate` handles this by scoring against the name and each alias, and reporting the ALIAS_EXACT band when the alias match is exact but the name differs.
|
|
- The TOKEN_SET tier: every whitespace-separated token of the query appears in the candidate's token set.
|
|
- The FUZZY tier: `fuzzySim * TIER.FUZZY` where `fuzzySim = max(wholeStringSim, tokenLevelSim)` and each `sim = 1 - damerau(a, b) / Math.max(a.length, b.length)` (see the `<scorer_contract>` block for the exact two-level shape).
|
|
- `scoreCandidate` MUST return the alias that produced the score (the raw, UN-normalized alias string — the corpus report wants the human-readable alias, not the normalized form).
|
|
|
|
**`search/test/distance.test.mjs`** — RED-first. One `test(...)` per vector from the `<distance_contract>` block, with the hand-computed expected value in the message. Load the module lazily (task 1's pattern).
|
|
|
|
**`search/test/score.test.mjs`** — RED-first. Cases:
|
|
- `S1 tier disjointness` — for each pair of adjacent tiers, `TIER[a] > TIER[b]` (including `TIER.TOKEN_SET` 0.65 > `TIER.FUZZY` 0.60); and `TIER.FUZZY === 0.60` (the band's CEILING) with every admitted FUZZY score strictly inside `(0, 0.60)` — `> 0` for a near miss, `< 0.60` because a similarity of 1.0 is caught by the EXACT tier first.
|
|
- `S2 min-length guard` — `scoreOne("ab", "abcd") === 0` and `scoreOne("kl", "kl") === 0` (both shorter than 3 → 0, EVEN for an exact match — the guard fires first).
|
|
- `S3 exact outranks fuzzy` — `scoreOne("klcc", "klcc") > scoreOne("klcc", "klca")` and `scoreOne("klcc", "klcc") > TIER.FUZZY` (an exact match always beats any fuzzy hit).
|
|
- `S4 prefix tier` — `scoreOne("jalan amp", "jalan ampang") === TIER.PREFIX`.
|
|
- `S5 token-set tier` — `scoreOne("taman bangsar", "taman bangsar kib") === TIER.TOKEN_SET` (every query token present) and `scoreOne("taman bangsar", "taman bukit") === 0` (not a subset — must NOT be TOKEN_SET).
|
|
- `S6 fuzzy residual + admission boundary` — `scoreOne("kclc", "klcc")` is in `(0, 0.60)` and equals `0.75 * 0.60`; and the admission boundary: a similarity of 0.72 admits, 0.667 does not (a "penang"-vs-"padang"-shaped near-miss stays below admission — assert via the two similarities' relationship to `FUZZY_ADMIT_SIMILARITY`).
|
|
- `S7 matchedAlias provenance` — `scoreCandidate("klcc", ["Suria KLCC", "KLCC"])` returns `{ score: TIER.EXACT or ALIAS_EXACT, matchedAlias: "KLCC" }` — the matched alias is the RAW alias string.
|
|
|
|
**Acceptance:** `npm run search:test` exits 0 and the output contains `PASS S1` … `PASS S7` and `PASS T1` (distance) — the runner names every case.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/C-Fu/araharah && npm run search:test 2>&1 | tail -30 ; npm run search:typecheck 2>&1 | tail -5 ; grep -c 'export const TIER' search/src/score.ts</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `npm run search:test` exits 0 and the output contains `PASS S1` through `PASS S7`
|
|
- `npm run search:typecheck` exits 0
|
|
- `grep -q 'export const TIER' search/src/score.ts` exits 0 AND `grep -q 'EXACT: *1.00' search/src/score.ts` exits 0 AND `grep -q 'ALIAS_EXACT: *0.90' search/src/score.ts` exits 0 AND `grep -q 'PREFIX: *0.75' search/src/score.ts` exits 0 AND `grep -q 'TOKEN_SET: *0.65' search/src/score.ts` exits 0 AND `grep -q 'FUZZY: *0.60' search/src/score.ts` exits 0 (the disjoint bands are exact literals — FUZZY is the 0.60 ceiling, never a 0.00 multiplier) AND `grep -q 'FUZZY_ADMIT_SIMILARITY = 0.72' search/src/score.ts` exits 0 (the admission floor is an exported constant, not an inlined literal)
|
|
- `grep -q 'export function damerau' search/src/distance.ts` exits 0
|
|
- `node -e "import('./search/src/distance.ts').then(m=>{const d=m.damerau; const cases=[['ab','ba',1],['a cat','an act',2],['','',0],['','abc',3],['abc','',3],['same','same',0],['klcc','kclc',1],['sunway','sunwya',1],['kitten','sitting',3]]; for(const [a,b,e] of cases){const g=d(a,b); if(g!==e) throw new Error(a+'/'+b+' => '+g+' expected '+e);} console.log('distance OK')})"` exits 0
|
|
- `node -e "import('./search/src/score.ts').then(m=>{if(m.scoreOne('ab','abcd')!==0) throw new Error('min-length guard'); if(!(m.scoreOne('klcc','klcc')>m.scoreOne('klcc','klca'))) throw new Error('exact must outrank fuzzy'); if(m.scoreOne('taman bangsar','taman bukit')===m.TIER.TOKEN_SET) throw new Error('token-set false positive'); console.log('score OK')})"` exits 0
|
|
- `grep -v '^\s*//' search/src/distance.ts | grep -c "require(\|from \"" | head -1` returns 0 (the distance module has ZERO imports — D-07)
|
|
</acceptance_criteria>
|
|
<done>The residual net is in place: a hand-rolled OSA distance proven against 9 hand-computed vectors, and a scorer whose 5 tiers occupy strictly disjoint descending bands so an exact match can never be outranked by a fuzzy hit.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>task 3: curated gazetteer JSON + lazy loader + three-environment proof + GPL/ODbL licensing</name>
|
|
<files>search/src/data/kl-gazetteer.json, search/src/gazetteer.ts, search/src/source.ts, search/src/index.ts, search/test/gazetteer.test.mjs, search/test/loader-env.test.mjs, LICENSE, LICENSES.md, README.md</files>
|
|
<read_first>
|
|
- `search/src/types.ts` (task 1's output — the exact contract the loader and source implement)
|
|
- `search/src/normalize.ts` (task 1's output — the loader normalizes every alias at index-build time)
|
|
- `search/src/score.ts` (task 2's output — the source ranks with it)
|
|
- `web/src/map/map.ts` (lines 19-20 — the build-time JSON-import precedent for a workspace subpath)
|
|
- `bot/test/initdata.test.mjs` (lines 42-51 — the lazy module-scope memoization pattern the loader copies)
|
|
- `bot/src/kv/sessions.ts` (lines 99-103 — the honest-null contract: `getClaimByToken` returns null on unknown, never a fabricated value)
|
|
- `README.md` (the §License section, lines ~221-225, which currently says "Private repository. No license file is present" — it must be rewritten in THIS task)
|
|
- `web/src/i18n/strings.ts` (line 73 — the `attribution: "© OpenStreetMap contributors"` string precedent)
|
|
</read_first>
|
|
<action>
|
|
**`search/src/data/kl-gazetteer.json`** — the curated gazetteer, EXACTLY per the `<gazetteer_seed_contract>` block: the `_license` / `_attribution` / `_notice` header object, `"version": 1`, and an `"entries"` array of at least 30 hand-curated KL-metro landmarks (the 10 required + at least 20 more from the suggested list). Every entry carries `id` (slug, `/^[a-z0-9-]{1,64}$/`), `name` (the canonical OSM landmark name), `aliases` (non-empty, includes the canonical name and the local shorthand), `lat`, `lng`, `category`, `address`.
|
|
|
|
The JSON must be PRETTY-PRINTED with 2-space indent (diff-friendly PRs — D-07's stated rationale: "diff-friendly PRs").
|
|
|
|
**`search/src/gazetteer.ts`** — the loader, per the `<gazetteer_loader_contract>` block's three laws. Header comment must state the three laws verbatim (lazy, static JSON import, no logging/storage). Exports `getEntries()`, `getEntryById(id)`, `getIndex()`.
|
|
|
|
The JSON import line (the mechanism under test in task 4's loader-env suite):
|
|
|
|
```typescript
|
|
import raw from "./data/kl-gazetteer.json" with { type: "json" };
|
|
```
|
|
|
|
If `npm run search:typecheck` or the loader-env suite rejects the import attribute, drop the attribute (esbuild and Vite inline JSON without it) — the loader-env suite is the arbiter, and the FINAL committed form must pass all three environments.
|
|
|
|
The index build: for each entry, normalize the entry's `name` and each of its `aliases`, and map each normalized string → the entry. The map is built ONCE, lazily, memoized in module scope (`let index: Map<string, GazetteerEntry[]> | null = null;`).
|
|
|
|
**`search/src/source.ts`** — the D-03 `SearchSource` implementation. Exports `createGazetteerSource(): SearchSource` and a synchronous `search(text: string, limit?: number): SearchHit[]` convenience wrapper. The module's HEADER COMMENT states the tiered admission rule with the numbers, verbatim-in-spirit from the `<index_contract>` block: non-fuzzy tiers always admitted; a FUZZY hit only at similarity >= `FUZZY_ADMIT_SIMILARITY` (0.72) — "KCLC" vs "KLCC" is sim 0.75 → admitted (score 0.45), "Penang" vs "Padang" is sim 0.667 → refused (an empty array, never a fabricated pin). Behaviour:
|
|
- Normalize the query text.
|
|
- For every candidate (each entry's normalized name + each alias), score with `scoreCandidate`.
|
|
- Keep only ADMITTED hits: EXACT / ALIAS_EXACT / PREFIX / TOKEN_SET always; a FUZZY hit only when its similarity >= `FUZZY_ADMIT_SIMILARITY` (0.72). Below admission, the query is "not in the v1 area" and the result is an EMPTY array, never a nearest-match pin (D-13/D-14).
|
|
- Sort by tier band DESC, then score DESC, then entry id ASC (deterministic).
|
|
- Return at most `limit ?? 5` hits.
|
|
- `byId(id)` validates `id` against `/^[a-z0-9-]{1,64}$/` and returns the entry or `null` (honest null).
|
|
|
|
**`search/src/index.ts`** — the public API, EXACTLY per the `<index_contract>` block. Named exports only.
|
|
|
|
**`search/test/gazetteer.test.mjs`** — RED-first. Cases:
|
|
- `G1 entry schema` — every entry has a non-empty `id`, `name`, a NON-EMPTY `aliases` array, finite numeric `lat`/`lng`, a `category` from the closed set, and a non-empty `address`.
|
|
- `G2 id uniqueness + charset` — no duplicate ids; every id matches `/^[a-z0-9-]{1,64}$/`.
|
|
- `G3 the landmark names are canonical, the shorthand is an ALIAS` — assert `suria-klcc`'s `name` is "Suria KLCC" AND that "KLCC" appears in its `aliases` (RESEARCH §Verification A3: "KLCC" has no usable OSM landmark name; the alias list IS the product).
|
|
- `G4 the alias table carries the local shorthand` — assert the entries for `the-duke` include "DUKE" as an alias and that no entry's `name` equals "DUKE" (it is a highway acronym, not a landmark name).
|
|
- `G5 coordinates are in the KL metro` — every `lat` in `[2.9, 3.4]` and every `lng` in `[101.4, 102.0]`.
|
|
- `G6 ODbL header` — the JSON carries `_license` containing "ODbL", `_attribution` containing "OpenStreetMap", and `_notice` containing "opendatacommons.org/licenses/odbl".
|
|
- `G7 the LICENSE file exists and is GPL-3.0` — `LICENSE` exists at the repo root and its text contains "GNU GENERAL PUBLIC LICENSE" and "Version 3".
|
|
- `G8 the data licence is documented` — `LICENSES.md` exists and names BOTH "GPL-3.0" and "ODbL" and "OpenStreetMap".
|
|
- `G9 the README licence section is no longer self-contradictory` — `README.md` contains "GPL-3.0" and does NOT contain "all rights reserved".
|
|
- `G10 the module has no logging and no storage` — read `search/src/*.ts` as text (comments stripped) and assert NO `console.log`, NO `localStorage`, NO `sessionStorage`, NO `indexedDB`, NO `document.cookie` (D-15).
|
|
- `G11 the loader is lazy` — read `search/src/gazetteer.ts` (comments stripped) and assert the index-building code is inside a function, NOT at module top level (RESEARCH Pitfall 8: Workers "avoid expensive work in global scope"). Practical assertion: the module's top-level statements (before the first `function`/`export function` declaration) contain no `new Map(` and no JSON traversal.
|
|
|
|
**`search/test/loader-env.test.mjs`** — THE PHASE'S TOP MECHANICAL RISK, proven mechanically. Three environments, three assertions:
|
|
- `L1 bare Node (type-stripping)` — `await import("../src/index.ts")` works and `getEntries().length >= 30`; the runner file itself IS this environment (Node 24 strips types by default).
|
|
- `L2 the Vite bundle inlines the gazetteer` — the web bundle is built by `npm run build -w theme && npm run build -w web` and the emitted `web/dist/assets/*.js` contains a gazetteer id marker (e.g. `"suria-klcc"`) — proving the JSON was inlined at BUILD time, not fetched at runtime. (The web workspace imports `@araharah/search` starting in plan 04-04; until then this case asserts the SEARCH workspace's own module can be bundled — assert it against the search workspace's own esbuild-free check: `node -e "import('./search/src/index.ts')"` PLUS a static-text assertion that `search/src/gazetteer.ts` contains a STATIC import of the JSON, not a `fetch(` or `readFileSync`. If plan 04-04 has landed by the time this suite runs, ALSO assert the web bundle marker; otherwise skip that half with an explicit `test.skip`-style guard that still asserts the static-import half. Make the skip visible in the runner output.)
|
|
- `L3 the esbuild/Worker bundler inlines the gazetteer` — TWO halves. **The inline half (provable at wave 1):** bundle the search module directly with esbuild — the same bundler family wrangler uses (esbuild 0.28.2 is hoisted at `node_modules/.bin/esbuild`): `node_modules/.bin/esbuild search/src/index.ts --bundle --outfile=/tmp/opencode/araharah-search-bundle/index.js`, then grep the emitted JS for the `suria-klcc` marker — a static JSON import MUST be inlined at build time, never fetched at runtime. **The Worker dry-run half (build regression guard ONLY):** `cd bot && npx wrangler deploy --dry-run --outdir=<tmp>` (wrangler 4.151.3 supports `--dry-run`; `wrangler dev` does NOT) must EXIT 0. Do NOT grep the dry-run bundle for the marker in this plan: at wave 1 no bot source imports `@araharah/search` (`bot.ts` gains `createGazetteerSource()` only in plan 04-03 task 2), so the Worker bundle cannot contain the marker yet and a marker grep here would red-light the whole suite. The Worker-bundle marker grep is plan 04-03 task 2's acceptance criterion. If plan 04-03 has already landed when this suite runs, ALSO assert the Worker-bundle marker — guarded by the same explicit skip-guard style L2 uses for the web bundle, with the skip made visible in the runner output.
|
|
|
|
**`LICENSE`** — the VERBATIM GPL-3.0 text from https://www.gnu.org/licenses/gpl-3.0.txt. Do NOT paraphrase, do NOT summarize, do NOT truncate the "how to apply" section. The file must contain the strings "GNU GENERAL PUBLIC LICENSE", "Version 3, 29 June 2007", and "TERMS AND CONDITIONS".
|
|
|
|
**`LICENSES.md`** — the dual-license note:
|
|
|
|
```markdown
|
|
# Licences
|
|
|
|
## Code — GPL-3.0
|
|
|
|
All araharah source code (the `bot/`, `web/`, `theme/`, and `search/` workspaces) is licensed under
|
|
the GNU General Public License v3.0. The full text is in [LICENSE](./LICENSE).
|
|
|
|
GPL-3.0 is a deliberate project law (D-04): a fork must be able to read, change, and redistribute the
|
|
whole system — including the parts that make it a working navigation product.
|
|
|
|
## Gazetteer data — ODbL-1.0
|
|
|
|
`search/src/data/kl-gazetteer.json` (and any index derived from it) is curated from OpenStreetMap
|
|
data and is made available under the Open Database License (ODbL) v1.0:
|
|
<https://opendatacommons.org/licenses/odbl/1-0/>
|
|
|
|
Attribution: © OpenStreetMap contributors.
|
|
|
|
The ODbL's share-alike obligation (ODbL §4.4) applies to a Derivative Database: the curated
|
|
gazetteer and any restructured form of it carry the same ODbL notice. The normalizer's index is a
|
|
derived, restructured form of the gazetteer and therefore carries the same notice.
|
|
|
|
Code and data are separately licensed — ODbL §2.3(a) excludes computer programs from the ODbL.
|
|
```
|
|
|
|
**`README.md` §License** — replace the current "Private repository. No license file is present; all
|
|
rights reserved by the project owner." section with a short statement pointing at `LICENSE` (GPL-3.0)
|
|
and `LICENSES.md` (the ODbL notice for the gazetteer). Also update the "Repository layout" code block
|
|
to include the `search/` workspace line:
|
|
|
|
```
|
|
search/ @araharah/search — src/types.ts, src/normalize.ts, src/score.ts, src/gazetteer.ts, src/data/kl-gazetteer.json
|
|
```
|
|
|
|
**Acceptance:** `npm run search:test` exits 0 (including `PASS G1`…`PASS G11` and `PASS L1`…`PASS L3`); `npm run search:typecheck` exits 0; the esbuild bundle of the search module contains the `suria-klcc` marker; `npx wrangler deploy --dry-run` in `bot/` exits 0 (the Worker bundle builds — its bundle-marker grep is plan 04-03 task 2's criterion, after `bot.ts` imports the source).
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/C-Fu/araharah && npm run search:test 2>&1 | tail -40 ; npm run search:typecheck 2>&1 | tail -5 ; node_modules/.bin/esbuild search/src/index.ts --bundle --outfile=/tmp/opencode/araharah-search-bundle/index.js 2>&1 | tail -3 ; grep -c 'suria-klcc' /tmp/opencode/araharah-search-bundle/index.js ; cd bot && npx wrangler deploy --dry-run --outdir /tmp/opencode/araharah-worker-dryrun 2>&1 | tail -3</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `npm run search:test` exits 0 and the output contains `PASS G1` through `PASS G11` and `PASS L1` through `PASS L3`
|
|
- `npm run search:typecheck` exits 0
|
|
- `cd bot && npx wrangler deploy --dry-run --outdir /tmp/opencode/araharah-worker-dryrun` exits 0 (the Worker bundle builds — a build regression guard; the bundle-marker grep is plan 04-03 task 2's criterion, where `bot.ts` actually imports the source)
|
|
- `node_modules/.bin/esbuild search/src/index.ts --bundle --outfile=/tmp/opencode/araharah-search-bundle/index.js` exits 0 AND `grep -c 'suria-klcc' /tmp/opencode/araharah-search-bundle/index.js` returns >= 1 (esbuild — the bundler family wrangler uses — INLINES the static JSON import at build time; a runtime fetch would be a hard failure in a Worker. This is the wave-1-provable inline proof, since no bot source imports `@araharah/search` until plan 04-03 task 2)
|
|
- `node -e "const g=require('./search/src/data/kl-gazetteer.json'); if(g.entries.length<30) throw new Error('seed too small: '+g.entries.length); if(!g._license||!g._license.includes('ODbL')) throw new Error('no ODbL header'); if(!g._attribution||!g._attribution.includes('OpenStreetMap')) throw new Error('no OSM attribution'); console.log('gazetteer OK '+g.entries.length+' entries')"` exits 0
|
|
- `node -e "const g=require('./search/src/data/kl-gazetteer.json'); const ids=g.entries.map(e=>e.id); if(new Set(ids).size!==ids.length) throw new Error('duplicate ids'); for(const e of g.entries){ if(!/^[a-z0-9-]{1,64}$/.test(e.id)) throw new Error('bad id '+e.id); if(!e.aliases||e.aliases.length===0) throw new Error('no aliases '+e.id); if(!Number.isFinite(e.lat)||!Number.isFinite(e.lng)) throw new Error('bad coords '+e.id); if(e.lat<2.9||e.lat>3.4||e.lng<101.4||e.lng>102.0) throw new Error('out of KL bbox '+e.id);} console.log('entries OK')"` exits 0
|
|
- `test -f LICENSE && grep -c 'GNU GENERAL PUBLIC LICENSE' LICENSE | head -1` returns >= 1 AND `grep -q 'Version 3, 29 June 2007' LICENSE` exits 0 (the VERBATIM GPL-3.0 text — D-04)
|
|
- `test -f LICENSES.md && grep -q 'ODbL' LICENSES.md && grep -q 'OpenStreetMap' LICENSES.md && grep -q 'GPL-3.0' LICENSES.md` exits 0
|
|
- `grep -q 'GPL-3.0' README.md` exits 0 AND `grep -c 'all rights reserved' README.md | head -1` returns 0 (the README no longer contradicts itself)
|
|
- `grep -v '^\s*\*\|^\s*//\|^/\*\|^\s*$' search/src/*.ts | grep -c 'console\.log\|localStorage\|sessionStorage\|indexedDB' | head -1` returns 0 (D-15: the shared module has no logging and no storage)
|
|
- `grep -q 'import raw from "./data/kl-gazetteer.json"' search/src/gazetteer.ts` exits 0 (the STATIC JSON import — never a runtime fetch, never `readFileSync`)
|
|
- `grep -c 'readFileSync\|fetch(' search/src/gazetteer.ts | head -1` returns 0 (Workers have no fs — the loader must be pure-import)
|
|
- `node -e "import('./search/src/index.ts').then(m=>{const hits=m.search('KLCC',5); if(!hits.length) throw new Error('KLCC found nothing'); if(hits[0].entry.id!=='suria-klcc') throw new Error('KLCC => '+hits[0].entry.id); if(m.search('Kuching',3).length!==0) throw new Error('negative leaked'); if(m.search('Penang',3).length!==0) throw new Error('negative leaked'); console.log('search OK: '+hits[0].entry.id)})"` exits 0 (SRCH-01 lands; the D-13 negatives are refused honestly)
|
|
- `node -e "import('./search/src/index.ts').then(m=>{const h=m.search('Jln Ampang',5); if(!h.some(x=>x.entry.id==='suria-klcc')) throw new Error('Jln Ampang missed KLCC'); const t=m.search('KCLC',5); if(!t.some(x=>x.entry.id==='suria-klcc')) throw new Error('typo missed'); console.log('SRCH-02 OK')})"` exits 0
|
|
</acceptance_criteria>
|
|
<done>The curated KL gazetteer ships as committed, ODbL-attributed JSON with locals' aliases; the loader parses it lazily and memoizes a normalized index that survives all three bundling environments with the JSON INLINED (proven at wave 1 by the esbuild bundle of the search module containing the id marker; the wrangler dry-run is asserted as an exit-0 build guard, with its bundle-marker grep landing in plan 04-03 task 2 once `bot.ts` imports the source); the repo carries the verbatim GPL-3.0 LICENSE and the ODbL data notice; and the README no longer says "all rights reserved".</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<threat_model>
|
|
## Trust Boundaries
|
|
|
|
| Boundary | Description |
|
|
|----------|-------------|
|
|
| Raw user query → normalizer/scorer | The query is untrusted, arbitrary text of arbitrary length; it enters the shared module and must never reach a log or a storage surface. |
|
|
| Committed JSON file → runtime index | The gazetteer is trusted repo content, but a malformed entry (duplicate id, non-finite coordinate) would corrupt every downstream shortlist and every venue card. |
|
|
| Bundle-time JSON import → runtime | The Worker has no `fs`; a runtime fetch of the dataset would fail at request time in a way that looks like "no results" — an honesty failure, not just an outage. |
|
|
| Module scope → Workers startup budget | The Worker has a 1 s startup limit and a 10 ms CPU/request budget; index construction at module scope is an availability threat, not a style issue. |
|
|
|
|
## STRIDE Threat Register
|
|
|
|
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
|
|-----------|----------|-----------|-------------|-----------------|
|
|
| T-04-01 | Information Disclosure | Query text reaching a log line (D-15 / V7) | mitigate | The shared module contains **no logging at all** — `search/src/*.ts` must contain zero `console.log` (G10 asserts it, comments stripped). The module also takes no logger and no storage dependency, so there is no seam through which a query could escape. The bot-side logging law (category-only) is enforced in plan 04-03. |
|
|
| T-04-02 | Information Disclosure | Query text reaching a storage surface | mitigate | The module imports no storage API and calls none (G10 bans `localStorage`/`sessionStorage`/`indexedDB`/`document.cookie` in `search/src/*.ts`). The gazetteer is a build-time constant, not user data. |
|
|
| T-04-03 | Denial of Service | Hostile query text — long string, pathological repetition — forcing an unbounded scan or allocation | mitigate | The query is bounded BEFORE this module is reached (plan 04-03 caps the raw command argument at 128 chars and rejects over-cap input). Inside the module the normalized query is tokenized once and compared against a finite, pre-built alias index; the fuzzy tier runs only on survivors of the token/prefix filters and only for tokens >= 3 characters, so the per-query work is bounded by the index size, not by the query length. The scorer refuses to score any side shorter than 3 characters. |
|
|
| T-04-04 | Denial of Service | Index construction at module scope blowing the Workers 1 s startup budget (RESEARCH Pitfall 8) | mitigate | LAW 1 in the loader: the index is built lazily inside `loadIndex()` and memoized in module scope — never at module top level (G11 asserts the top level contains no `new Map(` and no JSON traversal). The same pattern as `bot/test/initdata.test.mjs`'s lazy validator load. |
|
|
| T-04-05 | Denial of Service | Naive full-scan Damerau over ~2500 alias pairs costing ~8 ms/query against a 10 ms Workers Free CPU limit (RESEARCH §Verification A1 — REPRODUCED at 8.243 ms) | mitigate | The scorer is TIERED and the per-query cost is bounded by CANDIDATE COUNT, not query length: the EXACT / ALIAS_EXACT / PREFIX / TOKEN_SET tiers are comparisons against a pre-built hash index (O(1) per candidate), so a query with any cheap-tier hit never reaches the fuzzy scan. A pure-typo query (zero cheap-tier survivors) DOES pay a full fuzzy scan — that is the honest worst case, REPRODUCED at ~8.2 ms for ~2000 alias pairs — and every fuzzy comparison stays bounded: both sides >= 3 chars, OSA is O(len^2) on short strings, and the scan runs over a FIXED committed dataset. SCALING DECISION (RESEARCH directive #1, stated explicitly rather than hand-waved): the ~30-entry seed and D-07's 500-entry ceiling keep the full-scan worst case inside the 10 ms limit, so the trigram/blocking-key index is documented as the NEXT step, not built now; `docs/SEARCH-QUALITY.md`'s "Known limitations" (plan 04-02) records the trigger (top-up scale approaching the CPU limit) and the fix (a first-letter + length-band blocking key in front of the OSA tier), so the escalation is a recorded decision, not a surprise Error-1102. |
|
|
| T-04-06 | Tampering | A malformed committed entry (duplicate id, non-finite coordinate, empty alias list) reaching a venue card as a fabricated or broken pin | mitigate | G1/G2/G5 assert the dataset's invariants as a gate: unique slug-safe ids, non-empty aliases, finite coordinates inside the KL bbox. A non-finite coordinate would surface as `NaN` on the venue card — the gate makes that impossible to ship. |
|
|
| T-04-07 | Tampering | An out-of-area query answered with the nearest KL pin (a fabricated location) — an honesty failure (RESEARCH §Security V8, D-13/D-14) | mitigate | The tiered admission rule: EXACT/ALIAS_EXACT/PREFIX/TOKEN_SET always admitted; a FUZZY hit only at similarity >= `FUZZY_ADMIT_SIMILARITY` (0.72). "Kuching" and "Penang" must yield `[]` — asserted in this plan's acceptance criteria and re-asserted by the corpus runner in plan 04-02. An empty shortlist is a legitimate, load-bearing return value of the `SearchSource` contract, and the bot renders the honest refusal copy (plan 04-03). |
|
|
| T-04-08 | Tampering | The rule table growing a code-shaped entry point — a rule that changes behavior via code rather than a table edit (D-09's "extends by table edit, not code change" law) | mitigate | T9 asserts the rules file contains no algorithm and the row count is >= 47; the table is a `ReadonlyArray<readonly [string, string]>` of pairs, inverted once by a single generic builder. Any new abbreviation is a row, not a branch. |
|
|
| T-04-09 | Repudiation | The dataset's licence obligation unmet — the gazetteer shipped without the ODbL notice (ODbL §4.4 share-alike; OSMF attribution guidelines) | mitigate | The `_license` / `_attribution` / `_notice` header travels INSIDE the JSON (G6), `LICENSES.md` names both licences and the OSM attribution (G8), and the README points at both (G9). Three surfaces, so no single edit can leave the notice missing. |
|
|
| T-04-10 | Spoofing | A caller substituting a different `SearchSource` implementation to bypass the refusal floor | mitigate | The `SearchSource` interface is the seam (D-03), but the refusal contract is part of the INTERFACE's documented semantics, and the corpus runner exercises the concrete gazetteer source. A future backend (#2/#3) is a documented escalation path (D-05), not a runtime switch — there is no env flag that swaps the backend in production. |
|
|
| T-04-11 | Elevation of Privilege | The shared module reading env/config to reach a privileged backend | mitigate | The module reads NO environment, NO secrets, and NO network. It has exactly two imports (its own rules table and its own JSON data). The forkability law (D-04: "no hardcoded secrets — env only") is trivially satisfied because there is nothing to configure. |
|
|
</threat_model>
|
|
|
|
<verification>
|
|
- `npm run search:test` green — 9 normalizer vectors + 9 distance vectors + 7 score vectors + 11 gazetteer/license vectors + 3 loader-environment vectors, all named
|
|
- `npm run search:typecheck` green (`tsc --noEmit`, `strict` + `noUncheckedIndexedAccess`)
|
|
- the esbuild bundle of the search module (`node_modules/.bin/esbuild search/src/index.ts --bundle`) contains `suria-klcc` — esbuild (the bundler family wrangler uses) INLINES the dataset at build time, never a runtime fetch; `cd bot && npx wrangler deploy --dry-run` exits 0 as a build regression guard (its bundle-marker grep is asserted in plan 04-03 task 2, after `bot.ts` imports `createGazetteerSource()`)
|
|
- `npm test -w bot` and `npm test -w web` still green (this plan adds no bot/web source, so the existing suites must be unaffected)
|
|
- `node -e "import('./search/src/index.ts')..."` proves `search('KLCC')` → `suria-klcc` at rank 1 and `search('Kuching')` → `[]`
|
|
- `git status` shows no build output staged (`search/` is buildless — nothing under `search/dist/` may exist)
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- SRCH-01 (foundation): a query for "KLCC" resolves to the `suria-klcc` entry through the shared module with no network and no infrastructure; the same code path serves the bot, the Mini App, and the corpus runner
|
|
- SRCH-02 (foundation): "Jln Ampang" and "Jalan Ampang" normalize identically; "Tmn"→"taman"; "Kg"→"kampung"; "Prsn"→"presint"; "tamanbangsar" splits to "taman bangsar"; a transposition typo ("KCLC") still resolves to the right entry through the Damerau residual tier; an exact match always outranks a fuzzy one (disjoint tier bands, asserted)
|
|
- D-09: the abbreviation table is DATA with 47+ JUPEM-derived rows plus the explicit Presint/Prsn vowel-drop row; extending it is a file edit plus a vector, never a code change
|
|
- D-07/D-08: the dataset is typed, committed, diff-friendly JSON; each entry carries its locals' shorthand as native aliases ("KLCC" → suria-klcc, "DUKE" → the-duke) with the canonical OSM name as the `name`
|
|
- D-03: the `SearchSource` seam exists with an async `search()` and a synchronous `byId()`, so backend #2 (WASM geocoder) and #3 (Photon) are format changes rather than rewrites
|
|
- D-15: the shared module contains no logging and no storage surface at all
|
|
- D-04: the repo carries the verbatim GPL-3.0 LICENSE, `LICENSES.md` records the ODbL-1.0 + OpenStreetMap notice for the gazetteer, and the README no longer claims "all rights reserved"
|
|
- The three-environment loader risk is closed mechanically: bare Node resolves the dataset, the esbuild/Worker bundler family provably INLINES it (the search module's own esbuild bundle carries the id marker), and the Vite assertion follows L2's explicit skip-guard; the Worker dry-run exits 0 as a build guard, and its bundle-marker grep completes in plan 04-03 task 2 once `bot.ts` imports the source
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/04-destination-search-kl-gazetteer/04-01-SUMMARY.md`
|
|
</output>
|