diff --git a/.planning/phases/04-destination-search-kl-gazetteer/04-01-SUMMARY.md b/.planning/phases/04-destination-search-kl-gazetteer/04-01-SUMMARY.md new file mode 100644 index 0000000..4343aeb --- /dev/null +++ b/.planning/phases/04-destination-search-kl-gazetteer/04-01-SUMMARY.md @@ -0,0 +1,216 @@ +--- +phase: 04-destination-search-kl-gazetteer +plan: 01 +subsystem: search +tags: [search, gazetteer, normalizer, damerau, scoring, licensing, workers, tdd] +requires: + - npm workspaces root (theme/bot/web) + - Node 24 type-stripping (corpus-runner pattern) +provides: + - "@araharah/search workspace — the shared SearchSource seam (D-03)" + - "GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts (imported by plans 04-03/04-04)" + - "normalize() — symmetric Malay place-name normalizer (rule table + folding + glued split)" + - "damerau() — hand-rolled OSA distance, zero deps" + - "TIER + FUZZY_ADMIT_SIMILARITY — disjoint-tier scorer + the 0.72 admission floor" + - "createGazetteerSource() / search() / byId() — the in-Worker gazetteer backend" + - "30-entry curated kl-gazetteer.json with locals' aliases (D-08)" + - "GPL-3.0 LICENSE + LICENSES.md (ODbL-1.0 data notice) + corrected README" +affects: + - "plan 04-02 (corpus runner imports the module; docs/SEARCH-QUALITY.md)" + - "plan 04-03 (bot /cari handler imports createGazetteerSource; Worker bundle marker grep)" + - "plan 04-04 (web deep-link reader imports the same module; L2 web-bundle marker half arms)" +tech-stack: + added: + - "search/ — fifth npm workspace, buildless (pure .ts + .json, zero runtime deps)" + - "esbuild import attributes: import ... with { type: 'json' } (Node 24 + esbuild 0.28.2 + Vite)" + patterns: + - "hand-rolled zero-dependency test runner (bot/test precedent)" + - "lazy module-scope memoization (bot/test/initdata.test.mjs precedent)" + - "header-contract module comments (bot/src/security/initdata.ts precedent)" +key-files: + created: + - search/package.json + - search/tsconfig.json + - search/src/types.ts + - search/src/rules/abbreviations.ts + - search/src/normalize.ts + - search/src/distance.ts + - search/src/score.ts + - search/src/data/kl-gazetteer.json + - search/src/gazetteer.ts + - search/src/source.ts + - search/src/index.ts + - search/test/runner.mjs + - search/test/harness.mjs + - search/test/normalizer.test.mjs + - search/test/distance.test.mjs + - search/test/score.test.mjs + - search/test/gazetteer.test.mjs + - search/test/loader-env.test.mjs + - LICENSE + - LICENSES.md + modified: + - package.json (workspaces + search:test/search:typecheck delegates) + - package-lock.json (workspace linkage) + - README.md (licence section, four-package table, layout, commands) +decisions: + - "Abbreviation table is DATA: 49 vendored rows (JUPEM via Nominatim PR #4056) + explicit Presint/Prsn vowel-drop + MS 2039 A3 Kpg forms; extends by file edit (D-09)" + - "Canonical normalizer form is the LONG form — 'Jln Ampang' and 'Jalan Ampang' normalize identically (D-09 symmetry)" + - "TIER bands are disjoint literals: EXACT 1.00 / ALIAS_EXACT 0.90 / PREFIX 0.75 / TOKEN_SET 0.65 / FUZZY = 0.60 CEILING (a fuzzy score is fuzzySim × 0.60, never a 0.00 multiplier)" + - "Admission is tiered: cheap tiers always pass, FUZZY only at similarity >= 0.72 — 'KCLC' admits (0.45), 'Penang' refuses ([])" + - "TOKEN_SET evaluates before PREFIX (the plan's S5 vector pins the token-boundary case)" + - "Dataset is static-imported with the 'with { type: json }' attribute — proven to inline under esbuild; loader index built lazily (never at module scope)" +metrics: + duration: "~18 min executor time (three server restarts extended wall clock; state recovered from filesystem each time)" + completed: 2026-10-23 + tasks: 3 + filesCreated: 20 + filesModified: 3 +--- + +# Phase 4 Plan 1: Shared Search Module Summary + +**One-liner:** `@araharah/search` — a buildless, zero-runtime-dependency search workspace (JUPEM +abbreviation table as data, symmetric Malay normalizer, hand-rolled OSA Damerau, disjoint-tier +scorer with a 0.72 fuzzy-admission floor, 30-entry curated KL gazetteer with locals' aliases, lazy +three-environment loader) plus the repo's GPL-3.0/ODbL licensing — `search("KLCC")` resolves +`suria-klcc` at 0.90 and `search("Kuching")` honestly refuses `[]`. + +## What Was Built + +### Task 1 — workspace scaffold + contracts + normalizer (RED `ebcd542` → GREEN `a9f9ab3`) +- Fifth npm workspace `search/` (`@araharah/search`, private, type: module, **zero runtime deps**) + with root delegates `search:test` / `search:typecheck`; buildless — consumers import + `@araharah/search/src/index.ts` subpaths with extensions (the `@araharah/theme` precedent). +- `types.ts`: the verbatim `GazetteerEntry` / `SearchQuery` / `SearchHit` / `SearchSource` contracts + from RESEARCH §Pattern 1 — `search()` async (empty array = honest refusal), `byId()` separate and + synchronous. Downstream plans import these names. +- `rules/abbreviations.ts`: the JUPEM-derived table as pure DATA — 49 rows (48 from the Nominatim + transcription + the explicit `Presint → Prsn` vowel-drop row), inverted once by a single generic + `buildAbbreviationMap()` (abbreviation → full forms in table order; full form → itself). +- `normalize.ts`: NFKD fold → lowercase → punctuation collapse (keeps `/` for "Jln 3/1") → + whole-token expansion (first full form wins; canonical = long form) → glued-word split + (longest-starter-first, remainder ≥ 3 chars) → collapse. Idempotent (T7), symmetric (T2). +- T1–T9: rule vectors, symmetry, whole-token-only law ("Pdang" never expands), slash preservation, + glued split, vowel-drop, idempotence, folding, and the table-is-data gate (one function, ≥ 47 rows). + +### Task 2 — Damerau distance + disjoint-tier scorer (RED `25ee1e9` → GREEN `6368bad`) +- `distance.ts`: ~40-line OSA Damerau-Levenshtein, three-row band with the adjacent-transposition + term, **zero imports** (D-07). Pinned by 9 hand-computed vectors (D1–D9) incl. `klcc/kclc = 1` + and `sunway/sunwya = 1`. +- `score.ts`: `TIER = { EXACT: 1.00, ALIAS_EXACT: 0.90, PREFIX: 0.75, TOKEN_SET: 0.65, FUZZY: 0.60 }` + — strict literals, strictly disjoint bands; FUZZY is the band **ceiling** (score = fuzzySim × 0.60). + `FUZZY_ADMIT_SIMILARITY = 0.72` exported for source.ts and the corpus runner. `scoreOne()` order: + min-length guard (sub-3-char sides → 0 even on exact) → EXACT → TOKEN_SET (token-boundary superset) + → PREFIX (mid-word) → fuzzy residual (max of whole-string and token-level similarity × 0.60). + `scoreCandidate()` scores [name, ...aliases], demotes an exact-ALIAS hit to ALIAS_EXACT so a + name-exact outranks it, returns the RAW matched string (corpus provenance), and enforces the + tiered admission (sub-floor fuzzy → null). S1–S7 green. + +### Task 3 — gazetteer + lazy loader + licensing (RED `f1b31ca` → GREEN `8579da9`) +- `data/kl-gazetteer.json`: 30 hand-curated KL-metro landmarks (10 required + 20 suggested), + canonical OSM names with locals' aliases as first-class data (`"KLCC"` → suria-klcc, `"DUKE"` → + the-duke, `"Sg Wang"` → sungei-wang), ODbL-1.0 + OSM attribution header traveling inside the + file, pretty-printed 2-space. +- `gazetteer.ts`: the three laws — lazy memoized index never at module scope (Workers startup, + T-04-04); static `import ... with { type: "json" }` (D-07); no logging/storage (D-15). Names and + aliases normalized ONCE at index-build (D-09 symmetry). +- `source.ts`: `createGazetteerSource()` — name "gazetteer", async `search()` + synchronous + slug-validated `byId()` (honest null); deterministic ranking (score DESC, id ASC). The tiered + admission makes "Kuching"/"Penang" refuse `[]` — never a fabricated pin (T-04-07). +- `index.ts`: the named-only public API. +- Licensing: **LICENSE** = the verbatim GPL-3.0 text fetched from gnu.org (35,149 bytes); + **LICENSES.md** = the dual-licence note (code GPL-3.0, data ODbL-1.0 + © OpenStreetMap + contributors, §4.4 share-alike on derived indexes); **README** licence section corrected (no more + "all rights reserved"), search/ documented in the layout, package table, and commands. +- G1–G11 + L1–L3 green. L1 proves bare-Node loading + `search("KLCC")` → suria-klcc + honest + negatives; L2 proves the static-import law (fetch/readFileSync banned) with the web-bundle marker + half skipped VISIBLY until plan 04-04 lands; L3 proves esbuild INLINES the dataset (the bundle + carries the `suria-klcc` marker) and the wrangler dry-run exits 0. + +## Commits + +| task | commit | message | +| ---- | ------ | ------- | +| 1 RED | `ebcd542` | test(04-01): RED — search workspace scaffold + normalizer vector suite (T1-T9) | +| 1 GREEN | `a9f9ab3` | feat(04-01): search contracts + JUPEM abbreviation table as DATA + symmetric normalizer | +| 2 RED | `25ee1e9` | test(04-01): RED — OSA distance vectors (D1-D9) + disjoint-tier scorer suite (S1-S7) | +| 2 GREEN | `6368bad` | feat(04-01): OSA Damerau distance + disjoint-tier scorer (the residual net) | +| 3 RED | `f1b31ca` | test(04-01): RED — gazetteer schema/license gates (G1-G11) + three-environment loader proof (L1-L3) | +| 3 GREEN | `8579da9` | feat(04-01): curated KL gazetteer + lazy three-environment loader + GPL/ODbL licensing | + +## Verification Results + +- `npm run search:test` — **39 passed, 0 failed** (T1–T9, D1–D9, S1–S7, G1–G11, L1–L3, all named) +- `npm run search:typecheck` — green (`tsc --noEmit`, strict + noUncheckedIndexedAccess) +- esbuild bundle of the search module contains the `suria-klcc` marker (build-time inline proof); + `npx wrangler deploy --dry-run` in bot/ exits 0 (build guard; its marker grep is 04-03's gate) +- `npm test -w bot` (50) and `npm test -w web` (33) — unaffected +- Acceptance probes: `search("KLCC")[0]` = suria-klcc @ 0.90 via raw alias "KLCC"; + `search("Jln Ampang")` and `search("KCLC")` include suria-klcc; `search("Kuching")` and + `search("Penang")` = `[]` +- `search/` is buildless — no dist output, working tree clean + +## Deviations from Plan + +**1. [Rule 3 - Blocking] Added "Kpg" to the Kampung row beyond the 48 vendored rows** +- **Found during:** task 1 (GREEN) +- **Issue:** the plan's own T1 vector requires `"Kpg Ampang" → "kampung ampang"`, but "Kpg" is not + among the 48 minimum rows the contract lists. +- **Fix:** extended the Kampung row to `Kg,Kpg` per MS 2039:2006 Annex A3 (same source family as + the transcription), documented in the table header. T9's ≥ 47 row count still holds (49 rows). +- **Files:** search/src/rules/abbreviations.ts + +**2. [Rule 1 - Bug] S7 assertion refined during GREEN (task 2)** +- **Found during:** task 2 +- **Issue:** the RED suite asserted `scoreCandidate("kuching", [...]) === null`, but the plan's + contract makes admission the SOURCE's job — scoreCandidate legitimately returns sub-admission + fuzzy scores ("Kuching" vs suria-klcc ≈ 0.17). +- **Fix:** S7 now asserts the sub-admission similarity (`score / TIER.FUZZY < FUZZY_ADMIT_SIMILARITY`) + instead of null; the honest-refusal `[]` behavior is asserted at the source level (L1). +- **Files:** search/test/score.test.mjs + +**3. [Planner-note] S5's literal `=== 0` was unsatisfiable as written** +- **Issue:** the plan's S5 text says `scoreOne("taman bangsar", "taman bukit") === 0`, but the + plan's own S6 semantics (and its `node -e` acceptance check, which only forbids TOKEN_SET) give + every ≥3-char pair a nonzero fuzzy residual (~0.39 here). +- **Fix:** asserted `!== TIER.TOKEN_SET && < TIER.FUZZY` — matching the acceptance criterion and + the parenthetical intent ("must NOT be TOKEN_SET"). + +**4. [Planner-note] TOKEN_SET evaluates before PREFIX** +- **Issue:** the scorer_contract lists PREFIX before TOKEN_SET, but its own S5 vector pins + `scoreOne("taman bangsar", "taman bangsar kib") === TIER.TOKEN_SET` — which is also a string + prefix. S4 (mid-word PREFIX) still holds under TOKEN_SET-first ordering. +- **Fix:** token-set subset is checked before the string-prefix test. + +**5. [Recovery] Three server restarts mid-run** +- Lost tool results caused duplicate/variant file writes. Recovered from the filesystem each time: + re-verified content, re-ran gates, amended the task-2 GREEN commit (`6368bad`) to hold the final + variant of distance.ts/score.ts/distance.test.mjs. All commits verified green post-recovery. + +## Auth Gates + +None — the module reads no environment, no secrets, no network (D-04/D-11 trivially satisfied). + +## Known Stubs + +None — every export is wired to real data/behavior; no TODO/FIXME/placeholder anywhere in search/. + +## Threat Flags + +None — the built surface is exactly the plan's threat model (T-04-01…T-04-11): no new endpoints, +no auth paths, no storage, no env reads. T-04-01/02 (no logging/storage) are gated by G10; +T-04-04 (lazy loader) by G11; T-04-07 (honest refusal) by L1 + the SRCH probes. + +## TDD Gate Compliance + +PASSED — each task has a `test(...)` RED commit followed by a `feat(...)` GREEN commit: +`ebcd542→a9f9ab3`, `25ee1e9→6368bad`, `f1b31ca→8579da9`. RED phases were verified failing +(9, 16, and 13 failing cases respectively) before each GREEN. + +## Self-Check: PASSED + +- All 20 created files exist on disk (verified via `find search/ -type f` = 16 source/test files + + LICENSE + LICENSES.md + the 2 root manifests). +- All 6 commit hashes present in `git log --grep="04-01"` (verified above). +- Suite green at 39/39, typecheck green, bot/web suites green.