docs(04-01): complete shared-search-module plan

- 3 tasks executed RED→GREEN (6 commits): workspace scaffold + JUPEM normalizer, OSA distance + disjoint-tier scorer, curated 30-entry KL gazetteer + lazy three-environment loader + GPL/ODbL licensing - 39/39 suite green; typecheck clean; esbuild inlines the dataset; wrangler dry-run exit 0; bot + web suites unaffected - SRCH-01/SRCH-02 foundation proven: KLCC→suria-klcc, negatives refused
2026-10-24 00:38:58 +08:00
parent 8579da9d71
commit 8b6ad2047f
1 changed files with 216 additions and 0 deletions
@@ -0,0 +1,216 @@
+---
+phase: 04-destination-search-kl-gazetteer
+plan: 01
+subsystem: search
+tags: [search, gazetteer, normalizer, damerau, scoring, licensing, workers, tdd]
+requires:
+  - npm workspaces root (theme/bot/web)
+  - Node 24 type-stripping (corpus-runner pattern)
+provides:
+  - "@araharah/search workspace — the shared SearchSource seam (D-03)"
+  - "GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts (imported by plans 04-03/04-04)"
+  - "normalize() — symmetric Malay place-name normalizer (rule table + folding + glued split)"
+  - "damerau() — hand-rolled OSA distance, zero deps"
+  - "TIER + FUZZY_ADMIT_SIMILARITY — disjoint-tier scorer + the 0.72 admission floor"
+  - "createGazetteerSource() / search() / byId() — the in-Worker gazetteer backend"
+  - "30-entry curated kl-gazetteer.json with locals' aliases (D-08)"
+  - "GPL-3.0 LICENSE + LICENSES.md (ODbL-1.0 data notice) + corrected README"
+affects:
+  - "plan 04-02 (corpus runner imports the module; docs/SEARCH-QUALITY.md)"
+  - "plan 04-03 (bot /cari handler imports createGazetteerSource; Worker bundle marker grep)"
+  - "plan 04-04 (web deep-link reader imports the same module; L2 web-bundle marker half arms)"
+tech-stack:
+  added:
+    - "search/ — fifth npm workspace, buildless (pure .ts + .json, zero runtime deps)"
+    - "esbuild import attributes: import ... with { type: 'json' } (Node 24 + esbuild 0.28.2 + Vite)"
+  patterns:
+    - "hand-rolled zero-dependency test runner (bot/test precedent)"
+    - "lazy module-scope memoization (bot/test/initdata.test.mjs precedent)"
+    - "header-contract module comments (bot/src/security/initdata.ts precedent)"
+key-files:
+  created:
+    - search/package.json
+    - search/tsconfig.json
+    - search/src/types.ts
+    - search/src/rules/abbreviations.ts
+    - search/src/normalize.ts
+    - search/src/distance.ts
+    - search/src/score.ts
+    - search/src/data/kl-gazetteer.json
+    - search/src/gazetteer.ts
+    - search/src/source.ts
+    - search/src/index.ts
+    - search/test/runner.mjs
+    - search/test/harness.mjs
+    - search/test/normalizer.test.mjs
+    - search/test/distance.test.mjs
+    - search/test/score.test.mjs
+    - search/test/gazetteer.test.mjs
+    - search/test/loader-env.test.mjs
+    - LICENSE
+    - LICENSES.md
+  modified:
+    - package.json (workspaces + search:test/search:typecheck delegates)
+    - package-lock.json (workspace linkage)
+    - README.md (licence section, four-package table, layout, commands)
+decisions:
+  - "Abbreviation table is DATA: 49 vendored rows (JUPEM via Nominatim PR #4056) + explicit Presint/Prsn vowel-drop + MS 2039 A3 Kpg forms; extends by file edit (D-09)"
+  - "Canonical normalizer form is the LONG form — 'Jln Ampang' and 'Jalan Ampang' normalize identically (D-09 symmetry)"
+  - "TIER bands are disjoint literals: EXACT 1.00 / ALIAS_EXACT 0.90 / PREFIX 0.75 / TOKEN_SET 0.65 / FUZZY = 0.60 CEILING (a fuzzy score is fuzzySim × 0.60, never a 0.00 multiplier)"
+  - "Admission is tiered: cheap tiers always pass, FUZZY only at similarity >= 0.72 — 'KCLC' admits (0.45), 'Penang' refuses ([])"
+  - "TOKEN_SET evaluates before PREFIX (the plan's S5 vector pins the token-boundary case)"
+  - "Dataset is static-imported with the 'with { type: json }' attribute — proven to inline under esbuild; loader index built lazily (never at module scope)"
+metrics:
+  duration: "~18 min executor time (three server restarts extended wall clock; state recovered from filesystem each time)"
+  completed: 2026-10-23
+  tasks: 3
+  filesCreated: 20
+  filesModified: 3
+---
+
+# Phase 4 Plan 1: Shared Search Module Summary
+
+**One-liner:** `@araharah/search` — a buildless, zero-runtime-dependency search workspace (JUPEM
+abbreviation table as data, symmetric Malay normalizer, hand-rolled OSA Damerau, disjoint-tier
+scorer with a 0.72 fuzzy-admission floor, 30-entry curated KL gazetteer with locals' aliases, lazy
+three-environment loader) plus the repo's GPL-3.0/ODbL licensing — `search("KLCC")` resolves
+`suria-klcc` at 0.90 and `search("Kuching")` honestly refuses `[]`.
+
+## What Was Built
+
+### Task 1 — workspace scaffold + contracts + normalizer (RED `ebcd542` → GREEN `a9f9ab3`)
+- Fifth npm workspace `search/` (`@araharah/search`, private, type: module, **zero runtime deps**)
+  with root delegates `search:test` / `search:typecheck`; buildless — consumers import
+  `@araharah/search/src/index.ts` subpaths with extensions (the `@araharah/theme` precedent).
+- `types.ts`: the verbatim `GazetteerEntry` / `SearchQuery` / `SearchHit` / `SearchSource` contracts
+  from RESEARCH §Pattern 1 — `search()` async (empty array = honest refusal), `byId()` separate and
+  synchronous. Downstream plans import these names.
+- `rules/abbreviations.ts`: the JUPEM-derived table as pure DATA — 49 rows (48 from the Nominatim
+  transcription + the explicit `Presint → Prsn` vowel-drop row), inverted once by a single generic
+  `buildAbbreviationMap()` (abbreviation → full forms in table order; full form → itself).
+- `normalize.ts`: NFKD fold → lowercase → punctuation collapse (keeps `/` for "Jln 3/1") →
+  whole-token expansion (first full form wins; canonical = long form) → glued-word split
+  (longest-starter-first, remainder ≥ 3 chars) → collapse. Idempotent (T7), symmetric (T2).
+- T1–T9: rule vectors, symmetry, whole-token-only law ("Pdang" never expands), slash preservation,
+  glued split, vowel-drop, idempotence, folding, and the table-is-data gate (one function, ≥ 47 rows).
+
+### Task 2 — Damerau distance + disjoint-tier scorer (RED `25ee1e9` → GREEN `6368bad`)
+- `distance.ts`: ~40-line OSA Damerau-Levenshtein, three-row band with the adjacent-transposition
+  term, **zero imports** (D-07). Pinned by 9 hand-computed vectors (D1–D9) incl. `klcc/kclc = 1`
+  and `sunway/sunwya = 1`.
+- `score.ts`: `TIER = { EXACT: 1.00, ALIAS_EXACT: 0.90, PREFIX: 0.75, TOKEN_SET: 0.65, FUZZY: 0.60 }`
+  — strict literals, strictly disjoint bands; FUZZY is the band **ceiling** (score = fuzzySim × 0.60).
+  `FUZZY_ADMIT_SIMILARITY = 0.72` exported for source.ts and the corpus runner. `scoreOne()` order:
+  min-length guard (sub-3-char sides → 0 even on exact) → EXACT → TOKEN_SET (token-boundary superset)
+  → PREFIX (mid-word) → fuzzy residual (max of whole-string and token-level similarity × 0.60).
+  `scoreCandidate()` scores [name, ...aliases], demotes an exact-ALIAS hit to ALIAS_EXACT so a
+  name-exact outranks it, returns the RAW matched string (corpus provenance), and enforces the
+  tiered admission (sub-floor fuzzy → null). S1–S7 green.
+
+### Task 3 — gazetteer + lazy loader + licensing (RED `f1b31ca` → GREEN `8579da9`)
+- `data/kl-gazetteer.json`: 30 hand-curated KL-metro landmarks (10 required + 20 suggested),
+  canonical OSM names with locals' aliases as first-class data (`"KLCC"` → suria-klcc, `"DUKE"` →
+  the-duke, `"Sg Wang"` → sungei-wang), ODbL-1.0 + OSM attribution header traveling inside the
+  file, pretty-printed 2-space.
+- `gazetteer.ts`: the three laws — lazy memoized index never at module scope (Workers startup,
+  T-04-04); static `import ... with { type: "json" }` (D-07); no logging/storage (D-15). Names and
+  aliases normalized ONCE at index-build (D-09 symmetry).
+- `source.ts`: `createGazetteerSource()` — name "gazetteer", async `search()` + synchronous
+  slug-validated `byId()` (honest null); deterministic ranking (score DESC, id ASC). The tiered
+  admission makes "Kuching"/"Penang" refuse `[]` — never a fabricated pin (T-04-07).
+- `index.ts`: the named-only public API.
+- Licensing: **LICENSE** = the verbatim GPL-3.0 text fetched from gnu.org (35,149 bytes);
+  **LICENSES.md** = the dual-licence note (code GPL-3.0, data ODbL-1.0 + © OpenStreetMap
+  contributors, §4.4 share-alike on derived indexes); **README** licence section corrected (no more
+  "all rights reserved"), search/ documented in the layout, package table, and commands.
+- G1–G11 + L1–L3 green. L1 proves bare-Node loading + `search("KLCC")` → suria-klcc + honest
+  negatives; L2 proves the static-import law (fetch/readFileSync banned) with the web-bundle marker
+  half skipped VISIBLY until plan 04-04 lands; L3 proves esbuild INLINES the dataset (the bundle
+  carries the `suria-klcc` marker) and the wrangler dry-run exits 0.
+
+## Commits
+
+| task | commit | message |
+| ---- | ------ | ------- |
+| 1 RED | `ebcd542` | test(04-01): RED — search workspace scaffold + normalizer vector suite (T1-T9) |
+| 1 GREEN | `a9f9ab3` | feat(04-01): search contracts + JUPEM abbreviation table as DATA + symmetric normalizer |
+| 2 RED | `25ee1e9` | test(04-01): RED — OSA distance vectors (D1-D9) + disjoint-tier scorer suite (S1-S7) |
+| 2 GREEN | `6368bad` | feat(04-01): OSA Damerau distance + disjoint-tier scorer (the residual net) |
+| 3 RED | `f1b31ca` | test(04-01): RED — gazetteer schema/license gates (G1-G11) + three-environment loader proof (L1-L3) |
+| 3 GREEN | `8579da9` | feat(04-01): curated KL gazetteer + lazy three-environment loader + GPL/ODbL licensing |
+
+## Verification Results
+
+- `npm run search:test` — **39 passed, 0 failed** (T1–T9, D1–D9, S1–S7, G1–G11, L1–L3, all named)
+- `npm run search:typecheck` — green (`tsc --noEmit`, strict + noUncheckedIndexedAccess)
+- esbuild bundle of the search module contains the `suria-klcc` marker (build-time inline proof);
+  `npx wrangler deploy --dry-run` in bot/ exits 0 (build guard; its marker grep is 04-03's gate)
+- `npm test -w bot` (50) and `npm test -w web` (33) — unaffected
+- Acceptance probes: `search("KLCC")[0]` = suria-klcc @ 0.90 via raw alias "KLCC";
+  `search("Jln Ampang")` and `search("KCLC")` include suria-klcc; `search("Kuching")` and
+  `search("Penang")` = `[]`
+- `search/` is buildless — no dist output, working tree clean
+
+## Deviations from Plan
+
+**1. [Rule 3 - Blocking] Added "Kpg" to the Kampung row beyond the 48 vendored rows**
+- **Found during:** task 1 (GREEN)
+- **Issue:** the plan's own T1 vector requires `"Kpg Ampang" → "kampung ampang"`, but "Kpg" is not
+  among the 48 minimum rows the contract lists.
+- **Fix:** extended the Kampung row to `Kg,Kpg` per MS 2039:2006 Annex A3 (same source family as
+  the transcription), documented in the table header. T9's ≥ 47 row count still holds (49 rows).
+- **Files:** search/src/rules/abbreviations.ts
+
+**2. [Rule 1 - Bug] S7 assertion refined during GREEN (task 2)**
+- **Found during:** task 2
+- **Issue:** the RED suite asserted `scoreCandidate("kuching", [...]) === null`, but the plan's
+  contract makes admission the SOURCE's job — scoreCandidate legitimately returns sub-admission
+  fuzzy scores ("Kuching" vs suria-klcc ≈ 0.17).
+- **Fix:** S7 now asserts the sub-admission similarity (`score / TIER.FUZZY < FUZZY_ADMIT_SIMILARITY`)
+  instead of null; the honest-refusal `[]` behavior is asserted at the source level (L1).
+- **Files:** search/test/score.test.mjs
+
+**3. [Planner-note] S5's literal `=== 0` was unsatisfiable as written**
+- **Issue:** the plan's S5 text says `scoreOne("taman bangsar", "taman bukit") === 0`, but the
+  plan's own S6 semantics (and its `node -e` acceptance check, which only forbids TOKEN_SET) give
+  every ≥3-char pair a nonzero fuzzy residual (~0.39 here).
+- **Fix:** asserted `!== TIER.TOKEN_SET && < TIER.FUZZY` — matching the acceptance criterion and
+  the parenthetical intent ("must NOT be TOKEN_SET").
+
+**4. [Planner-note] TOKEN_SET evaluates before PREFIX**
+- **Issue:** the scorer_contract lists PREFIX before TOKEN_SET, but its own S5 vector pins
+  `scoreOne("taman bangsar", "taman bangsar kib") === TIER.TOKEN_SET` — which is also a string
+  prefix. S4 (mid-word PREFIX) still holds under TOKEN_SET-first ordering.
+- **Fix:** token-set subset is checked before the string-prefix test.
+
+**5. [Recovery] Three server restarts mid-run**
+- Lost tool results caused duplicate/variant file writes. Recovered from the filesystem each time:
+  re-verified content, re-ran gates, amended the task-2 GREEN commit (`6368bad`) to hold the final
+  variant of distance.ts/score.ts/distance.test.mjs. All commits verified green post-recovery.
+
+## Auth Gates
+
+None — the module reads no environment, no secrets, no network (D-04/D-11 trivially satisfied).
+
+## Known Stubs
+
+None — every export is wired to real data/behavior; no TODO/FIXME/placeholder anywhere in search/.
+
+## Threat Flags
+
+None — the built surface is exactly the plan's threat model (T-04-01…T-04-11): no new endpoints,
+no auth paths, no storage, no env reads. T-04-01/02 (no logging/storage) are gated by G10;
+T-04-04 (lazy loader) by G11; T-04-07 (honest refusal) by L1 + the SRCH probes.
+
+## TDD Gate Compliance
+
+PASSED — each task has a `test(...)` RED commit followed by a `feat(...)` GREEN commit:
+`ebcd542→a9f9ab3`, `25ee1e9→6368bad`, `f1b31ca→8579da9`. RED phases were verified failing
+(9, 16, and 13 failing cases respectively) before each GREEN.
+
+## Self-Check: PASSED
+
+- All 20 created files exist on disk (verified via `find search/ -type f` = 16 source/test files
+  + LICENSE + LICENSES.md + the 2 root manifests).
+- All 6 commit hashes present in `git log --grep="04-01"` (verified above).
+- Suite green at 39/39, typecheck green, bot/web suites green.