arah2/.planning/phases/04-destination-search-kl-gazetteer/04-01-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, decisions, metrics

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	decisions	metrics
04-destination-search-kl-gazetteer	01	search	search gazetteer normalizer damerau scoring licensing workers tdd	npm workspaces root (theme/bot/web) Node 24 type-stripping (corpus-runner pattern)	@araharah/search workspace — the shared SearchSource seam (D-03) GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts (imported by plans 04-03/04-04) normalize() — symmetric Malay place-name normalizer (rule table + folding + glued split) damerau() — hand-rolled OSA distance, zero deps TIER + FUZZY_ADMIT_SIMILARITY — disjoint-tier scorer + the 0.72 admission floor createGazetteerSource() / search() / byId() — the in-Worker gazetteer backend 30-entry curated kl-gazetteer.json with locals' aliases (D-08) GPL-3.0 LICENSE + LICENSES.md (ODbL-1.0 data notice) + corrected README	plan 04-02 (corpus runner imports the module; docs/SEARCH-QUALITY.md) plan 04-03 (bot /cari handler imports createGazetteerSource; Worker bundle marker grep) plan 04-04 (web deep-link reader imports the same module; L2 web-bundle marker half arms)	added patterns search/ — fifth npm workspace, buildless (pure .ts + .json, zero runtime deps) esbuild import attributes: import ... with { type: 'json' } (Node 24 + esbuild 0.28.2 + Vite) hand-rolled zero-dependency test runner (bot/test precedent) lazy module-scope memoization (bot/test/initdata.test.mjs precedent) header-contract module comments (bot/src/security/initdata.ts precedent)	created modified search/package.json search/tsconfig.json search/src/types.ts search/src/rules/abbreviations.ts search/src/normalize.ts search/src/distance.ts search/src/score.ts search/src/data/kl-gazetteer.json search/src/gazetteer.ts search/src/source.ts search/src/index.ts search/test/runner.mjs search/test/harness.mjs search/test/normalizer.test.mjs search/test/distance.test.mjs search/test/score.test.mjs search/test/gazetteer.test.mjs search/test/loader-env.test.mjs LICENSE LICENSES.md package.json (workspaces + search:test/search:typecheck delegates) package-lock.json (workspace linkage) README.md (licence section, four-package table, layout, commands)	Abbreviation table is DATA: 49 vendored rows (JUPEM via Nominatim PR #4056) + explicit Presint/Prsn vowel-drop + MS 2039 A3 Kpg forms; extends by file edit (D-09) Canonical normalizer form is the LONG form — 'Jln Ampang' and 'Jalan Ampang' normalize identically (D-09 symmetry) TIER bands are disjoint literals: EXACT 1.00 / ALIAS_EXACT 0.90 / PREFIX 0.75 / TOKEN_SET 0.65 / FUZZY = 0.60 CEILING (a fuzzy score is fuzzySim × 0.60, never a 0.00 multiplier) Admission is tiered: cheap tiers always pass, FUZZY only at similarity >= 0.72 — 'KCLC' admits (0.45), 'Penang' refuses ([]) TOKEN_SET evaluates before PREFIX (the plan's S5 vector pins the token-boundary case) Dataset is static-imported with the 'with { type: json }' attribute — proven to inline under esbuild; loader index built lazily (never at module scope)	duration completed tasks filesCreated filesModified ~18 min executor time (three server restarts extended wall clock; state recovered from filesystem each time) 2026-10-23 3 20 3

Phase 4 Plan 1: Shared Search Module Summary

One-liner: @araharah/search — a buildless, zero-runtime-dependency search workspace (JUPEM abbreviation table as data, symmetric Malay normalizer, hand-rolled OSA Damerau, disjoint-tier scorer with a 0.72 fuzzy-admission floor, 30-entry curated KL gazetteer with locals' aliases, lazy three-environment loader) plus the repo's GPL-3.0/ODbL licensing — search("KLCC") resolves suria-klcc at 0.90 and search("Kuching") honestly refuses [].

What Was Built

Task 1 — workspace scaffold + contracts + normalizer (RED ebcd542 → GREEN a9f9ab3)

• Fifth npm workspace search/ (@araharah/search, private, type: module, zero runtime deps) with root delegates search:test / search:typecheck; buildless — consumers import @araharah/search/src/index.ts subpaths with extensions (the @araharah/theme precedent).
• types.ts: the verbatim GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts from RESEARCH §Pattern 1 — search() async (empty array = honest refusal), byId() separate and synchronous. Downstream plans import these names.
• rules/abbreviations.ts: the JUPEM-derived table as pure DATA — 49 rows (48 from the Nominatim transcription + the explicit Presint → Prsn vowel-drop row), inverted once by a single generic buildAbbreviationMap() (abbreviation → full forms in table order; full form → itself).
• normalize.ts: NFKD fold → lowercase → punctuation collapse (keeps / for "Jln 3/1") → whole-token expansion (first full form wins; canonical = long form) → glued-word split (longest-starter-first, remainder ≥ 3 chars) → collapse. Idempotent (T7), symmetric (T2).
• T1–T9: rule vectors, symmetry, whole-token-only law ("Pdang" never expands), slash preservation, glued split, vowel-drop, idempotence, folding, and the table-is-data gate (one function, ≥ 47 rows).

Task 2 — Damerau distance + disjoint-tier scorer (RED 25ee1e9 → GREEN 6368bad)

• distance.ts: ~40-line OSA Damerau-Levenshtein, three-row band with the adjacent-transposition term, zero imports (D-07). Pinned by 9 hand-computed vectors (D1–D9) incl. klcc/kclc = 1 and sunway/sunwya = 1.
• score.ts: TIER = { EXACT: 1.00, ALIAS_EXACT: 0.90, PREFIX: 0.75, TOKEN_SET: 0.65, FUZZY: 0.60 } — strict literals, strictly disjoint bands; FUZZY is the band ceiling (score = fuzzySim × 0.60). FUZZY_ADMIT_SIMILARITY = 0.72 exported for source.ts and the corpus runner. scoreOne() order: min-length guard (sub-3-char sides → 0 even on exact) → EXACT → TOKEN_SET (token-boundary superset) → PREFIX (mid-word) → fuzzy residual (max of whole-string and token-level similarity × 0.60). scoreCandidate() scores [name, ...aliases], demotes an exact-ALIAS hit to ALIAS_EXACT so a name-exact outranks it, returns the RAW matched string (corpus provenance), and enforces the tiered admission (sub-floor fuzzy → null). S1–S7 green.

Task 3 — gazetteer + lazy loader + licensing (RED f1b31ca → GREEN 8579da9)

• data/kl-gazetteer.json: 30 hand-curated KL-metro landmarks (10 required + 20 suggested), canonical OSM names with locals' aliases as first-class data ("KLCC" → suria-klcc, "DUKE" → the-duke, "Sg Wang" → sungei-wang), ODbL-1.0 + OSM attribution header traveling inside the file, pretty-printed 2-space.
• gazetteer.ts: the three laws — lazy memoized index never at module scope (Workers startup, T-04-04); static import ... with { type: "json" } (D-07); no logging/storage (D-15). Names and aliases normalized ONCE at index-build (D-09 symmetry).
• source.ts: createGazetteerSource() — name "gazetteer", async search() + synchronous slug-validated byId() (honest null); deterministic ranking (score DESC, id ASC). The tiered admission makes "Kuching"/"Penang" refuse [] — never a fabricated pin (T-04-07).
• index.ts: the named-only public API.
• Licensing: LICENSE = the verbatim GPL-3.0 text fetched from gnu.org (35,149 bytes); LICENSES.md = the dual-licence note (code GPL-3.0, data ODbL-1.0 + © OpenStreetMap contributors, §4.4 share-alike on derived indexes); README licence section corrected (no more "all rights reserved"), search/ documented in the layout, package table, and commands.
• G1–G11 + L1–L3 green. L1 proves bare-Node loading + search("KLCC") → suria-klcc + honest negatives; L2 proves the static-import law (fetch/readFileSync banned) with the web-bundle marker half skipped VISIBLY until plan 04-04 lands; L3 proves esbuild INLINES the dataset (the bundle carries the suria-klcc marker) and the wrangler dry-run exits 0.

Commits

task	commit	message
1 RED	ebcd542	test(04-01): RED — search workspace scaffold + normalizer vector suite (T1-T9)
1 GREEN	a9f9ab3	feat(04-01): search contracts + JUPEM abbreviation table as DATA + symmetric normalizer
2 RED	25ee1e9	test(04-01): RED — OSA distance vectors (D1-D9) + disjoint-tier scorer suite (S1-S7)
2 GREEN	6368bad	feat(04-01): OSA Damerau distance + disjoint-tier scorer (the residual net)
3 RED	f1b31ca	test(04-01): RED — gazetteer schema/license gates (G1-G11) + three-environment loader proof (L1-L3)
3 GREEN	8579da9	feat(04-01): curated KL gazetteer + lazy three-environment loader + GPL/ODbL licensing

Verification Results

• npm run search:test — 39 passed, 0 failed (T1–T9, D1–D9, S1–S7, G1–G11, L1–L3, all named)
• npm run search:typecheck — green (tsc --noEmit, strict + noUncheckedIndexedAccess)
• esbuild bundle of the search module contains the suria-klcc marker (build-time inline proof); npx wrangler deploy --dry-run in bot/ exits 0 (build guard; its marker grep is 04-03's gate)
• npm test -w bot (50) and npm test -w web (33) — unaffected
• Acceptance probes: search("KLCC")[0] = suria-klcc @ 0.90 via raw alias "KLCC"; search("Jln Ampang") and search("KCLC") include suria-klcc; search("Kuching") and search("Penang") = []
• search/ is buildless — no dist output, working tree clean

Deviations from Plan

1. [Rule 3 - Blocking] Added "Kpg" to the Kampung row beyond the 48 vendored rows

• Found during: task 1 (GREEN)
• Issue: the plan's own T1 vector requires "Kpg Ampang" → "kampung ampang", but "Kpg" is not among the 48 minimum rows the contract lists.
• Fix: extended the Kampung row to Kg,Kpg per MS 2039:2006 Annex A3 (same source family as the transcription), documented in the table header. T9's ≥ 47 row count still holds (49 rows).
• Files: search/src/rules/abbreviations.ts

2. [Rule 1 - Bug] S7 assertion refined during GREEN (task 2)

• Found during: task 2
• Issue: the RED suite asserted scoreCandidate("kuching", [...]) === null, but the plan's contract makes admission the SOURCE's job — scoreCandidate legitimately returns sub-admission fuzzy scores ("Kuching" vs suria-klcc ≈ 0.17).
• Fix: S7 now asserts the sub-admission similarity (score / TIER.FUZZY < FUZZY_ADMIT_SIMILARITY) instead of null; the honest-refusal [] behavior is asserted at the source level (L1).
• Files: search/test/score.test.mjs

3. [Planner-note] S5's literal === 0 was unsatisfiable as written

• Issue: the plan's S5 text says scoreOne("taman bangsar", "taman bukit") === 0, but the plan's own S6 semantics (and its node -e acceptance check, which only forbids TOKEN_SET) give every ≥3-char pair a nonzero fuzzy residual (~0.39 here).
• Fix: asserted !== TIER.TOKEN_SET && < TIER.FUZZY — matching the acceptance criterion and the parenthetical intent ("must NOT be TOKEN_SET").

4. [Planner-note] TOKEN_SET evaluates before PREFIX

• Issue: the scorer_contract lists PREFIX before TOKEN_SET, but its own S5 vector pins scoreOne("taman bangsar", "taman bangsar kib") === TIER.TOKEN_SET — which is also a string prefix. S4 (mid-word PREFIX) still holds under TOKEN_SET-first ordering.
• Fix: token-set subset is checked before the string-prefix test.

5. [Recovery] Three server restarts mid-run

• Lost tool results caused duplicate/variant file writes. Recovered from the filesystem each time: re-verified content, re-ran gates, amended the task-2 GREEN commit (6368bad) to hold the final variant of distance.ts/score.ts/distance.test.mjs. All commits verified green post-recovery.

Auth Gates

None — the module reads no environment, no secrets, no network (D-04/D-11 trivially satisfied).

Known Stubs

None — every export is wired to real data/behavior; no TODO/FIXME/placeholder anywhere in search/.

Threat Flags

None — the built surface is exactly the plan's threat model (T-04-01…T-04-11): no new endpoints, no auth paths, no storage, no env reads. T-04-01/02 (no logging/storage) are gated by G10; T-04-04 (lazy loader) by G11; T-04-07 (honest refusal) by L1 + the SRCH probes.

TDD Gate Compliance

PASSED — each task has a test(...) RED commit followed by a feat(...) GREEN commit: ebcd542→a9f9ab3, 25ee1e9→6368bad, f1b31ca→8579da9. RED phases were verified failing (9, 16, and 13 failing cases respectively) before each GREEN.

Self-Check: PASSED

• All 20 created files exist on disk (verified via find search/ -type f = 16 source/test files
  • LICENSE + LICENSES.md + the 2 root manifests).
• All 6 commit hashes present in git log --grep="04-01" (verified above).
• Suite green at 39/39, typecheck green, bot/web suites green.