araharah-executor
8b6ad2047f
- 3 tasks executed RED→GREEN (6 commits): workspace scaffold + JUPEM normalizer, OSA distance + disjoint-tier scorer, curated 30-entry KL gazetteer + lazy three-environment loader + GPL/ODbL licensing - 39/39 suite green; typecheck clean; esbuild inlines the dataset; wrangler dry-run exit 0; bot + web suites unaffected - SRCH-01/SRCH-02 foundation proven: KLCC→suria-klcc, negatives refused
217 lines
13 KiB
Markdown
217 lines
13 KiB
Markdown
---
|
||
phase: 04-destination-search-kl-gazetteer
|
||
plan: 01
|
||
subsystem: search
|
||
tags: [search, gazetteer, normalizer, damerau, scoring, licensing, workers, tdd]
|
||
requires:
|
||
- npm workspaces root (theme/bot/web)
|
||
- Node 24 type-stripping (corpus-runner pattern)
|
||
provides:
|
||
- "@araharah/search workspace — the shared SearchSource seam (D-03)"
|
||
- "GazetteerEntry / SearchQuery / SearchHit / SearchSource contracts (imported by plans 04-03/04-04)"
|
||
- "normalize() — symmetric Malay place-name normalizer (rule table + folding + glued split)"
|
||
- "damerau() — hand-rolled OSA distance, zero deps"
|
||
- "TIER + FUZZY_ADMIT_SIMILARITY — disjoint-tier scorer + the 0.72 admission floor"
|
||
- "createGazetteerSource() / search() / byId() — the in-Worker gazetteer backend"
|
||
- "30-entry curated kl-gazetteer.json with locals' aliases (D-08)"
|
||
- "GPL-3.0 LICENSE + LICENSES.md (ODbL-1.0 data notice) + corrected README"
|
||
affects:
|
||
- "plan 04-02 (corpus runner imports the module; docs/SEARCH-QUALITY.md)"
|
||
- "plan 04-03 (bot /cari handler imports createGazetteerSource; Worker bundle marker grep)"
|
||
- "plan 04-04 (web deep-link reader imports the same module; L2 web-bundle marker half arms)"
|
||
tech-stack:
|
||
added:
|
||
- "search/ — fifth npm workspace, buildless (pure .ts + .json, zero runtime deps)"
|
||
- "esbuild import attributes: import ... with { type: 'json' } (Node 24 + esbuild 0.28.2 + Vite)"
|
||
patterns:
|
||
- "hand-rolled zero-dependency test runner (bot/test precedent)"
|
||
- "lazy module-scope memoization (bot/test/initdata.test.mjs precedent)"
|
||
- "header-contract module comments (bot/src/security/initdata.ts precedent)"
|
||
key-files:
|
||
created:
|
||
- search/package.json
|
||
- search/tsconfig.json
|
||
- search/src/types.ts
|
||
- search/src/rules/abbreviations.ts
|
||
- search/src/normalize.ts
|
||
- search/src/distance.ts
|
||
- search/src/score.ts
|
||
- search/src/data/kl-gazetteer.json
|
||
- search/src/gazetteer.ts
|
||
- search/src/source.ts
|
||
- search/src/index.ts
|
||
- search/test/runner.mjs
|
||
- search/test/harness.mjs
|
||
- search/test/normalizer.test.mjs
|
||
- search/test/distance.test.mjs
|
||
- search/test/score.test.mjs
|
||
- search/test/gazetteer.test.mjs
|
||
- search/test/loader-env.test.mjs
|
||
- LICENSE
|
||
- LICENSES.md
|
||
modified:
|
||
- package.json (workspaces + search:test/search:typecheck delegates)
|
||
- package-lock.json (workspace linkage)
|
||
- README.md (licence section, four-package table, layout, commands)
|
||
decisions:
|
||
- "Abbreviation table is DATA: 49 vendored rows (JUPEM via Nominatim PR #4056) + explicit Presint/Prsn vowel-drop + MS 2039 A3 Kpg forms; extends by file edit (D-09)"
|
||
- "Canonical normalizer form is the LONG form — 'Jln Ampang' and 'Jalan Ampang' normalize identically (D-09 symmetry)"
|
||
- "TIER bands are disjoint literals: EXACT 1.00 / ALIAS_EXACT 0.90 / PREFIX 0.75 / TOKEN_SET 0.65 / FUZZY = 0.60 CEILING (a fuzzy score is fuzzySim × 0.60, never a 0.00 multiplier)"
|
||
- "Admission is tiered: cheap tiers always pass, FUZZY only at similarity >= 0.72 — 'KCLC' admits (0.45), 'Penang' refuses ([])"
|
||
- "TOKEN_SET evaluates before PREFIX (the plan's S5 vector pins the token-boundary case)"
|
||
- "Dataset is static-imported with the 'with { type: json }' attribute — proven to inline under esbuild; loader index built lazily (never at module scope)"
|
||
metrics:
|
||
duration: "~18 min executor time (three server restarts extended wall clock; state recovered from filesystem each time)"
|
||
completed: 2026-10-23
|
||
tasks: 3
|
||
filesCreated: 20
|
||
filesModified: 3
|
||
---
|
||
|
||
# Phase 4 Plan 1: Shared Search Module Summary
|
||
|
||
**One-liner:** `@araharah/search` — a buildless, zero-runtime-dependency search workspace (JUPEM
|
||
abbreviation table as data, symmetric Malay normalizer, hand-rolled OSA Damerau, disjoint-tier
|
||
scorer with a 0.72 fuzzy-admission floor, 30-entry curated KL gazetteer with locals' aliases, lazy
|
||
three-environment loader) plus the repo's GPL-3.0/ODbL licensing — `search("KLCC")` resolves
|
||
`suria-klcc` at 0.90 and `search("Kuching")` honestly refuses `[]`.
|
||
|
||
## What Was Built
|
||
|
||
### Task 1 — workspace scaffold + contracts + normalizer (RED `ebcd542` → GREEN `a9f9ab3`)
|
||
- Fifth npm workspace `search/` (`@araharah/search`, private, type: module, **zero runtime deps**)
|
||
with root delegates `search:test` / `search:typecheck`; buildless — consumers import
|
||
`@araharah/search/src/index.ts` subpaths with extensions (the `@araharah/theme` precedent).
|
||
- `types.ts`: the verbatim `GazetteerEntry` / `SearchQuery` / `SearchHit` / `SearchSource` contracts
|
||
from RESEARCH §Pattern 1 — `search()` async (empty array = honest refusal), `byId()` separate and
|
||
synchronous. Downstream plans import these names.
|
||
- `rules/abbreviations.ts`: the JUPEM-derived table as pure DATA — 49 rows (48 from the Nominatim
|
||
transcription + the explicit `Presint → Prsn` vowel-drop row), inverted once by a single generic
|
||
`buildAbbreviationMap()` (abbreviation → full forms in table order; full form → itself).
|
||
- `normalize.ts`: NFKD fold → lowercase → punctuation collapse (keeps `/` for "Jln 3/1") →
|
||
whole-token expansion (first full form wins; canonical = long form) → glued-word split
|
||
(longest-starter-first, remainder ≥ 3 chars) → collapse. Idempotent (T7), symmetric (T2).
|
||
- T1–T9: rule vectors, symmetry, whole-token-only law ("Pdang" never expands), slash preservation,
|
||
glued split, vowel-drop, idempotence, folding, and the table-is-data gate (one function, ≥ 47 rows).
|
||
|
||
### Task 2 — Damerau distance + disjoint-tier scorer (RED `25ee1e9` → GREEN `6368bad`)
|
||
- `distance.ts`: ~40-line OSA Damerau-Levenshtein, three-row band with the adjacent-transposition
|
||
term, **zero imports** (D-07). Pinned by 9 hand-computed vectors (D1–D9) incl. `klcc/kclc = 1`
|
||
and `sunway/sunwya = 1`.
|
||
- `score.ts`: `TIER = { EXACT: 1.00, ALIAS_EXACT: 0.90, PREFIX: 0.75, TOKEN_SET: 0.65, FUZZY: 0.60 }`
|
||
— strict literals, strictly disjoint bands; FUZZY is the band **ceiling** (score = fuzzySim × 0.60).
|
||
`FUZZY_ADMIT_SIMILARITY = 0.72` exported for source.ts and the corpus runner. `scoreOne()` order:
|
||
min-length guard (sub-3-char sides → 0 even on exact) → EXACT → TOKEN_SET (token-boundary superset)
|
||
→ PREFIX (mid-word) → fuzzy residual (max of whole-string and token-level similarity × 0.60).
|
||
`scoreCandidate()` scores [name, ...aliases], demotes an exact-ALIAS hit to ALIAS_EXACT so a
|
||
name-exact outranks it, returns the RAW matched string (corpus provenance), and enforces the
|
||
tiered admission (sub-floor fuzzy → null). S1–S7 green.
|
||
|
||
### Task 3 — gazetteer + lazy loader + licensing (RED `f1b31ca` → GREEN `8579da9`)
|
||
- `data/kl-gazetteer.json`: 30 hand-curated KL-metro landmarks (10 required + 20 suggested),
|
||
canonical OSM names with locals' aliases as first-class data (`"KLCC"` → suria-klcc, `"DUKE"` →
|
||
the-duke, `"Sg Wang"` → sungei-wang), ODbL-1.0 + OSM attribution header traveling inside the
|
||
file, pretty-printed 2-space.
|
||
- `gazetteer.ts`: the three laws — lazy memoized index never at module scope (Workers startup,
|
||
T-04-04); static `import ... with { type: "json" }` (D-07); no logging/storage (D-15). Names and
|
||
aliases normalized ONCE at index-build (D-09 symmetry).
|
||
- `source.ts`: `createGazetteerSource()` — name "gazetteer", async `search()` + synchronous
|
||
slug-validated `byId()` (honest null); deterministic ranking (score DESC, id ASC). The tiered
|
||
admission makes "Kuching"/"Penang" refuse `[]` — never a fabricated pin (T-04-07).
|
||
- `index.ts`: the named-only public API.
|
||
- Licensing: **LICENSE** = the verbatim GPL-3.0 text fetched from gnu.org (35,149 bytes);
|
||
**LICENSES.md** = the dual-licence note (code GPL-3.0, data ODbL-1.0 + © OpenStreetMap
|
||
contributors, §4.4 share-alike on derived indexes); **README** licence section corrected (no more
|
||
"all rights reserved"), search/ documented in the layout, package table, and commands.
|
||
- G1–G11 + L1–L3 green. L1 proves bare-Node loading + `search("KLCC")` → suria-klcc + honest
|
||
negatives; L2 proves the static-import law (fetch/readFileSync banned) with the web-bundle marker
|
||
half skipped VISIBLY until plan 04-04 lands; L3 proves esbuild INLINES the dataset (the bundle
|
||
carries the `suria-klcc` marker) and the wrangler dry-run exits 0.
|
||
|
||
## Commits
|
||
|
||
| task | commit | message |
|
||
| ---- | ------ | ------- |
|
||
| 1 RED | `ebcd542` | test(04-01): RED — search workspace scaffold + normalizer vector suite (T1-T9) |
|
||
| 1 GREEN | `a9f9ab3` | feat(04-01): search contracts + JUPEM abbreviation table as DATA + symmetric normalizer |
|
||
| 2 RED | `25ee1e9` | test(04-01): RED — OSA distance vectors (D1-D9) + disjoint-tier scorer suite (S1-S7) |
|
||
| 2 GREEN | `6368bad` | feat(04-01): OSA Damerau distance + disjoint-tier scorer (the residual net) |
|
||
| 3 RED | `f1b31ca` | test(04-01): RED — gazetteer schema/license gates (G1-G11) + three-environment loader proof (L1-L3) |
|
||
| 3 GREEN | `8579da9` | feat(04-01): curated KL gazetteer + lazy three-environment loader + GPL/ODbL licensing |
|
||
|
||
## Verification Results
|
||
|
||
- `npm run search:test` — **39 passed, 0 failed** (T1–T9, D1–D9, S1–S7, G1–G11, L1–L3, all named)
|
||
- `npm run search:typecheck` — green (`tsc --noEmit`, strict + noUncheckedIndexedAccess)
|
||
- esbuild bundle of the search module contains the `suria-klcc` marker (build-time inline proof);
|
||
`npx wrangler deploy --dry-run` in bot/ exits 0 (build guard; its marker grep is 04-03's gate)
|
||
- `npm test -w bot` (50) and `npm test -w web` (33) — unaffected
|
||
- Acceptance probes: `search("KLCC")[0]` = suria-klcc @ 0.90 via raw alias "KLCC";
|
||
`search("Jln Ampang")` and `search("KCLC")` include suria-klcc; `search("Kuching")` and
|
||
`search("Penang")` = `[]`
|
||
- `search/` is buildless — no dist output, working tree clean
|
||
|
||
## Deviations from Plan
|
||
|
||
**1. [Rule 3 - Blocking] Added "Kpg" to the Kampung row beyond the 48 vendored rows**
|
||
- **Found during:** task 1 (GREEN)
|
||
- **Issue:** the plan's own T1 vector requires `"Kpg Ampang" → "kampung ampang"`, but "Kpg" is not
|
||
among the 48 minimum rows the contract lists.
|
||
- **Fix:** extended the Kampung row to `Kg,Kpg` per MS 2039:2006 Annex A3 (same source family as
|
||
the transcription), documented in the table header. T9's ≥ 47 row count still holds (49 rows).
|
||
- **Files:** search/src/rules/abbreviations.ts
|
||
|
||
**2. [Rule 1 - Bug] S7 assertion refined during GREEN (task 2)**
|
||
- **Found during:** task 2
|
||
- **Issue:** the RED suite asserted `scoreCandidate("kuching", [...]) === null`, but the plan's
|
||
contract makes admission the SOURCE's job — scoreCandidate legitimately returns sub-admission
|
||
fuzzy scores ("Kuching" vs suria-klcc ≈ 0.17).
|
||
- **Fix:** S7 now asserts the sub-admission similarity (`score / TIER.FUZZY < FUZZY_ADMIT_SIMILARITY`)
|
||
instead of null; the honest-refusal `[]` behavior is asserted at the source level (L1).
|
||
- **Files:** search/test/score.test.mjs
|
||
|
||
**3. [Planner-note] S5's literal `=== 0` was unsatisfiable as written**
|
||
- **Issue:** the plan's S5 text says `scoreOne("taman bangsar", "taman bukit") === 0`, but the
|
||
plan's own S6 semantics (and its `node -e` acceptance check, which only forbids TOKEN_SET) give
|
||
every ≥3-char pair a nonzero fuzzy residual (~0.39 here).
|
||
- **Fix:** asserted `!== TIER.TOKEN_SET && < TIER.FUZZY` — matching the acceptance criterion and
|
||
the parenthetical intent ("must NOT be TOKEN_SET").
|
||
|
||
**4. [Planner-note] TOKEN_SET evaluates before PREFIX**
|
||
- **Issue:** the scorer_contract lists PREFIX before TOKEN_SET, but its own S5 vector pins
|
||
`scoreOne("taman bangsar", "taman bangsar kib") === TIER.TOKEN_SET` — which is also a string
|
||
prefix. S4 (mid-word PREFIX) still holds under TOKEN_SET-first ordering.
|
||
- **Fix:** token-set subset is checked before the string-prefix test.
|
||
|
||
**5. [Recovery] Three server restarts mid-run**
|
||
- Lost tool results caused duplicate/variant file writes. Recovered from the filesystem each time:
|
||
re-verified content, re-ran gates, amended the task-2 GREEN commit (`6368bad`) to hold the final
|
||
variant of distance.ts/score.ts/distance.test.mjs. All commits verified green post-recovery.
|
||
|
||
## Auth Gates
|
||
|
||
None — the module reads no environment, no secrets, no network (D-04/D-11 trivially satisfied).
|
||
|
||
## Known Stubs
|
||
|
||
None — every export is wired to real data/behavior; no TODO/FIXME/placeholder anywhere in search/.
|
||
|
||
## Threat Flags
|
||
|
||
None — the built surface is exactly the plan's threat model (T-04-01…T-04-11): no new endpoints,
|
||
no auth paths, no storage, no env reads. T-04-01/02 (no logging/storage) are gated by G10;
|
||
T-04-04 (lazy loader) by G11; T-04-07 (honest refusal) by L1 + the SRCH probes.
|
||
|
||
## TDD Gate Compliance
|
||
|
||
PASSED — each task has a `test(...)` RED commit followed by a `feat(...)` GREEN commit:
|
||
`ebcd542→a9f9ab3`, `25ee1e9→6368bad`, `f1b31ca→8579da9`. RED phases were verified failing
|
||
(9, 16, and 13 failing cases respectively) before each GREEN.
|
||
|
||
## Self-Check: PASSED
|
||
|
||
- All 20 created files exist on disk (verified via `find search/ -type f` = 16 source/test files
|
||
+ LICENSE + LICENSES.md + the 2 root manifests).
|
||
- All 6 commit hashes present in `git log --grep="04-01"` (verified above).
|
||
- Suite green at 39/39, typecheck green, bot/web suites green.
|