Files

24 KiB
Raw Permalink Blame History

Phase 8: ALT Landmark Preprocessing (Routing Backend v2) - Pattern Map

Mapped: 2026-10-25 Files analyzed: 13 (11 modified, 2 new) Analogs found: 13 / 13 (all new/modified files have a strong in-repo analog; the baseline recorder is new but has two same-role templates)

File Classification

New/Modified File Role Data Flow Closest Analog Match Quality
routing/src/graph.ts (mod) model / codec transform (binary ⇄ typed-array views) itself (ARG1 version) — the phase extends it in place exact
routing/src/astar.ts (mod) service (pure compute) request-response (pure fn, no I/O) itself (BH/Ikeda version) — potential() internals swap only exact
routing/scripts/build-road-graph.mjs (mod) build script (maintainer-time) batch / file-I/O + network fetch itself — gains a preprocessing stage between edge assembly and encode exact
routing/scripts/measure-route-cpu.mjs (mod) measurement gate script batch (in-process timing) itself — gains extended-sample flag; scripts/lib/budget-gates.mjs is its CLI contract consumer exact
routing/scripts/route-corpus.mjs (mod or new sibling gate) measurement gate script batch / file-I/O (committed fixtures) itself — measurePairs() exported-core + CLI-guard pattern is the template for the new D-03 optimality gate exact
NEW routing/scripts/<baseline-recorder>.mjs + committed record measurement script + committed artifact file-I/O (JSON record) route-corpus.mjs (exported-core/CLI-guard) + measure-route-cpu.mjs (seeded sampling, printed measurements) role-match
routing/test/graph.test.mjs (mod) test (unit/codec gate) batch itself (G1–G8 suite), routing/test/harness.mjs + runner.mjs exact
routing/test/astar.test.mjs (mod) test (unit/optimality gate) batch itself (A1–A8 suite; A1 625-pair sweep + A8 structural pin) exact
routing/test/fixtures/build-mini-graph.mjs (mod) fixture-builder script file-I/O (writes committed .bin) itself — gains the landmark stage via encodeGraph exact
bot/src/routing/worker-router.ts (io-only check) controller (Worker handler) request-response itself — loadGraph → decodeGraph must stay untouched at its API exact
scripts/lib/budget-gates.mjs (compat constraint, likely no edit) gate consumer subprocess batch itself — runCpuLadderGate() executes measure-route-cpu.mjs with NO args and parses its stdout exact
docs/KNOWN-LIMITATIONS.md (mod) docs — itself — c3 section + roadmap-pointer table; the /cari fixed-record is the in-doc precedent exact
docs/ROAD-GRAPH.md (mod) docs (fork contract) — itself — envelope-law table + CPU-gate ladder table gain the ARG2 rung exact

Note on "itself as analog": this is a version-bump/extension phase — the strongest analog for every file is the file's own current code, which the phase extends or re-pins. Where behavior is genuinely NEW (baseline recorder, extended-sample flag, ARG2 section), the analog is the nearest same-role script listed above.

Pattern Assignments

routing/src/graph.ts (codec, transform — ARG1 → ARG2)

Analog: itself (routing/src/graph.ts, 299 lines). Excerpts below are the laws ARG2 must extend mechanically.

Layout arithmetic (layoutFor, lines 108–119) — ARG2's usable change: bump HEADER_SIZE to 72 (72 % 8 == 0 keeps f64 views aligned), add landmark section offsets, extend total:

const HEADER_SIZE = 64; // → 72 for ARG2; the pad keeps every f64 view 8-byte aligned
function layoutFor(nodeCount, edgeCount, nameCount, namePoolLen): Layout {
  const lat = HEADER_SIZE;
  ...
  return { lat, ..., total: namePool + namePoolLen };
}

Fail-honest decode (lines 125–144) — version/magic mismatch returns null; the exact-size check runs BEFORE any view allocation:

export function decodeGraph(buffer: ArrayBuffer): RoadGraph | null {
  if (buffer.byteLength < HEADER_SIZE) return null;
  const view = new DataView(buffer);
  if (view.getUint32(0, true) !== GRAPH_MAGIC) return null;
  if (view.getUint32(4, true) !== GRAPH_VERSION) return null;
  ...
  const lay = layoutFor(nodeCount, edgeCount, nameCount, namePoolLen);
  if (buffer.byteLength !== lay.total) return null;
  const lat = new Float64Array(buffer, lay.lat, nodeCount); // zero-copy views

Encode-side loud-validate (lines 180–222) — the fail() + field validation shape; GraphInput gains landmark input and the same validation joins these guards (research anti-pattern: NO second encodeGraph):

function fail(message: string): never {
  throw new Error(`encodeGraph: ${message}`);
}
// per-field guards fail(`edge ${edge.from}->${edge.to} has a non-finite/negative weight`) etc.
const lay = layoutFor(...);
const buffer = new ArrayBuffer(lay.total);
const view = new DataView(buffer);

Constants to re-pin (lines 37–48): GRAPH_MAGIC = 0x41524731 → add/keep as rejection data, 0x41524732 ("ARG2") for v2; GRAPH_VERSION = 1 → 2.

Header write block (lines 253–275): little-endian view.setUint32(...) field-by-field in offset order — the pattern for writing landmarkCount @56 (was pad) and landmarkQuantumS f32 @60.

routing/src/astar.ts (pure search, request-response — ALT potentials)

Analog: itself (routing/src/astar.ts, 470 lines). Only potential() (lines 232–251) changes; everything else is pinned.

The lazy-potential mechanism (lines 232–251) — ALT slots into this exact shape (Float64Array caches NaN-init, allocation-free):

const hfCache = new Float64Array(n).fill(Number.NaN);
const hbCache = new Float64Array(n).fill(Number.NaN);
const pf = new Float64Array(n);
function potential(v: number): number {
  let hf = hfCache[v]!;
  if (Number.isNaN(hf)) { hf = metersTo(graph, v, goalLat, goalLng) * heuristicScale; hfCache[v] = hf; }
  ... const value = (hf - hb) / 2; pf[v] = value; return value;
}

The SURVEY combination p_f(v) = (h_f − h_b)/2, p_b = −p_f (lines 199–218 doc-comment) carries over verbatim — only the h_f/h_b bodies swap to ALT triangle bounds (max over valid ℓ of to/from-table families, haversine fallback).

Per-search constant init precedent (lines 227–235, 263–268): goal/origin coords hoisted once per search; df[df[originNode] = 0; potential(originNode); ...] pattern is where the L-length landmark-constant tables (goalTo[ℓ] etc.) get initialized.

Per-graph memoization precedent (reverseFor, lines 146–192) — if landmark-derived search constants ever need memoizing, this is the pattern (module-scope WeakMap<RoadGraph, ...>):

Laws to preserve (pinned by tests): pure module (line 31), allocation-free hot loop — A8's structural test (astar.test.mjs lines 277–283) greps for new Map( after stripping comments; real-seconds recompute along the node path (lines 352–364) is where tie-path noise must not leak into returned seconds.

routing/scripts/build-road-graph.mjs (build script, batch — preprocessing stage)

Analog: itself (routing/scripts/build-road-graph.mjs, 375 lines). The new preprocessing stage joins between the edge-compaction block (lines 310–331) and the encode block (line 336).

Structural pattern of the tool (everything the stage must fit):

  • Named constants at top (lines 58–128) with rationale doc-comments — e.g. MAX_BINARY_BYTES, EXPECTED_MIN_BYTES, BBOX_MARGIN_DEG. ALT constants (QUANTUM_S, landmark-count formula, LANDMARK_MIN_COVERAGE_FRACTION) join here. Copy the ladder-history comment block style (lines 80–92).
  • function fail(message) { console.error(...); process.exit(1); } (lines 130–133) — the fail-honest exit for: coverage assertion failure, quantum-overflow assert (≤ 65,000 units), empty-graph refusal.
  • Print-every-stat + gate-then-write block (lines 339–373) — "collect-then-write: the binary is written ONLY after the encode and the stats gate both pass":
const buffer = encodeGraph({ lat: cLat, lng: cLng, edges, names, maxSpeedKmh: maxSeenKmh });
if (bytes > MAX_BINARY_BYTES) { fail(...) }
if (bytes <= MIN_BINARY_BYTES) { fail(...) }
if (bytes < EXPECTED_MIN_BYTES) { console.log("WARNING: ..."); }
mkdirSync(dirname(OUT_PATH), { recursive: true });
writeFileSync(OUT_PATH, Buffer.from(buffer));

The coverage/quantum/byte-budget gates go exactly here; --check mode (lines 137–159) prints node/edge stats via decodeGraph and must gain landmarkCount + coverage prints.

  • Edge array it consumes: edges = rawEdges.map(...) (line 327–329) with {from, to, seconds, flags, nameId} — the preprocessing stage derives ĉ(u,v) = Math.floor(seconds / QUANTUM_S) from these, never re-reading Overpass.

routing/scripts/measure-route-cpu.mjs (measurement gate — extended-sample flag)

Analog: itself (routing/scripts/measure-route-cpu.mjs, 185 lines), the gate this phase must pass (p95 ≤ P95_BUDGET_MS = 8).

Seed-sampling structure (lines 86–132) — the seeded-PRNG + labeled-pairs pattern; the D-04 extended sample re-uses this mechanic with a bigger N and a MIN_PAIR_KM-style filter:

function mulberry32(seed) { ... }           // deterministic, comparable re-runs
const rand = mulberry32(SAMPLE_SEED);       // SAMPLE_SEED = 0x06d4, named const
pairs.push({ label: `quadrant q${a}->q${b}`, from, to });

Pitfall note from RESEARCH (Pitfall 2): quadrant sampling is INDEX-keyed — fine for the CI-size CPU snapshot, NOT for cross-version optimality equality. The extended sample and baseline record must pin pairs by lat/lng (e.g. derive coordinates from bbox + seeded PRNG, snap at run time via snapToGraph).

Timing + gate structure (lines 136–185) — the print-the-measurement law, verbatim shape for the extended verdict:

function cpuMs(fn) { const t0 = process.hrtime.bigint(); const result = fn(); ... return { ms, result }; }
samples.sort((a, b) => a - b);
const p95 = samples[idx];
console.log(`p95: ${p95.toFixed(3)} ms (max ..., mean ..., n=${samples.length})`);
if (p95 > P95_BUDGET_MS) { console.error(`GATE FAIL: ...`); process.exit(1); }
console.log(`GATE PASS: p95 ...`);

HARD CLI-COMPAT CONSTRAINT — scripts/lib/budget-gates.mjs runCpuLadderGate() (lines 78–119) runs this script with zero arguments as a child process and parses its stdout:

out = execFileSync(process.execPath, [script], { cwd: ..., encoding: "utf8", ... });
const p95 = Number(out.match(/^p95: (\d+(?:\.\d+)?) ms/m)?.[1] ?? NaN);
const childGateLine = out.match(/^GATE (?:FAIL|PASS): .*$/m)?.[0] ?? null;

The extended sample MUST be an opt-in flag (--extended or similar); the default run must keep printing the same anchor lines (p95: <n> ms (max..., GATE PASS:|GATE FAIL: ...) so budget-audit (Phase 7 record) keeps passing unchanged. The extended run's own verdict lines can be additional, not mutations.

routing/scripts/route-corpus.mjs (corpus gate — the D-03 optimality-gate template)

Analog: itself (routing/scripts/route-corpus.mjs, 334 lines). A NEW D-03 optimality-gate script (either an edit here or a sibling like route-optimality.mjs) copies this file's architecture:

Exported measurement core + CLI guard (lines 68–73, 329–334) — one measurement, two reporters, no drift; test suites can assert the same numbers in-process:

export function measurePairs(corpus, fixtures, gazetteer, fixturesPath = DEFAULT_FIXTURES_PATH) { ... }
const invoked = process.argv[1] ? pathToFileURL(resolve(process.argv[1])).href : "";
if (invoked === import.meta.url) { main(); }

Check-registry + violations pattern (lines 74–92, 283–314):

const checks = {}; const check = (id, label) => { checks[id] = { id, label, violations: [] }; };
const violation = (id, pairId, reason) => { checks[id].violations.push({ pairId, reason }); };
...
const pass = allChecks.every((c) => c.violations.length === 0);

D-03's two checks map directly onto this: ("D03A seconds-equality ± named epsilon", "D03B verdict-class preserved") with per-pairId violations, and per the decision a tie-path difference must NOT emit a violation.

Named-constant bands (lines 41–53): LENGTH_RATIO_BAND = [0.85, 1.10] style — the epsilon is a named constant here (per the phase-6 constant law), not a magic number.

NEW: baseline recorder script + committed baseline record (Wave 0)

Analog (role-match): route-corpus.mjs for script architecture (exported core + CLI guard, committed-readFileSync inputs, header contract doc-comment) + measure-route-cpu.mjs for the run mechanics (decodeGraph → snapToGraph → routeAstar over the REAL binary, toArrayBuffer() helper at lines 75–77, printed measurements, exit codes).

Record format guidance (from RESEARCH Pitfall 1): committed JSON; each entry = pair id + origin/destination coordinates (lat/lng, never node indices) + route seconds (JSON number, f64-safe) + verdict class (reachable/unreachable) at recording time. Wave-0 task: recorded BEFORE any graph.ts version change, because ARG2-only decode (D-01) makes old-code+old-binary unreproducible from the working tree afterward.

routing/test/graph.test.mjs (codec tests — new ARG2 G-cases)

Analog: itself (routing/test/graph.test.mjs, 191 lines). Copy-cases for the new suite:

Case harness pattern (lazy-import RED compatibility, line 43–47; per-case named IDs, doc-comment case index at lines 17–31):

let graphPromise = null;
function loadGraph() { if (!graphPromise) graphPromise = import("../src/graph.ts"); return graphPromise; }
test("G2 bad magic -> null — a tampered header never half-decodes (T-06-03)", async () => { ... });

Tamper-vector pattern (lines 98–133) — the exact mechanics for ARG2's rejection matrix (version≠2, landmarkCount=0, oversized landmarkCount on a tiny buffer):

const tampered = buf.slice(0);
new DataView(tampered).setUint32(0, 0xdeadbeef, true);
assert.equal(decodeGraph(tampered), null, "wrong magic -> null, never a throw");
// hostile header: valid magic/version but huge count on a tiny buffer -> null "before any allocation"

Fixture-decode pin pattern (G7, lines 159–184) — re-pin for the committed ARG2 fixture: counts, bbox, one oneway flag, both speed classes. G8 (lines 186–191) re-pins GRAPH_MAGIC/GRAPH_VERSION exported constants → must expect 0x41524732/2 and gain the ARG1 rejection fixture (fixtures/mini-graph-arg1.bin, bytes preserved) as a new decode-null case (G3 merges into it — "version !== 2 refuses").

routing/test/astar.test.mjs (A-suite — A1 re-run + new island case)

Analog: itself (routing/test/astar.test.mjs, 283 lines). The load-bearing pattern for ALT's optimality seal:

The exhaustive Dijkstra-equivalence sweep (A1, lines 95–122) — the test-local brute-force Dijkstra (lines 69–91) with the SAME traversal law (edgeFlags bit0 never relaxed); every pair asserts cost equality within 1e-6, endpoints, and honest nulls. The ARG2 re-run with ALT potentials live must stay EXACTLY this shape: npm run routing:test, 625 pairs on the regenerated fixture. Synthetic disconnected graph pattern (A3, lines 147–171) — encodeGraph a hand-made 2-component graph inline; the new island case (ALT active + islanded fixture) copies this build-synthetic-vector mechanic and asserts routeAstar = Dijkstra on every pair touching the island. Structural pin pattern (A8, lines 277–283) — source-grep with comment stripping:

const src = readFileSync(path.join(here, "..", "src", "astar.ts"), "utf8");
const code = src.replace(/\/\*[\s\S]*?\*\//g, " ").replace(/(^|[^:\\])\/\/[^\n]*/g, "$1");
assert.ok(!code.includes("new Map("), "no Map allocation — the hot loop is typed arrays");

→ the new upload-unchanged structural pin (upload-road-graph.mjs byte-for-byte unchanged) copies THIS grep-shape, not RP-style subprocess git diffs.

Runner/registry context (routing/test/harness.mjs 10 lines + runner.mjs 48 lines): zero-dep auto-discovering runner; new case files named *.test.mjs in routing/test/ join automatically; run via npm run routing:test.

routing/test/fixtures/build-mini-graph.mjs (fixture builder — landmark stage)

Analog: itself (routing/test/fixtures/build-mini-graph.mjs, 85 lines). Pattern to extend:

const bin = encodeGraph({ lat, lng, edges, names, maxSpeedKmh: SPEED_PERIMETER_KMH }); // line 83
writeFileSync(new URL("./mini-graph.bin", import.meta.url), Buffer.from(bin));          // line 84

The landmark stage: tiny L for the fixture, computed in-script (farthest-point over the 5×5 grid or hand-picked anchors), passed into the extended encodeGraph/GraphInput; fail-loud invariant check before write (lines 78–81 style: if (edges.length !== 80) throw). Research requirement: old committed ARG1 bytes preserved first as fixtures/mini-graph-arg1.bin (the decode-rejection vector) BEFORE this script regenerates mini-graph.bin as ARG2.

bot/src/routing/worker-router.ts (io-only check — consumer unchanged)

Analog: itself (bot/src/routing/worker-router.ts, 200 lines). Planner note: NO code change here. The consumption seam that must stay untouched:

export const GRAPH_KEY = "kl-roadgraph.v1.bin";       // line 55 — name unchanged (D-01 lock)
memoized = env.ROAD_GRAPH.get(GRAPH_KEY)
  .then(async (obj) => (obj === null ? null : decodeGraph(await obj.arrayBuffer())))
  .catch(() => null);                                 // lines 72–74 — ONE decode path

ARG2's decode-path change (v1 → null) flows through this EXACT line — the new-graph → routeAstar → deriveSteps contract at lines 178–199 needs no edit. The io-only verification (_graph decodes at the same API) lands in a routing-bot suite pin case at most.

docs/KNOWN-LIMITATIONS.md (D-05 retirement record)

Analog: itself (docs/KNOWN-LIMITATIONS.md, 104 lines). Section to replace: "## Cross-metro routes may not resolve" (lines 29–41) — the c3 class paragraph, ending "The committed upgrade lever is ALT landmark preprocessing... it is a next-milestone item and is not in this release." In-doc precedent for the retirement wording: the /cari menu section (lines 43–57) — its structure is exactly the honest-fixed-record shape D-05 wants: ROOT CAUSE / what changed / verified date / what REMAINS. Copy: FIXED <date>: <what changed — ALT preprocessing>; measured <numbers from the D-04 run>; <remaining honest caveat: tie-path differences can appear, ARG1 binaries are router-pending until one re-upload>. Roadmap-pointer table to flip (lines 77–81, column "Honest status"):

| ALT landmark preprocessing | The cross-metro CPU degradation ... | Next-milestone item, not built |

→ status becomes built (Phase 8, with the measured p95). Keep the "none in this release" framing adjusted accordingly.

docs/ROAD-GRAPH.md (fork contract — ARG2 layout + gate rung)

Analog: itself (docs/ROAD-GRAPH.md, 130 lines). CPU-gate section to extend: "## The CPU gate (A1)" (lines 77–106) — the ladder table (lines 93–98) gains the (c2) rung with measured numbers, and the decision paragraph (lines 100–104) flips c2 to built/decided. Note the doc's path/knobs table (lines 119–130) listing routing/data/kl-roadgraph.v1.bin and "pinned CSR v1" — the ARG2 layout section documents: header 72 B, landmarkQuantumS knob + its scaling formula (Pitfall 5: 0.1 s ⇒ ≈6,553 s expressible ceiling; fork-raising instructions), the u16 table suffix, and the emitted warning that the STORAGE NAME keeps "v1" while the header version is the format truth (RESEARCH anti-pattern + docs-divorce Pitfall 6). Also note scripts/lib/source-gates.mjs cites measure-route-cpu.mjs lines 3–9 verbatim in a GPL notice — flag if the script's header line numbers drift.

docs/CONFIGURATION.md (check-touch only)

Line 24 and line 294 reference the build/upload pipeline by artifact path (name-only — no format claim); line 248 lists measure-route-cpu.mjs among maintainer scripts. Only touch if any statement becomes false after ARG2 (path and script names are unchanged; likely a no-op — verify, don't preemptively edit).

Shared Patterns

1. Named-constant laws (no inline magic)

Source: build-road-graph.mjs lines 56–128; measure-route-cpu.mjs lines 59–64 (P95_BUDGET_MS = 8, SAMPLE_SEED = 0x06d4); route-corpus.mjs lines 41–53. Apply to: every file this phase touches — epsilon (~±0.1%), QUANTUM_S = 0.1, LANDMARK_MIN_COVERAGE_FRACTION, landmark-count formula constants, MIN_PAIR_KM all get named, doc-commented constants with measured justification.

2. Fail-honest discrimination: null (decode/runtime) vs throw+exit-1 (build)

Source: graph.ts lines 120–144 (decode → null, never throws); graph.ts lines 180–182 + build-road-graph.mjs fail() (encode/build time → loud exit 1). Apply to: decodeGraph ARG2 laws (version/size/landmarkCount → null), build-stage asserts (coverage, quantum overflow) → exit 1, encodeGraph input validation → throw.

3. Print-the-measurement gates (verdict never assumed)

Source: measure-route-cpu.mjs lines 173–185; route-corpus.mjs report() lines 296–314; build-road-graph.mjs stats block lines 339–367. Apply to: the build stats gate (prints L, per-anchor coverage, quantum audit, byte budget), the CPU gate + extended verdict, the D-03 gate (prints per-pair equality/verdict-class and the epsilon used).

4. Collect-then-write / deterministic regeneration

Source: build-road-graph.mjs lines 22–24 + 369–373 ("written ONLY after the encode and the stats gate both pass"); build-mini-graph.mjs (deterministic rebuild through encodeGraph). Apply to: ARG2 build path and fixture regeneration — nothing overwrites a committed/good binary after a failed gate; the baseline record is written only after all pairs measure.

5. Hand-rolled TDD harness (zero-dep)

Source: routing/test/harness.mjs (10-line registry) + runner.mjs (auto-discovery); case-ID doc-index headers (G1–G8 at graph.test.mjs lines 17–31); lazy-import RED pattern for new modules. Apply to: all new G/A cases, the island case, the upload-unchanged pin — npm run routing:test must stay the single fast per-commit command.

6. CLI-contract compatibility for downstream parsres

Source: scripts/lib/budget-gates.mjs lines 78–119 — re-runs measure-route-cpu.mjs with no args, regex-parses ^p95: (\d+…) ms and GATE (?:FAIL|PASS):; Apply to: measure-route-cpu.mjs — extended sample behind an opt-in flag, default stdout byte-compatible (routing/data/kl-roadgraph.v1.bin path read also stays for the default mode... note budget-gates hard-codes that path at line 81; renaming the local artifact is prohibited anyway).

7. Allocation-free typed-array discipline + memo-once

Source: astar.ts potential/heap (no Map — enforced by A8), reverseFor's WeakMap memo (built once per graph, amortized per isolate), worker-router.ts graphMemo WeakMap. Apply to: the ALT potential internals (L-length constant Float64Arrays at search start, NaN skipping, sentinel-0xFFFF handled as integer BEFORE any float conversion) and any landmark-cache reuse.

No Analog Found

File Role Data Flow Reason Fallback
(none — every file maps) The two genuinely new artifacts (baseline record JSON; extended-sample flag output) are format-new but mechanism-covered: JSON-records precedent = route-corpus.mjs committed-JSON inputs; flag mechanics = trivial argv read like build-road-graph.mjs --check (lines 137–141). —

Metadata

Analog search scope: routing/src/, routing/scripts/, routing/test/, routing/test/fixtures/, bot/src/routing/, bot/test/ (spot), scripts/lib/ (budget-gates contract), docs/ Files read in full: graph.ts, astar.ts, build-road-graph.mjs, measure-route-cpu.mjs, route-corpus.mjs, upload-road-graph.mjs, graph.test.mjs, astar.test.mjs, build-mini-graph.mjs, worker-router.ts, KNOWN-LIMITATIONS.md, ROAD-GRAPH.md, budget-gates.mjs (targeted), harness.mjs, runner.mjs, route-privacy.test.mjs (partial, RP-pin style) Pattern extraction date: 2026-10-25