feat(08-04): ALT preprocessing stage in the build tool — landmark tables through the ONE module

- The stage between edge compaction and encode feeds the graph's edges to
  landmark.ts (farthestPointAnchors + landmarkTables + landmarkCoverage) —
  the script quantizes and implements nothing itself (single-authority law)
- D-02 scaling rule L = clamp(round(N/13000), 16, 32) (~24 at KL's 311k
  nodes) with a --landmarks <n> measured knob (positive int, hard cap 128,
  parsed before --check)
- Honest gates before any write: quantum-ceiling throw -> fail with the
  raise-quantum hint (Pitfall 5); per-anchor coverage >= 0.9 (the SAME
  fraction for any L — the override cannot lower it, T-08-11); anchor
  shortfall fails loud instead of encoding a weaker ALT
- encodeGraph receives landmarks: { quantumS, nodes, to, from } — the one
  ARG2 authority; no local fallback encode
- Stats gate prints L/quantum/coverage min-mean/table bytes/sentinel totals
  BEFORE the size gates; --check mirrors the facts from the decoded binary
  (landmarkCount/quantum/anchors/coverage/sentinels, no network)
- A1 ladder history gains rung (c2): EXECUTED in this preprocessing stage
This commit is contained in:
araharah-executor
2026-10-25 19:10:53 +08:00
parent 3ec3584fbd
commit 698fdf4689
+173 -16
View File
@@ -18,15 +18,24 @@
* Pitfall 1: an unfiltered highway=* graph is the classic CPU-budget kill). * Pitfall 1: an unfiltered highway=* graph is the classic CPU-budget kill).
* 3. Ways → nodes (deduplicated by rounded coordinate) → directed edges with * 3. Ways → nodes (deduplicated by rounded coordinate) → directed edges with
* oneway flags, speed-derived seconds, and street-name pool ids. * oneway flags, speed-derived seconds, and street-name pool ids.
* 4. encodeGraph (../src/graph.ts — the layout's SINGLE authority) produces * 4. THE ALT PREPROCESSING STAGE (Phase 8, ladder rung c2): the compacted
* the pinned CSR binary. COLLECT-THEN-WRITE: routing/data/ * edges feed the ONE landmark module (../src/landmark.ts — this script
* kl-roadgraph.v1.bin is written ONLY after the encode and the stats gate * quantizes nothing and implements nothing itself): farthestPointAnchors
* both pass — any failure exits 1 with the previous binary untouched. * picks L = clamp(round(N/13000), 16, 32) road-network anchors (D-02
* 5. THE STATS GATE prints nodeCount/edgeCount/nameCount/bbox/binary size and * scaling rule; anchors are NOT POIs), landmarkTables runs the integer
* enforces the D-01 envelope: size ≤ MAX_BINARY_BYTES or exit 1. A result * ĉ-metric Dijkstras, and landmarkCoverage gates every anchor at
* below the expected 30–80 MB band prints an ADVISORY warning (A2's * ≥ LANDMARK_MIN_COVERAGE_FRACTION node reach — any gate failure exits 1
* forecast) — the ENFORCED floor is the 1 MB honesty check in --verify, * before a byte is written.
* never the forecast band. * 5. encodeGraph (../src/graph.ts — the layout's SINGLE authority) produces
* the pinned ARG2 CSR binary from the edge set + the landmark block.
* COLLECT-THEN-WRITE: routing/data/kl-roadgraph.v1.bin is written ONLY
* after the encode and the stats gate both pass — any failure exits 1
* with the previous binary untouched.
* 6. THE STATS GATE prints nodeCount/edgeCount/nameCount/bbox/landmark
* facts/binary size and enforces the D-01 envelope: size ≤
* MAX_BINARY_BYTES or exit 1. A result below the expected 30–80 MB band
* prints an ADVISORY warning (A2's forecast) — the ENFORCED floor is the
* 1 MB honesty check in --verify, never the forecast band.
* *
* The binary is a BUILD ARTIFACT, not committed: routing/data/.gitignore * The binary is a BUILD ARTIFACT, not committed: routing/data/.gitignore
* ignores *.bin — forks build + upload their own (ODbL-derived data stays out * ignores *.bin — forks build + upload their own (ODbL-derived data stays out
@@ -36,14 +45,19 @@
* *
* Usage: * Usage:
* node routing/scripts/build-road-graph.mjs * node routing/scripts/build-road-graph.mjs
* # full build: fetch → encode → gate → write * # full build: fetch → landmarks → encode → gate → write
* # (network fetch; minutes on the KL bbox) * # (network fetch; minutes on the KL bbox)
* node routing/scripts/build-road-graph.mjs --landmarks <n>
* # full build with an explicit landmark count — the D-02
* # MEASURED knob (raise L within/at the top of the rule for a
* # CPU-gate retry; hard-capped at 128, never a table bomb)
* node routing/scripts/build-road-graph.mjs --check <file> * node routing/scripts/build-road-graph.mjs --check <file>
* # decode an existing binary with decodeGraph, print stats, * # decode an existing binary with decodeGraph, print stats
* # exit 0/1 — NO network * # incl. the landmark facts, exit 0/1 — NO network
* *
* Exit codes: 0 built and gated (or checked green); 1 pre-flight failure, * Exit codes: 0 built and gated (or checked green); 1 pre-flight failure,
* fetch failure, envelope-gate failure, or --check decode failure. * fetch failure, landmark coverage/quantum-gate failure, envelope-gate
* failure, or --check decode failure.
*/ */
import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path"; import { dirname, resolve } from "node:path";
@@ -89,7 +103,10 @@ const COORD_ROUND_DECIMALS = 6;
* CPU gate is carried by ladder step (b) — bidirectional A* in * CPU gate is carried by ladder step (b) — bidirectional A* in
* routing/src/astar.ts — and the recorded verdict in the plan summary. * routing/src/astar.ts — and the recorded verdict in the plan summary.
* Ladder step (c) — Workers Paid — breaks the $0 law and requires user * Ladder step (c) — Workers Paid — breaks the $0 law and requires user
* sign-off; it is NOT applied in this script. */ * sign-off; it is NOT applied in this script.
* Ladder rung (c2) — EXECUTED in this script's preprocessing stage (Phase 8):
* build-time landmark tables + ALT steering; the verdict lives in
* measure-route-cpu.mjs's history and docs/ROAD-GRAPH.md. */
const ACTIVE_CLASSES = new Set([ const ACTIVE_CLASSES = new Set([
"motorway", "motorway",
"trunk", "trunk",
@@ -126,12 +143,48 @@ const LINK_CLASS_SPEED_KMH = {
}; };
/** ENFORCED floor — an at/below-1 MB "KL road graph" is a broken build (D-01). */ /** ENFORCED floor — an at/below-1 MB "KL road graph" is a broken build (D-01). */
const MIN_BINARY_BYTES = 1 * 1024 * 1024; const MIN_BINARY_BYTES = 1 * 1024 * 1024;
/** D-02 scaling rule: the landmark count is NODE-COUNT-PROPORTIONAL, never a
* fixed generous constant — per-landmark table bytes scale with N (~1.19
* MB/landmark at KL's 311k nodes), so over-provisioning explodes at Malaysia
* scale exactly where ALT's advantage itself narrows (that tier is
* Contraction Hierarchies' job, still deferred). L = clamp(round(N/13,000),
* 16, 32) → ~24 at KL scale; a future Malaysia-wide build re-derives the
* count from the same rule at its own scale. */
const LANDMARK_COUNT_MIN = 16;
const LANDMARK_COUNT_MAX = 32;
const NODES_PER_LANDMARK = 13000; // 311,325 / 13,000 ≈ 24 at KL scale
/** Build-time coverage gate: every anchor's ĉ-Dijkstra must reach at least
* this fraction of nodes (the measured KL giant SCC covers ~0.98 — the
* headroom is honest). The SAME fraction gates ANY --landmarks value: the
* override cannot buy a weaker gate (T-08-11). */
const LANDMARK_MIN_COVERAGE_FRACTION = 0.9;
/** --landmarks hard cap — the knob exists for measured retries within/at the
* top of the D-02 rule, not for unbounded table bombs (T-08-10). The 80 MB
* envelope gate fires before any write regardless. */
const LANDMARK_COUNT_OVERRIDE_MAX = 128;
function fail(message) { function fail(message) {
console.error(`build-road-graph: ${message}`); console.error(`build-road-graph: ${message}`);
process.exit(1); process.exit(1);
} }
/* ————— --landmarks override (the D-02 measured knob; parsed BEFORE --check) ————— */
const landmarksIdx = process.argv.indexOf("--landmarks");
let landmarksOverride = null;
if (landmarksIdx !== -1) {
const raw = process.argv[landmarksIdx + 1];
const n = raw === undefined ? NaN : Number(raw);
if (!Number.isInteger(n) || n < 1) {
fail(`--landmarks requires a positive integer count, got ${raw === undefined ? "(missing value)" : `"${raw}"`}`);
}
if (n > LANDMARK_COUNT_OVERRIDE_MAX) {
fail(`--landmarks ${n} exceeds the hard cap ${LANDMARK_COUNT_OVERRIDE_MAX} — the knob exists for measured ` +
`retries within/at the top of the D-02 rule, not for unbounded table bombs`);
}
landmarksOverride = n;
}
/* ————— --check mode: decode + stats, no network ————— */ /* ————— --check mode: decode + stats, no network ————— */
const checkIdx = process.argv.indexOf("--check"); const checkIdx = process.argv.indexOf("--check");
@@ -144,6 +197,20 @@ if (checkIdx !== -1) {
const graph = decodeGraph(buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength)); const graph = decodeGraph(buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength));
if (graph === null) fail(`--check: ${file} is NOT a valid GRAPH_VERSION CSR binary (decodeGraph null)`); if (graph === null) fail(`--check: ${file} is NOT a valid GRAPH_VERSION CSR binary (decodeGraph null)`);
const size = statSync(file).size; const size = statSync(file).size;
// ARG2 landmark facts (Phase 8): decoded straight off the binary — the
// SAME ONE module computes the coverage audit here; NO network.
const { LANDMARK_SENTINEL, landmarkCoverage } = await import("../src/landmark.ts");
let toSentinels = 0;
for (let i = 0; i < graph.landmarkTo.length; i++) {
if (graph.landmarkTo[i] === LANDMARK_SENTINEL) toSentinels += 1;
}
let fromSentinels = 0;
for (let i = 0; i < graph.landmarkFrom.length; i++) {
if (graph.landmarkFrom[i] === LANDMARK_SENTINEL) fromSentinels += 1;
}
const coverage = landmarkCoverage(graph.landmarkTo, graph.landmarkNodes, graph.nodeCount);
const minCoverage = Math.min(...coverage);
const meanCoverage = coverage.reduce((a, b) => a + b, 0) / coverage.length;
console.log( console.log(
`--check ${file}\n` + `--check ${file}\n` +
` nodeCount: ${graph.nodeCount}\n` + ` nodeCount: ${graph.nodeCount}\n` +
@@ -151,7 +218,12 @@ if (checkIdx !== -1) {
` nameCount: ${graph.nameCount}\n` + ` nameCount: ${graph.nameCount}\n` +
` maxSpeedKmh: ${graph.maxSpeedKmh}\n` + ` maxSpeedKmh: ${graph.maxSpeedKmh}\n` +
` bbox: [${graph.bbox.minLat}, ${graph.bbox.minLng}, ${graph.bbox.maxLat}, ${graph.bbox.maxLng}]\n` + ` bbox: [${graph.bbox.minLat}, ${graph.bbox.minLng}, ${graph.bbox.maxLat}, ${graph.bbox.maxLng}]\n` +
` bytes: ${size}`, ` bytes: ${size}\n` +
` landmarkCount: ${graph.landmarkCount}\n` +
` landmarkQuantumS: ${graph.landmarkQuantumS.toFixed(3)}\n` +
` landmarkNodes: ${[...graph.landmarkNodes].join(", ")}\n` +
` landmark coverage min/mean: ${minCoverage.toFixed(3)} / ${meanCoverage.toFixed(3)}\n` +
` landmark sentinel cells: ${toSentinels} to / ${fromSentinels} from`,
); );
if (size > MAX_BINARY_BYTES) fail(`--check: ${size} bytes exceeds the ${MAX_BINARY_BYTES}-byte D-01 envelope`); if (size > MAX_BINARY_BYTES) fail(`--check: ${size} bytes exceeds the ${MAX_BINARY_BYTES}-byte D-01 envelope`);
if (size <= MIN_BINARY_BYTES) fail(`--check: ${size} bytes is at/below the ${MIN_BINARY_BYTES}-byte floor — not a real road graph`); if (size <= MIN_BINARY_BYTES) fail(`--check: ${size} bytes is at/below the ${MIN_BINARY_BYTES}-byte floor — not a real road graph`);
@@ -330,10 +402,83 @@ const edges = rawEdges
const droppedNodes = latOut.length - cLat.length; const droppedNodes = latOut.length - cLat.length;
if (cLat.length === 0 || edges.length === 0) fail("the extract produced an empty graph — refusing to write"); if (cLat.length === 0 || edges.length === 0) fail("the extract produced an empty graph — refusing to write");
/* ————— ALT preprocessing stage (Phase 8 rung c2): landmark tables through the ONE module —————
*
* This script quantizes nothing and implements nothing itself: the graph's
* edges feed ../src/landmark.ts (farthestPointAnchors + landmarkTables +
* landmarkCoverage) — there is no second integer-Dijkstra implementation here
* (single-authority law). Gates run BEFORE any write:
* - quantum ceiling: a landmarkTables throw means the ĉ metric overflowed
* the u16 domain → raise LANDMARK_QUANTUM_S in landmark.ts, never clamp
* (Pitfall 5);
* - coverage: every anchor reaches ≥ LANDMARK_MIN_COVERAGE_FRACTION of the
* graph — a fragmented extract fails honestly, it is never forced through.
*/
const { LANDMARK_QUANTUM_S, LANDMARK_SENTINEL, farthestPointAnchors, landmarkTables, landmarkCoverage } =
await import("../src/landmark.ts");
// D-02 scaling rule: the count derives from THIS graph's node count; a
// --landmarks override (validated at argv time) replaces the rule wholesale.
const ruleRaw = Math.round(cLat.length / NODES_PER_LANDMARK);
const ruleCount = Math.min(LANDMARK_COUNT_MAX, Math.max(LANDMARK_COUNT_MIN, ruleRaw));
const landmarkCount = landmarksOverride ?? ruleCount;
console.log(
`landmarks: rule round(${cLat.length} / ${NODES_PER_LANDMARK}) = ${ruleRaw} → clamped ${ruleCount}` +
(landmarksOverride !== null ? `; --landmarks override → L = ${landmarkCount}` : `; L = ${landmarkCount}`),
);
// The module's input shape — projected ONCE (no mutation of the encode-side
// edge array; encodeGraph re-groups edges itself).
const lmEdges = edges.map((e) => ({ from: e.from, to: e.to, seconds: e.seconds, flags: e.flags }));
const lmInput = { nodeCount: cLat.length, edges: lmEdges, quantumS: LANDMARK_QUANTUM_S };
const anchors = farthestPointAnchors(lmInput, landmarkCount);
if (anchors.length === 0) {
fail("farthestPointAnchors found no degree>=2 candidate — the extract has no junction-ish node (degenerate/fragmented graph)");
}
if (anchors.length < landmarkCount) {
fail(`farthestPointAnchors returned only ${anchors.length} of ${landmarkCount} requested anchors (candidates exhausted) — ` +
`refusing to encode a weaker ALT than the rule asked for; investigate the extract before rebuilding`);
}
let to;
let from;
try {
({ to, from } = landmarkTables(lmInput, anchors));
} catch (err) {
fail(`landmark tables exceeded the u16 quantum ceiling (${err.message}) — raise LANDMARK_QUANTUM_S in ` +
`routing/src/landmark.ts and rebuild (Pitfall 5: raise the quantum, never clamp)`);
}
// Coverage gate (research Pattern 3, layer 1): the SAME fraction for ANY L —
// the override cannot lower it (T-08-11).
const coverage = landmarkCoverage(to, anchors, cLat.length);
const minCoverage = Math.min(...coverage);
if (minCoverage < LANDMARK_MIN_COVERAGE_FRACTION) {
const worst = coverage.indexOf(minCoverage);
fail(
`landmark coverage gate FAILED at L=${landmarkCount}: anchor ${worst} (node ${anchors[worst]}) reaches only ` +
`${(minCoverage * 100).toFixed(1)}% of nodes (gate: ≥ ${(LANDMARK_MIN_COVERAGE_FRACTION * 100).toFixed(0)}%). ` +
`The KL extract is expected to be ONE giant SCC covering ~98% of nodes (2026-10-24 build fact) — ` +
`a miss this large means a fragmented extract; investigate the fetch, do NOT lower the gate.`,
);
}
const meanCoverage = coverage.reduce((a, b) => a + b, 0) / coverage.length;
// Sentinel audit — cheap honesty; --check mirrors it from the written bytes.
let toSentinels = 0;
for (let i = 0; i < to.length; i++) if (to[i] === LANDMARK_SENTINEL) toSentinels += 1;
let fromSentinels = 0;
for (let i = 0; i < from.length; i++) if (from[i] === LANDMARK_SENTINEL) fromSentinels += 1;
/* ————— encode (the layout's single authority) + stats gate ————— */ /* ————— encode (the layout's single authority) + stats gate ————— */
const names = [...nameIds.keys()]; const names = [...nameIds.keys()];
const buffer = encodeGraph({ lat: cLat, lng: cLng, edges, names, maxSpeedKmh: maxSeenKmh }); const buffer = encodeGraph({
lat: cLat,
lng: cLng,
edges,
names,
maxSpeedKmh: maxSeenKmh,
landmarks: { quantumS: LANDMARK_QUANTUM_S, nodes: anchors, to, from },
});
const bytes = buffer.byteLength; const bytes = buffer.byteLength;
const mb = (b) => `${(b / (1024 * 1024)).toFixed(1)} MB`; const mb = (b) => `${(b / (1024 * 1024)).toFixed(1)} MB`;
@@ -348,6 +493,18 @@ console.log(
` binary size: ${mb(bytes)} (${bytes} B)`, ` binary size: ${mb(bytes)} (${bytes} B)`,
); );
// The landmark facts print BEFORE the size gates — the gate numbers exist
// outside the terminal even when a gate then refuses the write.
const tableBytes = 2 * landmarkCount * cLat.length * 2 + 4 * landmarkCount;
console.log(
`landmark stats (ALT preprocessing through the ONE module):\n` +
` L: ${landmarkCount} (D-02 rule ${ruleCount}${landmarksOverride !== null ? `, --landmarks override ${landmarksOverride}` : ""})\n` +
` quantum: ${LANDMARK_QUANTUM_S} s (the ĉ unit)\n` +
` coverage min/mean: ${minCoverage.toFixed(3)} / ${meanCoverage.toFixed(3)} (gate ≥ ${LANDMARK_MIN_COVERAGE_FRACTION})\n` +
` table bytes: ${tableBytes} (${mb(tableBytes)} = 2 families × L·N u16 + L u32 anchors)\n` +
` sentinel cells: ${toSentinels} to / ${fromSentinels} from`,
);
// THE ENFORCED GATE (D-01): outside the 1-80 MB envelope → exit 1 WITHOUT writing. // THE ENFORCED GATE (D-01): outside the 1-80 MB envelope → exit 1 WITHOUT writing.
if (bytes > MAX_BINARY_BYTES) { if (bytes > MAX_BINARY_BYTES) {
fail(`binary is ${mb(bytes)} — over the ${mb(MAX_BINARY_BYTES)} D-01 envelope. ` + fail(`binary is ${mb(bytes)} — over the ${mb(MAX_BINARY_BYTES)} D-01 envelope. ` +