feat(08-04): ALT preprocessing stage in the build tool — landmark tables through the ONE module
- The stage between edge compaction and encode feeds the graph's edges to
landmark.ts (farthestPointAnchors + landmarkTables + landmarkCoverage) —
the script quantizes and implements nothing itself (single-authority law)
- D-02 scaling rule L = clamp(round(N/13000), 16, 32) (~24 at KL's 311k
nodes) with a --landmarks <n> measured knob (positive int, hard cap 128,
parsed before --check)
- Honest gates before any write: quantum-ceiling throw -> fail with the
raise-quantum hint (Pitfall 5); per-anchor coverage >= 0.9 (the SAME
fraction for any L — the override cannot lower it, T-08-11); anchor
shortfall fails loud instead of encoding a weaker ALT
- encodeGraph receives landmarks: { quantumS, nodes, to, from } — the one
ARG2 authority; no local fallback encode
- Stats gate prints L/quantum/coverage min-mean/table bytes/sentinel totals
BEFORE the size gates; --check mirrors the facts from the decoded binary
(landmarkCount/quantum/anchors/coverage/sentinels, no network)
- A1 ladder history gains rung (c2): EXECUTED in this preprocessing stage
This commit is contained in:
@@ -18,15 +18,24 @@
|
|||||||
* Pitfall 1: an unfiltered highway=* graph is the classic CPU-budget kill).
|
* Pitfall 1: an unfiltered highway=* graph is the classic CPU-budget kill).
|
||||||
* 3. Ways → nodes (deduplicated by rounded coordinate) → directed edges with
|
* 3. Ways → nodes (deduplicated by rounded coordinate) → directed edges with
|
||||||
* oneway flags, speed-derived seconds, and street-name pool ids.
|
* oneway flags, speed-derived seconds, and street-name pool ids.
|
||||||
* 4. encodeGraph (../src/graph.ts — the layout's SINGLE authority) produces
|
* 4. THE ALT PREPROCESSING STAGE (Phase 8, ladder rung c2): the compacted
|
||||||
* the pinned CSR binary. COLLECT-THEN-WRITE: routing/data/
|
* edges feed the ONE landmark module (../src/landmark.ts — this script
|
||||||
* kl-roadgraph.v1.bin is written ONLY after the encode and the stats gate
|
* quantizes nothing and implements nothing itself): farthestPointAnchors
|
||||||
* both pass — any failure exits 1 with the previous binary untouched.
|
* picks L = clamp(round(N/13000), 16, 32) road-network anchors (D-02
|
||||||
* 5. THE STATS GATE prints nodeCount/edgeCount/nameCount/bbox/binary size and
|
* scaling rule; anchors are NOT POIs), landmarkTables runs the integer
|
||||||
* enforces the D-01 envelope: size ≤ MAX_BINARY_BYTES or exit 1. A result
|
* ĉ-metric Dijkstras, and landmarkCoverage gates every anchor at
|
||||||
* below the expected 30–80 MB band prints an ADVISORY warning (A2's
|
* ≥ LANDMARK_MIN_COVERAGE_FRACTION node reach — any gate failure exits 1
|
||||||
* forecast) — the ENFORCED floor is the 1 MB honesty check in --verify,
|
* before a byte is written.
|
||||||
* never the forecast band.
|
* 5. encodeGraph (../src/graph.ts — the layout's SINGLE authority) produces
|
||||||
|
* the pinned ARG2 CSR binary from the edge set + the landmark block.
|
||||||
|
* COLLECT-THEN-WRITE: routing/data/kl-roadgraph.v1.bin is written ONLY
|
||||||
|
* after the encode and the stats gate both pass — any failure exits 1
|
||||||
|
* with the previous binary untouched.
|
||||||
|
* 6. THE STATS GATE prints nodeCount/edgeCount/nameCount/bbox/landmark
|
||||||
|
* facts/binary size and enforces the D-01 envelope: size ≤
|
||||||
|
* MAX_BINARY_BYTES or exit 1. A result below the expected 30–80 MB band
|
||||||
|
* prints an ADVISORY warning (A2's forecast) — the ENFORCED floor is the
|
||||||
|
* 1 MB honesty check in --verify, never the forecast band.
|
||||||
*
|
*
|
||||||
* The binary is a BUILD ARTIFACT, not committed: routing/data/.gitignore
|
* The binary is a BUILD ARTIFACT, not committed: routing/data/.gitignore
|
||||||
* ignores *.bin — forks build + upload their own (ODbL-derived data stays out
|
* ignores *.bin — forks build + upload their own (ODbL-derived data stays out
|
||||||
@@ -36,14 +45,19 @@
|
|||||||
*
|
*
|
||||||
* Usage:
|
* Usage:
|
||||||
* node routing/scripts/build-road-graph.mjs
|
* node routing/scripts/build-road-graph.mjs
|
||||||
* # full build: fetch → encode → gate → write
|
* # full build: fetch → landmarks → encode → gate → write
|
||||||
* # (network fetch; minutes on the KL bbox)
|
* # (network fetch; minutes on the KL bbox)
|
||||||
|
* node routing/scripts/build-road-graph.mjs --landmarks <n>
|
||||||
|
* # full build with an explicit landmark count — the D-02
|
||||||
|
* # MEASURED knob (raise L within/at the top of the rule for a
|
||||||
|
* # CPU-gate retry; hard-capped at 128, never a table bomb)
|
||||||
* node routing/scripts/build-road-graph.mjs --check <file>
|
* node routing/scripts/build-road-graph.mjs --check <file>
|
||||||
* # decode an existing binary with decodeGraph, print stats,
|
* # decode an existing binary with decodeGraph, print stats
|
||||||
* # exit 0/1 — NO network
|
* # incl. the landmark facts, exit 0/1 — NO network
|
||||||
*
|
*
|
||||||
* Exit codes: 0 built and gated (or checked green); 1 pre-flight failure,
|
* Exit codes: 0 built and gated (or checked green); 1 pre-flight failure,
|
||||||
* fetch failure, envelope-gate failure, or --check decode failure.
|
* fetch failure, landmark coverage/quantum-gate failure, envelope-gate
|
||||||
|
* failure, or --check decode failure.
|
||||||
*/
|
*/
|
||||||
import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
||||||
import { dirname, resolve } from "node:path";
|
import { dirname, resolve } from "node:path";
|
||||||
@@ -89,7 +103,10 @@ const COORD_ROUND_DECIMALS = 6;
|
|||||||
* CPU gate is carried by ladder step (b) — bidirectional A* in
|
* CPU gate is carried by ladder step (b) — bidirectional A* in
|
||||||
* routing/src/astar.ts — and the recorded verdict in the plan summary.
|
* routing/src/astar.ts — and the recorded verdict in the plan summary.
|
||||||
* Ladder step (c) — Workers Paid — breaks the $0 law and requires user
|
* Ladder step (c) — Workers Paid — breaks the $0 law and requires user
|
||||||
* sign-off; it is NOT applied in this script. */
|
* sign-off; it is NOT applied in this script.
|
||||||
|
* Ladder rung (c2) — EXECUTED in this script's preprocessing stage (Phase 8):
|
||||||
|
* build-time landmark tables + ALT steering; the verdict lives in
|
||||||
|
* measure-route-cpu.mjs's history and docs/ROAD-GRAPH.md. */
|
||||||
const ACTIVE_CLASSES = new Set([
|
const ACTIVE_CLASSES = new Set([
|
||||||
"motorway",
|
"motorway",
|
||||||
"trunk",
|
"trunk",
|
||||||
@@ -126,12 +143,48 @@ const LINK_CLASS_SPEED_KMH = {
|
|||||||
};
|
};
|
||||||
/** ENFORCED floor — an at/below-1 MB "KL road graph" is a broken build (D-01). */
|
/** ENFORCED floor — an at/below-1 MB "KL road graph" is a broken build (D-01). */
|
||||||
const MIN_BINARY_BYTES = 1 * 1024 * 1024;
|
const MIN_BINARY_BYTES = 1 * 1024 * 1024;
|
||||||
|
/** D-02 scaling rule: the landmark count is NODE-COUNT-PROPORTIONAL, never a
|
||||||
|
* fixed generous constant — per-landmark table bytes scale with N (~1.19
|
||||||
|
* MB/landmark at KL's 311k nodes), so over-provisioning explodes at Malaysia
|
||||||
|
* scale exactly where ALT's advantage itself narrows (that tier is
|
||||||
|
* Contraction Hierarchies' job, still deferred). L = clamp(round(N/13,000),
|
||||||
|
* 16, 32) → ~24 at KL scale; a future Malaysia-wide build re-derives the
|
||||||
|
* count from the same rule at its own scale. */
|
||||||
|
const LANDMARK_COUNT_MIN = 16;
|
||||||
|
const LANDMARK_COUNT_MAX = 32;
|
||||||
|
const NODES_PER_LANDMARK = 13000; // 311,325 / 13,000 ≈ 24 at KL scale
|
||||||
|
/** Build-time coverage gate: every anchor's ĉ-Dijkstra must reach at least
|
||||||
|
* this fraction of nodes (the measured KL giant SCC covers ~0.98 — the
|
||||||
|
* headroom is honest). The SAME fraction gates ANY --landmarks value: the
|
||||||
|
* override cannot buy a weaker gate (T-08-11). */
|
||||||
|
const LANDMARK_MIN_COVERAGE_FRACTION = 0.9;
|
||||||
|
/** --landmarks hard cap — the knob exists for measured retries within/at the
|
||||||
|
* top of the D-02 rule, not for unbounded table bombs (T-08-10). The 80 MB
|
||||||
|
* envelope gate fires before any write regardless. */
|
||||||
|
const LANDMARK_COUNT_OVERRIDE_MAX = 128;
|
||||||
|
|
||||||
function fail(message) {
|
function fail(message) {
|
||||||
console.error(`build-road-graph: ${message}`);
|
console.error(`build-road-graph: ${message}`);
|
||||||
process.exit(1);
|
process.exit(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ————— --landmarks override (the D-02 measured knob; parsed BEFORE --check) ————— */
|
||||||
|
|
||||||
|
const landmarksIdx = process.argv.indexOf("--landmarks");
|
||||||
|
let landmarksOverride = null;
|
||||||
|
if (landmarksIdx !== -1) {
|
||||||
|
const raw = process.argv[landmarksIdx + 1];
|
||||||
|
const n = raw === undefined ? NaN : Number(raw);
|
||||||
|
if (!Number.isInteger(n) || n < 1) {
|
||||||
|
fail(`--landmarks requires a positive integer count, got ${raw === undefined ? "(missing value)" : `"${raw}"`}`);
|
||||||
|
}
|
||||||
|
if (n > LANDMARK_COUNT_OVERRIDE_MAX) {
|
||||||
|
fail(`--landmarks ${n} exceeds the hard cap ${LANDMARK_COUNT_OVERRIDE_MAX} — the knob exists for measured ` +
|
||||||
|
`retries within/at the top of the D-02 rule, not for unbounded table bombs`);
|
||||||
|
}
|
||||||
|
landmarksOverride = n;
|
||||||
|
}
|
||||||
|
|
||||||
/* ————— --check mode: decode + stats, no network ————— */
|
/* ————— --check mode: decode + stats, no network ————— */
|
||||||
|
|
||||||
const checkIdx = process.argv.indexOf("--check");
|
const checkIdx = process.argv.indexOf("--check");
|
||||||
@@ -144,6 +197,20 @@ if (checkIdx !== -1) {
|
|||||||
const graph = decodeGraph(buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength));
|
const graph = decodeGraph(buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength));
|
||||||
if (graph === null) fail(`--check: ${file} is NOT a valid GRAPH_VERSION CSR binary (decodeGraph null)`);
|
if (graph === null) fail(`--check: ${file} is NOT a valid GRAPH_VERSION CSR binary (decodeGraph null)`);
|
||||||
const size = statSync(file).size;
|
const size = statSync(file).size;
|
||||||
|
// ARG2 landmark facts (Phase 8): decoded straight off the binary — the
|
||||||
|
// SAME ONE module computes the coverage audit here; NO network.
|
||||||
|
const { LANDMARK_SENTINEL, landmarkCoverage } = await import("../src/landmark.ts");
|
||||||
|
let toSentinels = 0;
|
||||||
|
for (let i = 0; i < graph.landmarkTo.length; i++) {
|
||||||
|
if (graph.landmarkTo[i] === LANDMARK_SENTINEL) toSentinels += 1;
|
||||||
|
}
|
||||||
|
let fromSentinels = 0;
|
||||||
|
for (let i = 0; i < graph.landmarkFrom.length; i++) {
|
||||||
|
if (graph.landmarkFrom[i] === LANDMARK_SENTINEL) fromSentinels += 1;
|
||||||
|
}
|
||||||
|
const coverage = landmarkCoverage(graph.landmarkTo, graph.landmarkNodes, graph.nodeCount);
|
||||||
|
const minCoverage = Math.min(...coverage);
|
||||||
|
const meanCoverage = coverage.reduce((a, b) => a + b, 0) / coverage.length;
|
||||||
console.log(
|
console.log(
|
||||||
`--check ${file}\n` +
|
`--check ${file}\n` +
|
||||||
` nodeCount: ${graph.nodeCount}\n` +
|
` nodeCount: ${graph.nodeCount}\n` +
|
||||||
@@ -151,7 +218,12 @@ if (checkIdx !== -1) {
|
|||||||
` nameCount: ${graph.nameCount}\n` +
|
` nameCount: ${graph.nameCount}\n` +
|
||||||
` maxSpeedKmh: ${graph.maxSpeedKmh}\n` +
|
` maxSpeedKmh: ${graph.maxSpeedKmh}\n` +
|
||||||
` bbox: [${graph.bbox.minLat}, ${graph.bbox.minLng}, ${graph.bbox.maxLat}, ${graph.bbox.maxLng}]\n` +
|
` bbox: [${graph.bbox.minLat}, ${graph.bbox.minLng}, ${graph.bbox.maxLat}, ${graph.bbox.maxLng}]\n` +
|
||||||
` bytes: ${size}`,
|
` bytes: ${size}\n` +
|
||||||
|
` landmarkCount: ${graph.landmarkCount}\n` +
|
||||||
|
` landmarkQuantumS: ${graph.landmarkQuantumS.toFixed(3)}\n` +
|
||||||
|
` landmarkNodes: ${[...graph.landmarkNodes].join(", ")}\n` +
|
||||||
|
` landmark coverage min/mean: ${minCoverage.toFixed(3)} / ${meanCoverage.toFixed(3)}\n` +
|
||||||
|
` landmark sentinel cells: ${toSentinels} to / ${fromSentinels} from`,
|
||||||
);
|
);
|
||||||
if (size > MAX_BINARY_BYTES) fail(`--check: ${size} bytes exceeds the ${MAX_BINARY_BYTES}-byte D-01 envelope`);
|
if (size > MAX_BINARY_BYTES) fail(`--check: ${size} bytes exceeds the ${MAX_BINARY_BYTES}-byte D-01 envelope`);
|
||||||
if (size <= MIN_BINARY_BYTES) fail(`--check: ${size} bytes is at/below the ${MIN_BINARY_BYTES}-byte floor — not a real road graph`);
|
if (size <= MIN_BINARY_BYTES) fail(`--check: ${size} bytes is at/below the ${MIN_BINARY_BYTES}-byte floor — not a real road graph`);
|
||||||
@@ -330,10 +402,83 @@ const edges = rawEdges
|
|||||||
const droppedNodes = latOut.length - cLat.length;
|
const droppedNodes = latOut.length - cLat.length;
|
||||||
if (cLat.length === 0 || edges.length === 0) fail("the extract produced an empty graph — refusing to write");
|
if (cLat.length === 0 || edges.length === 0) fail("the extract produced an empty graph — refusing to write");
|
||||||
|
|
||||||
|
/* ————— ALT preprocessing stage (Phase 8 rung c2): landmark tables through the ONE module —————
|
||||||
|
*
|
||||||
|
* This script quantizes nothing and implements nothing itself: the graph's
|
||||||
|
* edges feed ../src/landmark.ts (farthestPointAnchors + landmarkTables +
|
||||||
|
* landmarkCoverage) — there is no second integer-Dijkstra implementation here
|
||||||
|
* (single-authority law). Gates run BEFORE any write:
|
||||||
|
* - quantum ceiling: a landmarkTables throw means the ĉ metric overflowed
|
||||||
|
* the u16 domain → raise LANDMARK_QUANTUM_S in landmark.ts, never clamp
|
||||||
|
* (Pitfall 5);
|
||||||
|
* - coverage: every anchor reaches ≥ LANDMARK_MIN_COVERAGE_FRACTION of the
|
||||||
|
* graph — a fragmented extract fails honestly, it is never forced through.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const { LANDMARK_QUANTUM_S, LANDMARK_SENTINEL, farthestPointAnchors, landmarkTables, landmarkCoverage } =
|
||||||
|
await import("../src/landmark.ts");
|
||||||
|
|
||||||
|
// D-02 scaling rule: the count derives from THIS graph's node count; a
|
||||||
|
// --landmarks override (validated at argv time) replaces the rule wholesale.
|
||||||
|
const ruleRaw = Math.round(cLat.length / NODES_PER_LANDMARK);
|
||||||
|
const ruleCount = Math.min(LANDMARK_COUNT_MAX, Math.max(LANDMARK_COUNT_MIN, ruleRaw));
|
||||||
|
const landmarkCount = landmarksOverride ?? ruleCount;
|
||||||
|
console.log(
|
||||||
|
`landmarks: rule round(${cLat.length} / ${NODES_PER_LANDMARK}) = ${ruleRaw} → clamped ${ruleCount}` +
|
||||||
|
(landmarksOverride !== null ? `; --landmarks override → L = ${landmarkCount}` : `; L = ${landmarkCount}`),
|
||||||
|
);
|
||||||
|
|
||||||
|
// The module's input shape — projected ONCE (no mutation of the encode-side
|
||||||
|
// edge array; encodeGraph re-groups edges itself).
|
||||||
|
const lmEdges = edges.map((e) => ({ from: e.from, to: e.to, seconds: e.seconds, flags: e.flags }));
|
||||||
|
const lmInput = { nodeCount: cLat.length, edges: lmEdges, quantumS: LANDMARK_QUANTUM_S };
|
||||||
|
const anchors = farthestPointAnchors(lmInput, landmarkCount);
|
||||||
|
if (anchors.length === 0) {
|
||||||
|
fail("farthestPointAnchors found no degree>=2 candidate — the extract has no junction-ish node (degenerate/fragmented graph)");
|
||||||
|
}
|
||||||
|
if (anchors.length < landmarkCount) {
|
||||||
|
fail(`farthestPointAnchors returned only ${anchors.length} of ${landmarkCount} requested anchors (candidates exhausted) — ` +
|
||||||
|
`refusing to encode a weaker ALT than the rule asked for; investigate the extract before rebuilding`);
|
||||||
|
}
|
||||||
|
let to;
|
||||||
|
let from;
|
||||||
|
try {
|
||||||
|
({ to, from } = landmarkTables(lmInput, anchors));
|
||||||
|
} catch (err) {
|
||||||
|
fail(`landmark tables exceeded the u16 quantum ceiling (${err.message}) — raise LANDMARK_QUANTUM_S in ` +
|
||||||
|
`routing/src/landmark.ts and rebuild (Pitfall 5: raise the quantum, never clamp)`);
|
||||||
|
}
|
||||||
|
// Coverage gate (research Pattern 3, layer 1): the SAME fraction for ANY L —
|
||||||
|
// the override cannot lower it (T-08-11).
|
||||||
|
const coverage = landmarkCoverage(to, anchors, cLat.length);
|
||||||
|
const minCoverage = Math.min(...coverage);
|
||||||
|
if (minCoverage < LANDMARK_MIN_COVERAGE_FRACTION) {
|
||||||
|
const worst = coverage.indexOf(minCoverage);
|
||||||
|
fail(
|
||||||
|
`landmark coverage gate FAILED at L=${landmarkCount}: anchor ${worst} (node ${anchors[worst]}) reaches only ` +
|
||||||
|
`${(minCoverage * 100).toFixed(1)}% of nodes (gate: ≥ ${(LANDMARK_MIN_COVERAGE_FRACTION * 100).toFixed(0)}%). ` +
|
||||||
|
`The KL extract is expected to be ONE giant SCC covering ~98% of nodes (2026-10-24 build fact) — ` +
|
||||||
|
`a miss this large means a fragmented extract; investigate the fetch, do NOT lower the gate.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const meanCoverage = coverage.reduce((a, b) => a + b, 0) / coverage.length;
|
||||||
|
// Sentinel audit — cheap honesty; --check mirrors it from the written bytes.
|
||||||
|
let toSentinels = 0;
|
||||||
|
for (let i = 0; i < to.length; i++) if (to[i] === LANDMARK_SENTINEL) toSentinels += 1;
|
||||||
|
let fromSentinels = 0;
|
||||||
|
for (let i = 0; i < from.length; i++) if (from[i] === LANDMARK_SENTINEL) fromSentinels += 1;
|
||||||
|
|
||||||
/* ————— encode (the layout's single authority) + stats gate ————— */
|
/* ————— encode (the layout's single authority) + stats gate ————— */
|
||||||
|
|
||||||
const names = [...nameIds.keys()];
|
const names = [...nameIds.keys()];
|
||||||
const buffer = encodeGraph({ lat: cLat, lng: cLng, edges, names, maxSpeedKmh: maxSeenKmh });
|
const buffer = encodeGraph({
|
||||||
|
lat: cLat,
|
||||||
|
lng: cLng,
|
||||||
|
edges,
|
||||||
|
names,
|
||||||
|
maxSpeedKmh: maxSeenKmh,
|
||||||
|
landmarks: { quantumS: LANDMARK_QUANTUM_S, nodes: anchors, to, from },
|
||||||
|
});
|
||||||
const bytes = buffer.byteLength;
|
const bytes = buffer.byteLength;
|
||||||
|
|
||||||
const mb = (b) => `${(b / (1024 * 1024)).toFixed(1)} MB`;
|
const mb = (b) => `${(b / (1024 * 1024)).toFixed(1)} MB`;
|
||||||
@@ -348,6 +493,18 @@ console.log(
|
|||||||
` binary size: ${mb(bytes)} (${bytes} B)`,
|
` binary size: ${mb(bytes)} (${bytes} B)`,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// The landmark facts print BEFORE the size gates — the gate numbers exist
|
||||||
|
// outside the terminal even when a gate then refuses the write.
|
||||||
|
const tableBytes = 2 * landmarkCount * cLat.length * 2 + 4 * landmarkCount;
|
||||||
|
console.log(
|
||||||
|
`landmark stats (ALT preprocessing through the ONE module):\n` +
|
||||||
|
` L: ${landmarkCount} (D-02 rule ${ruleCount}${landmarksOverride !== null ? `, --landmarks override ${landmarksOverride}` : ""})\n` +
|
||||||
|
` quantum: ${LANDMARK_QUANTUM_S} s (the ĉ unit)\n` +
|
||||||
|
` coverage min/mean: ${minCoverage.toFixed(3)} / ${meanCoverage.toFixed(3)} (gate ≥ ${LANDMARK_MIN_COVERAGE_FRACTION})\n` +
|
||||||
|
` table bytes: ${tableBytes} (${mb(tableBytes)} = 2 families × L·N u16 + L u32 anchors)\n` +
|
||||||
|
` sentinel cells: ${toSentinels} to / ${fromSentinels} from`,
|
||||||
|
);
|
||||||
|
|
||||||
// THE ENFORCED GATE (D-01): outside the 1-80 MB envelope → exit 1 WITHOUT writing.
|
// THE ENFORCED GATE (D-01): outside the 1-80 MB envelope → exit 1 WITHOUT writing.
|
||||||
if (bytes > MAX_BINARY_BYTES) {
|
if (bytes > MAX_BINARY_BYTES) {
|
||||||
fail(`binary is ${mb(bytes)} — over the ${mb(MAX_BINARY_BYTES)} D-01 envelope. ` +
|
fail(`binary is ${mb(bytes)} — over the ${mb(MAX_BINARY_BYTES)} D-01 envelope. ` +
|
||||||
|
|||||||
Reference in New Issue
Block a user