tiennm99
c359a0b444
The repository now reads as the pipeline it is: crawler fetches, parser converts, assembler verifies and publishes, with data/ and web/ as the stores they hand work through. go-parser is renamed parser now that there is no other. The assembler replaces build-db.js and assemble-site.js. It compiles the parser, builds and verifies each database, compresses it, runs the Vite build and assembles _site — one command, and the only place that knows the order. It also closes a real hole: nothing previously asserted that a database reached the site. An empty staging directory assembled happily, so every page rendered, every query 404d and CI stayed green. The row-count and size guards could not catch that, since they only run when a database was built at all. Removing Node from the root forced the dataset list out of web/src/datasets.js, which the assembler cannot import. datasets.json is now the registry both sides read — JSON because Go and the browser both parse it without a dependency — while presentation stays in the web app, keyed by id and cross-checked against the registry so a half-added dataset fails instead of half-working. Guards verified by making each one fail: a missing database, and an expected row count one higher than the truth.
parser
Reads the .xls/.xlsx source spreadsheets in data/ and writes one SQLite
database per dataset.
go -C parser build -o bin/xlsxread ./cmd/xlsxread # compile
go -C parser test ./... # unit tests + the reader-fidelity suite
xlsxread build --schema parser/configs/<id>.yml --input data/<id> --output <db>
xlsxread audit --schema parser/configs/<id>.yml --input data/<id> --db <db>
This stage only produces a database. Verifying it against the expected row
count, compressing it and publishing it belong to assembler/, which compiles
this binary and drives it per dataset:
go -C assembler run ./cmd/assemble db
Layout
| path | role |
|---|---|
internal/reader |
spreadsheet reading; the only place that knows about file formats |
internal/ingest |
dataset policy — sheet selection, header skipping, blank rows, the build loop, and the 2016 per-sheet format detection |
internal/transform |
ToAscii, score-regex parsing, row validation |
internal/schema |
the canonical 22-column table: DDL, INSERT, subject regexes |
internal/config |
per-dataset YAML parse rules |
internal/writer |
SQLite lifecycle and the stats block |
internal/audit |
source-vs-database SBD comparison |
The reader deliberately knows nothing about datasets: it reports every sheet and
every row verbatim. All policy lives in ingest. That split is what made the
reader independently verifiable against a hash oracle.
Provenance
This is a port of a Rust crate that occupied this same path until the Go
implementation reached full parity, when it was built alongside as go-parser/
and moved back here once the Rust was removed. Source comments cite the original
by file and line (parser/src/transform.rs:56 and similar) — those refer to the
Rust tree and resolve at the tag pre-go-parser-removal, the last commit
containing it.
The port was gated on a field-by-field comparison of both implementations across
the four datasets that existed then — 3,265,641 rows with identical full-table
SHA-256, identical per-column non-NULL counts, identical schema metadata and
identical build stdout. scripts/differential-parity.mjs is that comparator and
still runs against any two sets of databases.
Behaviour was matched bug-for-bug, deliberately. Several quirks look like defects and are load-bearing for the published data:
- a parsed score of
0becomes NULL in the 2016 separate-scores layout, replicating a JavaScript falsy check; ToAsciistrips combining marks in the literal range U+0300–U+036F rather than by Unicode category, which is narrower;- gender is a two-value allowlist, and anything else becomes NULL;
diem_thiis read untrimmed while the other three fields are trimmed;- the
"SINH "header token carries a trailing space.
Each has a test naming it, so none can be tidied away by accident.
Verification
testdata/reader-fidelity-hashes.tsv holds a SHA-256 per input file over a
canonical dump of every cell of every sheet. It is frozen: it was produced
by the Rust reader, which no longer exists, so it cannot be regenerated. It
still fails if any single cell of any of the 182 files reads differently.
The assembler refuses to publish a database whose row count does not match the known figure, or whose artifact is under 90% of its usual size.