tiennm99
3fd137a233
The app was a single-entry React SPA: one index.html that the assembler copied to every dataset path, with a hand-rolled router resolving the dataset from window.location and the title patched in at runtime because one file had to serve every route. SvelteKit prerenders a real page per route instead. The entry generator in routes/[dataset]/+page.ts reads the same datasets.json the assembler does, so the set of pages and the set of databases cannot drift apart, and each dataset page ships its own <title> and description. Everything framework-free moved across unchanged in behaviour and became typed: the admission blocks, subject list, query classifier and SQL presets. `Student` now mirrors the 22-column table, so a mistyped column name fails the build rather than rendering blank. Tailwind replaces the stylesheet. Theme tokens are CSS variables, which keeps dark mode a single block of overrides rather than a `dark:` variant on every class. The score tiers stay hand-written CSS: the class is chosen at runtime from a score, and no utility generator can see that. Adds the tests the frontend never had, over the three modules where a silent wrong answer is possible — most importantly that toAscii here folds exactly as ToAscii does in the parser, which is what makes accent-insensitive search find anything. The assembler stops copying index.html per dataset and checks that the build prerendered each one instead. The database presence, raw-artifact and idempotence guards are untouched. Verified: 25 tests, ESLint and svelte-check clean, and a full `assemble site` producing /thptqg/, /thptqg/2016/, /thptqg/2017/ and 404.html with absolute asset URLs. Not verified in a browser — this machine has none.
3.4 KiB
parser
Reads the .xls/.xlsx source spreadsheets in data/ and writes one SQLite
database per dataset.
go -C parser build -o bin/xlsxread ./cmd/xlsxread # compile
go -C parser test ./... # unit tests + the reader-fidelity suite
xlsxread build --schema parser/configs/<id>.yml --input data/<id> --output <db>
xlsxread audit --schema parser/configs/<id>.yml --input data/<id> --db <db>
This stage only produces a database. Verifying it against the expected row
count, compressing it and publishing it belong to assembler/, which compiles
this binary and drives it per dataset:
go -C assembler run ./cmd/assemble db
Layout
| path | role |
|---|---|
internal/reader |
spreadsheet reading; the only place that knows about file formats |
internal/ingest |
dataset policy — sheet selection, header skipping, blank rows, the build loop, and the 2016 per-sheet layout detection |
internal/transform |
ToAscii, score-regex parsing, row validation |
internal/schema |
the canonical 22-column table: DDL, INSERT, subject regexes |
internal/config |
per-dataset YAML parse rules |
internal/writer |
SQLite lifecycle and the stats block |
internal/audit |
source-vs-database SBD comparison |
The reader deliberately knows nothing about datasets: it reports every sheet and
every row verbatim. All policy lives in ingest. That split is what made the
reader independently verifiable against a hash oracle.
Behaviour worth knowing
ToAsciistrips combining marks in the literal range U+0300–U+036F rather than by Unicode category. That covers every Vietnamese diacritic and must stay identical totoAsciiinweb/src/lib/to-ascii.ts, or accent-insensitive search misses rows.- Gender is normalised to
Nam/Nữ; the Cần Thơ files write0/1instead and are translated. Anything else becomes NULL. - A score of
0is a real score — the candidate sat the paper and scored nothing — and is stored, not dropped. - Birth dates are stored as
dd/mm/yyyy. The Cần Thơ files' compactddmmyyis expanded; the century is always 19xx, since a 2016 candidate born later would have sat the exam under age.
This code began as a port of a Rust crate that occupied the same path, and was
gated on a field-by-field comparison against it. That comparison is over:
correctness against the source spreadsheets decides behaviour now, not
agreement with the old implementation. assemble verify is the comparator
that gated it, and still compares any two sets of built databases.
Verification
testdata/reader-fidelity-hashes.tsv holds a SHA-256 per input file over a
canonical dump of every cell of every sheet. It is frozen: the tool that
produced it no longer exists, so it cannot be regenerated. It still fails if
any single cell of any input file reads differently, which is what makes it
useful — cell rendering and sheet geometry are settled, and a change there is
a regression until proven otherwise. Dataset policy lives in ingest, so
fixing a layout never touches it.
A mismatch names the file but not the cell. cmd/dumpcells prints the stream
the hash is taken over, so two runs can be diffed:
go -C parser run ./cmd/dumpcells ../data/2017/an-giang.xls out.tsv
The assembler refuses to publish a database whose row count does not match the known figure, or whose artifact is under 90% of its usual size.