tiennm99
219c7c6a69
Four of the 119 files in data/2016 publish one score column per subject instead of a DIEM_THI sentence, and none of them was being read correctly. The ĐH Công nghiệp Thực phẩm file puts a three-row ministry title block above its header, so no header was recognised and the positional fallback shifted every column by one: the serial number became so_bao_danh, the exam number became ho_ten, the name became ngay_sinh, and the national ID became the score cell. All 7,833 rows were unusable. The three ĐH Cần Thơ files name an SBD column but no DIEM_THI, so they fell to the same fallback: surname into ngay_sinh, given name into ten_cum_thi, birth date into the score cell, and 12,152 candidates with no scores at all. Both are now read by FormatSubjectColumns, which resolves identity and one column per subject from the header. The header is searched for in the first five rows, so a title block no longer hides it. The Cần Thơ score columns are numbered rather than named. They follow the order the exam was sat — each morning an essay paper, each afternoon a multiple-choice one — which is what identifies them: columns 1/3/5/7 quantise to 0.25 and 2/4/6/8 do not, and each column's mean lands within 0.5 of the same subject's mean across the rest of the dataset. The foreign language is filed under the subject its N1..N6 code names. Gender now accepts the 0/1 encoding those files use: of the rows marked 1, 53% carry "Thị" in the name against 1% of those marked 0. Birth dates in the compact ddmmyy form are expanded so the column holds one format. A score of 0 is stored rather than dropped, recovering 302 real scores that a JavaScript falsy check had been turning into NULL. Row count falls by one, to 877,460: the removed row is the title line "ĐƠN VỊ: / TRƯỜNG ĐẠI HỌC CÔNG NGHIỆP THỰC PHẨM TP. HỒ CHÍ MINH", which had been stored as a student. The dataset has no duplicate exam numbers; the three rows previously described as collapsing were that same file's title and header lines being counted and then rejected. Also drops behaviour that existed only to match the parser this one replaced: the inert "SINH " header token, the untrimmed diem_thi cell, an unreachable blank-row branch, and a cross-check test against a database that can no longer exist. None of them changes output. Verified by rebuilding both datasets: 877,460 and 861,068 rows, both artifacts through the assembler's row and size guards, and the reader fidelity suite unchanged across all 182 files.
3.4 KiB
parser
Reads the .xls/.xlsx source spreadsheets in data/ and writes one SQLite
database per dataset.
go -C parser build -o bin/xlsxread ./cmd/xlsxread # compile
go -C parser test ./... # unit tests + the reader-fidelity suite
xlsxread build --schema parser/configs/<id>.yml --input data/<id> --output <db>
xlsxread audit --schema parser/configs/<id>.yml --input data/<id> --db <db>
This stage only produces a database. Verifying it against the expected row
count, compressing it and publishing it belong to assembler/, which compiles
this binary and drives it per dataset:
go -C assembler run ./cmd/assemble db
Layout
| path | role |
|---|---|
internal/reader |
spreadsheet reading; the only place that knows about file formats |
internal/ingest |
dataset policy — sheet selection, header skipping, blank rows, the build loop, and the 2016 per-sheet layout detection |
internal/transform |
ToAscii, score-regex parsing, row validation |
internal/schema |
the canonical 22-column table: DDL, INSERT, subject regexes |
internal/config |
per-dataset YAML parse rules |
internal/writer |
SQLite lifecycle and the stats block |
internal/audit |
source-vs-database SBD comparison |
The reader deliberately knows nothing about datasets: it reports every sheet and
every row verbatim. All policy lives in ingest. That split is what made the
reader independently verifiable against a hash oracle.
Behaviour worth knowing
ToAsciistrips combining marks in the literal range U+0300–U+036F rather than by Unicode category. That covers every Vietnamese diacritic and must stay identical totoAsciiinweb/src/App.jsx, or accent-insensitive search misses rows.- Gender is normalised to
Nam/Nữ; the Cần Thơ files write0/1instead and are translated. Anything else becomes NULL. - A score of
0is a real score — the candidate sat the paper and scored nothing — and is stored, not dropped. - Birth dates are stored as
dd/mm/yyyy. The Cần Thơ files' compactddmmyyis expanded; the century is always 19xx, since a 2016 candidate born later would have sat the exam under age.
This code began as a port of a Rust crate that occupied the same path, and was
gated on a field-by-field comparison against it. That comparison is over:
correctness against the source spreadsheets decides behaviour now, not
agreement with the old implementation. assemble verify is the comparator
that gated it, and still compares any two sets of built databases.
Verification
testdata/reader-fidelity-hashes.tsv holds a SHA-256 per input file over a
canonical dump of every cell of every sheet. It is frozen: the tool that
produced it no longer exists, so it cannot be regenerated. It still fails if
any single cell of any input file reads differently, which is what makes it
useful — cell rendering and sheet geometry are settled, and a change there is
a regression until proven otherwise. Dataset policy lives in ingest, so
fixing a layout never touches it.
A mismatch names the file but not the cell. cmd/dumpcells prints the stream
the hash is taken over, so two runs can be diffed:
go -C parser run ./cmd/dumpcells ../data/2017/an-giang.xls out.tsv
The assembler refuses to publish a database whose row count does not match the known figure, or whose artifact is under 90% of its usual size.