tiennm99
a0420fdc37
The pipeline is now Go outside web/. differential-parity.mjs becomes
assembler/internal/verify, reachable as `assemble verify A B`. The port fixed a
real weakness: the JavaScript hashed each row's fields joined bare, so a value
shifted across a column boundary produced the same digest. A test now pins that.
The hub still rendered "Phiên bản cũ của trang 2017" above a permanently empty
list — it split datasets on id.includes("old"), and both such datasets are gone.
The heading and the filter are removed. index.html titled every page "THPT QG
2017", including 2016 and the hub, because one file is copied to every route;
the static title is now neutral and the app sets the dataset's own.
Dead code removed: the isOld2/containsOld branches in the stats block, which
only 2017-old2 could ever reach; SUBJECT_LABELS, DATASET_IDS and the unread
`short` subject field; an unused vite.svg and a favicon link to a file that
never existed; two unused CSS rules and --shadow-sm; site.Paths.Root.
Corrected comments that were confidently wrong rather than merely stale: the
reader claimed to be row-streaming when both implementations decode the whole
workbook into memory first, and the fidelity oracle still spoke of 299 input
files when it covers 182. Candidate counts in the hub now derive from
datasets.json instead of being written a second time as prose.
plans/ is emptied. The parity report it held was cited by docs/data-pipeline.md,
so the evidence that the recovered foreign-language scores are real — not the
citation, the four arguments themselves — is now inline there.
Verified: 2017 rebuilt after the writer change hashes identically to the build
before it.
3.8 KiB
thptqg
Tra cứu điểm thi THPT Quốc gia — exam-score lookup for Vietnam's national high
school graduation exam. Client-side SQL (sql.js) over a SQLite database built
from the ministry's raw .xls score files by the Go xlsxread parser.
Live at tiennm99.github.io/thptqg.
| Dataset | Exam | Candidates | Site |
|---|---|---|---|
2016 |
2016 | 877,461 | /2016/ |
2017 |
2017 | 861,068 | /2017/ |
Two earlier 2017 publications (2017-old, 2017-old2) were kept for a while
because they disagreed with the current one. They have been removed; they remain
in git history.
Layout
The repository is one directory per pipeline stage, plus the two stores they pass between them.
crawler/ Go — re-fetches the source spreadsheets → data/
parser/ Go — Excel to SQLite data/ → .db
assembler/ Go — verifies, compresses, builds, assembles .db + web/ → _site/
web/ npm — the frontend, one Vite app for every dataset
data/<id>/ raw Excel files, one directory per dataset
datasets.json the registry: which datasets exist, and their expected size
docs/ architecture, data pipeline, deployment
Each stage runs on its own and hands its output to the next through the stores.
web/ is the only npm project; the three stages are independent Go modules.
datasets.json is the contract between them. It is JSON because Go and the Vite
app both read it and neither needs a dependency to do so; presentation stays in
web/src/datasets.js, keyed by id, which fails loudly if the two disagree.
The dataset id is one identifier end to end:
data/2017/ → parser/configs/2017.yml → db/2017.db.gz → /thptqg/2017/
Build
(cd web && npm ci)
go -C assembler run ./cmd/assemble # databases, then the site, into _site/
npx serve _site
That one command compiles the parser, builds and verifies each database against
its registry row count, compresses it, builds the web app and assembles _site —
refusing to continue if a database is short, an artifact looks truncated, or one
is missing altogether. Sub-steps when iterating:
go -C assembler run ./cmd/assemble db 2017 # one database
go -C assembler run ./cmd/assemble site # web build and _site only
go -C assembler run ./cmd/assemble verify A B # compare two sets of databases
(cd web && npm run dev) # the app against staged databases
The source spreadsheets are committed, so a crawl is only needed to refresh them:
go -C crawler run ./cmd/crawl 2016
go -C crawler run ./cmd/crawl 2017
Each reads the download links out of the article that published the dataset, so no link list is kept in the repository. Crawling is idempotent — files already present are skipped — and is never part of the build.
Pushing to main runs the same steps in
.github/workflows/deploy-pages.yml and publishes to GitHub Pages.
Adding a dataset
- Put the Excel files in
data/<id>/ - Add
parser/configs/<id>.yml— sheet mode, column indices, validation guards. No SQL; the schema is canonical. - Add an entry to
datasets.jsonwith its expected row count and size - Add the matching presentation to
CONTENTinweb/src/datasets.js
Everything else follows: the assembler, the router and the hub all read the registry, and the UI adapts to whichever columns the dataset fills. Steps 3 and 4 check each other, so forgetting either one fails rather than half-working.
Docs
See docs/ — overview,
architecture,
data pipeline,
deployment.