tiennm99
4cb0a4f340
Reading the file where it lay never worked well enough. Two costs were structural rather than bugs: reads are serial, because the worker uses synchronous XHR, so a name search touching 390 pages waited 17 seconds to move 608 KB — roughly one request per result row, which no page size removes — and the first visitor after each deploy waited ~26 seconds for the CDN to fill its cache with a 288 MB object. The browser now downloads the whole database once and queries it in memory with sql.js. A dataset page is gated behind that: the gate states what it will cost, in transfer and in memory, and offers only the download, because there is nothing to show without it. Dropping the structures that existed to make range-request queries index-driven halved the file. name_word carried one row per word of every name, about 3.5 million of them, and with the partial score indexes it was more than half of what every visitor would now download. Measured on rebuilt databases: 2016 went 288.6 -> 142.5 MB (31 MB gzipped on the wire), 2017 237.7 -> 119.3 MB, both with row counts and audits unchanged. Queries on the result: an exam number is immediate, a name scans all 877,460 rows in about 240 ms. Alternatives were measured before choosing this. sqlite-wasm-http sizes files from a HEAD Content-Length with no override, so on a host that gzips it silently uses the compressed size. DuckDB-WASM ships 32-37 MB of WebAssembly before its Parquet extension, more than this whole download. Static pre-generated shards are the most robust option but cannot answer arbitrary SQL, and cannot stop early the way LIMIT does. The published name loses its chunk index, the byte budgets and the SQL consent modal go with the range reads that made them necessary, and the docs no longer describe a design the site does not use.
2.6 KiB
Project Overview
Goal
A public lookup tool for Vietnam's National High School Graduation Exam scores, running entirely in the browser and hosted for free on GitHub Pages. Covers the 2016 and 2017 exams — 1.7 million candidates across two datasets.
Scope
- Lookup by exam ID or full name, with Vietnamese diacritics handled
- Read-only SQL queries against a single
studenttable - Admission-block (khối thi) totals computed per candidate
- Static datasets — both exams are long over and the data is frozen
Target users
- Former candidates checking their scores
- Education researchers and data journalists running aggregate statistics
- Developers exploring SQL against a real-world dataset
Constraints
- Zero backend. The database (238–289 MB per dataset) stays on the server and the browser downloads it once and queries it in memory.
- Read-only.
INSERT/UPDATE/DELETEare rejected, so nobody is misled into thinking edits persist. The file is fetched, never written. - Row caps. 100 rows for lookups, 1000 for custom SQL, to prevent browser hangs.
- Vietnamese-first UI. App labels and data are Vietnamese; documentation is English.
Datasets
| id | Exam | Candidates | Notes |
|---|---|---|---|
2016 |
2016 | 877,460 | 119 files, four column layouts |
2017 |
2017 | 861,068 | current generation of three publications |
Two further 2017 datasets (2017-old, 2017-old2) were kept alongside these
because the three publications disagreed. They have been removed; git history
still has them.
Both datasets now have a crawler source. 2017 comes from the baotintuc.vn CDN,
which is still live. 2016 comes from an aggregator article on
dtnt.bacninh.edu.vn, also still online. Both datasets have been crawled
successfully, so either can be rebuilt from source — see
data-pipeline for the full article URLs.
History
Each year began as a standalone repository (thptqg2016, thptqg2017), merged
here with full history. They initially kept separate frontends and separate
copies of the same Rust parser, synchronised by hand. That duplication was
removed: there is now one frontend, one parser, and one canonical schema, with
per-dataset differences confined to one small config file and one registry
entry each.
The unification also fixed a latent data-loss bug — neither year's parser
config listed the complete set of subjects, so 1,691 candidates were missing
their foreign-language score. See data-pipeline.md.
Status
Stable, data frozen. Work is limited to UX polish and keeping the pipeline maintainable.