From 8ca6bfa7a8aeb3e1ef65a6a18da59cdffb03d912 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Tue, 14 Apr 2026 23:25:51 +0700 Subject: [PATCH] docs: expand README, add architecture / deployment / data-pipeline docs - README: live-site URLs, features, scripts, project layout, quickstart - docs/system-architecture.md: data flow, 3-variant deploy mechanism, schema, parse-quirk matrix, score-tier model, admission-block model, frontend key behaviors (deep-link, share, keyboard shortcuts) - docs/data-pipeline.md: source Excel shape, score regex, why three parsers, overflow-sheet gotcha, audit tooling, refresh flow - docs/deployment-guide.md: CI workflow, adding a new variant, rollback, GH Pages file-size note - docs/README.md: index for the docs dir --- 2017/README.md | 97 +++++++++++++++++++++++++++++++- 2017/docs/README.md | 5 ++ 2017/docs/data-pipeline.md | 88 +++++++++++++++++++++++++++++ 2017/docs/deployment-guide.md | 52 +++++++++++++++++ 2017/docs/system-architecture.md | 97 ++++++++++++++++++++++++++++++++ 5 files changed, 336 insertions(+), 3 deletions(-) create mode 100644 2017/docs/README.md create mode 100644 2017/docs/data-pipeline.md create mode 100644 2017/docs/deployment-guide.md create mode 100644 2017/docs/system-architecture.md diff --git a/2017/README.md b/2017/README.md index f373ed1..e43e864 100644 --- a/2017/README.md +++ b/2017/README.md @@ -1,4 +1,95 @@ -# thptqg2017 -Database điểm thi THPT QG 2017 +# Tra cứu điểm thi THPT Quốc gia 2017 -**Nguồn**: Sưu tầm từ trang báo thời đó +Static React + SQLite site for looking up Vietnamese high school graduation exam scores (2017). The full database (~861k students, 63 provinces) ships to the browser as a compressed SQLite file and queries run client-side via `sql.js` — no backend. + +## Live sites + +Three deployments, one per dataset: + +| URL | Data source | Rows | +|---|---|---| +| `/thptqg2017/` | `data/` — [baotintuc.vn](https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm) CDN, `.xls` | **861,068** | +| `/thptqg2017/old/` | `data-old/` — earlier xlsx collection | 847,348 | +| `/thptqg2017/old2/` | `data-old2/` — partial re-export (54 provinces) | 679,764 | + +## Features + +- Diacritics-insensitive name search (`"nguyen"` matches `"Nguyễn"`) +- Live debounced search with URL deep-link (`?q=49008235`) +- Single-result detail card with per-subject TFT rarity-tiered scores (6 tiers, ≤1 → 9-10) and all 49 admission blocks (khối A00 – D15) +- Share button: copies formatted summary + URL for Zalo/Messenger +- SQL query tab with grouped presets (rankings, Long An filters, statistics, schema) +- Light + dark mode (follows OS preference) +- Keyboard shortcut `/` to focus search + +## Requirements + +- Node.js 20+ +- npm + +## Quickstart + +```bash +npm install +npm run build:db # parse data/ into public/thptqg2017.db (~2 min, 159 MB) +gzip -kf -9 public/thptqg2017.db +npm run dev # http://localhost:5173 +``` + +## Scripts + +| Command | Action | +|---|---| +| `npm run dev` | Vite dev server | +| `npm run build` | Production build (main variant → `dist/`) | +| `npm run build:old` / `build:old2` | Build variant sites to `dist/old/`, `dist/old2/` | +| `npm run build:all` | All 3 web variants | +| `npm run build:db` | Build main DB from `data/` | +| `npm run build:db:old` / `build:db:old2` | Build old / old2 variant DBs | +| `npm run build:db:all` | All 3 DBs | +| `npm run lint` | ESLint | +| `node scripts/crawl-baotintuc.js` | Re-download all 63 province files from baotintuc.vn | +| `node scripts/audit-row-counts.js` | Verify source row count matches DB row count | +| `node scripts/check-duplicates.js` | MD5 + row-content duplicate audit | +| `node scripts/diff-datasets.js` | Compare `public/` vs `backup/` DB (when backup present) | + +## Project layout + +``` +. +├── data/ # 63 .xls files (source) +├── data-old/ # 63 .xlsx (previous export) +├── data-old2/ # 54 .xlsx (update/ overrides) +├── public/ # main variant assets + thptqg2017.db.gz +├── public-old/ # old variant assets +├── public-old2/ # old2 variant assets +├── scripts/ +│ ├── build-lib.js # shared schema + helpers +│ ├── build-database.js # parser for data/ +│ ├── build-database-old.js # parser for data-old/ +│ ├── build-database-old2.js # parser for data-old2/ +│ ├── crawl-baotintuc.js # downloader +│ ├── audit-row-counts.js # parse-loss audit +│ ├── check-duplicates.js # md5 dup detector +│ └── diff-datasets.js # DB-to-DB comparator +├── src/ +│ ├── App.jsx +│ ├── App.css +│ ├── components/{search-form, score-table, student-detail, custom-query}.jsx +│ ├── hooks/use-sqlite.js +│ └── lib/admission-blocks.js # 49 khối definitions + score-tier helper +├── docs/ # see docs/README.md +├── index.html +├── vite.config.js +└── package.json +``` + +See `docs/` for architecture + deployment details. + +## License + +See `LICENSE`. + +## Source + +Data collected from public Vietnamese news archives (2017). Intended for reference only. diff --git a/2017/docs/README.md b/2017/docs/README.md new file mode 100644 index 0000000..a16c8b6 --- /dev/null +++ b/2017/docs/README.md @@ -0,0 +1,5 @@ +# Docs + +- [`system-architecture.md`](./system-architecture.md) — data flow, schema, 3-variant deploy, score-tier model, admission-block catalog +- [`data-pipeline.md`](./data-pipeline.md) — Excel parse quirks, overflow-sheet gotcha, audit scripts, refresh flow +- [`deployment-guide.md`](./deployment-guide.md) — GitHub Pages workflow, adding a new variant, rollback diff --git a/2017/docs/data-pipeline.md b/2017/docs/data-pipeline.md new file mode 100644 index 0000000..d000e92 --- /dev/null +++ b/2017/docs/data-pipeline.md @@ -0,0 +1,88 @@ +# Data Pipeline + +From raw Excel files to a compressed SQLite file the browser can load. + +## Canonical source + +63 Excel files from baotintuc.vn CDN. Re-download anytime: + +```bash +node scripts/crawl-baotintuc.js +``` + +Idempotent — skips files already present. Saves to `data/.xls`. + +Source article URL: `https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm` + +## Source Excel shape + +Every file is a single sheet (for `data-old/`) or up to 3 sheets (for `data/` .xls) with columns: + +| Col | Name | Content | +|---|---|---| +| 0 | HO_TEN | full name in Vietnamese | +| 1 | NGAY_SINH | `dd/mm/yyyy` | +| 2 | SOBAODANH | 8-digit string, first 2 digits = province | +| 3 | DIEM_THI | concatenated Vietnamese score string, e.g. `"Toán: 6.80 Ngữ văn: 5.25 … Tiếng Anh: 5.80"` | + +Some files lack a header row. `isHeaderRow()` in `build-lib.js` detects and skips. + +## Score text parsing + +`SCORE_PATTERNS` in `build-lib.js` defines one regex per subject: + +```js +{ toan: /Toán:\s*(\d+(?:\.\d+)?)/, ngu_van: /Ngữ văn:…/, … } +``` + +Subjects supported: Toán, Ngữ văn, Vật lí, Hóa học, Sinh học, KHTN, Lịch sử, Địa lí, GDCD, KHXH, Tiếng Anh, Tiếng Pháp, Tiếng Nga, Tiếng Trung. + +Missing-in-string → `NULL` in DB (student didn't take that subject). + +## Why three parsers + +One shared lib + three thin parsers, not one parser with flags. Each dataset's quirks stay visible in its own file: + +| File | Source dir | Sheet strategy | Extra guard | +|---|---|---|---| +| `build-database.js` | `data/` | iterate ALL sheets (.xls row overflow) | — | +| `build-database-old.js` | `data-old/` | sheet 0 only | reject rows where `so_bao_danh` isn't digits-only (rejects an `HO_TEN` → `SOBAODANH` header leak present in this export) | +| `build-database-old2.js` | `data-old2/` | iterate ALL sheets (HCM overflow) | skip fully blank rows before counting | + +## Overflow-sheet gotcha + +The old `.xls` binary format caps at 65,536 rows per sheet. Hà Nội and HCM exceed that in the baotintuc export, so Excel auto-splits the overflow into `Sheet2`. **Only reading `Sheet1` silently drops 13,720 students** (Hà Nội +7,275, HCM +6,445). + +The audit `node scripts/audit-row-counts.js` catches this by comparing source-row-count (across all sheets) to DB-row-count. + +## Audits + +Run any time after a rebuild: + +```bash +node scripts/audit-row-counts.js # source rows vs DB rows (zero-loss check) +node scripts/check-duplicates.js # md5 + row-content duplicates in data/ +node scripts/diff-datasets.js # compare two DBs (needs backup/ dir) +``` + +Expected baseline: + +| DB | Source rows | Skipped | DB rows | +|---|---|---|---| +| main | 861,131 | 63 empty rows | **861,068** | +| old | 847,349 | 1 header-leak | **847,348** | +| old2 | 679,764 | 0 | **679,764** | + +## Refreshing the data + +```bash +rm data/*.xls # clear current snapshot +node scripts/crawl-baotintuc.js # re-download from CDN +npm run build:db # rebuild +node scripts/audit-row-counts.js # verify +gzip -kf -9 public/thptqg2017.db # compress for shipping +``` + +## Adding a new source dataset + +See `docs/deployment-guide.md` → "Adding a new variant". diff --git a/2017/docs/deployment-guide.md b/2017/docs/deployment-guide.md new file mode 100644 index 0000000..4410294 --- /dev/null +++ b/2017/docs/deployment-guide.md @@ -0,0 +1,52 @@ +# Deployment Guide + +Deploys to GitHub Pages via `.github/workflows/deploy.yml`. Every push to `main` rebuilds and redeploys all 3 variants. + +## What the workflow does + +1. Checkout, setup Node 20, `npm ci` +2. `npm run build:db:all` — builds 3 SQLite DBs (main, old, old2) +3. Gzip each DB at level 9 +4. `npm run build:all` — builds 3 Vite bundles into `dist/`, `dist/old/`, `dist/old2/` +5. Remove uncompressed `.db` files from `dist/` (only `.gz` ships) +6. `actions/upload-pages-artifact` + `actions/deploy-pages` + +Total CI time ≈ 4–6 min (DB build dominates). + +## Resulting URLs + +- `https://.github.io/thptqg2017/` +- `https://.github.io/thptqg2017/old/` +- `https://.github.io/thptqg2017/old2/` + +## Local reproduction + +```bash +npm ci +npm run build:db:all +gzip -kf -9 public/thptqg2017.db public-old/thptqg2017.db public-old2/thptqg2017.db +npm run build:all +npx serve dist # or any static server +``` + +## Adding a new variant + +1. Drop source Excel files into a new `data-vN/` folder +2. Add `public-vN/` to `.gitignore` patterns for the `.db` + `.db.gz` +3. Copy `scripts/build-database-old.js` → `scripts/build-database-vN.js`; update `SRC_DIR` / `DB_PATH`; adjust parse logic for the new quirks (multi-sheet? blank rows? numeric guard?) +4. Add variant to `VARIANT_CONFIG` in `vite.config.js`: + ```js + vN: { base: "/thptqg2017/vN/", publicDir: "public-vN", outDir: "dist/vN" } + ``` +5. Add `build:db:vN` and `build:vN` npm scripts; wire them into `build:db:all` and `build:all` +6. Update `.github/workflows/deploy.yml` — add gzip step + rm step for the new DB + +## Notes + +- **DB gzip is non-cacheable across deploys** — every rebuild produces a new `.db.gz` (not byte-identical due to SQLite page shuffling). First-visit users pay the 47 MB download; subsequent visits hit browser cache until the next deploy. +- **GitHub Pages file size limit**: individual files ≤ 100 MB. Main gzipped DB is ~47 MB — safe. Uncompressed 159 MB DB would not fit; that's why we ship `.gz` and the frontend decompresses via `DecompressionStream`. +- **No server-side compression assumption.** The app reads the `.gz` file directly (not `Content-Encoding: gzip`). GH Pages does not reliably gzip on-the-fly for arbitrary paths; shipping pre-gzipped bytes is deterministic. + +## Rollback + +Deploys are stateless snapshots. To rollback, revert the commit on `main` and push — the next workflow rebuilds the older state. There's no state to migrate. diff --git a/2017/docs/system-architecture.md b/2017/docs/system-architecture.md new file mode 100644 index 0000000..5a5eb48 --- /dev/null +++ b/2017/docs/system-architecture.md @@ -0,0 +1,97 @@ +# System Architecture + +Static site. No backend. Browser downloads a compressed SQLite file at boot, then every query runs locally via `sql.js` (WASM). + +## Data flow + +``` +Excel files (.xls / .xlsx) + │ + ▼ build-database*.js (Node) +SQLite DB (public*/thptqg2017.db) + │ + ▼ gzip -9 +thptqg2017.db.gz (~47 MB for main variant) + │ + ▼ Vite build copies public*/ into dist/ +Static site on GitHub Pages + │ + ▼ browser loads +sql.js (WASM) opens the .db → queries run client-side +``` + +## Three deployment variants + +One repo → three independent sites, same frontend, different dataset. + +| Variant | Route | Source dir | Public dir | Build cmd | +|---|---|---|---|---| +| main | `/thptqg2017/` | `data/` | `public/` | `npm run build` | +| old | `/thptqg2017/old/` | `data-old/` | `public-old/` | `npm run build:old` | +| old2 | `/thptqg2017/old2/` | `data-old2/` | `public-old2/` | `npm run build:old2` | + +`vite.config.js` reads `process.env.VARIANT` and switches `base`, `publicDir`, `outDir` accordingly. `emptyOutDir` is on only for the main build so the variant builds merge into `dist/old/`, `dist/old2/` cleanly. + +## Schema (all 3 DBs share this) + +```sql +CREATE TABLE student ( + so_bao_danh TEXT PRIMARY KEY, -- exam ID (8 digits, first 2 = province) + ho_ten TEXT NOT NULL, + ho_ten_ascii TEXT NOT NULL, -- NFD-stripped lowercase for accent-insensitive search + ngay_sinh TEXT, -- dd/mm/yyyy + toan, ngu_van, vat_ly, hoa_hoc, sinh_hoc, khtn, + lich_su, dia_ly, gdcd, khxh, + tieng_anh, tieng_phap, tieng_nga, tieng_trung REAL +); +CREATE INDEX idx_ho_ten ON student(ho_ten); +CREATE INDEX idx_ho_ten_ascii ON student(ho_ten_ascii); +``` + +No German / Japanese language columns — neither appears in any source file. + +Each score column is `NULL` when the student didn't take that subject. A student's "admission block" total is only computed when all 3 required subjects are non-null. + +## Parse quirks (why 3 builder scripts) + +Each dataset had a different quirk. Rather than one mega-parser with mode flags, each builder script is ~70 lines and isolates its own workarounds. + +| Dataset | Quirk | Mitigation | +|---|---|---| +| data/ (baotintuc .xls) | Hà Nội + HCM overflow past the 65,536 row-per-sheet .xls limit | Iterate all `wb.SheetNames` | +| data-old/ (xlsx) | Single-sheet always; one bogus header row (`SOBAODANH`/`HO_TEN`) leaked into an earlier DB | Strict numeric-SBD guard rejects the leak | +| data-old2/ (xlsx) | HCM overflow + many blank trailing rows | Multi-sheet walk + blank-row pre-filter | + +Shared concerns (regex, schema, `toAscii`, header detection) live in `scripts/build-lib.js`. + +## Admission blocks (khối thi) + +`src/lib/admission-blocks.js` lists all 49 official 2017 blocks computable from our schema (A00–A11, B00–B08, C00–C20, D01–D15). Each entry is `{ code, subjects: [3 keys], label }`. + +`computeBlocks(student)` returns only blocks where the student has all 3 subject scores, sorted by total desc. Used by the student detail card. + +The SQL preset "Top 10 điểm khối cao nhất – Long An" materialises all 49 blocks as a `UNION ALL` CTE, picks each student's best via `ROW_NUMBER() OVER (PARTITION BY so_bao_danh ORDER BY s DESC, k)`, then ranks across students. + +## Score tiers (UI coding) + +6-level TFT rarity ladder (white → green → blue → purple → gold → prismatic). Defined in `scoreTier()` in `src/lib/admission-blocks.js`. + +| Tier | Range | Label | +|---|---|---| +| common | ≤ 1 | Điểm liệt | +| uncommon | < 5 | Chưa đạt | +| rare | 5–6.5 | Trung bình | +| epic | 6.5–8 | Khá | +| legendary | 8–9 | Giỏi | +| prismatic | 9–10 | Xuất sắc | + +Color + icon + text label — never color-only. + +## Frontend key behaviors + +- Query state lives in `App.jsx` and is synced to URL `?q=...` via `history.replaceState` — bookmarkable, shareable. +- `SearchForm` is a controlled component that 300ms-debounces input changes and auto-triggers search once the query is long enough (3+ digits for SBD, 2+ chars for name). +- Detection: all-digits → exact `so_bao_danh` match; ASCII-only → folded `ho_ten_ascii LIKE`; has Vietnamese diacritics → search both `ho_ten` and folded column. +- Exactly 1 result → `StudentDetail` card; 0 or >1 → `ScoreTable`. +- Global `/` key focuses the search input when not already typing. +- Share button on the detail card uses Web Share API when present, else clipboard with a pre-formatted summary + deep-link URL.