tiennm99/thptqg
1
0
Fork 0
mirror of https://github.com/tiennm99/thptqg.git synced 2026-08-14 09:23:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: expand README, add architecture / deployment / data-pipeline docs

Browse Source 
- README: live-site URLs, features, scripts, project layout, quickstart
- docs/system-architecture.md: data flow, 3-variant deploy mechanism,
  schema, parse-quirk matrix, score-tier model, admission-block model,
  frontend key behaviors (deep-link, share, keyboard shortcuts)
- docs/data-pipeline.md: source Excel shape, score regex, why three
  parsers, overflow-sheet gotcha, audit tooling, refresh flow
- docs/deployment-guide.md: CI workflow, adding a new variant, rollback,
  GH Pages file-size note
- docs/README.md: index for the docs dir
This commit is contained in:
tiennm99
2026-04-14 23:25:51 +07:00
parent 1940e8936d
commit 8ca6bfa7a8
5 changed files with 336 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
2017/README.md
+94 -3
View File
@@ -1,4 +1,95 @@
# thptqg2017
Database điểm thi THPT QG 2017
# Tra cứu điểm thi THPT Quốc gia 2017
**Nguồn**: Sưu tầm từ trang báo thời đó
Static React + SQLite site for looking up Vietnamese high school graduation exam scores (2017). The full database (~861k students, 63 provinces) ships to the browser as a compressed SQLite file and queries run client-side via `sql.js` — no backend.
## Live sites
Three deployments, one per dataset:
| URL | Data source | Rows |
|---|---|---|
| `/thptqg2017/` | `data/` — [baotintuc.vn](https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm) CDN, `.xls` | **861,068** |
| `/thptqg2017/old/` | `data-old/` — earlier xlsx collection | 847,348 |
| `/thptqg2017/old2/` | `data-old2/` — partial re-export (54 provinces) | 679,764 |
## Features
- Diacritics-insensitive name search (`"nguyen"` matches `"Nguyễn"`)
- Live debounced search with URL deep-link (`?q=49008235`)
- Single-result detail card with per-subject TFT rarity-tiered scores (6 tiers, ≤1 → 9-10) and all 49 admission blocks (khối A00 – D15)
- Share button: copies formatted summary + URL for Zalo/Messenger
- SQL query tab with grouped presets (rankings, Long An filters, statistics, schema)
- Light + dark mode (follows OS preference)
- Keyboard shortcut `/` to focus search
## Requirements
- Node.js 20+
- npm
## Quickstart
```bash
npm install
npm run build:db # parse data/ into public/thptqg2017.db (~2 min, 159 MB)
gzip -kf -9 public/thptqg2017.db
npm run dev # http://localhost:5173
```
## Scripts
| Command | Action |
|---|---|
| `npm run dev` | Vite dev server |
| `npm run build` | Production build (main variant → `dist/`) |
| `npm run build:old` / `build:old2` | Build variant sites to `dist/old/`, `dist/old2/` |
| `npm run build:all` | All 3 web variants |
| `npm run build:db` | Build main DB from `data/` |
| `npm run build:db:old` / `build:db:old2` | Build old / old2 variant DBs |
| `npm run build:db:all` | All 3 DBs |
| `npm run lint` | ESLint |
| `node scripts/crawl-baotintuc.js` | Re-download all 63 province files from baotintuc.vn |
| `node scripts/audit-row-counts.js` | Verify source row count matches DB row count |
| `node scripts/check-duplicates.js` | MD5 + row-content duplicate audit |
| `node scripts/diff-datasets.js` | Compare `public/` vs `backup/` DB (when backup present) |
## Project layout
```
.
├── data/ # 63 .xls files (source)
├── data-old/ # 63 .xlsx (previous export)
├── data-old2/ # 54 .xlsx (update/ overrides)
├── public/ # main variant assets + thptqg2017.db.gz
├── public-old/ # old variant assets
├── public-old2/ # old2 variant assets
├── scripts/
│ ├── build-lib.js # shared schema + helpers
│ ├── build-database.js # parser for data/
│ ├── build-database-old.js # parser for data-old/
│ ├── build-database-old2.js # parser for data-old2/
│ ├── crawl-baotintuc.js # downloader
│ ├── audit-row-counts.js # parse-loss audit
│ ├── check-duplicates.js # md5 dup detector
│ └── diff-datasets.js # DB-to-DB comparator
├── src/
│ ├── App.jsx
│ ├── App.css
│ ├── components/{search-form, score-table, student-detail, custom-query}.jsx
│ ├── hooks/use-sqlite.js
│ └── lib/admission-blocks.js # 49 khối definitions + score-tier helper
├── docs/ # see docs/README.md
├── index.html
├── vite.config.js
└── package.json
```
See `docs/` for architecture + deployment details.
## License
See `LICENSE`.
## Source
Data collected from public Vietnamese news archives (2017). Intended for reference only.
2017/docs/README.md
+5
View File
@@ -0,0 +1,5 @@
# Docs
- [`system-architecture.md`](./system-architecture.md) — data flow, schema, 3-variant deploy, score-tier model, admission-block catalog
- [`data-pipeline.md`](./data-pipeline.md) — Excel parse quirks, overflow-sheet gotcha, audit scripts, refresh flow
- [`deployment-guide.md`](./deployment-guide.md) — GitHub Pages workflow, adding a new variant, rollback
2017/docs/data-pipeline.md
+88
View File
@@ -0,0 +1,88 @@
# Data Pipeline
From raw Excel files to a compressed SQLite file the browser can load.
## Canonical source
63 Excel files from baotintuc.vn CDN. Re-download anytime:
```bash
node scripts/crawl-baotintuc.js
```
Idempotent — skips files already present. Saves to `data/<ascii-kebab-province>.xls`.
Source article URL: `https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm`
## Source Excel shape
Every file is a single sheet (for `data-old/`) or up to 3 sheets (for `data/` .xls) with columns:
| Col | Name | Content |
|---|---|---|
| 0 | HO_TEN | full name in Vietnamese |
| 1 | NGAY_SINH | `dd/mm/yyyy` |
| 2 | SOBAODANH | 8-digit string, first 2 digits = province |
| 3 | DIEM_THI | concatenated Vietnamese score string, e.g. `"Toán: 6.80 Ngữ văn: 5.25 … Tiếng Anh: 5.80"` |
Some files lack a header row. `isHeaderRow()` in `build-lib.js` detects and skips.
## Score text parsing
`SCORE_PATTERNS` in `build-lib.js` defines one regex per subject:
```js
{ toan: /Toán:\s*(\d+(?:\.\d+)?)/, ngu_van: /Ngữ văn:…/, … }
```
Subjects supported: Toán, Ngữ văn, Vật lí, Hóa học, Sinh học, KHTN, Lịch sử, Địa lí, GDCD, KHXH, Tiếng Anh, Tiếng Pháp, Tiếng Nga, Tiếng Trung.
Missing-in-string → `NULL` in DB (student didn't take that subject).
## Why three parsers
One shared lib + three thin parsers, not one parser with flags. Each dataset's quirks stay visible in its own file:
| File | Source dir | Sheet strategy | Extra guard |
|---|---|---|---|
| `build-database.js` | `data/` | iterate ALL sheets (.xls row overflow) | — |
| `build-database-old.js` | `data-old/` | sheet 0 only | reject rows where `so_bao_danh` isn't digits-only (rejects an `HO_TEN` → `SOBAODANH` header leak present in this export) |
| `build-database-old2.js` | `data-old2/` | iterate ALL sheets (HCM overflow) | skip fully blank rows before counting |
## Overflow-sheet gotcha
The old `.xls` binary format caps at 65,536 rows per sheet. Hà Nội and HCM exceed that in the baotintuc export, so Excel auto-splits the overflow into `Sheet2`. **Only reading `Sheet1` silently drops 13,720 students** (Hà Nội +7,275, HCM +6,445).
The audit `node scripts/audit-row-counts.js` catches this by comparing source-row-count (across all sheets) to DB-row-count.
## Audits
Run any time after a rebuild:
```bash
node scripts/audit-row-counts.js # source rows vs DB rows (zero-loss check)
node scripts/check-duplicates.js # md5 + row-content duplicates in data/
node scripts/diff-datasets.js # compare two DBs (needs backup/ dir)
```
Expected baseline:
| DB | Source rows | Skipped | DB rows |
|---|---|---|---|
| main | 861,131 | 63 empty rows | **861,068** |
| old | 847,349 | 1 header-leak | **847,348** |
| old2 | 679,764 | 0 | **679,764** |
## Refreshing the data
```bash
rm data/*.xls # clear current snapshot
node scripts/crawl-baotintuc.js # re-download from CDN
npm run build:db # rebuild
node scripts/audit-row-counts.js # verify
gzip -kf -9 public/thptqg2017.db # compress for shipping
```
## Adding a new source dataset
See `docs/deployment-guide.md` → "Adding a new variant".
2017/docs/deployment-guide.md
+52
View File
@@ -0,0 +1,52 @@
# Deployment Guide
Deploys to GitHub Pages via `.github/workflows/deploy.yml`. Every push to `main` rebuilds and redeploys all 3 variants.
## What the workflow does
1. Checkout, setup Node 20, `npm ci`
2. `npm run build:db:all` — builds 3 SQLite DBs (main, old, old2)
3. Gzip each DB at level 9
4. `npm run build:all` — builds 3 Vite bundles into `dist/`, `dist/old/`, `dist/old2/`
5. Remove uncompressed `.db` files from `dist/` (only `.gz` ships)
6. `actions/upload-pages-artifact` + `actions/deploy-pages`
Total CI time ≈ 4–6 min (DB build dominates).
## Resulting URLs
- `https://<user>.github.io/thptqg2017/`
- `https://<user>.github.io/thptqg2017/old/`
- `https://<user>.github.io/thptqg2017/old2/`
## Local reproduction
```bash
npm ci
npm run build:db:all
gzip -kf -9 public/thptqg2017.db public-old/thptqg2017.db public-old2/thptqg2017.db
npm run build:all
npx serve dist # or any static server
```
## Adding a new variant
1. Drop source Excel files into a new `data-vN/` folder
2. Add `public-vN/` to `.gitignore` patterns for the `.db` + `.db.gz`
3. Copy `scripts/build-database-old.js` → `scripts/build-database-vN.js`; update `SRC_DIR` / `DB_PATH`; adjust parse logic for the new quirks (multi-sheet? blank rows? numeric guard?)
4. Add variant to `VARIANT_CONFIG` in `vite.config.js`:
```js
vN: { base: "/thptqg2017/vN/", publicDir: "public-vN", outDir: "dist/vN" }
```
5. Add `build:db:vN` and `build:vN` npm scripts; wire them into `build:db:all` and `build:all`
6. Update `.github/workflows/deploy.yml` — add gzip step + rm step for the new DB
## Notes
- **DB gzip is non-cacheable across deploys** — every rebuild produces a new `.db.gz` (not byte-identical due to SQLite page shuffling). First-visit users pay the 47 MB download; subsequent visits hit browser cache until the next deploy.
- **GitHub Pages file size limit**: individual files ≤ 100 MB. Main gzipped DB is ~47 MB — safe. Uncompressed 159 MB DB would not fit; that's why we ship `.gz` and the frontend decompresses via `DecompressionStream`.
- **No server-side compression assumption.** The app reads the `.gz` file directly (not `Content-Encoding: gzip`). GH Pages does not reliably gzip on-the-fly for arbitrary paths; shipping pre-gzipped bytes is deterministic.
## Rollback
Deploys are stateless snapshots. To rollback, revert the commit on `main` and push — the next workflow rebuilds the older state. There's no state to migrate.
2017/docs/system-architecture.md
+97
View File
@@ -0,0 +1,97 @@
# System Architecture
Static site. No backend. Browser downloads a compressed SQLite file at boot, then every query runs locally via `sql.js` (WASM).
## Data flow
```
Excel files (.xls / .xlsx)
│
▼ build-database*.js (Node)
SQLite DB (public*/thptqg2017.db)
│
▼ gzip -9
thptqg2017.db.gz (~47 MB for main variant)
│
▼ Vite build copies public*/ into dist/
Static site on GitHub Pages
│
▼ browser loads
sql.js (WASM) opens the .db → queries run client-side
```
## Three deployment variants
One repo → three independent sites, same frontend, different dataset.
| Variant | Route | Source dir | Public dir | Build cmd |
|---|---|---|---|---|
| main | `/thptqg2017/` | `data/` | `public/` | `npm run build` |
| old | `/thptqg2017/old/` | `data-old/` | `public-old/` | `npm run build:old` |
| old2 | `/thptqg2017/old2/` | `data-old2/` | `public-old2/` | `npm run build:old2` |
`vite.config.js` reads `process.env.VARIANT` and switches `base`, `publicDir`, `outDir` accordingly. `emptyOutDir` is on only for the main build so the variant builds merge into `dist/old/`, `dist/old2/` cleanly.
## Schema (all 3 DBs share this)
```sql
CREATE TABLE student (
so_bao_danh TEXT PRIMARY KEY, -- exam ID (8 digits, first 2 = province)
ho_ten TEXT NOT NULL,
ho_ten_ascii TEXT NOT NULL, -- NFD-stripped lowercase for accent-insensitive search
ngay_sinh TEXT, -- dd/mm/yyyy
toan, ngu_van, vat_ly, hoa_hoc, sinh_hoc, khtn,
lich_su, dia_ly, gdcd, khxh,
tieng_anh, tieng_phap, tieng_nga, tieng_trung REAL
);
CREATE INDEX idx_ho_ten ON student(ho_ten);
CREATE INDEX idx_ho_ten_ascii ON student(ho_ten_ascii);
```
No German / Japanese language columns — neither appears in any source file.
Each score column is `NULL` when the student didn't take that subject. A student's "admission block" total is only computed when all 3 required subjects are non-null.
## Parse quirks (why 3 builder scripts)
Each dataset had a different quirk. Rather than one mega-parser with mode flags, each builder script is ~70 lines and isolates its own workarounds.
| Dataset | Quirk | Mitigation |
|---|---|---|
| data/ (baotintuc .xls) | Hà Nội + HCM overflow past the 65,536 row-per-sheet .xls limit | Iterate all `wb.SheetNames` |
| data-old/ (xlsx) | Single-sheet always; one bogus header row (`SOBAODANH`/`HO_TEN`) leaked into an earlier DB | Strict numeric-SBD guard rejects the leak |
| data-old2/ (xlsx) | HCM overflow + many blank trailing rows | Multi-sheet walk + blank-row pre-filter |
Shared concerns (regex, schema, `toAscii`, header detection) live in `scripts/build-lib.js`.
## Admission blocks (khối thi)
`src/lib/admission-blocks.js` lists all 49 official 2017 blocks computable from our schema (A00–A11, B00–B08, C00–C20, D01–D15). Each entry is `{ code, subjects: [3 keys], label }`.
`computeBlocks(student)` returns only blocks where the student has all 3 subject scores, sorted by total desc. Used by the student detail card.
The SQL preset "Top 10 điểm khối cao nhất – Long An" materialises all 49 blocks as a `UNION ALL` CTE, picks each student's best via `ROW_NUMBER() OVER (PARTITION BY so_bao_danh ORDER BY s DESC, k)`, then ranks across students.
## Score tiers (UI coding)
6-level TFT rarity ladder (white → green → blue → purple → gold → prismatic). Defined in `scoreTier()` in `src/lib/admission-blocks.js`.
| Tier | Range | Label |
|---|---|---|
| common | ≤ 1 | Điểm liệt |
| uncommon | < 5 | Chưa đạt |
| rare | 5–6.5 | Trung bình |
| epic | 6.5–8 | Khá |
| legendary | 8–9 | Giỏi |
| prismatic | 9–10 | Xuất sắc |
Color + icon + text label — never color-only.
## Frontend key behaviors
- Query state lives in `App.jsx` and is synced to URL `?q=...` via `history.replaceState` — bookmarkable, shareable.
- `SearchForm` is a controlled component that 300ms-debounces input changes and auto-triggers search once the query is long enough (3+ digits for SBD, 2+ chars for name).
- Detection: all-digits → exact `so_bao_danh` match; ASCII-only → folded `ho_ten_ascii LIKE`; has Vietnamese diacritics → search both `ho_ten` and folded column.
- Exactly 1 result → `StudentDetail` card; 0 or >1 → `ScoreTable`.
- Global `/` key focuses the search input when not already typing.
- Share button on the detail card uses Web Share API when present, else clipboard with a pre-formatted summary + deep-link URL.