mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-04 18:13:58 +00:00
tiennm99
b38965dc4b
Every .ts becomes .js, every lang="ts" becomes lang-less, and the type declarations go with them: types.ts held nothing but types, so it is deleted outright. Tooling follows. typescript, svelte-check, typescript-eslint and @types/sql.js are uninstalled; tsconfig.json becomes jsconfig.json, which still extends the generated SvelteKit config so $lib and $app resolve in an editor; `npm run lint` is now ESLint alone, and CI's comment about it covering the type check goes too. What this gives up, stated plainly: a mistyped column name like row.nguvan used to fail the build and now renders blank, and the datasets.json-to-CONTENT cross-check is back to throwing at module load rather than at compile time. The runtime guard for the latter is still there and still throws loudly. Two mechanical notes. The svelte/no-navigation-without-resolve rule started flagging the footer's source link, which points at an off-site article — without type information the rule can no longer tell an external URL from a route, so that one line carries a disable comment. And Vitest's include pattern had to follow the tests to .js. Lint, 25 tests and the build all pass.
199 lines
8.8 KiB
Markdown
199 lines
8.8 KiB
Markdown
# System Architecture
|
||
|
||
Static site, no backend. The SQLite file stays on the server and the browser
|
||
reads the pages a query touches over HTTP range requests, via `sql.js-httpvfs`
|
||
(SQLite compiled to WebAssembly behind a virtual file system). A lookup costs a
|
||
few hundred KB; nothing downloads the database.
|
||
|
||
One frontend, one parser, one schema, two datasets.
|
||
|
||
## Data flow
|
||
|
||
Each stage is a directory; `data/` and `_site/` are the stores they hand work
|
||
through. `assembler/` sequences everything from the parser onwards.
|
||
|
||
```
|
||
▲ crawler/ (Go — manual refresh only, never part of the build)
|
||
data/<id>/*.xls(x)
|
||
│
|
||
▼ parser/ (Go, one binary, one config per dataset)
|
||
.build/public/db/<id>.sqlite3
|
||
│
|
||
▼ assembler/ — row count and size must match datasets.json
|
||
.build/public/db/<id>.sqlite3 (uncompressed: ranges of a gzip stream
|
||
│ are not ranges of the database)
|
||
▼ assembler/ → npm run build (SvelteKit static, assets = .build/public)
|
||
web/dist/
|
||
│
|
||
▼ assembler/ — one index.html per dataset; every database must be present
|
||
_site/ → GitHub Pages
|
||
│
|
||
▼ browser
|
||
sql.js-httpvfs asks for pages → HTTP range requests → results client-side
|
||
```
|
||
|
||
## The dataset id
|
||
|
||
One identifier ties the whole pipeline together:
|
||
|
||
```
|
||
data/2017/ → parser/configs/2017.yml → db/2017.sqlite3 → /thptqg/2017/
|
||
```
|
||
|
||
`datasets.json` at the repository root declares the ids once, with the row count
|
||
and artifact size the assembler enforces. It is JSON rather than a module
|
||
because the assembler is a Go program and the web app is not, and JSON is the
|
||
only format both parse without a dependency.
|
||
|
||
Presentation — titles, labels, search examples, SQL presets — stays in
|
||
`web/src/lib/datasets.js`, keyed by id. That file cross-checks the two: a registry
|
||
entry with no content, or content for a dataset that was never built, throws at
|
||
module load rather than rendering a page with no title or a link to a database
|
||
that does not exist.
|
||
|
||
| id | Exam | Rows | Source |
|
||
| --- | --- | --- | --- |
|
||
| `2016` | 2016 | 877,460 | `dtnt.bacninh.edu.vn` |
|
||
| `2017` | 2017 | 861,068 | `baotintuc.vn` |
|
||
|
||
Full source URLs are in [data-pipeline](./data-pipeline.md#sources); the web
|
||
footer links to them per dataset.
|
||
|
||
## Canonical schema
|
||
|
||
Defined once in `parser/internal/schema/schema.go` — DDL, INSERT, column order and the 16
|
||
subject regexes. The two YAML configs carry no SQL at all, only per-dataset
|
||
parse rules. Config parsing sets `KnownFields(true)`, so a leftover `schema:`
|
||
block fails loudly instead of looking effective while `schema.go` drives the
|
||
build.
|
||
|
||
```sql
|
||
CREATE TABLE student (
|
||
so_bao_danh TEXT PRIMARY KEY, -- 2017: 8 digits; 2016: 9 digits or a
|
||
-- 2-4 letter cluster code then digits
|
||
ho_ten TEXT NOT NULL,
|
||
ho_ten_ascii TEXT NOT NULL, -- NFD-stripped lowercase, for accent-insensitive search
|
||
ngay_sinh TEXT, -- dd/mm/yyyy
|
||
ten_cum_thi TEXT, -- 2016 only
|
||
gioi_tinh TEXT, -- 2016 only
|
||
toan, ngu_van, vat_ly, hoa_hoc, sinh_hoc, khtn,
|
||
lich_su, dia_ly, gdcd, khxh,
|
||
tieng_anh, tieng_phap, tieng_nga, tieng_duc, tieng_nhat, tieng_trung REAL
|
||
);
|
||
CREATE INDEX idx_ho_ten ON student(ho_ten);
|
||
CREATE INDEX idx_ho_ten_ascii ON student(ho_ten_ascii);
|
||
CREATE INDEX idx_ten_cum_thi ON student(ten_cum_thi) WHERE ten_cum_thi IS NOT NULL;
|
||
```
|
||
|
||
Every dataset gets all 22 columns; ones it has no data for are NULL, costing
|
||
about a byte per row. `khtn`, `khxh` and `gdcd` are empty on 2016;
|
||
`ten_cum_thi` and `gioi_tinh` are empty on 2017.
|
||
|
||
`idx_ten_cum_thi` is partial, so it holds zero entries where the column is
|
||
always NULL.
|
||
|
||
## Routing
|
||
|
||
URLs are flat, one segment per dataset, and the segment is the id:
|
||
|
||
```
|
||
/thptqg/ hub
|
||
/thptqg/2016/
|
||
/thptqg/2017/
|
||
```
|
||
|
||
The route is `web/src/routes/[dataset]/`, and its entry generator reads the same
|
||
`datasets.json` the assembler does, so the set of pages and the set of databases
|
||
cannot drift apart. Unknown paths fall through to the hub.
|
||
|
||
SvelteKit prerenders one HTML file per route, each with its own `<title>`.
|
||
Asset URLs stay absolute (`paths.relative: false`), so the copy of the hub that
|
||
serves as `404.html` resolves its assets from any depth. **No SPA 404-fallback
|
||
redirect is used** — the usual hack rewrites URLs and would interfere with the
|
||
deep links.
|
||
|
||
## Serving both exam years without branching
|
||
|
||
No component contains a per-dataset conditional. Two mechanisms do the work:
|
||
|
||
- **All-NULL columns are hidden.** `score-table.svelte` drops any column where
|
||
every row in the result set is NULL, so 2016 rows surface Cụm thi / GT / Đức /
|
||
Nhật and 2017 rows surface KHTN / KHXH / GDCD / Nga.
|
||
- **Incomplete admission blocks are skipped.** `computeBlocks()` only returns a
|
||
block when the student has all three subjects, so one block list covers both
|
||
years: GDCD blocks self-exclude on 2016, German and Japanese blocks
|
||
self-exclude wherever those languages were not sat.
|
||
|
||
Anything genuinely per-dataset — title, source, database size, search examples,
|
||
SQL presets — lives in `web/src/lib/datasets.js`.
|
||
|
||
## Exam ID formats
|
||
|
||
`web/src/lib/query-mode.js` decides whether a query is an exam ID or a name, and
|
||
is shared by the dataset page and `search-form.svelte` (they previously held
|
||
separate copies and had drifted apart on exactly this rule). `query-mode.test.js`
|
||
covers every form in the table below.
|
||
|
||
| Form | Example | Where |
|
||
| --- | --- | --- |
|
||
| 8 digits | `49008235` | 2017 — first two digits are the province |
|
||
| 9 digits with leading zero | `017006021` | 2016 |
|
||
| 2-4 letters then digits | `BAL000001` | 2016 — exam cluster code |
|
||
|
||
The letter-prefixed form is the majority case for 2016: 624,424 of 877,460
|
||
candidates (71.2%); the remaining 253,036 are all 9-digit. Letter prefixes are
|
||
upper-cased before lookup, so `bal000001` resolves.
|
||
|
||
## Score tiers
|
||
|
||
Six-level ladder in `scoreTier()` (`web/src/lib/admission-blocks.js`), paired with a
|
||
symbol so meaning is never colour-only.
|
||
|
||
| Tier | Range | Vietnamese |
|
||
| --- | --- | --- |
|
||
| common | ≤ 1 | Điểm liệt |
|
||
| uncommon | < 5 | Chưa đạt |
|
||
| rare | 5–6.5 | Trung bình |
|
||
| epic | 6.5–8 | Khá |
|
||
| legendary | 8–9 | Giỏi |
|
||
| prismatic | 9–10 | Xuất sắc |
|
||
|
||
## Admission blocks
|
||
|
||
Vietnamese universities admit on three-subject combinations (khối thi).
|
||
`web/src/lib/admission-blocks.js` lists the blocks computable from this schema
|
||
(A00–A11, B00–B08, C00–C20, D01–D15, plus D05/D06 for German and Japanese).
|
||
`computeBlocks(student)` returns those where all three scores exist, sorted by
|
||
total descending.
|
||
|
||
## Design decisions
|
||
|
||
| Concern | Choice | Rationale |
|
||
| --- | --- | --- |
|
||
| Storage | Static SQLite file, read by range request | No backend; the datasets are frozen, and a lookup needs a few pages of them |
|
||
| Compression | None | A byte range of a gzip stream is not a byte range of the database |
|
||
| WASM hosting | Bundled with the app | `sql.js-httpvfs` ships its own build; one less third-party runtime dependency |
|
||
| Diacritics search | Pre-computed `ho_ten_ascii`, indexed word by word | `LOWER(REPLACE(...))` at query time defeats the index, and `LIKE '%x%'` reads the whole table |
|
||
| Row count in the footer | Read from `datasets.json` | `COUNT(*)` scans an index — 20 MB over range requests |
|
||
| Page size | 1 KiB, matched by `requestChunkSize` | One HTTP request is one page; a row fetched by seek costs 1 KB rather than 4 KB, for about 5% more file |
|
||
| SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes cannot persist; the allowlist prevents confusion |
|
||
| Row caps | 100 (lookup), 1000 (SQL) | Keeps DOM render sizes reasonable |
|
||
| Routing | SvelteKit file routes, prerendered | Each dataset gets a real HTML file with its own title |
|
||
| Styling | Tailwind, with tier colours as CSS variables | Tier classes are chosen at runtime, which no utility generator can see |
|
||
|
||
## Risks and limitations
|
||
|
||
- **Unindexed queries are expensive.** The SQL tab can express a query that
|
||
walks the table, which over range requests means fetching 100+ MB. A byte
|
||
budget stops one before it gets that far, and the tab warns before it opens.
|
||
- **`Content-Encoding` breaks everything.** If the host ever compresses
|
||
`<id>.sqlite3` on the wire, ranges address compressed bytes and
|
||
`sql.js-httpvfs` refuses to open the file. Verify after a deploy:
|
||
`curl -sI …/db/2016.sqlite3` must show no `content-encoding`.
|
||
- **`sql.js-httpvfs` is unmaintained** (0.8.12, September 2022) and ships its
|
||
own SQLite WASM. `sqlite-wasm-http`, on the official build, is the fallback.
|
||
- **Hosted size.** 552 MB for both datasets against the 1 GB GitHub Pages
|
||
limit; a third dataset of this size would not fit.
|
||
- **Excel format drift.** A new source file with an unseen header layout needs a
|
||
new branch in `parser/internal/ingest/detect2016.go` or a new config.
|