mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-08-14 01:24:20 +00:00
tiennm99
a0420fdc37
The pipeline is now Go outside web/. differential-parity.mjs becomes
assembler/internal/verify, reachable as `assemble verify A B`. The port fixed a
real weakness: the JavaScript hashed each row's fields joined bare, so a value
shifted across a column boundary produced the same digest. A test now pins that.
The hub still rendered "Phiên bản cũ của trang 2017" above a permanently empty
list — it split datasets on id.includes("old"), and both such datasets are gone.
The heading and the filter are removed. index.html titled every page "THPT QG
2017", including 2016 and the hub, because one file is copied to every route;
the static title is now neutral and the app sets the dataset's own.
Dead code removed: the isOld2/containsOld branches in the stats block, which
only 2017-old2 could ever reach; SUBJECT_LABELS, DATASET_IDS and the unread
`short` subject field; an unused vite.svg and a favicon link to a file that
never existed; two unused CSS rules and --shadow-sm; site.Paths.Root.
Corrected comments that were confidently wrong rather than merely stale: the
reader claimed to be row-streaming when both implementations decode the whole
workbook into memory first, and the fidelity oracle still spoke of 299 input
files when it covers 182. Candidate counts in the hub now derive from
datasets.json instead of being written a second time as prose.
plans/ is emptied. The parity report it held was cited by docs/data-pipeline.md,
so the evidence that the recovered foreign-language scores are real — not the
citation, the four arguments themselves — is now inline there.
Verified: 2017 rebuilt after the writer change hashes identically to the build
before it.
105 lines
4.2 KiB
Markdown
105 lines
4.2 KiB
Markdown
# Deployment Guide
|
||
|
||
Deploys to GitHub Pages via `.github/workflows/deploy-pages.yml`. Every push to
|
||
`main` rebuilds both datasets and redeploys the whole site.
|
||
|
||
One-time setup: **Settings → Pages → Source: GitHub Actions**.
|
||
|
||
## What the workflow does
|
||
|
||
1. Checkout, Go toolchain, Node 24, `npm ci` in `web/`
|
||
2. Parser, crawler and assembler test suites, web lint, `govulncheck` over all three modules
|
||
3. `go -C assembler run ./cmd/assemble` — the whole pipeline: compile the
|
||
parser, build and verify each database, compress it into `.build/public/db/`,
|
||
run the Vite build, assemble `_site/`
|
||
4. `actions/upload-pages-artifact` + `actions/deploy-pages`
|
||
|
||
The database build dominates the runtime: roughly 348 MB of Excel is parsed on
|
||
every deploy.
|
||
|
||
## Resulting URLs
|
||
|
||
```
|
||
https://<user>.github.io/thptqg/
|
||
https://<user>.github.io/thptqg/2016/
|
||
https://<user>.github.io/thptqg/2017/
|
||
```
|
||
|
||
`/thptqg/2017/old/` and `/thptqg/2017/old2/` were the pre-flattening URLs for
|
||
the two removed 2017 archives. They are no longer served; like any unknown path
|
||
they now render the hub via `404.html`.
|
||
|
||
## Local reproduction
|
||
|
||
```bash
|
||
(cd web && npm ci)
|
||
go -C assembler run ./cmd/assemble
|
||
npx serve _site
|
||
```
|
||
|
||
To rebuild a single dataset, or only the site:
|
||
|
||
```bash
|
||
go -C assembler run ./cmd/assemble db 2017
|
||
go -C assembler run ./cmd/assemble site
|
||
```
|
||
|
||
## Base path
|
||
|
||
`vite.config.js` sets `base: "/thptqg/"`. If you fork under a different repo
|
||
name, update it to match — assets are referenced absolutely, so a mismatch shows
|
||
up as a blank page with 404s on `/assets/...`.
|
||
|
||
## Adding a dataset
|
||
|
||
1. Put the Excel files in `data/<id>/`
|
||
2. Add `parser/configs/<id>.yml` with the parse rules — sheet mode, column
|
||
indices, SBD validation, header tokens, blank-row stripping. No SQL: the
|
||
schema is canonical and lives in `parser/internal/schema/schema.go`
|
||
3. Add an entry to `datasets.json` — id, `expectedRows`, `dbSizeMb`
|
||
4. Add its presentation to `CONTENT` in `web/src/datasets.js`
|
||
|
||
Nothing else. The assembler and the router both read the registry, and the
|
||
frontend adapts to whichever columns the dataset populates. The last two steps
|
||
check each other, so forgetting either fails rather than half-working.
|
||
|
||
## Why no uncompressed database can ship
|
||
|
||
The assembler deletes the source once compression succeeds, so the raw file
|
||
does not survive the build, and it then fails the job if any `.db`,
|
||
`.db-journal`, `.db-wal` or `.db-shm` reached the output.
|
||
|
||
Both guards exist because the previous pipeline wrote a 100+ MB uncompressed
|
||
database into the source tree and relied on an `rm` step to keep it out of the
|
||
artifact — one missing line away from publishing it.
|
||
|
||
## Notes
|
||
|
||
- **The gzipped database is not cacheable across deploys.** Every rebuild
|
||
produces a different `.db.gz`, because SQLite does not lay pages out
|
||
deterministically. First-visit users pay the full download; later visits hit
|
||
browser cache until the next deploy.
|
||
- **GitHub Pages caps individual files at 100 MB.** The largest gzipped database
|
||
is about 48 MB. Uncompressed they run 135–234 MB and would not fit — which is
|
||
why the browser decompresses via `DecompressionStream`.
|
||
- **No server-side compression is assumed.** The app fetches the `.gz` bytes
|
||
directly rather than relying on `Content-Encoding: gzip`; Pages does not
|
||
reliably compress arbitrary paths on the fly.
|
||
- **Total artifact is about 93 MB**, well inside the 1 GB site limit.
|
||
|
||
## Rollback
|
||
|
||
Deploys are stateless snapshots. Revert the commit on `main` and push; the next
|
||
run rebuilds the older state. There is no data to migrate.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Typical cause |
|
||
| --- | --- |
|
||
| Blank page, 404 on assets | `base` in `vite.config.js` does not match the repo name |
|
||
| `Failed to fetch database: 404` | Dataset id in `datasets.json` does not match the file in `db/` |
|
||
| A route 404s | The site step did not run, or the id is missing from `datasets.json` |
|
||
| WASM fails to load | `sql.js.org` unreachable — self-host `sql-wasm.wasm` and update `SQL_WASM_URL` in `use-sqlite.js` |
|
||
| Deploy fails on assembly | An uncompressed database artefact reached the output; the error names the files |
|
||
| Missing rows after a data update | Unknown Excel header — check the per-file row counts the parser prints |
|