tiennm99/thptqg2017

0 Fork 0

mirror of https://github.com/tiennm99/thptqg2017.git synced 2026-10-03 09:13:53 +00:00

Files

T

tiennm99 f180c662a2 fix(web): read the database in chunked mode so the length can be supplied

The previous attempt passed fileLength in the inline config, and the
worker discarded it. sqlite.worker.ts builds the lazy file's config
itself and hardcodes:

  fileLength: config.serverMode === "chunked"
    ? config.databaseLengthBytes
    : undefined

So in full mode there is no way to supply a length, and the library
falls back to sizing the file with a HEAD request — which GitHub Pages
answers with the gzipped length, and then refuses to use.

Chunked mode is the only mode that takes a length. One chunk holds the
whole database, so the chunk index is always 0 and every request goes to
urlPrefix + "0"; the assembler therefore publishes <id>.sqlite30. The
length still comes from the range probe, which also checks the bytes are
a SQLite header.

dbPrefixOf and dbOf derive one form from the other and are handed to
RemoteDatabase together, so the prefix the library appends an index to
and the file the assembler writes cannot drift apart. A test pins that;
nothing else would catch it, because the symptom is a 404 per query.

The stray-artifact guard now also rejects a leftover <id>.sqlite3, which
after this change is a stale artifact rather than the published one.

sqlite-wasm-http was checked as an alternative and does not help: its
worker takes the size from a HEAD Content-Length too, and its options
expose no way to override it, so on Pages it would silently use the
compressed size. Its shared-cache backend wants COOP/COEP, but it ships
a fallback that does not, so isolation was never the blocker — the
architecture note claiming otherwise is corrected.

2026-08-14 16:01:59 +07:00

5.7 KiB

Raw Blame History

Deployment Guide

Deploys to GitHub Pages via .github/workflows/deploy-pages.yml. Every push to main rebuilds both datasets and redeploys the whole site.

One-time setup: Settings → Pages → Source: GitHub Actions.

What the workflow does

Checkout, Go toolchain, Node 24, npm ci in web/
Parser, crawler and assembler test suites, web lint, govulncheck over all three modules
go -C assembler run ./cmd/assemble — the whole pipeline: compile the parser, build and verify each database, compress it into .build/public/db/, run the web build, assemble _site/
actions/upload-pages-artifact + actions/deploy-pages

The database build dominates the runtime: roughly 348 MB of Excel is parsed on every deploy.

Resulting URLs

https://<user>.github.io/thptqg/
https://<user>.github.io/thptqg/2016/
https://<user>.github.io/thptqg/2017/

/thptqg/2017/old/ and /thptqg/2017/old2/ were the pre-flattening URLs for the two removed 2017 archives. They are no longer served; like any unknown path they now render the hub via 404.html.

Local reproduction

(cd web && npm ci)
go -C assembler run ./cmd/assemble
npx serve _site

To rebuild a single dataset, or only the site:

go -C assembler run ./cmd/assemble db 2017
go -C assembler run ./cmd/assemble site

Base path

svelte.config.js sets paths.base: "/thptqg". If you fork under a different repo name, update it to match — assets are referenced absolutely, so a mismatch shows up as a blank page with 404s on /_app/....

Adding a dataset

Put the Excel files in data/<id>/
Add parser/configs/<id>.yml with the parse rules — sheet mode, column indices, SBD validation, header tokens, blank-row stripping. No SQL: the schema is canonical and lives in parser/internal/schema/schema.go
Add an entry to datasets.json — id, expectedRows, dbSizeMb
Add its presentation to CONTENT in web/src/datasets.js

Nothing else. The assembler and the router both read the registry, and the frontend adapts to whichever columns the dataset populates. The last two steps check each other, so forgetting either fails rather than half-working.

Why no uncompressed database can ship

The assembler deletes the source once compression succeeds, so the raw file does not survive the build, and it then fails the job if any .db, .db-journal, .db-wal or .db-shm reached the output.

Both guards exist because the previous pipeline wrote a 100+ MB uncompressed database into the source tree and relied on an rm step to keep it out of the artifact — one missing line away from publishing it.

Notes

The database is not cacheable across deploys. Every rebuild lays SQLite pages out differently, so the file changes even when the data does not. Only the pages a query touches are fetched, so this costs far less than it used to, but a deploy does invalidate what a returning visitor had cached.
The 100 MB file limit is a Git limit, not a Pages one. It applies to files committed to a repository; the databases are built in CI and uploaded as a Pages artifact, and the documented Pages limits are a 1 GB published site and 100 GB/month of bandwidth, with no per-file figure. The two databases are 302 MB and 247 MB.
Total artifact is about 552 MB, inside the 1 GB site limit but with less headroom than before: a third dataset of this size would not fit. The fallback is sql.js-httpvfs's chunked mode, which splits a database into parts.
Pages does compress the databases, and that is survivable. The extension is unknown to Pages, so the file is served as application/octet-stream, which is marked compressible in mime-db and gzipped: a plain request returns Content-Encoding: gzip and the compressed length. Ranged reads are not affected, because the Fetch standard makes browsers send Accept-Encoding: identity on any request carrying a Range header. Only the length probe breaks, so the site supplies the length itself instead of trusting HEAD — see web/src/lib/db-probe.js and the chunked-mode note in system-architecture.
Verify the way a browser asks. A bare curl -sI advertises no encoding and so reports success whatever the host does; it is what let this reach production. Check ranged reads instead, and check the bytes, not the headers:
```
curl -s -r 0-15 -H 'Accept-Encoding: identity;q=1, *;q=0' \
  https://<user>.github.io/thptqg/db/2016.sqlite30 | head -c 16
# must print: SQLite format 3
```

Rollback

Deploys are stateless snapshots. Revert the commit on main and push; the next run rebuilds the older state. There is no data to migrate.

Troubleshooting

Symptom	Typical cause
Blank page, 404 on assets	`paths.base` in `svelte.config.js` does not match the repo name
`Failed to fetch database: 404`	Dataset id in `datasets.json` does not match the file in `db/`
A route 404s	The site step did not run, or the id is missing from `datasets.json`
`Length of the file not known`	The host gzipped the un-ranged response, so HEAD reports the compressed size. The site supplies `databaseLengthBytes` from a range probe; if this returns, the config is no longer reaching the worker in chunked mode
Database fails to open	A ranged read did not return raw database bytes. The range check above must print `SQLite format 3`
Every query is slow or huge	It is not using an index. `EXPLAIN QUERY PLAN` it: a `SCAN` means the browser is fetching the whole table
Deploy fails on assembly	An uncompressed database artefact reached the output; the error names the files
Missing rows after a data update	Unknown Excel header — check the per-file row counts the parser prints

5.7 KiB Raw Blame History