mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-08-24 11:25:10 +00:00
perf(parser): write the databases with 1 KiB pages
The browser fetches this file one page per HTTP request, so the page size is the granularity of every read. At SQLite's 4 KiB default a row reached by an index seek dragged 4 KB across the network; at 1 KiB it drags 1 KB. A name search returns up to 100 scattered rows, so its row fetches fall from about 400 KB to about 100 KB. Measured on the rebuilt 2016 file: 6.3 rows share a page where 27 did. The index walks are sequential and unaffected in bytes — the library's read-ahead already collapses those into few requests. Cost is 4% file size: 2016 288.6 -> 302.4 MB, 2017 237.7 -> 247.3 MB, the site 528 -> 552 MB against the 1 GB GitHub Pages limit. Both sql.js-httpvfs and sqlite-wasm-http recommend this page size. The PRAGMA has to run before the DDL, since a page size is fixed once a table exists, and requestChunkSize on the client has to match or every page read spans two requests. Row counts unchanged and through the assembler guards; query plans re-checked and still index-driven on the rebuilt files.
This commit is contained in:
+2
-2
@@ -20,12 +20,12 @@
|
||||
{
|
||||
"id": "2016",
|
||||
"expectedRows": 877460,
|
||||
"dbSizeMb": 289
|
||||
"dbSizeMb": 302
|
||||
},
|
||||
{
|
||||
"id": "2017",
|
||||
"expectedRows": 861068,
|
||||
"dbSizeMb": 238
|
||||
"dbSizeMb": 247
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -181,6 +181,10 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what
|
||||
|
||||
## Expected row counts
|
||||
|
||||
The databases are written with 1 KiB pages (`PRAGMA page_size` in
|
||||
`parser/internal/writer/writer.go`) because the browser fetches them a page per
|
||||
HTTP request. `CHUNK_BYTES` in `web/src/lib/sqlite.svelte.ts` must match.
|
||||
|
||||
| id | Source rows | Skipped | DB rows |
|
||||
| --- | --- | --- | --- |
|
||||
| `2016` | 877,460 | 0 | **877,460** |
|
||||
|
||||
@@ -83,8 +83,8 @@ artifact — one missing line away from publishing it.
|
||||
files committed to a repository; the databases are built in CI and uploaded
|
||||
as a Pages artifact, and the documented Pages limits are a 1 GB published
|
||||
site and 100 GB/month of bandwidth, with no per-file figure. The two
|
||||
databases are 289 MB and 238 MB.
|
||||
- **Total artifact is about 528 MB**, inside the 1 GB site limit but with less
|
||||
databases are 302 MB and 247 MB.
|
||||
- **Total artifact is about 552 MB**, inside the 1 GB site limit but with less
|
||||
headroom than before: a third dataset of this size would not fit. The fallback
|
||||
is `sql.js-httpvfs`'s chunked mode, which splits a database into parts.
|
||||
- **The server must not compress the databases.** Ranges of a compressed body
|
||||
|
||||
@@ -175,6 +175,7 @@ total descending.
|
||||
| WASM hosting | Bundled with the app | `sql.js-httpvfs` ships its own build; one less third-party runtime dependency |
|
||||
| Diacritics search | Pre-computed `ho_ten_ascii`, indexed word by word | `LOWER(REPLACE(...))` at query time defeats the index, and `LIKE '%x%'` reads the whole table |
|
||||
| Row count in the footer | Read from `datasets.json` | `COUNT(*)` scans an index — 20 MB over range requests |
|
||||
| Page size | 1 KiB, matched by `requestChunkSize` | One HTTP request is one page; a row fetched by seek costs 1 KB rather than 4 KB, for about 5% more file |
|
||||
| SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes cannot persist; the allowlist prevents confusion |
|
||||
| Row caps | 100 (lookup), 1000 (SQL) | Keeps DOM render sizes reasonable |
|
||||
| Routing | SvelteKit file routes, prerendered | Each dataset gets a real HTML file with its own title |
|
||||
@@ -191,7 +192,7 @@ total descending.
|
||||
`curl -sI …/db/2016.sqlite3` must show no `content-encoding`.
|
||||
- **`sql.js-httpvfs` is unmaintained** (0.8.12, September 2022) and ships its
|
||||
own SQLite WASM. `sqlite-wasm-http`, on the official build, is the fallback.
|
||||
- **Hosted size.** 528 MB for both datasets against the 1 GB GitHub Pages
|
||||
- **Hosted size.** 552 MB for both datasets against the 1 GB GitHub Pages
|
||||
limit; a third dataset of this size would not fit.
|
||||
- **Excel format drift.** A new source file with an unseen header layout needs a
|
||||
new branch in `parser/internal/ingest/detect2016.go` or a new config.
|
||||
|
||||
@@ -43,6 +43,19 @@ func OpenDB(dbPath string) (*sql.DB, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open db %s: %w", dbPath, err)
|
||||
}
|
||||
// Before the DDL, because a page size cannot change once a table exists —
|
||||
// only the VACUUM in Finish could rewrite it, and only to this same value.
|
||||
//
|
||||
// 1 KiB rather than SQLite's 4 KiB default because the browser reads this
|
||||
// file a page at a time over HTTP: a row fetched by index seek costs one
|
||||
// page, so a search that returns 100 scattered rows transfers 100 KB
|
||||
// instead of 400 KB. It costs about 5% file size, and both sql.js-httpvfs
|
||||
// and sqlite-wasm-http recommend it. web/src/lib/sqlite.svelte.ts must
|
||||
// request the same size.
|
||||
if _, err := db.Exec("PRAGMA page_size = 1024"); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("set page size: %w", err)
|
||||
}
|
||||
if _, err := db.Exec(schema.DDL); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("execute DDL: %w", err)
|
||||
|
||||
@@ -35,12 +35,16 @@ finds "Nguyễn Bửu Lộc", in a few hundred KB.
|
||||
|
||||
| Segment | 2016 |
|
||||
| --- | --- |
|
||||
| `student` | 127 MB |
|
||||
| `name_word` | 97 MB |
|
||||
| `idx_ten_cum_thi` | 37 MB |
|
||||
| PK autoindex | 15 MB |
|
||||
| score indexes | 12 MB |
|
||||
| **total** | **288.6 MB** (2017: 237.7 MB) |
|
||||
| `student` | 137.5 MB |
|
||||
| `name_word` | 98.7 MB |
|
||||
| `idx_ten_cum_thi` | 38.1 MB |
|
||||
| PK autoindex | 15.3 MB |
|
||||
| `idx_toan` | 12.6 MB |
|
||||
| **total** | **302.4 MB** (2017: 247.3 MB) |
|
||||
|
||||
Written with 1 KiB pages, so a row reached by an index seek costs one 1 KB
|
||||
request instead of 4 KB: 6.3 rows share a page rather than 27, which is what
|
||||
turns a 100-row search from ~400 KB of row fetches into ~100 KB.
|
||||
|
||||
**Assembler.** Publishes uncompressed; the size guard reads the raw size; the
|
||||
stray-artifact check now rejects journals, `.db` and `.gz`.
|
||||
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
# Brainstorm: which httpvfs best practices to adopt
|
||||
|
||||
2026-08-14 13:17. Follows
|
||||
[the research report](./web-sqlite-range-hosting-research-260814-1317-httpvfs-best-practices-report.md).
|
||||
Branch `feat/httpvfs-range-queries`.
|
||||
|
||||
## Problem
|
||||
|
||||
Research surfaced three upstream practices we do not follow. Decide which are
|
||||
worth the change, knowing nothing can be verified in a browser on this machine.
|
||||
|
||||
## Codebase context (scout)
|
||||
|
||||
| Concern | Touch points |
|
||||
| --- | --- |
|
||||
| Page size | `parser/internal/writer/writer.go:41-47` (PRAGMA must precede the DDL — page size is fixed once a table exists), `:185` (VACUUM already applies it), `web/src/lib/sqlite.svelte.ts:18,53` |
|
||||
| Chunked mode | `databases.go:41` Extension, size guard, **`Clean()` deletes anything not `<id>.sqlite3` — would eat every chunk**, `site.go:135-138,153`, `datasets.ts:91-97`, `datasets.json`, ~15 tests |
|
||||
| Library swap | `sqlite.svelte.ts:1-3,50-58,76,82`, `package.json:15`; all consumers go through `RemoteDatabase`, so the blast radius is one file |
|
||||
|
||||
## Options evaluated
|
||||
|
||||
### A. page_size 1024 + requestChunkSize 1024 — ADOPTED
|
||||
|
||||
- Upstream consensus: phiresky and mmomtchev both recommend 1024.
|
||||
- Honest sizing for *our* pattern: row fetches 400 KB → 100 KB per search;
|
||||
the index walk is sequential so bytes are unchanged and only the request
|
||||
count rises, which prefetch read-heads collapse. Net ≈ 300 KB saved per
|
||||
search — bandwidth, not latency.
|
||||
- Cost: ~10 lines; file size +5% (528 → ~555 MB total, still under the 1 GB
|
||||
Pages limit); both databases rebuilt.
|
||||
|
||||
### B. serverMode chunked — REJECTED
|
||||
|
||||
- Only benefit is CDN cache efficiency. GitHub Pages serves everything with
|
||||
`Cache-Control: max-age=600`, and each deploy relays out SQLite pages anyway,
|
||||
so cross-deploy caching is zero either way.
|
||||
- Cost: split step, config JSON, `Clean()`/guards/`dbOf()`/`datasets.json`
|
||||
rework, ~15 tests.
|
||||
- Complexity buying a benefit the host cancels. Revisit only behind a CDN with
|
||||
long TTLs.
|
||||
|
||||
### C. swap to sqlite-wasm-http — DEFERRED
|
||||
|
||||
- For: maintained (Dec 2025), official SQLite WASM instead of a 2022 fork;
|
||||
matches this repo's posture on stale dependencies. Swap is one file.
|
||||
- Against: its differentiator (shared cache) needs COOP/COEP headers GitHub
|
||||
Pages cannot send, so we would get the synchronous fallback and the
|
||||
maintenance benefit only. And the current integration has never run in a
|
||||
browser — swapping now means two unverified variables and no way to tell
|
||||
which broke.
|
||||
- Revisit after the current build is verified live.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt **A only**.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
1. `PRAGMA page_size = 1024` in `writer.OpenDB`, between `sql.Open` and
|
||||
`db.Exec(schema.DDL)`. The existing VACUUM in `Finish` applies it.
|
||||
2. `CHUNK_BYTES = 1024` in `web/src/lib/sqlite.svelte.ts`; it feeds
|
||||
`requestChunkSize` and must equal the page size.
|
||||
3. Rebuild both databases, update `dbSizeMb` in `datasets.json` to the new
|
||||
sizes, re-run the assembler guards.
|
||||
|
||||
## Risks
|
||||
|
||||
- The benefit is arithmetic plus upstream authority, not measurement. The byte
|
||||
counter in the SQL tab is the check, once deployed.
|
||||
- Prefetch deliberately overfetches ahead of the cursor, so the 25 MB search
|
||||
budget may trip earlier than a strict page count suggests. Tune after a real
|
||||
measurement, not before.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Does 1024 actually beat 4096 for our queries in a browser?
|
||||
2. Is Fastly's caching of ranges over a 300 MB object good enough that chunked
|
||||
mode stays unnecessary?
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
# Research Report: sql.js-httpvfs best practices
|
||||
|
||||
Conducted 2026-08-14 13:17 (Asia/Saigon). Context: two static SQLite files
|
||||
(2016 = 288.6 MB, 2017 = 237.7 MB) on GitHub Pages, branch
|
||||
`feat/httpvfs-range-queries`.
|
||||
|
||||
## Executive summary
|
||||
|
||||
Our implementation matches upstream guidance on the thing that matters most —
|
||||
index design — and diverges on one measurable parameter: **page size**. Both
|
||||
phiresky (sql.js-httpvfs) and mmomtchev (sqlite-wasm-http) recommend
|
||||
`page_size = 1024`; we shipped 4096. For our access pattern (~100 scattered
|
||||
single-row reads per search) that is a real 4× overfetch on the row-fetch half
|
||||
of a query.
|
||||
|
||||
Two findings reduce risk rather than add work. Prefetching with three virtual
|
||||
read heads makes a sequential scan cost a *logarithmic* number of requests, so
|
||||
our byte estimates hold but latency is better than assumed. And the canonical
|
||||
demo hosts a **670 MiB** database on GitHub Pages, so 289 MB is precedented.
|
||||
|
||||
One finding is new and worth a decision: **chunked mode** (split file + JSON
|
||||
config) exists specifically to make CDN caching effective for large databases,
|
||||
which matters because GitHub Pages serves everything with `Cache-Control:
|
||||
max-age=600`.
|
||||
|
||||
## Methodology
|
||||
|
||||
- Sources: 5 (1 primary blog, 2 project READMEs, 1 recent practitioner
|
||||
writeup, 1 search on Pages/Fastly caching), plus direct reading of the
|
||||
installed `sql.js-httpvfs@0.8.12` bundle in a previous session.
|
||||
- Date range: 2021 (canonical post) → March 2026 (practitioner writeup).
|
||||
- Gemini CLI absent → WebSearch/WebFetch.
|
||||
|
||||
## Key findings
|
||||
|
||||
### 1. Page size: recommended 1024, we use 4096
|
||||
|
||||
Both projects say the same thing. phiresky set 1 KiB pages "to balance request
|
||||
overhead against bandwidth efficiency"; sqlite-wasm-http says "it is highly
|
||||
recommended to decrease your SQLite page size to 1024 bytes for maximum
|
||||
performance" (`PRAGMA page_size=1024; VACUUM`).
|
||||
|
||||
`requestChunkSize` must match the page size.
|
||||
|
||||
Measured on our 2016 file *before* the schema change:
|
||||
|
||||
| page_size | file size |
|
||||
| --- | --- |
|
||||
| 1024 | 235.8 MB |
|
||||
| 4096 | 223.5 MB |
|
||||
| 8192 | 221.8 MB |
|
||||
|
||||
So 1024 costs ~5.5% file size. What it buys: a scattered row read fetches 1 KB
|
||||
instead of 4 KB. Our search does ~100 of those, so the row-fetch half of a
|
||||
search drops from ~400 KB to ~100 KB. The index-walk half is sequential and
|
||||
benefits from prefetch either way.
|
||||
|
||||
**Verdict: switch to 1024 + `requestChunkSize: 1024`.** Our workload is
|
||||
dominated by scattered single-row reads, which is exactly the case small pages
|
||||
serve.
|
||||
|
||||
### 2. Prefetch changes request count, not bytes
|
||||
|
||||
"Three separate virtual read heads" detect sequential access and grow request
|
||||
sizes exponentially, so "index scans or table scans reading more than a few KiB
|
||||
of data will only cause a number of requests that is logarithmic in the total
|
||||
byte length."
|
||||
|
||||
Consequence for our analysis: a full table scan still transfers ~127 MB (bytes
|
||||
are bytes), but in tens of requests rather than tens of thousands. Our byte
|
||||
budget is the right guardrail; a request-count budget would not be.
|
||||
|
||||
### 3. Index design — we already comply
|
||||
|
||||
- Covering indexes: put every column the query needs *in* the index, else
|
||||
SQLite does "another random access (unpredictable) read and thus HTTP request
|
||||
to retrieve the actual value for every data point". This is exactly why
|
||||
`name_word` carries `ho_ten_ascii`.
|
||||
- Column order decides which lookups are cheap.
|
||||
- Verify with `EXPLAIN QUERY PLAN`; a `SCAN` means the whole table crosses the
|
||||
network. We did this for every query the app issues.
|
||||
|
||||
### 4. Chunked mode exists for CDN caching
|
||||
|
||||
`serverMode: "chunked"` splits the database into parts (10 MB is the commonly
|
||||
cited size) with a JSON config. Stated benefit: "CDN caching much more
|
||||
effective" for large databases.
|
||||
|
||||
Relevant because **GitHub Pages sets `Cache-Control: max-age=600`** — ten
|
||||
minutes — on everything. Range responses are cached by Fastly per object; with
|
||||
one 289 MB object the practical caching story is weaker than with 29 chunks
|
||||
that a CDN edge can hold whole. Chunked mode also sidesteps any future per-file
|
||||
concern.
|
||||
|
||||
Cost: a build step to split files + emit config, and every deploy invalidates
|
||||
all chunks anyway (SQLite page layout is not deterministic across rebuilds).
|
||||
|
||||
### 5. Hosting facts confirmed
|
||||
|
||||
- Range requests work on Pages "out of the box" — matches our own probe (206 +
|
||||
correct `Content-Range`).
|
||||
- CORS headers (`Access-Control-Allow-Origin`, `Access-Control-Allow-Headers:
|
||||
Range`) only matter cross-origin. Ours is same-origin — non-issue.
|
||||
- 670 MiB database on Pages is the canonical demo. 289 MB is not exotic.
|
||||
- `Content-Encoding` remains the one fatal case: the library discards
|
||||
`Content-Length` and throws when a HEAD carries a non-identity encoding.
|
||||
Unknown extensions like `.sqlite3` are served `application/octet-stream` and
|
||||
left alone.
|
||||
|
||||
### 6. Alternatives
|
||||
|
||||
| | sql.js-httpvfs | sqlite-wasm-http |
|
||||
| --- | --- | --- |
|
||||
| WASM base | own sql.js fork (~3.36 era) | official `@sqlite.org/sqlite-wasm` |
|
||||
| Last release | 0.8.12, Sept 2022 | 1.2.0, Dec 2023; activity into Dec 2025 |
|
||||
| Self-description | "demo-level code… not for high stability" | "experimental" |
|
||||
| Concurrency | one worker | multiple connections, shared cache |
|
||||
| Shared cache needs | — | `SharedArrayBuffer` → COOP/COEP headers |
|
||||
| On GitHub Pages | works | works, but **Pages cannot set COOP/COEP**, so it falls back to the synchronous backend without shared cache |
|
||||
| Module format | CJS+ESM | **ES6 only** |
|
||||
|
||||
Both are self-declared experimental. sqlite-wasm-http's advantage (maintained,
|
||||
official WASM) is real; its headline feature (shared cache) is unavailable on
|
||||
Pages precisely because Pages cannot send cross-origin isolation headers.
|
||||
|
||||
## Implementation recommendations
|
||||
|
||||
1. **Change page size to 1024 and `requestChunkSize` to 1024.** Parser sets
|
||||
`PRAGMA page_size=1024` before DDL; VACUUM already runs. Cost ~+5% file
|
||||
size; benefit ~4× less overfetch per row read.
|
||||
2. **Keep `serverMode: "full"` for now.** Chunked mode's benefit is CDN cache
|
||||
efficiency, which Pages' 10-minute TTL blunts. Revisit if measured repeat-
|
||||
visit cost is bad.
|
||||
3. **Keep sql.js-httpvfs.** Switching to sqlite-wasm-http buys a maintained
|
||||
dependency but loses nothing we use, and its differentiator does not work on
|
||||
Pages. Note it as the escape hatch.
|
||||
4. **Verify `Content-Encoding` after deploy** — the single fatal hosting case.
|
||||
5. Keep the byte budget; drop any idea of a request-count budget.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
- Unindexed query → whole table over the network. `EXPLAIN QUERY PLAN` is the
|
||||
check.
|
||||
- Page size mismatched with `requestChunkSize` → every logical page read spans
|
||||
two requests.
|
||||
- Serving the database compressed → library refuses to open it.
|
||||
- Assuming CDN caching helps: on Pages, `max-age=600`.
|
||||
|
||||
## References
|
||||
|
||||
- [Hosting SQLite databases on GitHub Pages — phiresky](https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/)
|
||||
- [phiresky/sql.js-httpvfs](https://github.com/phiresky/sql.js-httpvfs)
|
||||
- [mmomtchev/sqlite-wasm-http](https://github.com/mmomtchev/sqlite-wasm-http)
|
||||
- [Query SQLite on GitHub Pages with sql.js-httpvfs (Mar 2026)](https://recca0120.github.io/en/2026/03/07/sql-js-httpvfs-static-hosting/)
|
||||
- [sqlite3 WebAssembly documentation](https://sqlite.org/wasm)
|
||||
- [GitHub Pages asset caching discussion](https://github.com/orgs/community/discussions/11884)
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Does 1024 measurably beat 4096 *for our queries*? Only a browser with the
|
||||
byte counter can answer; the estimate says yes for row fetches.
|
||||
2. Does Fastly cache 206 responses for a 289 MB object well enough that chunked
|
||||
mode is unnecessary? Needs a deployed measurement.
|
||||
3. Does the read-head prefetch overfetch on our index-range walks (fetching
|
||||
ahead beyond `LIMIT 100`)? Unknown without instrumentation.
|
||||
@@ -14,8 +14,10 @@ import wasmUrl from "sql.js-httpvfs/dist/sql-wasm.wasm?url";
|
||||
* the byte budget below is for.
|
||||
*/
|
||||
|
||||
// Matches the page size the parser writes, so one request is one page.
|
||||
const CHUNK_BYTES = 4096;
|
||||
// Must equal the page size the parser writes (PRAGMA page_size in
|
||||
// parser/internal/writer/writer.go), so one request is exactly one page. A
|
||||
// mismatch makes every logical page read span two requests.
|
||||
const CHUNK_BYTES = 1024;
|
||||
|
||||
/** Generous for indexed work: a name search costs well under 1 MB. */
|
||||
export const SEARCH_BUDGET_BYTES = 25 * 1024 * 1024;
|
||||
|
||||
Reference in New Issue
Block a user