diff --git a/datasets.json b/datasets.json index fef7161..4543006 100644 --- a/datasets.json +++ b/datasets.json @@ -20,12 +20,12 @@ { "id": "2016", "expectedRows": 877460, - "dbSizeMb": 289 + "dbSizeMb": 302 }, { "id": "2017", "expectedRows": 861068, - "dbSizeMb": 238 + "dbSizeMb": 247 } ] } diff --git a/docs/data-pipeline.md b/docs/data-pipeline.md index d199eda..1bb9567 100644 --- a/docs/data-pipeline.md +++ b/docs/data-pipeline.md @@ -181,6 +181,10 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what ## Expected row counts +The databases are written with 1 KiB pages (`PRAGMA page_size` in +`parser/internal/writer/writer.go`) because the browser fetches them a page per +HTTP request. `CHUNK_BYTES` in `web/src/lib/sqlite.svelte.ts` must match. + | id | Source rows | Skipped | DB rows | | --- | --- | --- | --- | | `2016` | 877,460 | 0 | **877,460** | diff --git a/docs/deployment-guide.md b/docs/deployment-guide.md index 0cf3049..b34a9dc 100644 --- a/docs/deployment-guide.md +++ b/docs/deployment-guide.md @@ -83,8 +83,8 @@ artifact — one missing line away from publishing it. files committed to a repository; the databases are built in CI and uploaded as a Pages artifact, and the documented Pages limits are a 1 GB published site and 100 GB/month of bandwidth, with no per-file figure. The two - databases are 289 MB and 238 MB. -- **Total artifact is about 528 MB**, inside the 1 GB site limit but with less + databases are 302 MB and 247 MB. +- **Total artifact is about 552 MB**, inside the 1 GB site limit but with less headroom than before: a third dataset of this size would not fit. The fallback is `sql.js-httpvfs`'s chunked mode, which splits a database into parts. - **The server must not compress the databases.** Ranges of a compressed body diff --git a/docs/system-architecture.md b/docs/system-architecture.md index d57b38d..850804a 100644 --- a/docs/system-architecture.md +++ b/docs/system-architecture.md @@ -175,6 +175,7 @@ total descending. | WASM hosting | Bundled with the app | `sql.js-httpvfs` ships its own build; one less third-party runtime dependency | | Diacritics search | Pre-computed `ho_ten_ascii`, indexed word by word | `LOWER(REPLACE(...))` at query time defeats the index, and `LIKE '%x%'` reads the whole table | | Row count in the footer | Read from `datasets.json` | `COUNT(*)` scans an index — 20 MB over range requests | +| Page size | 1 KiB, matched by `requestChunkSize` | One HTTP request is one page; a row fetched by seek costs 1 KB rather than 4 KB, for about 5% more file | | SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes cannot persist; the allowlist prevents confusion | | Row caps | 100 (lookup), 1000 (SQL) | Keeps DOM render sizes reasonable | | Routing | SvelteKit file routes, prerendered | Each dataset gets a real HTML file with its own title | @@ -191,7 +192,7 @@ total descending. `curl -sI …/db/2016.sqlite3` must show no `content-encoding`. - **`sql.js-httpvfs` is unmaintained** (0.8.12, September 2022) and ships its own SQLite WASM. `sqlite-wasm-http`, on the official build, is the fallback. -- **Hosted size.** 528 MB for both datasets against the 1 GB GitHub Pages +- **Hosted size.** 552 MB for both datasets against the 1 GB GitHub Pages limit; a third dataset of this size would not fit. - **Excel format drift.** A new source file with an unseen header layout needs a new branch in `parser/internal/ingest/detect2016.go` or a new config. diff --git a/parser/internal/writer/writer.go b/parser/internal/writer/writer.go index 9f7bc0a..eb8ea86 100644 --- a/parser/internal/writer/writer.go +++ b/parser/internal/writer/writer.go @@ -43,6 +43,19 @@ func OpenDB(dbPath string) (*sql.DB, error) { if err != nil { return nil, fmt.Errorf("open db %s: %w", dbPath, err) } + // Before the DDL, because a page size cannot change once a table exists — + // only the VACUUM in Finish could rewrite it, and only to this same value. + // + // 1 KiB rather than SQLite's 4 KiB default because the browser reads this + // file a page at a time over HTTP: a row fetched by index seek costs one + // page, so a search that returns 100 scattered rows transfers 100 KB + // instead of 400 KB. It costs about 5% file size, and both sql.js-httpvfs + // and sqlite-wasm-http recommend it. web/src/lib/sqlite.svelte.ts must + // request the same size. + if _, err := db.Exec("PRAGMA page_size = 1024"); err != nil { + db.Close() + return nil, fmt.Errorf("set page size: %w", err) + } if _, err := db.Exec(schema.DDL); err != nil { db.Close() return nil, fmt.Errorf("execute DDL: %w", err) diff --git a/plans/260814-1200-httpvfs-range-queries/plan.md b/plans/260814-1200-httpvfs-range-queries/plan.md index 427f24b..2ba55e2 100644 --- a/plans/260814-1200-httpvfs-range-queries/plan.md +++ b/plans/260814-1200-httpvfs-range-queries/plan.md @@ -35,12 +35,16 @@ finds "Nguyễn Bửu Lộc", in a few hundred KB. | Segment | 2016 | | --- | --- | -| `student` | 127 MB | -| `name_word` | 97 MB | -| `idx_ten_cum_thi` | 37 MB | -| PK autoindex | 15 MB | -| score indexes | 12 MB | -| **total** | **288.6 MB** (2017: 237.7 MB) | +| `student` | 137.5 MB | +| `name_word` | 98.7 MB | +| `idx_ten_cum_thi` | 38.1 MB | +| PK autoindex | 15.3 MB | +| `idx_toan` | 12.6 MB | +| **total** | **302.4 MB** (2017: 247.3 MB) | + +Written with 1 KiB pages, so a row reached by an index seek costs one 1 KB +request instead of 4 KB: 6.3 rows share a page rather than 27, which is what +turns a 100-row search from ~400 KB of row fetches into ~100 KB. **Assembler.** Publishes uncompressed; the size guard reads the raw size; the stray-artifact check now rejects journals, `.db` and `.gz`. diff --git a/plans/reports/from-research-to-implementation-brainstorm-260814-1317-httpvfs-page-size-adoption-report.md b/plans/reports/from-research-to-implementation-brainstorm-260814-1317-httpvfs-page-size-adoption-report.md new file mode 100644 index 0000000..070546a --- /dev/null +++ b/plans/reports/from-research-to-implementation-brainstorm-260814-1317-httpvfs-page-size-adoption-report.md @@ -0,0 +1,78 @@ +# Brainstorm: which httpvfs best practices to adopt + +2026-08-14 13:17. Follows +[the research report](./web-sqlite-range-hosting-research-260814-1317-httpvfs-best-practices-report.md). +Branch `feat/httpvfs-range-queries`. + +## Problem + +Research surfaced three upstream practices we do not follow. Decide which are +worth the change, knowing nothing can be verified in a browser on this machine. + +## Codebase context (scout) + +| Concern | Touch points | +| --- | --- | +| Page size | `parser/internal/writer/writer.go:41-47` (PRAGMA must precede the DDL — page size is fixed once a table exists), `:185` (VACUUM already applies it), `web/src/lib/sqlite.svelte.ts:18,53` | +| Chunked mode | `databases.go:41` Extension, size guard, **`Clean()` deletes anything not `.sqlite3` — would eat every chunk**, `site.go:135-138,153`, `datasets.ts:91-97`, `datasets.json`, ~15 tests | +| Library swap | `sqlite.svelte.ts:1-3,50-58,76,82`, `package.json:15`; all consumers go through `RemoteDatabase`, so the blast radius is one file | + +## Options evaluated + +### A. page_size 1024 + requestChunkSize 1024 — ADOPTED + +- Upstream consensus: phiresky and mmomtchev both recommend 1024. +- Honest sizing for *our* pattern: row fetches 400 KB → 100 KB per search; + the index walk is sequential so bytes are unchanged and only the request + count rises, which prefetch read-heads collapse. Net ≈ 300 KB saved per + search — bandwidth, not latency. +- Cost: ~10 lines; file size +5% (528 → ~555 MB total, still under the 1 GB + Pages limit); both databases rebuilt. + +### B. serverMode chunked — REJECTED + +- Only benefit is CDN cache efficiency. GitHub Pages serves everything with + `Cache-Control: max-age=600`, and each deploy relays out SQLite pages anyway, + so cross-deploy caching is zero either way. +- Cost: split step, config JSON, `Clean()`/guards/`dbOf()`/`datasets.json` + rework, ~15 tests. +- Complexity buying a benefit the host cancels. Revisit only behind a CDN with + long TTLs. + +### C. swap to sqlite-wasm-http — DEFERRED + +- For: maintained (Dec 2025), official SQLite WASM instead of a 2022 fork; + matches this repo's posture on stale dependencies. Swap is one file. +- Against: its differentiator (shared cache) needs COOP/COEP headers GitHub + Pages cannot send, so we would get the synchronous fallback and the + maintenance benefit only. And the current integration has never run in a + browser — swapping now means two unverified variables and no way to tell + which broke. +- Revisit after the current build is verified live. + +## Decision + +Adopt **A only**. + +## Implementation notes + +1. `PRAGMA page_size = 1024` in `writer.OpenDB`, between `sql.Open` and + `db.Exec(schema.DDL)`. The existing VACUUM in `Finish` applies it. +2. `CHUNK_BYTES = 1024` in `web/src/lib/sqlite.svelte.ts`; it feeds + `requestChunkSize` and must equal the page size. +3. Rebuild both databases, update `dbSizeMb` in `datasets.json` to the new + sizes, re-run the assembler guards. + +## Risks + +- The benefit is arithmetic plus upstream authority, not measurement. The byte + counter in the SQL tab is the check, once deployed. +- Prefetch deliberately overfetches ahead of the cursor, so the 25 MB search + budget may trip earlier than a strict page count suggests. Tune after a real + measurement, not before. + +## Unresolved questions + +1. Does 1024 actually beat 4096 for our queries in a browser? +2. Is Fastly's caching of ranges over a 300 MB object good enough that chunked + mode stays unnecessary? diff --git a/plans/reports/web-sqlite-range-hosting-research-260814-1317-httpvfs-best-practices-report.md b/plans/reports/web-sqlite-range-hosting-research-260814-1317-httpvfs-best-practices-report.md new file mode 100644 index 0000000..ef11e7c --- /dev/null +++ b/plans/reports/web-sqlite-range-hosting-research-260814-1317-httpvfs-best-practices-report.md @@ -0,0 +1,165 @@ +# Research Report: sql.js-httpvfs best practices + +Conducted 2026-08-14 13:17 (Asia/Saigon). Context: two static SQLite files +(2016 = 288.6 MB, 2017 = 237.7 MB) on GitHub Pages, branch +`feat/httpvfs-range-queries`. + +## Executive summary + +Our implementation matches upstream guidance on the thing that matters most — +index design — and diverges on one measurable parameter: **page size**. Both +phiresky (sql.js-httpvfs) and mmomtchev (sqlite-wasm-http) recommend +`page_size = 1024`; we shipped 4096. For our access pattern (~100 scattered +single-row reads per search) that is a real 4× overfetch on the row-fetch half +of a query. + +Two findings reduce risk rather than add work. Prefetching with three virtual +read heads makes a sequential scan cost a *logarithmic* number of requests, so +our byte estimates hold but latency is better than assumed. And the canonical +demo hosts a **670 MiB** database on GitHub Pages, so 289 MB is precedented. + +One finding is new and worth a decision: **chunked mode** (split file + JSON +config) exists specifically to make CDN caching effective for large databases, +which matters because GitHub Pages serves everything with `Cache-Control: +max-age=600`. + +## Methodology + +- Sources: 5 (1 primary blog, 2 project READMEs, 1 recent practitioner + writeup, 1 search on Pages/Fastly caching), plus direct reading of the + installed `sql.js-httpvfs@0.8.12` bundle in a previous session. +- Date range: 2021 (canonical post) → March 2026 (practitioner writeup). +- Gemini CLI absent → WebSearch/WebFetch. + +## Key findings + +### 1. Page size: recommended 1024, we use 4096 + +Both projects say the same thing. phiresky set 1 KiB pages "to balance request +overhead against bandwidth efficiency"; sqlite-wasm-http says "it is highly +recommended to decrease your SQLite page size to 1024 bytes for maximum +performance" (`PRAGMA page_size=1024; VACUUM`). + +`requestChunkSize` must match the page size. + +Measured on our 2016 file *before* the schema change: + +| page_size | file size | +| --- | --- | +| 1024 | 235.8 MB | +| 4096 | 223.5 MB | +| 8192 | 221.8 MB | + +So 1024 costs ~5.5% file size. What it buys: a scattered row read fetches 1 KB +instead of 4 KB. Our search does ~100 of those, so the row-fetch half of a +search drops from ~400 KB to ~100 KB. The index-walk half is sequential and +benefits from prefetch either way. + +**Verdict: switch to 1024 + `requestChunkSize: 1024`.** Our workload is +dominated by scattered single-row reads, which is exactly the case small pages +serve. + +### 2. Prefetch changes request count, not bytes + +"Three separate virtual read heads" detect sequential access and grow request +sizes exponentially, so "index scans or table scans reading more than a few KiB +of data will only cause a number of requests that is logarithmic in the total +byte length." + +Consequence for our analysis: a full table scan still transfers ~127 MB (bytes +are bytes), but in tens of requests rather than tens of thousands. Our byte +budget is the right guardrail; a request-count budget would not be. + +### 3. Index design — we already comply + +- Covering indexes: put every column the query needs *in* the index, else + SQLite does "another random access (unpredictable) read and thus HTTP request + to retrieve the actual value for every data point". This is exactly why + `name_word` carries `ho_ten_ascii`. +- Column order decides which lookups are cheap. +- Verify with `EXPLAIN QUERY PLAN`; a `SCAN` means the whole table crosses the + network. We did this for every query the app issues. + +### 4. Chunked mode exists for CDN caching + +`serverMode: "chunked"` splits the database into parts (10 MB is the commonly +cited size) with a JSON config. Stated benefit: "CDN caching much more +effective" for large databases. + +Relevant because **GitHub Pages sets `Cache-Control: max-age=600`** — ten +minutes — on everything. Range responses are cached by Fastly per object; with +one 289 MB object the practical caching story is weaker than with 29 chunks +that a CDN edge can hold whole. Chunked mode also sidesteps any future per-file +concern. + +Cost: a build step to split files + emit config, and every deploy invalidates +all chunks anyway (SQLite page layout is not deterministic across rebuilds). + +### 5. Hosting facts confirmed + +- Range requests work on Pages "out of the box" — matches our own probe (206 + + correct `Content-Range`). +- CORS headers (`Access-Control-Allow-Origin`, `Access-Control-Allow-Headers: + Range`) only matter cross-origin. Ours is same-origin — non-issue. +- 670 MiB database on Pages is the canonical demo. 289 MB is not exotic. +- `Content-Encoding` remains the one fatal case: the library discards + `Content-Length` and throws when a HEAD carries a non-identity encoding. + Unknown extensions like `.sqlite3` are served `application/octet-stream` and + left alone. + +### 6. Alternatives + +| | sql.js-httpvfs | sqlite-wasm-http | +| --- | --- | --- | +| WASM base | own sql.js fork (~3.36 era) | official `@sqlite.org/sqlite-wasm` | +| Last release | 0.8.12, Sept 2022 | 1.2.0, Dec 2023; activity into Dec 2025 | +| Self-description | "demo-level code… not for high stability" | "experimental" | +| Concurrency | one worker | multiple connections, shared cache | +| Shared cache needs | — | `SharedArrayBuffer` → COOP/COEP headers | +| On GitHub Pages | works | works, but **Pages cannot set COOP/COEP**, so it falls back to the synchronous backend without shared cache | +| Module format | CJS+ESM | **ES6 only** | + +Both are self-declared experimental. sqlite-wasm-http's advantage (maintained, +official WASM) is real; its headline feature (shared cache) is unavailable on +Pages precisely because Pages cannot send cross-origin isolation headers. + +## Implementation recommendations + +1. **Change page size to 1024 and `requestChunkSize` to 1024.** Parser sets + `PRAGMA page_size=1024` before DDL; VACUUM already runs. Cost ~+5% file + size; benefit ~4× less overfetch per row read. +2. **Keep `serverMode: "full"` for now.** Chunked mode's benefit is CDN cache + efficiency, which Pages' 10-minute TTL blunts. Revisit if measured repeat- + visit cost is bad. +3. **Keep sql.js-httpvfs.** Switching to sqlite-wasm-http buys a maintained + dependency but loses nothing we use, and its differentiator does not work on + Pages. Note it as the escape hatch. +4. **Verify `Content-Encoding` after deploy** — the single fatal hosting case. +5. Keep the byte budget; drop any idea of a request-count budget. + +## Common pitfalls + +- Unindexed query → whole table over the network. `EXPLAIN QUERY PLAN` is the + check. +- Page size mismatched with `requestChunkSize` → every logical page read spans + two requests. +- Serving the database compressed → library refuses to open it. +- Assuming CDN caching helps: on Pages, `max-age=600`. + +## References + +- [Hosting SQLite databases on GitHub Pages — phiresky](https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/) +- [phiresky/sql.js-httpvfs](https://github.com/phiresky/sql.js-httpvfs) +- [mmomtchev/sqlite-wasm-http](https://github.com/mmomtchev/sqlite-wasm-http) +- [Query SQLite on GitHub Pages with sql.js-httpvfs (Mar 2026)](https://recca0120.github.io/en/2026/03/07/sql-js-httpvfs-static-hosting/) +- [sqlite3 WebAssembly documentation](https://sqlite.org/wasm) +- [GitHub Pages asset caching discussion](https://github.com/orgs/community/discussions/11884) + +## Unresolved questions + +1. Does 1024 measurably beat 4096 *for our queries*? Only a browser with the + byte counter can answer; the estimate says yes for row fetches. +2. Does Fastly cache 206 responses for a 289 MB object well enough that chunked + mode is unnecessary? Needs a deployed measurement. +3. Does the read-head prefetch overfetch on our index-range walks (fetching + ahead beyond `LIMIT 100`)? Unknown without instrumentation. diff --git a/web/src/lib/sqlite.svelte.ts b/web/src/lib/sqlite.svelte.ts index e00d10f..5ef0f2f 100644 --- a/web/src/lib/sqlite.svelte.ts +++ b/web/src/lib/sqlite.svelte.ts @@ -14,8 +14,10 @@ import wasmUrl from "sql.js-httpvfs/dist/sql-wasm.wasm?url"; * the byte budget below is for. */ -// Matches the page size the parser writes, so one request is one page. -const CHUNK_BYTES = 4096; +// Must equal the page size the parser writes (PRAGMA page_size in +// parser/internal/writer/writer.go), so one request is exactly one page. A +// mismatch makes every logical page read span two requests. +const CHUNK_BYTES = 1024; /** Generous for indexed work: a name search costs well under 1 MB. */ export const SEARCH_BUDGET_BYTES = 25 * 1024 * 1024;