mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-08-14 01:24:20 +00:00
Merge pull request #6 from tiennm99/refactor/go-parser
refactor: split the repo into crawler, parser, assembler and web
This commit is contained in:
@@ -3,6 +3,11 @@ name: Deploy to GitHub Pages
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
# Pull requests run the build job only. Before this, the workflow triggered on
|
||||
# main-push and workflow_dispatch alone, so "verify on a branch first" was not
|
||||
# actually possible: pushing to a branch ran nothing, and dispatching from one
|
||||
# published that branch straight to the live site.
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
@@ -17,42 +22,74 @@ concurrency:
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
# Every Go module here is cgo-free — grate, excelize, yaml.v3, x/net,
|
||||
# x/text and modernc.org/sqlite — so no C toolchain is needed. Set
|
||||
# explicitly rather than relying on the default.
|
||||
CGO_ENABLED: '0'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
workspaces: parser
|
||||
go-version: '1.26'
|
||||
cache-dependency-path: |
|
||||
parser/go.sum
|
||||
crawler/go.sum
|
||||
assembler/go.sum
|
||||
|
||||
# web/ is the only npm project in the repository; the other stages are Go.
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '24'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: package-lock.json
|
||||
cache-dependency-path: web/package-lock.json
|
||||
|
||||
- run: npm ci
|
||||
- name: Install web dependencies
|
||||
working-directory: web
|
||||
run: npm ci
|
||||
|
||||
# One parser binary builds every dataset; build-db.js reads the dataset
|
||||
# list from src/datasets.js and gzips each database in place, leaving no
|
||||
# uncompressed file behind.
|
||||
- name: Build databases
|
||||
# The reader-fidelity suite compares every real input file against a
|
||||
# committed hash oracle, so it is the regression guard for the whole
|
||||
# reader. Runs before anything is built.
|
||||
- name: Test parser
|
||||
run: go -C parser test ./...
|
||||
|
||||
# The crawler is not part of the build — it only refreshes data/ by hand.
|
||||
# It is still tested here so it cannot rot unnoticed, and because its
|
||||
# fixture test guards the parser: input filenames decide which row
|
||||
# survives a duplicate exam number.
|
||||
- name: Test crawler
|
||||
run: go -C crawler test ./...
|
||||
|
||||
- name: Lint web
|
||||
working-directory: web
|
||||
run: npm run lint
|
||||
|
||||
# excelize carries an open advisory, and the 2017 refresh runbook feeds
|
||||
# network-downloaded spreadsheets straight into the parser.
|
||||
- name: Vulnerability scan
|
||||
run: |
|
||||
npm run build:rust
|
||||
npm run build:db
|
||||
go install golang.org/x/vuln/cmd/govulncheck@latest
|
||||
GOVULNCHECK="$(go env GOPATH)/bin/govulncheck"
|
||||
for m in parser crawler assembler; do (cd "$m" && "$GOVULNCHECK" ./...); done
|
||||
|
||||
# One Vite build produces every page. scripts/assemble-site.js copies the
|
||||
# emitted index.html to each dataset path (and the legacy nested URLs),
|
||||
# then fails the job if any uncompressed database reached the artifact.
|
||||
- name: Build and assemble site
|
||||
run: npm run build:site
|
||||
# One command runs the whole pipeline: compile the parser, build and
|
||||
# verify each database against its registry row count, compress it, build
|
||||
# the web app, and assemble _site — refusing to continue if a database is
|
||||
# short, an artifact looks truncated, or one is missing entirely.
|
||||
- name: Build site
|
||||
run: go -C assembler run ./cmd/assemble
|
||||
|
||||
- uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: _site
|
||||
|
||||
deploy:
|
||||
# Guarded to main. Without this, a workflow_dispatch from any branch would
|
||||
# publish that branch's output to the live site, and concurrency
|
||||
# cancel-in-progress would kill an in-flight good deploy on the way.
|
||||
if: github.ref == 'refs/heads/main'
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
environment:
|
||||
|
||||
+4
-3
@@ -17,8 +17,9 @@ dist/
|
||||
### Generated databases and Vite publicDir staging ###
|
||||
.build/
|
||||
|
||||
### Rust build artefacts ###
|
||||
parser/target/
|
||||
|
||||
### Assembled Pages artifact ###
|
||||
_site/
|
||||
|
||||
# Parser build output and regenerable ground-truth dumps
|
||||
parser/bin/
|
||||
parser/testdata/dumps/
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Tra cứu điểm thi THPT Quốc gia — exam-score lookup for Vietnam's national high
|
||||
school graduation exam. Client-side SQL (sql.js) over a SQLite database built
|
||||
from the ministry's raw `.xls` score files by the Rust `xlsxread` parser.
|
||||
from the ministry's raw `.xls` score files by the Go `xlsxread` parser.
|
||||
|
||||
Live at **[tiennm99.github.io/thptqg](https://tiennm99.github.io/thptqg/)**.
|
||||
|
||||
@@ -10,55 +10,84 @@ Live at **[tiennm99.github.io/thptqg](https://tiennm99.github.io/thptqg/)**.
|
||||
| --- | --- | --- | --- |
|
||||
| `2016` | 2016 | 877,461 | [/2016/](https://tiennm99.github.io/thptqg/2016/) |
|
||||
| `2017` | 2017 | 861,068 | [/2017/](https://tiennm99.github.io/thptqg/2017/) |
|
||||
| `2017-old` | 2017 | 847,348 | [/2017-old/](https://tiennm99.github.io/thptqg/2017-old/) |
|
||||
| `2017-old2` | 2017 | 679,764 | [/2017-old2/](https://tiennm99.github.io/thptqg/2017-old2/) |
|
||||
|
||||
The three 2017 datasets are successive publications of the same exam and they
|
||||
disagree; all three are kept so the differences stay inspectable.
|
||||
Two earlier 2017 publications (`2017-old`, `2017-old2`) were kept for a while
|
||||
because they disagreed with the current one. They have been removed; they remain
|
||||
in git history.
|
||||
|
||||
## Layout
|
||||
|
||||
The repository is one directory per pipeline stage, plus the two stores they
|
||||
pass between them.
|
||||
|
||||
```
|
||||
index.html + src/ the frontend — one app serving all four datasets and the hub
|
||||
datasets.js the four dataset ids and their per-dataset content
|
||||
router.js pathname → dataset
|
||||
data/<id>/ raw Excel files, one directory per dataset
|
||||
parser/ the Rust parser
|
||||
src/schema.rs canonical 22-column table: DDL, INSERT, subject regexes
|
||||
configs/<id>.toml per-dataset parse rules only, no SQL
|
||||
scripts/ database build, crawler, parity verification
|
||||
scripts/ site assembly
|
||||
docs/ architecture, data pipeline, deployment
|
||||
crawler/ Go — re-fetches the source spreadsheets → data/
|
||||
parser/ Go — Excel to SQLite data/ → .db
|
||||
assembler/ Go — verifies, compresses, builds, assembles .db + web/ → _site/
|
||||
web/ npm — the frontend, one Vite app for every dataset
|
||||
data/<id>/ raw Excel files, one directory per dataset
|
||||
datasets.json the registry: which datasets exist, and their expected size
|
||||
docs/ architecture, data pipeline, deployment
|
||||
```
|
||||
|
||||
Each stage runs on its own and hands its output to the next through the stores.
|
||||
`web/` is the only npm project; the three stages are independent Go modules.
|
||||
|
||||
`datasets.json` is the contract between them. It is JSON because Go and the Vite
|
||||
app both read it and neither needs a dependency to do so; presentation stays in
|
||||
`web/src/datasets.js`, keyed by id, which fails loudly if the two disagree.
|
||||
|
||||
The dataset id is one identifier end to end:
|
||||
|
||||
```
|
||||
data/2017-old/ → parser/configs/2017-old.toml → db/2017-old.db.gz → /thptqg/2017-old/
|
||||
data/2017/ → parser/configs/2017.yml → db/2017.db.gz → /thptqg/2017/
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
npm ci
|
||||
npm run build:rust # compile the parser
|
||||
npm run build:db # build + gzip all four databases (add an id for just one)
|
||||
npm run build:site # one Vite build, then assemble into _site/
|
||||
(cd web && npm ci)
|
||||
go -C assembler run ./cmd/assemble # databases, then the site, into _site/
|
||||
npx serve _site
|
||||
```
|
||||
|
||||
That one command compiles the parser, builds and verifies each database against
|
||||
its registry row count, compresses it, builds the web app and assembles `_site` —
|
||||
refusing to continue if a database is short, an artifact looks truncated, or one
|
||||
is missing altogether. Sub-steps when iterating:
|
||||
|
||||
```bash
|
||||
go -C assembler run ./cmd/assemble db 2017 # one database
|
||||
go -C assembler run ./cmd/assemble site # web build and _site only
|
||||
(cd web && npm run dev) # the app against staged databases
|
||||
```
|
||||
|
||||
The source spreadsheets are committed, so a crawl is only needed to refresh
|
||||
them:
|
||||
|
||||
```bash
|
||||
go -C crawler run ./cmd/crawl 2016
|
||||
go -C crawler run ./cmd/crawl 2017
|
||||
```
|
||||
|
||||
Each reads the download links out of the article that published the dataset, so
|
||||
no link list is kept in the repository. Crawling is idempotent — files already
|
||||
present are skipped — and is never part of the build.
|
||||
|
||||
Pushing to `main` runs the same steps in
|
||||
`.github/workflows/deploy-pages.yml` and publishes to GitHub Pages.
|
||||
|
||||
## Adding a dataset
|
||||
|
||||
1. Put the Excel files in `data/<id>/`
|
||||
2. Add `parser/configs/<id>.toml` — sheet mode, column indices, validation
|
||||
2. Add `parser/configs/<id>.yml` — sheet mode, column indices, validation
|
||||
guards. No SQL; the schema is canonical.
|
||||
3. Add an entry to `DATASETS` in `src/datasets.js`
|
||||
3. Add an entry to `datasets.json` with its expected row count and size
|
||||
4. Add the matching presentation to `CONTENT` in `web/src/datasets.js`
|
||||
|
||||
Everything else follows: the build script, the site assembly and the router all
|
||||
read that one list, and the UI adapts to whichever columns the dataset fills.
|
||||
Everything else follows: the assembler, the router and the hub all read the
|
||||
registry, and the UI adapts to whichever columns the dataset fills. Steps 3 and 4
|
||||
check each other, so forgetting either one fails rather than half-working.
|
||||
|
||||
## Docs
|
||||
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
// Command assemble turns source data and the web app into the directory
|
||||
// GitHub Pages publishes.
|
||||
//
|
||||
// assemble # databases, then the site
|
||||
// assemble db # databases only (add ids to limit: assemble db 2017)
|
||||
// assemble site # web build and _site only, reusing staged databases
|
||||
//
|
||||
// It sequences the other stages rather than doing their work: the parser reads
|
||||
// spreadsheets, Vite bundles the app, and this decides what runs, checks what
|
||||
// came out, and refuses to publish anything that looks wrong.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/tiennm99/thptqg/assembler/internal/databases"
|
||||
"github.com/tiennm99/thptqg/assembler/internal/registry"
|
||||
"github.com/tiennm99/thptqg/assembler/internal/site"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:]); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "assemble: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func usage() {
|
||||
fmt.Fprint(os.Stderr, `usage: assemble [step] [dataset...]
|
||||
|
||||
Steps:
|
||||
(none) databases, then the site
|
||||
db build, verify and compress the databases
|
||||
site build the web app and assemble _site
|
||||
|
||||
Naming datasets limits the db step to those; the site step always covers all of
|
||||
them, since a partial site would publish links to databases it did not build.
|
||||
`)
|
||||
}
|
||||
|
||||
func run(args []string) error {
|
||||
step := ""
|
||||
if len(args) > 0 {
|
||||
switch args[0] {
|
||||
case "db", "site":
|
||||
step, args = args[0], args[1:]
|
||||
case "-h", "--help":
|
||||
usage()
|
||||
return nil
|
||||
default:
|
||||
// Bare dataset ids are a natural thing to type; treat them as the
|
||||
// db step rather than rejecting them.
|
||||
step = "db"
|
||||
}
|
||||
}
|
||||
|
||||
root, err := repoRoot()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
all, err := registry.Load(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if step == "" || step == "db" {
|
||||
selected, err := registry.Select(all, args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := buildDatabases(root, all, selected); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if step == "" || step == "site" {
|
||||
sp := site.DefaultPaths(root)
|
||||
if err := site.BuildWeb(sp); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := site.Assemble(sp, all); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func buildDatabases(root string, all, selected []registry.Dataset) error {
|
||||
p := databases.DefaultPaths(root)
|
||||
|
||||
// Sweep first: a dataset dropped from the registry leaves its .db.gz behind,
|
||||
// and the site assembly copies the staging directory wholesale.
|
||||
if err := databases.Clean(p, all); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
bin, err := databases.BuildParser(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, d := range selected {
|
||||
if err := databases.Build(p, bin, d); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// repoRoot walks up from the working directory to the directory holding
|
||||
// datasets.json, so the command works from anywhere in the tree.
|
||||
func repoRoot() (string, error) {
|
||||
dir, err := os.Getwd()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for {
|
||||
if _, err := os.Stat(filepath.Join(dir, "datasets.json")); err == nil {
|
||||
return dir, nil
|
||||
}
|
||||
parent := filepath.Dir(dir)
|
||||
if parent == dir {
|
||||
return "", fmt.Errorf("no datasets.json found in any parent of the working directory")
|
||||
}
|
||||
dir = parent
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
module github.com/tiennm99/thptqg/assembler
|
||||
|
||||
go 1.26.5
|
||||
|
||||
require modernc.org/sqlite v1.56.0
|
||||
|
||||
require (
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/mattn/go-isatty v0.0.24 // indirect
|
||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
modernc.org/libc v1.74.4 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.11.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,50 @@
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
|
||||
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
|
||||
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
|
||||
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
||||
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI=
|
||||
modernc.org/cc/v4 v4.29.1/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
|
||||
modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=
|
||||
modernc.org/ccgo/v4 v4.34.6/go.mod h1:SZ8YcN9NG7XVsQYdm6jYBvi8PQP1qi+kqB6OhjqI3Fk=
|
||||
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
|
||||
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
|
||||
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
|
||||
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
|
||||
modernc.org/gc/v3 v3.1.4 h1:2g65LGVSmFQrXeITAw97x7hCRvZFcyE1uDP+7Vng7JI=
|
||||
modernc.org/gc/v3 v3.1.4/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
|
||||
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
|
||||
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
|
||||
modernc.org/libc v1.74.4 h1:fX1Omw4o2/1C2iRkkIsrQTasJQldLhRmuPreXLoWs9k=
|
||||
modernc.org/libc v1.74.4/go.mod h1:eeQAS9W3sZeKYMFubydxJpII9ybHWshk+7or7bLG9co=
|
||||
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
|
||||
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
|
||||
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
|
||||
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
|
||||
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
|
||||
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
|
||||
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
|
||||
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
|
||||
modernc.org/sqlite v1.56.0 h1:/D8e2RfFqoy/Zc6PuC76U28zFwmI/sYx1Kjm4yEn9e0=
|
||||
modernc.org/sqlite v1.56.0/go.mod h1:yCJ2cmAaIkHQ25oXWrF8H4O1lIfPYPR26yCEDj2P3pQ=
|
||||
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
|
||||
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
|
||||
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
|
||||
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
||||
@@ -0,0 +1,217 @@
|
||||
// Package databases builds, verifies and compresses one SQLite file per
|
||||
// dataset.
|
||||
//
|
||||
// VERIFICATION IS THE POINT OF THIS PACKAGE, not an extra.
|
||||
//
|
||||
// Nothing between the parser and the published site otherwise asserts that a
|
||||
// database has data in it. The parser logs a file-level failure and continues,
|
||||
// returns success regardless, and finishes cleanly even at zero rows; the site
|
||||
// assembly only inspects filenames. So a reader that silently under-produced
|
||||
// would publish a truncated dataset with green CI and no red signal anywhere.
|
||||
//
|
||||
// The guards below close that: a build whose row count does not match the
|
||||
// registry, or whose artifact is implausibly small, fails the pipeline.
|
||||
package databases
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
|
||||
_ "modernc.org/sqlite" // pure-Go driver: the pipeline stays cgo-free
|
||||
|
||||
"github.com/tiennm99/thptqg/assembler/internal/registry"
|
||||
)
|
||||
|
||||
// driverName is modernc.org/sqlite's registered name.
|
||||
const driverName = "sqlite"
|
||||
|
||||
// minSizeRatio: a gzipped database far below its usual size means a truncated
|
||||
// build, even if the row count somehow passed.
|
||||
const minSizeRatio = 0.9
|
||||
|
||||
// Paths locates the pieces this package needs.
|
||||
type Paths struct {
|
||||
// Root is the repository root.
|
||||
Root string
|
||||
// Parser is the parser module directory.
|
||||
Parser string
|
||||
// OutDir is where the databases are staged — the directory Vite publishes.
|
||||
OutDir string
|
||||
}
|
||||
|
||||
// DefaultPaths derives the standard layout from the repository root.
|
||||
func DefaultPaths(root string) Paths {
|
||||
return Paths{
|
||||
Root: root,
|
||||
Parser: filepath.Join(root, "parser"),
|
||||
OutDir: filepath.Join(root, ".build", "public", "db"),
|
||||
}
|
||||
}
|
||||
|
||||
// BuildParser compiles the parser binary and returns its path.
|
||||
//
|
||||
// Compiling here rather than expecting a prebuilt binary keeps the pipeline one
|
||||
// command. Go caches the work, so repeat runs cost almost nothing.
|
||||
func BuildParser(p Paths) (string, error) {
|
||||
bin := filepath.Join(p.Parser, "bin", "xlsxread")
|
||||
cmd := exec.Command("go", "-C", p.Parser, "build", "-o", "bin/xlsxread", "./cmd/xlsxread")
|
||||
cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
return "", fmt.Errorf("compiling the parser: %w", err)
|
||||
}
|
||||
return bin, nil
|
||||
}
|
||||
|
||||
// Build runs the parser for one dataset, verifies the result and compresses it.
|
||||
//
|
||||
// Only the .gz survives: shipping a 100+ MB uncompressed database is made
|
||||
// structurally impossible rather than left to a cleanup step.
|
||||
func Build(p Paths, bin string, d registry.Dataset) error {
|
||||
if err := os.MkdirAll(p.OutDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
db := filepath.Join(p.OutDir, d.ID+".db")
|
||||
|
||||
cmd := exec.Command(bin,
|
||||
"build",
|
||||
"--schema", filepath.Join(p.Parser, "configs", d.ID+".yml"),
|
||||
"--input", filepath.Join(p.Root, "data", d.ID),
|
||||
"--output", db,
|
||||
)
|
||||
cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
return fmt.Errorf("%s: parser failed: %w", d.ID, err)
|
||||
}
|
||||
|
||||
rows, err := countRows(db)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", d.ID, err)
|
||||
}
|
||||
if rows != d.ExpectedRows {
|
||||
return fmt.Errorf(
|
||||
"%s: row count %d, expected %d\nRefusing to publish — the build did not reproduce the known dataset",
|
||||
d.ID, rows, d.ExpectedRows)
|
||||
}
|
||||
fmt.Printf(" ✓ %s: %d rows (matches expected)\n", d.ID, rows)
|
||||
|
||||
gz, size, err := compress(db)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", d.ID, err)
|
||||
}
|
||||
|
||||
sizeMb := float64(size) / 1024 / 1024
|
||||
if min := d.DbSizeMb * minSizeRatio; sizeMb < min {
|
||||
return fmt.Errorf(
|
||||
"%s: %.1f MB is below %.1f MB (%.0f%% of the expected %.0f MB)\n"+
|
||||
"Refusing to publish — the artifact looks truncated",
|
||||
d.ID, sizeMb, min, minSizeRatio*100, d.DbSizeMb)
|
||||
}
|
||||
|
||||
fmt.Printf(" → %s (%.1f MB)\n\n", filepath.Base(gz), sizeMb)
|
||||
return nil
|
||||
}
|
||||
|
||||
// countRows opens the database read-only and counts what was written.
|
||||
func countRows(path string) (int64, error) {
|
||||
conn, err := sql.Open(driverName, "file:"+path+"?mode=ro")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
var n int64
|
||||
if err := conn.QueryRow("SELECT COUNT(*) FROM student").Scan(&n); err != nil {
|
||||
return 0, fmt.Errorf("counting rows: %w", err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// compress gzips path to path+".gz" and removes the original, returning the
|
||||
// compressed path and its size.
|
||||
//
|
||||
// The source is deleted only after the compressed file is closed successfully,
|
||||
// so a failure part-way through leaves the database rather than losing it.
|
||||
func compress(path string) (string, int64, error) {
|
||||
in, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
defer in.Close()
|
||||
|
||||
gzPath := path + ".gz"
|
||||
out, err := os.Create(gzPath)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
|
||||
zw, err := gzip.NewWriterLevel(out, gzip.BestCompression)
|
||||
if err != nil {
|
||||
out.Close()
|
||||
return "", 0, err
|
||||
}
|
||||
if _, err := io.Copy(zw, in); err != nil {
|
||||
zw.Close()
|
||||
out.Close()
|
||||
os.Remove(gzPath)
|
||||
return "", 0, err
|
||||
}
|
||||
if err := zw.Close(); err != nil {
|
||||
out.Close()
|
||||
os.Remove(gzPath)
|
||||
return "", 0, err
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
os.Remove(gzPath)
|
||||
return "", 0, err
|
||||
}
|
||||
|
||||
if err := in.Close(); err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
if err := os.Remove(path); err != nil {
|
||||
return "", 0, fmt.Errorf("removing the uncompressed database: %w", err)
|
||||
}
|
||||
|
||||
st, err := os.Stat(gzPath)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
return gzPath, st.Size(), nil
|
||||
}
|
||||
|
||||
// Clean removes staged artifacts for datasets that are no longer in the
|
||||
// registry. Without this a removed dataset's .db.gz lingers in the staging
|
||||
// directory, and the site assembly copies that directory wholesale — so the
|
||||
// dead database would be published again.
|
||||
func Clean(p Paths, keep []registry.Dataset) error {
|
||||
entries, err := os.ReadDir(p.OutDir)
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
wanted := make(map[string]bool, len(keep)*2)
|
||||
for _, d := range keep {
|
||||
wanted[d.ID+".db"] = true
|
||||
wanted[d.ID+".db.gz"] = true
|
||||
}
|
||||
|
||||
for _, e := range entries {
|
||||
if e.IsDir() || wanted[e.Name()] {
|
||||
continue
|
||||
}
|
||||
full := filepath.Join(p.OutDir, e.Name())
|
||||
if err := os.Remove(full); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf(" removed stale artifact %s\n", e.Name())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package databases
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/thptqg/assembler/internal/registry"
|
||||
)
|
||||
|
||||
// TestCompressRoundTripsAndRemovesTheSource: only the .gz may survive, so that
|
||||
// shipping a 100+ MB uncompressed database is structurally impossible rather
|
||||
// than left to a cleanup step.
|
||||
func TestCompressRoundTripsAndRemovesTheSource(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
src := filepath.Join(dir, "2016.db")
|
||||
body := []byte("pretend this is a SQLite file")
|
||||
if err := os.WriteFile(src, body, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
gzPath, size, err := compress(src)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if gzPath != src+".gz" || size <= 0 {
|
||||
t.Fatalf("gzPath=%q size=%d", gzPath, size)
|
||||
}
|
||||
if _, err := os.Stat(src); !os.IsNotExist(err) {
|
||||
t.Error("the uncompressed database must not survive")
|
||||
}
|
||||
|
||||
f, err := os.Open(gzPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer f.Close()
|
||||
zr, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := io.ReadAll(zr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(got) != string(body) {
|
||||
t.Errorf("round-trip gave %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanRemovesOnlyDroppedDatasets(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for _, name := range []string{
|
||||
"2016.db.gz", "2017.db.gz",
|
||||
"2017-old.db.gz", // dropped from the registry
|
||||
"2017-old2.db.gz", // dropped from the registry
|
||||
"2016.db-journal", // interrupted run
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
p := Paths{OutDir: dir}
|
||||
keep := []registry.Dataset{{ID: "2016"}, {ID: "2017"}}
|
||||
if err := Clean(p, keep); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var left []string
|
||||
for _, e := range entries {
|
||||
left = append(left, e.Name())
|
||||
}
|
||||
slices.Sort(left)
|
||||
|
||||
want := []string{"2016.db.gz", "2017.db.gz"}
|
||||
if !slices.Equal(left, want) {
|
||||
t.Errorf("left %v, want %v", left, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanToleratesAnAbsentStagingDirectory: a fresh checkout has never built
|
||||
// anything, and that is not an error.
|
||||
func TestCleanToleratesAnAbsentStagingDirectory(t *testing.T) {
|
||||
p := Paths{OutDir: filepath.Join(t.TempDir(), "never-created")}
|
||||
if err := Clean(p, nil); err != nil {
|
||||
t.Errorf("Clean on a missing directory should succeed, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultPaths(t *testing.T) {
|
||||
p := DefaultPaths("/repo")
|
||||
if p.Parser != filepath.Join("/repo", "parser") {
|
||||
t.Errorf("Parser = %q", p.Parser)
|
||||
}
|
||||
// The staging directory must be the one Vite publishes, or the databases
|
||||
// never reach the site.
|
||||
if p.OutDir != filepath.Join("/repo", ".build", "public", "db") {
|
||||
t.Errorf("OutDir = %q", p.OutDir)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
// Package registry reads the repository-root datasets.json.
|
||||
//
|
||||
// That file is the one place every stage agrees on what exists. The Vite app
|
||||
// reads it too, which is why it is JSON: Go and the browser both parse it
|
||||
// without a dependency.
|
||||
package registry
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Dataset is one entry in the registry.
|
||||
type Dataset struct {
|
||||
ID string `json:"id"`
|
||||
// ExpectedRows is exact. The inputs are frozen historical exam results, so
|
||||
// a deviation of even one row means something changed that nobody intended.
|
||||
ExpectedRows int64 `json:"expectedRows"`
|
||||
// DbSizeMb is the usual size of the gzipped database, used to catch a
|
||||
// build that produced a plausible row count but a truncated artifact.
|
||||
DbSizeMb float64 `json:"dbSizeMb"`
|
||||
}
|
||||
|
||||
type file struct {
|
||||
Datasets []Dataset `json:"datasets"`
|
||||
}
|
||||
|
||||
// Load reads datasets.json from the repository root.
|
||||
func Load(root string) ([]Dataset, error) {
|
||||
path := filepath.Join(root, "datasets.json")
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot read the dataset registry: %w", err)
|
||||
}
|
||||
|
||||
var f file
|
||||
if err := json.Unmarshal(b, &f); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
if len(f.Datasets) == 0 {
|
||||
return nil, fmt.Errorf("%s declares no datasets", path)
|
||||
}
|
||||
|
||||
for _, d := range f.Datasets {
|
||||
switch {
|
||||
case d.ID == "":
|
||||
return nil, fmt.Errorf("%s: a dataset has no id", path)
|
||||
case d.ExpectedRows <= 0:
|
||||
return nil, fmt.Errorf("%s: %s has no expectedRows; the build guard needs it", path, d.ID)
|
||||
case d.DbSizeMb <= 0:
|
||||
return nil, fmt.Errorf("%s: %s has no dbSizeMb; the size guard needs it", path, d.ID)
|
||||
}
|
||||
}
|
||||
return f.Datasets, nil
|
||||
}
|
||||
|
||||
// Select returns the named datasets, or all of them when none are named.
|
||||
func Select(all []Dataset, ids []string) ([]Dataset, error) {
|
||||
if len(ids) == 0 {
|
||||
return all, nil
|
||||
}
|
||||
byID := make(map[string]Dataset, len(all))
|
||||
known := make([]string, 0, len(all))
|
||||
for _, d := range all {
|
||||
byID[d.ID] = d
|
||||
known = append(known, d.ID)
|
||||
}
|
||||
|
||||
out := make([]Dataset, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
d, ok := byID[id]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("unknown dataset %q (known: %v)", id, known)
|
||||
}
|
||||
out = append(out, d)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package registry
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeRegistry(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "datasets.json"), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return dir
|
||||
}
|
||||
|
||||
func TestLoadsTheRealRegistry(t *testing.T) {
|
||||
root := filepath.Join("..", "..", "..")
|
||||
got, err := Load(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) == 0 {
|
||||
t.Fatal("no datasets")
|
||||
}
|
||||
for _, d := range got {
|
||||
if d.ID == "" || d.ExpectedRows <= 0 || d.DbSizeMb <= 0 {
|
||||
t.Errorf("incomplete entry: %+v", d)
|
||||
}
|
||||
// Every declared dataset must have somewhere to read from and a parse
|
||||
// config, or the build fails much later with a worse message.
|
||||
for _, p := range []string{
|
||||
filepath.Join(root, "data", d.ID),
|
||||
filepath.Join(root, "parser", "configs", d.ID+".yml"),
|
||||
} {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
t.Errorf("%s: missing %s", d.ID, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestIncompleteEntriesAreRejected: a dataset missing either guard figure would
|
||||
// otherwise publish unverified. Both are load-bearing, so neither may default.
|
||||
func TestIncompleteEntriesAreRejected(t *testing.T) {
|
||||
for name, body := range map[string]string{
|
||||
"no expectedRows": `{"datasets":[{"id":"x","dbSizeMb":1}]}`,
|
||||
"no dbSizeMb": `{"datasets":[{"id":"x","expectedRows":1}]}`,
|
||||
"no id": `{"datasets":[{"expectedRows":1,"dbSizeMb":1}]}`,
|
||||
"zero rows": `{"datasets":[{"id":"x","expectedRows":0,"dbSizeMb":1}]}`,
|
||||
"empty": `{"datasets":[]}`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
if _, err := Load(writeRegistry(t, body)); err == nil {
|
||||
t.Error("expected an error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMissingAndMalformedRegistry(t *testing.T) {
|
||||
if _, err := Load(t.TempDir()); err == nil {
|
||||
t.Error("a missing registry must fail")
|
||||
}
|
||||
if _, err := Load(writeRegistry(t, "{not json")); err == nil {
|
||||
t.Error("a malformed registry must fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelect(t *testing.T) {
|
||||
all := []Dataset{{ID: "2016"}, {ID: "2017"}}
|
||||
|
||||
got, err := Select(all, nil)
|
||||
if err != nil || len(got) != 2 {
|
||||
t.Errorf("no ids should select everything: %v %v", got, err)
|
||||
}
|
||||
|
||||
got, err = Select(all, []string{"2017"})
|
||||
if err != nil || len(got) != 1 || got[0].ID != "2017" {
|
||||
t.Errorf("Select(2017) = %v, %v", got, err)
|
||||
}
|
||||
|
||||
// A typo must not silently build nothing.
|
||||
_, err = Select(all, []string{"2018"})
|
||||
if err == nil {
|
||||
t.Fatal("an unknown id must fail")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "2018") || !strings.Contains(err.Error(), "2016") {
|
||||
t.Errorf("the error should name the bad id and the known ones, got: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,210 @@
|
||||
// Package site turns the web app and the staged databases into the directory
|
||||
// GitHub Pages publishes.
|
||||
//
|
||||
// The app resolves its dataset from the URL, so every page is the same
|
||||
// index.html. Because Vite's `base` is absolute (/thptqg/), that file references
|
||||
// /thptqg/assets/... no matter which directory it is served from — so copying it
|
||||
// to each dataset path produces a real static file at every URL.
|
||||
//
|
||||
// GitHub Pages serves those as directory indexes, which is why this needs no
|
||||
// SPA 404-fallback redirect. That matters beyond tidiness: the usual fallback
|
||||
// rewrites the URL and would interfere with the ?q= deep links the app relies
|
||||
// on.
|
||||
package site
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/tiennm99/thptqg/assembler/internal/registry"
|
||||
)
|
||||
|
||||
// Paths locates the pieces this package needs.
|
||||
type Paths struct {
|
||||
Root string
|
||||
// Web is the Vite project directory.
|
||||
Web string
|
||||
// Dist is where Vite emits, inside the web workspace.
|
||||
Dist string
|
||||
// Site is the artifact the deploy action uploads, at the repository root.
|
||||
Site string
|
||||
}
|
||||
|
||||
// DefaultPaths derives the standard layout from the repository root.
|
||||
func DefaultPaths(root string) Paths {
|
||||
web := filepath.Join(root, "web")
|
||||
return Paths{
|
||||
Root: root,
|
||||
Web: web,
|
||||
Dist: filepath.Join(web, "dist"),
|
||||
Site: filepath.Join(root, "_site"),
|
||||
}
|
||||
}
|
||||
|
||||
// BuildWeb runs the Vite build.
|
||||
//
|
||||
// Shelling out to npm is not a wart: Vite is a Node tool, and web/ is the only
|
||||
// npm project left in the repository. This stage owns the sequencing, not the
|
||||
// bundling.
|
||||
func BuildWeb(p Paths) error {
|
||||
cmd := exec.Command("npm", "run", "build")
|
||||
cmd.Dir = p.Web
|
||||
cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
return fmt.Errorf("vite build: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Assemble copies the build to one directory per dataset and checks the result.
|
||||
func Assemble(p Paths, datasets []registry.Dataset) error {
|
||||
index := filepath.Join(p.Dist, "index.html")
|
||||
if _, err := os.Stat(index); err != nil {
|
||||
return fmt.Errorf("no build found at %s — run the web build first", p.Dist)
|
||||
}
|
||||
|
||||
if err := os.RemoveAll(p.Site); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(p.Site, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Base build: index.html, assets/, and the gzipped databases from publicDir.
|
||||
if err := copyTree(p.Dist, p.Site); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Unknown paths render the hub rather than the default Pages 404.
|
||||
if err := copyFile(index, filepath.Join(p.Site, "404.html")); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// One entry point per dataset.
|
||||
for _, d := range datasets {
|
||||
dir := filepath.Join(p.Site, d.ID)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := copyFile(index, filepath.Join(dir, "index.html")); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := checkDatabasesPresent(p.Site, datasets); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := checkNoRawDatabases(p.Site); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Printf("assembled %s\n", p.Site)
|
||||
fmt.Printf(" /thptqg/\n /thptqg/404.html\n")
|
||||
for _, d := range datasets {
|
||||
fmt.Printf(" /thptqg/%s\n", d.ID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkDatabasesPresent: every dataset must have shipped its database.
|
||||
//
|
||||
// Without this the site assembles happily with an empty db/ directory — every
|
||||
// page renders, every query 404s, and CI stays green. That is the failure this
|
||||
// catches; the size and row-count guards only run when a database was built at
|
||||
// all.
|
||||
func checkDatabasesPresent(siteDir string, datasets []registry.Dataset) error {
|
||||
var missing []string
|
||||
for _, d := range datasets {
|
||||
gz := filepath.Join(siteDir, "db", d.ID+".db.gz")
|
||||
st, err := os.Stat(gz)
|
||||
if err != nil || st.Size() == 0 {
|
||||
missing = append(missing, d.ID+".db.gz")
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return fmt.Errorf(
|
||||
"no database in the site output for: %s\n"+
|
||||
"Every page would render and every query would 404. Build the databases first",
|
||||
strings.Join(missing, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// rawDatabase matches an uncompressed SQLite artifact, including the temporary
|
||||
// files SQLite leaves mid-build.
|
||||
var rawDatabase = regexp.MustCompile(`\.db(-journal|-wal|-shm)?$`)
|
||||
|
||||
// checkNoRawDatabases rejects an uncompressed database that reached the output.
|
||||
//
|
||||
// The build gzips without keeping the source, so none should exist — but the
|
||||
// staging directory is copied wholesale, and a leftover from an interrupted run
|
||||
// would go straight through. A raw database is 100+ MB.
|
||||
func checkNoRawDatabases(siteDir string) error {
|
||||
var stray []string
|
||||
err := filepath.WalkDir(siteDir, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !d.IsDir() && rawDatabase.MatchString(d.Name()) {
|
||||
stray = append(stray, path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(stray) > 0 {
|
||||
var b strings.Builder
|
||||
b.WriteString("uncompressed database artefact(s) found in the site output:\n")
|
||||
for _, f := range stray {
|
||||
st, _ := os.Stat(f)
|
||||
fmt.Fprintf(&b, " %s (%.1f MB)\n", f, float64(st.Size())/1048576)
|
||||
}
|
||||
b.WriteString("remove them from the staging directory and re-run")
|
||||
return fmt.Errorf("%s", b.String())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func copyTree(src, dst string) error {
|
||||
return filepath.WalkDir(src, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rel, err := filepath.Rel(src, path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
target := filepath.Join(dst, rel)
|
||||
if d.IsDir() {
|
||||
return os.MkdirAll(target, 0o755)
|
||||
}
|
||||
return copyFile(path, target)
|
||||
})
|
||||
}
|
||||
|
||||
func copyFile(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
return out.Close()
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package site
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/thptqg/assembler/internal/registry"
|
||||
)
|
||||
|
||||
var datasets = []registry.Dataset{{ID: "2016"}, {ID: "2017"}}
|
||||
|
||||
// fakeBuild stands in for a Vite build: an index.html, an asset, and whatever
|
||||
// databases the caller wants staged.
|
||||
func fakeBuild(t *testing.T, dbs ...string) Paths {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
dist := filepath.Join(root, "web", "dist")
|
||||
write(t, filepath.Join(dist, "index.html"), "<html>app</html>")
|
||||
write(t, filepath.Join(dist, "assets", "index.js"), "console.log(1)")
|
||||
for _, name := range dbs {
|
||||
write(t, filepath.Join(dist, "db", name), "gzipped-bytes")
|
||||
}
|
||||
return Paths{Root: root, Web: filepath.Join(root, "web"), Dist: dist, Site: filepath.Join(root, "_site")}
|
||||
}
|
||||
|
||||
func write(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssembleProducesAPageForEveryDataset(t *testing.T) {
|
||||
p := fakeBuild(t, "2016.db.gz", "2017.db.gz")
|
||||
if err := Assemble(p, datasets); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"index.html",
|
||||
"404.html",
|
||||
filepath.Join("2016", "index.html"),
|
||||
filepath.Join("2017", "index.html"),
|
||||
filepath.Join("assets", "index.js"),
|
||||
filepath.Join("db", "2016.db.gz"),
|
||||
} {
|
||||
if _, err := os.Stat(filepath.Join(p.Site, want)); err != nil {
|
||||
t.Errorf("missing from the artifact: %s", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestMissingDatabaseFailsTheBuild is the guard that closes the widest hole.
|
||||
//
|
||||
// Without it the site assembles happily with an empty db/ directory: every page
|
||||
// renders, every query 404s, and CI stays green. The row-count and size guards
|
||||
// cannot catch this — they only run when a database was built at all.
|
||||
func TestMissingDatabaseFailsTheBuild(t *testing.T) {
|
||||
p := fakeBuild(t, "2016.db.gz") // 2017 never built
|
||||
err := Assemble(p, datasets)
|
||||
if err == nil {
|
||||
t.Fatal("expected an error when a database is missing")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "2017.db.gz") {
|
||||
t.Errorf("the error should name the missing database, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEmptyDatabaseFailsTheBuild: a zero-byte file satisfies "exists" but is
|
||||
// not a database.
|
||||
func TestEmptyDatabaseFailsTheBuild(t *testing.T) {
|
||||
p := fakeBuild(t, "2016.db.gz", "2017.db.gz")
|
||||
write(t, filepath.Join(p.Dist, "db", "2017.db.gz"), "")
|
||||
if err := Assemble(p, datasets); err == nil {
|
||||
t.Fatal("expected an error for a zero-byte database")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRawDatabaseFailsTheBuild: the compression step deletes its source, so a
|
||||
// raw .db here means an interrupted run left one behind — and it is 100+ MB.
|
||||
func TestRawDatabaseFailsTheBuild(t *testing.T) {
|
||||
for _, name := range []string{"2016.db", "2016.db-journal", "2016.db-wal", "2016.db-shm"} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
p := fakeBuild(t, "2016.db.gz", "2017.db.gz")
|
||||
write(t, filepath.Join(p.Dist, "db", name), "raw sqlite")
|
||||
err := Assemble(p, datasets)
|
||||
if err == nil {
|
||||
t.Fatalf("expected an error for %s", name)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "uncompressed") {
|
||||
t.Errorf("unexpected error: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGzipIsNotMistakenForRaw: the reject pattern is anchored, so a .db.gz must
|
||||
// pass. Getting this wrong would fail every build.
|
||||
func TestGzipIsNotMistakenForRaw(t *testing.T) {
|
||||
if rawDatabase.MatchString("2016.db.gz") {
|
||||
t.Error("a .db.gz must not be treated as an uncompressed database")
|
||||
}
|
||||
for _, name := range []string{"2016.db", "x.db-journal", "x.db-wal", "x.db-shm"} {
|
||||
if !rawDatabase.MatchString(name) {
|
||||
t.Errorf("%s should be treated as an uncompressed artifact", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssembleRejectsAMissingBuild(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
p := Paths{Root: root, Web: root, Dist: filepath.Join(root, "dist"), Site: filepath.Join(root, "_site")}
|
||||
if err := Assemble(p, datasets); err == nil {
|
||||
t.Fatal("expected an error when there is no Vite build")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAssembleIsIdempotent: the site directory is rebuilt from scratch, so a
|
||||
// previous run's leftovers cannot survive into the artifact.
|
||||
func TestAssembleIsIdempotent(t *testing.T) {
|
||||
p := fakeBuild(t, "2016.db.gz", "2017.db.gz")
|
||||
if err := Assemble(p, datasets); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stale := filepath.Join(p.Site, "2015", "index.html")
|
||||
write(t, stale, "old dataset")
|
||||
if err := Assemble(p, datasets); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(stale); !os.IsNotExist(err) {
|
||||
t.Error("a directory from a previous run survived into the new artifact")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
// Command crawl downloads a dataset's source spreadsheets into data/<id>/.
|
||||
//
|
||||
// The argument is the dataset id, the same one used by the parser's configs and
|
||||
// the published site paths.
|
||||
//
|
||||
// crawl 2016 # 119 exam-cluster files
|
||||
// crawl 2017 # 63 province files
|
||||
// crawl 2017 --list # show what would be downloaded
|
||||
//
|
||||
// Each run reads the download links out of the article that published them, so
|
||||
// --list needs network access too. Runs are idempotent: a file already present
|
||||
// and non-empty is skipped, so an interrupted crawl can simply be re-run.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/tiennm99/thptqg/crawler/internal/article"
|
||||
"github.com/tiennm99/thptqg/crawler/internal/fetch"
|
||||
"github.com/tiennm99/thptqg/crawler/internal/sources"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:]); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "crawl: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func usage() {
|
||||
fmt.Fprint(os.Stderr, "usage: crawl <dataset> [flags]\n\nDatasets:\n")
|
||||
for _, s := range sources.All() {
|
||||
fmt.Fprintf(os.Stderr, " %-6s %s\n", s.ID, s.Summary)
|
||||
}
|
||||
fmt.Fprint(os.Stderr, "\nFlags:\n")
|
||||
fmt.Fprint(os.Stderr, " --out string output directory (default ../data/<dataset>)\n")
|
||||
fmt.Fprint(os.Stderr, " --concurrency int parallel downloads (default 6)\n")
|
||||
fmt.Fprint(os.Stderr, " --timeout duration per-file timeout (default 2m)\n")
|
||||
fmt.Fprint(os.Stderr, " --list print the file list and exit\n")
|
||||
}
|
||||
|
||||
func run(args []string) error {
|
||||
if len(args) == 0 || args[0] == "-h" || args[0] == "--help" {
|
||||
usage()
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("no dataset given")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
src, err := sources.Lookup(args[0])
|
||||
if err != nil {
|
||||
usage()
|
||||
return err
|
||||
}
|
||||
|
||||
fs := flag.NewFlagSet(src.ID, flag.ContinueOnError)
|
||||
// The default is relative to the crawler module directory, which is where
|
||||
// both `go -C crawler run ./cmd/crawl` and a manual `cd crawler` land.
|
||||
out := fs.String("out", filepath.Join("..", "data", src.ID), "output directory")
|
||||
concurrency := fs.Int("concurrency", 6, "parallel downloads")
|
||||
timeout := fs.Duration("timeout", 2*time.Minute, "per-file timeout")
|
||||
list := fs.Bool("list", false, "print the file list and exit")
|
||||
if err := fs.Parse(args[1:]); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Ctrl-C cancels the article fetch and any in-flight download; each worker
|
||||
// deletes its partial file on the way out, so an interrupted run leaves
|
||||
// nothing half-written.
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
client := &http.Client{Timeout: *timeout}
|
||||
|
||||
fmt.Printf("Reading %s\n", src.Article)
|
||||
links, err := article.Fetch(ctx, client, src.Article, src.Headers, src.Exts...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files, err := src.Resolve(links)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
outDir, err := filepath.Abs(*out)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
items := make([]fetch.Item, 0, len(files))
|
||||
for _, f := range files {
|
||||
items = append(items, fetch.Item{
|
||||
Name: f.Name,
|
||||
URL: f.URL,
|
||||
Path: filepath.Join(outDir, f.Dest),
|
||||
})
|
||||
}
|
||||
|
||||
if *list {
|
||||
for _, it := range items {
|
||||
fmt.Printf("%s\t%s\n", filepath.Base(it.Path), it.URL)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
fmt.Printf("Downloading %d files to %s...\n", len(items), outDir)
|
||||
|
||||
results, runErr := fetch.Run(ctx, items, fetch.Options{
|
||||
Concurrency: *concurrency,
|
||||
Timeout: *timeout,
|
||||
Headers: src.Headers,
|
||||
OnResult: printResult,
|
||||
})
|
||||
if runErr != nil {
|
||||
return runErr
|
||||
}
|
||||
|
||||
ok, skip, failed := fetch.Tally(results)
|
||||
fmt.Printf("\nDone. ok=%d skip=%d fail=%d\n", ok, skip, len(failed))
|
||||
if len(failed) > 0 {
|
||||
for _, r := range failed {
|
||||
fmt.Fprintf(os.Stderr, " %s: %v\n", r.Item.Name, r.Err)
|
||||
}
|
||||
return fmt.Errorf("%d file(s) failed", len(failed))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func printResult(done, total int, r fetch.Result) {
|
||||
tag := map[fetch.Status]string{
|
||||
fetch.StatusOK: "✓",
|
||||
fetch.StatusSkip: "·",
|
||||
fetch.StatusFail: "✗",
|
||||
}[r.Status]
|
||||
|
||||
detail := ""
|
||||
switch r.Status {
|
||||
case fetch.StatusFail:
|
||||
detail = r.Err.Error()
|
||||
default:
|
||||
detail = fmt.Sprintf("%.0f KB", float64(r.Size)/1024)
|
||||
}
|
||||
fmt.Printf(" %s [%d/%d] %-20s %s\n", tag, done, total, r.Item.Name, detail)
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
module github.com/tiennm99/thptqg/crawler
|
||||
|
||||
go 1.26.5
|
||||
|
||||
require (
|
||||
golang.org/x/net v0.58.0
|
||||
golang.org/x/text v0.41.0
|
||||
)
|
||||
@@ -0,0 +1,4 @@
|
||||
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
|
||||
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
|
||||
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
|
||||
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
|
||||
@@ -0,0 +1,144 @@
|
||||
// Package article reads download links out of a published web page.
|
||||
//
|
||||
// This is what makes the crawler a crawler: the file lists are not carried in
|
||||
// the repository, they are read from the articles that published them, at run
|
||||
// time. A source declares which page to read and how to name what it finds.
|
||||
package article
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/net/html"
|
||||
)
|
||||
|
||||
// Link is one downloadable file found on a page.
|
||||
type Link struct {
|
||||
// URL is absolute, resolved against the page it was found on.
|
||||
URL string
|
||||
// Text is the anchor's visible text, whitespace-collapsed. Some pages use
|
||||
// it for the thing being linked ("Bạc Liêu"), others for boilerplate
|
||||
// ("xem TẠI ĐÂY"), so a source decides whether it is worth anything.
|
||||
Text string
|
||||
// File is the last segment of the URL path, query string excluded.
|
||||
File string
|
||||
}
|
||||
|
||||
// Extract returns every anchor on the page whose target ends in one of exts.
|
||||
//
|
||||
// Pure: no network, so the parsing rules can be tested against a saved copy of
|
||||
// a real page. Order is document order, and duplicates are kept — deciding
|
||||
// whether two links to the same file is a problem belongs to the caller, which
|
||||
// knows what it would name them.
|
||||
func Extract(pageURL string, body io.Reader, exts ...string) ([]Link, error) {
|
||||
base, err := url.Parse(pageURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("bad page URL %q: %w", pageURL, err)
|
||||
}
|
||||
|
||||
doc, err := html.Parse(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse %s: %w", pageURL, err)
|
||||
}
|
||||
|
||||
var out []Link
|
||||
var walk func(*html.Node)
|
||||
walk = func(n *html.Node) {
|
||||
if n.Type == html.ElementNode && n.Data == "a" {
|
||||
if link, ok := linkOf(base, n, exts); ok {
|
||||
out = append(out, link)
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(doc)
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// linkOf turns one <a> element into a Link, or reports that it is not one of
|
||||
// the files being looked for.
|
||||
func linkOf(base *url.URL, n *html.Node, exts []string) (Link, bool) {
|
||||
var href string
|
||||
for _, a := range n.Attr {
|
||||
if strings.EqualFold(a.Key, "href") {
|
||||
href = strings.TrimSpace(a.Val)
|
||||
break
|
||||
}
|
||||
}
|
||||
if href == "" {
|
||||
return Link{}, false
|
||||
}
|
||||
|
||||
// Relative hrefs are the norm on the pages this reads; resolving against
|
||||
// the page URL is what lets a source name a mirror and get its files.
|
||||
ref, err := url.Parse(href)
|
||||
if err != nil {
|
||||
return Link{}, false
|
||||
}
|
||||
abs := base.ResolveReference(ref)
|
||||
|
||||
file := path.Base(abs.Path)
|
||||
if !hasExt(file, exts) {
|
||||
return Link{}, false
|
||||
}
|
||||
|
||||
return Link{URL: abs.String(), Text: textOf(n), File: file}, true
|
||||
}
|
||||
|
||||
func hasExt(name string, exts []string) bool {
|
||||
lower := strings.ToLower(name)
|
||||
for _, e := range exts {
|
||||
if strings.HasSuffix(lower, strings.ToLower(e)) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// textOf collects an element's visible text, collapsing whitespace. Anchors on
|
||||
// these pages wrap the label in styling tags, so the text is rarely a single
|
||||
// child node.
|
||||
func textOf(n *html.Node) string {
|
||||
var b strings.Builder
|
||||
var walk func(*html.Node)
|
||||
walk = func(n *html.Node) {
|
||||
if n.Type == html.TextNode {
|
||||
b.WriteString(n.Data)
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(n)
|
||||
return strings.Join(strings.Fields(b.String()), " ")
|
||||
}
|
||||
|
||||
// Fetch downloads a page and runs Extract on it.
|
||||
func Fetch(ctx context.Context, client *http.Client, pageURL string, headers map[string]string, exts ...string) ([]Link, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, pageURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for k, v := range headers {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fetch %s: %w", pageURL, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("fetch %s: HTTP %d", pageURL, resp.StatusCode)
|
||||
}
|
||||
|
||||
return Extract(pageURL, resp.Body, exts...)
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package article
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const page = "https://example.test/news/scores.html"
|
||||
|
||||
func extract(t *testing.T, body string, exts ...string) []Link {
|
||||
t.Helper()
|
||||
links, err := Extract(page, strings.NewReader(body), exts...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return links
|
||||
}
|
||||
|
||||
// TestResolvesRelativeHrefs is the behaviour the 2016 source depends on: its
|
||||
// article links files as /upload/..., so they only become fetchable once
|
||||
// resolved against the page they were found on.
|
||||
func TestResolvesRelativeHrefs(t *testing.T) {
|
||||
got := extract(t, `
|
||||
<a href="/upload/s/a.xlsx">root-relative</a>
|
||||
<a href="sub/b.xlsx">page-relative</a>
|
||||
<a href="https://cdn.other/c.xlsx">already absolute</a>
|
||||
<a href="//cdn.other/d.xlsx">protocol-relative</a>
|
||||
`, ".xlsx")
|
||||
|
||||
want := []string{
|
||||
"https://example.test/upload/s/a.xlsx",
|
||||
"https://example.test/news/sub/b.xlsx",
|
||||
"https://cdn.other/c.xlsx",
|
||||
"https://cdn.other/d.xlsx",
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %d links, want %d", len(got), len(want))
|
||||
}
|
||||
for i, w := range want {
|
||||
if got[i].URL != w {
|
||||
t.Errorf("link %d = %q, want %q", i, got[i].URL, w)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFiltersByExtension(t *testing.T) {
|
||||
got := extract(t, `
|
||||
<a href="/a.xls">keep</a>
|
||||
<a href="/b.xlsx">keep</a>
|
||||
<a href="/c.pdf">drop</a>
|
||||
<a href="/d.html">drop</a>
|
||||
<a href="/e.XLSX">keep, case-insensitive</a>
|
||||
<a>no href at all</a>
|
||||
`, ".xls", ".xlsx")
|
||||
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("got %d links, want 3: %+v", len(got), got)
|
||||
}
|
||||
// ".xls" must not swallow ".xlsx" or vice versa when only one is asked for.
|
||||
only := extract(t, `<a href="/a.xls">x</a><a href="/b.xlsx">y</a>`, ".xlsx")
|
||||
if len(only) != 1 || only[0].File != "b.xlsx" {
|
||||
t.Errorf("extension filter is too loose: %+v", only)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileIgnoresQueryString: a query string is not part of the filename, and
|
||||
// writing one to disk would produce a name the parser never sees.
|
||||
func TestFileIgnoresQueryString(t *testing.T) {
|
||||
got := extract(t, `<a href="/d/report.xlsx?v=2&t=3">x</a>`, ".xlsx")
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("got %d links", len(got))
|
||||
}
|
||||
if got[0].File != "report.xlsx" {
|
||||
t.Errorf("File = %q, want report.xlsx", got[0].File)
|
||||
}
|
||||
if !strings.Contains(got[0].URL, "v=2") {
|
||||
t.Errorf("the query string must survive in the URL: %q", got[0].URL)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTextCollectsNestedMarkup: both real articles wrap the label in styling
|
||||
// tags, so the anchor text is never a single child node. 2017 names its files
|
||||
// from this text.
|
||||
func TestTextCollectsNestedMarkup(t *testing.T) {
|
||||
got := extract(t, `<a href="/a.xls"><span style="x"><b>Bà Rịa</b> -Vũng Tàu</span></a>`, ".xls")
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("got %d links", len(got))
|
||||
}
|
||||
if got[0].Text != "Bà Rịa -Vũng Tàu" {
|
||||
t.Errorf("Text = %q, want %q", got[0].Text, "Bà Rịa -Vũng Tàu")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEntitiesInHrefAreDecoded(t *testing.T) {
|
||||
got := extract(t, `<a href="/d/a.xlsx?x=1&y=2">x</a>`, ".xlsx")
|
||||
if len(got) != 1 || !strings.Contains(got[0].URL, "x=1&y=2") {
|
||||
t.Errorf("href entity not decoded: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMalformedHTMLStillParses: these are hand-edited CMS pages, and the parser
|
||||
// must not give up on unclosed tags.
|
||||
func TestMalformedHTMLStillParses(t *testing.T) {
|
||||
got := extract(t, `<p><b>list<a href="/a.xls">one<a href="/b.xls">two`, ".xls")
|
||||
if len(got) != 2 {
|
||||
t.Errorf("got %d links, want 2: %+v", len(got), got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractRejectsBadPageURL(t *testing.T) {
|
||||
if _, err := Extract("://nonsense", strings.NewReader("<a href=/a.xls>x</a>"), ".xls"); err == nil {
|
||||
t.Error("expected an error for an unparseable page URL")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchSendsHeadersAndParses(t *testing.T) {
|
||||
var ua, ref string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
ua, ref = r.Header.Get("User-Agent"), r.Header.Get("Referer")
|
||||
w.Write([]byte(`<a href="/x/a.xls">An Giang</a>`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
links, err := Fetch(t.Context(), srv.Client(), srv.URL,
|
||||
map[string]string{"User-Agent": "test-agent", "Referer": "https://ref.test/"}, ".xls")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ua != "test-agent" || ref != "https://ref.test/" {
|
||||
t.Errorf("headers not sent: ua=%q referer=%q", ua, ref)
|
||||
}
|
||||
if len(links) != 1 || links[0].Text != "An Giang" || links[0].File != "a.xls" {
|
||||
t.Errorf("unexpected links: %+v", links)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchFailsLoudly: a CMS that answers 404 or 403 with an HTML error page
|
||||
// would otherwise yield zero links and read as "nothing to download".
|
||||
func TestFetchFailsLoudly(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "<html>not found</html>", http.StatusNotFound)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
if _, err := Fetch(t.Context(), srv.Client(), srv.URL, nil, ".xls"); err == nil {
|
||||
t.Fatal("expected an error for a non-200 response")
|
||||
} else if !strings.Contains(err.Error(), "404") {
|
||||
t.Errorf("error should name the status, got: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
// Package fetch downloads a list of files concurrently, skipping any that are
|
||||
// already on disk.
|
||||
//
|
||||
// It is deliberately source-agnostic: it knows nothing about provinces, exam
|
||||
// years or spreadsheet formats. Each source in internal/sources produces a
|
||||
// []Item and this package moves the bytes.
|
||||
package fetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Item is one file to download.
|
||||
type Item struct {
|
||||
// Name is the human label used in progress output, e.g. "An Giang".
|
||||
Name string
|
||||
URL string
|
||||
// Path is the absolute destination, including filename.
|
||||
Path string
|
||||
}
|
||||
|
||||
// Status is the outcome of one item.
|
||||
type Status int
|
||||
|
||||
const (
|
||||
// StatusOK means the file was downloaded during this run.
|
||||
StatusOK Status = iota
|
||||
// StatusSkip means a non-empty file was already present.
|
||||
StatusSkip
|
||||
// StatusFail means the download did not complete; Result.Err says why.
|
||||
StatusFail
|
||||
)
|
||||
|
||||
// Result records what happened to one item.
|
||||
type Result struct {
|
||||
Item Item
|
||||
Status Status
|
||||
Size int64
|
||||
Err error
|
||||
}
|
||||
|
||||
// Options configures a run. The zero value is usable: Concurrency and Timeout
|
||||
// fall back to defaults.
|
||||
type Options struct {
|
||||
// Concurrency is the number of files downloaded at once.
|
||||
Concurrency int
|
||||
// Timeout bounds each individual request, headers and body together.
|
||||
// The original JS crawler had no timeout, so one stalled connection could
|
||||
// hang the whole run indefinitely.
|
||||
Timeout time.Duration
|
||||
// Headers are sent with every request. The CDN this was written against
|
||||
// rejects requests without a browser User-Agent and a matching Referer.
|
||||
Headers map[string]string
|
||||
// OnResult, if set, is called once per completed item. Calls are
|
||||
// serialised, so it does not need its own locking, but they arrive in
|
||||
// completion order rather than list order.
|
||||
OnResult func(done, total int, r Result)
|
||||
}
|
||||
|
||||
const (
|
||||
defaultConcurrency = 6
|
||||
defaultTimeout = 2 * time.Minute
|
||||
)
|
||||
|
||||
// Run downloads every item, returning one Result each. It does not return an
|
||||
// error for a failed download — that is reported per item — only for a problem
|
||||
// that stops the run as a whole, such as an unusable output directory.
|
||||
//
|
||||
// Results come back in completion order, not input order.
|
||||
func Run(ctx context.Context, items []Item, opts Options) ([]Result, error) {
|
||||
if opts.Concurrency <= 0 {
|
||||
opts.Concurrency = defaultConcurrency
|
||||
}
|
||||
if opts.Timeout <= 0 {
|
||||
opts.Timeout = defaultTimeout
|
||||
}
|
||||
|
||||
// Every item's parent directory must exist before any worker starts, so a
|
||||
// missing output directory fails once here rather than N times in parallel.
|
||||
for _, it := range items {
|
||||
if err := os.MkdirAll(filepath.Dir(it.Path), 0o755); err != nil {
|
||||
return nil, fmt.Errorf("cannot create output directory: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
client := &http.Client{Timeout: opts.Timeout}
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
results = make([]Result, 0, len(items))
|
||||
next int
|
||||
)
|
||||
|
||||
work := make(chan Item)
|
||||
var wg sync.WaitGroup
|
||||
for range opts.Concurrency {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for it := range work {
|
||||
r := download(ctx, client, it, opts.Headers)
|
||||
mu.Lock()
|
||||
results = append(results, r)
|
||||
next++
|
||||
if opts.OnResult != nil {
|
||||
opts.OnResult(next, len(items), r)
|
||||
}
|
||||
mu.Unlock()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
for _, it := range items {
|
||||
select {
|
||||
case work <- it:
|
||||
case <-ctx.Done():
|
||||
close(work)
|
||||
wg.Wait()
|
||||
return results, ctx.Err()
|
||||
}
|
||||
}
|
||||
close(work)
|
||||
wg.Wait()
|
||||
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// download fetches one item, or reports that it was already present.
|
||||
//
|
||||
// The body is streamed to a .part file and renamed into place only once it is
|
||||
// complete. Without that, an interrupted run leaves a truncated file at the
|
||||
// final path — and because the skip check only tests for a non-empty file,
|
||||
// every later run would skip it and the corruption would persist silently.
|
||||
//
|
||||
// A .part left behind by an abrupt kill needs no cleanup: it does not satisfy
|
||||
// the skip check, os.Create truncates it, and the next run re-fetches the file.
|
||||
// The parser ignores it in the meantime, since it reads only .xls and .xlsx.
|
||||
func download(ctx context.Context, client *http.Client, it Item, headers map[string]string) Result {
|
||||
if st, err := os.Stat(it.Path); err == nil && st.Size() > 0 {
|
||||
return Result{Item: it, Status: StatusSkip, Size: st.Size()}
|
||||
}
|
||||
|
||||
fail := func(err error) Result {
|
||||
return Result{Item: it, Status: StatusFail, Err: err}
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, it.URL, nil)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
for k, v := range headers {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fail(fmt.Errorf("HTTP %d", resp.StatusCode))
|
||||
}
|
||||
|
||||
part := it.Path + ".part"
|
||||
f, err := os.Create(part)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
n, err := io.Copy(f, resp.Body)
|
||||
if cerr := f.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(part)
|
||||
return fail(err)
|
||||
}
|
||||
if n == 0 {
|
||||
os.Remove(part)
|
||||
return fail(fmt.Errorf("empty response body"))
|
||||
}
|
||||
if err := os.Rename(part, it.Path); err != nil {
|
||||
os.Remove(part)
|
||||
return fail(err)
|
||||
}
|
||||
|
||||
return Result{Item: it, Status: StatusOK, Size: n}
|
||||
}
|
||||
|
||||
// Tally counts results by status.
|
||||
func Tally(results []Result) (ok, skip int, failed []Result) {
|
||||
for _, r := range results {
|
||||
switch r.Status {
|
||||
case StatusOK:
|
||||
ok++
|
||||
case StatusSkip:
|
||||
skip++
|
||||
case StatusFail:
|
||||
failed = append(failed, r)
|
||||
}
|
||||
}
|
||||
return ok, skip, failed
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package fetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func serve(t *testing.T, h http.HandlerFunc) string {
|
||||
t.Helper()
|
||||
s := httptest.NewServer(h)
|
||||
t.Cleanup(s.Close)
|
||||
return s.URL
|
||||
}
|
||||
|
||||
func TestDownloadsAndSkips(t *testing.T) {
|
||||
var hits int
|
||||
url := serve(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
hits++
|
||||
w.Write([]byte("payload"))
|
||||
})
|
||||
|
||||
dir := t.TempDir()
|
||||
items := []Item{{Name: "one", URL: url, Path: filepath.Join(dir, "one.xls")}}
|
||||
|
||||
results, err := Run(context.Background(), items, Options{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if results[0].Status != StatusOK {
|
||||
t.Fatalf("status = %v, err = %v", results[0].Status, results[0].Err)
|
||||
}
|
||||
if got, _ := os.ReadFile(items[0].Path); string(got) != "payload" {
|
||||
t.Errorf("content = %q", got)
|
||||
}
|
||||
|
||||
// Second run must not re-fetch.
|
||||
results, err = Run(context.Background(), items, Options{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if results[0].Status != StatusSkip {
|
||||
t.Errorf("status = %v, want StatusSkip", results[0].Status)
|
||||
}
|
||||
if hits != 1 {
|
||||
t.Errorf("server hit %d times, want 1", hits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNon200Fails(t *testing.T) {
|
||||
url := serve(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "gone", http.StatusNotFound)
|
||||
})
|
||||
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "missing.xls")
|
||||
results, err := Run(context.Background(), []Item{{Name: "x", URL: url, Path: path}}, Options{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if results[0].Status != StatusFail {
|
||||
t.Fatalf("status = %v, want StatusFail", results[0].Status)
|
||||
}
|
||||
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
||||
t.Error("a failed download must leave no file at the destination")
|
||||
}
|
||||
assertNoPartFiles(t, dir)
|
||||
}
|
||||
|
||||
// TestFailureLeavesNoPartial is the reason downloads land on a .part file
|
||||
// first. Writing straight to the destination would leave a truncated file
|
||||
// there, and the skip check — which only tests for a non-empty file — would
|
||||
// skip it on every later run, so the corruption would never be re-fetched.
|
||||
func TestFailureLeavesNoPartial(t *testing.T) {
|
||||
url := serve(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Length", "1000")
|
||||
w.Write([]byte("short"))
|
||||
// Closing early makes the body read fail mid-copy.
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
panic(http.ErrAbortHandler)
|
||||
})
|
||||
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "truncated.xls")
|
||||
results, err := Run(context.Background(), []Item{{Name: "x", URL: url, Path: path}}, Options{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if results[0].Status != StatusFail {
|
||||
t.Fatalf("status = %v, want StatusFail", results[0].Status)
|
||||
}
|
||||
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
||||
t.Error("a truncated download must not be left at the destination")
|
||||
}
|
||||
assertNoPartFiles(t, dir)
|
||||
}
|
||||
|
||||
func TestEmptyBodyFails(t *testing.T) {
|
||||
url := serve(t, func(w http.ResponseWriter, _ *http.Request) {})
|
||||
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "empty.xls")
|
||||
results, err := Run(context.Background(), []Item{{Name: "x", URL: url, Path: path}}, Options{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if results[0].Status != StatusFail {
|
||||
t.Errorf("an empty body must fail, got %v", results[0].Status)
|
||||
}
|
||||
assertNoPartFiles(t, dir)
|
||||
}
|
||||
|
||||
func TestHeadersAreSent(t *testing.T) {
|
||||
var gotUA, gotRef string
|
||||
url := serve(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
gotUA, gotRef = r.Header.Get("User-Agent"), r.Header.Get("Referer")
|
||||
w.Write([]byte("ok"))
|
||||
})
|
||||
|
||||
_, err := Run(context.Background(),
|
||||
[]Item{{Name: "x", URL: url, Path: filepath.Join(t.TempDir(), "x.xls")}},
|
||||
Options{Headers: map[string]string{"User-Agent": "test-agent", "Referer": "https://example.test/"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if gotUA != "test-agent" || gotRef != "https://example.test/" {
|
||||
t.Errorf("headers not sent: ua=%q referer=%q", gotUA, gotRef)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllItemsRun(t *testing.T) {
|
||||
url := serve(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Write([]byte("x"))
|
||||
})
|
||||
|
||||
dir := t.TempDir()
|
||||
var items []Item
|
||||
for i := range 20 {
|
||||
items = append(items, Item{
|
||||
Name: "f",
|
||||
URL: url,
|
||||
Path: filepath.Join(dir, "sub", string(rune('a'+i))+".xls"),
|
||||
})
|
||||
}
|
||||
|
||||
results, err := Run(context.Background(), items, Options{Concurrency: 4})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(results) != len(items) {
|
||||
t.Fatalf("got %d results, want %d", len(results), len(items))
|
||||
}
|
||||
ok, _, failed := Tally(results)
|
||||
if ok != len(items) {
|
||||
t.Errorf("ok = %d, want %d (failures: %v)", ok, len(items), failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTally(t *testing.T) {
|
||||
ok, skip, failed := Tally([]Result{
|
||||
{Status: StatusOK}, {Status: StatusOK}, {Status: StatusSkip}, {Status: StatusFail},
|
||||
})
|
||||
if ok != 2 || skip != 1 || len(failed) != 1 {
|
||||
t.Errorf("ok=%d skip=%d fail=%d", ok, skip, len(failed))
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoPartFiles(t *testing.T, dir string) {
|
||||
t.Helper()
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if filepath.Ext(e.Name()) == ".part" {
|
||||
t.Errorf("leftover partial file: %s", e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package sources
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/tiennm99/thptqg/crawler/internal/article"
|
||||
)
|
||||
|
||||
// source2016 fetches the 2016 dataset: one spreadsheet per exam cluster
|
||||
// (cụm thi), 4 .xls and 115 .xlsx.
|
||||
//
|
||||
// The article is a mirror. The site that first published this list
|
||||
// (dtntbacgiang.edu.vn) no longer resolves; this copy of the same article is
|
||||
// still online, and its links are read at crawl time like any other source.
|
||||
//
|
||||
// Dest keeps the server's own filename verbatim — a 32-hex content hash, the
|
||||
// cluster slug, then a millisecond timestamp. Two reasons not to prettify it:
|
||||
//
|
||||
// - parser sorts inputs bytewise and inserts last-wins, so filenames decide
|
||||
// which row survives a duplicate exam number. That is live here, not
|
||||
// hypothetical: 877,464 source rows collapse to 877,461, so three rows'
|
||||
// contents depend on this ordering.
|
||||
// - parser/testdata/reader-fidelity-hashes.tsv is keyed by full path, and
|
||||
// it is frozen — it was produced by the Rust reader, which no longer exists.
|
||||
//
|
||||
// Unlike 2017 this needs no transliteration: the name comes from the URL, so it
|
||||
// is exact by construction rather than derived from a label.
|
||||
var source2016 = Source{
|
||||
ID: "2016",
|
||||
Summary: "119 exam-cluster files (4 .xls + 115 .xlsx)",
|
||||
Article: "https://dtnt.bacninh.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html",
|
||||
Exts: []string{".xls", ".xlsx"},
|
||||
WantFiles: 119,
|
||||
Dest: func(l article.Link) (string, error) {
|
||||
if l.File == "" {
|
||||
return "", fmt.Errorf("link %s has no filename", l.URL)
|
||||
}
|
||||
return l.File, nil
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package sources
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/text/unicode/norm"
|
||||
|
||||
"github.com/tiennm99/thptqg/crawler/internal/article"
|
||||
)
|
||||
|
||||
// browserUA and the Referer are both required: the CDN 403s an unrecognised
|
||||
// User-Agent, and rejects requests that do not carry the article as Referer.
|
||||
const browserUA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
|
||||
|
||||
const article2017 = "https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm"
|
||||
|
||||
// source2017 fetches the 2017 dataset: one .xls per province, still served from
|
||||
// the CDN that originally published it.
|
||||
//
|
||||
// Dest cannot use the CDN's filenames the way 2016 does, because they are
|
||||
// inconsistent — Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls, Gia-Lai.xls —
|
||||
// carrying upload-order prefixes and arbitrary capitalisation. The province name
|
||||
// in the link text is the stable identifier, so the local name is derived from
|
||||
// that instead, which is what produced the files in data/2017/.
|
||||
var source2017 = Source{
|
||||
ID: "2017",
|
||||
Summary: "63 province .xls files from the baotintuc.vn CDN",
|
||||
Article: article2017,
|
||||
Exts: []string{".xls"},
|
||||
WantFiles: 63,
|
||||
Headers: map[string]string{
|
||||
"User-Agent": browserUA,
|
||||
"Referer": article2017,
|
||||
},
|
||||
Dest: func(l article.Link) (string, error) {
|
||||
s := slug(l.Text)
|
||||
if s == "" {
|
||||
return "", fmt.Errorf("link %s has no usable label to name it by", l.URL)
|
||||
}
|
||||
return s + ".xls", nil
|
||||
},
|
||||
}
|
||||
|
||||
var nonAlphanumeric = regexp.MustCompile(`[^a-z0-9]+`)
|
||||
|
||||
// slug turns a province name into a filename stem: transliterated to ASCII,
|
||||
// lowercased, with every run of other characters collapsed to one hyphen.
|
||||
//
|
||||
// "Bà Rịa -Vũng Tàu" -> "ba-ria-vung-tau"
|
||||
// "Bắc Kạn" -> "bac-kan"
|
||||
//
|
||||
// The article writes the names with diacritics, so this has to strip them.
|
||||
// Combining marks are removed by the literal range U+0300–U+036F rather than by
|
||||
// the unicode.Mn category, matching what parser does to build ho_ten_ascii.
|
||||
func slug(name string) string {
|
||||
var b strings.Builder
|
||||
for _, r := range norm.NFD.String(name) {
|
||||
switch {
|
||||
case r >= 0x0300 && r <= 0x036F:
|
||||
// combining mark: drop
|
||||
case r == 'đ' || r == 'Đ':
|
||||
b.WriteByte('d')
|
||||
default:
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
return strings.Trim(nonAlphanumeric.ReplaceAllString(strings.ToLower(b.String()), "-"), "-")
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
// Package sources says where each dataset's spreadsheets are published and
|
||||
// what to call them locally.
|
||||
//
|
||||
// It carries no link lists. Each source names the article that published the
|
||||
// files; internal/article reads the links out of that page at run time, and
|
||||
// internal/fetch moves the bytes.
|
||||
package sources
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/tiennm99/thptqg/crawler/internal/article"
|
||||
)
|
||||
|
||||
// Source is one crawlable dataset.
|
||||
type Source struct {
|
||||
// ID is the dataset id: the subcommand, the directory under data/, and the
|
||||
// parser config name, all at once. Keeping it single means a source
|
||||
// cannot be pointed at the wrong dataset's directory.
|
||||
ID string
|
||||
Summary string
|
||||
|
||||
// Article is the published page listing this dataset's files.
|
||||
Article string
|
||||
|
||||
// Exts are the file extensions to pick out of that page.
|
||||
Exts []string
|
||||
|
||||
// Headers are sent with every request for this source, for the article and
|
||||
// the files alike.
|
||||
Headers map[string]string
|
||||
|
||||
// WantFiles is how many links the article is expected to yield. A page that
|
||||
// suddenly yields fewer has changed shape, and silently crawling a partial
|
||||
// dataset is the failure this exists to prevent — parser would happily
|
||||
// build a short database and only the row-count guard would catch it, after
|
||||
// the fact.
|
||||
WantFiles int
|
||||
|
||||
// Dest names the local file for one discovered link.
|
||||
//
|
||||
// This is the load-bearing part. parser sorts input files bytewise and
|
||||
// inserts last-wins, so the names chosen here decide which row survives a
|
||||
// duplicate exam number. Two sources answer it differently and both have a
|
||||
// reason: see source_2016.go and source_2017.go.
|
||||
Dest func(article.Link) (string, error)
|
||||
}
|
||||
|
||||
// File is one spreadsheet to download.
|
||||
type File struct {
|
||||
// Name is the human label used in progress output.
|
||||
Name string
|
||||
URL string
|
||||
// Dest is the filename within the dataset directory.
|
||||
Dest string
|
||||
}
|
||||
|
||||
// Resolve turns the links found on a source's article into the files to fetch,
|
||||
// enforcing the count and rejecting any two links that would write to the same
|
||||
// name — which would silently cost the dataset a file.
|
||||
func (s Source) Resolve(links []article.Link) ([]File, error) {
|
||||
if len(links) != s.WantFiles {
|
||||
return nil, fmt.Errorf(
|
||||
"%s: found %d links on %s, expected %d — the page has changed shape; "+
|
||||
"check it before crawling, a partial dataset builds without complaint",
|
||||
s.ID, len(links), s.Article, s.WantFiles)
|
||||
}
|
||||
|
||||
out := make([]File, 0, len(links))
|
||||
seen := make(map[string]string, len(links))
|
||||
for _, l := range links {
|
||||
dest, err := s.Dest(l)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", s.ID, err)
|
||||
}
|
||||
if prev, dup := seen[dest]; dup {
|
||||
return nil, fmt.Errorf("%s: %q and %q both name the local file %q",
|
||||
s.ID, prev, l.URL, dest)
|
||||
}
|
||||
seen[dest] = l.URL
|
||||
|
||||
name := l.Text
|
||||
if name == "" {
|
||||
name = dest
|
||||
}
|
||||
out = append(out, File{Name: name, URL: l.URL, Dest: dest})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// registry is ordered: it drives the help output.
|
||||
var registry = []Source{source2016, source2017}
|
||||
|
||||
// All returns every known source, in help order.
|
||||
func All() []Source { return registry }
|
||||
|
||||
// Lookup finds a source by ID.
|
||||
func Lookup(id string) (Source, error) {
|
||||
for _, s := range registry {
|
||||
if s.ID == id {
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
return Source{}, fmt.Errorf("unknown dataset %q", id)
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package sources
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/thptqg/crawler/internal/article"
|
||||
)
|
||||
|
||||
// The fixtures are saved copies of the two source articles, kept so the link
|
||||
// extraction and the naming rules can be exercised without the network.
|
||||
//
|
||||
// article-2017 is the live page. article-2016 came from the Internet Archive's
|
||||
// copy of the site that first published that list (dtntbacgiang.edu.vn, which
|
||||
// no longer resolves); the mirror the crawler actually reads carries the same
|
||||
// article. Its hrefs are relative, so they resolve against whichever host the
|
||||
// source names, and the filenames are identical either way.
|
||||
func fixture(t *testing.T, id string) *gzip.Reader {
|
||||
t.Helper()
|
||||
f, err := os.Open(filepath.Join("testdata", "article-"+id+".html.gz"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { f.Close() })
|
||||
gz, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { gz.Close() })
|
||||
return gz
|
||||
}
|
||||
|
||||
// resolveFixture runs a source's full pipeline over its saved article.
|
||||
func resolveFixture(t *testing.T, src Source) []File {
|
||||
t.Helper()
|
||||
links, err := article.Extract(src.Article, fixture(t, src.ID), src.Exts...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files, err := src.Resolve(links)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return files
|
||||
}
|
||||
|
||||
// TestReproducesFilesOnDisk is the guard that matters.
|
||||
//
|
||||
// parser sorts its inputs and inserts last-wins, so filenames decide which
|
||||
// row survives a duplicate exam number. If extraction or naming drifted, a
|
||||
// re-crawl could rebuild a database with the same row count and different
|
||||
// content, which the row-count guard in build-db.js would not catch.
|
||||
//
|
||||
// The committed data/<id>/ is the oracle: reading the source article and
|
||||
// applying the source's naming rule must reproduce it exactly, both directions.
|
||||
func TestReproducesFilesOnDisk(t *testing.T) {
|
||||
for _, src := range All() {
|
||||
t.Run(src.ID, func(t *testing.T) {
|
||||
dir := filepath.Join("..", "..", "..", "data", src.ID)
|
||||
entries, err := os.ReadDir(dir)
|
||||
if os.IsNotExist(err) {
|
||||
t.Skipf("%s not present", dir)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
onDisk := map[string]bool{}
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() {
|
||||
onDisk[e.Name()] = true
|
||||
}
|
||||
}
|
||||
|
||||
for _, f := range resolveFixture(t, src) {
|
||||
if !onDisk[f.Dest] {
|
||||
t.Errorf("crawler would write %q, which is not in %s — "+
|
||||
"a renamed input can change which duplicate row survives", f.Dest, dir)
|
||||
}
|
||||
delete(onDisk, f.Dest)
|
||||
}
|
||||
for name := range onDisk {
|
||||
t.Errorf("%s/%s exists but the article produces no link for it", dir, name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolvedURLsAreAbsolute: 2016's article uses relative hrefs, so a
|
||||
// mis-resolved base would produce URLs that cannot be fetched at all.
|
||||
func TestResolvedURLsAreAbsolute(t *testing.T) {
|
||||
for _, src := range All() {
|
||||
t.Run(src.ID, func(t *testing.T) {
|
||||
for _, f := range resolveFixture(t, src) {
|
||||
if !strings.HasPrefix(f.URL, "https://") {
|
||||
t.Errorf("%s: %q is not an absolute https URL", f.Dest, f.URL)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtensionsMatchWhatIsExpected(t *testing.T) {
|
||||
counts := map[string]map[string]int{}
|
||||
for _, src := range All() {
|
||||
c := map[string]int{}
|
||||
for _, f := range resolveFixture(t, src) {
|
||||
c[filepath.Ext(f.Dest)]++
|
||||
}
|
||||
counts[src.ID] = c
|
||||
}
|
||||
if got := counts["2016"]; got[".xls"] != 4 || got[".xlsx"] != 115 {
|
||||
t.Errorf("2016: %d .xls and %d .xlsx, want 4 and 115", got[".xls"], got[".xlsx"])
|
||||
}
|
||||
if got := counts["2017"]; got[".xls"] != 63 {
|
||||
t.Errorf("2017: %d .xls, want 63", got[".xls"])
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveRejectsShortPage: a page that has changed shape must stop the
|
||||
// crawl, not quietly produce a partial dataset.
|
||||
func TestResolveRejectsShortPage(t *testing.T) {
|
||||
src := source2017
|
||||
_, err := src.Resolve([]article.Link{{URL: "https://x/a.xls", Text: "An Giang", File: "a.xls"}})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error for a short link list")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "expected 63") {
|
||||
t.Errorf("error should name the expected count, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveRejectsCollidingNames: two links naming the same local file would
|
||||
// cost the dataset a file with no error anywhere downstream.
|
||||
func TestResolveRejectsCollidingNames(t *testing.T) {
|
||||
src := Source{
|
||||
ID: "t", WantFiles: 2,
|
||||
Dest: func(article.Link) (string, error) { return "same.xls", nil },
|
||||
}
|
||||
_, err := src.Resolve([]article.Link{
|
||||
{URL: "https://x/a.xls", File: "a.xls"},
|
||||
{URL: "https://x/b.xls", File: "b.xls"},
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "same.xls") {
|
||||
t.Errorf("expected a collision error naming the file, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlug(t *testing.T) {
|
||||
// The article writes province names with diacritics; the committed
|
||||
// filenames are ASCII. These are the awkward ones.
|
||||
for in, want := range map[string]string{
|
||||
"An Giang": "an-giang",
|
||||
"Bạc Liêu": "bac-lieu",
|
||||
"Bắc Kạn": "bac-kan",
|
||||
"Bà Rịa -Vũng Tàu": "ba-ria-vung-tau",
|
||||
"Thừa Thiên - Huế": "thua-thien-hue",
|
||||
"Đắk Lắk": "dak-lak",
|
||||
} {
|
||||
if got := slug(in); got != want {
|
||||
t.Errorf("slug(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookup(t *testing.T) {
|
||||
for _, s := range All() {
|
||||
got, err := Lookup(s.ID)
|
||||
if err != nil {
|
||||
t.Errorf("Lookup(%q): %v", s.ID, err)
|
||||
}
|
||||
if got.ID != s.ID || got.Article == "" || got.WantFiles == 0 {
|
||||
t.Errorf("Lookup(%q) returned an incomplete source: %+v", s.ID, got)
|
||||
}
|
||||
}
|
||||
if _, err := Lookup("nope"); err == nil {
|
||||
t.Error("Lookup of an unknown dataset must fail")
|
||||
}
|
||||
}
|
||||
|
||||
// TestIDsAreDatasetIDs: the ID doubles as the directory under data/, so a typo
|
||||
// would send a crawl into a directory the parser never reads.
|
||||
func TestIDsAreDatasetIDs(t *testing.T) {
|
||||
for _, s := range All() {
|
||||
dir := filepath.Join("..", "..", "..", "data", s.ID)
|
||||
if _, err := os.Stat(dir); os.IsNotExist(err) {
|
||||
t.Errorf("source %q has no dataset directory at %s", s.ID, dir)
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user