diff --git a/.gitignore b/.gitignore index d86052c..1ea055c 100644 --- a/.gitignore +++ b/.gitignore @@ -18,7 +18,6 @@ dist/ .build/ ### Rust build artefacts ### -parser/target/ ### Assembled Pages artifact ### _site/ diff --git a/README.md b/README.md index 883ccf6..02c569d 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ docs/ architecture, data pipeline, deployment The dataset id is one identifier end to end: ``` -data/2017-old/ → parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/ +data/2017-old/ → go-parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/ ``` ## Build @@ -53,7 +53,7 @@ Pushing to `main` runs the same steps in ## Adding a dataset 1. Put the Excel files in `data//` -2. Add `parser/configs/.yml` — sheet mode, column indices, validation +2. Add `go-parser/configs/.yml` — sheet mode, column indices, validation guards. No SQL; the schema is canonical. 3. Add an entry to `DATASETS` in `src/datasets.js` diff --git a/docs/data-pipeline.md b/docs/data-pipeline.md index ef3e1d6..bbfe48d 100644 --- a/docs/data-pipeline.md +++ b/docs/data-pipeline.md @@ -2,9 +2,9 @@ From raw Excel files to a compressed SQLite file the browser can load. -One Rust binary (`parser/`) builds every dataset. What differs per dataset is +One Rust binary (`go-parser/`) builds every dataset. What differs per dataset is parse rules only — sheet strategy, column layout, validation guards — declared -in `parser/configs/.yml`. The table shape, the INSERT and the subject +in `go-parser/configs/.yml`. The table shape, the INSERT and the subject regexes are canonical and live in `go-parser/internal/schema/schema.go`. ## Sources @@ -19,7 +19,7 @@ regexes are canonical and live in `go-parser/internal/schema/schema.go`. Only `2017` can be re-fetched: ```bash -node parser/scripts/crawl-baotintuc.js +node go-parser/scripts/crawl-baotintuc.js ``` Idempotent — skips files already present, saves to `data/2017/`. Source article: @@ -116,34 +116,44 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what ## Verifying a rebuild -`parser/scripts/db-stats.js` dumps row counts, per-column non-NULL counts, file -size and a deterministic student sample as JSON. -`parser/scripts/verify-parity.js` diffs two such files and exits non-zero on any -mismatch. +`npm run build:db` verifies itself: each database's row count must match the +figure in the table above, and each `.db.gz` must be at least 90% of its usual +size, or the build fails rather than publishing. That guard is the reason a +truncated dataset cannot reach the site with a green pipeline. + +For a deeper check, `go-parser/scripts/differential-parity.mjs` compares two sets +of databases field-by-field — row counts, per-column non-NULL counts, a +full-table SHA-256 over every row ordered by `so_bao_danh`, schema metadata, and +build stdout: ```bash -node parser/scripts/db-stats.js 2016=.db … > current.json -node parser/scripts/verify-parity.js plans/reports/parser-parity-baseline.json current.json +node go-parser/scripts/differential-parity.mjs \ + --rust /path/to/a-{id}.db --go /path/to/b-{id}.db ``` -The committed baseline was built with the pre-refactor code and cannot be -regenerated — the two old crates no longer exist. Both scripts use the built-in -`node:sqlite`, so they need no dependencies. +It exits non-zero on any mismatch and fails loudly if a dataset is missing rather +than skipping it. Written for the Rust-to-Go migration, it works for any two +builds. Uses the built-in `node:sqlite`, so it needs no dependencies. + +`go-parser/internal/reader` additionally carries a frozen oracle of per-file +cell-dump hashes covering all 299 inputs; `npm run test:go` fails if any single +cell of any input file reads differently. ## Refreshing the 2017 data ```bash rm data/2017/*.xls -node parser/scripts/crawl-baotintuc.js +node go-parser/scripts/crawl-baotintuc.js node go-parser/scripts/build-db.js 2017 ``` -Then re-run the parity check above and confirm the row count still matches. +The row-count guard in `build:db` confirms the rebuild matches the expected total. -## Legacy scripts +## Removed scripts -`parser/scripts/check-duplicates.js` and `diff-datasets.js` are one-off audits -that were already broken before the repo was unified — a hardcoded Windows path -in one, an undeclared `better-sqlite3` dependency and stale paths in the other. -Each carries a comment saying so. For comparing two builds, use `db-stats.js` -plus `verify-parity.js` instead. +`check-duplicates.js`, `diff-datasets.js`, `db-stats.js` and `verify-parity.js` +were dropped with the Rust parser. The first two had been broken since before the +repo was unified (a hardcoded Windows path in one, an undeclared +`better-sqlite3` dependency in the other) and neither had any automated caller. +The latter two are superseded by `differential-parity.mjs`, which compares more +and cannot silently skip a dataset. diff --git a/docs/deployment-guide.md b/docs/deployment-guide.md index 65af318..80710bc 100644 --- a/docs/deployment-guide.md +++ b/docs/deployment-guide.md @@ -56,7 +56,7 @@ up as a blank page with 404s on `/assets/...`. ## Adding a dataset 1. Put the Excel files in `data//` -2. Add `parser/configs/.yml` with the parse rules — sheet mode, column +2. Add `go-parser/configs/.yml` with the parse rules — sheet mode, column indices, SBD validation, header tokens, blank-row stripping. No SQL: the schema is canonical and lives in `go-parser/internal/schema/schema.go` 3. Add an entry to `DATASETS` in `src/datasets.js` diff --git a/docs/system-architecture.md b/docs/system-architecture.md index e525488..ab5040d 100644 --- a/docs/system-architecture.md +++ b/docs/system-architecture.md @@ -31,7 +31,7 @@ data//*.xls(x) One identifier ties the whole pipeline together: ``` -data/2017-old/ → parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/ +data/2017-old/ → go-parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/ ``` `src/datasets.js` declares the four ids once. The frontend, the database build diff --git a/eslint.config.js b/eslint.config.js index 097adc2..7ce9419 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -28,7 +28,7 @@ export default defineConfig([ }, { // Node-executed files (Vite config, parser tooling) run with Node globals. - files: ['vite.config.js', 'scripts/**/*.js', 'go-parser/scripts/**/*.js', 'parser/scripts/**/*.js'], + files: ['vite.config.js', 'scripts/**/*.js', 'go-parser/scripts/**/*.js'], languageOptions: { globals: { ...globals.node }, }, diff --git a/go-parser/README.md b/go-parser/README.md new file mode 100644 index 0000000..9d25b24 --- /dev/null +++ b/go-parser/README.md @@ -0,0 +1,67 @@ +# go-parser + +Reads the `.xls`/`.xlsx` source spreadsheets in `data/` and writes one SQLite +database per dataset. + +```bash +npm run build:go # compile go-parser/bin/xlsxread +npm run build:db # build + verify + gzip all four datasets +npm run test:go # unit tests + the 299-file reader-fidelity suite +``` + +``` +xlsxread build --schema go-parser/configs/.yml --input data/ --output +xlsxread audit --schema go-parser/configs/.yml --input data/ --db +``` + +## Layout + +| path | role | +|---|---| +| `internal/reader` | spreadsheet reading; the only place that knows about file formats | +| `internal/ingest` | dataset policy — sheet selection, header skipping, blank rows, the build loop, and the 2016 per-sheet format detection | +| `internal/transform` | `ToAscii`, score-regex parsing, row validation | +| `internal/schema` | the canonical 22-column table: DDL, INSERT, subject regexes | +| `internal/config` | per-dataset YAML parse rules | +| `internal/writer` | SQLite lifecycle and the stats block | +| `internal/audit` | source-vs-database SBD comparison | + +The reader deliberately knows nothing about datasets: it reports every sheet and +every row verbatim. All policy lives in `ingest`. That split is what made the +reader independently verifiable against a hash oracle. + +## Provenance + +This is a port of a Rust crate that lived at `parser/` until the Go +implementation reached full parity. Source comments cite the original by file +and line (`parser/src/transform.rs:56` and similar); those paths resolve at the +tag **`pre-go-parser-removal`**, the last commit containing the Rust code. + +The port was gated on a field-by-field comparison of both implementations across +all four datasets — 3,265,641 rows with identical full-table SHA-256, identical +per-column non-NULL counts, identical schema metadata and identical build +stdout. `scripts/differential-parity.mjs` is that comparator and still runs +against any two sets of databases. + +Behaviour was matched bug-for-bug, deliberately. Several quirks look like +defects and are load-bearing for the published data: + +- a parsed score of `0` becomes NULL in the 2016 separate-scores layout, + replicating a JavaScript falsy check; +- `ToAscii` strips combining marks in the literal range U+0300–U+036F rather + than by Unicode category, which is narrower; +- gender is a two-value allowlist, and anything else becomes NULL; +- `diem_thi` is read untrimmed while the other three fields are trimmed; +- the `"SINH "` header token carries a trailing space. + +Each has a test naming it, so none can be tidied away by accident. + +## Verification + +`testdata/reader-fidelity-hashes.tsv` holds a SHA-256 per input file over a +canonical dump of every cell of every sheet. It is **frozen**: it was produced +by the Rust reader, which no longer exists, so it cannot be regenerated. It +still fails if any single cell of any of the 299 files reads differently. + +`npm run build:db` refuses to publish a database whose row count does not match +the known figure, or whose artifact is under 90% of its usual size. diff --git a/parser/configs/2016.yml b/go-parser/configs/2016.yml similarity index 100% rename from parser/configs/2016.yml rename to go-parser/configs/2016.yml diff --git a/parser/configs/2017-old.yml b/go-parser/configs/2017-old.yml similarity index 100% rename from parser/configs/2017-old.yml rename to go-parser/configs/2017-old.yml diff --git a/parser/configs/2017-old2.yml b/go-parser/configs/2017-old2.yml similarity index 100% rename from parser/configs/2017-old2.yml rename to go-parser/configs/2017-old2.yml diff --git a/parser/configs/2017.yml b/go-parser/configs/2017.yml similarity index 100% rename from parser/configs/2017.yml rename to go-parser/configs/2017.yml diff --git a/go-parser/internal/config/config_test.go b/go-parser/internal/config/config_test.go index 8a88ad5..c3a1a1c 100644 --- a/go-parser/internal/config/config_test.go +++ b/go-parser/internal/config/config_test.go @@ -155,7 +155,7 @@ func TestLoadRealConfigs(t *testing.T) { } for id, w := range want { t.Run(id, func(t *testing.T) { - cfg, err := Load(filepath.Join(root, "parser", "configs", id+".yml")) + cfg, err := Load(filepath.Join(root, "go-parser", "configs", id+".yml")) if err != nil { t.Fatalf("load: %v", err) } @@ -217,7 +217,7 @@ func repoRoot(t *testing.T) string { t.Fatalf("getwd: %v", err) } for i := 0; i < 6; i++ { - if _, err := os.Stat(filepath.Join(dir, "parser", "configs")); err == nil { + if _, err := os.Stat(filepath.Join(dir, "go-parser", "configs")); err == nil { return dir } dir = filepath.Dir(dir) diff --git a/go-parser/scripts/build-db.js b/go-parser/scripts/build-db.js index 874bbe9..f3ac9da 100644 --- a/go-parser/scripts/build-db.js +++ b/go-parser/scripts/build-db.js @@ -81,7 +81,7 @@ for (const id of targets) { [ "build", "--schema", - resolve(ROOT, `parser/configs/${id}.yml`), + resolve(ROOT, `go-parser/configs/${id}.yml`), "--input", resolve(ROOT, `data/${id}`), "--output", diff --git a/parser/scripts/crawl-baotintuc.js b/go-parser/scripts/crawl-baotintuc.js similarity index 100% rename from parser/scripts/crawl-baotintuc.js rename to go-parser/scripts/crawl-baotintuc.js diff --git a/go-parser/scripts/regen-fidelity-hashes.sh b/go-parser/scripts/regen-fidelity-hashes.sh deleted file mode 100755 index c26d5d2..0000000 --- a/go-parser/scripts/regen-fidelity-hashes.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env bash -# Regenerates the reader-fidelity oracle from the Rust/calamine ground truth. -# -# Emits SHA-256 per input file over the canonical cell dump. Only hashes are -# committed: the dumps are real student PII, and parser/tests/fixtures/README.md -# establishes that fixtures in this repo carry synthetic data only. -# -# Requires the Rust parser to still build. Run from the repo root. -set -euo pipefail - -OUT=go-parser/testdata/reader-fidelity-hashes.tsv -CANON='BEGIN{OFS="\t"} - $1=="FILE"{next} - $1=="SHEETCOUNT"{print;next} - $1=="SHEET"{print $1,$2,$3,$4,$5;next} - $1=="ROW"{print;next} - $1=="CELL"{print $1,$2,$3,$4,$7;next}' - -{ - echo "# Canonical cell-dump SHA-256 per input file, produced by the Rust/calamine" - echo "# ground truth (parser/examples/dump_cells.rs). The Go reader must reproduce" - echo "# each hash exactly. Hashes only - the dumps themselves are real student PII" - echo "# and are never committed, per parser/tests/fixtures/README.md." - echo "# Regenerate: go-parser/scripts/regen-fidelity-hashes.sh" -} > "$OUT" - -for f in data/2016/* data/2017/* data/2017-old/* data/2017-old2/*; do - h=$(cargo run --release --quiet --manifest-path parser/Cargo.toml \ - --example dump_cells -- "$f" /dev/stdout \ - | awk -F'\t' "$CANON" | sha256sum | cut -d' ' -f1) - printf '%s\t%s\n' "$f" "$h" >> "$OUT" -done - -echo "wrote $(grep -cv '^#' "$OUT") hashes to $OUT" diff --git a/go-parser/testdata/reader-fidelity-hashes.tsv b/go-parser/testdata/reader-fidelity-hashes.tsv index 72d1d5c..8c9ce2f 100644 --- a/go-parser/testdata/reader-fidelity-hashes.tsv +++ b/go-parser/testdata/reader-fidelity-hashes.tsv @@ -2,7 +2,12 @@ # ground truth (parser/examples/dump_cells.rs). The Go reader must reproduce # each hash exactly. Hashes only - the dumps themselves are real student PII # and are never committed, per parser/tests/fixtures/README.md. -# Regenerate: go-parser/scripts/regen-fidelity-hashes.sh +# FROZEN. These hashes were produced by the Rust/calamine parser, which has +# since been removed, so they cannot be regenerated. They remain a real +# regression guard: any change to the Go reader that alters a single cell of any +# of the 299 input files fails the suite. If the inputs themselves ever change, +# the affected rows must be re-derived deliberately and reviewed, not refreshed +# in bulk. data/2016/023718c7d3cf7ace3a7116fabb12bd9cdhyduoccantho-1468920829104.xlsx fc3afd8da9ed28b0fc177192fab7b19a06535cb45d86841835d8e8828fc61c1c data/2016/08cdeb3636dcb2adaef829d62968274atravinh-1468902443859.xlsx 71f0e61918c63b95f08a16d502f5e9f2054e3c830bd83c725f1285bb1aabead1 data/2016/0b583c9875443e65ffa449d5ca76fae8ninhbinh-1468899778557.xlsx b0c2cfb9ab419442f151be5a014439f7f477fa7b31a9ae3f37b83073276bdb59 diff --git a/parser/Cargo.lock b/parser/Cargo.lock deleted file mode 100644 index bcd7270..0000000 --- a/parser/Cargo.lock +++ /dev/null @@ -1,1131 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "adler2" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" - -[[package]] -name = "aes" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" -dependencies = [ - "cfg-if", - "cipher", - "cpufeatures", -] - -[[package]] -name = "ahash" -version = "0.8.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" -dependencies = [ - "cfg-if", - "once_cell", - "version_check", - "zerocopy", -] - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys", -] - -[[package]] -name = "anyhow" -version = "1.0.102" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" - -[[package]] -name = "arbitrary" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" -dependencies = [ - "derive_arbitrary", -] - -[[package]] -name = "bitflags" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" - -[[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - -[[package]] -name = "bumpalo" -version = "3.20.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" - -[[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" - -[[package]] -name = "bzip2" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47" -dependencies = [ - "bzip2-sys", -] - -[[package]] -name = "bzip2-sys" -version = "0.1.13+1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" -dependencies = [ - "cc", - "pkg-config", -] - -[[package]] -name = "calamine" -version = "0.26.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "138646b9af2c5d7f1804ea4bf93afc597737d2bd4f7341d67c48b03316976eb1" -dependencies = [ - "byteorder", - "codepage", - "encoding_rs", - "log", - "quick-xml", - "serde", - "zip", -] - -[[package]] -name = "cc" -version = "1.2.62" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1dce859f0832a7d088c4f1119888ab94ef4b5d6795d1ce05afb7fe159d79f98" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cipher" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" -dependencies = [ - "crypto-common", - "inout", -] - -[[package]] -name = "clap" -version = "4.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "codepage" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48f68d061bc2828ae826206326e61251aca94c1e4a5305cf52d9138639c918b4" -dependencies = [ - "encoding_rs", -] - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "constant_time_eq" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6" - -[[package]] -name = "cpufeatures" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" -dependencies = [ - "libc", -] - -[[package]] -name = "crc" -version = "3.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" -dependencies = [ - "crc-catalog", -] - -[[package]] -name = "crc-catalog" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" - -[[package]] -name = "crc32fast" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" - -[[package]] -name = "crypto-common" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" -dependencies = [ - "generic-array", - "typenum", -] - -[[package]] -name = "deflate64" -version = "0.1.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" - -[[package]] -name = "deranged" -version = "0.5.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" -dependencies = [ - "powerfmt", -] - -[[package]] -name = "derive_arbitrary" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer", - "crypto-common", - "subtle", -] - -[[package]] -name = "displaydoc" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "encoding_rs" -version = "0.8.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "fallible-iterator" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649" - -[[package]] -name = "fallible-streaming-iterator" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "flate2" -version = "1.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" -dependencies = [ - "crc32fast", - "miniz_oxide", -] - -[[package]] -name = "generic-array" -version = "0.14.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" -dependencies = [ - "typenum", - "version_check", -] - -[[package]] -name = "getrandom" -version = "0.3.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" -dependencies = [ - "cfg-if", - "js-sys", - "libc", - "r-efi", - "wasip2", - "wasm-bindgen", -] - -[[package]] -name = "glob" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" - -[[package]] -name = "hashbrown" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" -dependencies = [ - "ahash", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "hashlink" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af" -dependencies = [ - "hashbrown 0.14.5", -] - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "hmac" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" -dependencies = [ - "digest", -] - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown 0.17.1", -] - -[[package]] -name = "inout" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" -dependencies = [ - "generic-array", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jobserver" -version = "0.1.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.98" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67df7112613f8bfd9150013a0314e196f4800d3201ae742489d999db2f979f08" -dependencies = [ - "cfg-if", - "once_cell", - "wasm-bindgen", -] - -[[package]] -name = "libc" -version = "0.2.186" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" - -[[package]] -name = "libsqlite3-sys" -version = "0.30.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e99fb7a497b1e3339bc746195567ed8d3e24945ecd636e3619d20b9de9e9149" -dependencies = [ - "cc", - "pkg-config", - "vcpkg", -] - -[[package]] -name = "log" -version = "0.4.29" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" - -[[package]] -name = "lzma-rs" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "297e814c836ae64db86b36cf2a557ba54368d03f6afcd7d947c266692f71115e" -dependencies = [ - "byteorder", - "crc", -] - -[[package]] -name = "lzma-sys" -version = "0.1.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27" -dependencies = [ - "cc", - "libc", - "pkg-config", -] - -[[package]] -name = "memchr" -version = "2.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" - -[[package]] -name = "miniz_oxide" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" -dependencies = [ - "adler2", - "simd-adler32", -] - -[[package]] -name = "num-conv" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "pbkdf2" -version = "0.12.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" -dependencies = [ - "digest", - "hmac", -] - -[[package]] -name = "pkg-config" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" - -[[package]] -name = "powerfmt" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" - -[[package]] -name = "proc-macro2" -version = "1.0.106" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quick-xml" -version = "0.31.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1004a344b30a54e2ee58d66a71b32d2db2feb0a31f9a2d302bf0536f15de2a33" -dependencies = [ - "encoding_rs", - "memchr", -] - -[[package]] -name = "quote" -version = "1.0.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "5.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" - -[[package]] -name = "regex" -version = "1.12.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" - -[[package]] -name = "rusqlite" -version = "0.32.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7753b721174eb8ff87a9a0e799e2d7bc3749323e773db92e0984debb00019d6e" -dependencies = [ - "bitflags", - "fallible-iterator", - "fallible-streaming-iterator", - "hashlink", - "libsqlite3-sys", - "smallvec", -] - -[[package]] -name = "rustversion" -version = "1.0.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" - -[[package]] -name = "ryu" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" - -[[package]] -name = "serde" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "serde_yaml" -version = "0.9.34+deprecated" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" -dependencies = [ - "indexmap", - "itoa", - "ryu", - "serde", - "unsafe-libyaml", -] - -[[package]] -name = "sha1" -version = "0.10.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" -dependencies = [ - "cfg-if", - "cpufeatures", - "digest", -] - -[[package]] -name = "shlex" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" - -[[package]] -name = "simd-adler32" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" - -[[package]] -name = "smallvec" -version = "1.15.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "subtle" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" - -[[package]] -name = "syn" -version = "2.0.117" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "thiserror" -version = "1.0.69" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" -dependencies = [ - "thiserror-impl 1.0.69", -] - -[[package]] -name = "thiserror" -version = "2.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" -dependencies = [ - "thiserror-impl 2.0.18", -] - -[[package]] -name = "thiserror-impl" -version = "1.0.69" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "time" -version = "0.3.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" -dependencies = [ - "deranged", - "num-conv", - "powerfmt", - "serde_core", - "time-core", -] - -[[package]] -name = "time-core" -version = "0.1.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" - -[[package]] -name = "tinyvec" -version = "1.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" -dependencies = [ - "tinyvec_macros", -] - -[[package]] -name = "tinyvec_macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" - -[[package]] -name = "typenum" -version = "1.20.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40ce102ab67701b8526c123c1bab5cbe42d7040ccfd0f64af1a385808d2f43de" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "unicode-normalization" -version = "0.1.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" -dependencies = [ - "tinyvec", -] - -[[package]] -name = "unsafe-libyaml" -version = "0.2.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "vcpkg" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" - -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - -[[package]] -name = "wasip2" -version = "1.0.3+wasi-0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" -dependencies = [ - "wit-bindgen", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.121" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49ace1d07c165b0864824eee619580c4689389afa9dc9ed3a4c75040d82e6790" -dependencies = [ - "cfg-if", - "once_cell", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.121" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e68e6f4afd367a562002c05637acb8578ff2dea1943df76afb9e83d177c8578" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.121" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d95a9ec35c64b2a7cb35d3fead40c4238d0940c86d107136999567a4703259f2" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.121" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4e0100b01e9f0d03189a92b96772a1fb998639d981193d7dbab487302513441" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "wit-bindgen" -version = "0.57.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" - -[[package]] -name = "xlsxread" -version = "0.1.0" -dependencies = [ - "anyhow", - "calamine", - "clap", - "glob", - "regex", - "rusqlite", - "serde", - "serde_yaml", - "thiserror 1.0.69", - "unicode-normalization", - "zip", -] - -[[package]] -name = "xz2" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2" -dependencies = [ - "lzma-sys", -] - -[[package]] -name = "zerocopy" -version = "0.8.48" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.48" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "zeroize" -version = "1.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" -dependencies = [ - "zeroize_derive", -] - -[[package]] -name = "zeroize_derive" -version = "1.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85a5b4158499876c763cb03bc4e49185d3cccbabb15b33c627f7884f43db852e" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "zip" -version = "2.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fabe6324e908f85a1c52063ce7aa26b68dcb7eb6dbc83a2d148403c9bc3eba50" -dependencies = [ - "aes", - "arbitrary", - "bzip2", - "constant_time_eq", - "crc32fast", - "crossbeam-utils", - "deflate64", - "displaydoc", - "flate2", - "getrandom", - "hmac", - "indexmap", - "lzma-rs", - "memchr", - "pbkdf2", - "sha1", - "thiserror 2.0.18", - "time", - "xz2", - "zeroize", - "zopfli", - "zstd", -] - -[[package]] -name = "zopfli" -version = "0.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" -dependencies = [ - "bumpalo", - "crc32fast", - "log", - "simd-adler32", -] - -[[package]] -name = "zstd" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" -dependencies = [ - "zstd-safe", -] - -[[package]] -name = "zstd-safe" -version = "7.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" -dependencies = [ - "zstd-sys", -] - -[[package]] -name = "zstd-sys" -version = "2.0.16+zstd.1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" -dependencies = [ - "cc", - "pkg-config", -] diff --git a/parser/Cargo.toml b/parser/Cargo.toml deleted file mode 100644 index ec0ef94..0000000 --- a/parser/Cargo.toml +++ /dev/null @@ -1,30 +0,0 @@ -[package] -name = "xlsxread" -version = "0.1.0" -edition = "2021" -description = "Rust CLI replacing SheetJS xlsx build scripts for thptqg2017/thptqg2016" - -[dependencies] -calamine = "0.26" -rusqlite = { version = "0.32", features = ["bundled"] } -clap = { version = "4", features = ["derive"] } -serde = { version = "1", features = ["derive"] } -serde_yaml = "0.9" # deprecated upstream but stable; this crate is deleted at cutover -regex = "1" -unicode-normalization = "0.1" -thiserror = "1" -anyhow = "1" -glob = "0.3" - -# zip is already a transitive dep of calamine; pin explicitly so tests can use it -[dev-dependencies] -zip = "2" -rusqlite = { version = "0.32", features = ["bundled"] } - -[[bin]] -name = "xlsxread" -path = "src/main.rs" - -[[test]] -name = "golden" -path = "tests/golden.rs" diff --git a/parser/examples/dump_cells.rs b/parser/examples/dump_cells.rs deleted file mode 100644 index b6d15ca..0000000 --- a/parser/examples/dump_cells.rs +++ /dev/null @@ -1,106 +0,0 @@ -//! Ground-truth cell dumper for the Go reader fidelity gate. -//! -//! Emits a canonical, reader-agnostic text rendering of every sheet and every -//! cell of a spreadsheet exactly as calamine sees it. The Go reader must -//! reproduce this byte-for-byte; the two dumps are compared by hash. -//! -//! Deliberately dumps the RAW used range: every sheet, every row, header rows -//! included, no config applied. This gate is about cell fidelity, not build -//! semantics — sheet selection and header skipping are exercised later. -//! -//! Usage: cargo run --release --example dump_cells -- [out-file] -//! With no out-file the dump goes to stdout. - -use std::env; -use std::fs::File; -use std::io::{self, BufWriter, Write}; - -use calamine::{open_workbook_auto, Data, Reader, Sheets}; - -/// Escapes the field separators so a cell value can never break the line format. -fn escape(s: &str) -> String { - let mut out = String::with_capacity(s.len()); - for ch in s.chars() { - match ch { - '\\' => out.push_str("\\\\"), - '\t' => out.push_str("\\t"), - '\n' => out.push_str("\\n"), - '\r' => out.push_str("\\r"), - _ => out.push(ch), - } - } - out -} - -/// Discriminates the calamine variant so a Go port can be checked against the -/// actual type, not just the rendered string. -fn kind(d: &Data) -> &'static str { - match d { - Data::Empty => "empty", - Data::String(_) => "str", - Data::Float(_) => "float", - Data::Int(_) => "int", - Data::Bool(_) => "bool", - Data::Error(_) => "err", - Data::DateTime(_) => "datetime", - Data::DateTimeIso(_) => "datetimeiso", - Data::DurationIso(_) => "durationiso", - } -} - -fn main() -> Result<(), Box> { - let args: Vec = env::args().collect(); - if args.len() < 2 { - eprintln!("usage: dump_cells [out-file]"); - std::process::exit(2); - } - let path = &args[1]; - - let mut out: Box = match args.get(2) { - Some(p) => Box::new(BufWriter::new(File::create(p)?)), - None => Box::new(BufWriter::new(io::stdout())), - }; - - let mut workbook: Sheets<_> = open_workbook_auto(path)?; - let sheet_names: Vec = workbook.sheet_names().to_vec(); - - writeln!(out, "FILE\t{}", escape(path))?; - writeln!(out, "SHEETCOUNT\t{}", sheet_names.len())?; - - for (idx, name) in sheet_names.iter().enumerate() { - let range = workbook.worksheet_range(name)?; - // start() is the used-range origin — the key question for any Go reader, - // which may index absolutely from A1 instead. - let (srow, scol) = range.start().unwrap_or((0, 0)); - writeln!( - out, - "SHEET\t{}\t{}\t{}\t{}\t{}\t{}", - idx, - escape(name), - range.height(), - range.width(), - srow, - scol - )?; - - for (r, row) in range.rows().enumerate() { - writeln!(out, "ROW\t{}\t{}\t{}", idx, r, row.len())?; - for (c, cell) in row.iter().enumerate() { - let s = cell.to_string(); - writeln!( - out, - "CELL\t{}\t{}\t{}\t{}\t{}\t{}", - idx, - r, - c, - kind(cell), - if matches!(cell, Data::Empty) { 1 } else { 0 }, - escape(&s) - )?; - } - } - } - - out.flush()?; - Ok(()) -} diff --git a/parser/examples/scan_kinds.rs b/parser/examples/scan_kinds.rs deleted file mode 100644 index fc06a05..0000000 --- a/parser/examples/scan_kinds.rs +++ /dev/null @@ -1,76 +0,0 @@ -//! Corpus-wide cell-kind and sheet-geometry scan, for the Go reader fidelity gate. -//! -//! For every file given, prints one TSV line per sheet and one summary line per -//! file. Aggregates only — never materialises the cell text — so the whole -//! 418 MB corpus can be scanned quickly. -//! -//! The point is to find out which calamine `Data` variants actually occur in -//! real inputs. `DateTime` and `Float` are the variants whose rendering differs -//! between readers; if they never appear, the divergence risk is theoretical. -//! -//! Usage: cargo run --release --example scan_kinds -- ... - -use std::env; - -use calamine::{open_workbook_auto, Data, Reader, Sheets}; - -fn main() { - let args: Vec = env::args().skip(1).collect(); - if args.is_empty() { - eprintln!("usage: scan_kinds ..."); - std::process::exit(2); - } - - println!("#TYPE\tpath\tsheet_idx\tname\theight\twidth\tstart_row\tstart_col\tempty\tstr\tfloat\tint\tbool\tdatetime\tdtiso\tduriso\terr"); - - for path in &args { - let mut wb: Sheets<_> = match open_workbook_auto(path) { - Ok(w) => w, - Err(e) => { - println!("ERR\t{path}\t{e}"); - continue; - } - }; - - let names: Vec = wb.sheet_names().to_vec(); - for (idx, name) in names.iter().enumerate() { - let range = match wb.worksheet_range(name) { - Ok(r) => r, - Err(e) => { - println!("SHEETERR\t{path}\t{idx}\t{e}"); - continue; - } - }; - let (sr, sc) = range.start().unwrap_or((0, 0)); - let mut k = [0usize; 9]; // empty,str,float,int,bool,datetime,dtiso,duriso,err - for row in range.rows() { - for cell in row { - let i = match cell { - Data::Empty => 0, - Data::String(_) => 1, - Data::Float(_) => 2, - Data::Int(_) => 3, - Data::Bool(_) => 4, - Data::DateTime(_) => 5, - Data::DateTimeIso(_) => 6, - Data::DurationIso(_) => 7, - Data::Error(_) => 8, - }; - k[i] += 1; - } - } - println!( - "SHEET\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}", - path, - idx, - name.replace('\t', " "), - range.height(), - range.width(), - sr, - sc, - k[0], k[1], k[2], k[3], k[4], k[5], k[6], k[7], k[8] - ); - } - println!("FILE\t{}\t{}", path, names.len()); - } -} diff --git a/parser/scripts/build-db.js b/parser/scripts/build-db.js deleted file mode 100644 index 5bdd991..0000000 --- a/parser/scripts/build-db.js +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env node -/** - * Build the SQLite database for one or all datasets, then gzip it. - * - * Replaces the six per-dataset npm scripts the two old projects carried. The - * dataset list comes from src/datasets.js so it is written in exactly one place. - * - * Output goes to .build/public/db/ — the directory Vite copies as its publicDir. - * Only the .gz survives: shipping a 100+ MB uncompressed database is made - * structurally impossible rather than left to a cleanup step. - * - * Usage: - * node parser/scripts/build-db.js # all four datasets - * node parser/scripts/build-db.js 2017-old # just one - */ - -import { execFileSync } from "node:child_process"; -import { mkdirSync, rmSync, existsSync } from "node:fs"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -import { DATASET_IDS } from "../../src/datasets.js"; - -const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../.."); -const BIN = resolve(ROOT, "parser/target/release/xlsxread"); -const OUT_DIR = resolve(ROOT, ".build/public/db"); - -const requested = process.argv.slice(2); -const unknown = requested.filter((id) => !DATASET_IDS.includes(id)); -if (unknown.length) { - console.error(`unknown dataset(s): ${unknown.join(", ")}`); - console.error(`known: ${DATASET_IDS.join(", ")}`); - process.exit(2); -} -const targets = requested.length ? requested : DATASET_IDS; - -if (!existsSync(BIN)) { - console.error(`parser binary not found at ${BIN}`); - console.error("run: npm run build:rust"); - process.exit(1); -} - -mkdirSync(OUT_DIR, { recursive: true }); - -for (const id of targets) { - const db = resolve(OUT_DIR, `${id}.db`); - - execFileSync( - BIN, - [ - "build", - "--schema", - resolve(ROOT, `parser/configs/${id}.yml`), - "--input", - resolve(ROOT, `data/${id}`), - "--output", - db, - ], - { stdio: "inherit" }, - ); - - // -9 without -k: the raw .db must not reach the published artifact. - rmSync(`${db}.gz`, { force: true }); - execFileSync("gzip", ["-9", db], { stdio: "inherit" }); - console.log(` → db/${id}.db.gz\n`); -} diff --git a/parser/scripts/check-duplicates.js b/parser/scripts/check-duplicates.js deleted file mode 100644 index c0b4a53..0000000 --- a/parser/scripts/check-duplicates.js +++ /dev/null @@ -1,30 +0,0 @@ -// One-off audit: detect content-identical Excel files via md5. -// -// BROKEN as committed — `dirs` below is a hardcoded Windows path from the -// original author's machine. Predates the repo unification; left unchanged -// rather than half-fixed. Point `dirs` at data/ to use it. -import crypto from "crypto"; -import fs from "fs"; -import path from "path"; - -const dirs = ["D:/tiennm99/thptqg2017/data"]; - -const byHash = {}; -for (const d of dirs) { - for (const f of fs.readdirSync(d)) { - const full = path.join(d, f); - if (!fs.statSync(full).isFile() || !f.endsWith(".xlsx")) continue; - const h = crypto.createHash("md5").update(fs.readFileSync(full)).digest("hex"); - (byHash[h] ||= []).push(full); - } -} - -const total = Object.values(byHash).reduce((s, a) => s + a.length, 0); -const dupes = Object.entries(byHash).filter(([, a]) => a.length > 1); -console.log(`Total files: ${total}`); -console.log(`Unique by md5: ${Object.keys(byHash).length}`); -console.log(`Duplicate groups: ${dupes.length}`); -for (const [h, a] of dupes) { - console.log(` ${h.slice(0, 12)}:`); - a.forEach((p) => console.log(` ${p}`)); -} diff --git a/parser/scripts/db-stats.js b/parser/scripts/db-stats.js deleted file mode 100644 index 8582821..0000000 --- a/parser/scripts/db-stats.js +++ /dev/null @@ -1,79 +0,0 @@ -#!/usr/bin/env node -/** - * Dump per-dataset statistics from built SQLite databases as JSON. - * - * Used twice: once against the pre-refactor databases to capture a baseline, - * and again after the schema unification. Comparing the two outputs is what - * proves no score data was lost or invented. - * - * Schema-agnostic on purpose — columns come from PRAGMA table_info, so the same - * script runs against the old 18/20-column tables and the new 22-column one. - * - * Usage: - * node db-stats.js