mirror of
https://github.com/tiennm99/thptqg2017.git
synced 2026-09-18 20:19:47 +00:00
tiennm99
6ff2ed99ec
The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.
New layout:
index.html + src/ the 2017 frontend, now the only one
data/<id>/ 2016, 2017, 2017-old, 2017-old2
parser/ the single Rust crate, configs renamed to <id>.toml
docs/ both projects' docs, 2016 copies suffixed -2016-legacy
pending the merge pass
<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.
pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.
Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.
Follow-on fixes the move made necessary:
- eslint's Node-globals override pointed at scripts/, now parser/scripts/
- crawl-baotintuc.js wrote to <root>/data, now data/2017
- golden tests loaded configs by their old thptqg*-data.toml names
Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.
63 Rust tests pass and clippy is clean from the new location.
31 lines
1.1 KiB
JavaScript
31 lines
1.1 KiB
JavaScript
// One-off audit: detect content-identical Excel files via md5.
|
|
//
|
|
// BROKEN as committed — `dirs` below is a hardcoded Windows path from the
|
|
// original author's machine. Predates the repo unification; left unchanged
|
|
// rather than half-fixed. Point `dirs` at data/<dataset> to use it.
|
|
import crypto from "crypto";
|
|
import fs from "fs";
|
|
import path from "path";
|
|
|
|
const dirs = ["D:/tiennm99/thptqg2017/data"];
|
|
|
|
const byHash = {};
|
|
for (const d of dirs) {
|
|
for (const f of fs.readdirSync(d)) {
|
|
const full = path.join(d, f);
|
|
if (!fs.statSync(full).isFile() || !f.endsWith(".xlsx")) continue;
|
|
const h = crypto.createHash("md5").update(fs.readFileSync(full)).digest("hex");
|
|
(byHash[h] ||= []).push(full);
|
|
}
|
|
}
|
|
|
|
const total = Object.values(byHash).reduce((s, a) => s + a.length, 0);
|
|
const dupes = Object.entries(byHash).filter(([, a]) => a.length > 1);
|
|
console.log(`Total files: ${total}`);
|
|
console.log(`Unique by md5: ${Object.keys(byHash).length}`);
|
|
console.log(`Duplicate groups: ${dupes.length}`);
|
|
for (const [h, a] of dupes) {
|
|
console.log(` ${h.slice(0, 12)}:`);
|
|
a.forEach((p) => console.log(` ${p}`));
|
|
}
|