mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-05 08:14:55 +00:00
tiennm99
933adf70c9
Comments across the tree justified the code by pointing at a Rust implementation that is no longer in the repository, citing files and line numbers (config.rs:132, schema.rs:26-54, reader.rs:42) that cannot be opened, plus crates and datasets that are equally gone. A reader could not check any of it. Every invariant those comments carried is kept and restated so it stands on its own: the bytewise sort that decides which row survives a duplicate exam number, the literal U+0300..U+036F range that must match the site's toAscii, the trailing space in "SINH ", the BIFF and shared-string corrections, the VACUUM-after-COMMIT rule, the deploy-from-main guard. The reader's contract is now anchored to the frozen oracle in parser/testdata, which still exists and is still checked, rather than to the tool that originally produced it. TestDDLMatchesRust becomes TestDDLIsFrozen: it compares against a copy of the DDL inside the test and never read schema.rs, so both the name and the failure message were misleading. ToAscii no longer claims the d-replacement must precede lowercasing. Both cases map to 'd' and ToLower runs last, so the order has no effect.
39 lines
1.4 KiB
Go
39 lines
1.4 KiB
Go
package sources
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"github.com/tiennm99/thptqg/crawler/internal/article"
|
|
)
|
|
|
|
// source2016 fetches the 2016 dataset: one spreadsheet per exam cluster
|
|
// (cụm thi), 4 .xls and 115 .xlsx.
|
|
//
|
|
// The article is a school site that aggregated every cluster's file.
|
|
//
|
|
// Dest keeps the server's own filename verbatim — a 32-hex content hash, the
|
|
// cluster slug, then a millisecond timestamp. Two reasons not to prettify it:
|
|
//
|
|
// - parser sorts inputs bytewise and inserts last-wins, so filenames decide
|
|
// which row survives a duplicate exam number. That is live here, not
|
|
// hypothetical: 877,464 source rows collapse to 877,461, so three rows'
|
|
// contents depend on this ordering.
|
|
// - parser/testdata/reader-fidelity-hashes.tsv is keyed by full path and is
|
|
// frozen, so a renamed file no longer matches its oracle entry.
|
|
//
|
|
// Unlike 2017 this needs no transliteration: the name comes from the URL, so it
|
|
// is exact by construction rather than derived from a label.
|
|
var source2016 = Source{
|
|
ID: "2016",
|
|
Summary: "119 exam-cluster files (4 .xls + 115 .xlsx)",
|
|
Article: "https://dtnt.bacninh.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html",
|
|
Exts: []string{".xls", ".xlsx"},
|
|
WantFiles: 119,
|
|
Dest: func(l article.Link) (string, error) {
|
|
if l.File == "" {
|
|
return "", fmt.Errorf("link %s has no filename", l.URL)
|
|
}
|
|
return l.File, nil
|
|
},
|
|
}
|