feat: refresh data from baotintuc.vn source, fix overflow sheet loss

Browse Source

Dataset update:
- Crawl all 63 .xls province files from baotintuc.vn CDN (original source)
- Old xlsx dataset moved to data-old/ for reference
- Net: +13,719 students (Hà Nội +7,275, HCM +6,445) — the old .xls → xlsx
  conversion silently dropped rows beyond the 65,536 per-sheet cap
- Also removes 1 bogus header row that had leaked into the old DB
- 100% identical scores on the 847,348 SBDs present in both datasets

Build pipeline:
- build-database.js: iterate ALL sheets per workbook (fixes the overflow
  loss) and accept .xls in addition to .xlsx

Audit tooling:
- scripts/crawl-baotintuc.js: idempotent 63-province downloader
- scripts/diff-datasets.js: compares two DBs by SBD set and per-column
  score deltas

...

This commit is contained in:

tiennm99

2026-04-14 21:42:29 +07:00

parent d8dc1e0955

commit 8f86e4dc3b

129 changed files with 274 additions and 17 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/10_BinhThuan_RIIW.xls.xlsx → data-old/10_BinhThuan_RIIW.xls.xlsx

data/10_Ca_Mau_BKXT.xls.xlsx → data-old/10_Ca_Mau_BKXT.xls.xlsx

data/10_LamDong_GNFT.xls.xlsx → data-old/10_LamDong_GNFT.xls.xlsx

data/10_Soc_Trang_XCGJ.xls.xlsx → data-old/10_Soc_Trang_XCGJ.xls.xlsx

data/11_BinhDuong_RYQL.xls.xlsx → data-old/11_BinhDuong_RYQL.xls.xlsx

data/11_LaoCai_GMSU.xls.xlsx → data-old/11_LaoCai_GMSU.xls.xlsx

data/12_BenTre_DKWF.xls.xlsx → data-old/12_BenTre_DKWF.xls.xlsx

data/12_LongAn_ZZUK.xls.xlsx → data-old/12_LongAn_ZZUK.xls.xlsx

data/13_NamDinh_ESEL.xls.xlsx → data-old/13_NamDinh_ESEL.xls.xlsx

data/13_TraVinh_LKUJ.xls.xlsx → data-old/13_TraVinh_LKUJ.xls.xlsx

data/14_NgheAn_BSLY.xls.xlsx → data-old/14_NgheAn_BSLY.xls.xlsx

data/15_PhuTho_ABWQ.xls.xlsx → data-old/15_PhuTho_ABWQ.xls.xlsx

data/16_QuangBinh_KGEU.xls.xlsx → data-old/16_QuangBinh_KGEU.xls.xlsx

data/17_QuangNam_AMTK.xls.xlsx → data-old/17_QuangNam_AMTK.xls.xlsx

data/18_QuangNgai_KOFP.xls.xlsx → data-old/18_QuangNgai_KOFP.xls.xlsx

data/19_QuangTri_OMZF.xls.xlsx → data-old/19_QuangTri_OMZF.xls.xlsx

data/1_BaRia_VungTau_HJKG.xls.xlsx → data-old/1_BaRia_VungTau_HJKG.xls.xlsx

data/1_Da_Nang_AHWJ.xls.xlsx → data-old/1_Da_Nang_AHWJ.xls.xlsx

data/1_Ha_Noi_CVXG.xls.xlsx → data-old/1_Ha_Noi_CVXG.xls.xlsx

data/1_Son_La_JIDP.xls.xlsx → data-old/1_Son_La_JIDP.xls.xlsx

data/1_TuyenQuang_JBYF.xls.xlsx → data-old/1_TuyenQuang_JBYF.xls.xlsx

data/20_TayNinh_ILFA.xls.xlsx → data-old/20_TayNinh_ILFA.xls.xlsx

data/21_ThaiBinh_FTVG.xls.xlsx → data-old/21_ThaiBinh_FTVG.xls.xlsx

data/22_ThaiNguyen_TLTW.xls.xlsx → data-old/22_ThaiNguyen_TLTW.xls.xlsx

data/23_HaiPhong_HXBV.xls.xlsx → data-old/23_HaiPhong_HXBV.xls.xlsx

data/24_HCM_XULN.xls.xlsx → data-old/24_HCM_XULN.xls.xlsx

data/2_BacKan_GFVQ.xls.xlsx → data-old/2_BacKan_GFVQ.xls.xlsx

data/2_Ha_Giang_QNCM.xls.xlsx → data-old/2_Ha_Giang_QNCM.xls.xlsx

data/2_Ninh_Thuan_VHLY.xls.xlsx → data-old/2_Ninh_Thuan_VHLY.xls.xlsx

data/2_Thanh_Hoa_AUFV.xls.xlsx → data-old/2_Thanh_Hoa_AUFV.xls.xlsx

data/2_VinhPhuc_GUDK.xls.xlsx → data-old/2_VinhPhuc_GUDK.xls.xlsx

data/3_BacGiang_TOIF.xls.xlsx → data-old/3_BacGiang_TOIF.xls.xlsx

data/3_BinhPhuoc_YFMU.xls.xlsx → data-old/3_BinhPhuoc_YFMU.xls.xlsx

data/3_Cao_Bang_CIEY.xls.xlsx → data-old/3_Cao_Bang_CIEY.xls.xlsx

data/3_Dong_Thap_GSXQ.xls.xlsx → data-old/3_Dong_Thap_GSXQ.xls.xlsx

data/3_Thua_Thien_Hue_HMDB.xls.xlsx → data-old/3_Thua_Thien_Hue_HMDB.xls.xlsx

data/4_An_Giang_JNOS.xls.xlsx → data-old/4_An_Giang_JNOS.xls.xlsx

data/4_BacNinh_STLR.xls.xlsx → data-old/4_BacNinh_STLR.xls.xlsx

data/4_Binh_Dinh_WWZW.xls.xlsx → data-old/4_Binh_Dinh_WWZW.xls.xlsx

data/4_DienBien_SOJG.xls.xlsx → data-old/4_DienBien_SOJG.xls.xlsx

data/4_Lang_Son_NYQL.xls.xlsx → data-old/4_Lang_Son_NYQL.xls.xlsx

data/5_Bac_Lieu_XEKH.xls.xlsx → data-old/5_Bac_Lieu_XEKH.xls.xlsx

data/5_Gia_Lai_ABZI.xls.xlsx → data-old/5_Gia_Lai_ABZI.xls.xlsx

data/5_HaiDuong_WWWG.xls.xlsx → data-old/5_HaiDuong_WWWG.xls.xlsx

data/5_Hanam_QGJS.xls.xlsx → data-old/5_Hanam_QGJS.xls.xlsx

data/5_Yen_Bai_FAQR.xls.xlsx → data-old/5_Yen_Bai_FAQR.xls.xlsx

data/6_Dong_Nai_WOTM.xls.xlsx → data-old/6_Dong_Nai_WOTM.xls.xlsx

data/6_Hau_Giang_KWDM.xls.xlsx → data-old/6_Hau_Giang_KWDM.xls.xlsx

data/6_HoaBinh_TPZY.xls.xlsx → data-old/6_HoaBinh_TPZY.xls.xlsx

data/6_NinhBinh_IGFT.xls.xlsx → data-old/6_NinhBinh_IGFT.xls.xlsx

data/6_Quang_Ninh_DQCJ.xls.xlsx → data-old/6_Quang_Ninh_DQCJ.xls.xlsx

data/7_HaTinh_DDHD.xls.xlsx → data-old/7_HaTinh_DDHD.xls.xlsx

data/7_HungYen_LTIK.xls.xlsx → data-old/7_HungYen_LTIK.xls.xlsx

data/7_Kon_Tum_RSLR.xls.xlsx → data-old/7_Kon_Tum_RSLR.xls.xlsx

data/7_Tien_Giang_EFHX.xls.xlsx → data-old/7_Tien_Giang_EFHX.xls.xlsx

data/8_Can_Tho_RQZM.xls.xlsx → data-old/8_Can_Tho_RQZM.xls.xlsx

data/8_Dak_Lak_YKPR.xls.xlsx → data-old/8_Dak_Lak_YKPR.xls.xlsx

data/8_KienGiang_OTOB.xls.xlsx → data-old/8_KienGiang_OTOB.xls.xlsx

data/8_PhuYen_OGIM.xls.xlsx → data-old/8_PhuYen_OGIM.xls.xlsx

data/9_DakNong_FBOP.xls.xlsx → data-old/9_DakNong_FBOP.xls.xlsx

data/9_Khanh_Hoa_KPKQ.xls.xlsx → data-old/9_Khanh_Hoa_KPKQ.xls.xlsx

data/9_LaiChau_ALKN.xls.xlsx → data-old/9_LaiChau_ALKN.xls.xlsx

data/9_Vinh_Long_OLWA.xls.xlsx → data-old/9_Vinh_Long_OLWA.xls.xlsx

data/an-giang.xls

BIN

Binary file not shown.

data/ba-ria-vung-tau.xls

BIN

Binary file not shown.

data/bac-giang.xls

BIN

Binary file not shown.

data/bac-kan.xls

BIN

Binary file not shown.

data/bac-lieu.xls

BIN

Binary file not shown.

data/bac-ninh.xls

BIN

Binary file not shown.

data/ben-tre.xls

BIN

Binary file not shown.

data/binh-dinh.xls

BIN

Binary file not shown.

data/binh-duong.xls

BIN

Binary file not shown.

data/binh-phuoc.xls

BIN

Binary file not shown.

data/binh-thuan.xls

BIN

Binary file not shown.

data/ca-mau.xls

BIN

Binary file not shown.

data/can-tho.xls

BIN

Binary file not shown.

data/cao-bang.xls

BIN

Binary file not shown.

data/da-nang.xls

BIN

Binary file not shown.

data/dak-lak.xls

BIN

Binary file not shown.

data/dak-nong.xls

BIN

Binary file not shown.

data/dien-bien.xls

BIN

Binary file not shown.

data/dong-nai.xls

BIN

Binary file not shown.

data/dong-thap.xls

BIN

Binary file not shown.

data/gia-lai.xls

BIN

Binary file not shown.

data/ha-giang.xls

BIN

Binary file not shown.

data/ha-nam.xls

BIN

Binary file not shown.

data/ha-noi.xls

BIN

Binary file not shown.

data/ha-tinh.xls

BIN

Binary file not shown.

data/hai-duong.xls

BIN

Binary file not shown.

data/hai-phong.xls

BIN

Binary file not shown.

data/hau-giang.xls

BIN

Binary file not shown.

data/ho-chi-minh.xls

BIN

Binary file not shown.

data/hoa-binh.xls

BIN

Binary file not shown.

data/hung-yen.xls

BIN

Binary file not shown.

data/khanh-hoa.xls

BIN

Binary file not shown.

data/kien-giang.xls

BIN

Binary file not shown.

data/kon-tum.xls

BIN

Binary file not shown.

data/lai-chau.xls

BIN

Binary file not shown.

data/lam-dong.xls

BIN

Binary file not shown.

data/lang-son.xls

BIN

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More