mirror of
https://github.com/tiennm99/hako-crawler.git
synced 2026-09-02 04:20:39 +00:00
tiennm99
0b6cfdaf74
The crawler took one hardcoded novel URL, wrote per-chapter text files into ./data, and fetched serially until the site answered 429. Rename the module to hako-crawler and split fetching, page parsing and crawl orchestration into a hako package that takes any novel URL. The chapter body is no longer served as markup: #chapter-content now carries a shuffled, XOR-obfuscated payload that the site's own script expands in the browser, so the previous p[id=<digits>] extractor returned nothing at all. Decode it, keeping the plain-markup path as a fallback. The scheme is read from the page and an unrecognised one is a hard error, because the alternative is exporting a book of mojibake. Space requests globally instead of per-goroutine, so concurrency no longer raises the request rate, and retry only what a retry can fix. Cache raw pages so re-runs cost no requests. BREAKING CHANGE: module path is now github.com/tiennm99/hako-crawler, and the novel URL is a required argument rather than a constant.
15 lines
265 B
AMPL
15 lines
265 B
AMPL
module github.com/tiennm99/hako-crawler
|
|
|
|
go 1.25.0
|
|
|
|
require (
|
|
github.com/PuerkitoBio/goquery v1.12.0
|
|
github.com/go-pdf/fpdf v0.9.0
|
|
golang.org/x/sync v0.22.0
|
|
)
|
|
|
|
require (
|
|
github.com/andybalholm/cascadia v1.3.3 // indirect
|
|
golang.org/x/net v0.52.0 // indirect
|
|
)
|