mirror of
https://github.com/tiennm99/monkeyd-crawler.git
synced 2026-08-04 19:30:38 +00:00
tiennm99
c3fd484c92
Fetches every chapter of a monkeydd.com novel and renders it as a single
PDF laid out for reading on a phone.
Two site behaviours drive the extractor design:
- Roughly a fifth of each chapter's words are not in the markup. The page
emits empty spans and supplies the word from the stylesheet via
":before { content: ... }" rules, so reading DOM text alone drops them
with no error. The extractor resolves those rules and substitutes the
words back; a test asserts they disappear when the rule is removed.
- Chapter URLs cannot be generated. Numbering has gaps and slugs are not
uniform across novels, so chapter links are always parsed from the page.
The chapter list is read from the landing page and cross-checked against
the dropdown embedded in each chapter page, so a truncated list cannot
silently shorten the export.
PDF defaults to a 90x160mm page rather than A4 with large type: viewers
scale a whole page to fit the screen, so a phone-shaped page fills it at
100% zoom where 12pt stays comfortable. A5 and A4 remain available.
Requests are spaced globally, so raising worker count does not raise the
request rate. Raw pages cache to disk so re-exporting at different font
or page settings needs no network.
7 lines
471 B
Plaintext
7 lines
471 B
Plaintext
github.com/go-pdf/fpdf v0.9.0 h1:PPvSaUuo1iMi9KkaAn90NuKi+P4gwMedWPHhj8YlJQw=
|
|
github.com/go-pdf/fpdf v0.9.0/go.mod h1:oO8N111TkmKb9D7VvWGLvLJlaZUQVPM+6V42pp3iV4Y=
|
|
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
|
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
|
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
|
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|