monkeyd-crawler

Downloads every chapter of a monkeydd.com novel and exports it as a single PDF sized for reading on a phone.

Install

Requires Go 1.22+.

go build ./cmd/monkeyd-crawler

Usage

Pass the novel's landing page URL:

./monkeyd-crawler -url https://monkeydd.com/tro-lai-nam-thang-cu.html

The PDF is named after the novel unless you pass -out.

# Bigger type, A5 page for a tablet
./monkeyd-crawler -url https://monkeydd.com/tro-lai-nam-thang-cu.html \
  -page a5 -font-size 14 -out truyen.pdf

# Try the layout on 3 chapters before fetching the whole book
./monkeyd-crawler -url https://monkeydd.com/tro-lai-nam-thang-cu.html -limit 3

Flags

Flag	Default	Purpose
-url	required	Novel landing page URL
-out	novel title	Output PDF path
-page	phone	Page size: phone, a5, a4
-font-size	10	Body font size in points
-line-spacing	1.55	Line height as a multiple of font size
-margin	6	Page margin in mm
-font	auto	Path to a .ttf; defaults to a system font, else the bundled one
-workers	4	Concurrent chapter fetches
-delay	400ms	Minimum delay between requests
-retries	3	Retries per request
-limit	0	Fetch only the first N chapters (0 = all)
-cache	.cache	Cache directory for raw pages (empty to disable)

Why the default page is 90×160 mm

Phone readability is governed by page shape more than by font size. A PDF viewer scales a whole page to fit the screen, so a large font on an A4 page still ends up small: the page is about three times wider than a phone screen and gets shrunk to match. The default page is cut to a 9:16 ratio so it fills the screen at 100% zoom, where the default 10 pt renders at a comfortable size with roughly 43 characters per line and 26 lines per page. Raising it to 12 pt gives about 36 characters per line; use -font-size to suit your screen and eyes.

Use -page a5 or -page a4 for a tablet or for printing.

How it works

1. Fetch the landing page and read the chapter list from div.list-chapters.
2. Cross-check that list against the #selected_chapter dropdown embedded in the first chapter page. When the dropdown is a superset it wins; a disagreement is reported.
3. Fetch each chapter concurrently and extract its text.
4. Render one PDF, each chapter starting on a new page.

Two site behaviours the extractor has to handle

Chapter text is partly served through CSS. The markup contains empty elements, and the stylesheet supplies the missing word:

Nghe <span class="t-3e625e…"></span> trưởng tử

.t-3e625e…:before { content: "vị"; }

Reading DOM text alone silently drops these — about 19% of the words in a sampled chapter. The extractor parses the :before rules and substitutes each word back in.

Chapter URLs cannot be generated. Numbering has gaps (the sample novel has no chapter 4) and slugs are not uniform across novels (/14.html on one, /chuong-12.html on another), so chapter links are always parsed from the page rather than constructed from a count.

Politeness and caching

Requests are spaced by -delay globally, so raising -workers does not raise the request rate. Raw pages are cached under .cache/, so re-exporting with different font or page settings costs no requests. Delete the directory to refetch.

Tests

go test ./...

Tests run against synthetic fixtures that reproduce the CSS-injected words, the newest-first chapter ordering, and the numbering gap. No network access required.

Layout

cmd/monkeyd-crawler/    CLI
export/                 URL -> PDF in one call; shared by the CLI and importers
monkeyd/                fetching, HTML/CSS parsing, crawl orchestration
pdfout/                 PDF rendering, font resolution, bundled fallback font

Use as a library

The packages are importable, so another Go program can produce the same PDF without shelling out to the binary. export.Export is the whole pipeline — chapter list, fetch, font resolution, render:

result, err := export.Export(ctx, export.Request{
	NovelURL: "https://monkeydd.com/tro-lai-nam-thang-cu.html",
	OutDir:   tmpDir,
})

Only NovelURL is required; each zero-valued field falls back to the same default as the matching CLI flag. Because zero means "unset", ask for no cache or no request delay with the NoCache and NoDelay fields rather than by zeroing CacheDir or Delay. Pass a Log function to receive the progress messages the CLI prints to stderr.

Fonts

The PDF embeds a TrueType font, and Vietnamese needs one that covers the Latin Extended Additional block — a basic-Latin font silently drops the diacritics. The font is resolved in this order:

1. the path given to -font / Request.FontFile, which is an error if it cannot be read — a named font is not silently substituted;
2. a system font known to cover Vietnamese (see pdfout.FindFont);
3. the bundled DejaVu Sans, compiled into the binary.

Step 3 means rendering never depends on the host having fonts installed, which is what a minimal container usually looks like. See pdfout/fonts/NOTICE.md for the bundled font's provenance and licensing.

Font data is handed to the PDF writer as bytes, not as a path. fpdf's path-taking AddUTF8Font joins the name onto its own font directory, which it defaults to "."; an absolute path is thereby rewritten into a working-directory-relative one and fails wherever the process does not run from the filesystem root.

Scope

Downloaded text stays on your machine; only fetch content you are allowed to read offline, and respect the site's terms.