From 84e22fa167764883bcfd4d7959f93f87862b15ad Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Mon, 11 May 2026 20:44:47 +0700 Subject: [PATCH] =?UTF-8?q?docs:=20expand=20README=20=E2=80=94=20config=20?= =?UTF-8?q?schema,=20output=20format,=20error=20handling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- miti-scraper/README.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/miti-scraper/README.md b/miti-scraper/README.md index 8f9a259..580bbb4 100644 --- a/miti-scraper/README.md +++ b/miti-scraper/README.md @@ -1,14 +1,40 @@ # miti-scraper -Simple configurable website scraper written in Go. +Configurable single-site web scraper in Go. Crawls one root URL, follows internal links that match whitelist regexes, strips HTML to plain text, and writes one `.txt` per page under `data/`. -## Quick start +Built on [`gocolly/colly`](https://github.com/gocolly/colly). + +## Config — `config.yaml` + +```yaml +root_url: "https://example.com/" + +# Only URLs matching at least one regex are crawled and saved +whitelist: + - "^https?://([^/]*\\.)?example\\.com(/[^?]*)?$" + +# Newline-delimited list of already-processed URLs. Auto-resumes across runs. +data_file: "processed_urls.txt" + +# Politeness — seconds between requests +delay_seconds: 1 +``` + +## Run ```bash -# Edit config.yaml to set target URLs and rules go run . ``` +## Output + +- `data/_.txt` — one file per scraped page, HTML stripped to whitespace-collapsed text (drops `