diff --git a/miti-scraper/README.md b/miti-scraper/README.md index 8f9a259..580bbb4 100644 --- a/miti-scraper/README.md +++ b/miti-scraper/README.md @@ -1,14 +1,40 @@ # miti-scraper -Simple configurable website scraper written in Go. +Configurable single-site web scraper in Go. Crawls one root URL, follows internal links that match whitelist regexes, strips HTML to plain text, and writes one `.txt` per page under `data/`. -## Quick start +Built on [`gocolly/colly`](https://github.com/gocolly/colly). + +## Config — `config.yaml` + +```yaml +root_url: "https://example.com/" + +# Only URLs matching at least one regex are crawled and saved +whitelist: + - "^https?://([^/]*\\.)?example\\.com(/[^?]*)?$" + +# Newline-delimited list of already-processed URLs. Auto-resumes across runs. +data_file: "processed_urls.txt" + +# Politeness — seconds between requests +delay_seconds: 1 +``` + +## Run ```bash -# Edit config.yaml to set target URLs and rules go run . ``` +## Output + +- `data/_.txt` — one file per scraped page, HTML stripped to whitespace-collapsed text (drops `