mirror of
https://github.com/tiennm99/store-scraper-bot.git
synced 2026-09-01 10:19:49 +00:00
tiennm99
1799ebf29f
Vercel `nodejs` runtime passes IncomingMessage/ServerResponse with shouldAddHelpers=true (auto-parsed JSON body, .status/.send helpers), not the Web Standards Request/Response. Calling `req.headers.get(...)` on the classic IncomingMessage threw `TypeError: req.headers.get is not a function` and crashed every webhook + cron invocation with 500. Switch both handlers to (req, res) signature, read headers as plain object (lowercased keys), use req.body for parsed JSON, and respond via res.status().send(). Caught during Phase 6 smoke test of the first prod deploy.
store-scraper-bot
JavaScript (Node.js) implementation. Ports java-store-scraper-bot.
⚠️ Preview / unstable — use at your own risk. This port was produced largely with AI assistance and has not been tested end-to-end against a live Telegram bot or the upstream Java implementation. Behavior parity is intended but unverified. Do not run against a production database.
The Java version remains the reference implementation.
Status
- Mongo schema matches Java/Go (collections:
common,group,apple_app,google_app; string_id;classdiscriminator). - Telegram command identifiers match Java exactly:
/info,/addgroup,/delgroup,/listgroup,/addapple,/delapple,/addgoogle,/delgoogle,/listapp,/checkapp,/checkappscore,/rawappleapp,/rawgoogleapp. - HTML parse mode; weekend-silent daily report; configurable API cache (default 10 min).
Requirements
- Node.js 20+ (uses built-in
fetch) - MongoDB 4.4+
Configuration
See .env.example:
| Name | Notes |
|---|---|
TELEGRAM_BOT_TOKEN |
Telegram bot token (required) |
TELEGRAM_BOT_USERNAME |
Bot username (required) |
MONGODB_CONNECTION_STRING |
Preferred (Java parity); falls back to MONGO_URI |
MONGO_DATABASE |
Optional; inferred from URI path if omitted |
ADMIN_IDS |
Comma-separated Telegram user IDs (required) |
ENV |
DEVELOPMENT or PRODUCTION |
SOURCE_COMMIT |
Optional; shown on startup |
APP_CACHE_SECONDS |
Cache TTL for upstream API responses (default 600) |
NUM_DAYS_WARNING_NOT_UPDATED |
Threshold for daily warning (default 30) |
SCHEDULE_CHECK_APP_TIME |
Cron expression in Vietnam timezone (default 0 7 * * *) |
Run
npm install
cp .env.example .env # then edit credentials
npm start
Or via Docker Compose:
docker compose up --build
Migrating from MongoDB Atlas (one-time)
If you're moving an existing bot off Atlas to Cloudflare KV:
# 1. Make sure .env has MONGODB_URI pointing at the Atlas cluster (read-only is fine).
npm install # pulls the mongodb devDep needed by the migration script
npm run migrate # writes scripts/.atlas-export.json (admin + groups)
# Optionally also migrate cached app entries with their TTL preserved:
# npm run migrate -- --include-cache
npm run migrate:bulk # uploads the JSON to the production KV namespace
rm scripts/.atlas-export.json # contains your bot state — delete after success
Cache collections (apple_app, google_app) are skipped by default since they auto-rebuild from upstream APIs within APP_CACHE_SECONDS. Re-running the migration is idempotent.
Project Layout
src/
├── index.js # entry point: wire up config, mongo, scrapers, bot, scheduler
├── config.js
├── logger.js
├── api/
│ ├── apple-scraper.js
│ └── google-scraper.js
├── models/ # plain object factories matching Mongo docs
├── repository/ # Mongo collection wrappers (admin / group / cached app)
├── bot/
│ ├── bot.js # Telegram polling, command dispatch, sender
│ └── commands/ # one file per /command
├── scheduler/scheduler.js # daily 7am Vietnam-time check
└── util/ # table renderer, time helpers
Differences vs Go / Java
- Group / admin / chat IDs are JS
numbers. Telegram chat IDs fit in safe-int range, so this is intentional and matches Telegram's documented limits. - Pino logging instead of Java/Go's structured loggers; semantics equivalent.
- HTTP via Node 20's built-in
fetch(no extra dependency).