mirror of
https://github.com/tiennm99/store-scraper-bot.git
synced 2026-08-31 02:25:49 +00:00
tiennm99
797f040574
Refactors source to be Worker-shaped. No live deploy yet — sister deploy plan
runs Atlas provisioning + smoke later.
- wrangler.toml with nodejs_compat_v2, daily UTC 0 cron (= 7am Asia/Ho_Chi_Minh)
- package.json: drop node-telegram-bot-api, node-cron, dotenv, pino,
pino-pretty; add wrangler devDep; bump to 0.2.0
- src/bot/telegram-api.js: raw fetch wrapper for Telegram Bot API
- src/bot/dispatch.js: per-message dispatcher extracted from polling loop
- src/repository/mongodb.js: memoized MongoClient per warm isolate, typed
MongoUnavailable error, fast-fail timeouts
- src/repository/store.js: factory binding env once
- All 4 repositories converted to factory shape
- src/api/{apple,google}-scraper.js: take store instead of importing repos
- src/index.js: Worker entry exporting { fetch, scheduled }; webhook validates
X-Telegram-Bot-Api-Secret-Token; ack-then-waitUntil pattern
- src/scheduler/scheduler.js: trimmed; runDailyCheck only (no node-cron)
- src/config.js, src/logger.js: env-driven, console.log JSON output
- scripts/register-webhook.js: setWebhook + setMyCommands; --dry-run supported
- scripts/check-secret-leaks.js: lint blocks console.log(env.<SECRET>)
- plans/260426-2015-cloudflare-worker-code-port: this code port plan
- plans/260426-2327-cloudflare-deploy-and-smoke: sister deploy plan
Validated via node --check on all 32 source files; lint clean. Real deploy
gates (bundle size, cold-start CPU) run in deploy plan.
js-store-scraper-bot
JavaScript (Node.js) port of store-scraper-bot.
⚠️ Preview / unstable — use at your own risk. This port was produced largely with AI assistance and has not been tested end-to-end against a live Telegram bot or the upstream Java implementation. Behavior parity is intended but unverified. Do not run against a production database.
The Java version remains the reference implementation.
Status
- Mongo schema matches Java/Go (collections:
common,group,apple_app,google_app; string_id;classdiscriminator). - Telegram command identifiers match Java exactly:
/info,/addgroup,/delgroup,/listgroup,/addapple,/delapple,/addgoogle,/delgoogle,/listapp,/checkapp,/checkappscore,/rawappleapp,/rawgoogleapp. - HTML parse mode; weekend-silent daily report; configurable API cache (default 10 min).
Requirements
- Node.js 20+ (uses built-in
fetch) - MongoDB 4.4+
Configuration
See .env.example:
| Name | Notes |
|---|---|
TELEGRAM_BOT_TOKEN |
Telegram bot token (required) |
TELEGRAM_BOT_USERNAME |
Bot username (required) |
MONGODB_CONNECTION_STRING |
Preferred (Java parity); falls back to MONGO_URI |
MONGO_DATABASE |
Optional; inferred from URI path if omitted |
ADMIN_IDS |
Comma-separated Telegram user IDs (required) |
ENV |
DEVELOPMENT or PRODUCTION |
SOURCE_COMMIT |
Optional; shown on startup |
APP_CACHE_SECONDS |
Cache TTL for upstream API responses (default 600) |
NUM_DAYS_WARNING_NOT_UPDATED |
Threshold for daily warning (default 30) |
SCHEDULE_CHECK_APP_TIME |
Cron expression in Vietnam timezone (default 0 7 * * *) |
Run
npm install
cp .env.example .env # then edit credentials
npm start
Or via Docker Compose:
docker compose up --build
Project Layout
src/
├── index.js # entry point: wire up config, mongo, scrapers, bot, scheduler
├── config.js
├── logger.js
├── api/
│ ├── apple-scraper.js
│ └── google-scraper.js
├── models/ # plain object factories matching Mongo docs
├── repository/ # Mongo collection wrappers (admin / group / cached app)
├── bot/
│ ├── bot.js # Telegram polling, command dispatch, sender
│ └── commands/ # one file per /command
├── scheduler/scheduler.js # daily 7am Vietnam-time check
└── util/ # table renderer, time helpers
Differences vs Go / Java
- Group / admin / chat IDs are JS
numbers. Telegram chat IDs fit in safe-int range, so this is intentional and matches Telegram's documented limits. - Pino logging instead of Java/Go's structured loggers; semantics equivalent.
- HTTP via Node 20's built-in
fetch(no extra dependency).