Commit Graph
134 Commits
Author SHA1 Message Date
tiennm99 5fbc25f674 fix: gate /rawappleapp + /rawgoogleapp on authorizeGroup
Match the auth check used by every other command. Without it,
any chat that knows the bot username could dump arbitrary
App Store / Play Store JSON.
2026-05-09 23:01:19 +07:00
tiennm99 2a78a34733 fix: /delgroup wipes group key + scheduler/checkapp guard non-finite updated
/delgroup previously only removed the chatId from the admin allowlist;
the matching group:{chatId} key (with its tracked-app state) was left
in Redis with no TTL. Re-adding the same group later resurrected the
old subscription list. Now the command calls store.group.deleteGroup
after the admin removal succeeds.

The scheduler's Google branch and /checkapp's Google rows path both
called daysBetween(updatedMs, now) without guarding non-finite
updatedMs values, unlike the parallel Apple branches. A garbage
upstream value would have produced NaN days. Skip those entries.
2026-05-09 22:37:39 +07:00
tiennm99 413dc62d4c refactor: remove redundant inner try/catch in commands
The dispatcher already wraps every handler in try/catch and sends the
'Internal server error' fallback on failure. Each command's inner
try/catch around its Redis ops was masking that path — the dispatcher
never saw the error, so logger.error('command failed') never fired.
Removing the inner catches restores observability and shortens each
file. The semantically-different try/catch blocks (mapping upstream
API failures to a different user message) are kept.
2026-05-09 22:37:02 +07:00
tiennm99 3f586d1d9f refactor: dispatcher pre-parses args, drop arg helpers + info.js
The dispatcher now extracts both the command name and its arguments
once and passes the args array to handlers via a 3rd parameter. Each
command file drops its splitArgs(getCommandArguments(msg.text)) call
and the corresponding imports. command-utils.js loses the now-unused
arg helpers (only auth helpers remain). info.js — a 12-line file with
one sendMessage — folds into bot.js's commands map.
2026-05-09 22:35:38 +07:00
tiennm99 ee491f9eb5 refactor: merge apple/google app caches into one parametrized repo
Two near-identical per-store repository files collapse into a single
createAppCacheRepository(handle, prefix, ttl) factory used twice from
app-builder. Scrapers now receive the cache directly (one less layer),
and the cache entry shape drops the obsolete _id field — the Redis key
already encodes the appId.
2026-05-09 22:32:56 +07:00
tiennm99 9bcdaa9130 refactor: drop dead repo + telegram-api exports, inline createStore
Delete repository/store.js (one-line aggregator) — wiring now inline in
app-builder.js. Drop unused exports: scan + UpstashUnavailable from
upstash.js, getMe + TelegramApiError class from telegram-api.js,
init/getAdmin/save from admin-repository.js, exists/saveGroup from
group-repository.js. Generic Error replaces the named error classes.
2026-05-09 22:32:12 +07:00
tiennm99 32b14217cc refactor: drop Mongo class discriminator + delete src/models/
Inline trivial factory bodies into the repos and scrapers that used them.
The class:/_id: fields were Java-Mongo parity artifacts that nothing
in this codebase reads — Redis docs with the old fields still parse
fine; the next write drops them.
2026-05-09 22:30:48 +07:00
tiennm99 c61b77f3c0 chore: journal Phase 7 cleanup of vercel-upstash consolidation
Closes the 260509-1656-consolidate-vercel-upstash plan with a record
of the cleanup commit (0a395bd) and the operator follow-ups left open.
2026-05-09 21:50:58 +07:00
tiennm99 f1710a5698 chore: remove cloudflare + docker + legacy migration scripts
Phase 7 cleanup of the Vercel + Upstash consolidation plan:

- delete wrangler.toml, Dockerfile, docker-compose{,.dev}.yml,
  scripts/migrate-atlas-to-upstash.js (one-shot migration done)
- drop wrangler + mongodb devDeps and migrate* npm scripts;
  regenerate package-lock.json (-70 packages)
- prune CF/Wrangler/Atlas-export entries from .gitignore + .vercelignore
- drop MONGODB_URI from .env.deploy.example
- rewrite README for Vercel + Upstash architecture
- refresh stale Cloudflare comments in src/{logger,models,repository}
2026-05-09 21:49:48 +07:00
tiennm99 1799ebf29f fix: use Vercel classic Node runtime API instead of Web Standards Request
Vercel `nodejs` runtime passes IncomingMessage/ServerResponse with
shouldAddHelpers=true (auto-parsed JSON body, .status/.send helpers),
not the Web Standards Request/Response. Calling `req.headers.get(...)`
on the classic IncomingMessage threw `TypeError: req.headers.get is
not a function` and crashed every webhook + cron invocation with 500.

Switch both handlers to (req, res) signature, read headers as plain
object (lowercased keys), use req.body for parsed JSON, and respond
via res.status().send().

Caught during Phase 6 smoke test of the first prod deploy.
2026-05-09 20:46:29 +07:00
tiennm99 458772ff15 chore: gitignore .vercel/ (CLI artifacts from vercel build/pull) 2026-05-09 20:36:48 +07:00
tiennm99 cad84e21a4 feat: accept KV_REST_API_* env vars as Upstash fallback
Vercel Marketplace Upstash integration injects KV_REST_API_URL and
KV_REST_API_TOKEN — different names from vanilla Upstash signup
(UPSTASH_REDIS_REST_URL / UPSTASH_REDIS_REST_TOKEN). Adapter and
migration script now accept either form, so the operator doesn't have
to duplicate values when sharing an Upstash DB provisioned via the
Vercel integration. UPSTASH_* takes precedence when both are set.
2026-05-09 20:17:06 +07:00
tiennm99 a5af074062 chore: refresh .env.deploy.example for Vercel + Upstash + add session journal
- Add Upstash REST creds, KEY_PREFIX, MONGODB_URI (one-shot migration)
- Update WORKER_URL example to Vercel format (var name retained for
  register-webhook.js compatibility; rename deferred to Phase 7)
- Track docs/journals/ entry from cook session
2026-05-09 20:13:15 +07:00
tiennm99 73762fa0a0 feat: migrate to Vercel + Upstash with KEY_PREFIX namespacing
Phases 1-5 of consolidate-vercel-upstash plan. Replaces Cloudflare
Workers + KV with Vercel serverless functions + Upstash Redis. Inlines
app-store-scraper / google-play-scraper npm libs (drops the
store-scraper.vercel.app HTTP roundtrip). KEY_PREFIX (default
'store-scraper-bot:') namespaces all Redis keys so the Upstash DB can
be safely shared with other Vercel projects.

- vercel.json + .vercelignore + Vercel-aware package.json scripts
- api/webhook.js + api/cron.js Vercel functions (with shared
  src/app-builder.js); cron auth fails closed when CRON_SECRET unset
- src/repository/upstash.js replaces kv.js; all 4 repos take a handle
  bundling client + prefix
- scripts/migrate-atlas-to-upstash.js writes legacy Java Atlas state
  directly to Upstash with --dry-run + --include-cache flags
- .env.example refreshed for the new env surface

Phases 6 (Vercel deploy + webhook cutover) and 7 (Docker + wrangler
cleanup) remain operator-driven post-deploy.
2026-05-09 20:07:07 +07:00
tiennm99 cf17071679 docs: update Java reference to java-store-scraper-bot
Java repo renamed legacy-store-scraper-bot → java-store-scraper-bot for symmetry with go-store-scraper-bot. Status (deprecated/maintained) belongs in README banners, not URLs.
2026-05-09 17:38:31 +07:00
tiennm99 9eab9e07ba refactor: rename project to store-scraper-bot
GitHub repo rename: js-store-scraper-bot becomes the canonical store-scraper-bot. The Java reference impl is renamed to legacy-store-scraper-bot in parallel.

Updates: package.json name + description, README header + reference link, wrangler.toml worker name (file slated for removal in cleanup phase but kept consistent in the interim).
2026-05-09 17:18:50 +07:00
tiennm99 b55ff19611 docs: lock Vercel + Upstash consolidation plan
Brainstorm + research concluded: bot moves off CF Workers + Vercel split-architecture onto a single Vercel deployment with Upstash Redis storage. 7-phase implementation plan added.

Supersedes the in-progress CF KV migration plan; code-level KV migration already shipped, deploy phases never executed.
2026-05-09 17:18:39 +07:00
tiennm99 0169c09b0f docs: add Phase 06 (Atlas → KV data migration) to KV plan
Adds the phase doc that drove the migration script and updates plan.md
with the new phase, dependency edges (02 → 06 → 04), and removes the
prior "Out of scope: Data migration" line.
2026-05-05 21:15:27 +07:00
tiennm99 d33bfeae3d feat: add Atlas → KV one-shot migration script
Operator runs `npm run migrate` (reads admin + group docs from Atlas)
followed by `npm run migrate:bulk` (uploads via wrangler kv bulk put).
Cache collections are skipped by default since they auto-rebuild from
upstream APIs; --include-cache flag migrates them with TTL preserved.

- mongodb is added as a devDependency only — never enters the Worker
  bundle, the Worker still talks to KV exclusively.
- scripts/.atlas-export.json is gitignored (contains exported state).
- README documents the one-time runbook.
2026-05-05 21:15:18 +07:00
tiennm99 971c9c311f docs: add KV migration plan, supersede Atlas deploy plan
Adds the 260505-1425 plan that drove the KV swap (Phase 01 already
applied in the preceding commit; Phases 02-05 are operator-driven).
Marks 260426-2327-cloudflare-deploy-and-smoke as superseded since
the KV pivot was taken pre-emptively rather than after a hard-gate trip.
2026-05-05 20:39:12 +07:00
tiennm99 9898f408a2 feat: replace MongoDB driver with Cloudflare KV storage
Drops the mongodb dependency entirely; all four logical collections
(admin singleton, group, apple_app, google_app) now live in a single KV
namespace bound as STORE_KV with prefixed keys. Cache TTL is delegated
to KV via expirationTtl (clamped to the 60s minimum). Document shape,
field names, and Java parity at the doc level are preserved.

- Adds src/repository/kv.js helper (getJson/putJson/del with TTL clamp)
- Rewrites all four *-repository.js modules on top of KV
- Removes src/repository/mongodb.js and the MONGODB_URI env requirement
- Adds an early STORE_KV-binding guard in src/index.js
- Bumps to 0.3.0
2026-05-05 20:39:01 +07:00
tiennm99 00f49531e0 docs: add Java vs JS parity reports from /ck:xia
researcher report maps 47 Java files to 35 JS files (11 PARITY, 2
MINOR, 0 GAP, 3 EXTRA). xia synthesis records source manifest, decision
matrix, and risk score. No port work needed; remaining items are
operational.
2026-04-29 16:07:17 +07:00
tiennm99 00a6415778 chore: archive code-port plan, add plans/todo.md index
Code port (commit bff1d32) is done; move plan to plans/archive/.
New plans/todo.md is the forward-looking index pointing at the active deploy
plan, pre-flight checks, hard gates, and open questions for the operator.
2026-04-26 23:46:22 +07:00
tiennm99 797f040574 feat: Cloudflare Workers code port (deploy pending)
Refactors source to be Worker-shaped. No live deploy yet — sister deploy plan
runs Atlas provisioning + smoke later.

- wrangler.toml with nodejs_compat_v2, daily UTC 0 cron (= 7am Asia/Ho_Chi_Minh)
- package.json: drop node-telegram-bot-api, node-cron, dotenv, pino,
  pino-pretty; add wrangler devDep; bump to 0.2.0
- src/bot/telegram-api.js: raw fetch wrapper for Telegram Bot API
- src/bot/dispatch.js: per-message dispatcher extracted from polling loop
- src/repository/mongodb.js: memoized MongoClient per warm isolate, typed
  MongoUnavailable error, fast-fail timeouts
- src/repository/store.js: factory binding env once
- All 4 repositories converted to factory shape
- src/api/{apple,google}-scraper.js: take store instead of importing repos
- src/index.js: Worker entry exporting { fetch, scheduled }; webhook validates
  X-Telegram-Bot-Api-Secret-Token; ack-then-waitUntil pattern
- src/scheduler/scheduler.js: trimmed; runDailyCheck only (no node-cron)
- src/config.js, src/logger.js: env-driven, console.log JSON output
- scripts/register-webhook.js: setWebhook + setMyCommands; --dry-run supported
- scripts/check-secret-leaks.js: lint blocks console.log(env.<SECRET>)
- plans/260426-2015-cloudflare-worker-code-port: this code port plan
- plans/260426-2327-cloudflare-deploy-and-smoke: sister deploy plan

Validated via node --check on all 32 source files; lint clean. Real deploy
gates (bundle size, cold-start CPU) run in deploy plan.
2026-04-26 23:36:39 +07:00
tiennm99 3c97553c69 feat: initial JavaScript port of store-scraper-bot
Node.js 20+ ESM port mirroring Java/Go implementations.

- 13 Telegram commands matching Java identifiers
- MongoDB schema parity (common, group, apple_app, google_app collections)
- Apple/Google scrapers calling store-scraper.vercel.app with 10-min cache
- Daily 7am Vietnam-time cron with weekend-silent mode
- HTML table renderer matching Java/Go output
- Docker + Compose (prod and dev)

Untested end-to-end against live Telegram or upstream API.
2026-04-26 20:16:00 +07:00
tiennm99 ca1a409610 Clear and rewrite
Retire the Go port; history continues from the JavaScript implementation (store-scraper-bot).
2026-08-04 23:00:01 +07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
824279e1fe build(deps): bump golang.org/x/crypto from 0.26.0 to 0.45.0 (#1)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.26.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.26.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-23 19:17:21 +07:00
tiennm99 6bc7756596 docs: update reference to java-store-scraper-bot 2026-05-09 17:38:42 +07:00
tiennm99 a5cade43cf docs: point reference link at legacy-store-scraper-bot
Java reference impl renamed from store-scraper-bot to legacy-store-scraper-bot. The new canonical store-scraper-bot is the JS implementation.
2026-05-09 17:29:21 +07:00
tiennm99 fdb5bee896 docs: mark project as preview, AI-assisted, untested 2026-04-25 22:01:35 +07:00
tiennm99 e695d9b223 feat: port logic from Java original to align with current Java behavior
- Switch Telegram parse mode to HTML to match Java client.
- Rename command identifiers to Java's: delgroup/delapple/delgoogle/
  checkappscore/rawappleapp/rawgoogleapp.
- Move admin singleton to "common" collection at _id="admin".
- Store group _id as string-of-int64 to match Java AbstractModel schema.
- Add `class` discriminator field on persisted models.
- Expand AppleAppResponse (42 fields) and GoogleAppResponse (63 fields,
  with nested Category/Feature) to match Java records.
- Add AppleAppRequest (track id or bundle id) / GoogleAppRequest types.
- Implement Raw* commands as Telegram document attachments instead of
  truncated text.
- Group-authorization gate (admin.HasGroup(chatId)) on non-admin commands.
- Cache TTL via millis timestamp; round score to 1 decimal; weekend silent
  send in scheduler.
- Match Java table renderer: " │ " separators, "─┼─" rows every 5 lines.
- Prefer MONGODB_CONNECTION_STRING env (Java parity); auto-extract DB
  name from URI with fallback to "store-scraper-bot".

Note: This is an AI-assisted port (not human-verified). Behavior parity
with the Java implementation has not been tested end-to-end; treat as a
starting point.
2026-04-25 20:26:25 +07:00
tiennm99 cfc8d67641 Merge branch 'feature/claude-converted' 2025-12-03 20:46:02 +07:00
tiennm99 9b03eb007a feat(core): add logic converted from original repo by Claude Code 2025-11-15 18:54:06 +07:00
tiennm99 f243bc9b88 Update README.md 2025-12-01 00:08:58 +07:00
Tien Nguyen MinhandGitHub 165023f539 Initial commit 2025-11-15 18:52:19 +07:00
tiennm99 c5f3628f01 Clear and rewrite
Retire the Java implementation; history continues from the Go port (store-scraper-bot-go).
2026-08-04 22:59:44 +07:00
tiennm99 e794000df3 feat(mongodb): manual handle expire 2025-12-23 21:15:33 +07:00
tiennm99 d85dc72aab feat(deps): cleanup 2025-12-16 22:20:49 +07:00
tiennm99 d99798fae7 feat(repo): cleanup 2025-12-16 22:08:25 +07:00
tiennm99 bb4c12e29e fix(mongo): codec 2025-12-16 21:57:57 +07:00
tiennm99 b048cbc0a5 fix(deserialize): 2025-12-15 22:56:37 +07:00
tiennm99 c060aa891d feat(codec): try to auto handle (de)serialize 2025-12-15 22:49:57 +07:00
tiennm99 8394f909ae feat(mongo): cleanup 2025-12-15 22:46:21 +07:00
tiennm99 0eae89ac16 feat(db): use mongodb instead of couchbase (WIP) 2025-11-22 23:05:39 +07:00
tiennm99 dae676da69 fix(schedule): exception break schedule 2025-11-17 12:07:21 +07:00
tiennm99 23fd181c0f Update log4j2.xml 2025-11-14 18:55:22 +07:00
tiennm99 d5467f16bc Update log4j2.xml 2025-11-14 18:32:53 +07:00
tiennm99 56b6dc5cb3 feat(loki): try to enable loki again 2025-11-14 18:25:46 +07:00
tiennm99 61e1dcfe00 Update Dockerfile 2025-11-14 18:11:10 +07:00
tiennm99 5879bdda67 Update Dockerfile 2025-11-14 18:05:32 +07:00