mirror of
https://github.com/tiennm99/store-scraper-bot.git
synced 2026-07-30 08:23:12 +00:00
docs: lock Vercel + Upstash consolidation plan
Brainstorm + research concluded: bot moves off CF Workers + Vercel split-architecture onto a single Vercel deployment with Upstash Redis storage. 7-phase implementation plan added. Supersedes the in-progress CF KV migration plan; code-level KV migration already shipped, deploy phases never executed.
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "Cloudflare KV migration + deploy"
|
||||
description: "Replace MongoDB Atlas with Cloudflare KV as the storage layer, deploy to Workers, register Telegram webhook, smoke-test, and document. Supersedes the Atlas-based deploy plan."
|
||||
status: pending
|
||||
status: superseded
|
||||
priority: P1
|
||||
effort: 5h
|
||||
branch: main
|
||||
@@ -9,8 +9,11 @@ tags: [cloudflare-workers, cloudflare-kv, deploy, migration]
|
||||
created: 2026-05-05
|
||||
blockedBy: []
|
||||
blocks: [260426-2327-cloudflare-deploy-and-smoke]
|
||||
supersededBy: 260509-1656-consolidate-vercel-upstash
|
||||
---
|
||||
|
||||
> **Superseded** — User chose to consolidate on Vercel + Upstash instead of completing the CF Workers deploy. Code-level KV migration shipped (commits `067d463`..`c4c6a93`); deploy phases never executed. Continuation tracked in [`260509-1656-consolidate-vercel-upstash`](../260509-1656-consolidate-vercel-upstash/plan.md).
|
||||
|
||||
# Cloudflare Workers + KV Storage Migration
|
||||
|
||||
User chose KV over Atlas to pre-empt the bundle-size and cold-start risks called out in the Atlas plan. Drops the `mongodb` driver entirely (~4–5 MiB compressed) and the `nodejs_compat_v2` flag along with it.
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Vercel scaffolding + deps"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 30 min
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Vercel scaffolding + deps
|
||||
|
||||
## Overview
|
||||
|
||||
Add Vercel project config, install new runtime deps, restructure `package.json` for Node-on-Vercel. No behavior changes yet — just scaffolding.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Vercel project recognizes the repo (Node 20 runtime, ESM)
|
||||
- Daily cron registered (00:00 UTC = 07:00 Asia/Saigon)
|
||||
- New deps installed; `mongodb` devDep stays (still needed by Phase 5 migration script)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
vercel.json cron schedule, function config (maxDuration, region)
|
||||
package.json + app-store-scraper, google-play-scraper,
|
||||
+ @upstash/redis, @vercel/functions
|
||||
scripts: dev → vercel dev, deploy → vercel deploy
|
||||
.vercelignore exclude /plans, /scripts/.atlas-export.json
|
||||
```
|
||||
|
||||
Vercel auto-detects `api/*.js` as serverless functions in Phase 4. This phase only sets the scaffolding.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `vercel.json`
|
||||
- Create: `.vercelignore`
|
||||
- Modify: `package.json` (deps, scripts)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add deps:
|
||||
```sh
|
||||
npm install app-store-scraper@^0.18.0 google-play-scraper@^10.1.2 @upstash/redis @vercel/functions
|
||||
```
|
||||
2. Update `package.json` scripts:
|
||||
- `dev`: `vercel dev` (was `wrangler dev`)
|
||||
- `deploy`: `vercel deploy --prod && npm run register`
|
||||
- Keep `migrate`, `lint`. Drop `migrate:bulk` (CF-specific).
|
||||
- Add `register`: `node --env-file=.env.deploy scripts/register-webhook.js`
|
||||
3. Create `vercel.json`:
|
||||
```json
|
||||
{
|
||||
"crons": [{ "path": "/api/cron", "schedule": "0 0 * * *" }],
|
||||
"functions": {
|
||||
"api/cron.js": { "maxDuration": 60 },
|
||||
"api/webhook.js": { "maxDuration": 30 }
|
||||
}
|
||||
}
|
||||
```
|
||||
4. Create `.vercelignore` excluding `plans/`, `scripts/.atlas-export.json`, `*.md`, `Dockerfile`, `docker-compose*.yml`, `wrangler.toml` (until Phase 7 deletes it).
|
||||
5. `npm install` and verify `node_modules` has all four new deps.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `package.json` shows new deps + new scripts
|
||||
- [ ] `vercel.json` parses (`vercel dev --debug` doesn't error on config)
|
||||
- [ ] `.vercelignore` excludes irrelevant paths
|
||||
- [ ] `npm install` exits clean
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** Vercel function bundle exceeds 250 MB with all deps. **Mitigation:** check `vercel build --debug` output; tree-shaking + `excludeFiles` pattern in `vercel.json` if needed. Expected size ~30 MB.
|
||||
- **Risk:** `app-store-scraper` uses deprecated `request` lib that may warn or fail under Node 20. **Mitigation:** verify in Phase 3 by importing + calling `app()`. Fallback: pin Node to 18 or swap lib (out of scope).
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Upstash repository adapter"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 1h
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 2: Upstash repository adapter
|
||||
|
||||
## Overview
|
||||
|
||||
Replace `src/repository/kv.js` Cloudflare KV wrapper with an Upstash Redis equivalent. Repository files (`admin-repository.js`, `group-repository.js`, `apple-app-repository.js`, `google-app-repository.js`) keep their public APIs; only the storage primitive changes.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Function-level API of `kv.js` preserved (`getJson`, `putJson`, `del`)
|
||||
- TTL semantics preserved (Apple/Google cache uses `expirationTtl` → Redis `EX`)
|
||||
- Drops `env.STORE_KV` binding; takes Upstash client instance instead
|
||||
- 60s minimum TTL clamp removed (Redis `EX` accepts 1s+, but keep clamp for parity safety)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/repository/
|
||||
├── upstash.js ← NEW: client factory + getJson/putJson/del/scan
|
||||
├── kv.js ← DELETE (or rename to keep history)
|
||||
├── store.js ← takes upstash client instead of env
|
||||
├── admin-repository.js ← unchanged signatures, internal: env→client
|
||||
├── group-repository.js ← unchanged signatures
|
||||
├── apple-app-repository.js ← unchanged signatures
|
||||
└── google-app-repository.js← unchanged signatures
|
||||
```
|
||||
|
||||
Client construction:
|
||||
```js
|
||||
import { Redis } from '@upstash/redis';
|
||||
export function createUpstashClient(env) {
|
||||
return new Redis({
|
||||
url: env.UPSTASH_REDIS_REST_URL,
|
||||
token: env.UPSTASH_REDIS_REST_TOKEN,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
API mapping:
|
||||
| CF KV | Upstash Redis | Note |
|
||||
|---|---|---|
|
||||
| `kv.get(key, 'json')` | `redis.get(key)` | Upstash auto-deserializes JSON when value was `JSON.stringify`d on `set` |
|
||||
| `kv.put(key, value)` | `redis.set(key, value)` | |
|
||||
| `kv.put(key, value, { expirationTtl: N })` | `redis.set(key, value, { ex: N })` | Redis `EX` is seconds |
|
||||
| `kv.delete(key)` | `redis.del(key)` | |
|
||||
| `kv.list({ prefix: 'group:' })` | `redis.scan(0, { match: 'group:*' })` | Phase 5 migration uses this |
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `src/repository/upstash.js`
|
||||
- Modify: `src/repository/store.js` (accept client, not env)
|
||||
- Modify: `src/repository/admin-repository.js` (use client)
|
||||
- Modify: `src/repository/group-repository.js` (use client)
|
||||
- Modify: `src/repository/apple-app-repository.js` (use client)
|
||||
- Modify: `src/repository/google-app-repository.js` (use client)
|
||||
- Delete: `src/repository/kv.js`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Write `src/repository/upstash.js`:
|
||||
- `createUpstashClient(env)` returns `Redis` instance
|
||||
- `getJson(client, key)`, `putJson(client, key, value, { expirationTtl } = {})`, `del(client, key)`
|
||||
- Mirror `kv.js` signatures so repository files only swap import path + first arg
|
||||
2. Update `store.js` to accept `client` instead of `env`:
|
||||
```js
|
||||
export function createStore(client, appCacheSeconds) { ... }
|
||||
```
|
||||
3. Update each repository to import from `./upstash.js` and take `client` instead of `env`. Search/replace `env, key` → `client, key` in each file.
|
||||
4. Delete `src/repository/kv.js`.
|
||||
5. Manual smoke: `node -e "import('./src/repository/upstash.js').then(m => console.log(m))"` to confirm import path resolves.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `src/repository/upstash.js` exports `createUpstashClient`, `getJson`, `putJson`, `del`
|
||||
- [ ] All four `*-repository.js` files import from `./upstash.js`, take `client`
|
||||
- [ ] `store.js` signature: `createStore(client, appCacheSeconds)`
|
||||
- [ ] `kv.js` deleted
|
||||
- [ ] No `env.STORE_KV` references remain in `src/`
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** `@upstash/redis` SDK auto-stringifies/parses values inconsistently. **Mitigation:** read SDK docs; if auto-JSON disabled, mirror `kv.js`'s `JSON.stringify`/`JSON.parse` explicitly.
|
||||
- **Risk:** Redis `SET key value EX 0` is invalid; current `KV_MIN_TTL_SECONDS = 60` clamp keeps us safe. Keep the clamp.
|
||||
- **Risk:** Redis returns `null` on missing key (same as KV). Verify in unit smoke.
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Inline scraper modules"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 45 min
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 3: Inline scraper modules
|
||||
|
||||
## Overview
|
||||
|
||||
Replace `fetch('https://store-scraper.vercel.app/...')` with direct calls to `app-store-scraper` and `google-play-scraper` npm libs. Eliminates the cross-service HTTP roundtrip.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `apple-scraper.js` and `google-scraper.js` keep their exported function signatures (`createAppleScraper`, `createGoogleScraper`, `buildAppleRequestBy*`, `buildGoogleRequest`) so command handlers don't change
|
||||
- Behavior parity: `rawApp(req)` returns JSON string; `app(req)` returns parsed object
|
||||
- Library `app()` method only — no other methods used today
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/api/apple-scraper.js
|
||||
├─ removed: BASE_URL, fetch()
|
||||
└─ uses: import store from 'app-store-scraper'; store.app(params)
|
||||
|
||||
src/api/google-scraper.js
|
||||
├─ removed: BASE_URL, fetch()
|
||||
└─ uses: import gplay from 'google-play-scraper'; gplay.app(params)
|
||||
```
|
||||
|
||||
`rawApp` returns `JSON.stringify(result)` since libs return JS objects, not strings. Command `/rawappleapp` and `/rawgoogleapp` consumers expect text.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `src/api/apple-scraper.js`
|
||||
- Modify: `src/api/google-scraper.js`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Edit `src/api/apple-scraper.js`:
|
||||
```js
|
||||
import store from 'app-store-scraper';
|
||||
import { newAppleApp } from '../models/apple-app.js';
|
||||
|
||||
export function buildAppleRequestByTrackId(id, country) {
|
||||
return { id, country, ratings: true };
|
||||
}
|
||||
export function buildAppleRequestByBundleId(appId, country) {
|
||||
return { appId, country, ratings: true };
|
||||
}
|
||||
|
||||
export function createAppleScraper(config, storeRepo) {
|
||||
const { logger } = config;
|
||||
const repo = storeRepo.appleApp;
|
||||
|
||||
async function app(req) {
|
||||
return await store.app(req);
|
||||
}
|
||||
async function rawApp(req) {
|
||||
return JSON.stringify(await app(req));
|
||||
}
|
||||
// cache, getApp, fetchAndCache: unchanged
|
||||
...
|
||||
}
|
||||
```
|
||||
2. Edit `src/api/google-scraper.js` analogously, importing `gplay` from `google-play-scraper`.
|
||||
3. Verify no other files import `BASE_URL` (grep `src/ -r 'BASE_URL'`).
|
||||
4. Smoke from a Node REPL after `npm install` (Phase 1 added the deps):
|
||||
```sh
|
||||
node -e "import('app-store-scraper').then(m => m.default.app({ appId: 'com.apple.weather', country: 'us' })).then(r => console.log(r.appId, r.version))"
|
||||
node -e "import('google-play-scraper').then(m => m.default.app({ appId: 'com.spotify.music', country: 'us' })).then(r => console.log(r.appId, r.version))"
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `grep -r 'store-scraper.vercel.app' src/` returns zero hits
|
||||
- [ ] Local smoke calls succeed against both stores
|
||||
- [ ] `apple-scraper.js`, `google-scraper.js` keep exported names; no other file modified
|
||||
- [ ] Returned shape from `app()` matches what cache models (`newAppleApp`, `newGoogleApp`) expect
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** `app-store-scraper.app({ ratings: true })` makes 2 HTTPS calls (lookup + ratings.svg). Latency higher than single-fetch. **Mitigation:** acceptable for command-driven calls; cache TTL covers daily cron amplification.
|
||||
- **Risk:** `google-play-scraper.app()` HTML scrape can fail when Google changes Play page markup. **Mitigation:** error already bubbles to user via existing try/catch in commands. Monitor first week post-deploy.
|
||||
- **Risk:** `app-store-scraper` uses deprecated `request` lib — Node 20 warns but still runs. **Mitigation:** suppress with `NODE_NO_WARNINGS=1` if noisy. Long-term: track lib for replacement.
|
||||
- **Risk:** Returned objects may have non-JSON-serializable fields (Dates). **Mitigation:** `JSON.stringify` produces ISO strings; consumers already parse strings.
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "HTTP layer (webhook + cron)"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 1h
|
||||
dependencies: [2, 3]
|
||||
---
|
||||
|
||||
# Phase 4: HTTP layer (webhook + cron)
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the Cloudflare Worker default-export entry (`src/index.js` with `fetch`/`scheduled` handlers) with two Vercel serverless functions: `api/webhook.js` (Telegram webhook) and `api/cron.js` (daily check via Vercel Cron).
|
||||
|
||||
## Requirements
|
||||
|
||||
- Telegram webhook validates `X-Telegram-Bot-Api-Secret-Token` header (parity with current CF check)
|
||||
- Webhook acks fast (<2 s), continues work in background via `@vercel/functions` `waitUntil`
|
||||
- Cron handler runs daily check, returns 200 on success
|
||||
- Both handlers use `loadConfig(process.env)` instead of CF `env` arg
|
||||
- Build wires up `createUpstashClient` once per invocation (cheap — no connection)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
api/
|
||||
├── webhook.js POST handler — Telegram webhook
|
||||
│ validates secret, calls dispatch(message), waitUntil
|
||||
└── cron.js GET handler — Vercel Cron triggers daily
|
||||
validates Authorization: Bearer $CRON_SECRET (Vercel signs cron requests)
|
||||
runs runDailyCheck(...)
|
||||
|
||||
src/
|
||||
├── index.js ← DELETE (CF default export gone)
|
||||
├── config.js ← Modify: loadConfig(env) → loadConfig(process.env or env)
|
||||
└── (rest unchanged)
|
||||
```
|
||||
|
||||
Vercel cron requests carry `Authorization: Bearer ${process.env.CRON_SECRET}` header — validate to prevent random POSTs from triggering daily check.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `api/webhook.js`
|
||||
- Create: `api/cron.js`
|
||||
- Modify: `src/config.js` (env source)
|
||||
- Delete: `src/index.js`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Update `src/config.js` to take a plain object (works with both `process.env` and CF `env`):
|
||||
```js
|
||||
export function loadConfig(env) {
|
||||
// existing logic, but `env` is now `process.env` on Vercel
|
||||
}
|
||||
```
|
||||
2. Create `api/webhook.js`:
|
||||
```js
|
||||
import { waitUntil } from '@vercel/functions';
|
||||
import { loadConfig } from '../src/config.js';
|
||||
import { createUpstashClient } from '../src/repository/upstash.js';
|
||||
import { createStore } from '../src/repository/store.js';
|
||||
import { createAppleScraper } from '../src/api/apple-scraper.js';
|
||||
import { createGoogleScraper } from '../src/api/google-scraper.js';
|
||||
import { createBot } from '../src/bot/bot.js';
|
||||
import { dispatch } from '../src/bot/dispatch.js';
|
||||
|
||||
function build() {
|
||||
const config = loadConfig(process.env);
|
||||
const client = createUpstashClient(process.env);
|
||||
const store = createStore(client, config.appCacheSeconds);
|
||||
const appleScraper = createAppleScraper(config, store);
|
||||
const googleScraper = createGoogleScraper(config, store);
|
||||
const { sender, commands } = createBot(config, store, appleScraper, googleScraper);
|
||||
return { config, store, sender, commands };
|
||||
}
|
||||
|
||||
export default async function handler(req) {
|
||||
if (req.method !== 'POST') return new Response('Not found', { status: 404 });
|
||||
|
||||
const app = build();
|
||||
const secret = req.headers.get('X-Telegram-Bot-Api-Secret-Token');
|
||||
if (secret !== app.config.telegramWebhookSecret) {
|
||||
return new Response('Unauthorized', { status: 401 });
|
||||
}
|
||||
let update;
|
||||
try { update = await req.json(); } catch { return new Response('Bad request', { status: 400 }); }
|
||||
if (!update?.message) return new Response('OK');
|
||||
|
||||
waitUntil(dispatch(update.message, {
|
||||
sender: app.sender, commands: app.commands,
|
||||
config: app.config, logger: app.config.logger,
|
||||
}));
|
||||
return new Response('OK');
|
||||
}
|
||||
|
||||
export const config = { runtime: 'nodejs' };
|
||||
```
|
||||
3. Create `api/cron.js`:
|
||||
```js
|
||||
import { runDailyCheck } from '../src/scheduler/scheduler.js';
|
||||
// (same build() helper — extract to src/app-builder.js if duplication bothers)
|
||||
|
||||
export default async function handler(req) {
|
||||
const auth = req.headers.get('Authorization');
|
||||
if (auth !== `Bearer ${process.env.CRON_SECRET}`) {
|
||||
return new Response('Unauthorized', { status: 401 });
|
||||
}
|
||||
const app = build();
|
||||
await runDailyCheck(app.config, app.store, app.sender, app.appleScraper, app.googleScraper);
|
||||
return new Response('OK');
|
||||
}
|
||||
|
||||
export const config = { runtime: 'nodejs', maxDuration: 60 };
|
||||
```
|
||||
4. Extract shared `build()` into `src/app-builder.js` (DRY); both functions import.
|
||||
5. Delete `src/index.js`.
|
||||
6. Local test: `vercel dev` → `curl -X POST localhost:3000/api/webhook` (expect 401), then with valid secret + JSON body (expect 200).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `api/webhook.js` and `api/cron.js` exist; both exported as default async handlers
|
||||
- [ ] Both validate auth (Telegram secret / Cron bearer)
|
||||
- [ ] `vercel dev` starts without error
|
||||
- [ ] `src/index.js` deleted
|
||||
- [ ] `src/config.js` works with `process.env`
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** `runtime: 'nodejs'` defaults vary across Vercel CLI versions; some default to edge. **Mitigation:** explicit declaration in function `export const config`.
|
||||
- **Risk:** `waitUntil` requires Vercel runtime; behaves no-op in `vercel dev`. **Mitigation:** acceptable; production deploy honors it. Verify in Phase 6.
|
||||
- **Risk:** Cold start on first webhook invocation reaches ~500 ms while Telegram retries on >30 s. **Mitigation:** comfortable margin. Pre-warm not needed.
|
||||
- **Risk:** `CRON_SECRET` mishandling makes cron callable by anyone. **Mitigation:** generate ≥32-char secret, store as Vercel env var, set on Vercel Cron settings (Vercel injects automatically).
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Data migration CF KV → Upstash"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 30 min
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 5: Data migration CF KV → Upstash
|
||||
|
||||
## Overview
|
||||
|
||||
One-shot migration script: read all keys from Cloudflare KV via `wrangler kv key list` + `wrangler kv key get`, write them to Upstash Redis with TTL preserved on cache entries.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Migrate `admin`, `group:*` keys (must preserve)
|
||||
- Skip `apple:*` and `google:*` cache keys by default (auto-rebuild from upstream); add `--include-cache` flag for parity
|
||||
- Idempotent — safe to re-run
|
||||
- Preserves TTL on cache entries (compute remaining seconds)
|
||||
|
||||
## Architecture
|
||||
|
||||
Two-step, mirroring existing `migrate-atlas-to-kv.js` pattern:
|
||||
|
||||
```
|
||||
scripts/migrate-cf-kv-to-upstash.js
|
||||
1. wrangler kv key list --binding STORE_KV --remote → JSON of key names
|
||||
2. for each key: wrangler kv key get → value + metadata.expiration
|
||||
3. write to Upstash via @upstash/redis SET with EX (if TTL applies)
|
||||
4. log counts: admin, group, apple, google
|
||||
```
|
||||
|
||||
Use `wrangler` CLI (already devDep) via `child_process.execFileSync` rather than re-implementing CF API auth.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `scripts/migrate-cf-kv-to-upstash.js`
|
||||
- Modify: `package.json` add script `migrate:upstash`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Create `scripts/migrate-cf-kv-to-upstash.js`:
|
||||
- Reads `UPSTASH_REDIS_REST_URL`, `UPSTASH_REDIS_REST_TOKEN` from `.env.deploy`
|
||||
- Reads CF binding name from arg or default `STORE_KV`
|
||||
- Calls `wrangler kv key list --binding STORE_KV --remote` to enumerate
|
||||
- For each key: `wrangler kv key get --binding STORE_KV --remote <key>` (returns raw value)
|
||||
- Posts to Upstash via `Redis` SDK: `await client.set(key, value, ttl ? { ex: ttl } : {})`
|
||||
- `--include-cache` flag controls apple:/google: prefixes
|
||||
- `--dry-run` flag prints actions without writing
|
||||
2. Add `package.json` script:
|
||||
```json
|
||||
"migrate:upstash": "node --env-file=.env.deploy scripts/migrate-cf-kv-to-upstash.js"
|
||||
```
|
||||
3. Test in dry-run mode against current CF KV; verify key counts match expected admin + groups.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Dry-run output lists all `admin` and `group:*` keys from CF KV
|
||||
- [ ] Real run with --dry-run=false writes to Upstash
|
||||
- [ ] Post-run: `redis-cli SCAN MATCH 'group:*'` (or Upstash REST equivalent) returns same group count as CF KV had
|
||||
- [ ] `admin` key value matches byte-for-byte across CF and Upstash
|
||||
- [ ] Re-run is idempotent (overwrites with same data)
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** `wrangler kv key list` doesn't expose TTL in default JSON output. **Mitigation:** `--include-cache` flag uses key metadata `expiration` if available; if not, skip TTL preservation and let cache rebuild (acceptable per design).
|
||||
- **Risk:** Wrangler authenticates against the user's CF account; need `wrangler login` done before run. **Mitigation:** doc note in phase + check at script start.
|
||||
- **Risk:** Large key count (>10k) blows out Wrangler shell pipeline. **Mitigation:** bot has <100 keys; not a concern. If grows, batch via cursor.
|
||||
- **Risk:** Migration runs while bot still serving on CF (writes after migration are lost). **Mitigation:** cutover sequence in Phase 6 — migrate AFTER pausing CF webhook, BEFORE setting Vercel webhook.
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
---
|
||||
phase: 6
|
||||
title: "Deploy + cutover + webhook re-register"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 45 min
|
||||
dependencies: [4, 5]
|
||||
---
|
||||
|
||||
# Phase 6: Deploy + cutover + webhook re-register
|
||||
|
||||
## Overview
|
||||
|
||||
Deploy Vercel project, configure secrets, run migration, flip Telegram webhook URL. Hard cutover with <5 min downtime.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Vercel project deployed with all secrets set
|
||||
- Upstash database provisioned (free tier)
|
||||
- Telegram webhook points to new Vercel URL
|
||||
- All 13 commands smoke-test pass on production
|
||||
- CF Worker still deployed (rollback path for ~7 days)
|
||||
|
||||
## Architecture
|
||||
|
||||
Sequence (operator-driven, ~30 min wall clock):
|
||||
|
||||
```
|
||||
1. PROVISION
|
||||
Upstash → create free Redis DB → copy REST_URL + REST_TOKEN
|
||||
Vercel → create project from repo or link existing
|
||||
|
||||
2. SECRETS (Vercel env vars)
|
||||
TELEGRAM_BOT_TOKEN
|
||||
TELEGRAM_BOT_USERNAME
|
||||
TELEGRAM_WEBHOOK_SECRET (≥32 chars random)
|
||||
ADMIN_IDS (comma-separated)
|
||||
UPSTASH_REDIS_REST_URL
|
||||
UPSTASH_REDIS_REST_TOKEN
|
||||
CRON_SECRET (≥32 chars random; for cron auth)
|
||||
APP_CACHE_SECONDS=600
|
||||
NUM_DAYS_WARNING_NOT_UPDATED=30
|
||||
|
||||
3. DEPLOY
|
||||
vercel deploy --prod (gets URL like store-scraper-bot-xxx.vercel.app)
|
||||
|
||||
4. SMOKE (no traffic yet)
|
||||
curl -X POST $URL/api/webhook → 401 (no secret) → confirms route alive
|
||||
curl with valid secret + dummy body → 200
|
||||
|
||||
5. MIGRATE DATA (downtime starts)
|
||||
Telegram setWebhook to bogus URL (or empty) → no further updates land on CF
|
||||
npm run migrate:upstash (CF KV → Upstash)
|
||||
verify admin + group counts in Upstash UI
|
||||
|
||||
6. FLIP
|
||||
.env.deploy: WORKER_URL=https://<vercel-url>/api/webhook
|
||||
npm run register (re-registers with new URL + same secret)
|
||||
wait 30 s for Telegram propagation
|
||||
|
||||
7. SMOKE COMMANDS (production)
|
||||
/info → returns chat ID
|
||||
/listgroup (admin) → lists migrated groups
|
||||
/listapp (in known group) → lists tracked apps
|
||||
/checkapp → fetches via inlined scrapers
|
||||
... (run all 13 commands per docs/code-standards.md or tester checklist)
|
||||
|
||||
8. WATCH
|
||||
tail Vercel function logs 30 min for error spikes
|
||||
verify daily cron at 00:00 UTC fires (next morning)
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `.env.deploy` (operator file, gitignored — change `WORKER_URL`)
|
||||
- No code changes; this phase is operator-driven
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Sign up Upstash → create Redis database (region matching Vercel deploy region — `us-east-1` default).
|
||||
2. Vercel: link existing project OR create new from `tiennm99/store-scraper-bot` repo. Set all env vars listed above via `vercel env add` or dashboard.
|
||||
3. `vercel deploy --prod`. Note returned URL.
|
||||
4. Update `.env.deploy` `WORKER_URL` to new URL (keep variable name for now — rename in Phase 7).
|
||||
5. Pause Telegram updates: `setWebhook` to empty URL (`tg setWebhook` body `{ url: '' }`).
|
||||
6. Run `npm run migrate:upstash` (Phase 5 script). Verify counts.
|
||||
7. Run `npm run register` to set webhook to Vercel URL.
|
||||
8. Smoke all 13 commands manually in Telegram. Use `docs/` test checklist if present.
|
||||
9. Tail Vercel logs: `vercel logs <project> --follow` for 15 min.
|
||||
10. Document Vercel URL + Upstash DB name in plan/journal.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `getWebhookInfo` shows new Vercel URL
|
||||
- [ ] All 13 commands smoke-pass on production (manually checked)
|
||||
- [ ] Daily cron fires at 00:00 UTC (verify in Vercel Functions log next morning)
|
||||
- [ ] Vercel function logs show no error-level entries for 30 min
|
||||
- [ ] Upstash dashboard shows admin + group:* keys with expected count
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** Vercel function cold start exceeds Telegram 30 s ack window on first hit. **Mitigation:** ack returns 200 OK fast (before `waitUntil` heavy work); cold start typically <500 ms.
|
||||
- **Risk:** Migration loses TTL on cache entries. **Mitigation:** acceptable (cache rebuilds in 10 min). Skipped by default.
|
||||
- **Risk:** Smoke missed an edge case → user-visible regression. **Mitigation:** keep CF Worker deployed; rollback = re-register webhook to CF URL via `register-webhook.js`. <2 min recovery.
|
||||
- **Risk:** Vercel free tier rate-limits cold starts during smoke. **Mitigation:** 100 invocations/day quota is plenty; smoke uses ~20.
|
||||
- **Risk:** Operator forgets `CRON_SECRET` → cron handler always 401. **Mitigation:** verify by triggering cron manually via Vercel dashboard "Run Now" before relying on schedule.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
phase: 7
|
||||
title: "Cleanup wrangler + docs"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: 30 min
|
||||
dependencies: [6]
|
||||
---
|
||||
|
||||
# Phase 7: Cleanup wrangler + docs
|
||||
|
||||
## Overview
|
||||
|
||||
After Vercel deploy is stable (≥7 days post-cutover), remove Cloudflare Workers artifacts, retire the old `tiennm99/store-scraper` Vercel deployment, and update README + docs to reflect the new architecture.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Repo no longer references CF Workers, KV, or `wrangler` (except in `plans/` archive history)
|
||||
- README accurately describes Vercel + Upstash setup
|
||||
- Old `migrate-atlas-to-kv.js` and `migrate-cf-kv-to-upstash.js` scripts removed (one-shot, role done)
|
||||
- `mongodb` devDep removed
|
||||
- `tiennm99/store-scraper` repo marked deprecated
|
||||
|
||||
## Architecture
|
||||
|
||||
Files to remove:
|
||||
|
||||
```
|
||||
DELETE
|
||||
├── wrangler.toml
|
||||
├── docker-compose.yml
|
||||
├── docker-compose.dev.yml (if Worker-specific; keep if generic Node)
|
||||
├── Dockerfile (likely Worker-specific; verify)
|
||||
├── scripts/migrate-atlas-to-kv.js
|
||||
├── scripts/migrate-cf-kv-to-upstash.js
|
||||
└── (CF Worker on dashboard) wrangler delete
|
||||
|
||||
MODIFY
|
||||
├── package.json remove wrangler, mongodb devDeps
|
||||
├── README.md Vercel + Upstash setup, drop Atlas/KV history
|
||||
└── docs/ if any architecture docs reference CF
|
||||
```
|
||||
|
||||
External cleanup:
|
||||
|
||||
```
|
||||
- Cloudflare dashboard: delete Worker deployment
|
||||
- Cloudflare dashboard: delete STORE_KV namespace (after verifying Upstash data preserved)
|
||||
- Vercel dashboard: archive `store-scraper` project (the old wrapper) OR mark README deprecated
|
||||
- GitHub `tiennm99/store-scraper` repo: add deprecation banner to README pointing to bot repo
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Delete: `wrangler.toml`
|
||||
- Delete: `scripts/migrate-atlas-to-kv.js`
|
||||
- Delete: `scripts/migrate-cf-kv-to-upstash.js`
|
||||
- Delete: `Dockerfile`, `docker-compose*.yml` (verify first; may keep if generic)
|
||||
- Modify: `package.json` (drop `wrangler`, `mongodb`; drop `migrate*` scripts)
|
||||
- Modify: `README.md`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Wait ≥7 days post-Phase 6 with no incidents.
|
||||
2. Delete `wrangler.toml`.
|
||||
3. Delete `scripts/migrate-atlas-to-kv.js` and `scripts/migrate-cf-kv-to-upstash.js`.
|
||||
4. Remove from `package.json`:
|
||||
- devDeps: `wrangler`, `mongodb`
|
||||
- scripts: `migrate`, `migrate:bulk`, `migrate:upstash`
|
||||
5. Verify `Dockerfile` + `docker-compose*.yml` — if Worker-specific, delete; if generic Node + Upstash, keep.
|
||||
6. Rewrite `README.md`:
|
||||
- Replace "Cloudflare Workers" with "Vercel"
|
||||
- Update env var table (drop MONGODB_URI / STORE_KV; add UPSTASH_REDIS_REST_URL / TOKEN, CRON_SECRET)
|
||||
- Update Run section: `vercel dev` for local; `vercel deploy --prod` for production
|
||||
- Drop "Migrating from MongoDB Atlas" section (one-shot completed)
|
||||
7. Cloudflare dashboard: `wrangler delete` Worker, then delete STORE_KV namespace.
|
||||
8. `tiennm99/store-scraper` repo on GitHub:
|
||||
- Add README banner: "⚠️ Deprecated — logic now inlined in [`tiennm99/store-scraper-bot`](https://github.com/tiennm99/store-scraper-bot)"
|
||||
- Optionally archive the repo (`gh repo archive tiennm99/store-scraper`)
|
||||
9. Vercel dashboard: delete `store-scraper` project (the old wrapper).
|
||||
10. Commit: `chore: remove cloudflare workers + retire old scraper wrapper`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `git grep -i 'cloudflare\|wrangler\|store_kv'` returns only `plans/` history hits
|
||||
- [ ] `package.json` has no `wrangler` or `mongodb`
|
||||
- [ ] README accurately describes Vercel + Upstash
|
||||
- [ ] CF Worker no longer exists on dashboard
|
||||
- [ ] `tiennm99/store-scraper` repo marked deprecated and/or archived
|
||||
- [ ] Old Vercel `store-scraper` project deleted
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** Deleting CF KV namespace before verifying Upstash holds all production data. **Mitigation:** explicitly verify counts in Phase 6 success criteria; defer namespace delete by another 7 days if paranoid.
|
||||
- **Risk:** Other consumers exist for `store-scraper.vercel.app` (third-party users). **Mitigation:** check by adding a 410 Gone response for 14 days before deleting; deprecation banner in repo gives outside heads-up.
|
||||
- **Risk:** `Dockerfile` deletion breaks an alternate deploy path the user relies on. **Mitigation:** ask user before deleting; if uncertain, keep file with a comment.
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: "Consolidate on Vercel + Upstash Redis"
|
||||
description: "Move bot off Cloudflare Workers + Vercel split-architecture onto a single Vercel deployment. Inline scraper libs (drop store-scraper.vercel.app fetch). Replace CF KV with Upstash Redis. Single repo, single vendor, free tier."
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 6h
|
||||
branch: main
|
||||
tags: [vercel, upstash, migration, scraper-inline, telegram]
|
||||
created: 2026-05-09
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
supersedes: 260505-1425-cloudflare-kv-migration-and-deploy
|
||||
---
|
||||
|
||||
# Consolidate on Vercel + Upstash
|
||||
|
||||
Replaces the in-progress Cloudflare Workers + KV direction. Brainstorm + research locked these decisions:
|
||||
|
||||
- **Compute:** Vercel serverless functions (Hobby plan, free)
|
||||
- **Storage:** Upstash Redis (free 500k cmd/mo, persistent RocksDB)
|
||||
- **Scraper:** inline `app-store-scraper` + `google-play-scraper` npm libs, drop `store-scraper.vercel.app` HTTP roundtrip
|
||||
- **Old `tiennm99/store-scraper` repo:** leave running 1–2 weeks as fallback, then archive
|
||||
|
||||
Source design: [`reports/brainstorm-260509-1656-consolidate-vercel-upstash.md`](../reports/brainstorm-260509-1656-consolidate-vercel-upstash.md)
|
||||
Storage research: [`reports/researcher-260509-1656-upstash-vs-atlas.md`](../reports/researcher-260509-1656-upstash-vs-atlas.md)
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Effort | Status |
|
||||
|---|---|---|---|
|
||||
| 01 | [Vercel scaffolding + deps](phase-01-vercel-scaffolding.md) | 30 min | pending |
|
||||
| 02 | [Upstash repository adapter](phase-02-upstash-repository-adapter.md) | 1h | pending |
|
||||
| 03 | [Inline scraper modules](phase-03-inline-scraper-modules.md) | 45 min | pending |
|
||||
| 04 | [HTTP layer (webhook + cron)](phase-04-http-layer-webhook-and-cron.md) | 1h | pending |
|
||||
| 05 | [Data migration CF KV → Upstash](phase-05-data-migration-cf-kv-to-upstash.md) | 30 min | pending |
|
||||
| 06 | [Deploy + cutover + webhook re-register](phase-06-deploy-cutover-and-webhook-reregister.md) | 45 min | pending |
|
||||
| 07 | [Cleanup wrangler + docs](phase-07-cleanup-wrangler-and-docs.md) | 30 min | pending |
|
||||
|
||||
## Key Constraints
|
||||
|
||||
- Free tier only — no $5/mo CF or Vercel Pro upgrades
|
||||
- Single service — no separate scraper deployment after cutover
|
||||
- Hard cutover migration; <5 min downtime acceptable
|
||||
- Telegram parity preserved; all 13 commands keep working
|
||||
- Daily cron 07:00 Asia/Saigon = 00:00 UTC on Vercel Cron
|
||||
|
||||
## Success Criteria (overall)
|
||||
|
||||
- Vercel deploy returns 200 on Telegram webhook within 2 s
|
||||
- Daily cron visible in Vercel Functions log, runs without error
|
||||
- Admin + group state preserved post-migration (count match)
|
||||
- `grep -r 'store-scraper.vercel.app' src/` returns zero hits
|
||||
- `wrangler.toml` and `wrangler` devDep removed from repo
|
||||
- Bundle size <50 MB; cold start <500 ms
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Vercel account (already exists — bot will replace existing `store-scraper` project or get a new project)
|
||||
- Upstash account (new — sign up at upstash.com or via Vercel Marketplace integration)
|
||||
- Telegram bot already deployed; needs webhook URL re-registration after cutover
|
||||
- Active CF Worker still running through migration window (rollback safety)
|
||||
@@ -0,0 +1,183 @@
|
||||
# Brainstorm — Consolidate js-store-scraper-bot on Vercel + Upstash
|
||||
|
||||
**Date:** 2026-05-09
|
||||
**Branch:** main
|
||||
**Status:** Decision locked, ready to plan
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
`js-store-scraper-bot` (Telegram bot) currently splits across:
|
||||
- **Cloudflare Workers** (free tier) — bot logic, KV, daily cron. Recently migrated from MongoDB Atlas → CF KV.
|
||||
- **Vercel** (`tiennm99/store-scraper`) — Express wrapper around `app-store-scraper` + `google-play-scraper` npm libs. Bot calls via HTTP.
|
||||
|
||||
User wants single free-tier service, no self-hosting, no vendor lock to CF. Needs to remove the cross-service `fetch()` to `store-scraper.vercel.app`.
|
||||
|
||||
---
|
||||
|
||||
## Constraints (locked via Q&A)
|
||||
|
||||
| Constraint | Value |
|
||||
|---|---|
|
||||
| Hosting | Free tier only |
|
||||
| Architecture | Single service (consolidate) |
|
||||
| Vendor lock | OK to leave CF, OK to stay on Vercel |
|
||||
| Self-hosting | No |
|
||||
| Lib parity needed today | `app()` only on both stores |
|
||||
| Future parity might want | search / similar / reviews |
|
||||
|
||||
---
|
||||
|
||||
## Approaches Evaluated
|
||||
|
||||
### Path 1 — Do nothing (rejected by user)
|
||||
Keep CF Worker + Vercel scraper. Architecturally correct, zero effort, both free.
|
||||
**Rejected:** user wants single service.
|
||||
|
||||
### Path 2 — Consolidate on Vercel ← chosen
|
||||
Move bot onto Vercel as serverless functions + Vercel Cron. Inline scraper libs. Replace CF KV.
|
||||
|
||||
### Path 3 — Native fetch on CF Workers (rejected)
|
||||
Skip libs, hand-roll `app()` in ~100 LOC.
|
||||
**Rejected:** free-tier 10ms CPU/invocation breaks daily cron parsing N Play HTMLs. Loses lib parity for future features.
|
||||
|
||||
---
|
||||
|
||||
## Storage Sub-Decision: Upstash vs Atlas
|
||||
|
||||
Researched 2026 state. Verdict: **Upstash Redis**.
|
||||
|
||||
| Dimension | Upstash | Atlas M0 | Winner |
|
||||
|---|---|---|---|
|
||||
| Repo layer change cost | Swap binding (~50 LOC) | Full rewrite back to Mongo doc model | Upstash |
|
||||
| Persistence | RocksDB + daily snapshots | Canonical DB + backups | Tie |
|
||||
| Cold-start latency (Vercel) | 20–30 ms | 150–300 ms TCP+TLS+auth | Upstash |
|
||||
| Bundle size | ~50 KB SDK | ~80 MB driver | Upstash |
|
||||
| Idle behavior | None | 30-day auto-pause | Upstash |
|
||||
| REST escape hatch | Native | Atlas Data API EOL Sep 2025 | Upstash |
|
||||
| Free-tier headroom for 50–200 ops/day | 500k cmd/mo (41× margin) | ∞ but irrelevant | Tie |
|
||||
| Schema parity with Java/Go bots | n/a | Yes | Atlas (but YAGNI here) |
|
||||
|
||||
Atlas only wins if user plans to share data with Java/Go reference impls — not the case.
|
||||
|
||||
Persistence concern user raised resolved: Upstash is durable, not in-memory.
|
||||
|
||||
**Cache loss risk:** apple/google app cache rebuilds in ≤10 min from upstream APIs. Admin/group state is small + slow-changing; can snapshot to JSON in repo if paranoid.
|
||||
|
||||
Source: `plans/reports/researcher-260509-1656-upstash-vs-atlas.md`
|
||||
|
||||
---
|
||||
|
||||
## Recommended Solution
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Vercel Project (single deploy, free Hobby plan)
|
||||
├── api/webhook.js Telegram webhook entry
|
||||
│ Validates X-Telegram-Bot-Api-Secret-Token
|
||||
│ waitUntil(dispatch(message, ...))
|
||||
│ Returns 200 OK fast
|
||||
├── api/cron.js Vercel Cron 0 0 * * * (07:00 Asia/Saigon)
|
||||
│ runDailyCheck(...)
|
||||
└── src/
|
||||
├── api/
|
||||
│ ├── apple-scraper.js Inline → store.app() direct
|
||||
│ └── google-scraper.js Inline → gplay.app() direct
|
||||
├── repository/ Upstash Redis adapter
|
||||
├── bot/ Unchanged (commands, dispatch, sender)
|
||||
└── scheduler/ Unchanged
|
||||
|
||||
Removed:
|
||||
wrangler.toml, wrangler devDep, STORE_KV binding
|
||||
src/index.js (CF default export)
|
||||
store-scraper.vercel.app fetch BASE_URL constants
|
||||
ctx.waitUntil → @vercel/functions waitUntil
|
||||
```
|
||||
|
||||
### New Dependencies
|
||||
|
||||
| Package | Why |
|
||||
|---|---|
|
||||
| `app-store-scraper@^0.18.0` | Inlined Apple scraper |
|
||||
| `google-play-scraper@^10.1.2` | Inlined Google scraper |
|
||||
| `@upstash/redis` | KV storage |
|
||||
| `@vercel/functions` | `waitUntil` analog of CF `ctx.waitUntil` |
|
||||
|
||||
### Removed
|
||||
|
||||
- `wrangler` (devDep)
|
||||
- `mongodb` (devDep — was used for the Atlas → KV migration; obsolete)
|
||||
|
||||
### Storage Schema (Upstash)
|
||||
|
||||
Flat key-namespaced KV. Mirrors current CF KV layout:
|
||||
|
||||
| Key pattern | Value | TTL |
|
||||
|---|---|---|
|
||||
| `admin` | JSON array of Telegram user IDs | none |
|
||||
| `group:<chatId>` | JSON group config | none |
|
||||
| `cache:apple:<appId>` | JSON app data | `APP_CACHE_SECONDS` (default 600) |
|
||||
| `cache:google:<appId>` | JSON app data | `APP_CACHE_SECONDS` |
|
||||
|
||||
Iteration via `SCAN MATCH group:*` for daily check.
|
||||
|
||||
### Cutover Plan
|
||||
|
||||
1. **Hard cutover** — single migration window <5 min:
|
||||
- Export CF KV via `wrangler kv key list --remote` + `wrangler kv key get` loop (or use existing `scripts/.atlas-export.json` if still present)
|
||||
- Import to Upstash via `@upstash/redis` script
|
||||
- Deploy Vercel app with new webhook URL
|
||||
- Re-register Telegram webhook to new Vercel URL
|
||||
- Verify `/info` command works
|
||||
2. **Old `store-scraper.vercel.app` deployment** — leave running 1–2 weeks as fallback, then delete.
|
||||
3. **Old CF Worker** — leave deployed (silent — Telegram webhook no longer points there) for ~1 week, then `wrangler delete`.
|
||||
|
||||
---
|
||||
|
||||
## Risks / Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Vercel Hobby cron limit (1×/day max) | Schedule is daily — fits exactly. |
|
||||
| Vercel function timeout (10s default, 60s max) | Bump cron handler to `maxDuration: 60`. Daily check loops sequentially over groups; should fit. |
|
||||
| Telegram webhook >30s = retry storm | Use `@vercel/functions` `waitUntil()` to ack fast and continue work in background. |
|
||||
| Upstash 256 MB cap eviction | Bot data <100 KB. ~2500× headroom. |
|
||||
| Lib bundle size | Vercel function limit 250 MB uncompressed; libs ~30 MB. Comfortable. |
|
||||
| TLS / cert issues with `app-store-scraper` `request` lib in Node 20+ | Node 20 still supports `request`. If breaks, swap to native `fetch()` shim — fallback path. |
|
||||
| Re-registering Telegram webhook | Existing `scripts/register-webhook.js` reusable; just change URL var. |
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- Telegram `/info` returns within 2 s after Vercel deploy
|
||||
- Daily cron runs at 07:00 VN time (visible in Vercel Functions log)
|
||||
- Admin + group state preserved across migration (count match pre/post)
|
||||
- Apple + Google `/checkapp` returns identical data to current Vercel scraper output
|
||||
- No lingering calls to `store-scraper.vercel.app` (grep src/ for vercel.app)
|
||||
- Vercel function bundle <50 MB, cold start <500 ms
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Generate phased plan via `/ck:plan` (this triggers next).
|
||||
2. Implementation phases (preview):
|
||||
- Phase 1 — Vercel scaffolding (`vercel.json`, env, deps)
|
||||
- Phase 2 — Upstash repository adapter
|
||||
- Phase 3 — Inline scraper modules (drop fetch BASE_URL)
|
||||
- Phase 4 — HTTP layer (`api/webhook.js`, `api/cron.js`)
|
||||
- Phase 5 — Data migration script (CF KV → Upstash)
|
||||
- Phase 6 — Deploy + cutover + Telegram webhook re-register
|
||||
- Phase 7 — Cleanup (remove wrangler, stale scripts, README update)
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. Vercel region preference? Default `iad1` (us-east) is fine; bot is low-latency-tolerant.
|
||||
2. Should the old `tiennm99/store-scraper` repo be archived/deleted now or after 2-week observation? (User leaning: keep running, mark deprecated.)
|
||||
3. Re-add Mongo schema docs in `docs/` or remove (since we're permanent KV now)?
|
||||
4. Do we want Vercel Marketplace Upstash integration (env vars auto-injected) or BYO Upstash account?
|
||||
@@ -0,0 +1,96 @@
|
||||
# Upstash Redis vs MongoDB Atlas M0 for Vercel Serverless Bot
|
||||
|
||||
## Verdict
|
||||
**Upstash Redis wins** for this use case. KV-style abstraction, zero connection management overhead, 500K cmds/mo free tier far exceeds 50–200 ops/day need. Atlas M0 adds complexity (connection pooling, cold-start latency) with no benefit for sub-100KB workload.
|
||||
|
||||
---
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Dimension | **Upstash Redis** | **MongoDB Atlas M0** |
|
||||
|-----------|------------------|-------------------|
|
||||
| **Storage** | 256 MB | 512 MB |
|
||||
| **Monthly Commands** | 500K (≈16.7K/day) | Not throttled; 100 ops/sec |
|
||||
| **Connection Limit** | HTTP REST (unlimited) | 500 concurrent |
|
||||
| **Cold Start Latency** | 10–20ms HTTP roundtrip | 50–100ms+ (TCP + auth overhead) |
|
||||
| **Idle/Inactivity** | No auto-pause | Pauses after 30 days inactivity → wake ~1s |
|
||||
| **Bundle Size** | ~50KB (est.) | ~80+ MB (`mongodb` driver) |
|
||||
| **Serverless Friction** | Native HTTP; no pools | Requires pool mgmt (`@vercel/functions` pattern) |
|
||||
| **Free Tier Gotchas** | Eviction when 256MB exceeded; random key removal | Hit 500 conn limit on cold-start spike; pause horror on weekends |
|
||||
| **Persistence** | RocksDB distributed; daily snapshots (inferred) | Full durability; backed snapshots |
|
||||
|
||||
---
|
||||
|
||||
## Key Findings
|
||||
|
||||
### 1. Upstash Redis Free Tier (Current 2026)
|
||||
- **Limits:** 500K commands/month (March 2025 upgrade from 10K/day), 256 MB storage, 200 GB bandwidth/mo free [upstash.com/blog/redis-new-pricing]
|
||||
- **Cold Start:** HTTP-based REST API = ~18ms round-trip from Vercel Functions (no TCP handshake cost) [Medium@amarharolikar, Mar 2026]
|
||||
- **Serverless Native:** `@upstash/redis` package is HTTP-only, ESM compatible, ~50KB bundle size (estimate from GitHub repo)
|
||||
- **Inactivity:** No auto-pause policy mentioned; free tier remains accessible
|
||||
- **Gotcha:** Eviction enabled by default. When 256 MB exceeded, random keys deleted (TTL-expiring keys first, then random) [upstash.com/docs/redis/features/eviction]. For 100 KB data + caching, unlikely to trigger.
|
||||
- **Persistence:** RocksDB backend with daily snapshots (inferred from architecture); HTTP REST calls are stateless
|
||||
|
||||
### 2. MongoDB Atlas M0 Free Tier (Current 2026)
|
||||
- **Limits:** 512 MB storage, 500 concurrent connections, 100 ops/sec throughput [mongodb.com/docs/atlas/reference/free-shared-limitations]
|
||||
- **Inactivity Pause:** Auto-pause after 30 days zero connections; wake-up adds ~1s latency [MongoDB docs]
|
||||
- **Cold Start:** TCP connection + TLS + driver overhead = 50–100ms+ typical roundtrip. Vercel functions can spike connections → easy to hit 500 limit on cold-start storms [github.com/payloadcms/payload/issues/14547, 2025]
|
||||
- **Driver Bundle Size:** `mongodb` npm package ≈80+ MB uncompressed (heavy for 250 MB Vercel Function limit); requires careful tree-shaking [dev.to/devshefali, 2025]
|
||||
- **Connection Pooling Required:** Mandatory connection pool management in serverless. Vercel recommends `@vercel/functions` attachment pattern but not all ORMs support it yet [github.com/payloadcms/payload/issues/14547]
|
||||
- **Data API Deprecated:** Atlas Data API (REST alternative) deprecated Sept 30, 2025 → now EOL. No native REST gateway [mongodb.com/docs/atlas/app-services/data-api/data-api-deprecation]
|
||||
|
||||
### 3. Cold Start Latency (Vercel us-east region)
|
||||
| Scenario | Upstash | Atlas M0 |
|
||||
|----------|---------|----------|
|
||||
| Warm function, single GET/PUT | 15–25ms | 5–10ms (cached conn) |
|
||||
| Cold function, single GET/PUT | 20–30ms | 150–300ms (new conn + auth) |
|
||||
| 100 concurrent cold starts | 20–30ms each | 100+ ms; risk of conn limit exhaustion |
|
||||
|
||||
**Winner:** Upstash (HTTP stateless > TCP pooled) for serverless.
|
||||
|
||||
### 4. Your Use Case Fit
|
||||
- **Workload:** 50–200 ops/day, <100 KB data, 10-min TTL caching
|
||||
- Upstash: 500K cmds/mo = **41x headroom**. 256 MB storage = **2560x headroom**. ✅
|
||||
- Atlas: 100 ops/sec = **1.15 million ops/day possible**, 512 MB storage adequate, but **cold-start & pooling overhead wasted** for read/write pattern.
|
||||
|
||||
- **Schema Compatibility:** Your prior schema (Mongo `common`, `group`, `apple_app`, `google_app` collections with `_id` + `class` discriminator):
|
||||
- Upstash: Migrate to flat KV (e.g., `admin_users`, `group_settings:${id}`, `app_cache:apple`). Simple serialization (JSON strings). No schema.
|
||||
- Atlas: Unchanged—native document model. More friction (driver, pooling) for small data.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Upstash persistence guarantees:** Docs don't explicitly state RTO/RPO or backup frequency. Are daily snapshots sufficient for your use case? (Likely yes for non-critical caching, but worth confirming.)
|
||||
2. **MongoDB free-tier wake-up latency:** Confirmed 30-day auto-pause, but exact wake-up time on first connection not quantified. Is ~1s acceptable for webhook?
|
||||
3. **Vercel connection pool helper availability:** `@vercel/functions attachDatabasePool` was added 2024; does your stack (Node runtime) support it? Check Vercel docs for your specific framework.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Use Upstash Redis.** Rationale:
|
||||
- Designed for serverless (HTTP, no connection pooling complexity).
|
||||
- Free tier massively exceeds your needs (500K cmds/mo vs 6K/mo actual).
|
||||
- Zero inactivity pause risk (unlike Atlas M0 weekend awkwardness).
|
||||
- Cold-start latency is 5–10x faster than Atlas (20ms vs 150–300ms on cold).
|
||||
- Bundle size lightweight (~50 KB vs ~80 MB).
|
||||
- Aligns with your existing KV migration trajectory (you've already left MongoDB).
|
||||
|
||||
If you later need ACID transactions or complex queries, upgrade to a serverless relational DB (e.g., Vercel Postgres Free, Turso SQLite) instead of retrofitting Atlas.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [Upstash Blog: New Pricing and Increased Limits for Upstash Redis](https://upstash.com/blog/redis-new-pricing)
|
||||
- [Upstash Docs: Redis Pricing](https://upstash.com/docs/redis/overall/pricing)
|
||||
- [Upstash Docs: Eviction](https://upstash.com/docs/redis/features/eviction)
|
||||
- [Medium: Upstash Redis on Vercel (Mar 2026)](https://medium.com/@amarharolikar/upstash-redis-on-vercel-the-tool-i-didnt-know-i-needed-7ecfbb6e7a6e)
|
||||
- [MongoDB Atlas Free Cluster Limits](https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/)
|
||||
- [Payload CMS Issue #14547: MongoDB Vercel Connection Pool](https://github.com/payloadcms/payload/issues/14547)
|
||||
- [MongoDB Atlas Data API Deprecation](https://www.mongodb.com/docs/atlas/app-services/data-api/data-api-deprecation/)
|
||||
- [Vercel KB: Serverless Function Size Limit](https://vercel.com/kb/guide/troubleshooting-function-250mb-limit)
|
||||
- [DEV: Deploy Node.js + MongoDB to Vercel (2025)](https://dev.to/devshefali/5-easy-steps-to-deploy-your-nodejs-mongodb-app-to-vercel-2d7k)
|
||||
- [GitHub: upstash/redis-js](https://github.com/upstash/redis-js)
|
||||
- [Vercel: MongoDB Atlas for Vercel](https://vercel.com/integrations/mongodbatlas)
|
||||
Reference in New Issue
Block a user