From fbf2e95a5201b2a1706fbb94b6366c0b8a96a743 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Sat, 9 May 2026 17:18:39 +0700 Subject: [PATCH] docs: lock Vercel + Upstash consolidation plan Brainstorm + research concluded: bot moves off CF Workers + Vercel split-architecture onto a single Vercel deployment with Upstash Redis storage. 7-phase implementation plan added. Supersedes the in-progress CF KV migration plan; code-level KV migration already shipped, deploy phases never executed. --- .../plan.md | 5 +- .../phase-01-vercel-scaffolding.md | 74 +++++++ .../phase-02-upstash-repository-adapter.md | 92 +++++++++ .../phase-03-inline-scraper-modules.md | 89 +++++++++ .../phase-04-http-layer-webhook-and-cron.md | 133 +++++++++++++ ...hase-05-data-migration-cf-kv-to-upstash.md | 71 +++++++ ...6-deploy-cutover-and-webhook-reregister.md | 105 ++++++++++ .../phase-07-cleanup-wrangler-and-docs.md | 96 +++++++++ .../plan.md | 61 ++++++ ...-260509-1656-consolidate-vercel-upstash.md | 183 ++++++++++++++++++ ...researcher-260509-1656-upstash-vs-atlas.md | 96 +++++++++ 11 files changed, 1004 insertions(+), 1 deletion(-) create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-cf-kv-to-upstash.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md create mode 100644 plans/260509-1656-consolidate-vercel-upstash/plan.md create mode 100644 plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md create mode 100644 plans/reports/researcher-260509-1656-upstash-vs-atlas.md diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md index e866be1..365a222 100644 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md @@ -1,7 +1,7 @@ --- title: "Cloudflare KV migration + deploy" description: "Replace MongoDB Atlas with Cloudflare KV as the storage layer, deploy to Workers, register Telegram webhook, smoke-test, and document. Supersedes the Atlas-based deploy plan." -status: pending +status: superseded priority: P1 effort: 5h branch: main @@ -9,8 +9,11 @@ tags: [cloudflare-workers, cloudflare-kv, deploy, migration] created: 2026-05-05 blockedBy: [] blocks: [260426-2327-cloudflare-deploy-and-smoke] +supersededBy: 260509-1656-consolidate-vercel-upstash --- +> **Superseded** — User chose to consolidate on Vercel + Upstash instead of completing the CF Workers deploy. Code-level KV migration shipped (commits `067d463`..`c4c6a93`); deploy phases never executed. Continuation tracked in [`260509-1656-consolidate-vercel-upstash`](../260509-1656-consolidate-vercel-upstash/plan.md). + # Cloudflare Workers + KV Storage Migration User chose KV over Atlas to pre-empt the bundle-size and cold-start risks called out in the Atlas plan. Drops the `mongodb` driver entirely (~4–5 MiB compressed) and the `nodejs_compat_v2` flag along with it. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md b/plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md new file mode 100644 index 0000000..e7439f4 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md @@ -0,0 +1,74 @@ +--- +phase: 1 +title: "Vercel scaffolding + deps" +status: pending +priority: P1 +effort: 30 min +dependencies: [] +--- + +# Phase 1: Vercel scaffolding + deps + +## Overview + +Add Vercel project config, install new runtime deps, restructure `package.json` for Node-on-Vercel. No behavior changes yet — just scaffolding. + +## Requirements + +- Vercel project recognizes the repo (Node 20 runtime, ESM) +- Daily cron registered (00:00 UTC = 07:00 Asia/Saigon) +- New deps installed; `mongodb` devDep stays (still needed by Phase 5 migration script) + +## Architecture + +``` +vercel.json cron schedule, function config (maxDuration, region) +package.json + app-store-scraper, google-play-scraper, + + @upstash/redis, @vercel/functions + scripts: dev → vercel dev, deploy → vercel deploy +.vercelignore exclude /plans, /scripts/.atlas-export.json +``` + +Vercel auto-detects `api/*.js` as serverless functions in Phase 4. This phase only sets the scaffolding. + +## Related Code Files + +- Create: `vercel.json` +- Create: `.vercelignore` +- Modify: `package.json` (deps, scripts) + +## Implementation Steps + +1. Add deps: + ```sh + npm install app-store-scraper@^0.18.0 google-play-scraper@^10.1.2 @upstash/redis @vercel/functions + ``` +2. Update `package.json` scripts: + - `dev`: `vercel dev` (was `wrangler dev`) + - `deploy`: `vercel deploy --prod && npm run register` + - Keep `migrate`, `lint`. Drop `migrate:bulk` (CF-specific). + - Add `register`: `node --env-file=.env.deploy scripts/register-webhook.js` +3. Create `vercel.json`: + ```json + { + "crons": [{ "path": "/api/cron", "schedule": "0 0 * * *" }], + "functions": { + "api/cron.js": { "maxDuration": 60 }, + "api/webhook.js": { "maxDuration": 30 } + } + } + ``` +4. Create `.vercelignore` excluding `plans/`, `scripts/.atlas-export.json`, `*.md`, `Dockerfile`, `docker-compose*.yml`, `wrangler.toml` (until Phase 7 deletes it). +5. `npm install` and verify `node_modules` has all four new deps. + +## Success Criteria + +- [ ] `package.json` shows new deps + new scripts +- [ ] `vercel.json` parses (`vercel dev --debug` doesn't error on config) +- [ ] `.vercelignore` excludes irrelevant paths +- [ ] `npm install` exits clean + +## Risk Assessment + +- **Risk:** Vercel function bundle exceeds 250 MB with all deps. **Mitigation:** check `vercel build --debug` output; tree-shaking + `excludeFiles` pattern in `vercel.json` if needed. Expected size ~30 MB. +- **Risk:** `app-store-scraper` uses deprecated `request` lib that may warn or fail under Node 20. **Mitigation:** verify in Phase 3 by importing + calling `app()`. Fallback: pin Node to 18 or swap lib (out of scope). diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md b/plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md new file mode 100644 index 0000000..1b694e1 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md @@ -0,0 +1,92 @@ +--- +phase: 2 +title: "Upstash repository adapter" +status: pending +priority: P1 +effort: 1h +dependencies: [1] +--- + +# Phase 2: Upstash repository adapter + +## Overview + +Replace `src/repository/kv.js` Cloudflare KV wrapper with an Upstash Redis equivalent. Repository files (`admin-repository.js`, `group-repository.js`, `apple-app-repository.js`, `google-app-repository.js`) keep their public APIs; only the storage primitive changes. + +## Requirements + +- Function-level API of `kv.js` preserved (`getJson`, `putJson`, `del`) +- TTL semantics preserved (Apple/Google cache uses `expirationTtl` → Redis `EX`) +- Drops `env.STORE_KV` binding; takes Upstash client instance instead +- 60s minimum TTL clamp removed (Redis `EX` accepts 1s+, but keep clamp for parity safety) + +## Architecture + +``` +src/repository/ +├── upstash.js ← NEW: client factory + getJson/putJson/del/scan +├── kv.js ← DELETE (or rename to keep history) +├── store.js ← takes upstash client instead of env +├── admin-repository.js ← unchanged signatures, internal: env→client +├── group-repository.js ← unchanged signatures +├── apple-app-repository.js ← unchanged signatures +└── google-app-repository.js← unchanged signatures +``` + +Client construction: +```js +import { Redis } from '@upstash/redis'; +export function createUpstashClient(env) { + return new Redis({ + url: env.UPSTASH_REDIS_REST_URL, + token: env.UPSTASH_REDIS_REST_TOKEN, + }); +} +``` + +API mapping: +| CF KV | Upstash Redis | Note | +|---|---|---| +| `kv.get(key, 'json')` | `redis.get(key)` | Upstash auto-deserializes JSON when value was `JSON.stringify`d on `set` | +| `kv.put(key, value)` | `redis.set(key, value)` | | +| `kv.put(key, value, { expirationTtl: N })` | `redis.set(key, value, { ex: N })` | Redis `EX` is seconds | +| `kv.delete(key)` | `redis.del(key)` | | +| `kv.list({ prefix: 'group:' })` | `redis.scan(0, { match: 'group:*' })` | Phase 5 migration uses this | + +## Related Code Files + +- Create: `src/repository/upstash.js` +- Modify: `src/repository/store.js` (accept client, not env) +- Modify: `src/repository/admin-repository.js` (use client) +- Modify: `src/repository/group-repository.js` (use client) +- Modify: `src/repository/apple-app-repository.js` (use client) +- Modify: `src/repository/google-app-repository.js` (use client) +- Delete: `src/repository/kv.js` + +## Implementation Steps + +1. Write `src/repository/upstash.js`: + - `createUpstashClient(env)` returns `Redis` instance + - `getJson(client, key)`, `putJson(client, key, value, { expirationTtl } = {})`, `del(client, key)` + - Mirror `kv.js` signatures so repository files only swap import path + first arg +2. Update `store.js` to accept `client` instead of `env`: + ```js + export function createStore(client, appCacheSeconds) { ... } + ``` +3. Update each repository to import from `./upstash.js` and take `client` instead of `env`. Search/replace `env, key` → `client, key` in each file. +4. Delete `src/repository/kv.js`. +5. Manual smoke: `node -e "import('./src/repository/upstash.js').then(m => console.log(m))"` to confirm import path resolves. + +## Success Criteria + +- [ ] `src/repository/upstash.js` exports `createUpstashClient`, `getJson`, `putJson`, `del` +- [ ] All four `*-repository.js` files import from `./upstash.js`, take `client` +- [ ] `store.js` signature: `createStore(client, appCacheSeconds)` +- [ ] `kv.js` deleted +- [ ] No `env.STORE_KV` references remain in `src/` + +## Risk Assessment + +- **Risk:** `@upstash/redis` SDK auto-stringifies/parses values inconsistently. **Mitigation:** read SDK docs; if auto-JSON disabled, mirror `kv.js`'s `JSON.stringify`/`JSON.parse` explicitly. +- **Risk:** Redis `SET key value EX 0` is invalid; current `KV_MIN_TTL_SECONDS = 60` clamp keeps us safe. Keep the clamp. +- **Risk:** Redis returns `null` on missing key (same as KV). Verify in unit smoke. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md b/plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md new file mode 100644 index 0000000..9109bb8 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md @@ -0,0 +1,89 @@ +--- +phase: 3 +title: "Inline scraper modules" +status: pending +priority: P1 +effort: 45 min +dependencies: [1] +--- + +# Phase 3: Inline scraper modules + +## Overview + +Replace `fetch('https://store-scraper.vercel.app/...')` with direct calls to `app-store-scraper` and `google-play-scraper` npm libs. Eliminates the cross-service HTTP roundtrip. + +## Requirements + +- `apple-scraper.js` and `google-scraper.js` keep their exported function signatures (`createAppleScraper`, `createGoogleScraper`, `buildAppleRequestBy*`, `buildGoogleRequest`) so command handlers don't change +- Behavior parity: `rawApp(req)` returns JSON string; `app(req)` returns parsed object +- Library `app()` method only — no other methods used today + +## Architecture + +``` +src/api/apple-scraper.js + ├─ removed: BASE_URL, fetch() + └─ uses: import store from 'app-store-scraper'; store.app(params) + +src/api/google-scraper.js + ├─ removed: BASE_URL, fetch() + └─ uses: import gplay from 'google-play-scraper'; gplay.app(params) +``` + +`rawApp` returns `JSON.stringify(result)` since libs return JS objects, not strings. Command `/rawappleapp` and `/rawgoogleapp` consumers expect text. + +## Related Code Files + +- Modify: `src/api/apple-scraper.js` +- Modify: `src/api/google-scraper.js` + +## Implementation Steps + +1. Edit `src/api/apple-scraper.js`: + ```js + import store from 'app-store-scraper'; + import { newAppleApp } from '../models/apple-app.js'; + + export function buildAppleRequestByTrackId(id, country) { + return { id, country, ratings: true }; + } + export function buildAppleRequestByBundleId(appId, country) { + return { appId, country, ratings: true }; + } + + export function createAppleScraper(config, storeRepo) { + const { logger } = config; + const repo = storeRepo.appleApp; + + async function app(req) { + return await store.app(req); + } + async function rawApp(req) { + return JSON.stringify(await app(req)); + } + // cache, getApp, fetchAndCache: unchanged + ... + } + ``` +2. Edit `src/api/google-scraper.js` analogously, importing `gplay` from `google-play-scraper`. +3. Verify no other files import `BASE_URL` (grep `src/ -r 'BASE_URL'`). +4. Smoke from a Node REPL after `npm install` (Phase 1 added the deps): + ```sh + node -e "import('app-store-scraper').then(m => m.default.app({ appId: 'com.apple.weather', country: 'us' })).then(r => console.log(r.appId, r.version))" + node -e "import('google-play-scraper').then(m => m.default.app({ appId: 'com.spotify.music', country: 'us' })).then(r => console.log(r.appId, r.version))" + ``` + +## Success Criteria + +- [ ] `grep -r 'store-scraper.vercel.app' src/` returns zero hits +- [ ] Local smoke calls succeed against both stores +- [ ] `apple-scraper.js`, `google-scraper.js` keep exported names; no other file modified +- [ ] Returned shape from `app()` matches what cache models (`newAppleApp`, `newGoogleApp`) expect + +## Risk Assessment + +- **Risk:** `app-store-scraper.app({ ratings: true })` makes 2 HTTPS calls (lookup + ratings.svg). Latency higher than single-fetch. **Mitigation:** acceptable for command-driven calls; cache TTL covers daily cron amplification. +- **Risk:** `google-play-scraper.app()` HTML scrape can fail when Google changes Play page markup. **Mitigation:** error already bubbles to user via existing try/catch in commands. Monitor first week post-deploy. +- **Risk:** `app-store-scraper` uses deprecated `request` lib — Node 20 warns but still runs. **Mitigation:** suppress with `NODE_NO_WARNINGS=1` if noisy. Long-term: track lib for replacement. +- **Risk:** Returned objects may have non-JSON-serializable fields (Dates). **Mitigation:** `JSON.stringify` produces ISO strings; consumers already parse strings. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md b/plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md new file mode 100644 index 0000000..f0df547 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md @@ -0,0 +1,133 @@ +--- +phase: 4 +title: "HTTP layer (webhook + cron)" +status: pending +priority: P1 +effort: 1h +dependencies: [2, 3] +--- + +# Phase 4: HTTP layer (webhook + cron) + +## Overview + +Replace the Cloudflare Worker default-export entry (`src/index.js` with `fetch`/`scheduled` handlers) with two Vercel serverless functions: `api/webhook.js` (Telegram webhook) and `api/cron.js` (daily check via Vercel Cron). + +## Requirements + +- Telegram webhook validates `X-Telegram-Bot-Api-Secret-Token` header (parity with current CF check) +- Webhook acks fast (<2 s), continues work in background via `@vercel/functions` `waitUntil` +- Cron handler runs daily check, returns 200 on success +- Both handlers use `loadConfig(process.env)` instead of CF `env` arg +- Build wires up `createUpstashClient` once per invocation (cheap — no connection) + +## Architecture + +``` +api/ +├── webhook.js POST handler — Telegram webhook +│ validates secret, calls dispatch(message), waitUntil +└── cron.js GET handler — Vercel Cron triggers daily + validates Authorization: Bearer $CRON_SECRET (Vercel signs cron requests) + runs runDailyCheck(...) + +src/ +├── index.js ← DELETE (CF default export gone) +├── config.js ← Modify: loadConfig(env) → loadConfig(process.env or env) +└── (rest unchanged) +``` + +Vercel cron requests carry `Authorization: Bearer ${process.env.CRON_SECRET}` header — validate to prevent random POSTs from triggering daily check. + +## Related Code Files + +- Create: `api/webhook.js` +- Create: `api/cron.js` +- Modify: `src/config.js` (env source) +- Delete: `src/index.js` + +## Implementation Steps + +1. Update `src/config.js` to take a plain object (works with both `process.env` and CF `env`): + ```js + export function loadConfig(env) { + // existing logic, but `env` is now `process.env` on Vercel + } + ``` +2. Create `api/webhook.js`: + ```js + import { waitUntil } from '@vercel/functions'; + import { loadConfig } from '../src/config.js'; + import { createUpstashClient } from '../src/repository/upstash.js'; + import { createStore } from '../src/repository/store.js'; + import { createAppleScraper } from '../src/api/apple-scraper.js'; + import { createGoogleScraper } from '../src/api/google-scraper.js'; + import { createBot } from '../src/bot/bot.js'; + import { dispatch } from '../src/bot/dispatch.js'; + + function build() { + const config = loadConfig(process.env); + const client = createUpstashClient(process.env); + const store = createStore(client, config.appCacheSeconds); + const appleScraper = createAppleScraper(config, store); + const googleScraper = createGoogleScraper(config, store); + const { sender, commands } = createBot(config, store, appleScraper, googleScraper); + return { config, store, sender, commands }; + } + + export default async function handler(req) { + if (req.method !== 'POST') return new Response('Not found', { status: 404 }); + + const app = build(); + const secret = req.headers.get('X-Telegram-Bot-Api-Secret-Token'); + if (secret !== app.config.telegramWebhookSecret) { + return new Response('Unauthorized', { status: 401 }); + } + let update; + try { update = await req.json(); } catch { return new Response('Bad request', { status: 400 }); } + if (!update?.message) return new Response('OK'); + + waitUntil(dispatch(update.message, { + sender: app.sender, commands: app.commands, + config: app.config, logger: app.config.logger, + })); + return new Response('OK'); + } + + export const config = { runtime: 'nodejs' }; + ``` +3. Create `api/cron.js`: + ```js + import { runDailyCheck } from '../src/scheduler/scheduler.js'; + // (same build() helper — extract to src/app-builder.js if duplication bothers) + + export default async function handler(req) { + const auth = req.headers.get('Authorization'); + if (auth !== `Bearer ${process.env.CRON_SECRET}`) { + return new Response('Unauthorized', { status: 401 }); + } + const app = build(); + await runDailyCheck(app.config, app.store, app.sender, app.appleScraper, app.googleScraper); + return new Response('OK'); + } + + export const config = { runtime: 'nodejs', maxDuration: 60 }; + ``` +4. Extract shared `build()` into `src/app-builder.js` (DRY); both functions import. +5. Delete `src/index.js`. +6. Local test: `vercel dev` → `curl -X POST localhost:3000/api/webhook` (expect 401), then with valid secret + JSON body (expect 200). + +## Success Criteria + +- [ ] `api/webhook.js` and `api/cron.js` exist; both exported as default async handlers +- [ ] Both validate auth (Telegram secret / Cron bearer) +- [ ] `vercel dev` starts without error +- [ ] `src/index.js` deleted +- [ ] `src/config.js` works with `process.env` + +## Risk Assessment + +- **Risk:** `runtime: 'nodejs'` defaults vary across Vercel CLI versions; some default to edge. **Mitigation:** explicit declaration in function `export const config`. +- **Risk:** `waitUntil` requires Vercel runtime; behaves no-op in `vercel dev`. **Mitigation:** acceptable; production deploy honors it. Verify in Phase 6. +- **Risk:** Cold start on first webhook invocation reaches ~500 ms while Telegram retries on >30 s. **Mitigation:** comfortable margin. Pre-warm not needed. +- **Risk:** `CRON_SECRET` mishandling makes cron callable by anyone. **Mitigation:** generate ≥32-char secret, store as Vercel env var, set on Vercel Cron settings (Vercel injects automatically). diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-cf-kv-to-upstash.md b/plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-cf-kv-to-upstash.md new file mode 100644 index 0000000..b462ae5 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-cf-kv-to-upstash.md @@ -0,0 +1,71 @@ +--- +phase: 5 +title: "Data migration CF KV → Upstash" +status: pending +priority: P1 +effort: 30 min +dependencies: [2] +--- + +# Phase 5: Data migration CF KV → Upstash + +## Overview + +One-shot migration script: read all keys from Cloudflare KV via `wrangler kv key list` + `wrangler kv key get`, write them to Upstash Redis with TTL preserved on cache entries. + +## Requirements + +- Migrate `admin`, `group:*` keys (must preserve) +- Skip `apple:*` and `google:*` cache keys by default (auto-rebuild from upstream); add `--include-cache` flag for parity +- Idempotent — safe to re-run +- Preserves TTL on cache entries (compute remaining seconds) + +## Architecture + +Two-step, mirroring existing `migrate-atlas-to-kv.js` pattern: + +``` +scripts/migrate-cf-kv-to-upstash.js + 1. wrangler kv key list --binding STORE_KV --remote → JSON of key names + 2. for each key: wrangler kv key get → value + metadata.expiration + 3. write to Upstash via @upstash/redis SET with EX (if TTL applies) + 4. log counts: admin, group, apple, google +``` + +Use `wrangler` CLI (already devDep) via `child_process.execFileSync` rather than re-implementing CF API auth. + +## Related Code Files + +- Create: `scripts/migrate-cf-kv-to-upstash.js` +- Modify: `package.json` add script `migrate:upstash` + +## Implementation Steps + +1. Create `scripts/migrate-cf-kv-to-upstash.js`: + - Reads `UPSTASH_REDIS_REST_URL`, `UPSTASH_REDIS_REST_TOKEN` from `.env.deploy` + - Reads CF binding name from arg or default `STORE_KV` + - Calls `wrangler kv key list --binding STORE_KV --remote` to enumerate + - For each key: `wrangler kv key get --binding STORE_KV --remote ` (returns raw value) + - Posts to Upstash via `Redis` SDK: `await client.set(key, value, ttl ? { ex: ttl } : {})` + - `--include-cache` flag controls apple:/google: prefixes + - `--dry-run` flag prints actions without writing +2. Add `package.json` script: + ```json + "migrate:upstash": "node --env-file=.env.deploy scripts/migrate-cf-kv-to-upstash.js" + ``` +3. Test in dry-run mode against current CF KV; verify key counts match expected admin + groups. + +## Success Criteria + +- [ ] Dry-run output lists all `admin` and `group:*` keys from CF KV +- [ ] Real run with --dry-run=false writes to Upstash +- [ ] Post-run: `redis-cli SCAN MATCH 'group:*'` (or Upstash REST equivalent) returns same group count as CF KV had +- [ ] `admin` key value matches byte-for-byte across CF and Upstash +- [ ] Re-run is idempotent (overwrites with same data) + +## Risk Assessment + +- **Risk:** `wrangler kv key list` doesn't expose TTL in default JSON output. **Mitigation:** `--include-cache` flag uses key metadata `expiration` if available; if not, skip TTL preservation and let cache rebuild (acceptable per design). +- **Risk:** Wrangler authenticates against the user's CF account; need `wrangler login` done before run. **Mitigation:** doc note in phase + check at script start. +- **Risk:** Large key count (>10k) blows out Wrangler shell pipeline. **Mitigation:** bot has <100 keys; not a concern. If grows, batch via cursor. +- **Risk:** Migration runs while bot still serving on CF (writes after migration are lost). **Mitigation:** cutover sequence in Phase 6 — migrate AFTER pausing CF webhook, BEFORE setting Vercel webhook. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md b/plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md new file mode 100644 index 0000000..1558f26 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md @@ -0,0 +1,105 @@ +--- +phase: 6 +title: "Deploy + cutover + webhook re-register" +status: pending +priority: P1 +effort: 45 min +dependencies: [4, 5] +--- + +# Phase 6: Deploy + cutover + webhook re-register + +## Overview + +Deploy Vercel project, configure secrets, run migration, flip Telegram webhook URL. Hard cutover with <5 min downtime. + +## Requirements + +- Vercel project deployed with all secrets set +- Upstash database provisioned (free tier) +- Telegram webhook points to new Vercel URL +- All 13 commands smoke-test pass on production +- CF Worker still deployed (rollback path for ~7 days) + +## Architecture + +Sequence (operator-driven, ~30 min wall clock): + +``` +1. PROVISION + Upstash → create free Redis DB → copy REST_URL + REST_TOKEN + Vercel → create project from repo or link existing + +2. SECRETS (Vercel env vars) + TELEGRAM_BOT_TOKEN + TELEGRAM_BOT_USERNAME + TELEGRAM_WEBHOOK_SECRET (≥32 chars random) + ADMIN_IDS (comma-separated) + UPSTASH_REDIS_REST_URL + UPSTASH_REDIS_REST_TOKEN + CRON_SECRET (≥32 chars random; for cron auth) + APP_CACHE_SECONDS=600 + NUM_DAYS_WARNING_NOT_UPDATED=30 + +3. DEPLOY + vercel deploy --prod (gets URL like store-scraper-bot-xxx.vercel.app) + +4. SMOKE (no traffic yet) + curl -X POST $URL/api/webhook → 401 (no secret) → confirms route alive + curl with valid secret + dummy body → 200 + +5. MIGRATE DATA (downtime starts) + Telegram setWebhook to bogus URL (or empty) → no further updates land on CF + npm run migrate:upstash (CF KV → Upstash) + verify admin + group counts in Upstash UI + +6. FLIP + .env.deploy: WORKER_URL=https:///api/webhook + npm run register (re-registers with new URL + same secret) + wait 30 s for Telegram propagation + +7. SMOKE COMMANDS (production) + /info → returns chat ID + /listgroup (admin) → lists migrated groups + /listapp (in known group) → lists tracked apps + /checkapp → fetches via inlined scrapers + ... (run all 13 commands per docs/code-standards.md or tester checklist) + +8. WATCH + tail Vercel function logs 30 min for error spikes + verify daily cron at 00:00 UTC fires (next morning) +``` + +## Related Code Files + +- Modify: `.env.deploy` (operator file, gitignored — change `WORKER_URL`) +- No code changes; this phase is operator-driven + +## Implementation Steps + +1. Sign up Upstash → create Redis database (region matching Vercel deploy region — `us-east-1` default). +2. Vercel: link existing project OR create new from `tiennm99/store-scraper-bot` repo. Set all env vars listed above via `vercel env add` or dashboard. +3. `vercel deploy --prod`. Note returned URL. +4. Update `.env.deploy` `WORKER_URL` to new URL (keep variable name for now — rename in Phase 7). +5. Pause Telegram updates: `setWebhook` to empty URL (`tg setWebhook` body `{ url: '' }`). +6. Run `npm run migrate:upstash` (Phase 5 script). Verify counts. +7. Run `npm run register` to set webhook to Vercel URL. +8. Smoke all 13 commands manually in Telegram. Use `docs/` test checklist if present. +9. Tail Vercel logs: `vercel logs --follow` for 15 min. +10. Document Vercel URL + Upstash DB name in plan/journal. + +## Success Criteria + +- [ ] `getWebhookInfo` shows new Vercel URL +- [ ] All 13 commands smoke-pass on production (manually checked) +- [ ] Daily cron fires at 00:00 UTC (verify in Vercel Functions log next morning) +- [ ] Vercel function logs show no error-level entries for 30 min +- [ ] Upstash dashboard shows admin + group:* keys with expected count + +## Risk Assessment + +- **Risk:** Vercel function cold start exceeds Telegram 30 s ack window on first hit. **Mitigation:** ack returns 200 OK fast (before `waitUntil` heavy work); cold start typically <500 ms. +- **Risk:** Migration loses TTL on cache entries. **Mitigation:** acceptable (cache rebuilds in 10 min). Skipped by default. +- **Risk:** Smoke missed an edge case → user-visible regression. **Mitigation:** keep CF Worker deployed; rollback = re-register webhook to CF URL via `register-webhook.js`. <2 min recovery. +- **Risk:** Vercel free tier rate-limits cold starts during smoke. **Mitigation:** 100 invocations/day quota is plenty; smoke uses ~20. +- **Risk:** Operator forgets `CRON_SECRET` → cron handler always 401. **Mitigation:** verify by triggering cron manually via Vercel dashboard "Run Now" before relying on schedule. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md b/plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md new file mode 100644 index 0000000..31e8b0c --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md @@ -0,0 +1,96 @@ +--- +phase: 7 +title: "Cleanup wrangler + docs" +status: pending +priority: P2 +effort: 30 min +dependencies: [6] +--- + +# Phase 7: Cleanup wrangler + docs + +## Overview + +After Vercel deploy is stable (≥7 days post-cutover), remove Cloudflare Workers artifacts, retire the old `tiennm99/store-scraper` Vercel deployment, and update README + docs to reflect the new architecture. + +## Requirements + +- Repo no longer references CF Workers, KV, or `wrangler` (except in `plans/` archive history) +- README accurately describes Vercel + Upstash setup +- Old `migrate-atlas-to-kv.js` and `migrate-cf-kv-to-upstash.js` scripts removed (one-shot, role done) +- `mongodb` devDep removed +- `tiennm99/store-scraper` repo marked deprecated + +## Architecture + +Files to remove: + +``` +DELETE +├── wrangler.toml +├── docker-compose.yml +├── docker-compose.dev.yml (if Worker-specific; keep if generic Node) +├── Dockerfile (likely Worker-specific; verify) +├── scripts/migrate-atlas-to-kv.js +├── scripts/migrate-cf-kv-to-upstash.js +└── (CF Worker on dashboard) wrangler delete + +MODIFY +├── package.json remove wrangler, mongodb devDeps +├── README.md Vercel + Upstash setup, drop Atlas/KV history +└── docs/ if any architecture docs reference CF +``` + +External cleanup: + +``` +- Cloudflare dashboard: delete Worker deployment +- Cloudflare dashboard: delete STORE_KV namespace (after verifying Upstash data preserved) +- Vercel dashboard: archive `store-scraper` project (the old wrapper) OR mark README deprecated +- GitHub `tiennm99/store-scraper` repo: add deprecation banner to README pointing to bot repo +``` + +## Related Code Files + +- Delete: `wrangler.toml` +- Delete: `scripts/migrate-atlas-to-kv.js` +- Delete: `scripts/migrate-cf-kv-to-upstash.js` +- Delete: `Dockerfile`, `docker-compose*.yml` (verify first; may keep if generic) +- Modify: `package.json` (drop `wrangler`, `mongodb`; drop `migrate*` scripts) +- Modify: `README.md` + +## Implementation Steps + +1. Wait ≥7 days post-Phase 6 with no incidents. +2. Delete `wrangler.toml`. +3. Delete `scripts/migrate-atlas-to-kv.js` and `scripts/migrate-cf-kv-to-upstash.js`. +4. Remove from `package.json`: + - devDeps: `wrangler`, `mongodb` + - scripts: `migrate`, `migrate:bulk`, `migrate:upstash` +5. Verify `Dockerfile` + `docker-compose*.yml` — if Worker-specific, delete; if generic Node + Upstash, keep. +6. Rewrite `README.md`: + - Replace "Cloudflare Workers" with "Vercel" + - Update env var table (drop MONGODB_URI / STORE_KV; add UPSTASH_REDIS_REST_URL / TOKEN, CRON_SECRET) + - Update Run section: `vercel dev` for local; `vercel deploy --prod` for production + - Drop "Migrating from MongoDB Atlas" section (one-shot completed) +7. Cloudflare dashboard: `wrangler delete` Worker, then delete STORE_KV namespace. +8. `tiennm99/store-scraper` repo on GitHub: + - Add README banner: "⚠️ Deprecated — logic now inlined in [`tiennm99/store-scraper-bot`](https://github.com/tiennm99/store-scraper-bot)" + - Optionally archive the repo (`gh repo archive tiennm99/store-scraper`) +9. Vercel dashboard: delete `store-scraper` project (the old wrapper). +10. Commit: `chore: remove cloudflare workers + retire old scraper wrapper` + +## Success Criteria + +- [ ] `git grep -i 'cloudflare\|wrangler\|store_kv'` returns only `plans/` history hits +- [ ] `package.json` has no `wrangler` or `mongodb` +- [ ] README accurately describes Vercel + Upstash +- [ ] CF Worker no longer exists on dashboard +- [ ] `tiennm99/store-scraper` repo marked deprecated and/or archived +- [ ] Old Vercel `store-scraper` project deleted + +## Risk Assessment + +- **Risk:** Deleting CF KV namespace before verifying Upstash holds all production data. **Mitigation:** explicitly verify counts in Phase 6 success criteria; defer namespace delete by another 7 days if paranoid. +- **Risk:** Other consumers exist for `store-scraper.vercel.app` (third-party users). **Mitigation:** check by adding a 410 Gone response for 14 days before deleting; deprecation banner in repo gives outside heads-up. +- **Risk:** `Dockerfile` deletion breaks an alternate deploy path the user relies on. **Mitigation:** ask user before deleting; if uncertain, keep file with a comment. diff --git a/plans/260509-1656-consolidate-vercel-upstash/plan.md b/plans/260509-1656-consolidate-vercel-upstash/plan.md new file mode 100644 index 0000000..e7c7a53 --- /dev/null +++ b/plans/260509-1656-consolidate-vercel-upstash/plan.md @@ -0,0 +1,61 @@ +--- +title: "Consolidate on Vercel + Upstash Redis" +description: "Move bot off Cloudflare Workers + Vercel split-architecture onto a single Vercel deployment. Inline scraper libs (drop store-scraper.vercel.app fetch). Replace CF KV with Upstash Redis. Single repo, single vendor, free tier." +status: pending +priority: P1 +effort: 6h +branch: main +tags: [vercel, upstash, migration, scraper-inline, telegram] +created: 2026-05-09 +blockedBy: [] +blocks: [] +supersedes: 260505-1425-cloudflare-kv-migration-and-deploy +--- + +# Consolidate on Vercel + Upstash + +Replaces the in-progress Cloudflare Workers + KV direction. Brainstorm + research locked these decisions: + +- **Compute:** Vercel serverless functions (Hobby plan, free) +- **Storage:** Upstash Redis (free 500k cmd/mo, persistent RocksDB) +- **Scraper:** inline `app-store-scraper` + `google-play-scraper` npm libs, drop `store-scraper.vercel.app` HTTP roundtrip +- **Old `tiennm99/store-scraper` repo:** leave running 1–2 weeks as fallback, then archive + +Source design: [`reports/brainstorm-260509-1656-consolidate-vercel-upstash.md`](../reports/brainstorm-260509-1656-consolidate-vercel-upstash.md) +Storage research: [`reports/researcher-260509-1656-upstash-vs-atlas.md`](../reports/researcher-260509-1656-upstash-vs-atlas.md) + +## Phases + +| # | Phase | Effort | Status | +|---|---|---|---| +| 01 | [Vercel scaffolding + deps](phase-01-vercel-scaffolding.md) | 30 min | pending | +| 02 | [Upstash repository adapter](phase-02-upstash-repository-adapter.md) | 1h | pending | +| 03 | [Inline scraper modules](phase-03-inline-scraper-modules.md) | 45 min | pending | +| 04 | [HTTP layer (webhook + cron)](phase-04-http-layer-webhook-and-cron.md) | 1h | pending | +| 05 | [Data migration CF KV → Upstash](phase-05-data-migration-cf-kv-to-upstash.md) | 30 min | pending | +| 06 | [Deploy + cutover + webhook re-register](phase-06-deploy-cutover-and-webhook-reregister.md) | 45 min | pending | +| 07 | [Cleanup wrangler + docs](phase-07-cleanup-wrangler-and-docs.md) | 30 min | pending | + +## Key Constraints + +- Free tier only — no $5/mo CF or Vercel Pro upgrades +- Single service — no separate scraper deployment after cutover +- Hard cutover migration; <5 min downtime acceptable +- Telegram parity preserved; all 13 commands keep working +- Daily cron 07:00 Asia/Saigon = 00:00 UTC on Vercel Cron + +## Success Criteria (overall) + +- Vercel deploy returns 200 on Telegram webhook within 2 s +- Daily cron visible in Vercel Functions log, runs without error +- Admin + group state preserved post-migration (count match) +- `grep -r 'store-scraper.vercel.app' src/` returns zero hits +- `wrangler.toml` and `wrangler` devDep removed from repo +- Bundle size <50 MB; cold start <500 ms + +## Dependencies + +- Vercel account (already exists — bot will replace existing `store-scraper` project or get a new project) +- Upstash account (new — sign up at upstash.com or via Vercel Marketplace integration) +- Telegram bot already deployed; needs webhook URL re-registration after cutover +- Active CF Worker still running through migration window (rollback safety) diff --git a/plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md b/plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md new file mode 100644 index 0000000..9e94434 --- /dev/null +++ b/plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md @@ -0,0 +1,183 @@ +# Brainstorm — Consolidate js-store-scraper-bot on Vercel + Upstash + +**Date:** 2026-05-09 +**Branch:** main +**Status:** Decision locked, ready to plan + +--- + +## Problem Statement + +`js-store-scraper-bot` (Telegram bot) currently splits across: +- **Cloudflare Workers** (free tier) — bot logic, KV, daily cron. Recently migrated from MongoDB Atlas → CF KV. +- **Vercel** (`tiennm99/store-scraper`) — Express wrapper around `app-store-scraper` + `google-play-scraper` npm libs. Bot calls via HTTP. + +User wants single free-tier service, no self-hosting, no vendor lock to CF. Needs to remove the cross-service `fetch()` to `store-scraper.vercel.app`. + +--- + +## Constraints (locked via Q&A) + +| Constraint | Value | +|---|---| +| Hosting | Free tier only | +| Architecture | Single service (consolidate) | +| Vendor lock | OK to leave CF, OK to stay on Vercel | +| Self-hosting | No | +| Lib parity needed today | `app()` only on both stores | +| Future parity might want | search / similar / reviews | + +--- + +## Approaches Evaluated + +### Path 1 — Do nothing (rejected by user) +Keep CF Worker + Vercel scraper. Architecturally correct, zero effort, both free. +**Rejected:** user wants single service. + +### Path 2 — Consolidate on Vercel ← chosen +Move bot onto Vercel as serverless functions + Vercel Cron. Inline scraper libs. Replace CF KV. + +### Path 3 — Native fetch on CF Workers (rejected) +Skip libs, hand-roll `app()` in ~100 LOC. +**Rejected:** free-tier 10ms CPU/invocation breaks daily cron parsing N Play HTMLs. Loses lib parity for future features. + +--- + +## Storage Sub-Decision: Upstash vs Atlas + +Researched 2026 state. Verdict: **Upstash Redis**. + +| Dimension | Upstash | Atlas M0 | Winner | +|---|---|---|---| +| Repo layer change cost | Swap binding (~50 LOC) | Full rewrite back to Mongo doc model | Upstash | +| Persistence | RocksDB + daily snapshots | Canonical DB + backups | Tie | +| Cold-start latency (Vercel) | 20–30 ms | 150–300 ms TCP+TLS+auth | Upstash | +| Bundle size | ~50 KB SDK | ~80 MB driver | Upstash | +| Idle behavior | None | 30-day auto-pause | Upstash | +| REST escape hatch | Native | Atlas Data API EOL Sep 2025 | Upstash | +| Free-tier headroom for 50–200 ops/day | 500k cmd/mo (41× margin) | ∞ but irrelevant | Tie | +| Schema parity with Java/Go bots | n/a | Yes | Atlas (but YAGNI here) | + +Atlas only wins if user plans to share data with Java/Go reference impls — not the case. + +Persistence concern user raised resolved: Upstash is durable, not in-memory. + +**Cache loss risk:** apple/google app cache rebuilds in ≤10 min from upstream APIs. Admin/group state is small + slow-changing; can snapshot to JSON in repo if paranoid. + +Source: `plans/reports/researcher-260509-1656-upstash-vs-atlas.md` + +--- + +## Recommended Solution + +### Architecture + +``` +Vercel Project (single deploy, free Hobby plan) +├── api/webhook.js Telegram webhook entry +│ Validates X-Telegram-Bot-Api-Secret-Token +│ waitUntil(dispatch(message, ...)) +│ Returns 200 OK fast +├── api/cron.js Vercel Cron 0 0 * * * (07:00 Asia/Saigon) +│ runDailyCheck(...) +└── src/ + ├── api/ + │ ├── apple-scraper.js Inline → store.app() direct + │ └── google-scraper.js Inline → gplay.app() direct + ├── repository/ Upstash Redis adapter + ├── bot/ Unchanged (commands, dispatch, sender) + └── scheduler/ Unchanged + +Removed: + wrangler.toml, wrangler devDep, STORE_KV binding + src/index.js (CF default export) + store-scraper.vercel.app fetch BASE_URL constants + ctx.waitUntil → @vercel/functions waitUntil +``` + +### New Dependencies + +| Package | Why | +|---|---| +| `app-store-scraper@^0.18.0` | Inlined Apple scraper | +| `google-play-scraper@^10.1.2` | Inlined Google scraper | +| `@upstash/redis` | KV storage | +| `@vercel/functions` | `waitUntil` analog of CF `ctx.waitUntil` | + +### Removed + +- `wrangler` (devDep) +- `mongodb` (devDep — was used for the Atlas → KV migration; obsolete) + +### Storage Schema (Upstash) + +Flat key-namespaced KV. Mirrors current CF KV layout: + +| Key pattern | Value | TTL | +|---|---|---| +| `admin` | JSON array of Telegram user IDs | none | +| `group:` | JSON group config | none | +| `cache:apple:` | JSON app data | `APP_CACHE_SECONDS` (default 600) | +| `cache:google:` | JSON app data | `APP_CACHE_SECONDS` | + +Iteration via `SCAN MATCH group:*` for daily check. + +### Cutover Plan + +1. **Hard cutover** — single migration window <5 min: + - Export CF KV via `wrangler kv key list --remote` + `wrangler kv key get` loop (or use existing `scripts/.atlas-export.json` if still present) + - Import to Upstash via `@upstash/redis` script + - Deploy Vercel app with new webhook URL + - Re-register Telegram webhook to new Vercel URL + - Verify `/info` command works +2. **Old `store-scraper.vercel.app` deployment** — leave running 1–2 weeks as fallback, then delete. +3. **Old CF Worker** — leave deployed (silent — Telegram webhook no longer points there) for ~1 week, then `wrangler delete`. + +--- + +## Risks / Mitigations + +| Risk | Mitigation | +|---|---| +| Vercel Hobby cron limit (1×/day max) | Schedule is daily — fits exactly. | +| Vercel function timeout (10s default, 60s max) | Bump cron handler to `maxDuration: 60`. Daily check loops sequentially over groups; should fit. | +| Telegram webhook >30s = retry storm | Use `@vercel/functions` `waitUntil()` to ack fast and continue work in background. | +| Upstash 256 MB cap eviction | Bot data <100 KB. ~2500× headroom. | +| Lib bundle size | Vercel function limit 250 MB uncompressed; libs ~30 MB. Comfortable. | +| TLS / cert issues with `app-store-scraper` `request` lib in Node 20+ | Node 20 still supports `request`. If breaks, swap to native `fetch()` shim — fallback path. | +| Re-registering Telegram webhook | Existing `scripts/register-webhook.js` reusable; just change URL var. | + +--- + +## Success Metrics + +- Telegram `/info` returns within 2 s after Vercel deploy +- Daily cron runs at 07:00 VN time (visible in Vercel Functions log) +- Admin + group state preserved across migration (count match pre/post) +- Apple + Google `/checkapp` returns identical data to current Vercel scraper output +- No lingering calls to `store-scraper.vercel.app` (grep src/ for vercel.app) +- Vercel function bundle <50 MB, cold start <500 ms + +--- + +## Next Steps + +1. Generate phased plan via `/ck:plan` (this triggers next). +2. Implementation phases (preview): + - Phase 1 — Vercel scaffolding (`vercel.json`, env, deps) + - Phase 2 — Upstash repository adapter + - Phase 3 — Inline scraper modules (drop fetch BASE_URL) + - Phase 4 — HTTP layer (`api/webhook.js`, `api/cron.js`) + - Phase 5 — Data migration script (CF KV → Upstash) + - Phase 6 — Deploy + cutover + Telegram webhook re-register + - Phase 7 — Cleanup (remove wrangler, stale scripts, README update) + +--- + +## Unresolved Questions + +1. Vercel region preference? Default `iad1` (us-east) is fine; bot is low-latency-tolerant. +2. Should the old `tiennm99/store-scraper` repo be archived/deleted now or after 2-week observation? (User leaning: keep running, mark deprecated.) +3. Re-add Mongo schema docs in `docs/` or remove (since we're permanent KV now)? +4. Do we want Vercel Marketplace Upstash integration (env vars auto-injected) or BYO Upstash account? diff --git a/plans/reports/researcher-260509-1656-upstash-vs-atlas.md b/plans/reports/researcher-260509-1656-upstash-vs-atlas.md new file mode 100644 index 0000000..2c5f7b1 --- /dev/null +++ b/plans/reports/researcher-260509-1656-upstash-vs-atlas.md @@ -0,0 +1,96 @@ +# Upstash Redis vs MongoDB Atlas M0 for Vercel Serverless Bot + +## Verdict +**Upstash Redis wins** for this use case. KV-style abstraction, zero connection management overhead, 500K cmds/mo free tier far exceeds 50–200 ops/day need. Atlas M0 adds complexity (connection pooling, cold-start latency) with no benefit for sub-100KB workload. + +--- + +## Comparison Table + +| Dimension | **Upstash Redis** | **MongoDB Atlas M0** | +|-----------|------------------|-------------------| +| **Storage** | 256 MB | 512 MB | +| **Monthly Commands** | 500K (≈16.7K/day) | Not throttled; 100 ops/sec | +| **Connection Limit** | HTTP REST (unlimited) | 500 concurrent | +| **Cold Start Latency** | 10–20ms HTTP roundtrip | 50–100ms+ (TCP + auth overhead) | +| **Idle/Inactivity** | No auto-pause | Pauses after 30 days inactivity → wake ~1s | +| **Bundle Size** | ~50KB (est.) | ~80+ MB (`mongodb` driver) | +| **Serverless Friction** | Native HTTP; no pools | Requires pool mgmt (`@vercel/functions` pattern) | +| **Free Tier Gotchas** | Eviction when 256MB exceeded; random key removal | Hit 500 conn limit on cold-start spike; pause horror on weekends | +| **Persistence** | RocksDB distributed; daily snapshots (inferred) | Full durability; backed snapshots | + +--- + +## Key Findings + +### 1. Upstash Redis Free Tier (Current 2026) +- **Limits:** 500K commands/month (March 2025 upgrade from 10K/day), 256 MB storage, 200 GB bandwidth/mo free [upstash.com/blog/redis-new-pricing] +- **Cold Start:** HTTP-based REST API = ~18ms round-trip from Vercel Functions (no TCP handshake cost) [Medium@amarharolikar, Mar 2026] +- **Serverless Native:** `@upstash/redis` package is HTTP-only, ESM compatible, ~50KB bundle size (estimate from GitHub repo) +- **Inactivity:** No auto-pause policy mentioned; free tier remains accessible +- **Gotcha:** Eviction enabled by default. When 256 MB exceeded, random keys deleted (TTL-expiring keys first, then random) [upstash.com/docs/redis/features/eviction]. For 100 KB data + caching, unlikely to trigger. +- **Persistence:** RocksDB backend with daily snapshots (inferred from architecture); HTTP REST calls are stateless + +### 2. MongoDB Atlas M0 Free Tier (Current 2026) +- **Limits:** 512 MB storage, 500 concurrent connections, 100 ops/sec throughput [mongodb.com/docs/atlas/reference/free-shared-limitations] +- **Inactivity Pause:** Auto-pause after 30 days zero connections; wake-up adds ~1s latency [MongoDB docs] +- **Cold Start:** TCP connection + TLS + driver overhead = 50–100ms+ typical roundtrip. Vercel functions can spike connections → easy to hit 500 limit on cold-start storms [github.com/payloadcms/payload/issues/14547, 2025] +- **Driver Bundle Size:** `mongodb` npm package ≈80+ MB uncompressed (heavy for 250 MB Vercel Function limit); requires careful tree-shaking [dev.to/devshefali, 2025] +- **Connection Pooling Required:** Mandatory connection pool management in serverless. Vercel recommends `@vercel/functions` attachment pattern but not all ORMs support it yet [github.com/payloadcms/payload/issues/14547] +- **Data API Deprecated:** Atlas Data API (REST alternative) deprecated Sept 30, 2025 → now EOL. No native REST gateway [mongodb.com/docs/atlas/app-services/data-api/data-api-deprecation] + +### 3. Cold Start Latency (Vercel us-east region) +| Scenario | Upstash | Atlas M0 | +|----------|---------|----------| +| Warm function, single GET/PUT | 15–25ms | 5–10ms (cached conn) | +| Cold function, single GET/PUT | 20–30ms | 150–300ms (new conn + auth) | +| 100 concurrent cold starts | 20–30ms each | 100+ ms; risk of conn limit exhaustion | + +**Winner:** Upstash (HTTP stateless > TCP pooled) for serverless. + +### 4. Your Use Case Fit +- **Workload:** 50–200 ops/day, <100 KB data, 10-min TTL caching + - Upstash: 500K cmds/mo = **41x headroom**. 256 MB storage = **2560x headroom**. ✅ + - Atlas: 100 ops/sec = **1.15 million ops/day possible**, 512 MB storage adequate, but **cold-start & pooling overhead wasted** for read/write pattern. + +- **Schema Compatibility:** Your prior schema (Mongo `common`, `group`, `apple_app`, `google_app` collections with `_id` + `class` discriminator): + - Upstash: Migrate to flat KV (e.g., `admin_users`, `group_settings:${id}`, `app_cache:apple`). Simple serialization (JSON strings). No schema. + - Atlas: Unchanged—native document model. More friction (driver, pooling) for small data. + +--- + +## Unresolved Questions + +1. **Upstash persistence guarantees:** Docs don't explicitly state RTO/RPO or backup frequency. Are daily snapshots sufficient for your use case? (Likely yes for non-critical caching, but worth confirming.) +2. **MongoDB free-tier wake-up latency:** Confirmed 30-day auto-pause, but exact wake-up time on first connection not quantified. Is ~1s acceptable for webhook? +3. **Vercel connection pool helper availability:** `@vercel/functions attachDatabasePool` was added 2024; does your stack (Node runtime) support it? Check Vercel docs for your specific framework. + +--- + +## Recommendation + +**Use Upstash Redis.** Rationale: +- Designed for serverless (HTTP, no connection pooling complexity). +- Free tier massively exceeds your needs (500K cmds/mo vs 6K/mo actual). +- Zero inactivity pause risk (unlike Atlas M0 weekend awkwardness). +- Cold-start latency is 5–10x faster than Atlas (20ms vs 150–300ms on cold). +- Bundle size lightweight (~50 KB vs ~80 MB). +- Aligns with your existing KV migration trajectory (you've already left MongoDB). + +If you later need ACID transactions or complex queries, upgrade to a serverless relational DB (e.g., Vercel Postgres Free, Turso SQLite) instead of retrofitting Atlas. + +--- + +## Sources + +- [Upstash Blog: New Pricing and Increased Limits for Upstash Redis](https://upstash.com/blog/redis-new-pricing) +- [Upstash Docs: Redis Pricing](https://upstash.com/docs/redis/overall/pricing) +- [Upstash Docs: Eviction](https://upstash.com/docs/redis/features/eviction) +- [Medium: Upstash Redis on Vercel (Mar 2026)](https://medium.com/@amarharolikar/upstash-redis-on-vercel-the-tool-i-didnt-know-i-needed-7ecfbb6e7a6e) +- [MongoDB Atlas Free Cluster Limits](https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/) +- [Payload CMS Issue #14547: MongoDB Vercel Connection Pool](https://github.com/payloadcms/payload/issues/14547) +- [MongoDB Atlas Data API Deprecation](https://www.mongodb.com/docs/atlas/app-services/data-api/data-api-deprecation/) +- [Vercel KB: Serverless Function Size Limit](https://vercel.com/kb/guide/troubleshooting-function-250mb-limit) +- [DEV: Deploy Node.js + MongoDB to Vercel (2025)](https://dev.to/devshefali/5-easy-steps-to-deploy-your-nodejs-mongodb-app-to-vercel-2d7k) +- [GitHub: upstash/redis-js](https://github.com/upstash/redis-js) +- [Vercel: MongoDB Atlas for Vercel](https://vercel.com/integrations/mongodbatlas)