 tiennm99
|
0859356ec7
|
feat(scripts): phase 05 — backfill + verify + wipe (local node, no admin routes)
Operator-run migration scripts for KV→Mongo and D1→trading_trades, plus a
parity verifier and a rollback wiper. Pure local Node — no Worker code,
no /__admin/* routes, no new Worker secrets. Complies with
docs/architecture.md §10.
Scripts
- backfill-kv-to-mongo.js: paginates CF KV REST API per module, fetches
values, $setOnInsert upsert into per-module Mongo collection. Resumes
from .backfill-cursor-<module>.json on restart. Throttles 50 ops/sec.
expiresAt derived from KV metadata.expiration (debugger #10). --dry-run
and --module flags for incremental work.
- backfill-d1-to-mongo.js: wrangler d1 execute --remote --json → parse →
insertMany batches into trading_trades, preserving original integer id
as legacy_id (code-reviewer #13). Pre-flight aborts if collection
non-empty unless --force.
- verify-mongo-parity.js: count parity ±1%, SHA256 value compare,
expiresAt ±5min bucket. Full-scan when <10K docs, sqrt-sample
capped at 500 otherwise (code-reviewer #21). Trading: full-scan
on legacy_id/ts/user_id/symbol/qty.
- wipe-mongo.js: rollback helper. deleteMany across all collections
with readline confirm. --yes for CI.
- lib/migration-helpers.js: shared sleep, sha256, checkpoint I/O,
cfKvList/cfKvGet, MongoClient singleton, sample strategy.
Surface updates
- .env.deploy.example: CF account/token/namespace placeholders.
- package.json: backfill:kv[:dry], backfill:d1[:dry], verify:mongo,
wipe:mongo scripts.
- check-secret-leaks.js: SECRETS array gains CLOUDFLARE_API_TOKEN +
CLOUDFLARE_ACCOUNT_ID for defense-in-depth.
- .gitignore: .backfill-cursor-*.json excluded.
Tests: 638 → 667 (+29 pure-logic tests for sha256, checkpoint round-trip,
count-diff, sample-size, fetch-mocked CF REST). Lint clean.
Operator-run sequence (after Phase 06 deploy):
npm run backfill:kv:dry # preview
npm run backfill:kv
npm run backfill:d1:dry
npm run backfill:d1
npm run verify:mongo # exit 0 = parity ok
|
2026-04-26 09:13:00 +07:00 |
|