mirror of
https://github.com/tiennm99/miti99bot.git
synced 2026-09-05 04:19:36 +00:00
fix(plans): align cf-to-aws migration and cutover docs
Lock the Cloudflare-to-AWS migration matrix and runbook against the live DynamoDB runtime shape, and gate AWS cutover on verified migration parity instead of assuming a symmetric rollback.
This commit is contained in:
@@ -0,0 +1,143 @@
|
||||
# Runbook: Cloudflare data → AWS migration
|
||||
|
||||
This doc is the operator runbook for moving durable Cloudflare KV / D1 data into the live AWS DynamoDB shape used by `miti99bot`.
|
||||
|
||||
> Scope here is only durable state that the current Go runtime still reads. Do not bulk-copy legacy Cloudflare data.
|
||||
|
||||
## Live AWS target shape
|
||||
|
||||
DynamoDB runtime contract:
|
||||
- partition key: `pk = moduleName`
|
||||
- sort key: `sk = caller key`
|
||||
- payload attr: `value`
|
||||
|
||||
Examples:
|
||||
- `wordle` + `stats:<subject>`
|
||||
- `loldle` + `config:<subject>`
|
||||
- `lolschedule` + `subscribers`
|
||||
- `misc` + `last_ping`
|
||||
- `trading` + `user:<telegram_id>`
|
||||
|
||||
## Migration matrix (locked from live code)
|
||||
|
||||
| Source dataset / prefix | Current consumer | Action | AWS target | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `wordle:stats:*` | `internal/modules/wordle/state.go` | migrate | `pk=wordle`, `sk=stats:<subject>` | durable player stats |
|
||||
| `wordle:game:*` | `internal/modules/wordle/state.go` | skip | none | ephemeral in-flight game state |
|
||||
| `loldle:stats:*` | `internal/modules/loldle/state.go` | migrate | `pk=loldle`, `sk=stats:<subject>` | durable player stats |
|
||||
| `loldle:config:*` | `internal/modules/loldle/state.go` | migrate | `pk=loldle`, `sk=config:<subject>` | durable per-subject config |
|
||||
| `loldle:game:*` | `internal/modules/loldle/state.go` | skip | none | ephemeral in-flight game state |
|
||||
| `twentyq:stats:*` | `internal/modules/twentyq/state.go` | migrate | `pk=twentyq`, `sk=stats:<subject>` | durable player stats |
|
||||
| `twentyq:game:*` | `internal/modules/twentyq/state.go` | skip | none | ephemeral in-flight game state |
|
||||
| `lolschedule:subscribers` | `internal/modules/lolschedule/subscribers.go` | migrate | `pk=lolschedule`, `sk=subscribers` | durable subscriber list |
|
||||
| `lolschedule:matches:*` | `internal/modules/lolschedule/api_client.go` | skip | none | cache only |
|
||||
| `misc:last_ping` | `internal/modules/misc/misc.go` | migrate | `pk=misc`, `sk=last_ping` | `/mstats` still reads it |
|
||||
| `trading:user:*` or equivalent legacy portfolio state | `internal/modules/trading/portfolio.go` | migrate (transform) | `pk=trading`, `sk=user:<telegram_id>` | must become current `Portfolio` JSON |
|
||||
| `trading:sym:*` | `internal/modules/trading/symbols.go` | skip | none | cache only |
|
||||
| retired module namespaces (`loldle_emoji`, `loldle_quote`, `loldle_ability`, `loldle_splash`, `semantle`, `doantu`, etc.) | no live Go consumer | archive | none | export only if operator wants cold backup |
|
||||
|
||||
## Trading source inventory
|
||||
|
||||
What is confirmed in-repo:
|
||||
- legacy D1 table `trading_trades` exists
|
||||
- known columns from prior work: `id`, `user_id`, `symbol`, `side`, `qty`, `price_vnd`, `ts`
|
||||
- current AWS runtime target is `internal/modules/trading/portfolio.go`:
|
||||
- `currency map[string]float64`
|
||||
- `assets map[string]int64`
|
||||
- `meta.invested float64`
|
||||
- `meta.createdAt int64`
|
||||
|
||||
What is **not** confirmed in-repo:
|
||||
- authoritative legacy source table/column for `meta.invested`
|
||||
- authoritative legacy source table/column for `meta.createdAt`
|
||||
- whether old Cloudflare runtime stored a user portfolio snapshot separately from `trading_trades`
|
||||
- whether legacy `users` / `holdings` tables referenced in older plans still exist in production D1
|
||||
|
||||
## Phase 01 operator procedure
|
||||
|
||||
### 1) Inventory D1 tables
|
||||
|
||||
List all tables:
|
||||
|
||||
```sh
|
||||
wrangler d1 execute <database> --remote \
|
||||
--command "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name" \
|
||||
--json
|
||||
```
|
||||
|
||||
Dump table definitions for all discovered tables, then inspect every name that looks trading-, portfolio-, user-, or holding-related:
|
||||
|
||||
```sh
|
||||
wrangler d1 execute <database> --remote \
|
||||
--command "SELECT name, sql FROM sqlite_master WHERE type='table' ORDER BY name" \
|
||||
--json
|
||||
```
|
||||
|
||||
Inspect columns for every candidate table that could hold portfolio snapshots or metadata:
|
||||
|
||||
```sh
|
||||
wrangler d1 execute <database> --remote \
|
||||
--command "PRAGMA table_info(<table_name>)" \
|
||||
--json
|
||||
```
|
||||
|
||||
Do not stop at historical names like `trading_trades`, `users`, or `holdings`; the goal is to inspect whatever production D1 actually contains today.
|
||||
|
||||
### 2) Lock the trading transform inputs
|
||||
|
||||
Before any import code is written, prove one of these is true:
|
||||
- a portfolio snapshot table already exists and contains the exact source for `currency`, `assets`, `meta.invested`, and `meta.createdAt`, or
|
||||
- those fields must be derived, and the derivation is written down explicitly and accepted.
|
||||
|
||||
Minimum decision record to capture:
|
||||
- source table(s)
|
||||
- source column(s)
|
||||
- mapping rule into `Portfolio`
|
||||
- whether historical `trading_trades` rows are import inputs only or audit-only exports
|
||||
|
||||
### 3) Inventory Cloudflare KV prefixes
|
||||
|
||||
For each durable and skip candidate prefix above, list keys and capture representative values.
|
||||
Use Wrangler KV listing for at least:
|
||||
- `wordle:stats:`
|
||||
- `wordle:game:`
|
||||
- `loldle:stats:`
|
||||
- `loldle:config:`
|
||||
- `loldle:game:`
|
||||
- `twentyq:stats:`
|
||||
- `twentyq:game:`
|
||||
- `lolschedule:subscribers`
|
||||
- `lolschedule:matches:`
|
||||
- `misc:last_ping`
|
||||
- `trading:user:` (candidate only; inspect if a legacy portfolio snapshot prefix exists)
|
||||
- `trading:sym:`
|
||||
|
||||
### 4) Freeze the matrix
|
||||
|
||||
Do not proceed to import tooling until each discovered dataset is tagged as one of:
|
||||
- `migrate`
|
||||
- `skip`
|
||||
- `archive`
|
||||
|
||||
Anything not in the matrix is out of scope by default.
|
||||
|
||||
## Phase 01 done checklist
|
||||
|
||||
- [ ] Every live Cloudflare KV prefix is classified as `migrate`, `skip`, or `archive`
|
||||
- [ ] Every migrated KV dataset has an exact DynamoDB `(pk, sk)` target
|
||||
- [ ] Trading source table(s) and column(s) are locked
|
||||
- [ ] `meta.invested` source is explicit
|
||||
- [ ] `meta.createdAt` source is explicit
|
||||
- [ ] Retired module namespaces are explicitly excluded from runtime import
|
||||
- [ ] AWS cutover remains gated on a green parity report from the migration plan
|
||||
|
||||
## Current blocker
|
||||
|
||||
The repo proves the live AWS target shape and most KV policy, but it does **not** prove the authoritative legacy source for trading `meta.invested` and `meta.createdAt`. Phase 03 should stay blocked until the operator finishes the D1 inventory above.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Does production D1 still have `users` / `holdings` tables, or only `trading_trades`?
|
||||
2. If only `trading_trades` exists, what exact derivation should define `meta.createdAt`?
|
||||
3. If only `trading_trades` exists, should `meta.invested` be reconstructed from surviving positions only or from full historical cost basis?
|
||||
4. Are any retired Cloudflare KV namespaces still carrying data the operator wants archived before teardown?
|
||||
@@ -20,7 +20,7 @@ Stand up the AWS account with strict $0 footprint: IAM OIDC trust for GitHub Act
|
||||
```
|
||||
GitHub Actions ─OIDC─► AWS IAM Role (github-deploy)
|
||||
│ └─ trust: token.actions.githubusercontent.com
|
||||
│ └─ scoped: repo:tiennm99/miti99bot-go:ref:refs/heads/main
|
||||
│ └─ scoped: repo:tiennm99/miti99bot:ref:refs/heads/main
|
||||
│
|
||||
└─► CloudFormation (SAM) ─► Lambda + DynamoDB + ParamStore + EventBridge + Logs
|
||||
```
|
||||
@@ -36,7 +36,7 @@ GitHub Actions ─OIDC─► AWS IAM Role (github-deploy)
|
||||
## Implementation Steps
|
||||
1. Create AWS account (or reuse existing). Enable MFA on root, create IAM admin user for one-time bootstrap. Set region default `ap-southeast-1`.
|
||||
2. Create the GitHub OIDC identity provider in IAM: thumbprint, audience `sts.amazonaws.com`. (One-time, manual or via small CloudFormation snippet.)
|
||||
3. Create IAM role `github-deploy-miti99bot` with trust policy scoped to `repo:tiennm99/miti99bot-go:ref:refs/heads/main` and `repo:tiennm99/miti99bot-go:ref:refs/heads/dev`. Attach managed policies for SAM deploy: CloudFormation, Lambda, DynamoDB, EventBridge, IAM (PassRole only), SSM Parameter Store, Logs, S3 (SAM staging bucket).
|
||||
3. Create IAM role `github-deploy-miti99bot` with trust policy scoped to `repo:tiennm99/miti99bot:ref:refs/heads/main` and `repo:tiennm99/miti99bot:ref:refs/heads/dev`. Attach managed policies for SAM deploy: CloudFormation, Lambda, DynamoDB, EventBridge, IAM (PassRole only), SSM Parameter Store, Logs, S3 (SAM staging bucket).
|
||||
4. Write `template.yaml` skeleton:
|
||||
- `AWSTemplateFormatVersion: '2010-09-09'`, `Transform: AWS::Serverless-2016-10-31`
|
||||
- `Globals.Function`: `Runtime: provided.al2023`, `Architectures: [arm64]`, `MemorySize: 256`, `Timeout: 15`, `Tracing: Active` (still free at this volume)
|
||||
|
||||
@@ -37,7 +37,7 @@ GitHub push to main
|
||||
- Modify: `Makefile` — `deploy` target runs `sam build && sam deploy --no-confirm-changeset`
|
||||
|
||||
## Implementation Steps
|
||||
1. Confirm Phase 01's IAM role trust policy includes `repo:tiennm99/miti99bot-go:ref:refs/heads/main` and `repo:tiennm99/miti99bot-go:pull_request` (for PR validation flow if `sam validate` against AWS).
|
||||
1. Confirm Phase 01's IAM role trust policy includes `repo:tiennm99/miti99bot:ref:refs/heads/main` and the matching repo subject for any manual deploy path.
|
||||
2. Write `deploy.yml`:
|
||||
```yaml
|
||||
name: Deploy to AWS
|
||||
@@ -59,7 +59,7 @@ GitHub push to main
|
||||
with: { use-installer: true }
|
||||
- uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-deploy-miti99bot
|
||||
role-to-assume: arn:aws:iam::225603493174:role/github-deploy-miti99bot
|
||||
aws-region: ap-southeast-1
|
||||
- run: make build-lambda
|
||||
- run: sam build
|
||||
@@ -69,7 +69,7 @@ GitHub push to main
|
||||
URL=$(aws cloudformation describe-stacks --stack-name miti99bot-aws-port --query "Stacks[0].Outputs[?OutputKey=='FunctionUrl'].OutputValue" --output text)
|
||||
curl -fsSL "$URL/" | jq .
|
||||
```
|
||||
3. Add `AWS_ACCOUNT_ID` to GitHub repo secrets (it's not a credential, but keeping the ARN out of the repo file is hygiene).
|
||||
3. Keep the AWS account ID in the committed role ARN for this repo. If the deploy account changes later, update both the workflow ARN and the IAM trust policy together.
|
||||
4. PR validation workflow (`ci.yml`): runs `go vet`, `go test`, `sam validate` (no AWS creds needed for validate).
|
||||
5. Test the full path: open a PR with a trivial change → CI green; merge → deploy fires; smoke step prints health JSON.
|
||||
6. Add a `rollback.yml` workflow_dispatch path: re-run with a chosen commit SHA. CloudFormation handles the rollback inherently.
|
||||
|
||||
@@ -10,21 +10,23 @@ dependencies: [2, 3, 4, 5, 6]
|
||||
# Phase 07: Cutover + README + retire GCP paths
|
||||
|
||||
## Overview
|
||||
Flip the production Telegram webhook to the AWS Function URL, soak for 7 days, then mark the AWS port as default in README and code defaults. Keep GCP code paths in tree (Firestore impl, Cloud Run Dockerfile) but unwired by default.
|
||||
Flip the production Telegram webhook to the AWS Function URL only after the Cloudflare→AWS migration plan has produced a green parity report and a rehearsed final-delta procedure. Keep GCP code paths in tree (Firestore impl, Cloud Run Dockerfile) but unwired by default.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Real production bot serves users from Lambda. No regressions vs prior baseline (whatever ran before — JS Worker or partial GCP).
|
||||
- **Non-functional:** Zero downtime cutover (Telegram webhook flip is atomic). 7-day soak with logs reviewed daily. Fallback path documented.
|
||||
- **Functional:** Real production bot serves users from Lambda with durable Cloudflare data already migrated or intentionally archived. No regressions vs prior baseline (whatever ran before — JS Worker or partial GCP).
|
||||
- **Non-functional:** Accept a brief operator-controlled freeze window for the final delta import; no silent data loss. 7-day soak with logs reviewed daily. Fallback path documented.
|
||||
|
||||
## Architecture
|
||||
- **Cutover op:** `setWebhook` Telegram API call pointing to the AWS Function URL with the production webhook secret.
|
||||
- **Rollback path:** identical `setWebhook` call to the prior URL; ~5 second op.
|
||||
- **Migration gate:** `plans/260515-2250-cf-data-to-aws-migration/` must finish first; Phase 04 parity report there is the go/no-go input for this phase.
|
||||
- **Freeze-window cutover:** pause Cloudflare cron/webhook writes, run final delta export/import + verify, then call `setWebhook` to point Telegram at the AWS Function URL with the production webhook secret.
|
||||
- **Rollback path:** if the final delta verify fails or AWS smoke fails before the first AWS-served write, restore the prior webhook target. After AWS starts accepting new writes, this cutover is forward-fix only unless a reverse-sync path exists.
|
||||
- **Code:** Firestore impl stays compilable, gated by `KV_PROVIDER=firestore`. Default `KV_PROVIDER=dynamodb` in Lambda env. Cloud Run Dockerfile retained for offline / non-AWS users.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `README.md` — full rewrite of "Run locally", "Build", new "Deploy to AWS" section, status table updated, link to AWS plan, archive link to GCP plan
|
||||
- Modify: `cmd/server/main.go` — default `KV_PROVIDER` selection logic: `dynamodb` if `AWS_LAMBDA_FUNCTION_NAME` set, else `memory`
|
||||
- Create: `docs/deploy-aws.md` — single source of truth for AWS deploy ops (parameter store names, IAM role ARN, smoke commands)
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md` — freeze-window delta import + rollback sequence
|
||||
- Modify: `plans/260508-2222-go-port-cloud-run/plan.md` — top-of-file note: "Deploy phases 01, 09–12 superseded by `plans/260510-0114-aws-port/`. Module work (phases 03–07) reused unchanged."
|
||||
- Optional remove: `Dockerfile` retained for now; revisit in 30 days
|
||||
- Optional remove: GCP-specific docs in `docs/` if any (none observed)
|
||||
@@ -37,7 +39,10 @@ Flip the production Telegram webhook to the AWS Function URL, soak for 7 days, t
|
||||
- [ ] Phase 05 push-to-main auto-deploys
|
||||
- [ ] Phase 06 budget alert email confirmed
|
||||
- [ ] Cold-start P95 < 1.5s confirmed
|
||||
2. Run `setWebhook` against production bot:
|
||||
- [ ] `plans/260515-2250-cf-data-to-aws-migration/phase-04-parity-verification-and-rehearsal.md` passed with a saved green report
|
||||
- [ ] Final delta import commands rehearsed during the freeze window
|
||||
2. Pause Cloudflare writes and run the final delta import + verify.
|
||||
3. Run `setWebhook` against production bot:
|
||||
```sh
|
||||
curl -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/setWebhook" \
|
||||
-d "url=$AWS_FUNCTION_URL/webhook" \
|
||||
@@ -45,24 +50,26 @@ Flip the production Telegram webhook to the AWS Function URL, soak for 7 days, t
|
||||
-d "drop_pending_updates=false" \
|
||||
-d "allowed_updates=[\"message\",\"callback_query\"]"
|
||||
```
|
||||
3. Verify with `getWebhookInfo`:
|
||||
4. Verify with `getWebhookInfo`:
|
||||
```sh
|
||||
curl "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getWebhookInfo" | jq .
|
||||
```
|
||||
Confirm `url`, `pending_update_count` near 0, `last_error_date` empty.
|
||||
4. Send a test command (`/start`, `/wordle`, `/twentyq` to exercise Gemini path). Confirm responses match prior behavior.
|
||||
5. Soak for 7 days: each morning, check CloudWatch Logs for ERROR / WARN, DynamoDB throttle metrics (should be zero), budget current spend (should be $0), Gemini RPD usage (should be far under cap).
|
||||
6. After 7-day soak, update README:
|
||||
5. Send a test command (`/start`, `/wordle`, `/twentyq` to exercise Gemini path). Confirm responses match prior behavior and expected migrated state is visible.
|
||||
6. Verify at least one migrated trading account, one existing lolschedule subscriber path, and `/mstats` if `last_ping` was migrated.
|
||||
7. Soak for 7 days: each morning, check CloudWatch Logs for ERROR / WARN, DynamoDB throttle metrics (should be zero), budget current spend (should be $0), Gemini RPD usage (should be far under cap).
|
||||
8. After 7-day soak, update README:
|
||||
- Status table: AWS port phases marked "done"
|
||||
- Replace "Run locally" with two paths: in-memory (no AWS) and DynamoDB Local (with AWS deps)
|
||||
- "Deploy" section: link `docs/deploy-aws.md`, drop Cloud Run instructions
|
||||
- Status badge / link to `plans/260510-0114-aws-port/`
|
||||
7. Add a top-of-file note in `plans/260508-2222-go-port-cloud-run/plan.md` redirecting deploy questions to the AWS plan.
|
||||
8. Tag a release: `git tag v1.0.0-aws -m "AWS deploy default"` and push.
|
||||
9. Add a top-of-file note in `plans/260508-2222-go-port-cloud-run/plan.md` redirecting deploy questions to the AWS plan.
|
||||
10. Tag a release: `git tag v1.0.0-aws -m "AWS deploy default"` and push.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Telegram webhook `getWebhookInfo` shows AWS Function URL
|
||||
- [ ] Production bot answers `/start` from real users with normal latency
|
||||
- [ ] Production bot answers `/start` from real users with normal latency and expected migrated data
|
||||
- [ ] Final Cloudflare→AWS migration report is green before any CF teardown
|
||||
- [ ] 7-day soak: zero unrecovered errors, zero throttles, zero unexpected spend
|
||||
- [ ] README accurately reflects AWS as the default deploy
|
||||
- [ ] `docs/deploy-aws.md` is sufficient for a fresh dev to redeploy from scratch
|
||||
@@ -70,7 +77,7 @@ Flip the production Telegram webhook to the AWS Function URL, soak for 7 days, t
|
||||
- [ ] Release tag pushed
|
||||
|
||||
## Risk Assessment
|
||||
- **Webhook flip causes message loss** during the few-second propagation — Telegram retries failed deliveries automatically; `drop_pending_updates=false` preserves backlog.
|
||||
- **Final import misses writes** if Cloudflare stays writable during cutover — Mitigation: use the migration plan's freeze-window delta import before `setWebhook`, then verify parity again.
|
||||
- **Latency regression vs JS Worker** — Cloud Run / JS Worker had different cold-start profiles; if users complain, document and consider ARM→x86 swap or provisioned concurrency (kills free tier).
|
||||
- **Hidden Firestore dependency** still wired in some module — Mitigation: grep for `firestore.NewClient` and confirm all paths are gated by `KV_PROVIDER=firestore` env. Add a CI test that builds with `KV_PROVIDER=dynamodb` and asserts Firestore client is not initialized.
|
||||
- **GCP project quietly billing** because resources weren't deleted — Mitigation: explicit step in this phase: `gcloud projects delete <project>` OR `gcloud run services delete` for any deployed services. Check Cloud Console for any orphaned resources.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: "Migrate miti99bot-go from GCP to AWS (Lambda + DynamoDB + EventBridge, free tier)"
|
||||
title: "Migrate miti99bot from GCP to AWS (Lambda + DynamoDB + EventBridge, free tier)"
|
||||
description: "Re-target the deploy/runtime layer from Cloud Run + Firestore + Cloud Scheduler to Lambda (Go ZIP + LWA + Function URL) + DynamoDB on-demand + EventBridge Scheduler, region ap-southeast-1, IaC via SAM, CI via GH Actions OIDC. Module code unchanged."
|
||||
status: in-progress
|
||||
priority: P2
|
||||
@@ -7,7 +7,7 @@ effort: 3-4d
|
||||
branch: main
|
||||
tags: [aws, lambda, dynamodb, eventbridge, sam, port, telegram-bot, free-tier]
|
||||
created: 2026-05-10
|
||||
blockedBy: []
|
||||
blockedBy: [260515-2250-cf-data-to-aws-migration]
|
||||
blocks: []
|
||||
supersedes-deploy-of: [260508-2222-go-port-cloud-run]
|
||||
---
|
||||
@@ -24,7 +24,7 @@ Re-target only the deploy/runtime layer. Module work (Phases 03–07 of GCP plan
|
||||
## Locked decisions
|
||||
- **Compute:** Lambda Go on `provided.al2023`, **ARM64**, ZIP package, binary `bootstrap`, build with `-tags lambda.norpc -ldflags="-s -w"`.
|
||||
- **HTTP:** Lambda Function URL (`AuthType: NONE`) + AWS Lambda Web Adapter layer → existing `http.Handler` runs unchanged.
|
||||
- **KV:** DynamoDB single-table `miti99bot`, PK=`pk` (`{module}#{key}`), attr `value` (Binary). On-demand billing.
|
||||
- **KV:** DynamoDB single-table `miti99bot`, composite key `(pk, sk)` where `pk = moduleName` and `sk = caller key`, attr `value` (Binary). On-demand billing.
|
||||
- **Cron:** EventBridge Scheduler → HTTPS target = Function URL `/cron/{name}` with `X-Cron-Token` header (token in Parameter Store). Preserves existing route shape; alternative (direct Lambda invoke) deferred.
|
||||
- **Secrets:** SSM Parameter Store SecureString. Names: `/miti99bot/{env}/telegram-token`, `…/webhook-secret`, `…/gemini-api-key`, `…/cron-token`. Fetched at cold start.
|
||||
- **Region:** `ap-southeast-1` (Singapore).
|
||||
@@ -43,7 +43,7 @@ Re-target only the deploy/runtime layer. Module work (Phases 03–07 of GCP plan
|
||||
| 04 | [EventBridge cron wiring](phase-04-eventbridge-cron.md) | pending (blocked on cron handlers, see 260510-0234-pre-deploy-wrapup) | 3h | Scheduler → `/cron/{name}` with token, two crons firing on schedule |
|
||||
| 05 | [GitHub Actions deploy (OIDC + SAM)](phase-05-gha-deploy.md) | done | 3h | `deploy.yml` runs on push to `main`, builds + sam deploys idempotently |
|
||||
| 06 | [Observability + budget alert](phase-06-observability.md) | partial (budget shipped; metric filter in 260510-0234) | 2h | Logs retention set, $1 budget alert, cold-start P95 captured |
|
||||
| 07 | [Cutover + README + retire GCP paths](phase-07-cutover.md) | pending (deploy-gated) | 3h | Webhook flipped to Function URL, README rewritten, GCP code paths kept but unwired by default |
|
||||
| 07 | [Cutover + README + retire GCP paths](phase-07-cutover.md) | pending (deploy + migration gated) | 3h | Webhook flipped to Function URL after green CF→AWS migration report; README rewritten; GCP code paths kept but unwired by default |
|
||||
|
||||
## Dependency graph
|
||||
```
|
||||
@@ -69,7 +69,7 @@ Re-target only the deploy/runtime layer. Module work (Phases 03–07 of GCP plan
|
||||
- **Function URL auth-bypass risk** discovered: switch to API Gateway HTTP API (12-month free, then $1/M).
|
||||
|
||||
## Rollback
|
||||
Until Phase 07 webhook flip, the GCP runtime path remains intact. Per-phase rollback documented in each phase. Phase 07 is the only irreversible cutover step.
|
||||
Until Phase 07 webhook flip, the GCP runtime path remains intact. Per-phase rollback documented in each phase. Phase 07 now depends on `plans/260515-2250-cf-data-to-aws-migration/` producing a green parity report before the webhook moves or any Cloudflare data source is deleted.
|
||||
|
||||
## Open questions
|
||||
1. Direct Lambda invoke for cron vs HTTP loopback via Function URL — final call deferred to Phase 04 implementation.
|
||||
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Source inventory and migration policy"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 01: Source inventory and migration policy
|
||||
|
||||
## Overview
|
||||
Identify the exact Cloudflare KV namespaces and D1 tables still carrying production data, then lock a per-key policy: migrate, skip, or archive. The goal is to prevent a noisy "copy everything" migration that drags stale caches, retired modules, or incompatible schemas into DynamoDB.
|
||||
|
||||
## Requirements
|
||||
- Functional: produce a concrete inventory of live CF data sources, active key prefixes, D1 tables, and the AWS target shape for each kept dataset.
|
||||
- Non-functional: decisions are explicit, reversible, and tied to current code paths — not guesses from old plans.
|
||||
|
||||
## Architecture
|
||||
- Inspect current AWS consumers first: `wordle stats:*`, `loldle stats:*` / `config:*`, `twentyq stats:*`, `lolschedule subscribers`, `misc:last_ping`, `trading user:*` portfolios.
|
||||
- Inspect legacy CF sources second: KV namespace(s), D1 trading tables, and any retired-module prefixes still present.
|
||||
- Lock default policy:
|
||||
- **Migrate:** long-lived, user-visible state.
|
||||
- **Skip:** `game:*`, `matches:*`, `sym:*`, other caches.
|
||||
- **Archive-only:** retired modules and optional historical trade rows not consumed by current AWS runtime.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Modify: `plans/260510-0114-aws-port/phase-07-cutover.md`
|
||||
- Read only: `internal/modules/wordle/state.go`, `internal/modules/loldle/state.go`, `internal/modules/twentyq/state.go`, `internal/modules/lolschedule/subscribers.go`, `internal/modules/trading/portfolio.go`
|
||||
|
||||
## Implementation Steps
|
||||
1. Enumerate current AWS key shapes from live code.
|
||||
2. Pull a source inventory from Cloudflare KV and D1 using operator credentials.
|
||||
3. Build a migration matrix: source dataset → target DynamoDB key → action (`migrate|skip|archive`).
|
||||
4. Lock the exact D1 source tables/columns for `Portfolio.Meta.CreatedAt` and `Portfolio.Meta.Invested`.
|
||||
5. Mark retired namespaces explicitly so they are not silently reintroduced.
|
||||
6. Update the AWS cutover phase to say final webhook flip is gated on this migration matrix.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Every live CF dataset is classified as migrate, skip, or archive.
|
||||
- [ ] Every migrated dataset has an explicit AWS target key shape.
|
||||
- [ ] Trading `meta.createdAt` and `meta.invested` have authoritative source fields.
|
||||
- [ ] Retired-module data is excluded by policy.
|
||||
- [ ] The cutover plan references this migration gate.
|
||||
|
||||
## Risk Assessment
|
||||
Main risk is misclassifying a dataset as disposable when users still care about it. Mitigation: classify by current runtime consumers first, then validate Cloudflare inventory against those exact consumers before any tooling is written.
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Backfill toolchain and safety rails"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "3-4h"
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 02: Backfill toolchain and safety rails
|
||||
|
||||
## Overview
|
||||
Build operator-run migration tooling in Go that can read legacy Cloudflare data, write DynamoDB records idempotently, and support dry-runs plus checkpoints. This phase is about controlled mechanics, not the actual production import yet.
|
||||
|
||||
## Requirements
|
||||
- Functional: provide commands for KV export/import, trading import, and parity verification inputs.
|
||||
- Non-functional: idempotent writes, dry-run mode, resumable progress, zero admin HTTP surface, and no dependency on the running AWS bot process.
|
||||
|
||||
## Architecture
|
||||
- Keep tooling inside this repo and language stack, but keep it small:
|
||||
- `cmd/migrate_cf_data/` for inventory + KV import + trading import modes
|
||||
- `cmd/verify_cf_aws_parity/` for verification only
|
||||
- Shared logic lives under `internal/migration/` for Cloudflare REST reads, DynamoDB writes, and report formatting.
|
||||
- D1 source extraction stays simple: operator uses `wrangler d1 execute ... --json --remote` to create local JSON exports; Go import code consumes those files instead of re-implementing remote SQL access.
|
||||
- Checkpoint/resume is conditional: add it only if Phase 01 proves the keyspace is large enough to justify it.
|
||||
- No writes happen during `--dry-run`; output is a machine-readable summary plus human-readable progress logs.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/migrate_cf_data/main.go`
|
||||
- Create: `cmd/verify_cf_aws_parity/main.go`
|
||||
- Create: `internal/migration/cloudflare_kv_client.go`
|
||||
- Create: `internal/migration/dynamodb_writer.go`
|
||||
- Create: `internal/migration/report.go`
|
||||
- Optional create: `internal/migration/checkpoint_store.go`
|
||||
- Modify: `go.mod`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. Define CLI flags and env contract for Cloudflare and AWS credentials.
|
||||
2. Implement KV list/get readers against Cloudflare REST with pagination support.
|
||||
3. Implement DynamoDB writers against the live runtime shape: `pk = moduleName`, `sk = caller key`.
|
||||
4. Add checkpoint files only if Phase 01 proves resume support is worth the extra surface area.
|
||||
5. Add dry-run and report output before any real import path is allowed.
|
||||
6. Document the exact operator workflow in the runbook.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Tooling reads CF KV metadata and values without touching app code paths.
|
||||
- [ ] Trading import mode accepts local D1 JSON exports.
|
||||
- [ ] Every command supports `--dry-run`.
|
||||
- [ ] Import path is idempotent or safely merge-based.
|
||||
- [ ] Checkpoint/resume behavior is either justified and documented or intentionally omitted.
|
||||
|
||||
## Risk Assessment
|
||||
The main risk is embedding too much migration logic into one giant binary. Mitigation: split command entrypoints and keep shared logic in small `internal/migration/` helpers so each command stays reviewable and under the repo's file-size guidance.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Trading and durable KV import"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "4-6h"
|
||||
dependencies: [1, 2]
|
||||
---
|
||||
|
||||
# Phase 03: Trading and durable KV import
|
||||
|
||||
## Overview
|
||||
Run the first real data movement into DynamoDB: import durable KV records and transform old D1 trading data into the current AWS portfolio shape. This phase intentionally does not move ephemeral state or try to preserve abandoned feature surfaces.
|
||||
|
||||
## Requirements
|
||||
- Functional: import approved KV keys plus trading data into DynamoDB while preserving the full current `Portfolio` contract, not just balances/holdings.
|
||||
- Non-functional: preserve current runtime schemas exactly, avoid duplicate writes on rerun, and keep historical-only data out of the hot path unless explicitly requested.
|
||||
|
||||
## Architecture
|
||||
- KV import target uses the same live DynamoDB shape the runtime already expects: `pk = moduleName`, `sk = caller key`.
|
||||
- Durable key policy from Phase 01 drives import filters:
|
||||
- keep `stats:*`, `config:*`, `subscribers`, `last_ping`, approved trading records
|
||||
- reject `game:*`, `matches:*`, `sym:*`
|
||||
- Trading import is a transform:
|
||||
- source: D1 exports from the exact authoritative tables locked in Phase 01
|
||||
- target: `trading` module KV entries with keys `user:<telegram_id>`
|
||||
- value: `internal/modules/trading/portfolio.go` JSON shape (`currency`, `assets`, `meta`)
|
||||
- `meta.createdAt` and `meta.invested` must be mapped from explicit source fields or derivations approved in Phase 01 before this phase starts
|
||||
- Historical trade rows are exported for audit only unless the user later asks to restore history as a separate feature.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `cmd/migrate_cf_data/main.go` — add the trading import mode once Phase 01 locks authoritative source tables
|
||||
- Create: `internal/migration/trading_transform.go`
|
||||
- Create: `internal/migration/kv_filter.go`
|
||||
- Create: `internal/migration/import_report.go`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Read only: `internal/modules/trading/portfolio.go`, `internal/modules/wordle/state.go`, `internal/modules/loldle/state.go`, `internal/modules/twentyq/state.go`, `internal/modules/lolschedule/subscribers.go`
|
||||
|
||||
## Implementation Steps
|
||||
1. Implement KV filters from the Phase 01 migration matrix.
|
||||
2. Build the D1-to-portfolio transform using the current `Portfolio` JSON contract.
|
||||
3. Import durable KV records into DynamoDB using module/key parity.
|
||||
4. Import transformed trading portfolios into the `trading` module namespace.
|
||||
5. Emit an import report with counts for imported, skipped, archived, and failed records.
|
||||
6. Capture raw trading exports locally for audit if retention is needed outside runtime state.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Durable KV keys land in DynamoDB under the exact runtime key names.
|
||||
- [ ] Trading users get correct `Portfolio` JSON records, including `meta.createdAt` and `meta.invested`.
|
||||
- [ ] Re-running the import does not duplicate or corrupt data.
|
||||
- [ ] Skipped datasets are reported explicitly, not silently dropped.
|
||||
- [ ] Historical-only trading rows are archived or intentionally ignored by policy.
|
||||
|
||||
## Risk Assessment
|
||||
Biggest risk is a wrong trading transform that gives users the wrong balances or holdings. Mitigation: derive the target object from the current `Portfolio` type only, spot-check several real users, and keep raw D1 exports so any discrepancy can be recomputed without touching Cloudflare again.
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Parity verification and rehearsal"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: [2, 3]
|
||||
---
|
||||
|
||||
# Phase 04: Parity verification and rehearsal
|
||||
|
||||
## Overview
|
||||
Prove the imported AWS data matches the Cloudflare source closely enough to trust a real cutover. This phase turns migration from a one-off script run into a repeatable, auditable procedure with a staging-table rehearsal.
|
||||
|
||||
## Requirements
|
||||
- Functional: verify counts, sample payload parity, and trading portfolio correctness between CF exports and DynamoDB.
|
||||
- Non-functional: produce a saved report, support reruns, and define rollback steps before the production webhook is moved.
|
||||
|
||||
## Architecture
|
||||
- Verifier compares source exports against the AWS target table using the same module/key selectors from Phase 01.
|
||||
- Checks by dataset type:
|
||||
- KV durable records: count parity + payload/hash comparisons
|
||||
- trading portfolios: count parity + deep field comparison on currency, assets, and invested metadata
|
||||
- Rehearsal happens against a staging DynamoDB table only. No destructive rerun path is allowed against the live table.
|
||||
- Final output is a migration report under `plans/reports/` plus runbook updates.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/verify_cf_aws_parity/main.go`
|
||||
- Create: `internal/migration/parity_checks.go`
|
||||
- Create: `internal/migration/rollback_scope.go`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Create during execution: `plans/reports/migration-260515-2250-cf-data-to-aws-parity.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. Implement count and payload verification per migrated dataset.
|
||||
2. Add trading-specific deep checks against the current `Portfolio` shape.
|
||||
3. Save the verifier result as a markdown report under `plans/reports/`.
|
||||
4. Rehearse import + verify against a staging DynamoDB table.
|
||||
5. Promote the exact same procedure to the live table only after staging is green.
|
||||
6. Mark the migration runbook ready only after a green verifier report.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Verifier reports pass for all migrated datasets.
|
||||
- [ ] Trading portfolios match expected balances and holdings on spot checks.
|
||||
- [ ] A staging-table rehearsal completes successfully without touching the live table.
|
||||
- [ ] The migration report is saved and linked from the runbook.
|
||||
- [ ] The cutover checklist now depends on a green parity report.
|
||||
|
||||
## Risk Assessment
|
||||
The main risks are false confidence from count-only checks and accidental destructive rehearsal against production storage. Mitigation: include dataset-specific deep comparisons, especially for trading portfolios, and require a saved report plus staging-table-only rehearsal before the cutover phase can begin.
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Cutover integration and Cloudflare decommission"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: [1, 2, 3, 4]
|
||||
---
|
||||
|
||||
# Phase 05: Cutover integration and Cloudflare decommission
|
||||
|
||||
## Overview
|
||||
Fold the verified migration into the AWS cutover path, then decommission Cloudflare resources only after a successful freeze-window migration and AWS soak. This phase closes the data-consistency gap left by the original AWS port plan.
|
||||
|
||||
## Requirements
|
||||
- Functional: production cutover moves webhook ownership to AWS without losing durable writes from the old Cloudflare stack.
|
||||
- Non-functional: rollback is fast only before the first AWS-served write; after that, the plan is forward-fix only unless reverse sync is built later. Cloudflare resources are deleted only after verification, and operator steps are explicit.
|
||||
|
||||
## Architecture
|
||||
- There is no legacy dual-write path, so final cutover uses a short **write-freeze window**:
|
||||
1. disable/pause Cloudflare cron triggers
|
||||
2. stop Cloudflare webhook intake so no new writes land there
|
||||
3. run final delta export/import + parity verify
|
||||
4. point Telegram webhook to AWS
|
||||
5. begin AWS soak
|
||||
- Before the first AWS-served write, rollback is still a webhook restore.
|
||||
- After the first AWS-served write, rollback is not symmetry; it becomes forward-fix only unless a reverse-sync mechanism exists.
|
||||
- This keeps migration correctness simple and avoids inventing temporary cross-runtime replication.
|
||||
- Cloudflare teardown is a separate final step after the AWS soak, not part of the initial webhook flip.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `plans/260510-0114-aws-port/plan.md`
|
||||
- Modify: `plans/260510-0114-aws-port/phase-07-cutover.md`
|
||||
- Modify: `docs/deploy-aws.md`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Optional create: `docs/cf-decommission-checklist.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. Update the AWS cutover phase to depend on a green migration report.
|
||||
2. Add the freeze-window sequence and pre-flip vs post-flip rollback semantics to the runbook.
|
||||
3. Define the final delta import and verification command set.
|
||||
4. Flip the Telegram webhook only after the final delta verify succeeds.
|
||||
5. Add post-flip smoke checks for a migrated trading account, an existing lolschedule subscriber, and `/mstats` if `last_ping` is kept.
|
||||
6. Soak on AWS, then remove CF Worker/KV/D1 only when rollback is no longer needed.
|
||||
7. Archive or document any intentionally skipped legacy datasets before teardown.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] AWS cutover docs explicitly require a green migration report.
|
||||
- [ ] Freeze-window steps are documented end-to-end.
|
||||
- [ ] Pre-flip rollback and post-flip forward-fix semantics are documented explicitly.
|
||||
- [ ] Final delta import and rollback commands are ready before webhook flip.
|
||||
- [ ] Cloudflare resources are not deleted during the initial cutover window.
|
||||
- [ ] After soak, CF teardown is documented and low-risk.
|
||||
|
||||
## Risk Assessment
|
||||
The biggest risk is write drift between an early backfill and the final webhook flip. Mitigation: use a short freeze window for the last delta import instead of trying to add temporary dual-write behavior to a legacy system that lives outside this repo.
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
title: "Migrate Cloudflare data to AWS DynamoDB"
|
||||
description: "Export durable data from the legacy Cloudflare Worker stack, import it into the live AWS DynamoDB store, verify parity, and gate final cutover on a proven migration runbook."
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 1-2d
|
||||
branch: main
|
||||
tags: [migration, cloudflare, aws, dynamodb, cutover, data]
|
||||
created: 2026-05-15
|
||||
blockedBy: []
|
||||
blocks: [260510-0114-aws-port]
|
||||
---
|
||||
|
||||
# Plan: Cloudflare data → AWS DynamoDB
|
||||
|
||||
This plan adds the missing data-migration leg to the in-progress AWS port. AWS runtime + DynamoDB already exist; the gap is getting durable user data out of the legacy Cloudflare KV/D1 stack before final decommission.
|
||||
|
||||
## Why separate plan
|
||||
- `plans/260510-0114-aws-port/` covers runtime + deploy cutover.
|
||||
- This plan covers source-data inventory, export/import tooling, parity verification, and rollback.
|
||||
- `aws-port` should not be considered done until this plan passes.
|
||||
|
||||
## Locked decisions
|
||||
- Migrate **durable user-visible data only**.
|
||||
- Skip ephemeral or disposable data: in-flight game state, schedule caches, stale price caches.
|
||||
- Keep `misc:last_ping` unless the user explicitly accepts that reset; `/mstats` reads it today.
|
||||
- No admin HTTP routes. Migration runs as operator-invoked one-shot tooling.
|
||||
- Rehearsal uses a staging DynamoDB table only. No wipe-and-rerun flow is allowed against the live table.
|
||||
- Trading import is a **transform**, not a table copy: old D1 rows must become current KV portfolio JSON in the `trading` module.
|
||||
- After the first AWS-served write, rollback is forward-fix only unless a reverse-sync path is built later.
|
||||
- Retired module namespaces from the old CF stack are archived or ignored, not revived into AWS.
|
||||
|
||||
## Source data classes to classify in Phase 01
|
||||
- Migrate: `wordle stats:*`, `loldle stats:*`, `loldle config:*`, `twentyq stats:*`, `lolschedule subscribers`, `misc:last_ping`, trading balances/holdings.
|
||||
- Skip by default: `game:*`, `matches:*`, `sym:*`.
|
||||
- Decide explicitly: historical trading ledger rows, retired-module data, and the authoritative D1 fields for trading `meta.createdAt` + `meta.invested`.
|
||||
|
||||
## Related current code
|
||||
- `cmd/server/main.go:167` — runtime storage backend selection (`dynamodb|firestore|memory`)
|
||||
- `internal/storage/dynamodb_provider.go:7` — live DynamoDB partitioning (`pk = moduleName`)
|
||||
- `internal/storage/dynamodb_kv.go:24` — live DynamoDB sort-key contract (`sk = caller key`)
|
||||
- `internal/modules/wordle/state.go:50` — `game:*` + `stats:*`
|
||||
- `internal/modules/loldle/state.go:48` — `game:*`, `stats:*`, `config:*`
|
||||
- `internal/modules/twentyq/state.go:37` — `game:*` + `stats:*`
|
||||
- `internal/modules/lolschedule/subscribers.go:14` — `subscribers`
|
||||
- `internal/modules/trading/portfolio.go:39` — current AWS target shape: per-user KV portfolio JSON
|
||||
- `plans/260508-2222-go-port-cloud-run/phase-12-cutover.md:37` — prior CF→Go cutover notes (trading-only import assumption)
|
||||
- `plans/260510-0114-aws-port/phase-07-cutover.md:13` — current AWS final cutover phase
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Key deliverable |
|
||||
|---|-------|--------|--------|-----------------|
|
||||
| 01 | [Source inventory and migration policy](phase-01-source-inventory-and-migration-policy.md) | pending | 2-3h | exact CF namespaces/tables mapped to migrate vs skip vs archive |
|
||||
| 02 | [Backfill toolchain and safety rails](phase-02-backfill-toolchain-and-safety-rails.md) | pending | 3-4h | operator-run export/import binaries + dry-run support |
|
||||
| 03 | [Trading and durable KV import](phase-03-trading-and-durable-kv-import.md) | pending | 4-6h | transformed trading portfolios + durable KV records loaded into DynamoDB |
|
||||
| 04 | [Parity verification and rehearsal](phase-04-parity-verification-and-rehearsal.md) | pending | 2-3h | repeatable verifier, mismatch report, rollback drill |
|
||||
| 05 | [Cutover integration and Cloudflare decommission](phase-05-cutover-integration-and-cloudflare-decommission.md) | pending | 2-3h | AWS cutover checklist updated; CF teardown gated on verified migration |
|
||||
|
||||
## Key dependencies
|
||||
- Blocks: `plans/260510-0114-aws-port/phase-07-cutover.md`
|
||||
- Uses the already-live AWS target from `plans/260510-0114-aws-port/`
|
||||
- Should finish before deleting CF Worker/KV/D1 resources referenced in `plans/260508-2222-go-port-cloud-run/phase-12-cutover.md`
|
||||
|
||||
## Success bar
|
||||
- Durable CF data imported into DynamoDB with counts + sampled payload parity.
|
||||
- Trading balances/holdings and required portfolio metadata match the old system after transform.
|
||||
- Cutover runbook explicitly distinguishes pre-flip rollback from post-flip forward-fix semantics.
|
||||
- CF resources are not deleted until parity report is green.
|
||||
Reference in New Issue
Block a user