mirror of
https://github.com/tiennm99/miti99bot.git
synced 2026-09-10 02:17:35 +00:00
chore(plans): remove completed planning artifacts
This commit is contained in:
@@ -1,63 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "GCP setup + free-tier baseline"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "3h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 01: GCP setup + free-tier baseline
|
||||
|
||||
## Overview
|
||||
Stand up the GCP project with all needed APIs enabled, deploy a throwaway Go hello-world to Cloud Run, capture cold-start P95 baseline. Validates that all free-tier services work together before any port code is written.
|
||||
|
||||
## Requirements
|
||||
- Functional: GCP project ready, Cloud Run accepts deploys, Firestore Native initialized, Secret Manager + Artifact Registry usable, Gemini API key works.
|
||||
- Non-functional: cold-start P95 measured (target ≤1.5s for static-link Go binary), free-tier budgets confirmed in Billing dashboard.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
GCP project (free tier)
|
||||
├── Cloud Run service (region: asia-southeast1)
|
||||
├── Firestore Native database (region: asia-southeast1, default db)
|
||||
├── Artifact Registry repo (region: asia-southeast1, format: docker)
|
||||
├── Secret Manager
|
||||
├── Cloud Scheduler (jobs created later in Phase 09)
|
||||
└── Generative Language API (Gemini, key-based, region-agnostic)
|
||||
```
|
||||
|
||||
Region pinned to `asia-southeast1` (Singapore) to keep RTT to VN users low. Same region for Cloud Run + Firestore + Artifact Registry to avoid cross-region egress.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `scripts/gcp-bootstrap.sh` — idempotent setup script
|
||||
- Create: `docs/gcp-free-tier.md` — captured caps + measured baseline
|
||||
|
||||
## Implementation Steps
|
||||
1. Create GCP project: `miti99bot-prod`. Confirm billing account linked but free-tier-only.
|
||||
2. Enable APIs: `run.googleapis.com`, `firestore.googleapis.com`, `cloudscheduler.googleapis.com`, `artifactregistry.googleapis.com`, `secretmanager.googleapis.com`, `generativelanguage.googleapis.com`, `cloudbuild.googleapis.com`.
|
||||
3. Initialize Firestore Native in `asia-southeast1`. Create `(default)` database.
|
||||
4. Create Artifact Registry docker repo `miti99bot-go` in `asia-southeast1`.
|
||||
5. Create runtime service account `miti99bot-runtime@…iam.gserviceaccount.com` with roles: `roles/datastore.user`, `roles/secretmanager.secretAccessor`, `roles/run.invoker` (for Scheduler→Run OIDC).
|
||||
6. Create deployer service account `miti99bot-deployer@…` for CI with `roles/run.admin`, `roles/artifactregistry.writer`, `roles/iam.serviceAccountUser`.
|
||||
7. Get a Gemini API key from AI Studio → store as a Secret Manager secret `gemini-api-key` (value-only test now; real wiring in Phase 07).
|
||||
8. Write a 30-line Go hello-world (stdlib `net/http`) responding "ok" on `/`. Build static binary, multi-stage Dockerfile (`golang:1.23 → distroless/static`).
|
||||
9. Push image, deploy: `gcloud run deploy miti99bot-baseline --image=… --region=asia-southeast1 --allow-unauthenticated --min-instances=0 --max-instances=2 --memory=128Mi --cpu=1 --timeout=30s`.
|
||||
10. Cold-start measurement: hit `/` 10× with 5-min spacing (force scale-to-zero between). Record P50/P95/P99 from `gcloud run services logs` or curl `-w "%{time_total}"`.
|
||||
11. Document baseline in `docs/gcp-free-tier.md`. Tear down baseline service: `gcloud run services delete miti99bot-baseline`.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] All 7 APIs enabled, billing-free confirmed
|
||||
- [ ] Firestore Native exists in `asia-southeast1`
|
||||
- [ ] Hello-world deploys end-to-end
|
||||
- [ ] Cold-start P95 documented (any number — used as Phase 11 soak gate)
|
||||
- [ ] Baseline service torn down (no idle resources)
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: GCP requires billing account even for free tier → unexpected charges. **Mitigation**: enable Billing alert at $1, $5, $10 thresholds.
|
||||
- **Risk**: Firestore in `asia-southeast1` has higher per-op cost than `us-central1` once free tier exceeded. **Mitigation**: latency wins for VN users; monitor reads in Phase 11.
|
||||
- **Risk**: Gemini API key has no fine-grained quota controls. **Mitigation**: handle 429s gracefully in Phase 07.
|
||||
|
||||
## Rollback
|
||||
Delete project. No state to preserve.
|
||||
@@ -1,82 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "New repo bootstrap + webhook skeleton"
|
||||
status: partial
|
||||
priority: P1
|
||||
effort: "3h"
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 02: New repo bootstrap + webhook skeleton
|
||||
|
||||
## Overview
|
||||
Create `miti99bot-go` GitHub repo, init Go module, scaffold HTTP server with `/`, `/webhook`, `/cron/{name}` routes. Wire Telegram webhook secret-token validation. End-to-end test with a dev bot to prove the loop works before any module logic.
|
||||
|
||||
## Requirements
|
||||
- Functional: `POST /webhook` accepts a Telegram update, validates `X-Telegram-Bot-Api-Secret-Token`, replies 200 OK with no-op handler. `GET /` returns 200 "miti99bot-go ok". Unknown routes 404.
|
||||
- Non-functional: stdlib `net/http` only (no router framework yet — KISS). Static-linked binary, ≤15 MiB. Cold-start ≤500ms target.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
cmd/server/main.go ← entrypoint, wires deps + http.ListenAndServe
|
||||
internal/server/router.go ← HTTP routes, secret-token middleware
|
||||
internal/server/health.go ← GET / handler
|
||||
internal/telegram/webhook.go ← /webhook handler, no-op dispatch
|
||||
internal/telegram/client.go ← grammY-equivalent: github.com/go-telegram/bot wrapper
|
||||
go.mod (module github.com/<owner>/miti99bot-go)
|
||||
Dockerfile ← multi-stage, golang:1.23-alpine → gcr.io/distroless/static
|
||||
.github/workflows/ci.yml ← go vet + go test + go build (no deploy yet)
|
||||
README.md
|
||||
```
|
||||
|
||||
Choice of `github.com/go-telegram/bot` (not `go-telegram-bot-api/v5`) — actively maintained, generic-handler API, `bot.MatchTypeCommand`, idiomatic.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/server/main.go`
|
||||
- Create: `internal/server/router.go`, `internal/server/health.go`
|
||||
- Create: `internal/telegram/webhook.go`, `internal/telegram/client.go`
|
||||
- Create: `Dockerfile`, `.dockerignore`, `.gitignore`
|
||||
- Create: `.github/workflows/ci.yml`
|
||||
- Create: `go.mod`, `go.sum`, `README.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. `gh repo create <owner>/miti99bot-go --public --description "Go port of miti99bot for Cloud Run"` (or private — user choice).
|
||||
2. `git clone` locally. `go mod init github.com/<owner>/miti99bot-go`. Require Go 1.23.
|
||||
3. Add deps: `go get github.com/go-telegram/bot`.
|
||||
4. Write `cmd/server/main.go`:
|
||||
- Read `PORT`, `TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET` from env.
|
||||
- Construct `*bot.Bot`. Register a single dummy `/ping` command (returns "pong") for smoke test.
|
||||
- Build `http.ServeMux`, attach `/`, `/webhook`, `/cron/{name}` handlers.
|
||||
- `http.ListenAndServe(":"+port, mux)`.
|
||||
5. Webhook handler:
|
||||
- Reject non-POST → 405.
|
||||
- Compare `X-Telegram-Bot-Api-Secret-Token` header to `TELEGRAM_WEBHOOK_SECRET`. Mismatch → 401.
|
||||
- Decode JSON `models.Update`, call `b.ProcessUpdate(ctx, &update)`. Return 200.
|
||||
6. Cron handler stub: returns 200 OK, logs `cron name=<name>`. Real dispatch in Phase 09.
|
||||
7. Dockerfile multi-stage:
|
||||
- Builder: `FROM golang:1.23-alpine`, `CGO_ENABLED=0 go build -ldflags="-s -w" -o /server ./cmd/server`.
|
||||
- Runtime: `FROM gcr.io/distroless/static`, `COPY --from=builder /server /`, `ENTRYPOINT ["/server"]`.
|
||||
8. `.github/workflows/ci.yml`: matrix on Go 1.23, run `go vet ./...`, `go test ./...`, `go build ./...`. No deploy step yet (Phase 10).
|
||||
9. Local smoke test: `TELEGRAM_BOT_TOKEN=… TELEGRAM_WEBHOOK_SECRET=local PORT=8080 go run ./cmd/server`. Use `ngrok http 8080`, call Telegram `setWebhook` against the dev bot, `/ping` → "pong".
|
||||
10. Manual deploy to Cloud Run for end-to-end check: `gcloud run deploy miti99bot-go --source=. --region=asia-southeast1 --set-env-vars=… --set-secrets=TELEGRAM_BOT_TOKEN=…,TELEGRAM_WEBHOOK_SECRET=…`. Point dev bot's webhook at the Cloud Run URL. Verify `/ping`.
|
||||
|
||||
## Success Criteria
|
||||
- [x] Repo exists, CI workflow defined (`go vet` + `go test -race` + `go build`)
|
||||
- [ ] `/ping` works against dev bot via Cloud Run URL — **deferred until Phase 01 (GCP setup) lands**
|
||||
- [x] Secret-token mismatch returns 401 (constant-time compare; covered by `internal/telegram/webhook_test.go`)
|
||||
- [x] Image size ≤20 MiB (binary 6.4 MB; distroless image ≈9 MB)
|
||||
- [x] No-secrets-in-git audit clean (no creds present; secrets stripped from `Deps.Env`)
|
||||
|
||||
## Implementation deviations
|
||||
- Step 1 (`gh repo create`) skipped — repo already exists at `github.com/tiennm99/miti99bot-go`.
|
||||
- Steps 9–10 (ngrok smoke test, Cloud Run deploy) deferred to Phase 01.
|
||||
- Step 4's direct `/ping` registration replaced by the Phase 03 module dispatcher; no example module is shipped yet (`MODULES=""` boots cleanly).
|
||||
- Webhook + cron handlers carry hardenings beyond spec: constant-time secret compare, `MaxBytesReader`, shared-secret bridge for `/cron/{name}` (env: `CRON_SHARED_SECRET`; absent → endpoint disabled), bounded handler context via `bot.WithNotAsyncHandlers`, `bot.WithSkipGetMe` to avoid 5s cold-start blocking call. Code review recommendations C1–C3, H1–H7, M1, M4, L5 applied; remaining M-class items tracked in [report](reports/code-reviewer-260508-2254-phase02-03-bootstrap.md).
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: `gcloud run deploy --source` uses Cloud Build, which has its own free tier (120 build-min/day). **Mitigation**: small Go build is ~30s; well within. CI/CD in Phase 10 may move builds to GHA to keep Cloud Build for fallback.
|
||||
- **Risk**: `go-telegram/bot` API surface differs from grammY — handler signatures + middleware patterns require relearning. **Mitigation**: stick to common patterns (commands + plain handlers); avoid grammY's plugins/middleware where translation is fuzzy.
|
||||
|
||||
## Rollback
|
||||
Delete repo + Cloud Run service. CF Worker still owns prod webhook so no impact.
|
||||
@@ -1,132 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Module framework + storage interfaces"
|
||||
status: done
|
||||
priority: P1
|
||||
effort: "4h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 03: Module framework + storage interfaces
|
||||
|
||||
## Overview
|
||||
Replicate the JS plug-n-play module system in Go. Define `Module`, `Command`, `Cron` types. Build static module registry with conflict detection. Define `KVStore` interface (SQL pattern dropped — trading uses Firestore directly). Wire dispatcher to `bot.RegisterHandler` calls.
|
||||
|
||||
## Requirements
|
||||
- Functional: at runtime, `MODULES` env var (CSV) selects which modules load. Each module exposes `Commands []Command` and optional `Crons []Cron`. Registry detects name conflicts across all visibility levels and aborts on conflict (fail-fast at startup).
|
||||
- Non-functional: zero reflection, no plugins. Static slice of constructors registered in `internal/modules/registry.go`. Idiomatic Go (interfaces small, structs concrete).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
internal/modules/
|
||||
├── module.go ← Module, Command, Cron types + Visibility enum
|
||||
├── registry.go ← static map + Build() + name-conflict detection
|
||||
├── dispatcher.go ← installCommands(b *bot.Bot, reg *Registry)
|
||||
├── cron_dispatcher.go ← DispatchScheduled(name, deps)
|
||||
├── validate.go ← validateCommand / validateCron
|
||||
└── modules.go ← static import map (slice of factories)
|
||||
|
||||
internal/storage/
|
||||
├── kv_store.go ← KVStore interface
|
||||
├── memory_kv.go ← in-memory fake (for tests + smoke)
|
||||
└── prefix.go ← per-module key prefixing wrapper
|
||||
```
|
||||
|
||||
Module type:
|
||||
|
||||
```go
|
||||
type Visibility int
|
||||
const (
|
||||
VisibilityPublic Visibility = iota
|
||||
VisibilityProtected
|
||||
VisibilityPrivate
|
||||
)
|
||||
|
||||
type Command struct {
|
||||
Name string // ^[a-z0-9_]{1,32}$
|
||||
Visibility Visibility
|
||||
Description string // required
|
||||
Handler func(ctx context.Context, b *bot.Bot, u *models.Update) error
|
||||
}
|
||||
|
||||
type Cron struct {
|
||||
Schedule string // documentation only
|
||||
Name string // unique within module
|
||||
Handler func(ctx context.Context, deps Deps) error
|
||||
}
|
||||
|
||||
type Module struct {
|
||||
Name string
|
||||
Commands []Command
|
||||
Crons []Cron
|
||||
Init func(ctx context.Context, deps Deps) error // optional
|
||||
}
|
||||
|
||||
type Deps struct {
|
||||
KV KVStore // already prefixed per-module
|
||||
Firestore *firestore.Client
|
||||
Gemini *genai.Client
|
||||
Env map[string]string
|
||||
}
|
||||
|
||||
type Factory func() Module
|
||||
```
|
||||
|
||||
`KVStore` interface mirrors the JS contract:
|
||||
|
||||
```go
|
||||
type KVStore interface {
|
||||
Get(ctx context.Context, key string) ([]byte, error)
|
||||
GetJSON(ctx context.Context, key string, dst any) error // returns ErrNotFound if missing
|
||||
Put(ctx context.Context, key string, val []byte) error
|
||||
PutJSON(ctx context.Context, key string, val any) error
|
||||
Delete(ctx context.Context, key string) error
|
||||
List(ctx context.Context, prefix string) ([]string, error)
|
||||
}
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/modules/{module,registry,dispatcher,cron_dispatcher,validate,modules}.go`
|
||||
- Create: `internal/storage/{kv_store,memory_kv,prefix}.go`
|
||||
- Modify: `cmd/server/main.go` to construct registry, pass to dispatcher
|
||||
|
||||
## Implementation Steps
|
||||
1. Define types in `internal/modules/module.go`. Visibility enum + Command/Cron/Module/Deps structs.
|
||||
2. `internal/modules/validate.go`:
|
||||
- `validateCommand(c Command) error` — name regex, visibility known, description nonempty, handler nonnil.
|
||||
- `validateCron(c Cron) error` — name nonempty, handler nonnil.
|
||||
3. `internal/modules/modules.go`: empty `var Factories = []Factory{}` for now. Each module registers itself in subsequent phases.
|
||||
4. `internal/modules/registry.go`:
|
||||
- `Build(env []string, factories []Factory) (*Registry, error)`.
|
||||
- For each name in `env` ∩ factory map: call factory, validate every command/cron, accumulate into `publicCmds`, `protectedCmds`, `privateCmds`, `allCmds`.
|
||||
- Detect duplicate command names across all 3 maps → error `command conflict: /foo defined in <a> and <b>`.
|
||||
5. `internal/modules/dispatcher.go`:
|
||||
- `Install(b *bot.Bot, reg *Registry)`: iterate `reg.AllCommands`, call `b.RegisterHandler(bot.HandlerTypeMessageText, "/"+name, bot.MatchTypeCommand, handler)`.
|
||||
6. `internal/modules/cron_dispatcher.go`:
|
||||
- `DispatchScheduled(ctx, cronName string, reg *Registry, deps Deps)`: look up cron by name across all modules, run all matching handlers concurrently (errgroup).
|
||||
7. `internal/storage/memory_kv.go`: `sync.Map`-backed KVStore for tests + smoke runs.
|
||||
8. `internal/storage/prefix.go`: `Prefixed(s KVStore, prefix string) KVStore` wrapper that prepends `<prefix>:` to all keys.
|
||||
9. Wire in `cmd/server/main.go`: build registry, install commands, pass to webhook + cron handlers.
|
||||
10. Unit tests: `registry_test.go` (conflict detection, validation errors), `prefix_test.go` (round-trip).
|
||||
|
||||
## Success Criteria
|
||||
- [x] Empty `MODULES=""` boots cleanly (no fallback handler today; grammY's `/start` parity deferred to a future phase)
|
||||
- [x] Two modules with same command name → startup fails with clear error (`TestBuild_DetectsCommandConflict`)
|
||||
- [x] Per-module KVStore prefix isolation verified by test (`TestBuild_PerModulePrefixedKV`, `TestPrefixed_RoundTrip`, `TestDispatchScheduled_PassesPrefixedDeps`)
|
||||
- [x] `go vet ./...` + `go test -race -count=1 ./...` green
|
||||
|
||||
## Implementation deviations
|
||||
- `Factory func() Module` → `Factory func(deps Deps) Module`: handler closures capture deps directly. Eliminates a separate `Module.Init` lifecycle step.
|
||||
- `Factories []Factory` → `Factories map[string]Factory`: required for `MODULES`-env name lookup; prevents duplicate names at compile-load.
|
||||
- `Deps` ships only `KV` + `Env` today. `Firestore` + `Gemini` fields land in Phases 04 / 07 (YAGNI).
|
||||
- Cron uniqueness enforced across modules (registry-level), instead of "concurrent errgroup of all matches" — simpler and matches the one-cron-per-name reality.
|
||||
- `cmd/server` strips `TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `CRON_SHARED_SECRET` from `Deps.Env` to prevent accidental leakage.
|
||||
- Module names validated against `^[a-z0-9_]{1,32}$` (same regex as commands) so KV prefix isolation cannot be subverted by a `:` in the name.
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: `go-telegram/bot` `RegisterHandler` is more general than grammY's `bot.command`. Need to confirm behavior on `/cmd@botname` (group chats). **Mitigation**: library docs say `MatchTypeCommand` strips `@botname`; verify with a group chat test before Phase 05.
|
||||
- **Risk**: Static factory slice means new modules require code change — same constraint as JS `index.js` static map. Acceptable.
|
||||
|
||||
## Rollback
|
||||
Revert to Phase 02 main.go. Module framework is purely additive.
|
||||
@@ -1,86 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Firestore KVStore + per-module prefixing"
|
||||
status: done
|
||||
priority: P1
|
||||
effort: "4h"
|
||||
dependencies: [3]
|
||||
---
|
||||
|
||||
# Phase 04: Firestore KVStore + per-module prefixing
|
||||
|
||||
## Overview
|
||||
Implement `FirestoreKVStore` against the `KVStore` interface from Phase 03. One Firestore collection per module (`<module>`), each KV entry one document. Test against the local Firestore emulator. Provide an in-memory fake for module-level unit tests so they don't need the emulator.
|
||||
|
||||
## Requirements
|
||||
- Functional: `Get/GetJSON/Put/PutJSON/Delete/List` work against Firestore. Per-module isolation via collection name. JSON values stored as `value` field on the document.
|
||||
- Non-functional: P50 read ≤80ms warm, ≤500ms cold. Connection reused across requests via package-level `*firestore.Client`. Free-tier-aware: avoid `Query.GetAll` on hot paths.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
internal/storage/
|
||||
├── firestore_kv.go ← FirestoreKVStore impl
|
||||
├── firestore_client.go ← package-level client (lazy init, project ID from env)
|
||||
└── firestore_kv_test.go ← runs against emulator if FIRESTORE_EMULATOR_HOST set
|
||||
```
|
||||
|
||||
Firestore document shape:
|
||||
```
|
||||
collection: <moduleName>
|
||||
document id: <key> ← URL-safe key (rejects `/` per Firestore rules)
|
||||
fields:
|
||||
value: bytes | string | map ← raw bytes for Put, JSON-marshaled struct for PutJSON
|
||||
updatedAt: timestamp
|
||||
```
|
||||
|
||||
`List(prefix)` uses `collection.Where(firestore.DocumentID(), ">=", prefix).Where(firestore.DocumentID(), "<", prefixSuccessor(prefix))`.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/storage/firestore_kv.go`, `firestore_client.go`, `firestore_kv_test.go`
|
||||
- Modify: `cmd/server/main.go` to initialize Firestore client, pass to module Deps
|
||||
- Modify: `internal/modules/dispatcher.go` Deps construction
|
||||
- Create: `Makefile` target `test-emulator` (start emulator, run tests)
|
||||
|
||||
## Implementation Steps
|
||||
1. Add dep: `go get cloud.google.com/go/firestore`.
|
||||
2. `firestore_client.go`: singleton `func Client(ctx) (*firestore.Client, error)` reading `GOOGLE_CLOUD_PROJECT` from env. Reuse across requests.
|
||||
3. `firestore_kv.go`:
|
||||
- Struct `FirestoreKVStore { c *firestore.Client; collection string }`.
|
||||
- `Get(ctx, key)`: `c.Collection(collection).Doc(key).Get(ctx)`. Map `codes.NotFound` → `ErrNotFound`. Return `value` field as bytes.
|
||||
- `Put(ctx, key, val)`: `Doc(key).Set(ctx, map{"value": val, "updatedAt": time.Now()})`.
|
||||
- `GetJSON/PutJSON`: marshal/unmarshal via `encoding/json`.
|
||||
- `Delete`: `Doc(key).Delete(ctx)`.
|
||||
- `List(prefix)`: `Where(DocumentID >= prefix).Where(DocumentID < successor)`. Iterator → slice of doc IDs.
|
||||
4. Key validation: reject `/`, empty string, length >1500 bytes (Firestore limit).
|
||||
5. `firestore_kv_test.go`: skip if `FIRESTORE_EMULATOR_HOST` not set. Round-trip Put/Get/Delete/List/PutJSON/GetJSON/NotFound.
|
||||
6. Update `internal/storage/memory_kv.go` to support `List(prefix)` symmetrically (iterate map keys).
|
||||
7. Update `cmd/server/main.go`:
|
||||
- Init Firestore client at startup.
|
||||
- For each module, pass `Prefixed(NewFirestoreKVStore(client, module.Name), module.Name)` (collection name = module name = prefix; equivalent to single-collection prefixing).
|
||||
- Actually: drop the `Prefixed` wrapper for Firestore — collection itself isolates. `Prefixed` only used with `MemoryKV` for tests.
|
||||
8. Add `Makefile`: `firestore-emulator: gcloud emulators firestore start --host-port=localhost:8085` and `test: FIRESTORE_EMULATOR_HOST=localhost:8085 go test ./...`.
|
||||
|
||||
## Success Criteria
|
||||
- [x] All KV ops round-trip against emulator (`firestore_kv_test.go`, runs via `make test-emulator`)
|
||||
- [x] In-memory fake matches Firestore semantics for List ordering + ErrNotFound
|
||||
- [x] Two modules writing to same key name → no collision (`TestBuild_PerModulePrefixedKV` for memory backend; collection-per-module IS the isolation for Firestore — test exists in registry layer)
|
||||
- [x] `go test -race -count=1 ./...` green (Firestore tests skip cleanly without emulator)
|
||||
|
||||
## Implementation deviations
|
||||
- Spec step 3 (wrap Firestore in `Prefixed`) contradicts step 7 (drop the wrapper). We followed step 7 — collection-per-module IS isolation. Memory backend keeps `Prefixed` because all modules share one in-process store.
|
||||
- Introduced `KVProvider` interface (not in spec): `MemoryProvider` wraps base+Prefixed, `FirestoreProvider` returns one collection per module. `modules.Build` now takes `KVProvider`+env map instead of base `Deps`. Cleaner: modules never see the backend choice.
|
||||
- Backend selection in `cmd/server/main.go`: Firestore when `GOOGLE_CLOUD_PROJECT` or `FIRESTORE_EMULATOR_HOST` is set (latter supplies a placeholder project ID for the SDK); otherwise in-memory.
|
||||
- `validateKey` rejects more than spec required: empty, `/`, `.`, `..`, `__namespace__`, > 1500 bytes. `validatePrefix` runs the same check on `List` arguments.
|
||||
- Binary size 6.4 MB → 17 MB after Firestore SDK + gRPC. Within Phase 02's ≤20 MiB target. Distroless image ≈19 MB.
|
||||
|
||||
## Code review
|
||||
[Phase 04 review](reports/code-reviewer-260508-2333-phase04-firestore-kv.md) — 0 critical, 3 high (H1 emulator-only-no-project trap, H2 List-prefix unvalidated, H3 bytes-vs-runes doc) all addressed in same session; M1 prefixSuccessor all-0xFF degeneracy documented; remaining mediums deferred.
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: Firestore document IDs reject `/` but JS keys may contain them (e.g. nested loldle state). **Mitigation**: encode `/` → `_` in Put, decode on Get. Document the mapping. Or use base64 for arbitrary keys.
|
||||
- **Risk**: 50k reads/day hard cap. Listing leaderboards on every request hits this fast. **Mitigation**: cache hot reads in process memory with 5-minute TTL — free for warm instance, costs 0 reads.
|
||||
- **Risk**: Emulator behavior diverges from prod (e.g. timestamp resolution, indexes). **Mitigation**: smoke a 50-key Put/List against real Firestore at end of phase.
|
||||
|
||||
## Rollback
|
||||
Drop the Firestore client init, revert to `MemoryKV` for all modules. Modules continue working but lose persistence.
|
||||
@@ -1,118 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Port simple modules (util, misc, wordle, loldle classic)"
|
||||
status: done
|
||||
priority: P2
|
||||
effort: "6h"
|
||||
dependencies: [4]
|
||||
---
|
||||
|
||||
# Phase 05: Port simple modules
|
||||
|
||||
## Overview
|
||||
Port the four KV-only, AI-free modules: `util` (info/help renderer), `misc` (stub easter eggs), `wordle` (5-letter game with 14k-word dict), `loldle` classic (LoL champion guesser). Validates the framework end-to-end before the more complex modules.
|
||||
|
||||
## Requirements
|
||||
- Functional: command parity with JS — same names, same behaviors, same KV state shape (so a future export-import migration is feasible).
|
||||
- Non-functional: each module file ≤200 lines per code-standards.md. Static word/champion datasets embedded via `go:embed`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
internal/modules/util/
|
||||
├── util.go ← Module factory, registers /info /help
|
||||
├── info.go ← /info handler
|
||||
└── help.go ← /help renderer (groups by visibility)
|
||||
|
||||
internal/modules/misc/
|
||||
└── misc.go ← stub commands
|
||||
|
||||
internal/modules/wordle/
|
||||
├── wordle.go ← factory, registers /wordle /wguess /wgiveup /wstats
|
||||
├── game.go ← session state struct, Get/Save via KV
|
||||
├── guess.go ← scoring (green/yellow/gray)
|
||||
├── data/words.txt ← 14k word dict (embedded)
|
||||
└── words.go ← go:embed loader
|
||||
|
||||
internal/modules/loldle/
|
||||
├── loldle.go ← factory
|
||||
├── game.go ← session state
|
||||
├── champions.go ← go:embed champion JSON
|
||||
├── data/champions.json
|
||||
└── compare.go ← attribute comparison logic
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Create: above tree under `internal/modules/{util,misc,wordle,loldle}`
|
||||
- Modify: `internal/modules/modules.go` Factories slice — append `util.New, misc.New, wordle.New, loldle.New`
|
||||
- Copy: word list + champion JSON from JS repo (verbatim)
|
||||
|
||||
## Implementation Steps
|
||||
1. Copy `src/modules/util/*` JS source as reference. Implement `/info` (returns env-derived bot info) + `/help` (groups commands public+protected, omits private).
|
||||
2. `/help` queries the registry — already accessible via `Deps`. Format as Telegram MarkdownV2.
|
||||
3. Misc module: port commands as-is (mostly text replies).
|
||||
4. Wordle:
|
||||
- Copy `src/modules/wordle/words.txt` to `internal/modules/wordle/data/words.txt`.
|
||||
- `go:embed data/words.txt` into a `string`, split lines, build a `map[string]struct{}` for O(1) validity checks.
|
||||
- Game state: `{ word string; guesses []string; status string }` saved per user.
|
||||
- KV key: `game:<userID>`.
|
||||
5. Loldle classic:
|
||||
- Copy champion JSON dataset into `data/champions.json`.
|
||||
- State: `{ targetID string; guesses []string }` per user per UTC day. Key: `game:<userID>:<yyyy-mm-dd>`.
|
||||
- Comparison: gender, position, species, resource, range, region, release year. Yields green/yellow/red per attribute.
|
||||
6. Port unit tests from JS:
|
||||
- `wordle/format_test.go` — score formatting
|
||||
- `wordle/guess_test.go` — green/yellow/gray correctness, double-letter edge case
|
||||
- `loldle/compare_test.go` — each attribute comparison
|
||||
- `loldle/game_test.go` — daily reset, max-guesses gate
|
||||
7. Wire into `Factories` slice. `MODULES=util,misc,wordle,loldle` env var enables them.
|
||||
8. Smoke test on Cloud Run with dev bot.
|
||||
|
||||
## Success Criteria
|
||||
- [x] `/wordle`, `/wordle <word>`, `/wordle_new`, `/wordle_giveup`, `/wordle_stats` ported (commands renamed from spec's `/wguess` etc. to match JS source)
|
||||
- [x] `/loldle`, `/loldle <champion>`, `/loldle_giveup`, `/loldle_stats`, `/loldle_setmax` (private) ported
|
||||
- [x] `/help` lists all loaded modules' public + protected commands (util + misc + wordle + loldle)
|
||||
- [x] All ported tests pass — wordle and loldle JS vitest suites ported verbatim, plus Go-only coverage for race-free pickers, pool exhaustion, render alignment, keylock fan-out
|
||||
- [x] Image size stays ≤25 MiB after embedding word + champion data (binary 17 MB; 88 KB words.txt + 65 KB champions.json are noise vs the 10 MB Firestore SDK)
|
||||
|
||||
## Cook scope split
|
||||
This phase shipped in three sub-cooks:
|
||||
- **5a (done):** util + misc — small, validates the module-loading pipeline end-to-end. ✅
|
||||
- **5b (done):** wordle — 14855-word dict, scoring, sessions. ✅
|
||||
- **5c (this cook):** loldle classic — 172-champion JSON, attribute comparison, sticker pools. ✅
|
||||
|
||||
## Implementation deviations (5a)
|
||||
- `modules.Deps` gained a `Registry *Registry` pointer so `/help` can introspect at runtime. Pointer is captured at factory time and stable thereafter; Registry is documented read-only after Build returns.
|
||||
- Static factory catalog (`modules.Factories`) moved to `cmd/server/main.go::factories()` to avoid an import cycle (`modules → util → modules`). The empty `internal/modules/modules.go` file remains as a doc anchor.
|
||||
- `misc.lastPing.At` stored as int64 ms-epoch (matches JS `Date.now()`) — preserves byte-for-byte KV parity for the future export-import migration.
|
||||
- Telegram-side handler tests intentionally skipped — would require a fake bot HTTP server for negligible coverage gain. Renderer + KV behaviour ARE tested.
|
||||
|
||||
## Implementation deviations (5b — wordle)
|
||||
- KV TTL: JS uses Cloudflare KV's `expirationTtl: 60*60*24*7`. Firestore has no equivalent per-doc TTL; `gameTTLSeconds` constant is informational. Old games linger — Phase 11 GC if needed.
|
||||
- `pickDaily` ported but unused (handlers call `pickRandom`). Kept for parity so future "daily wordle" mode is a one-line swap.
|
||||
- Added `subjectLocks` (per-subject `sync.Mutex` map) to serialise `Get → mutate → Put` in handlers. Cloudflare Workers' isolate model gave the JS source this for free; Go + Firestore needs explicit locking or two concurrent guesses to the same group chat silently lose one.
|
||||
- `pickRandom(words, nil)` falls through to `math/rand.Intn` (package-level, mutex-protected globals) instead of a singleton `*rand.Rand` so the bot dispatcher's per-update goroutines don't race on RNG state.
|
||||
- KV wire-format parity: `GameState.Giveup` always emitted (no omitempty); `Stats.LastResultAt` is `*int64` so unplayed accounts marshal as `null` matching JS shape; `StartedAt` is ms-epoch int64.
|
||||
- Subject IDs converted to strings for KV keys (`game:<subject>`); JS uses numbers but Cloudflare KV stringifies on the wire so Firestore round-trips identically.
|
||||
- Word-list loader panics on malformed embedded data — corrupt regen of `words.txt` is a build-time bug, not a runtime concern worth recovering from.
|
||||
|
||||
## Implementation deviations (5c — loldle)
|
||||
- Per-subject lock extracted from wordle into `internal/keylock` (shared package). Both wordle and loldle now import it. Naming chosen as a peer to `internal/storage` and `internal/telegram` rather than nesting under `internal/modules/`.
|
||||
- KV TTL deferred — Cloudflare KV's `expirationTtl` has no Firestore equivalent. Phase 11 GC if old games become a cost concern.
|
||||
- Sticker pools (win/lose/giveup) preserved verbatim from `stickers.js`; file_ids are bot-scoped to `@miti99bot` and were already valid against the new bot per the test-bot policy.
|
||||
- `lastResultAt` deliberately omitted from loldle stats (parity with JS source — different from wordle's stats which DOES include it; that asymmetry exists in the JS source).
|
||||
- `pickRandomChampion` and `pickSticker` use `math/rand.Intn` (package-level mutex-protected globals) so concurrent /loldle handlers don't race on RNG state. Same pattern as wordle 5b.
|
||||
- `winRate` uses `math.Round` not `int(...)` truncation, after Phase 5c review caught the JS-parity bug. The same fix was retroactively applied to wordle's `/wordle_stats`.
|
||||
|
||||
## Code reviews
|
||||
- [Phase 5a review](reports/code-reviewer-260509-0813-phase5a-util-misc.md) — 1 critical (`/info` nil-deref), 2 high (1 informational + 1 perf-deferred), 4 mediums/lows. C1, L2, M1, L3, H1 doc applied.
|
||||
- [Phase 5b review](reports/code-reviewer-260509-0918-phase5b-wordle.md) — 1 critical (`defaultRNG` data race) + 2 high (Get-mutate-Put logical race; dead `debugPickerError`) + extra compare test + race test for `pickRandom`. All addressed in same session. Mediums (M1 giveup-on-never-played JS-faithful gotcha; M2 `subjectFor` test) deferred — JS-parity intentional.
|
||||
- [Phase 5c review](reports/code-reviewer-260509-0940-phase5c-loldle.md) — 1 high (`winRate` truncation across both wordle + loldle) + 4 mediums (test gaps). H1 fixed in both modules in same session; M1 (render alignment golden test) and M2 (keylock fan-out + serialisation tests) added; M3/M4 deferred — covered transitively elsewhere.
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: 14k-word file embedded → ~120 KiB. `go:embed` puts it in the binary; no runtime IO. Acceptable.
|
||||
- **Risk**: Wordle scoring has a known JS-side edge case (double-letter); ensure ported logic matches. **Mitigation**: bring the failing-cases test verbatim.
|
||||
- **Risk**: Loldle daily reset uses UTC in JS; confirm Go uses same. **Mitigation**: explicit `time.Now().UTC()` in date key.
|
||||
|
||||
## Rollback
|
||||
Remove modules from `Factories` slice or `MODULES` env. Each module is independent.
|
||||
@@ -1,102 +0,0 @@
|
||||
---
|
||||
phase: 6
|
||||
title: "Port loldle variants + lolschedule"
|
||||
status: partial
|
||||
priority: P2
|
||||
effort: "5h"
|
||||
dependencies: [5]
|
||||
---
|
||||
|
||||
# Phase 06: Port loldle variants + lolschedule
|
||||
|
||||
## Overview
|
||||
Port the four loldle variants (`loldle-emoji`, `loldle-quote`, `loldle-ability`, `loldle-splash`) plus `lolschedule`. They share the per-day session pattern from classic loldle, differ only in clue-reveal mechanics.
|
||||
|
||||
## Requirements
|
||||
- Functional: command parity — same commands, same data sources (Riot Data Dragon for ability icons + splash arts; loldle.net derived for emoji + quote pools).
|
||||
- Non-functional: image-bearing commands (ability icons, splash) reuse remote URLs — do not embed binaries. Reply uses Telegram `sendPhoto` with URL string.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
internal/modules/loldle-emoji/
|
||||
├── module.go
|
||||
├── data/emoji-pool.json ← embedded
|
||||
└── game.go
|
||||
|
||||
internal/modules/loldle-quote/
|
||||
├── module.go
|
||||
├── data/quotes.json
|
||||
└── game.go
|
||||
|
||||
internal/modules/loldle-ability/
|
||||
├── module.go
|
||||
├── ability.go ← URL pattern: ddragon ability icon
|
||||
└── game.go
|
||||
|
||||
internal/modules/loldle-splash/
|
||||
├── module.go
|
||||
├── data/skin-pool.json ← scraped from loldle.net (per credits in README)
|
||||
├── splash.go ← ddragon splash URL builder
|
||||
└── game.go
|
||||
|
||||
internal/modules/lolschedule/
|
||||
├── module.go
|
||||
├── client.go ← lolesports/leaguepedia HTTP client
|
||||
└── format.go ← schedule formatter
|
||||
```
|
||||
|
||||
A small shared package would help, but keep modules independent (KISS) until duplication exceeds 3 callers — then extract.
|
||||
|
||||
## Related Code Files
|
||||
- Create: above 5 module trees
|
||||
- Reuse: copy data files from `src/modules/<name>/data/*` verbatim
|
||||
- Modify: `internal/modules/modules.go` Factories slice
|
||||
- Update: `MODULES` env var in Cloud Run service yaml
|
||||
|
||||
## Implementation Steps
|
||||
1. **loldle-emoji**: Port emoji clue pool. Game state `{ targetID; guesses []; cluesShown int }`. Reveal one emoji per wrong guess, max 4.
|
||||
2. **loldle-quote**: Port quote pool. Reveal up to 3 quote chunks across guesses.
|
||||
3. **loldle-ability**: Build ability icon URL from champion ID + ability slot (Q/W/E/R), e.g. `https://ddragon.leagueoflegends.com/cdn/<v>/img/spell/<spellId>.png`. Cache the latest ddragon version once per cold start.
|
||||
4. **loldle-splash**: URL pattern `https://ddragon.leagueoflegends.com/cdn/img/champion/splash/<key>_<skin>.jpg`.
|
||||
5. **lolschedule**: HTTP client to lolesports/leaguepedia API for upcoming match schedule. Format with `/lolschedule [date]` syntax (recent commit shows this is current behavior).
|
||||
6. Use the same KV `game:<userID>:<yyyy-mm-dd>` namespace pattern (one game per variant per day).
|
||||
7. Port unit tests for clue-reveal logic, URL builders, schedule formatter.
|
||||
8. Wire factories.
|
||||
9. Smoke each command against dev bot.
|
||||
|
||||
## Cook scope split
|
||||
This phase ships in five sub-cooks (one per module — each is large enough to risk context exhaustion):
|
||||
- **6a:** loldle-emoji — 172-record emoji clue dict, binary scoring, simplest variant. ✅
|
||||
- **6b:** loldle-quote — quote-pool variant, default 6 guesses. ✅ (consumes the shared `chathelper` + `champname` packages extracted in fix-all-review-findings Phase 03)
|
||||
- **6c:** loldle-ability — DDragon ability-icon URL builder, sendPhoto reply, gameState gains a `slot` field so the same icon shows across guesses. ✅
|
||||
- **6d:** loldle-splash — DDragon splash URL, sendPhoto reply, gameState locks `skinId` so the same splash shows across guesses. Default 4 guesses. ✅
|
||||
- **6e:** lolschedule — HTTP client to lolesports.com persisted API (cache-first with 60-min stale fallback), ICT-anchored date parsing (dd-mm-yyyy / dd/mm/yyyy / ddmmyyyy), today/week renderers, subscriber list. 5 user commands shipped. Daily-push cron deferred to Phase 09 (Cloud Scheduler) since `Deps` doesn't currently expose a `*bot.Bot` reference. ✅
|
||||
|
||||
## Success Criteria
|
||||
- [x] loldle-emoji responds to `/loldle_emoji`, `/loldle_emoji_giveup`, `/loldle_emoji_stats`, `/loldle_emoji_setmax`
|
||||
- [x] loldle-quote responds to `/loldle_quote`, `/loldle_quote_giveup`, `/loldle_quote_stats`, `/loldle_quote_setmax`
|
||||
- [x] loldle-ability responds to `/loldle_ability`, `/loldle_ability_giveup`, `/loldle_ability_stats`, `/loldle_ability_setmax`; sendPhoto path uses the DDragon icon URL directly
|
||||
- [x] loldle-splash responds to `/loldle_splash`, `/loldle_splash_giveup`, `/loldle_splash_stats`, `/loldle_splash_setmax`; sendPhoto path uses the DDragon splash URL directly
|
||||
- [x] `/lolschedule [date]`, `/lolschedule_today`, `/lolschedule_week`, `/lolschedule_subscribe`, `/lolschedule_unsubscribe` match JS behavior; daily-push cron deferred to Phase 09
|
||||
- [x] All variants share consistent guess-count limits matching JS (emoji 5, quote 6 — JS parity)
|
||||
- [x] Ported tests pass for loldle-emoji + loldle-quote (lookup, state, render, JS-wire-format decode, handler integration)
|
||||
|
||||
## Implementation deviations (6a — loldle-emoji)
|
||||
- `moduleNameRe` relaxed from `^[a-z0-9_]{1,32}$` to `^[a-z0-9_-]{1,32}$` so JS-source module names like `loldle-emoji` pass validation. The storage prefix delimiter (`:`) remains rejected; tests cover both shapes.
|
||||
- Go package directory + package name use `loldleemoji` (no separator) per Go convention; the registered MODULE name is `loldle-emoji` (hyphenated, byte-identical to JS) for KV-prefix migration parity.
|
||||
- `normalize`, `subjectFor`, `argAfterCommand` duplicated from classic loldle. Marked for extraction at the start of cook 6b — three callers will exist by then, past the YAGNI threshold.
|
||||
- `winRate` uses `math.Round` from day one (lesson from Phase 5c review).
|
||||
- KV TTL deferred — Cloudflare KV's `expirationTtl` has no Firestore equivalent.
|
||||
- No sticker pools — JS source has none for emoji mode.
|
||||
|
||||
## Code reviews (6a)
|
||||
- [Phase 6a review](reports/code-reviewer-260509-1206-phase6a-loldle-emoji.md) — 0 critical, 0 high. Concerns: F#1 (JS-wire-format decode test) added in same session; F#2 (`getOrInitGame` cap-reduction edge case) deferred — defensive branch only; E (extract shared helpers) earmarked as 6b prep work.
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: Riot Data Dragon version pinning — JS version may use different ddragon version than fresh fetch. **Mitigation**: pin version in env or fetch latest at cold start; document in README.
|
||||
- **Risk**: lolschedule API surface may have changed since JS implementation. **Mitigation**: re-test against live API; fix forward if drifted.
|
||||
- **Risk**: Splash skin pool was scraped from loldle.net; legality + freshness. **Mitigation**: reuse the same JSON file already in repo (no re-scrape).
|
||||
|
||||
## Rollback
|
||||
Remove from Factories. Per-variant rollback works independently.
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
phase: 7
|
||||
title: "Gemini AI + port semantle/doantu/twentyq"
|
||||
status: done
|
||||
priority: P2
|
||||
effort: "6h"
|
||||
dependencies: [4]
|
||||
---
|
||||
|
||||
# Phase 07: Gemini AI + port semantle/doantu/twentyq
|
||||
|
||||
## Overview
|
||||
Wire Gemini API as the Workers AI replacement. Port the three AI-using modules: `semantle` and `doantu` use embeddings; `twentyq` uses chat-style text generation. All must respect Gemini free-tier RPM/RPD; degrade gracefully on 429.
|
||||
|
||||
## Requirements
|
||||
- Functional:
|
||||
- `semantle`/`doantu`: target word + user guess → cosine similarity score via embeddings.
|
||||
- `twentyq`: 20-question style game, model plays the responder (yes/no/sometimes), tracks remaining questions.
|
||||
- Non-functional:
|
||||
- Free-tier-aware: cache embeddings of game targets (rare changes), retry-with-jitter on 429.
|
||||
- Gemini client reused as package-level singleton (gRPC connection).
|
||||
- Per-user RPM soft-limit in process to prevent abuse from blowing through 1500 RPD shared quota.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
internal/ai/
|
||||
├── gemini.go ← package-level *genai.Client init
|
||||
├── embeddings.go ← Embed(ctx, text) ([]float32, error) using text-embedding-004
|
||||
├── chat.go ← Generate(ctx, prompt, history) (string, error) using gemini-1.5-flash
|
||||
└── ratelimit.go ← per-user token bucket (in-memory, sync.Map of buckets)
|
||||
|
||||
internal/modules/semantle/
|
||||
├── module.go
|
||||
├── data/targets-en.json ← curated daily target pool
|
||||
├── game.go ← session state
|
||||
├── score.go ← cosine similarity
|
||||
└── targets.go ← daily target selection (deterministic from date)
|
||||
|
||||
internal/modules/doantu/
|
||||
├── module.go ← Vietnamese variant (different target pool, same algorithm)
|
||||
└── data/targets-vi.json
|
||||
|
||||
internal/modules/twentyq/
|
||||
├── module.go
|
||||
├── prompt.go ← system prompt + history serialization
|
||||
├── game.go
|
||||
└── parser.go ← yes/no/maybe extractor from model output
|
||||
```
|
||||
|
||||
`bge-m3` (1024d, multilingual) is replaced by `text-embedding-004` (768d). Different vector space — pre-cached target vectors must be re-computed; do not migrate vectors from CF KV.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/ai/{gemini,embeddings,chat,ratelimit}.go`
|
||||
- Create: `internal/modules/{semantle,doantu,twentyq}/...`
|
||||
- Modify: `Deps` struct (already contains `Gemini *genai.Client` from Phase 03)
|
||||
- Modify: `cmd/server/main.go` to init Gemini client
|
||||
|
||||
## Implementation Steps
|
||||
1. Add dep: `go get google.golang.org/genai` (official Google GenAI Go SDK).
|
||||
2. `internal/ai/gemini.go`: lazy client init from `GEMINI_API_KEY` (Secret Manager → env var injection at deploy).
|
||||
3. `internal/ai/embeddings.go`: `Embed(ctx, texts []string) ([][]float32, error)` using `text-embedding-004`. Batch up to 100 inputs per call.
|
||||
4. `internal/ai/chat.go`: `Generate(ctx, system, history []Msg) (string, error)` using `gemini-1.5-flash`. Output ≤200 tokens, temperature 0.7.
|
||||
5. `internal/ai/ratelimit.go`: per-user 5 req/min bucket via `golang.org/x/time/rate`. Drop-on-exceed with user-visible "slow down" reply.
|
||||
6. Pre-compute target embeddings:
|
||||
- At cold start, load target pool, embed any not yet cached in Firestore (`semantle_target_cache:<word>` → `[]float32`).
|
||||
- 1500 RPD limit means ≤1500 fresh target embeds/day. Curated pool of ~365 targets (one per day) embedded once = ~30 minutes work amortized.
|
||||
7. Semantle/doantu game flow: user `/semantle`, target picked deterministically from `today's UTC date`. Each `/sguess <word>` → embed user word → cosine similarity → reply with score.
|
||||
8. Twentyq: user picks a topic, model (system prompt: "you're answering 20-questions about X, reply only yes/no/maybe"). Track Q count, end at 20.
|
||||
9. Tests:
|
||||
- `embeddings_test.go` — fake `*genai.Client` interface; verify cache hit/miss
|
||||
- `score_test.go` — cosine math
|
||||
- `parser_test.go` — twentyq response parsing
|
||||
- `ratelimit_test.go` — bucket refill + drop
|
||||
10. Smoke each module on dev bot.
|
||||
|
||||
## Success Criteria
|
||||
- [x] `internal/ai` package wraps `google.golang.org/genai` (v1.56) with Embedder/Chatter interfaces; per-user `PerUserLimiter` (5 req / 60s burst).
|
||||
- [x] `Deps` extended with `Embedder`/`Chatter` (nil when GEMINI_API_KEY unset → modules refuse with config-error).
|
||||
- [x] `/semantle` ported: 9894-word google-10k pool, JS-parity sigmoid calibration, OOV gate, fast-path dedup, render board with sort+top-15.
|
||||
- [x] `/doantu` ported via JS-parity `phow2sim` HTTP client (NOT Gemini — see Deviations below).
|
||||
- [x] `/twentyq` ported with prompts.go (verbatim JS prompt strings), parser.go (JSON-with-fence extraction), redact-secret defense, fallback round-start.
|
||||
- [x] 429 from Gemini mapped to `ai.ErrRateLimited` → user-visible "rate-limited" reply.
|
||||
- [x] All factories registered in `cmd/server/main.go`; `go vet ./...` and `go test -race -count=1 ./...` clean.
|
||||
|
||||
## Deviations from original plan
|
||||
- **doantu uses phow2sim HTTP, not Gemini embeddings.** Rationale: text-embedding-004 was not trained for Vietnamese semantic relatedness; phow2sim is a domain-trained PhoW2V model. The JS bot already uses it; switching to embeddings would diverge behaviour, not preserve it. `PHOW2SIM_API_URL` overridable via env (allowlisted in `cmd/server/main.go`).
|
||||
- **No Firestore-backed target embedding cache** (plan step 6). semantle embeds both target+guess on every call (matches JS bge-m3 path). Cache adds complexity without measurable savings until Phase 11 soak data shows the 1500 RPD ceiling is real.
|
||||
- **gemini-2.5-flash, not 1.5.** SDK default is the newer flash; behaviour-equivalent for the twentyq use case.
|
||||
- **Per-day cap deferred.** Token bucket only; if Phase 11 soak shows abuse, add a Firestore counter.
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: 768d vs 1024d means similarity scores have different distribution. Game tuning constants (winning threshold) need re-calibration. **Mitigation**: empirical tune against dev bot; document in module file.
|
||||
- **Risk**: 1500 RPD shared across all users. Heavy semantle play could exhaust. **Mitigation**: per-user 50 req/day soft cap. Cache user-guess embeddings too (most users guess common words).
|
||||
- **Risk**: `gemini-1.5-flash` cold-start latency (gRPC TLS handshake) on Cloud Run. **Mitigation**: client init at process start, not per-request.
|
||||
- **Risk**: Gemini may be deprecated or repriced. **Mitigation**: AI ops abstracted behind `internal/ai` package — switching providers (e.g. Vertex AI, OpenRouter free-tier) is a single-package change.
|
||||
|
||||
## Rollback
|
||||
Remove from Factories. AI modules are isolated by package; main framework continues without them.
|
||||
@@ -1,87 +0,0 @@
|
||||
---
|
||||
phase: 8
|
||||
title: "Port trading + Firestore composite indexes"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "6h"
|
||||
dependencies: [4]
|
||||
---
|
||||
|
||||
# Phase 08: Port trading + Firestore composite indexes
|
||||
|
||||
## Overview
|
||||
Port the most complex module: VN-stocks paper trading. Original used D1 (relational SQL) for trades + leaderboards. Translate to Firestore document model with composite indexes for the leaderboard query path.
|
||||
|
||||
## Requirements
|
||||
- Functional: `/trade`, `/buy <ticker> <qty>`, `/sell …`, `/portfolio`, `/leaderboard`, plus the daily price-update cron at `0 17 * * *`.
|
||||
- Non-functional: leaderboard query stays under 100ms warm. Daily cron fits within 50k-reads/20k-writes per-day cap (≤300 active users, ≤50 unique tickets traded).
|
||||
|
||||
## Architecture
|
||||
|
||||
Firestore data model (replacing D1's `trading_trades` table):
|
||||
|
||||
```
|
||||
collection: trading_users ← user state
|
||||
doc id: <userID>
|
||||
fields:
|
||||
balanceVnd: number
|
||||
createdAt: timestamp
|
||||
lastTradeAt: timestamp
|
||||
pnlVnd: number ← denormalized for leaderboard
|
||||
|
||||
subcollection: trades ← per-user trade log
|
||||
doc id: <auto>
|
||||
fields: { ticker, side, qty, priceVnd, ts }
|
||||
|
||||
subcollection: holdings ← current positions (one per ticker)
|
||||
doc id: <ticker>
|
||||
fields: { qty, avgCostVnd }
|
||||
|
||||
collection: trading_prices ← current ticker prices
|
||||
doc id: <ticker>
|
||||
fields: { priceVnd, updatedAt }
|
||||
```
|
||||
|
||||
Composite index: `trading_users` on `(pnlVnd DESC)` for leaderboard. Single-field default indexes cover everything else.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/modules/trading/{module,buy,sell,portfolio,leaderboard,prices,cron_daily_update}.go`
|
||||
- Create: `internal/modules/trading/store.go` — direct Firestore access (bypassing KVStore for relational queries)
|
||||
- Create: `firestore.indexes.json` (committed) — composite indexes deployed via `gcloud firestore indexes composite create`
|
||||
- Modify: `Deps` to include `*firestore.Client` (already present)
|
||||
- Modify: `MODULES` env var in deploy yaml — add `trading`
|
||||
|
||||
## Implementation Steps
|
||||
1. **Schema**: Define structs `User`, `Trade`, `Holding`, `Price` in `store.go`. Use `firestore` struct tags.
|
||||
2. **Buy flow**:
|
||||
- Read user balance + ticker price.
|
||||
- Validate sufficient balance + qty > 0.
|
||||
- In a Firestore `RunTransaction`: decrement balance, increment holding (compute new avgCost), append trade, update `lastTradeAt`.
|
||||
3. **Sell flow**:
|
||||
- Symmetric. Realized PnL = (sellPrice - avgCost) * qty. Update `pnlVnd` denorm.
|
||||
4. **Portfolio**: list holdings + current prices (one read per ticker — typical user holds <10).
|
||||
5. **Leaderboard**: `Where(pnlVnd > 0).OrderBy(pnlVnd DESC).Limit(10)`. Requires composite index.
|
||||
6. **Daily price update cron**:
|
||||
- Triggered by Cloud Scheduler at `0 17 * * *` (set up in Phase 09).
|
||||
- Fetches VN stock prices from existing data source (port URL/parsing from JS module).
|
||||
- Writes ~50 ticker docs into `trading_prices`. Stays under 20k writes/day cap easily.
|
||||
7. **One-time data import** (optional, decided in Phase 12 cutover): script to read D1 dump, transform, write to Firestore. Skip if user opts to start fresh.
|
||||
8. **Tests**: emulator-based — buy → sell → portfolio → leaderboard parity with JS expectations.
|
||||
9. **firestore.indexes.json**: capture the composite index definition; `gcloud firestore indexes composite create --collection-group=trading_users --field-config=field-path=pnlVnd,order=descending`.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Buy/sell round-trips correctly compute balance + avgCost
|
||||
- [ ] Leaderboard query returns top 10 by pnl in <100ms
|
||||
- [ ] Daily price cron runs (manual trigger via `/cron/trading-daily-update` for now)
|
||||
- [ ] Composite index deployed and active
|
||||
- [ ] Tests pass against emulator
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: Firestore transactions have a 500-doc / 5MB / 10s limit. Trading transactions are tiny — fine.
|
||||
- **Risk**: Leaderboard composite index requires explicit creation (Firestore prompts in console on first failed query). **Mitigation**: capture in `firestore.indexes.json` + deploy via gcloud in CI.
|
||||
- **Risk**: Denormalized `pnlVnd` can drift if a sell update partially fails. **Mitigation**: always update inside transaction with the trade write.
|
||||
- **Risk**: Free tier 20k writes/day. Per active user, a buy+sell = 4 writes (user, trade, holding, price-touched). 300 users × 5 trades/day = 6k writes — well within.
|
||||
- **Risk**: VN stock data source may be unstable. **Mitigation**: port the same source used by JS; if cron fails, retry on next run.
|
||||
|
||||
## Rollback
|
||||
Remove `trading` from `MODULES`. Existing data in `trading_users` collection persists harmlessly; no orphan refs since modules are isolated.
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
phase: 9
|
||||
title: "Cloud Scheduler cron wiring"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "2h"
|
||||
dependencies: [3]
|
||||
---
|
||||
|
||||
# Phase 09: Cloud Scheduler cron wiring
|
||||
|
||||
## Overview
|
||||
Replace CF Worker `[triggers] crons` with Cloud Scheduler. Each module-declared cron becomes a Scheduler job that POSTs to `/cron/{name}` on the Cloud Run service with an OIDC token, which the service validates before dispatching to module cron handlers.
|
||||
|
||||
## Requirements
|
||||
- Functional: 2 jobs run on schedule (`0 17 * * *`, `0 1 * * *`). Each invocation reaches the corresponding module cron handlers and completes within Cloud Run timeout.
|
||||
- Non-functional: free-tier — 3 jobs/mo cap, fits with 33% headroom. OIDC auth so the `/cron/*` endpoint stays Cloud-Scheduler-only (private). No public bypass.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Cloud Scheduler Cloud Run
|
||||
┌───────────────────────┐ ┌───────────────────────────┐
|
||||
│ job: cron-0-17 │ POST + OIDC│ /cron/0_17_star_star_star │
|
||||
│ schedule: 0 17 * * * │────────────►│ ──► validate OIDC │
|
||||
│ target: /cron/0_17... │ │ ──► dispatcher.Dispatch │
|
||||
│ auth: OIDC │ │ (cron name = "0 17 * *│
|
||||
└───────────────────────┘ │ *") │
|
||||
└───────────────────────────┘
|
||||
```
|
||||
|
||||
Path structure: encode the cron expression in the URL (URL-safe form), e.g. `0 17 * * *` → `/cron/0_17_star_star_star`. Or simpler: use a stable name per scheduler job (e.g. `/cron/daily-eod` and `/cron/daily-cleanup`), with the registry mapping name → cron handlers.
|
||||
|
||||
Adopt the named-job approach: cleaner than escaping cron syntax in URLs.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `internal/modules/cron_dispatcher.go` — `DispatchByName(ctx, name string, reg *Registry, deps Deps) error`
|
||||
- Modify: `internal/server/router.go` — `/cron/{name}` handler, validates OIDC token via `google.golang.org/api/idtoken`
|
||||
- Create: `scripts/setup-scheduler.sh` — idempotent `gcloud scheduler jobs create http …` for each cron
|
||||
- Modify: per-module `Cron` declarations to use **stable names** (e.g. `daily-eod-update`, `nightly-cleanup`) instead of cron syntax
|
||||
|
||||
## Implementation Steps
|
||||
1. Refactor `Cron.Schedule` field's role: keep as **documentation only**. Add `Cron.Name` as the stable identifier. Wrangler-style auto-registration is no longer needed.
|
||||
2. Update `cron_dispatcher.go`:
|
||||
- `DispatchByName(ctx, name, reg, deps)`: find all crons across all modules where `c.Name == name`. Run with errgroup. Return aggregate error.
|
||||
3. Update `/cron/{name}` handler:
|
||||
- Reject non-POST → 405.
|
||||
- Validate `Authorization: Bearer <id-token>` header via `idtoken.Validate(ctx, token, audience=cloudRunURL)`. Confirm `email` claim matches the runtime SA. Mismatch → 401.
|
||||
- Call `DispatchByName(ctx, mux.Vars["name"], reg, deps)`.
|
||||
- 200 on success, 500 on dispatcher error (Scheduler retries with backoff).
|
||||
4. `scripts/setup-scheduler.sh`:
|
||||
- For each known cron (currently 2): `gcloud scheduler jobs create http <name> --schedule=<cron> --uri=<cloudrun-url>/cron/<name> --http-method=POST --oidc-service-account-email=<runtime-sa> --oidc-token-audience=<cloudrun-url> --location=asia-southeast1`.
|
||||
- Idempotent: try `update` first, fall back to `create` on not-found.
|
||||
5. Local test: simulate Scheduler call with `gcloud scheduler jobs run <name>`; verify Cloud Run logs show successful dispatch.
|
||||
6. Document in `docs/using-cron.md` (port from JS repo, adjusted for Cloud Scheduler model).
|
||||
|
||||
## Success Criteria
|
||||
- [ ] 2 Scheduler jobs created in `asia-southeast1`
|
||||
- [ ] OIDC validation rejects unsigned POSTs (401)
|
||||
- [ ] Manual `gcloud scheduler jobs run` triggers handler
|
||||
- [ ] Cron handler error → Scheduler retries (configured retry policy)
|
||||
- [ ] Stays within 3-job free cap
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: 3-job hard cap. Adding a 4th cron later → paid tier. **Mitigation**: collapse multiple module crons into a single dispatcher endpoint sharing one Scheduler job; or rely on internal-time-based-fan-out (cheaper but less precise).
|
||||
- **Risk**: OIDC token validation requires correct audience. Misconfig → 401 in prod. **Mitigation**: Phase 09 ends only after manual `jobs run` succeeds.
|
||||
- **Risk**: Cron handler exceeding Cloud Run timeout (default 5 min, our config 30s). Trading daily update fetches ~50 prices serially. **Mitigation**: parallelize price fetches with errgroup + worker pool of 5.
|
||||
|
||||
## Rollback
|
||||
`gcloud scheduler jobs delete <name>`. CF Worker still owns prod cron triggers — no missed runs during transition.
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
phase: 10
|
||||
title: "CI/CD + Dockerfile + Secret Manager"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "4h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 10: CI/CD + Dockerfile + Secret Manager
|
||||
|
||||
## Overview
|
||||
Production-grade build + deploy pipeline. GitHub Actions builds image, pushes to Artifact Registry, deploys to Cloud Run. Secrets pulled from Secret Manager at runtime via Cloud Run's `--set-secrets`. Post-deploy hook runs `setWebhook` + `setMyCommands` against Telegram (replacing JS `scripts/register.js`).
|
||||
|
||||
## Requirements
|
||||
- Functional: PR → CI green; merge to `main` → auto-deploy to Cloud Run; deploy includes Telegram registration.
|
||||
- Non-functional: build time ≤2 min; no secrets in image, in env yaml, or in repo. Workload Identity Federation between GHA + GCP (no long-lived JSON key).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
.github/workflows/
|
||||
├── ci.yml ← PRs: vet, test, build (no deploy)
|
||||
└── deploy.yml ← main: build, push to AR, deploy Cloud Run, register Telegram
|
||||
|
||||
cmd/register/main.go ← Go port of scripts/register.js (setWebhook + setMyCommands)
|
||||
Dockerfile ← finalized multi-stage
|
||||
firestore.indexes.json ← composite indexes (Phase 08)
|
||||
.dockerignore
|
||||
```
|
||||
|
||||
Secret Manager secrets (created in Phase 01, populated here):
|
||||
- `telegram-bot-token`
|
||||
- `telegram-webhook-secret`
|
||||
- `gemini-api-key`
|
||||
|
||||
Cloud Run service env (non-secret):
|
||||
- `MODULES=util,misc,wordle,loldle,loldle-emoji,loldle-quote,loldle-ability,loldle-splash,trading,lolschedule,semantle,doantu,twentyq`
|
||||
- `GOOGLE_CLOUD_PROJECT`
|
||||
- `LOG_LEVEL=info`
|
||||
|
||||
## Related Code Files
|
||||
- Create: `.github/workflows/{ci,deploy}.yml`
|
||||
- Create: `cmd/register/main.go`
|
||||
- Modify: `Dockerfile` (finalize from Phase 02)
|
||||
- Create: `infra/cloud-run.yaml` (declarative service spec) OR keep imperative `gcloud run deploy` flags
|
||||
|
||||
## Implementation Steps
|
||||
1. **Workload Identity Federation setup** (one-time):
|
||||
- `gcloud iam workload-identity-pools create github-pool --location=global`.
|
||||
- `gcloud iam workload-identity-pools providers create-oidc github-provider …`.
|
||||
- Bind `roles/iam.workloadIdentityUser` from GHA repo → `miti99bot-deployer` SA.
|
||||
2. **Dockerfile finalization**:
|
||||
- Builder: `FROM golang:1.23-alpine`, install ca-certs, `CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /server ./cmd/server`.
|
||||
- Runtime: `FROM gcr.io/distroless/static-debian12:nonroot`, `COPY --from=builder /server /server`, `USER nonroot`, `ENTRYPOINT ["/server"]`.
|
||||
3. **`ci.yml`**:
|
||||
- Triggers: `pull_request`, `push: branches: [main]`.
|
||||
- Steps: checkout, setup-go, `go vet ./...`, `go test ./...`, `go build ./...`. (No emulator integration tests in CI — run locally.)
|
||||
4. **`deploy.yml`**:
|
||||
- Trigger: `push: branches: [main]` after `ci` workflow succeeds.
|
||||
- Auth via WIF (`google-github-actions/auth@v2`).
|
||||
- Build + push: `docker build -t asia-southeast1-docker.pkg.dev/$PROJECT/miti99bot-go/server:$SHA .; docker push …`.
|
||||
- Deploy: `gcloud run deploy miti99bot-go --image=… --region=asia-southeast1 --service-account=miti99bot-runtime@… --set-env-vars="MODULES=…,GOOGLE_CLOUD_PROJECT=$PROJECT" --set-secrets="TELEGRAM_BOT_TOKEN=telegram-bot-token:latest,TELEGRAM_WEBHOOK_SECRET=telegram-webhook-secret:latest,GEMINI_API_KEY=gemini-api-key:latest" --min-instances=0 --max-instances=2 --memory=256Mi --cpu=1 --timeout=30s --allow-unauthenticated`.
|
||||
- Apply Firestore indexes: `gcloud firestore indexes composite create --collection-group=trading_users --field-config=field-path=pnlVnd,order=descending` (idempotent — errors on already-exists, swallow).
|
||||
- Apply Scheduler jobs: invoke `scripts/setup-scheduler.sh` with current Cloud Run URL.
|
||||
- Post-deploy: `go run ./cmd/register` reads `MODULES` + Cloud Run URL + bot token from env, calls Telegram `setWebhook` (with secret token) + `setMyCommands` (public commands only). Idempotent.
|
||||
5. **`cmd/register/main.go`**:
|
||||
- Build registry locally (no Firestore — embed an `OfflineKVStore` that no-ops). Walk public commands.
|
||||
- HTTP POST to `https://api.telegram.org/bot<TOKEN>/setWebhook` with `{url, secret_token, allowed_updates: ["message"]}`.
|
||||
- HTTP POST to `…/setMyCommands` with `{commands: [{command, description}]}`.
|
||||
- `--dry-run` flag prints payloads without calling API (parity with JS `register:dry`).
|
||||
6. **Concurrency-1 lock** on `deploy.yml` to prevent overlapping deploys (Cloud Run handles multiple revisions, but webhook race is annoying).
|
||||
7. **Smoke after deploy**: GHA waits 10s, curls `/` → expect 200 "miti99bot-go ok"; if not, fail the workflow.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] PR triggers CI, all checks pass
|
||||
- [ ] Merge → deploy runs end-to-end, Cloud Run revision served
|
||||
- [ ] No secret values appear in workflow logs
|
||||
- [ ] Telegram webhook is set after deploy (verify `getWebhookInfo`)
|
||||
- [ ] `setMyCommands` reflects current `MODULES`
|
||||
- [ ] Image size ≤30 MiB
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: WIF setup is tricky; bad bind → GHA can't auth. **Mitigation**: validate via a manual workflow run before relying on auto-deploy.
|
||||
- **Risk**: Deploy runs Telegram register before Cloud Run is healthy → Telegram pings new URL, gets 503. **Mitigation**: smoke `/` first, register only after.
|
||||
- **Risk**: A bad deploy auto-flips webhook to broken revision. **Mitigation**: Cloud Run keeps prior revision; manual `gcloud run services update-traffic` is the rollback. Document in deployment-guide.md.
|
||||
- **Risk**: Cost spike if `--max-instances` set too high under attack. **Mitigation**: capped at 2 — handles VN-side org load comfortably; raise only if measured.
|
||||
|
||||
## Rollback
|
||||
`gcloud run services update-traffic miti99bot-go --to-revisions=<previous>=100`. Re-register webhook against prev URL not needed (URL is service-level, not revision-level).
|
||||
@@ -1,77 +0,0 @@
|
||||
---
|
||||
phase: 11
|
||||
title: "Test parity + observability"
|
||||
status: partial
|
||||
priority: P3
|
||||
effort: "4h"
|
||||
dependencies: [8]
|
||||
---
|
||||
|
||||
# Phase 11: Test parity + observability
|
||||
|
||||
## Overview
|
||||
Reach test-count parity with the JS suite where applicable. Wire structured JSON logs to Cloud Logging. Add lightweight metrics (counters for command invocations, errors, AI calls). Soak the Go service against a test bot for 48 hours before cutover.
|
||||
|
||||
## Requirements
|
||||
- Functional: every JS test that covers logic (not framework/transport) has a Go counterpart. Logs are JSON-shaped, consumable by Cloud Logging severity filters.
|
||||
- Non-functional: no external metrics backend (free-tier discipline) — Cloud Logging structured fields used as the metrics surface (Log Explorer + Log-based Metrics, all free up to default quota).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
internal/log/
|
||||
├── logger.go ← slog.Logger configured with JSON handler, severity → Cloud Logging convention
|
||||
└── middleware.go ← request log: msg=req method= path= status= ms=
|
||||
|
||||
internal/metrics/
|
||||
└── counters.go ← incrCommand(name), incrError(kind), incrAI(model). Logged at info severity.
|
||||
|
||||
tests/integration/ ← optional: emulator-based end-to-end (not run in CI)
|
||||
```
|
||||
|
||||
`slog` (Go 1.21+) handles JSON output. Cloud Logging auto-parses structured `severity` + `message` + custom fields when written to stdout.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/log/{logger,middleware}.go`
|
||||
- Create: `internal/metrics/counters.go`
|
||||
- Modify: every module command handler — add `metrics.IncCommand("/wordle")` etc.
|
||||
- Modify: `internal/ai/*` — add `metrics.IncAI("embedding")` and `metrics.IncError("ai-429")` paths
|
||||
- Add: per-module `*_test.go` files until parity reached
|
||||
|
||||
## Implementation Steps
|
||||
1. **Logger**: `slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo, ReplaceAttr: replaceLevelKey}))`. Map slog `level` → Cloud Logging `severity` convention (DEBUG/INFO/WARNING/ERROR).
|
||||
2. **Request middleware**: wraps `/webhook`, `/cron/*`. Logs `{msg: "req", method, path, status, ms}` at info. Mirrors JS index.js shape.
|
||||
3. **Counters**: `metrics.IncCommand(name)` increments an in-memory `sync.Map[string]*atomic.Int64`. Periodic flush every 60s logs `{msg: "metrics", commands: {...}, errors: {...}, ai: {...}}` then resets. Graceful shutdown flushes once on SIGTERM.
|
||||
4. **Log-based metrics in GCP** (one-time setup, document in deployment-guide.md):
|
||||
- Counter on `severity=ERROR` → alerts.
|
||||
- Counter on `jsonPayload.msg=req AND jsonPayload.status>=500` → 5xx rate.
|
||||
- Counter on `jsonPayload.msg=metrics` → daily aggregation by command.
|
||||
5. **Test parity audit**:
|
||||
- Run `find . -name "*.test.js" | wc -l` against JS repo for baseline.
|
||||
- Run `find . -name "*_test.go"` count in Go repo.
|
||||
- Aim ≥80% of JS tests have a Go counterpart. Skip framework-only tests (e.g. CF Worker fetch handler tests) — they have no analogue.
|
||||
6. **48-hour soak**:
|
||||
- Point a separate test bot at the Cloud Run service.
|
||||
- Manual playthrough of every module's commands × 3 users.
|
||||
- Watch Cloud Logging for errors. Watch Firestore reads/writes per day.
|
||||
- Capture cold-start P95 (`severity=INFO AND jsonPayload.msg=req` filtered to first request after gap).
|
||||
7. **Compare to Phase 01 baseline**: if Phase 11 cold-start P95 > Phase 01 baseline × 1.5, investigate before cutover (gRPC client init usually the suspect).
|
||||
|
||||
## Success Criteria
|
||||
- [x] **Logger** ported: `internal/log/log.go` exposes `slog.JSONHandler` writing to stdout, severity-aware via `LOG_LEVEL` env (Phase 04 of fix-all-review-findings forward-ported this).
|
||||
- [x] **Request middleware** ported: `internal/server/log_middleware.go` wraps every route and emits `{msg:"req", method, path, status, ms}` per request.
|
||||
- [x] **In-memory counters** ported: `internal/metrics/counters.go` exposes `IncCommand`/`IncError`/`IncAI` with 60s periodic `Flush` to `{msg:"metrics", commands, errors, ai}`. Wired into the dispatcher so every command invocation + handler error is counted; `cmd/server/main.go` runs the flush loop bound to rootCtx (one final flush on SIGTERM).
|
||||
- [x] Test coverage 69.8% across 20 packages (`fix-all-review-findings` Phase 05 raised it from 44.7% baseline). Module-level coverage: champname/keylock/telegram 100%, util 90%, log/chathelper/loldle/wordle/misc 77-81%, others ≥70%.
|
||||
- [ ] All errors during 48h soak triaged — **deferred** (requires Cloud Run deployment).
|
||||
- [ ] Cold-start P95 ≤1.5s — **deferred** (requires Phase 01 GCP baseline).
|
||||
- [ ] Daily Firestore reads <40k cap — **deferred** (production observation).
|
||||
- [ ] Cloud Logging log-based metrics setup — **deferred** (one-time GCP console / `gcloud logging metrics create`; document in `docs/deployment-guide.md` once Phase 01 lands).
|
||||
- [ ] No memory leaks check — **deferred** (production observation).
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: in-memory counters lost when instance scales to zero. **Mitigation**: acceptable — Cloud Logging is the source of truth via per-request log lines; in-memory counters are just convenience for debugging.
|
||||
- **Risk**: 48h soak reveals a cold-start regression we can't fix without major rework. **Mitigation**: trigger an abort criterion → keep CF Worker as primary, treat Go as standby.
|
||||
- **Risk**: log-based metrics setup is fiddly. **Mitigation**: use `gcloud logging metrics create` in `scripts/setup-logging.sh`, idempotent.
|
||||
|
||||
## Rollback
|
||||
None needed — observability is read-only. If logging is overly noisy, lower default level via env var.
|
||||
@@ -1,98 +0,0 @@
|
||||
---
|
||||
phase: 12
|
||||
title: "Cutover + decommission CF Worker"
|
||||
status: pending
|
||||
priority: P3
|
||||
effort: "3h"
|
||||
dependencies: [10, 11]
|
||||
---
|
||||
|
||||
# Phase 12: Cutover + decommission CF Worker
|
||||
|
||||
## Overview
|
||||
Final phase: flip the prod Telegram webhook from CF Worker to Cloud Run, observe a 7-day soak with both deployments side by side (CF still receives bg cron triggers, Cloud Run owns webhook), then decommission the Worker.
|
||||
|
||||
## Requirements
|
||||
- Functional: prod bot answers from Cloud Run after webhook flip. No commands regress. Daily cron jobs continue running.
|
||||
- Non-functional: rollback to CF Worker is one Telegram API call away (`setWebhook` back to Worker URL) at any time during the soak.
|
||||
|
||||
## Architecture
|
||||
|
||||
Cutover sequence:
|
||||
|
||||
```
|
||||
Day 0 test-bot soak passed (Phase 11)
|
||||
↓
|
||||
Day 0 setWebhook(prod-bot, CloudRunURL)
|
||||
↓ ↑ rollback: setWebhook(prod-bot, WorkerURL)
|
||||
Day 0–7 observe error rate, AI quota, Firestore quota
|
||||
↓
|
||||
Day 7 delete CF Worker, KV namespace, D1 database
|
||||
↓
|
||||
Day 7 update README to point at miti99bot-go repo
|
||||
↓
|
||||
Day 7+ journal entry, archive plan
|
||||
```
|
||||
|
||||
Data migration (one-shot, optional):
|
||||
- Trading: export D1 → transform → write to Firestore. Done before cutover so user balances persist. Script: `cmd/migrate-trading/main.go` reads from D1 export JSON, writes via Firestore client.
|
||||
- Loldle/wordle/semantle: session state is per-day; users start fresh on Day 0. No migration.
|
||||
- Twentyq: stateless (history embedded in conversation). No migration.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/migrate-trading/main.go` (one-shot, optional based on user choice)
|
||||
- Modify: `README.md` of JS repo — point to Go repo, mark archived
|
||||
- Modify: `wrangler.toml` — comment out crons, set up auto-disable webhook on next deploy
|
||||
- Create: `docs/cutover-runbook.md` — sequence + rollback exact commands
|
||||
|
||||
## Implementation Steps
|
||||
1. **Pre-flight checklist** (before flipping webhook):
|
||||
- [ ] Phase 11 success criteria all green
|
||||
- [ ] Cloud Run service has prod-secret values, not test
|
||||
- [ ] Telegram `setMyCommands` reflects production module set
|
||||
- [ ] Trading data exported from D1 (if migrating)
|
||||
2. **Trading data import** (only if user opted in):
|
||||
- Export D1: `wrangler d1 execute miti99bot-db --command="SELECT * FROM trading_trades" --json > trades.json`. Repeat for users + holdings tables.
|
||||
- Transform script: read JSON, write to Firestore `trading_users/{id}` + subcollections.
|
||||
- Verify counts match: `gcloud firestore export` count = D1 row count.
|
||||
3. **Webhook flip**:
|
||||
- `curl -F "url=<CloudRunURL>/webhook" -F "secret_token=<secret>" https://api.telegram.org/bot<token>/setWebhook`.
|
||||
- `getWebhookInfo` to confirm.
|
||||
- Send a `/info` to prod bot, verify response from Cloud Run (check Cloud Logging).
|
||||
4. **Soak (Day 0–7)**:
|
||||
- Daily check: Firestore quota usage, error rate from Cloud Logging, Gemini RPD usage.
|
||||
- User-facing channel for issue reports (existing channel).
|
||||
- If criticals → `setWebhook` back to CF Worker URL → diagnose → re-attempt cutover.
|
||||
5. **Decommission (Day 7)**:
|
||||
- `wrangler deployments list` — note current revision (rollback insurance).
|
||||
- `wrangler delete miti99bot` — removes service.
|
||||
- Drop CF KV namespace + D1 database via dashboard.
|
||||
- Remove `wrangler.toml` cron triggers + secrets via `wrangler secret delete`.
|
||||
- JS repo: add archive notice to README, push final commit `chore: archive — superseded by miti99bot-go`.
|
||||
6. **Wrap-up**:
|
||||
- Run `/ck:journal` to capture lessons learned.
|
||||
- `ck plan archive` on this plan.
|
||||
- Mark `260425-1945-mongodb-atlas-migration` plan as superseded (already documented in its frontmatter once this plan is created).
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Prod webhook hits Cloud Run, no commands regress
|
||||
- [ ] 7-day soak completes without rollback
|
||||
- [ ] CF resources fully torn down (Worker, KV, D1)
|
||||
- [ ] JS repo archived, README points to Go repo
|
||||
- [ ] Free-tier budget unbroken throughout soak (no surprise bills)
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk**: Webhook flip is atomic in Telegram but Cloud Run cold-start delays first replies. **Mitigation**: schedule flip during VN low-traffic hours (3-4am Saigon).
|
||||
- **Risk**: User complaints about lost game state (loldle/wordle in-flight). **Mitigation**: announce cutover in channel, urge users to finish open games.
|
||||
- **Risk**: Trading data import has subtle schema mismatch leading to corrupt balances. **Mitigation**: import to a `trading_users_staging` collection first, eyeball-verify a few users, then rename collection (Firestore lacks rename — copy then delete original).
|
||||
- **Risk**: A pending CF cron at the moment of cutover runs against decommissioned data. **Mitigation**: pause CF crons (set `[triggers] crons = []` and deploy) before flipping webhook. Cloud Scheduler picks up at next interval.
|
||||
- **Risk**: Telegram caches commands client-side; `setMyCommands` takes minutes to propagate. **Mitigation**: tolerate; cosmetic only.
|
||||
|
||||
## Rollback
|
||||
- **During soak**: `setWebhook(prod-bot, WorkerURL)` reverts. CF Worker still alive.
|
||||
- **After Day 7 decommission**: rollback requires re-deploying Worker from git history. Document last-known-good wrangler version in cutover-runbook.md before deletion.
|
||||
|
||||
## Next Steps
|
||||
- Archive this plan: `ck plan archive 260508-2222-go-port-cloud-run`
|
||||
- Run `/ck:journal` for retrospective
|
||||
- Update `docs/development-roadmap.md` of Go repo with post-cutover priorities (e.g. observability dashboards, additional modules)
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
title: "Go port of miti99bot to Google Cloud Run (free tier)"
|
||||
description: "Full rewrite of the grammY/Cloudflare Worker bot in Go, deployed to Cloud Run with Firestore + Gemini + Cloud Scheduler — all free-tier."
|
||||
status: in-progress
|
||||
priority: P2
|
||||
effort: 5-7d
|
||||
branch: main
|
||||
tags: [go, cloud-run, gcp, firestore, gemini, port, telegram-bot]
|
||||
created: 2026-05-08
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
supersedes: [260425-1945-mongodb-atlas-migration]
|
||||
---
|
||||
|
||||
# Plan: Go port → Google Cloud Run (free tier)
|
||||
|
||||
> **2026-05-10:** Deploy phases (01, 09–12) **superseded by [`plans/260510-0114-aws-port/`](../260510-0114-aws-port/plan.md)** — strict $0 free-tier goal motivated switch to AWS Lambda + DynamoDB + EventBridge. Module work (phases 03–07) is done and **reused unchanged** by the AWS plan. Phase 08 (trading) remains pending, cloud-agnostic, can be tackled before or after the AWS cutover.
|
||||
|
||||
Full rewrite of miti99bot in Go for deployment on Cloud Run, swapping CF KV+D1+Workers AI for Firestore Native + Gemini API + Cloud Scheduler. Source repo lives at a new `miti99bot-go` repo (separate). Cutover via dual-run + soak.
|
||||
|
||||
## Locked decisions
|
||||
- **Compute**: Cloud Run (min-instances=0, scale-to-zero). Free tier: 2M req/mo, 360k vCPU-s, 180k GiB-s.
|
||||
- **Storage**: Firestore Native, region `asia-southeast1`. Free: 1 GiB, 50k reads/d, 20k writes/d.
|
||||
- **AI**: Gemini API via `google.golang.org/genai`. `text-embedding-004` (768d) + `gemini-1.5-flash`. Free: 15 RPM / 1500 RPD per model.
|
||||
- **Cron**: Cloud Scheduler. 3 jobs/mo free → fits 2 current crons (`0 17 * * *`, `0 1 * * *`).
|
||||
- **Secrets**: Secret Manager. Free: 6 active secret versions, 10k access ops/mo.
|
||||
- **Image registry**: Artifact Registry. Free: 0.5 GiB storage.
|
||||
- **Telegram lib**: `github.com/go-telegram/bot` (modern, idiomatic, active).
|
||||
- **Repo layout**: separate `miti99bot-go` repo. JS/TS repo stays as-is during port.
|
||||
- **Cutover**: dual-run on test bot → flip prod webhook → 7-day overlap → decommission CF Worker.
|
||||
|
||||
## Reports
|
||||
- [code-reviewer 2026-05-08 — Phase 02-03 bootstrap](reports/code-reviewer-260508-2254-phase02-03-bootstrap.md) (3 critical + 7 high addressed in same session; 2 medium + nits deferred)
|
||||
- [code-reviewer 2026-05-08 — Phase 04 Firestore](reports/code-reviewer-260508-2333-phase04-firestore-kv.md) (0 critical, 3 high all addressed; mediums deferred)
|
||||
- [code-reviewer 2026-05-09 — Phase 5a util+misc](reports/code-reviewer-260509-0813-phase5a-util-misc.md) (1 critical /info nil-deref + L2 KV wire-format mismatch with JS, both fixed; M1 doc + L3 escape test applied)
|
||||
- [code-reviewer 2026-05-09 — Phase 5b wordle](reports/code-reviewer-260509-0918-phase5b-wordle.md) (1 critical defaultRNG race + 1 high Get-mutate-Put race + dead-code; all fixed in same session; per-subject mutex added to serialise compound KV ops)
|
||||
- [code-reviewer 2026-05-09 — Phase 5c loldle](reports/code-reviewer-260509-0940-phase5c-loldle.md) (1 high winRate truncation in both loldle AND wordle; both fixed; render + keylock test gaps closed)
|
||||
- [code-reviewer 2026-05-09 — Phase 6a loldle-emoji](reports/code-reviewer-260509-1206-phase6a-loldle-emoji.md) (0 critical/high; JS-wire-format decode test added; shared-helper extraction queued for 6b)
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Key deliverable |
|
||||
|---|-------|--------|--------|-----------------|
|
||||
| 01 | [GCP setup + free-tier baseline](phase-01-gcp-setup.md) | pending | 3h | Hello-world Go on Cloud Run, cold-start P95 captured |
|
||||
| 02 | [New repo bootstrap + webhook skeleton](phase-02-repo-bootstrap.md) | partial | 3h | `miti99bot-go` repo, `/webhook` validates secret token (Cloud Run deploy + Telegram smoke test deferred to Phase 01) |
|
||||
| 03 | [Module framework + storage interfaces](phase-03-module-framework.md) | done | 4h | Module/Command/Cron interfaces, registry, dispatcher |
|
||||
| 04 | [Firestore KVStore + per-module prefixing](phase-04-firestore-kv.md) | done | 4h | `FirestoreKVStore`, emulator tests, KVProvider abstraction (Memory + Firestore) |
|
||||
| 05 | [Port simple modules (util/misc/wordle/loldle)](phase-05-port-simple-modules.md) | done | 6h | 4 KV-only modules at JS parity; shared `internal/keylock` extracted |
|
||||
| 06 | [Port loldle variants + lolschedule](phase-06-port-loldle-variants.md) | done | 5h | All five sub-modules ported (emoji, quote, ability, splash, lolschedule); lolschedule daily-push cron deferred to Phase 09 |
|
||||
| 07 | [Gemini AI + port semantle/doantu/twentyq](phase-07-gemini-ai-modules.md) | done | 6h | `internal/ai` (Embedder/Chatter + per-user bucket); semantle (text-embedding-004), doantu (phow2sim HTTP — JS-parity deviation), twentyq (gemini-2.5-flash) |
|
||||
| 08 | [Port trading + composite indexes](phase-08-port-trading.md) | pending | 6h | VN-stocks paper trading + daily price cron |
|
||||
| 09 | [Cloud Scheduler cron wiring](phase-09-cloud-scheduler.md) | pending | 2h | 2 jobs → `/cron/{name}` with OIDC |
|
||||
| 10 | [CI/CD + Dockerfile + Secret Manager](phase-10-ci-cd.md) | pending | 4h | GHA pipeline → AR → Cloud Run, idempotent |
|
||||
| 11 | [Test parity + observability](phase-11-tests-observability.md) | partial | 4h | Code-side done: `internal/log` (slog JSON), request log middleware, `internal/metrics` counters + 60s flush, dispatcher instrumented. 48h soak + cold-start measurement + log-based metrics setup deferred to post-deploy. |
|
||||
| 12 | [Cutover + decommission CF Worker](phase-12-cutover.md) | pending | 3h | Prod webhook flipped, soak passed, Worker retired |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
```
|
||||
01 ──► 02 ──► 03 ──► 04 ──► 05 ──► 06 ─┐
|
||||
├──► 07 ─────┤
|
||||
└──► 08 ─────┤
|
||||
03 ──────────► 09 ───────┤
|
||||
02 ──────────► 10 ───────┤
|
||||
08 ──► 11 ──► 12 ◄── 10
|
||||
```
|
||||
|
||||
## Free-tier budget at peak
|
||||
|
||||
| Resource | Cap | Expected | Headroom |
|
||||
|---|---|---|---|
|
||||
| Cloud Run req | 2M/mo | ~30k/mo | 99% |
|
||||
| Cloud Run vCPU-s | 360k | ~5k | 99% |
|
||||
| Firestore reads | 50k/day | ~5k/day | 90% |
|
||||
| Firestore writes | 20k/day | ~2k/day | 90% |
|
||||
| Cloud Scheduler jobs | 3 | 2 | 33% |
|
||||
| Gemini RPM (flash) | 15 | <5 burst | 67% |
|
||||
| Gemini RPD | 1500 | ~200 | 87% |
|
||||
| Secret Manager versions | 6 | 3 | 50% |
|
||||
| Artifact Registry storage | 0.5 GiB | <50 MiB | 90% |
|
||||
|
||||
If Firestore reads cap is hit → enable Cloud Run instance-level cache (warm-instance memo). If Gemini RPD cap is hit → degrade twentyq with a "free tier exhausted, retry tomorrow" reply.
|
||||
|
||||
## Abort criteria
|
||||
- **Cold-start P95 > 1.5s** sustained (Phase 01 baseline + Phase 11 soak): retain JS Worker for time-sensitive surfaces.
|
||||
- **Firestore reads > 80% of cap** during Phase 11 soak: add KV-style instance cache before cutover.
|
||||
- **Gemini quota exhaustion** during normal use: switch to lower-RPM-friendly Vertex AI (still free under credit) or accept degraded UX.
|
||||
|
||||
## Rollback
|
||||
Per-phase rollback documented in each phase file. Phase 12 is the only irreversible step; until then, the CF Worker continues to serve prod via existing webhook.
|
||||
|
||||
## Open questions
|
||||
_Resolved 2026-05-08:_
|
||||
1. ~~Scheduler cron names~~ → **Keep `0 17 * * *` UTC** (= midnight Saigon). Cloud Scheduler stays UTC, no behavior change vs. JS Worker.
|
||||
2. ~~Migrate KV/D1 data~~ → **Migrate everything**. One-shot export of D1 + KV → Firestore on cutover. Phase 12 owns the migration script.
|
||||
3. ~~Test Telegram bot~~ → **User creates the bot manually**, token + webhook secret injected via Cloud Run env vars (`TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`).
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "AWS bootstrap + IAM OIDC + SAM skeleton"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "3h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 01: AWS bootstrap + IAM OIDC + SAM skeleton
|
||||
|
||||
## Overview
|
||||
Stand up the AWS account with strict $0 footprint: IAM OIDC trust for GitHub Actions, baseline SAM stack that deploys an empty Lambda + DynamoDB table + Function URL placeholder. Nothing wired to real bot yet.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Empty stack deploys via `sam deploy --guided` from local. GitHub Actions can assume the deploy role via OIDC (no long-lived keys).
|
||||
- **Non-functional:** Region `ap-southeast-1`. Single AWS account. Stack name `miti99bot-aws-port`. All resources tagged `app=miti99bot, env=prod`. Strict free-tier resources only.
|
||||
|
||||
## Architecture
|
||||
```
|
||||
GitHub Actions ─OIDC─► AWS IAM Role (github-deploy)
|
||||
│ └─ trust: token.actions.githubusercontent.com
|
||||
│ └─ scoped: repo:tiennm99/miti99bot:ref:refs/heads/main
|
||||
│
|
||||
└─► CloudFormation (SAM) ─► Lambda + DynamoDB + ParamStore + EventBridge + Logs
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Create: `template.yaml` (SAM root, all resources declared here)
|
||||
- Create: `samconfig.toml` (stack name, region, capabilities)
|
||||
- Create: `aws/iam-github-oidc-trust.json` (one-shot reference doc, not deployed)
|
||||
- Create: `aws/README.md` (commands cheat sheet for first-time setup)
|
||||
- Create: `Makefile` (targets: `build`, `package`, `deploy`, `logs`)
|
||||
- Modify: `.gitignore` (add `.aws-sam/`, `samconfig.toml.local`)
|
||||
|
||||
## Implementation Steps
|
||||
1. Create AWS account (or reuse existing). Enable MFA on root, create IAM admin user for one-time bootstrap. Set region default `ap-southeast-1`.
|
||||
2. Create the GitHub OIDC identity provider in IAM: thumbprint, audience `sts.amazonaws.com`. (One-time, manual or via small CloudFormation snippet.)
|
||||
3. Create IAM role `github-deploy-miti99bot` with trust policy scoped to `repo:tiennm99/miti99bot:ref:refs/heads/main` and `repo:tiennm99/miti99bot:ref:refs/heads/dev`. Attach managed policies for SAM deploy: CloudFormation, Lambda, DynamoDB, EventBridge, IAM (PassRole only), SSM Parameter Store, Logs, S3 (SAM staging bucket).
|
||||
4. Write `template.yaml` skeleton:
|
||||
- `AWSTemplateFormatVersion: '2010-09-09'`, `Transform: AWS::Serverless-2016-10-31`
|
||||
- `Globals.Function`: `Runtime: provided.al2023`, `Architectures: [arm64]`, `MemorySize: 256`, `Timeout: 15`, `Tracing: Active` (still free at this volume)
|
||||
- `Resources.BotFunction`: empty handler (`bootstrap` not yet built), Function URL with `AuthType: NONE`
|
||||
- `Resources.BotTable`: DynamoDB on-demand, PK=`pk` (S), no GSI yet
|
||||
- Outputs: function URL, table name
|
||||
5. Write `samconfig.toml` with stack name, region, capabilities (`CAPABILITY_IAM`).
|
||||
6. First deploy: `sam build && sam deploy --guided` from local using bootstrap admin credentials. Confirm stack reaches `CREATE_COMPLETE`. Save the Function URL.
|
||||
7. Verify GH Actions OIDC by running a one-shot workflow that calls `aws sts get-caller-identity` — confirms trust works without keys.
|
||||
8. Manual smoke: `curl <function-url>` returns 502 (no handler yet) — proves URL is reachable.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] AWS account active, MFA on root, region default `ap-southeast-1`
|
||||
- [ ] GitHub OIDC provider created
|
||||
- [ ] `github-deploy-miti99bot` IAM role assumes successfully from a test GH Actions run
|
||||
- [ ] `sam deploy` succeeds; stack `miti99bot-aws-port` in `CREATE_COMPLETE`
|
||||
- [ ] DynamoDB table `miti99bot` exists, on-demand billing mode
|
||||
- [ ] Function URL reachable (502 expected)
|
||||
- [ ] AWS Cost Explorer shows $0 spend after 24h
|
||||
|
||||
## Risk Assessment
|
||||
- **OIDC trust scope too loose** (any branch / any repo) → Mitigation: scope to specific repo + ref pattern; review `sub` claim in CloudTrail after first successful run.
|
||||
- **IAM policy over-broad** → Mitigation: start with managed policies for speed, tighten in Phase 06 once resource ARNs stable.
|
||||
- **SAM staging bucket created in wrong region / accumulates artifacts** → Mitigation: pin region in samconfig; add lifecycle rule (7-day expiration) on staging bucket.
|
||||
- **CloudFormation drift** if user edits via console → Mitigation: forbid console edits, document in `aws/README.md`.
|
||||
|
||||
## Open questions
|
||||
1. Single account vs separate dev/prod accounts? Single is simpler for solo dev; defer split until usage warrants.
|
||||
2. Reuse SAM staging bucket from existing AWS work or fresh one? Fresh, scoped to this stack, easier to clean up.
|
||||
3. Pin SAM CLI version in `Makefile`? Yes, document expected version (current latest works); rely on `setup-sam` action in CI to pin.
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Lambda runtime (Go ZIP + LWA + Function URL)"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "4h"
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 02: Lambda runtime (Go ZIP + LWA + Function URL)
|
||||
|
||||
## Overview
|
||||
Make the existing Go HTTP server run as a Lambda behind a Function URL with zero handler-code changes, using AWS Lambda Web Adapter. Routes `/` (healthcheck) and `/webhook` work end-to-end with secret verification.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Function URL responds to `GET /` with the existing health JSON, and to `POST /webhook` with the existing Telegram dispatcher logic. Secret-token verification (`X-Telegram-Bot-Api-Secret-Token`) preserved.
|
||||
- **Non-functional:** Cold start P95 < 1.5s for ARM64 Go ZIP. Memory 256 MiB. Timeout 15s. Binary size <30 MiB.
|
||||
|
||||
## Architecture
|
||||
```
|
||||
Telegram ──HTTPS──► Function URL ──► Lambda runtime
|
||||
│
|
||||
├── LWA layer (extension) translates Lambda event → HTTP
|
||||
│ └── localhost:8080 (LWA listens here)
|
||||
│
|
||||
└── bootstrap binary (existing Go server)
|
||||
starts http.ListenAndServe(":8080", ...)
|
||||
dispatcher → modules → DynamoDB / Gemini
|
||||
```
|
||||
|
||||
LWA is added as a Lambda layer; binary just runs `http.ListenAndServe` — no Lambda SDK import required.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/server/lambda.go` (build-tag `lambda`, sets `PORT=8080` defaults; minimal — possibly empty)
|
||||
- Modify: `cmd/server/main.go` — accept `PORT` env (likely already does), confirm graceful shutdown on `SIGTERM` (LWA sends it on shutdown)
|
||||
- Modify: `template.yaml` — wire `BotFunction` properly:
|
||||
- `CodeUri: build/` (ZIP staging)
|
||||
- `Handler: bootstrap`
|
||||
- `Layers: [arn:aws:lambda:ap-southeast-1:753240598075:layer:LambdaAdapterLayerArm64:<latest>]`
|
||||
- `Environment.Variables`: `AWS_LAMBDA_EXEC_WRAPPER=/opt/bootstrap`, `PORT=8080`, `READINESS_CHECK_PATH=/`, `MODULES=util,misc,wordle,...`, `TELEGRAM_BOT_TOKEN={{resolve:ssm-secure:...}}`, etc.
|
||||
- Modify: `Makefile` — add `build-lambda` target: `GOOS=linux GOARCH=arm64 go build -tags lambda.norpc -ldflags="-s -w" -o build/bootstrap ./cmd/server && chmod +x build/bootstrap`
|
||||
- Reference: `internal/server/router.go` (unchanged)
|
||||
- Reference: `internal/telegram/*.go` (unchanged)
|
||||
|
||||
## Implementation Steps
|
||||
1. Confirm `cmd/server/main.go` reads `PORT` env (it does per inspection). Confirm graceful shutdown on `SIGTERM`/`SIGINT`.
|
||||
2. Add `build-lambda` Makefile target. Test locally: `make build-lambda && file build/bootstrap` shows ARM64 ELF.
|
||||
3. Pick latest LWA layer ARN for `ap-southeast-1` ARM64 — pin major version in `template.yaml` with comment linking to release notes.
|
||||
4. Add Function env vars in `template.yaml`. Use `{{resolve:ssm-secure:...}}` for secrets so values never appear in template. Reference Phase 01's Parameter Store names.
|
||||
5. Wire DynamoDB IAM permissions (read/write on table) via `Policies: - DynamoDBCrudPolicy`. Wire SSM read perms via `SSMParameterReadPolicy`.
|
||||
6. Build + deploy: `sam build && sam deploy`. Tail logs: `sam logs --tail`.
|
||||
7. Smoke test:
|
||||
- `curl <function-url>/` → 200 with health JSON
|
||||
- `curl -XPOST <function-url>/webhook -H "X-Telegram-Bot-Api-Secret-Token: wrong"` → 401
|
||||
- `curl -XPOST <function-url>/webhook -H "X-Telegram-Bot-Api-Secret-Token: <real>" -d '{"update_id":1,"message":{"text":"/start","chat":{"id":1},"from":{"id":1}}}'` → 200 (or expected dispatcher response)
|
||||
8. Set Telegram dev-bot webhook to Function URL. Send `/start` from real client. Confirm response in chat.
|
||||
9. Capture cold-start P95 from CloudWatch Logs `Init Duration` field over 20+ invocations (use Powertools or grep). Record in this phase's "Risks" if >1s.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] `make build-lambda` produces ARM64 binary <30 MiB
|
||||
- [ ] `sam deploy` updates `BotFunction` successfully
|
||||
- [ ] `curl <function-url>/` returns health JSON
|
||||
- [ ] Wrong webhook secret → 401
|
||||
- [ ] Correct webhook secret + valid update → dispatcher responds
|
||||
- [ ] Telegram dev bot exchanges messages end-to-end via Function URL
|
||||
- [ ] Cold start P95 < 1.5s
|
||||
|
||||
## Risk Assessment
|
||||
- **LWA cold-start tax** adds ~100ms — acceptable; if not, fall back to `lambda.Start()` adapter path (rewrite handler, more invasive).
|
||||
- **`{{resolve:ssm-secure:...}}` requires CloudFormation perms** — SAM handles this if role has `ssm:GetParameter*`.
|
||||
- **Webhook secret leakage in logs** — confirm `internal/server/router.go` does not log header value; if it does, redact.
|
||||
- **Binary too large** (>50 MiB unzipped) — strip with `-ldflags="-s -w"` (already done); if still too big, audit deps with `go build -ldflags="-s -w" -trimpath` + `goweight`.
|
||||
- **ARM64 incompatibility** with any cgo dep — confirm `CGO_ENABLED=0` in build (existing Dockerfile does this).
|
||||
|
||||
## Open questions
|
||||
1. Should LWA `READINESS_CHECK_PATH` be `/` or a dedicated `/healthz`? `/` works since handler is cheap; revisit if `/` ever does work.
|
||||
2. Telegram delivery reliability with cold-start 1–3s — acceptable in practice (Telegram retries), but document in README.
|
||||
3. Provisioned concurrency to eliminate cold start — kills free tier, defer indefinitely.
|
||||
@@ -1,88 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "DynamoDB KV provider"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "4h"
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 03: DynamoDB KV provider
|
||||
|
||||
## Overview
|
||||
Add `DynamoDBKVStore` + `DynamoDBProvider` as a sibling to the existing Firestore impl, satisfying the same `KVStore` / `KVProvider` interface. Selectable via `KV_PROVIDER=dynamodb|firestore|memory` env. Default in production: `dynamodb`. Firestore impl preserved for parity tests.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** All existing modules' KV ops (Get/Put/Delete/List + JSON convenience methods) work against DynamoDB with byte-for-byte parity to Firestore where observable.
|
||||
- **Non-functional:** Single-table design. On-demand billing. P99 < 50ms for Get/Put. List() with prefix uses `Query` (not `Scan`) — must be cheap.
|
||||
|
||||
## Architecture
|
||||
**Single-table schema (composite key):**
|
||||
```
|
||||
TableName: miti99bot-data
|
||||
PK (pk): string = moduleName (e.g. "wordle")
|
||||
SK (sk): string = caller-provided key (e.g. "user:42:state")
|
||||
attrs:
|
||||
value: binary raw bytes
|
||||
updatedAt: number epoch nanos (parity with Firestore impl)
|
||||
```
|
||||
|
||||
Composite key is the canonical DynamoDB shape for prefix-scan workloads: `Query` supports `begins_with(sk, :prefix)` on the **sort key**, but only `=` on the partition key — so the sort key holds the user-supplied key and the partition key holds the module name (which gives free isolation by partition).
|
||||
|
||||
**Operations:**
|
||||
- `Get(key)` → `GetItem(pk=module, sk=key)` with `ConsistentRead: true` (parity with Firestore strong read)
|
||||
- `Put(key, val)` → `PutItem(pk=module, sk=key, value=val, updatedAt=now)`
|
||||
- `Delete(key)` → `DeleteItem(pk=module, sk=key)`
|
||||
- `List(prefix)` → `Query(pk=module AND begins_with(sk, prefix))` paginated
|
||||
|
||||
**Provider isolation:** `For(moduleName)` returns a `DynamoDBKVStore` bound to that module name; the partition key is the isolation boundary. No prefix wrapping needed at this layer.
|
||||
|
||||
**Reserved word handling:** `value` is reserved in DynamoDB expressions; resolved via `ExpressionAttributeNames` (`#v` → `value`).
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/storage/dynamodb_client.go` — AWS SDK v2 client init, region from env
|
||||
- Create: `internal/storage/dynamodb_kv.go` — `DynamoDBKVStore` (Get/Put/Delete/List + JSON helpers)
|
||||
- Create: `internal/storage/dynamodb_provider.go` — `DynamoDBProvider`, `For()` returns module-bound store
|
||||
- Create: `internal/storage/dynamodb_kv_test.go` — uses `localstack` or DynamoDB Local via `testcontainers-go`
|
||||
- Create: `internal/storage/dynamodb_provider_test.go` — cross-module isolation
|
||||
- Create: `internal/storage/parity_test.go` (optional) — runs the same op sequence against Memory + Firestore + DynamoDB and asserts identical observables
|
||||
- Modify: `cmd/server/main.go` — read `KV_PROVIDER`, branch on value; default `dynamodb` when running on Lambda (detect via `AWS_LAMBDA_FUNCTION_NAME` env)
|
||||
- Modify: `go.mod` — add `github.com/aws/aws-sdk-go-v2`, `…/config`, `…/service/dynamodb`, `…/feature/dynamodb/attributevalue`, `…/feature/dynamodb/expression`
|
||||
|
||||
## Implementation Steps
|
||||
1. Add SDK deps. `go mod tidy`.
|
||||
2. Implement `DynamoDBKVStore` with the same method set as `FirestoreKVStore`. Key composition: `pk = moduleName + "#" + key`.
|
||||
3. `List(prefix)` implementation — `Query` with `KeyConditionExpression` on PK begins-with semantics (use range trick or `BEGINS_WITH` on PK; AWS docs: `BEGINS_WITH` works on sort key only, so use the start/end range trick on PK directly).
|
||||
4. JSON helpers (`GetJSON`, `PutJSON`) — mirror Firestore impl exactly: marshal/unmarshal with `encoding/json`, store as binary, `ErrNotFound` semantics preserved.
|
||||
5. Tests with DynamoDB Local (Docker image `amazon/dynamodb-local`). Add `make dynamodb-local` target. Skip if `DYNAMODB_LOCAL_URL` env unset (so CI without Docker can still build).
|
||||
6. Cross-module isolation test: Put `wordle#k=A`, `loldle#k=B`, assert `wordleStore.Get("k") == A`, `loldleStore.Get("k") == B`, `loldleStore.List("") returns ["k"]` (not `[wordle#k, loldle#k]`).
|
||||
7. Wire provider selection in `main.go`:
|
||||
```go
|
||||
switch os.Getenv("KV_PROVIDER") {
|
||||
case "dynamodb": kv = storage.NewDynamoDBProvider(...)
|
||||
case "firestore": kv = storage.NewFirestoreProvider(...)
|
||||
default: kv = storage.NewMemoryProvider()
|
||||
}
|
||||
```
|
||||
8. Manual smoke against deployed Lambda: send `/start`, then verify `aws dynamodb scan --table-name miti99bot --max-items 5` shows expected keys.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] `dynamodb_kv_test.go` passes against DynamoDB Local
|
||||
- [ ] `dynamodb_provider_test.go` passes (cross-module isolation)
|
||||
- [ ] `parity_test.go` (if added) passes — Memory ≡ Firestore ≡ DynamoDB on observables
|
||||
- [ ] `KV_PROVIDER=dynamodb` works in deployed Lambda end-to-end
|
||||
- [ ] One full game session of `/wordle` → state persists across invocations (cold-start safe)
|
||||
- [ ] List() with prefix returns expected keys, no `Scan` calls in CloudWatch metrics
|
||||
|
||||
## Risk Assessment
|
||||
- **`List` performance** if a module accumulates >1k keys — DynamoDB Query handles this fine via paginated results. Confirm caller iterates pages (or all calls fit in one page).
|
||||
- **Item size limit (400 KB)** — modules generally store small JSON; document the cap and add a `len(val) > 380*1024 → error` guard.
|
||||
- **Eventual consistency** — DynamoDB defaults to eventually consistent reads. Use `ConsistentRead: true` in `Get` to match Firestore's strong default. Costs 2× the RCU but on-demand absorbs it.
|
||||
- **Reserved word `value`** — DynamoDB reserves many names; use `ExpressionAttributeNames` `#v = "value"` to avoid the conflict.
|
||||
- **AWS SDK v2 cold-start tax** (~80ms) — acceptable; cache client instance globally in `init()` or pkg-level var.
|
||||
|
||||
## Open questions
|
||||
1. TTL attribute for ephemeral keys (e.g. wordle daily state)? Add optional `ttl` Number attr; modules opt in via a new method or skip for v1.
|
||||
2. Use single PK or PK+SK? Sticking with single PK for KISS — no current module needs sort-key queries.
|
||||
3. Encryption — DynamoDB uses AWS-owned KMS by default (free); switch to AWS-managed only if compliance demands it.
|
||||
4. Backup strategy — point-in-time recovery is paid; for free-tier hobby use, accept "no backup" and document.
|
||||
@@ -1,92 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "EventBridge cron wiring"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "3h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 04: EventBridge cron wiring
|
||||
|
||||
## Overview
|
||||
Replace the planned Cloud Scheduler design with EventBridge Scheduler. Preserve the existing `/cron/{name}` HTTP route shape inside the Lambda by invoking the Function URL via Scheduler's HTTPS target. Auth via `X-Cron-Token` header sourced from Parameter Store.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Two scheduled jobs fire on cron expressions matching the GCP plan (`0 17 * * *` for daily push, `0 1 * * *` for cleanup or whatever Phase 09 of GCP plan defined). Both routes execute against the live module dispatcher and complete within Lambda timeout.
|
||||
- **Non-functional:** Token rotates without code changes (Parameter Store update). Failure retried 2× with exponential backoff. Dead letters logged.
|
||||
|
||||
## Architecture
|
||||
**Decision:** HTTPS target (Function URL) over direct Lambda invoke. **Why:**
|
||||
- Preserves the existing `/cron/{name}` route + dispatcher code from Phase 03 of GCP plan
|
||||
- Local dev still works: `curl localhost:8080/cron/dailypush -H "X-Cron-Token: ..."`
|
||||
- Single ingress path for observability (one URL, one log group)
|
||||
- Direct invoke would require a separate Lambda entrypoint or routing on event shape — more code, less testable
|
||||
|
||||
**Trade-off accepted:** Slightly less AWS-idiomatic; HTTPS adds ~10ms latency vs direct invoke; not material here.
|
||||
|
||||
```
|
||||
EventBridge Scheduler ─cron─► HTTPS POST <function-url>/cron/{name}
|
||||
+ Header: X-Cron-Token: <from ParamStore>
|
||||
+ AWS Sigv4 NOT used (Function URL AuthType: NONE)
|
||||
│
|
||||
└─► Lambda → router → dispatcher → cron handler
|
||||
```
|
||||
|
||||
**Auth model:** Function URL `AuthType: NONE` (already set in Phase 02 for Telegram). Cron auth = shared-secret header verified server-side. The token lives in Parameter Store (`/miti99bot/prod/cron-token`) and is fetched by Scheduler at invoke time via `SECRETSMANAGER_SECRET` reference (Scheduler supports referencing Parameter Store via `secret reference` in target input transformer, OR plain text in target — for KISS, store the token in Scheduler's invocation HTTP target headers as a templated literal, but referenced from Parameter Store via SAM resource attribute).
|
||||
|
||||
**Simpler concrete approach:** SAM template reads the Parameter Store value at deploy time using `{{resolve:ssm-secure:...}}` in the schedule target's HTTP header config. Token rotation = update parameter, redeploy.
|
||||
|
||||
## Related Code Files
|
||||
- Create: SAM resources `Resources.DailyPushSchedule` (AWS::Scheduler::Schedule)
|
||||
- Create: SAM resources `Resources.CleanupSchedule` (or whichever second cron)
|
||||
- Create: SAM resource `Resources.SchedulerExecutionRole` with `events:InvokeApiDestination` / equivalent for HTTPS targets (or use built-in `aws.UniversalTarget` for `https`)
|
||||
- Modify: `internal/server/router.go` — confirm `/cron/{name}` validates `X-Cron-Token` against env-loaded value (currently has `cronAuthHeader = "X-Cron-Token"`, good)
|
||||
- Modify: `cmd/server/main.go` — load `CRON_TOKEN` env from Parameter Store reference, pass to `Config.CronToken`
|
||||
- Reference: existing module cron registrations in each module's `Cron()` method
|
||||
|
||||
## Implementation Steps
|
||||
1. Define SAM `AWS::Scheduler::Schedule` for each cron job:
|
||||
```yaml
|
||||
DailyPushSchedule:
|
||||
Type: AWS::Scheduler::Schedule
|
||||
Properties:
|
||||
ScheduleExpression: "cron(0 17 * * ? *)" # 17:00 UTC = 00:00 Saigon
|
||||
FlexibleTimeWindow: { Mode: 'OFF' }
|
||||
Target:
|
||||
Arn: arn:aws:scheduler:::http-invoke
|
||||
RoleArn: !GetAtt SchedulerRole.Arn
|
||||
Input: '{"name":"dailypush"}'
|
||||
HttpParameters:
|
||||
HeaderParameters: { X-Cron-Token: '{{resolve:ssm-secure:/miti99bot/prod/cron-token:1}}' }
|
||||
RetryPolicy: { MaximumRetryAttempts: 2, MaximumEventAgeInSeconds: 600 }
|
||||
DeadLetterConfig: { Arn: !GetAtt CronDLQ.Arn }
|
||||
FlexibleTimeWindow: { Mode: OFF }
|
||||
```
|
||||
*(Pseudo — confirm exact `aws.HttpInvoke` target syntax against current AWS SAM docs at deploy time; AWS docs note the API surface is evolving.)*
|
||||
2. Add `CronDLQ` (SQS queue, free tier 1M req/mo).
|
||||
3. Add `SchedulerRole` IAM with `lambda:InvokeFunctionUrl` (or `events:InvokeApiDestination` if going via API destination).
|
||||
4. Provision `/miti99bot/prod/cron-token` in Parameter Store with a 32-byte random value (`openssl rand -hex 32`).
|
||||
5. Verify router rejects requests with wrong/missing token (test exists; confirm).
|
||||
6. Deploy. From AWS console, "run now" each schedule. Confirm CloudWatch log entry shows successful 200 from Lambda.
|
||||
7. Wait one full schedule window (or change to `rate(2 minutes)` temporarily) to confirm automatic firing.
|
||||
8. Restore production cron expressions. Confirm next-fire timestamp.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Both schedules deploy via SAM
|
||||
- [ ] Manual "run now" returns HTTP 200 from Function URL
|
||||
- [ ] Server logs show cron handler executing the right module
|
||||
- [ ] Wrong/missing token → 401, no module side effects
|
||||
- [ ] DLQ receives failed invocations on simulated Lambda error
|
||||
- [ ] Schedule fires automatically once on production cron expression
|
||||
|
||||
## Risk Assessment
|
||||
- **AWS Scheduler HTTPS target maturity** — relatively new feature; if SAM transform doesn't support `aws.HttpInvoke` cleanly, fall back to: Scheduler → SNS → Lambda subscription → existing handler (one extra hop, identical effect). Document fallback in this file.
|
||||
- **Token in template via `resolve`** — at deploy time the value is fetched and embedded into the schedule target's static config; rotation requires redeploy. If frequent rotation needed, switch to a Lambda authorizer pattern (out of scope for v1).
|
||||
- **Cron drift / TZ confusion** — EventBridge `cron()` uses UTC by default (matches Cloud Scheduler behavior in GCP plan). Use `?` for day-of-week-or-month constraint per AWS syntax.
|
||||
- **Cold-start during cron** — first invocation after idle = 1–3s; cron handler logic must complete within Lambda timeout (15s). If a cron handler calls Gemini and exceeds 15s, raise function timeout to 30s (still free).
|
||||
|
||||
## Open questions
|
||||
1. Direct Lambda invoke vs HTTPS target — locked to HTTPS for the reasons above; revisit only if HTTPS proves flaky.
|
||||
2. Single schedule with dynamic `name` vs one schedule per cron — one per cron is clearer in console; switch to dynamic only if cron count grows past ~5.
|
||||
3. Cleanup cron (`0 1 * * *`) — confirm what it does in original miti99bot. Likely TTL-style sweep; review and decide if DynamoDB TTL attribute can replace it (eliminates the cron entirely).
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "GitHub Actions deploy (OIDC + SAM)"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "3h"
|
||||
dependencies: [2, 3, 4]
|
||||
---
|
||||
|
||||
# Phase 05: GitHub Actions deploy (OIDC + SAM)
|
||||
|
||||
## Overview
|
||||
Push to `main` → CI builds the ARM64 Go binary, packages into ZIP, runs `sam deploy` against the existing stack via OIDC-assumed role. No long-lived AWS keys. Idempotent (zero-diff deploys are no-ops).
|
||||
|
||||
## Requirements
|
||||
- **Functional:** PR validates (`go vet`, `go test`, `sam validate`). Push to `main` deploys. Manual workflow_dispatch redeploy supported.
|
||||
- **Non-functional:** Deploy < 4 min. Concurrency: only one deploy at a time per ref. Stack name parameterized by env (default `prod`).
|
||||
|
||||
## Architecture
|
||||
```
|
||||
GitHub push to main
|
||||
└─► .github/workflows/deploy.yml
|
||||
1. checkout
|
||||
2. setup-go (1.25)
|
||||
3. setup-sam
|
||||
4. configure-aws-credentials (OIDC) ─► assume github-deploy-miti99bot
|
||||
5. make build-lambda
|
||||
6. sam build --use-container=false
|
||||
7. sam deploy --no-confirm-changeset --no-fail-on-empty-changeset
|
||||
8. post-deploy smoke (curl <function-url>/)
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Create: `.github/workflows/deploy.yml`
|
||||
- Create: `.github/workflows/ci.yml` — maybe split out validate-only path; or add `if:` guard in `deploy.yml`
|
||||
- Modify: existing `.github/workflows/ci.yml` — add `sam validate` step
|
||||
- Modify: `Makefile` — `deploy` target runs `sam build && sam deploy --no-confirm-changeset`
|
||||
|
||||
## Implementation Steps
|
||||
1. Confirm Phase 01's IAM role trust policy includes `repo:tiennm99/miti99bot:ref:refs/heads/main` and the matching repo subject for any manual deploy path.
|
||||
2. Write `deploy.yml`:
|
||||
```yaml
|
||||
name: Deploy to AWS
|
||||
on:
|
||||
push: { branches: [main] }
|
||||
workflow_dispatch:
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
concurrency: { group: deploy-prod, cancel-in-progress: false }
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-go@v5
|
||||
with: { go-version: '1.25' }
|
||||
- uses: aws-actions/setup-sam@v2
|
||||
with: { use-installer: true }
|
||||
- uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::225603493174:role/github-deploy-miti99bot
|
||||
aws-region: ap-southeast-1
|
||||
- run: make build-lambda
|
||||
- run: sam build
|
||||
- run: sam deploy --no-confirm-changeset --no-fail-on-empty-changeset --stack-name miti99bot-aws-port
|
||||
- name: Smoke test
|
||||
run: |
|
||||
URL=$(aws cloudformation describe-stacks --stack-name miti99bot-aws-port --query "Stacks[0].Outputs[?OutputKey=='FunctionUrl'].OutputValue" --output text)
|
||||
curl -fsSL "$URL/" | jq .
|
||||
```
|
||||
3. Keep the AWS account ID in the committed role ARN for this repo. If the deploy account changes later, update both the workflow ARN and the IAM trust policy together.
|
||||
4. PR validation workflow (`ci.yml`): runs `go vet`, `go test`, `sam validate` (no AWS creds needed for validate).
|
||||
5. Test the full path: open a PR with a trivial change → CI green; merge → deploy fires; smoke step prints health JSON.
|
||||
6. Add a `rollback.yml` workflow_dispatch path: re-run with a chosen commit SHA. CloudFormation handles the rollback inherently.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] PR triggers `ci.yml` only (no AWS deploy)
|
||||
- [ ] Merge to `main` triggers `deploy.yml`
|
||||
- [ ] Deploy succeeds without manual intervention
|
||||
- [ ] Post-deploy smoke step returns 200 from Function URL
|
||||
- [ ] Concurrency lock prevents overlapping deploys
|
||||
- [ ] Re-running a no-op deploy reports "no changes" and exits 0
|
||||
- [ ] No AWS access keys in repo, GitHub Actions secrets, or anywhere
|
||||
|
||||
## Risk Assessment
|
||||
- **OIDC trust misconfiguration** locks deploy out — Mitigation: keep bootstrap admin user (Phase 01) as glass-break recovery; rotate after 90 days.
|
||||
- **`sam build` with native deps fails** — pure Go has no native deps; should be fine.
|
||||
- **CloudFormation drift** between manual changes and CI — Mitigation: forbid console edits; daily `sam deploy --no-execute-changeset` to detect.
|
||||
- **Build cache cold each run** (~90s for Go deps) — Mitigation: `actions/setup-go` cache enabled by default.
|
||||
- **Workflow secret leak via debug logs** — Mitigation: never echo `secrets.*`, GH masks them automatically.
|
||||
|
||||
## Open questions
|
||||
1. Separate dev/staging stacks for PR previews? Out of scope for v1; `prod` only.
|
||||
2. Slack/Telegram notification on deploy success/failure? Defer; CloudWatch logs + `gh run list` suffice initially.
|
||||
3. Pin SAM version in `setup-sam`? Yes — pin to a specific version to avoid surprise breakage.
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
phase: 6
|
||||
title: "Observability + budget alert"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "2h"
|
||||
dependencies: [5]
|
||||
---
|
||||
|
||||
# Phase 06: Observability + budget alert
|
||||
|
||||
## Overview
|
||||
Wire CloudWatch Logs retention, metric filters for key counters, AWS Budgets $1/mo alert, and capture cold-start P95 baseline for the abort criterion in `plan.md`.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Log retention 7 days. Budget alert fires at $1.00 actual. Cold-start P95 measurable from logs.
|
||||
- **Non-functional:** All observability stays in free tier (5 GB log ingest/mo, 10 custom metrics free, 1k AWS Budgets API ops free).
|
||||
|
||||
## Architecture
|
||||
- **Logs:** Lambda's auto-created log group `/aws/lambda/miti99bot-aws-port-BotFunction-*`. SAM sets retention.
|
||||
- **Metrics:** Existing `internal/metrics` package emits counters. Lambda env can ship them via stdout — CloudWatch Logs ingests, log-based metric filters extract `request.duration`, `module.dispatched`, `cron.fired`. Free metric filter quota: unlimited filters, paid for resulting metrics past 10/mo.
|
||||
- **Budget:** `AWS::Budgets::Budget` in SAM template, threshold $1, email alert.
|
||||
- **Cold start:** parse `REPORT` log lines → `Init Duration` field → P50/P95/P99.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `template.yaml` — add `LogRetentionInDays: 7` on `BotFunction` (SAM `LoggingConfig`); add `AWS::Budgets::Budget` resource; add `AWS::Logs::MetricFilter` for key metrics
|
||||
- Create: `aws/dashboards/cold-start-coldwatch.json` (optional, manual import)
|
||||
- Reference: `internal/log/*.go` (slog JSON emitter — already exists)
|
||||
- Reference: `internal/metrics/*.go` (counters + 60s flush — already exists)
|
||||
|
||||
## Implementation Steps
|
||||
1. Add to `template.yaml` under `BotFunction.Properties`:
|
||||
```yaml
|
||||
LoggingConfig:
|
||||
LogFormat: JSON
|
||||
ApplicationLogLevel: INFO
|
||||
SystemLogLevel: WARN
|
||||
LogGroup: !Ref BotFunctionLogGroup
|
||||
```
|
||||
2. Add explicit log group to control retention:
|
||||
```yaml
|
||||
BotFunctionLogGroup:
|
||||
Type: AWS::Logs::LogGroup
|
||||
Properties:
|
||||
LogGroupName: /aws/lambda/miti99bot-aws-port-bot
|
||||
RetentionInDays: 7
|
||||
```
|
||||
3. Add metric filter for cold start:
|
||||
```yaml
|
||||
ColdStartFilter:
|
||||
Type: AWS::Logs::MetricFilter
|
||||
Properties:
|
||||
LogGroupName: !Ref BotFunctionLogGroup
|
||||
FilterPattern: '[report="REPORT", ..., init_label="Init", init_dur_label="Duration:", init_dur, ...]'
|
||||
MetricTransformations:
|
||||
- MetricName: ColdStartInitDuration
|
||||
MetricNamespace: miti99bot
|
||||
MetricValue: $init_dur
|
||||
```
|
||||
4. Add `AWS::Budgets::Budget` (sends email at 80% and 100% of $1):
|
||||
```yaml
|
||||
MonthlyBudget:
|
||||
Type: AWS::Budgets::Budget
|
||||
Properties:
|
||||
Budget:
|
||||
BudgetName: miti99bot-monthly
|
||||
BudgetLimit: { Amount: '1.00', Unit: 'USD' }
|
||||
TimeUnit: MONTHLY
|
||||
BudgetType: COST
|
||||
NotificationsWithSubscribers:
|
||||
- Notification: { ComparisonOperator: GREATER_THAN, NotificationType: ACTUAL, Threshold: 80, ThresholdType: PERCENTAGE }
|
||||
Subscribers: [{ Address: <email>, SubscriptionType: EMAIL }]
|
||||
```
|
||||
5. Deploy. Trigger cold start (`aws lambda update-function-configuration --function-name … --environment 'Variables={…,FORCE_RESTART=$(date +%s)}'`).
|
||||
6. Capture 50 cold starts manually or via a one-shot load script (`hey -n 50 -c 1 -i 30s <function-url>/`) — concurrency=1 with delay forces fresh inits. Compute P95 from CloudWatch Insights:
|
||||
```
|
||||
filter @type = "REPORT"
|
||||
| stats avg(@initDuration), pct(@initDuration, 95)
|
||||
```
|
||||
7. Record P95 in `plan.md`'s "Free-tier budget at peak" or as an addendum here.
|
||||
8. Confirm budget shows up in AWS Console > Budgets and has the email subscriber.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Log group `RetentionInDays: 7` set
|
||||
- [ ] Cold-start P95 captured and < 1.5s (per abort criterion)
|
||||
- [ ] Budget alert visible in console, email subscriber confirmed (test mail received)
|
||||
- [ ] No log group accumulates >500 MiB after 7 days of normal traffic
|
||||
- [ ] CloudWatch Insights query for cold start works without manual setup
|
||||
|
||||
## Risk Assessment
|
||||
- **Email subscriber not confirmed** → first alert silently dropped — Mitigation: send a test from console before relying on it.
|
||||
- **Log retention deleted by SAM redeploy** if log group not explicit → Mitigation: declare log group explicitly (step 2).
|
||||
- **Cold start drift** as deps grow — Mitigation: re-measure quarterly; add a CI step that fails if `bootstrap` binary > 30 MiB.
|
||||
- **Budget delays** (AWS Budgets evaluates ~3× day, not real-time) → Mitigation: also enable Cost Anomaly Detection (free) for spike alerts.
|
||||
|
||||
## Open questions
|
||||
1. Email vs SNS topic for budget alerts? Email is simpler; SNS lets fan-out to webhook later. Start with email, migrate if needed.
|
||||
2. Custom CloudWatch dashboard? Skip for v1 — Insights queries are enough for solo-dev.
|
||||
3. Trace via AWS X-Ray? `Tracing: Active` already set in Phase 02 globals — free at this volume; review traces post-Phase 07.
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
phase: 7
|
||||
title: "Cutover + README + retire GCP paths"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "3h"
|
||||
dependencies: [2, 3, 4, 5, 6]
|
||||
---
|
||||
|
||||
# Phase 07: Cutover + README + retire GCP paths
|
||||
|
||||
## Overview
|
||||
Flip the production Telegram webhook to the AWS Function URL only after the Cloudflare→AWS migration plan has produced a green parity report and a rehearsed final-delta procedure. Keep GCP code paths in tree (Firestore impl, Cloud Run Dockerfile) but unwired by default.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Real production bot serves users from Lambda with durable Cloudflare data already migrated or intentionally archived. No regressions vs prior baseline (whatever ran before — JS Worker or partial GCP).
|
||||
- **Non-functional:** Accept a brief operator-controlled freeze window for the final delta import; no silent data loss. 7-day soak with logs reviewed daily. Fallback path documented.
|
||||
|
||||
## Architecture
|
||||
- **Migration gate:** `plans/260515-2250-cf-data-to-aws-migration/` must finish first; Phase 04 parity report there is the go/no-go input for this phase.
|
||||
- **Freeze-window cutover:** pause Cloudflare cron/webhook writes, run final delta export/import + verify, then call `setWebhook` to point Telegram at the AWS Function URL with the production webhook secret.
|
||||
- **Rollback path:** if the final delta verify fails or AWS smoke fails before the first AWS-served write, restore the prior webhook target. After AWS starts accepting new writes, this cutover is forward-fix only unless a reverse-sync path exists.
|
||||
- **Code:** Firestore impl stays compilable, gated by `KV_PROVIDER=firestore`. Default `KV_PROVIDER=dynamodb` in Lambda env. Cloud Run Dockerfile retained for offline / non-AWS users.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `README.md` — full rewrite of "Run locally", "Build", new "Deploy to AWS" section, status table updated, link to AWS plan, archive link to GCP plan
|
||||
- Modify: `cmd/server/main.go` — default `KV_PROVIDER` selection logic: `dynamodb` if `AWS_LAMBDA_FUNCTION_NAME` set, else `memory`
|
||||
- Create: `docs/deploy-aws.md` — single source of truth for AWS deploy ops (parameter store names, IAM role ARN, smoke commands)
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md` — freeze-window delta import + rollback sequence
|
||||
- Modify: `plans/260508-2222-go-port-cloud-run/plan.md` — top-of-file note: "Deploy phases 01, 09–12 superseded by `plans/260510-0114-aws-port/`. Module work (phases 03–07) reused unchanged."
|
||||
- Optional remove: `Dockerfile` retained for now; revisit in 30 days
|
||||
- Optional remove: GCP-specific docs in `docs/` if any (none observed)
|
||||
|
||||
## Implementation Steps
|
||||
1. Pre-flight checklist (run inside this phase):
|
||||
- [ ] Phase 02 smoke green (manual curl)
|
||||
- [ ] Phase 03 wordle daily state survives deploy + cold start
|
||||
- [ ] Phase 04 cron fired at least one real trigger
|
||||
- [ ] Phase 05 push-to-main auto-deploys
|
||||
- [ ] Phase 06 budget alert email confirmed
|
||||
- [ ] Cold-start P95 < 1.5s confirmed
|
||||
- [ ] `plans/260515-2250-cf-data-to-aws-migration/phase-04-parity-verification-and-rehearsal.md` passed with a saved green report
|
||||
- [ ] Final delta import commands rehearsed during the freeze window
|
||||
2. Pause Cloudflare writes and run the final delta import + verify.
|
||||
3. Run `setWebhook` against production bot:
|
||||
```sh
|
||||
curl -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/setWebhook" \
|
||||
-d "url=$AWS_FUNCTION_URL/webhook" \
|
||||
-d "secret_token=$TELEGRAM_WEBHOOK_SECRET" \
|
||||
-d "drop_pending_updates=false" \
|
||||
-d "allowed_updates=[\"message\",\"callback_query\"]"
|
||||
```
|
||||
4. Verify with `getWebhookInfo`:
|
||||
```sh
|
||||
curl "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getWebhookInfo" | jq .
|
||||
```
|
||||
Confirm `url`, `pending_update_count` near 0, `last_error_date` empty.
|
||||
5. Send a test command (`/start`, `/wordle`, `/twentyq` to exercise Gemini path). Confirm responses match prior behavior and expected migrated state is visible.
|
||||
6. Verify at least one migrated trading account, one existing lolschedule subscriber path, and `/mstats` if `last_ping` was migrated.
|
||||
7. Soak for 7 days: each morning, check CloudWatch Logs for ERROR / WARN, DynamoDB throttle metrics (should be zero), budget current spend (should be $0), Gemini RPD usage (should be far under cap).
|
||||
8. After 7-day soak, update README:
|
||||
- Status table: AWS port phases marked "done"
|
||||
- Replace "Run locally" with two paths: in-memory (no AWS) and DynamoDB Local (with AWS deps)
|
||||
- "Deploy" section: link `docs/deploy-aws.md`, drop Cloud Run instructions
|
||||
- Status badge / link to `plans/260510-0114-aws-port/`
|
||||
9. Add a top-of-file note in `plans/260508-2222-go-port-cloud-run/plan.md` redirecting deploy questions to the AWS plan.
|
||||
10. Tag a release: `git tag v1.0.0-aws -m "AWS deploy default"` and push.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Telegram webhook `getWebhookInfo` shows AWS Function URL
|
||||
- [ ] Production bot answers `/start` from real users with normal latency and expected migrated data
|
||||
- [ ] Final Cloudflare→AWS migration report is green before any CF teardown
|
||||
- [ ] 7-day soak: zero unrecovered errors, zero throttles, zero unexpected spend
|
||||
- [ ] README accurately reflects AWS as the default deploy
|
||||
- [ ] `docs/deploy-aws.md` is sufficient for a fresh dev to redeploy from scratch
|
||||
- [ ] GCP plan file annotated; old phase files preserved for history
|
||||
- [ ] Release tag pushed
|
||||
|
||||
## Risk Assessment
|
||||
- **Final import misses writes** if Cloudflare stays writable during cutover — Mitigation: use the migration plan's freeze-window delta import before `setWebhook`, then verify parity again.
|
||||
- **Latency regression vs JS Worker** — Cloud Run / JS Worker had different cold-start profiles; if users complain, document and consider ARM→x86 swap or provisioned concurrency (kills free tier).
|
||||
- **Hidden Firestore dependency** still wired in some module — Mitigation: grep for `firestore.NewClient` and confirm all paths are gated by `KV_PROVIDER=firestore` env. Add a CI test that builds with `KV_PROVIDER=dynamodb` and asserts Firestore client is not initialized.
|
||||
- **GCP project quietly billing** because resources weren't deleted — Mitigation: explicit step in this phase: `gcloud projects delete <project>` OR `gcloud run services delete` for any deployed services. Check Cloud Console for any orphaned resources.
|
||||
- **Lambda Web Adapter unsupported on a future runtime** — Mitigation: pin LWA layer version, monitor AWS Labs repo.
|
||||
|
||||
## Open questions
|
||||
1. Delete the old GCP project entirely or leave it dormant? Dormant is safe (no GCP free-tier abandonment penalty); delete after 30 days if no regret.
|
||||
2. Keep Dockerfile in repo? Yes — useful for non-Lambda local runs and as reference for any future Cloud Run revival.
|
||||
3. Keep Firestore impl forever or drop after 90 days? Drop only if the parity test proves redundant; the impl itself is small and tested.
|
||||
4. Announce the change anywhere (README badge, release notes) — depends on whether this is a public bot. User decides.
|
||||
@@ -1,77 +0,0 @@
|
||||
---
|
||||
title: "Migrate miti99bot from GCP to AWS (Lambda + DynamoDB + EventBridge, free tier)"
|
||||
description: "Re-target the deploy/runtime layer from Cloud Run + Firestore + Cloud Scheduler to Lambda (Go ZIP + LWA + Function URL) + DynamoDB on-demand + EventBridge Scheduler, region ap-southeast-1, IaC via SAM, CI via GH Actions OIDC. Module code unchanged."
|
||||
status: in-progress
|
||||
priority: P2
|
||||
effort: 3-4d
|
||||
branch: main
|
||||
tags: [aws, lambda, dynamodb, eventbridge, sam, port, telegram-bot, free-tier]
|
||||
created: 2026-05-10
|
||||
blockedBy: [260515-2250-cf-data-to-aws-migration]
|
||||
blocks: []
|
||||
supersedes-deploy-of: [260508-2222-go-port-cloud-run]
|
||||
---
|
||||
|
||||
# Plan: AWS port (Lambda + DynamoDB + EventBridge, free tier)
|
||||
|
||||
Re-target only the deploy/runtime layer. Module work (Phases 03–07 of GCP plan) is **done and reused unchanged**. The KVStore interface (`internal/storage/`) absorbs the swap; `http.Handler` code (`internal/server/`) is preserved via Lambda Web Adapter.
|
||||
|
||||
## Context
|
||||
- **Why switch:** Strict $0 free-tier goal — DynamoDB 25 GiB / 200M req-mo, EventBridge unlimited rules, 100 GB egress all-region beat Firestore 1 GiB, Cloud Scheduler 3-job cap, GCP NA-only egress. See `plans/reports/research-260510-0021-aws-vs-gcp-greenfield-rethink.md`.
|
||||
- **Reused as-is:** module framework, registry, dispatcher, Telegram lib, AI clients, all 11 modules, Firestore impl (kept as sibling for parity tests).
|
||||
- **Replaced:** Cloud Run → Lambda; Firestore → DynamoDB (sibling provider, default switchable via env); Cloud Scheduler → EventBridge Scheduler; Secret Manager → Parameter Store; Artifact Registry → none (ZIP); Cloud Logging → CloudWatch Logs; CF Worker / GCP CI → GH Actions + SAM.
|
||||
|
||||
## Locked decisions
|
||||
- **Compute:** Lambda Go on `provided.al2023`, **ARM64**, ZIP package, binary `bootstrap`, build with `-tags lambda.norpc -ldflags="-s -w"`.
|
||||
- **HTTP:** Lambda Function URL (`AuthType: NONE`) + AWS Lambda Web Adapter layer → existing `http.Handler` runs unchanged.
|
||||
- **KV:** DynamoDB single-table `miti99bot`, composite key `(pk, sk)` where `pk = moduleName` and `sk = caller key`, attr `value` (Binary). On-demand billing.
|
||||
- **Cron:** EventBridge Scheduler → HTTPS target = Function URL `/cron/{name}` with `X-Cron-Token` header (token in Parameter Store). Preserves existing route shape; alternative (direct Lambda invoke) deferred.
|
||||
- **Secrets:** SSM Parameter Store SecureString. Names: `/miti99bot/{env}/telegram-token`, `…/webhook-secret`, `…/gemini-api-key`, `…/cron-token`. Fetched at cold start.
|
||||
- **Region:** `ap-southeast-1` (Singapore).
|
||||
- **IaC:** AWS SAM (`template.yaml`).
|
||||
- **CI:** GitHub Actions, OIDC role, `aws-actions/configure-aws-credentials@v4` + `aws-actions/setup-sam@v2`.
|
||||
- **Logs:** CloudWatch Logs, 7-day retention.
|
||||
- **Cost guard:** AWS Budgets $1/mo alert.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Key deliverable |
|
||||
|---|-------|--------|--------|-----------------|
|
||||
| 01 | [AWS bootstrap + IAM OIDC + SAM skeleton](phase-01-aws-bootstrap.md) | pending (manual) | 3h | AWS account, OIDC trust, empty SAM stack deployable |
|
||||
| 02 | [Lambda runtime (Go ZIP + LWA + Function URL)](phase-02-lambda-runtime.md) | code-done; awaits first deploy | 4h | `/` and `/webhook` served from Lambda; secret-token check passes |
|
||||
| 03 | [DynamoDB KV provider](phase-03-dynamodb-kv.md) | code-done; integration tests skip without DDB Local | 4h | `dynamodb_kv.go` + `dynamodb_provider.go` sibling to Firestore impl, parity tests pass |
|
||||
| 04 | [EventBridge cron wiring](phase-04-eventbridge-cron.md) | pending (blocked on cron handlers, see 260510-0234-pre-deploy-wrapup) | 3h | Scheduler → `/cron/{name}` with token, two crons firing on schedule |
|
||||
| 05 | [GitHub Actions deploy (OIDC + SAM)](phase-05-gha-deploy.md) | done | 3h | `deploy.yml` runs on push to `main`, builds + sam deploys idempotently |
|
||||
| 06 | [Observability + budget alert](phase-06-observability.md) | partial (budget shipped; metric filter in 260510-0234) | 2h | Logs retention set, $1 budget alert, cold-start P95 captured |
|
||||
| 07 | [Cutover + README + retire GCP paths](phase-07-cutover.md) | pending (deploy + migration gated) | 3h | Webhook flipped to Function URL after green CF→AWS migration report; README rewritten; GCP code paths kept but unwired by default |
|
||||
|
||||
## Dependency graph
|
||||
```
|
||||
01 ──► 02 ──► 03 ──► 04 ──► 05 ──► 06 ──► 07
|
||||
└──► 04 ─────►┘
|
||||
```
|
||||
|
||||
## Free-tier budget at peak
|
||||
| Resource | Cap | Expected | Headroom |
|
||||
|---|---|---|---|
|
||||
| Lambda req | 1M/mo | ~30k/mo | 97% |
|
||||
| Lambda compute | 400k GB-s | <5k | 99% |
|
||||
| DynamoDB req | 200M/mo | <100k | 99.9% |
|
||||
| DynamoDB storage | 25 GiB | <50 MiB | 99.8% |
|
||||
| EventBridge invocations | 14M/mo | ~60 (2 crons × ~30 days) | 99.9% |
|
||||
| Parameter Store accesses | unlimited (Standard) | <100/cold-start × ~30 starts | n/a |
|
||||
| Egress | 100 GB/mo | <50 MiB | 99.95% |
|
||||
| CloudWatch Logs ingest | 5 GB/mo | <500 MiB | 90% |
|
||||
|
||||
## Abort criteria
|
||||
- **Cold-start P95 > 1.5s** sustained: investigate ARM64→x86_64 swap or pre-warm with provisioned concurrency (kills free tier; only if user-facing latency unacceptable).
|
||||
- **DynamoDB throttle** under normal load: switch to provisioned mode (still free under 25 RCU/WCU).
|
||||
- **Function URL auth-bypass risk** discovered: switch to API Gateway HTTP API (12-month free, then $1/M).
|
||||
|
||||
## Rollback
|
||||
Until Phase 07 webhook flip, the GCP runtime path remains intact. Per-phase rollback documented in each phase. Phase 07 now depends on `plans/260515-2250-cf-data-to-aws-migration/` producing a green parity report before the webhook moves or any Cloudflare data source is deleted.
|
||||
|
||||
## Open questions
|
||||
1. Direct Lambda invoke for cron vs HTTP loopback via Function URL — final call deferred to Phase 04 implementation.
|
||||
2. Whether to delete Firestore impl after parity confirmed, or keep as offline test backend permanently.
|
||||
3. Single SAM stack vs split (data + compute) — start single, split if iteration speed suffers.
|
||||
@@ -1,57 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Cosmetics: README + plan status sync"
|
||||
status: pending
|
||||
priority: P3
|
||||
effort: "30m"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 01: Cosmetics — README + plan status sync
|
||||
|
||||
## Overview
|
||||
README still says "Cloud Run + Firestore"; AWS-port phases 02/03/05 say "pending" though their code already shipped. Sync both before any new clone or visitor reads stale docs.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** README describes AWS as default deploy, with link to `docs/deploy-aws.md`. AWS-port plan's phase-status table reflects shipped code. GCP plan's tagline note about supersession remains.
|
||||
- **Non-functional:** No code changes. Markdown-only.
|
||||
|
||||
## Architecture
|
||||
N/A — documentation update.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `README.md`
|
||||
- Modify: `plans/260510-0114-aws-port/plan.md` (status column)
|
||||
- Reference (no edit): `plans/260508-2222-go-port-cloud-run/plan.md` (already annotated)
|
||||
|
||||
## Implementation Steps
|
||||
1. Rewrite `README.md`:
|
||||
- Tagline: "Plug-n-play Telegram bot framework in Go. Default deploy: AWS Lambda + DynamoDB + EventBridge (free tier). Cloud Run path retained as alt."
|
||||
- Status table: collapse to one row per work-phase; mark current state honestly.
|
||||
- "Run locally" section: keep in-memory KV path; add note about `make dynamodb-local` for DynamoDB integration testing; keep `make firestore-emulator` line.
|
||||
- "Deploy" section: link `docs/deploy-aws.md` as canonical; mention Dockerfile for non-Lambda hosts.
|
||||
- "Test" section: add `make test-dynamodb` line.
|
||||
2. Update `plans/260510-0114-aws-port/plan.md` phases table:
|
||||
- Phase 01 → still pending (manual user steps)
|
||||
- Phase 02 → "code-done; awaiting first deploy"
|
||||
- Phase 03 → "code-done; integration tests skip without DynamoDB Local"
|
||||
- Phase 04 → still pending (this plan unblocks it)
|
||||
- Phase 05 → "done"
|
||||
- Phase 06 → "partial" — budget alert in template; metric filter deferred to this plan's Phase 02
|
||||
- Phase 07 → still pending (deploy-gated)
|
||||
3. Smoke-render the README locally (`grip` or just open in editor) — confirm headings, links, and code blocks render cleanly.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] README's intro line names AWS as default
|
||||
- [ ] README's status table accurate (no "pending" rows that are actually done)
|
||||
- [ ] AWS-port plan.md phase statuses reflect shipped code
|
||||
- [ ] All link targets resolve (no 404 in `docs/deploy-aws.md`, `aws/README.md`, both plan files)
|
||||
- [ ] No broken markdown rendering
|
||||
|
||||
## Risk Assessment
|
||||
- **Drift between README and plan.md** if updated separately later — Mitigation: this phase is the single place both get touched together; future drift caught in Phase 06 cutover.
|
||||
- **Stale Cloud Run instructions misleading new contributors** — Mitigation: prefix the alt-path section with "Alternative: Cloud Run (deferred)" so the canonical path is unambiguous.
|
||||
|
||||
## Open questions
|
||||
1. Move Cloud Run instructions into `docs/deploy-gcp-cloud-run.md` instead of inlining? Cleaner README but adds a file. Default: inline a short note + link to old plan.
|
||||
2. Add CI badge to README? Skip for v1 — no public bot, no marketing pressure.
|
||||
@@ -1,62 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Cold-start metric filter"
|
||||
status: pending
|
||||
priority: P3
|
||||
effort: "30m"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 02: Cold-start metric filter
|
||||
|
||||
## Overview
|
||||
Add a CloudWatch Logs metric filter that extracts Lambda's `Init Duration` from the auto-emitted `REPORT` line so the AWS-port plan's "P95 < 1.5s" abort criterion is measurable from day one.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** A custom metric `miti99bot/ColdStartInitDuration` exists; samples appear in CloudWatch Metrics within 5 minutes of cold-start.
|
||||
- **Non-functional:** Stays inside CloudWatch's always-free 10 custom metrics. No additional ingest cost (filter operates on existing log stream).
|
||||
|
||||
## Architecture
|
||||
Lambda emits a synthetic `REPORT` line at the end of every invocation. On a cold start, that line includes `Init Duration: <ms>`. A `AWS::Logs::MetricFilter` parses the line and emits a custom metric value into `miti99bot/ColdStartInitDuration` namespace.
|
||||
|
||||
```
|
||||
Lambda invocation → CloudWatch log stream →
|
||||
filter pattern matches "REPORT ... Init Duration: <n>" →
|
||||
publish metric (Namespace: miti99bot, Name: ColdStartInitDuration, Value: <n>)
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `template.yaml` — add `ColdStartMetricFilter` resource
|
||||
|
||||
## Implementation Steps
|
||||
1. Append to `template.yaml` after `BotFunctionLogGroup`:
|
||||
```yaml
|
||||
ColdStartMetricFilter:
|
||||
Type: AWS::Logs::MetricFilter
|
||||
Properties:
|
||||
LogGroupName: !Ref BotFunctionLogGroup
|
||||
FilterPattern: '[report="REPORT", reqid_label="RequestId:", reqid, dur_label="Duration:", dur, dur_unit="ms", bill_label="Billed", bill_dur_label, bill_dur, bill_unit, mem_label, mem_size_label, mem_size, mem_unit, max_label="Max", max_used_label="Memory", max_used_label2="Used:", max_used, max_used_unit, init_label="Init", init_dur_label="Duration:", init_dur, init_unit="ms"]'
|
||||
MetricTransformations:
|
||||
- MetricName: ColdStartInitDuration
|
||||
MetricNamespace: miti99bot
|
||||
MetricValue: $init_dur
|
||||
Unit: Milliseconds
|
||||
```
|
||||
2. Validate locally: `make sam-validate` (offline lint). Should pass.
|
||||
3. After AWS-port Phase 01 manual deploy completes, verify:
|
||||
- `aws logs describe-metric-filters --log-group-name /aws/lambda/miti99bot-aws-port-bot`
|
||||
- Trigger a cold start (`aws lambda update-function-configuration --environment ... ` flip).
|
||||
- `aws cloudwatch get-metric-statistics --namespace miti99bot --metric-name ColdStartInitDuration --statistics Average,Maximum --start-time ... --end-time ... --period 300`
|
||||
|
||||
## Success Criteria
|
||||
- [ ] `template.yaml` has `ColdStartMetricFilter` resource
|
||||
- [ ] `sam validate` passes
|
||||
- [ ] Post-deploy: metric filter visible in AWS console, samples flow within 5 min of a cold start
|
||||
|
||||
## Risk Assessment
|
||||
- **Filter pattern brittleness** — Lambda's REPORT format is stable but unofficial. Mitigation: pattern uses positional + label parsing, tolerant of whitespace; if AWS changes format, filter just stops matching (no error, just zero data).
|
||||
- **Warm-only invocations don't include `Init Duration`** — pattern won't match those, which is correct; we only want cold-start samples.
|
||||
|
||||
## Open questions
|
||||
1. Capture `Duration:` (warm + cold) too as a separate metric? YAGNI — request latency is already in CloudWatch's built-in `Duration` metric for the function.
|
||||
2. Per-region dashboard? Skip — solo dev uses Insights queries.
|
||||
@@ -1,109 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "lolschedule daily-push cron handler"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "3h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 03: lolschedule daily-push cron handler
|
||||
|
||||
## Overview
|
||||
Implement the deferred `lolschedule_daily_push` cron — fans out today's match schedule to subscribers at 08:00 ICT. Requires extending `modules.Deps` to expose `*bot.Bot` (current blocker noted in `internal/modules/lolschedule/lolschedule.go:8-12`).
|
||||
|
||||
## Requirements
|
||||
- **Functional:**
|
||||
- `lolschedule.Module.Crons()` returns one entry: name `daily_push`, schedule `0 1 * * *` (UTC = 08:00 ICT).
|
||||
- Handler reads subscribers, fetches today's matches via existing `api_client.go`, sends formatted message to each chat via `*bot.Bot`.
|
||||
- Failed sends per-chat are logged, do not abort the batch (one bad chat doesn't take down the whole push).
|
||||
- **Non-functional:**
|
||||
- Handler completes within Lambda's 30s default timeout for typical subscriber counts (<100). Past that, paginate or move to async.
|
||||
- Rate-limit-aware: respect Telegram's 30 messages/sec global cap. For low subscriber counts, no batching needed.
|
||||
|
||||
## Architecture
|
||||
**Deps extension (the real work):**
|
||||
```go
|
||||
// internal/modules/module.go
|
||||
type Deps struct {
|
||||
KV storage.KVStore
|
||||
Embedder ai.Embedder
|
||||
Chatter ai.Chatter
|
||||
Env map[string]string
|
||||
Bot *bot.Bot // NEW — nil-safe; modules check before use
|
||||
}
|
||||
```
|
||||
|
||||
`*bot.Bot` is already constructed in `cmd/server/main.go` before `modules.Build`. Wire it into `BuildOptions` (typed, like `Embedder`/`Chatter`) and have `modules.Build` thread it into each module's `Deps`. Modules that don't need it ignore it — same pattern as Gemini.
|
||||
|
||||
**Cron handler:**
|
||||
```go
|
||||
// internal/modules/lolschedule/cron.go (new file)
|
||||
func (m *Module) dailyPush(ctx context.Context) error {
|
||||
if m.deps.Bot == nil {
|
||||
return errors.New("lolschedule: daily push requires bot reference")
|
||||
}
|
||||
subs, err := listSubscribers(ctx, m.kv)
|
||||
if err != nil { return err }
|
||||
matches, err := m.api.TodayMatches(ctx) // existing api_client method
|
||||
if err != nil { return err }
|
||||
msg := formatMatches(matches) // existing format.go helper
|
||||
|
||||
var sent, failed int
|
||||
for _, chatID := range subs {
|
||||
if _, err := m.deps.Bot.SendMessage(ctx, &bot.SendMessageParams{ChatID: chatID, Text: msg}); err != nil {
|
||||
log.Warn("lolschedule push failed", "chat", chatID, "err", err)
|
||||
failed++
|
||||
continue
|
||||
}
|
||||
sent++
|
||||
}
|
||||
log.Info("lolschedule daily push complete", "sent", sent, "failed", failed)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `internal/modules/module.go` — add `Bot *bot.Bot` to `Deps` and `BuildOptions`
|
||||
- Modify: `internal/modules/registry.go` (or wherever `modules.Build` constructs Deps) — thread `Bot` through
|
||||
- Modify: `cmd/server/main.go` — pass `b` (already constructed) into `modules.BuildOptions{Bot: b}`
|
||||
- Create: `internal/modules/lolschedule/cron.go` — `dailyPush` handler + helper
|
||||
- Modify: `internal/modules/lolschedule/lolschedule.go` — implement `Crons()` returning the registration; remove the deferred-cron comment block at line 8-12
|
||||
- Create: `internal/modules/lolschedule/cron_test.go` — table tests for handler with mock bot + KV
|
||||
|
||||
## Implementation Steps
|
||||
1. **Deps extension:**
|
||||
- Add `Bot *bot.Bot` field to `Deps` and `BuildOptions` (in `internal/modules/module.go`)
|
||||
- Update `modules.Build` to copy `BuildOptions.Bot` into each constructed `Deps`
|
||||
- Update `cmd/server/main.go` to pass `Bot: b` in the options literal
|
||||
- Run `go vet ./...` + `go build ./...` — should be clean (additive change)
|
||||
2. **lolschedule cron registration:**
|
||||
- Add `Crons() []modules.Cron` to `Module`, returning `{{Name: "daily_push", Schedule: "0 1 * * *", Handler: m.dailyPush}}` (verify the exact struct shape from `modules.Cron` definition)
|
||||
- Remove the deferred-cron comment block in `lolschedule.go`
|
||||
3. **Handler implementation:**
|
||||
- Write `cron.go` per architecture above
|
||||
- Reuse existing `api_client.TodayMatches` and `format.go` helpers (read these to confirm signatures; adjust if needed)
|
||||
4. **Tests:**
|
||||
- Mock `*bot.Bot` via interface or wrapper; assert SendMessage called once per subscriber
|
||||
- Cover: happy path (3 subs, all succeed); partial failure (1 of 3 fails, batch continues); empty subscribers (no-op, no error); API failure (returns error)
|
||||
5. **Wire dispatch:** confirm `internal/modules/cron_dispatcher.go` (or wherever crons are surfaced to `internal/server/router.go`) picks up the new registration without further wiring. Check by hitting `/cron/lolschedule_daily_push` locally with the right secret token; should call the handler.
|
||||
6. **Local smoke:** `go test ./internal/modules/lolschedule/...` green; manual `curl` against running server with at least one subscriber.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] `modules.Deps.Bot` exposed; nil-safe (modules without it work unchanged)
|
||||
- [ ] `lolschedule.Module.Crons()` returns one entry
|
||||
- [ ] `dailyPush` handler implemented per architecture
|
||||
- [ ] Cron-handler unit tests pass (happy path + partial failure + empty subs)
|
||||
- [ ] `go vet`, `go build`, full `go test` green
|
||||
- [ ] Manual `/cron/lolschedule_daily_push` invocation works locally and triggers fan-out
|
||||
|
||||
## Risk Assessment
|
||||
- **Deps extension breaks every module** if not nil-safe — Mitigation: zero-value `*bot.Bot` is `nil`, all current modules ignore the field, additive change. Add a registry test that builds a module without Bot to confirm.
|
||||
- **Telegram global rate limit** (30 msg/sec) on large subscriber counts — Mitigation: add a 50ms sleep between sends if subs > 30; below that, send hot. Document threshold in cron handler.
|
||||
- **Handler exceeds Lambda 30s timeout** at very large sub counts — Mitigation: estimate at 100 subs × 100ms each = 10s, comfortably under. If breached, raise function timeout to 60s in `template.yaml` (still free).
|
||||
- **Long-poll bot client used elsewhere** vs cron-time short-lived calls — current `bot.Bot` instance is shared; SendMessage is goroutine-safe per the lib's design. Confirm in upstream go-telegram/bot docs if uncertain.
|
||||
|
||||
## Open questions
|
||||
1. Do we want per-subscriber timezone awareness, or push to all at the global 08:00 ICT? Original miti99bot pushes globally — match for parity.
|
||||
2. Failure handling: store failed-chat IDs for retry on next push, or just log? Default: log only; transient Telegram failures resolve naturally.
|
||||
3. Should daily push respect a "no matches today" outcome with a quiet skip vs. sending an empty message? Quiet skip; matches parity with original.
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Trading module port (VN stocks paper trading)"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "6h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 04: Trading module port (VN stocks paper trading)
|
||||
|
||||
## Overview
|
||||
Port the `trading` module from the original miti99bot to Go. Paper-trading on Vietnam-listed stocks: per-user portfolio + buy/sell commands + daily price refresh cron. Largest remaining cloud-agnostic chunk; carries from `plans/260508-2222-go-port-cloud-run/` Phase 08 unchanged in scope.
|
||||
|
||||
## Requirements
|
||||
- **Functional:**
|
||||
- Commands (parity with original): `/buy <ticker> <qty>`, `/sell <ticker> <qty>`, `/portfolio`, `/price <ticker>`, `/leaderboard`
|
||||
- Daily price refresh cron at market-close (Vietnam: 15:00 ICT = UTC 08:00) — fetch latest closes for tracked tickers, store snapshot, recompute portfolio P&L
|
||||
- Per-user paper-money starting balance, persistent ledger of trades
|
||||
- Leaderboard: top-N users by total portfolio value
|
||||
- **Non-functional:**
|
||||
- Stays inside Firestore + DynamoDB free tiers (per-user state in 1-3 KV keys, leaderboard a single derived doc)
|
||||
- Daily price API call counts: <50/day (well inside any reasonable free tier on the data source)
|
||||
|
||||
## Architecture
|
||||
**Module shape** mirrors existing modules (e.g. `wordle`):
|
||||
```
|
||||
internal/modules/trading/
|
||||
trading.go Module struct + factory + Commands() + Crons()
|
||||
api_client.go HTTP client to VN-stocks data source
|
||||
api_client_test.go
|
||||
portfolio.go core domain: Portfolio struct, buy/sell, mark-to-market
|
||||
portfolio_test.go
|
||||
handlers.go command handlers (/buy, /sell, /portfolio, /price, /leaderboard)
|
||||
handlers_test.go
|
||||
cron.go daily price refresh handler
|
||||
cron_test.go
|
||||
format.go message formatting helpers
|
||||
format_test.go
|
||||
```
|
||||
|
||||
**KV layout** (per-module partition, no cross-keys):
|
||||
- `user:<id>:portfolio` → JSON Portfolio (cash, positions, trade history)
|
||||
- `prices:<ticker>` → JSON {price, timestamp}
|
||||
- `leaderboard` → JSON sorted list (recomputed by cron)
|
||||
- `tickers` → JSON array (set of tracked tickers across all users; cron iterates this)
|
||||
|
||||
**Concurrency:** Buy/sell mutate the same user portfolio; reuse existing `internal/keylock` (already in repo from wordle work) keyed by `user:<id>:portfolio`.
|
||||
|
||||
**Cron:** registered with name `daily_refresh`, schedule `0 8 * * *` (UTC = 15:00 ICT, market close). Iterates `tickers`, fetches each price, updates `prices:*`, recomputes leaderboard.
|
||||
|
||||
## Related Code Files
|
||||
- Create: all files under `internal/modules/trading/` (per architecture above)
|
||||
- Modify: `cmd/server/main.go` — add `"trading": trading.New` to factories map
|
||||
- Reference: `internal/modules/wordle/` as the closest existing template (commands + state + per-user mutex)
|
||||
- Reference: `internal/modules/lolschedule/api_client.go` as the closest HTTP-fetching template
|
||||
|
||||
## Implementation Steps
|
||||
1. **Locate original miti99bot trading source** — review the JS implementation in https://github.com/tiennm99/miti99bot to nail down exact command shape, message formats, leaderboard rules, and the data source URL.
|
||||
2. **Verify data source** — confirm the API used by original miti99bot is still free + accessible. If not, evaluate alternatives (TCBS public API, VPS public API, etc.). Document the choice in `api_client.go` header.
|
||||
3. **Stub `api_client.go`** with the chosen endpoint + request shape; unit-test with golden HTTP fixtures (no network calls in tests).
|
||||
4. **Domain (`portfolio.go`):** pure Go, no I/O — Portfolio struct, Buy/Sell methods returning new state + delta, mark-to-market against a price map. Heavily unit-tested (this is the easiest part to get wrong silently).
|
||||
5. **Handlers (`handlers.go`):** parse args, load portfolio from KV under `keylock`, call domain method, persist, format reply. Mirror error paths from original (insufficient funds, unknown ticker, etc.).
|
||||
6. **Cron (`cron.go`):** fetch tickers list, iterate, fetch each price, update KV, recompute leaderboard. Returns aggregate counts in log.
|
||||
7. **Wire in `cmd/server/main.go`:** add factory line; bump `MODULES` env default in `template.yaml` to include `trading`.
|
||||
8. **Tests:** ≥80% coverage on `portfolio.go` (domain), happy paths on handlers + cron with mock `api_client`. Match the bar set by `wordle`.
|
||||
9. **Smoke locally:** `MODULES=trading go run ./cmd/server`; exercise `/buy`, `/sell`, `/portfolio`; manually trigger `/cron/trading_daily_refresh`.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] All five commands implemented at parity with original miti99bot
|
||||
- [ ] Daily refresh cron registered (Phase 05 wires it to AWS Scheduler)
|
||||
- [ ] Portfolio domain has ≥80% test coverage
|
||||
- [ ] No flaky tests; no network calls in unit tests (HTTP fixtures only)
|
||||
- [ ] `go vet`, `go test ./internal/modules/trading/...`, full `go build` green
|
||||
- [ ] Local smoke against in-memory KV exercises buy/sell/portfolio/leaderboard end-to-end
|
||||
- [ ] `template.yaml` MODULES default updated to include `trading`
|
||||
|
||||
## Risk Assessment
|
||||
- **Original API source no longer free** — Mitigation: pre-flight check in step 2; if blocked, document fallback (web scrape with caching, or paid tier acceptance, or feature gate the module) and re-scope this phase.
|
||||
- **Time zone bugs in market close** — Mitigation: store all timestamps as UTC, format for display only; unit-test the cron's UTC→ICT translation explicitly.
|
||||
- **Leaderboard recomputation cost** at scale — Mitigation: full recompute is O(users); under 1k users this is single-digit ms. Past that, switch to incremental updates triggered on each trade.
|
||||
- **Schema drift between paper-money currency and real ticker prices** (VND vs USD vs cents) — Mitigation: portfolio stores integer minor units (VND đồng, no fractional); document this loudly in `portfolio.go` header.
|
||||
- **Concurrent buy/sell on same user** — Mitigation: `keylock` per `user:<id>:portfolio` already proven in wordle.
|
||||
|
||||
## Open questions
|
||||
1. Starting balance default — match original (likely 100M VND)? Confirm in step 1.
|
||||
2. Allow short selling? Original likely doesn't; default to "long only, can't sell what you don't own."
|
||||
3. Price ticks during market hours: refresh on `/price <ticker>` command, or only on cron? Cron-only is simpler and free-tier-friendlier; confirm against original behavior.
|
||||
4. Should the cron also run on weekends (Vietnam market closed)? Skip Sat/Sun in handler; emit a no-op log line.
|
||||
5. Multi-user leaderboard privacy — show user IDs or display names? Match original behavior; default to display name with fallback to "user-<id>".
|
||||
@@ -1,105 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Wire EventBridge schedules to live cron handlers"
|
||||
status: deferred
|
||||
priority: P3
|
||||
effort: "30m"
|
||||
dependencies: [3, 4]
|
||||
---
|
||||
|
||||
> **Status update 2026-05-10:** Deferred to first-deploy decision. Two issues surfaced during Phase 03/04 implementation:
|
||||
> 1. **Trading module has no cron** in upstream — only one schedule needed (lolschedule_daily_push), not two
|
||||
> 2. **Lambda Web Adapter only handles HTTP-shape events** — direct Scheduler→Lambda invokes bypass LWA, requiring either an event-shape detector in `main.go` or the HTTPS-target universal-invoke pattern (`arn:aws:scheduler:::http-invoke`, added in 2024)
|
||||
>
|
||||
> The HTTPS-target syntax in `AWS::Scheduler::Schedule` needs validation against the deploy-region SAM transform; doing this offline without `sam validate` access risks committing infra that won't deploy. Decision deferred to deploy-time. Once user runs Phase 01 of AWS-port plan and has SAM available, add a single schedule for `lolschedule_daily_push` per the prose below — pick HTTPS or direct invoke based on what `sam validate` accepts.
|
||||
|
||||
# Phase 05: Wire EventBridge schedules to live cron handlers
|
||||
|
||||
## Overview
|
||||
With Phases 03 + 04 landed, two cron routes exist (`/cron/lolschedule_daily_push`, `/cron/trading_daily_refresh`). This phase adds concrete `AWS::Scheduler::Schedule` resources to `template.yaml` so AWS Scheduler invokes them on schedule via the existing `SchedulerExecutionRole` + `CronDLQ` already provisioned by AWS-port Phase 01.
|
||||
|
||||
## Requirements
|
||||
- **Functional:** Two schedules deploy via SAM. Each fires at the correct cron expression with `X-Cron-Token` header sourced from Parameter Store. Failures land in `CronDLQ`.
|
||||
- **Non-functional:** Stays inside EventBridge Scheduler free tier (14M invocations/mo; we use ~60). Token rotation = update SSM param + redeploy (acceptable trade-off).
|
||||
|
||||
## Architecture
|
||||
```
|
||||
EventBridge Scheduler (rule: 0 1 * * ? *) ─HTTPS POST─► <FunctionURL>/cron/lolschedule_daily_push
|
||||
+ Headers: X-Cron-Token: {{from SSM}}
|
||||
+ Retry: max 2, max-age 600s
|
||||
+ DLQ: CronDLQ on permanent failure
|
||||
|
||||
EventBridge Scheduler (rule: 0 8 * * ? *) ─HTTPS POST─► <FunctionURL>/cron/trading_daily_refresh
|
||||
(same auth + retry + DLQ shape)
|
||||
```
|
||||
|
||||
**HTTPS target syntax:** EventBridge Scheduler uses `arn:aws:scheduler:::http-invoke` with `HttpParameters` carrying headers. SAM's `AWS::Scheduler::Schedule` resource passes through to this; no SAM transform magic needed.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `template.yaml` — append `LolscheduleDailyPushSchedule` + `TradingDailyRefreshSchedule` resources
|
||||
- Reference (no edit): existing `SchedulerExecutionRole` + `CronDLQ` in `template.yaml`
|
||||
- Reference: `aws/README.md` (SSM parameter setup for `/miti99bot/prod/cron-shared-secret`)
|
||||
|
||||
## Implementation Steps
|
||||
1. Confirm AWS SDK / CloudFormation supports `aws.HttpInvoke` target via `AWS::Scheduler::Schedule` for the deploy region (`ap-southeast-1`). Check via `aws cloudformation describe-type --type RESOURCE --type-name AWS::Scheduler::Schedule` if uncertain.
|
||||
2. Append to `template.yaml`:
|
||||
```yaml
|
||||
LolscheduleDailyPushSchedule:
|
||||
Type: AWS::Scheduler::Schedule
|
||||
Properties:
|
||||
Name: !Sub "${AWS::StackName}-lolschedule-daily-push"
|
||||
ScheduleExpression: "cron(0 1 * * ? *)" # 01:00 UTC = 08:00 ICT
|
||||
FlexibleTimeWindow: { Mode: OFF }
|
||||
State: ENABLED
|
||||
Target:
|
||||
Arn: !GetAtt BotFunction.Arn # Lambda direct? Or HTTPS? Decide per step 1
|
||||
RoleArn: !GetAtt SchedulerExecutionRole.Arn
|
||||
RetryPolicy: { MaximumRetryAttempts: 2, MaximumEventAgeInSeconds: 600 }
|
||||
DeadLetterConfig: { Arn: !GetAtt CronDLQ.Arn }
|
||||
Input: '{"name":"lolschedule_daily_push"}'
|
||||
# IF using HTTPS invoke (preferred for route preservation):
|
||||
# Replace `Arn: !GetAtt BotFunction.Arn` with the universal target
|
||||
# `Arn: arn:aws:scheduler:::http-invoke` and add HttpParameters.
|
||||
|
||||
TradingDailyRefreshSchedule:
|
||||
Type: AWS::Scheduler::Schedule
|
||||
Properties:
|
||||
Name: !Sub "${AWS::StackName}-trading-daily-refresh"
|
||||
ScheduleExpression: "cron(0 8 * * ? *)" # 08:00 UTC = 15:00 ICT (market close)
|
||||
FlexibleTimeWindow: { Mode: OFF }
|
||||
State: ENABLED
|
||||
Target:
|
||||
# Same shape as above
|
||||
RoleArn: !GetAtt SchedulerExecutionRole.Arn
|
||||
RetryPolicy: { MaximumRetryAttempts: 2, MaximumEventAgeInSeconds: 600 }
|
||||
DeadLetterConfig: { Arn: !GetAtt CronDLQ.Arn }
|
||||
Input: '{"name":"trading_daily_refresh"}'
|
||||
```
|
||||
3. **Decide direct-invoke vs HTTPS** at implementation time:
|
||||
- **HTTPS (preferred):** preserves `/cron/{name}` route; works with existing dispatcher; same shape as local-dev `curl` smoke. Need `HttpParameters` block with `X-Cron-Token` header.
|
||||
- **Direct Lambda invoke:** simpler IAM, lower latency, bypasses HTTP layer. Requires a Lambda event-shape branch in `cmd/server/main.go` to detect Scheduler events vs Function URL events.
|
||||
- Default: HTTPS for KISS; switch only if HTTPS proves flaky.
|
||||
4. Validate locally: `make sam-validate` should pass.
|
||||
5. After AWS-port Phase 01 deploy:
|
||||
- Console → EventBridge Scheduler → "Run now" each rule. Confirm 200 from Lambda.
|
||||
- Check CloudWatch log group for the cron handler executing.
|
||||
- Send a synthetic invocation that fails (wrong token) — confirm DLQ receives the failed message.
|
||||
6. Watch first scheduled fire from the AWS console (use a temporary `rate(2 minutes)` to verify, then revert).
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Two schedules in `template.yaml`
|
||||
- [ ] `sam validate` passes
|
||||
- [ ] Post-deploy: Manual "run now" returns 200 and triggers handler
|
||||
- [ ] DLQ receives failed invocations (synthetic test)
|
||||
- [ ] First scheduled fire happens at the correct UTC time
|
||||
|
||||
## Risk Assessment
|
||||
- **`AWS::Scheduler::Schedule` HTTPS-target syntax** still evolving — mitigated by step 1 confirmation and ability to fall back to direct invoke.
|
||||
- **Token mismatch between SSM and Lambda env** — both resolve at deploy time from the same parameter; no drift unless one is rotated independently.
|
||||
- **Cron firing before Lambda is deployed** during stack creation — CloudFormation orders dependencies; Schedules `DependsOn: BotFunction` if needed (probably auto from Arn ref).
|
||||
- **Time-zone confusion** — cron expressions use UTC; verified in comments next to each expression.
|
||||
|
||||
## Open questions
|
||||
1. Direct invoke vs HTTPS — final decision lives here, not Phase 04 of AWS-port plan.
|
||||
2. Add a third schedule for a manual "ad-hoc" endpoint (e.g. for testing without console)? YAGNI — `aws scheduler invoke-now` works.
|
||||
3. Schedule `State: ENABLED` vs `DISABLED` initially? ENABLED — first deploy implicitly trusts the cron handlers; if either causes prod issues, disable via console immediately.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
title: "Pre-deploy wrap-up: cron handlers + trading + cosmetics"
|
||||
description: "Cloud-agnostic Go work + small SAM additions to land before Phase 01 AWS bootstrap. Outputs feed directly into AWS-port plan's Phase 04 + 06 + 07 verification."
|
||||
status: in-progress
|
||||
priority: P2
|
||||
effort: 8h
|
||||
branch: main
|
||||
tags: [aws, modules, lolschedule, trading, observability, readme]
|
||||
created: 2026-05-10
|
||||
blockedBy: []
|
||||
blocks: [260510-0114-aws-port]
|
||||
---
|
||||
|
||||
# Plan: Pre-deploy wrap-up
|
||||
|
||||
Five focused phases that finish all **non-deploy** remaining work. Designed to land *before* the user runs Phase 01 of the AWS-port plan (manual AWS account + first `sam deploy`). After this plan ships, AWS-port Phases 04, 06, 07 become genuinely meaningful (real cron handlers, real metric data, accurate README at cutover).
|
||||
|
||||
## Why these five
|
||||
|
||||
From the punch-list:
|
||||
1. **Cosmetics** — README still GCP-flavored; AWS-port plan statuses say "pending" for code that already shipped
|
||||
2. **Metric filter** — small additive SAM change; trivial to land now
|
||||
3. **lolschedule daily-push cron** — was deferred from GCP plan; without it Phase 04 schedules nothing real
|
||||
4. **Trading module** — biggest pending Go chunk; cloud-agnostic; from old GCP Phase 08
|
||||
5. **EventBridge schedules** — wires the new cron handlers to AWS Scheduler
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Key deliverable |
|
||||
|---|-------|--------|--------|-----------------|
|
||||
| 01 | [Cosmetics: README + plan status sync](phase-01-cosmetics.md) | done | 30m | README rewritten for AWS default; AWS-port phases marked code-done |
|
||||
| 02 | [Cold-start metric filter](phase-02-metric-filter.md) | done | 30m | `AWS::Logs::MetricFilter` for `Init Duration` in `template.yaml` |
|
||||
| 03 | [lolschedule daily-push cron](phase-03-lolschedule-cron.md) | done | 3h | `Crons()` registered; Deps exposes bot for fan-out; daily push at 08:00 ICT |
|
||||
| 04 | [Trading module port](phase-04-trading-module.md) | done (scope-trimmed: no daily refresh cron, no leaderboard — neither in upstream) | 4h | VN-stocks paper trading: topup/buy/sell/stats/convert; KBS price source |
|
||||
| 05 | [Wire EventBridge schedules](phase-05-eventbridge-schedules.md) | **deferred to first-deploy decision** | 30m | `AWS::Scheduler::Schedule` resource for lolschedule cron — needs HTTPS-vs-direct-invoke call validated against live SAM CLI |
|
||||
|
||||
## Dependency graph
|
||||
```
|
||||
01 ──┐ (README + status — independent)
|
||||
02 ──┤ (metric filter — independent)
|
||||
03 ──┐
|
||||
├──► 05 (schedules need real handlers from 03 + 04)
|
||||
04 ──┘
|
||||
```
|
||||
|
||||
01 and 02 can ship in any order, including parallel. 05 blocks on 03 + 04 having registered crons.
|
||||
|
||||
## Relation to other plans
|
||||
- Builds on: `plans/260510-0114-aws-port/` (the offline artifacts already shipped)
|
||||
- Carried-over from `plans/260508-2222-go-port-cloud-run/` Phase 08 (trading) — that phase is fulfilled by this plan's Phase 04
|
||||
- After this lands, AWS-port Phase 04 (EventBridge) and Phase 06 (metric capture) become verifiable on first deploy
|
||||
|
||||
## Out of scope (explicit non-goals)
|
||||
- AWS account creation / IAM OIDC / first `sam deploy` (= AWS-port Phase 01, user-manual)
|
||||
- Telegram webhook flip (= AWS-port Phase 07, deploy-gated)
|
||||
- 7-day soak observations (= AWS-port Phase 07)
|
||||
- Provisioned concurrency, DynamoDB TTL, X-Ray dashboard customization (YAGNI for v1)
|
||||
|
||||
## Abort criteria
|
||||
- Phase 03 hits architectural friction extending `modules.Deps` to expose `*bot.Bot` cleanly → split into a smaller Deps refactor PR first, defer cron handler.
|
||||
- Phase 04 trading API source unavailable / paywalled → stub the data layer, mark module disabled by default.
|
||||
|
||||
## Open questions
|
||||
1. Daily-push timezone: ICT 08:00 = UTC 01:00 — confirm cron expression matches in Phase 03.
|
||||
2. Trading data source: original miti99bot uses VN stocks API — confirm it's still free + accessible in Phase 04.
|
||||
3. Should Phase 01 (README) wait until trading module is done, so the README can advertise it? Tradeoff: ship docs sooner vs. ship complete picture. Default: ship now, update README when trading lands.
|
||||
-189
@@ -1,189 +0,0 @@
|
||||
# Code review — lolschedule cron + trading module + framework changes
|
||||
|
||||
Date: 2026-05-10
|
||||
Reviewer: code-reviewer (staff)
|
||||
Scope: 1× framework change, 1× new cron, 1× new module (~30 files)
|
||||
Verification: build clean, `go vet` clean, `go test -race` 24/24 green; **but `gofmt -l` reports 1 file dirty** (see B1).
|
||||
|
||||
---
|
||||
|
||||
## Critical (blocks merge)
|
||||
|
||||
### C1. `gofmt -l` fails on `internal/modules/trading/handlers.go`
|
||||
Line 22-28 of `state` struct has misaligned field tags. `gofmt -d` proposes:
|
||||
```
|
||||
- kv storage.KVStore
|
||||
- prices *PriceClient
|
||||
- locks keylock.Map
|
||||
- nowFn func() time.Time
|
||||
+ kv storage.KVStore
|
||||
+ prices *PriceClient
|
||||
+ locks keylock.Map
|
||||
+ nowFn func() time.Time
|
||||
commingSoonMessage string
|
||||
```
|
||||
golangci-lint v2.12 enforces `gofmt`; CI will fail. Run `gofmt -w internal/modules/trading/handlers.go`.
|
||||
|
||||
### C2. Sell-rollback can silently lose user shares
|
||||
`internal/modules/trading/handlers.go:188-189`:
|
||||
```go
|
||||
p.AddAsset(symbol, qty)
|
||||
_ = SavePortfolio(ctx, s.kv, userID, p)
|
||||
```
|
||||
The deduction-rollback Save's error is dropped with `_ =`. If KBS is down (already true at this codepath) AND the rollback write also fails (transient DynamoDB throttle, ctx deadline near expiry), the user's prior `DeductAsset` is in-memory only, never reverted, and on the next Load they will see the previous (post-deduct) state. Net result: shares deleted, no VND credited.
|
||||
|
||||
Fix: log + replace user-facing message when rollback fails so an op is alerted, e.g.:
|
||||
```go
|
||||
if err := SavePortfolio(ctx, s.kv, userID, p); err != nil {
|
||||
log.Error("trading_sell_rollback_failed", "user", userID, "symbol", symbol, "qty", qty, "err", err)
|
||||
return chathelper.Reply(ctx, b, chatID, "Sell failed and rollback errored — contact support before retrying.")
|
||||
}
|
||||
```
|
||||
Or — better — fetch the price BEFORE acquiring the lock (parallel to handleBuy's structure) so no rollback path is needed. handleSell currently fetches *under* the lock too, which also blocks the user's mutex on a 10-second HTTP call (see H1).
|
||||
|
||||
---
|
||||
|
||||
## High
|
||||
|
||||
### H1. handleSell holds the per-user lock across a 10s HTTP call
|
||||
`handlers.go:174-185`: lock acquired → Load → DeductAsset → **FetchPrice (10s timeout)** → AddCurrency → Save. Concurrent operations on the same user serialise behind a transient HTTP call to KBS. handleBuy correctly fetches the price *before* the lock. Rewrite handleSell to mirror handleBuy's order: validate, FetchPrice, then acquire lock, then Load → check holdings → Deduct + AddCurrency → Save. Eliminates the lock-held HTTP call AND removes the rollback hazard from C2.
|
||||
|
||||
### H2. lolschedule cron will exceed the 60s server timeout above ~1100 subscribers
|
||||
`internal/server/timeouts.go:9`: `defaultCronTimeout = 60 * time.Second`.
|
||||
`internal/modules/lolschedule/cron.go:31`: `telegramRateLimitDelay = 50 * time.Millisecond` when subs > 30.
|
||||
At N subscribers, throttled inter-send delay alone is `(N-1) * 50ms`. SendMessage HTTP latency adds ~50–200ms each. Effective ceiling ≈ 600–800 subscribers before the cron context cancels mid-batch. The handler does check `ctx.Done()` (good — returns ctx.Err) but the run is then logged as a failure with no resume state; the next day's run starts from chat[0] again so early subscribers are over-served and tail subscribers are starved (not fair).
|
||||
|
||||
Mitigations (pick one for v1; defer the rest):
|
||||
- a. Cap N: refuse new subscribers above e.g. 800 (warn user).
|
||||
- b. Shard schedule: emit one cron per 500-sub group with offset times. Requires Phase 05 EventBridge work.
|
||||
- c. Async fan-out: cron enqueues N SQS messages, each consumer SendMessages → done in parallel. Best long-term but needs new IaC.
|
||||
|
||||
For v1 with realistic JS-source subscriber counts (<100), this is probably fine; flag for monitoring after deploy.
|
||||
|
||||
### H3. handleSell: silent rollback save error swallows user data loss
|
||||
Already covered in C2 — also high-severity from the data-integrity angle.
|
||||
|
||||
### H4. No `From.ID == 0` defense
|
||||
The user's task description says "we explicitly refuse but verify" — code does NOT verify. `senderInfo` (handlers.go:50) only refuses `nil` From, not `From.ID == 0`. If Telegram (or a malicious local-dev fixture) ever produces a User with ID=0, all such users would key into `user:0` and share a portfolio. Telegram's spec says IDs are positive, so this is defense-in-depth, but trivial to add:
|
||||
```go
|
||||
if msg == nil || msg.From == nil || msg.From.ID == 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Medium
|
||||
|
||||
### M1. `Currency` is `map[string]float64` for VND — should be int64
|
||||
VND has no sub-unit; the smallest legal denomination is 1 VND. `float64` arithmetic on `cost := float64(qty) * price` and `Meta.Invested += amount` accumulates IEEE-754 drift. After a few hundred buys at non-round prices (24,500 × 137 = 3,356,500 — exact, ok; but 18,750 × 31 = 581,250 — also exact, but compounded sums of non-power-of-two integers eventually drift). At Vietnamese stock-trade volumes this is unlikely to materialise as user-visible cents, but flagging because:
|
||||
- (a) JSON decode `float64` of a saved 24,500,000 then `× 137` could round-trip-shift if KV ever stores e.g. "1.5e7";
|
||||
- (b) `FormatVND` uses `math.Round` which masks drift in the UI but not in the stored ledger.
|
||||
|
||||
Severity is medium (not high) because the upstream JS likely had the same issue and no incident has been reported. Recommend documenting the trade-off in `portfolio.go` or migrating to int64 in v2.
|
||||
|
||||
### M2. No ticker length / alphabet validation
|
||||
`symbols.go:30-35` accepts any non-empty `args[1]` after upper+trim. There's no length cap, no `[A-Z0-9]` enforcement, no defence against unicode lookalikes. While `url.PathEscape` makes the HTTP call safe and `ErrNoPrice` paths skip cache writes (so no KV pollution from invalid lookups), a user could still spam:
|
||||
```
|
||||
/trade_buy 1 АAA (Cyrillic А, looks like ASCII A)
|
||||
```
|
||||
which generates KBS HTTP calls + a partial DoS amplification through your Lambda. Add a regex check, e.g. `^[A-Z0-9]{1,16}$` after upper+trim, return `ErrUnknownTicker` for misses. Cheap, principled.
|
||||
|
||||
### M3. Field typo: `commingSoonMessage`
|
||||
`handlers.go:27, 118, 212` — should be `comingSoonMessage` (one m). User-invisible (it's a private field), but CI linters with spell-check rules flag this. Style grep'd consistently (3 occurrences); a single rename works.
|
||||
|
||||
### M4. `chatIDString` in cron_test.go is dead/wrong code
|
||||
`cron_test.go:35-37`:
|
||||
```go
|
||||
func chatIDString(id int64) string {
|
||||
return time.Unix(id, 0).Format("00") // arbitrary stringification
|
||||
}
|
||||
```
|
||||
`Format("00")` returns the literal string `"00"` because "00" contains no Go time-format directives. So every chat error message is identical: `"fakeSender: induced failure for chat 00"`. Replace with `strconv.FormatInt(id, 10)` or just inline `fmt.Errorf("fakeSender: induced failure for chat %d", id)`.
|
||||
|
||||
### M5. lolschedule daily-push has no retry / dead-chat unsubscribe
|
||||
A subscriber who blocks the bot returns 403 from SendMessage. The cron logs `failed++` and never removes them. Over weeks, the failure count grows. Not a correctness issue but an operational drag. Consider, in a future PR, trimming subscribers whose SendMessage returns specific 403/400 error codes.
|
||||
|
||||
### M6. `Phase 05 EventBridge schedule` deferred — daily-push is dead-on-deploy
|
||||
Per `plan.md:35`, Phase 05 is deferred. Without the `AWS::Scheduler::Schedule`, the registered `lolschedule_daily_push` cron will never fire in production. This is intentional per the plan, but I'm flagging because the README / changelog should not advertise the daily-push feature until Phase 05 ships. Verify the README copy doesn't promise active push.
|
||||
|
||||
---
|
||||
|
||||
## Low
|
||||
|
||||
### L1. `runDailyPush`: throttle decision is binary on `len(subs) > 30`
|
||||
At N=31, the cron suddenly serialises with 50ms delays. Telegram's 30/sec global limit is a target rate not a hard ceiling — 30 contiguous sends is fine. The threshold is conservative; not wrong, just unnecessarily slow at N=31..100.
|
||||
|
||||
### L2. `handleStats` allocates a per-call `heldList` slice — tiny GC churn at scale, fine for v1.
|
||||
|
||||
### L3. `prices.go:103` shadows builtin `close`
|
||||
```go
|
||||
close := body.DataDay[0].C
|
||||
if close <= 0 { ... }
|
||||
```
|
||||
`close` is a Go builtin (channel close). Shadowing is legal but lint-noisy. Rename to `c` or `lastClose`.
|
||||
|
||||
### L4. Test `TestRunDailyPush_SendsToAllSubscribers` asserts ordering of `sender.calls`
|
||||
Subscribers come back from `listSubscribers` in JSON-array order, which is the order they were added — *currently*. If the persistence layer ever switches to a set-like backend, the test breaks. Either lock the contract in `listSubscribers`'s godoc or sort before asserting in the test.
|
||||
|
||||
### L5. README / template.yaml — `trading` enabled by default
|
||||
`template.yaml:17` adds `trading` to ModulesCSV. Trading is a financial-looking command surface (paper or not). For a personal bot this is fine, but consider whether it should be opt-in via a `--with-trading` deploy flag in case future operators want to disable it without editing the template. v1: leave as-is.
|
||||
|
||||
---
|
||||
|
||||
## Edge cases / scout findings
|
||||
|
||||
- **handleStats** reads portfolio without keylock; safe because LoadPortfolio JSON-decodes a fresh struct each call (no shared map memory with concurrent buy/sell). **No race.**
|
||||
- **`defer s.locks.Acquire(key)()` semantics** — verified correct: outer call evaluates immediately (acquires), Unlock is deferred. Both buy and sell hold the lock over the right region.
|
||||
- **ResolveSymbol cache-write fallback** — `_ = kv.PutJSON(...)` on cache miss + successful KBS lookup is intentional and safe (next call will reresolve). Acceptable.
|
||||
- **`from.ID` collision** — see H4.
|
||||
- **KBS HTTP error semantics** — 4xx/5xx → ErrNoPrice (verified by test). Network errors → wrapped. JSON decode errors → wrapped. Negative close → ErrNoPrice. Empty data_day → ErrNoPrice. **All paths covered.**
|
||||
- **Cron auth** — `subtle.ConstantTimeCompare` used (router.go:68). No constant-time bypass via header probing. Good.
|
||||
- **Cron name regex** — `^[a-z0-9_]{1,32}$`, blocks log injection. `lolschedule_daily_push` matches. Good.
|
||||
- **No PII / secret leak** — error messages to users are generic ("Could not load portfolio. Try again later."); KBS upstream URLs not echoed; SendMessage params not logged with chat content; no stack traces propagated.
|
||||
- **Stats fan-out latency** — sequential per-ticker FetchPrice; for a portfolio of 50 tickers at 100ms KBS latency that's 5s of dead time before the user sees anything. Below the 60s ceiling but bad UX. Probably fine for v1 (typical user holds <10).
|
||||
- **Integer overflow** — `int64` for `qty` and `Assets` map values; max 9.2e18, never reachable for stock counts. `float64` for VND has 53-bit mantissa (~9e15 = 9 quadrillion VND ≈ $360 billion); not reachable.
|
||||
|
||||
---
|
||||
|
||||
## Positive observations
|
||||
|
||||
- Lock granularity (per-user) is correct, not over-broad. Distinct users never block each other.
|
||||
- handleBuy correctly fetches price *before* acquiring lock — minimises lock duration.
|
||||
- Tests use `httptest.NewServer` everywhere; **no real KBS calls in `go test`**. Hermetic.
|
||||
- Dependency injection via `messageSender` interface in cron.go is exemplary: enables real-bot test without mocking the full `*bot.Bot` API.
|
||||
- `BuildOptions` extension pattern: future deps (Bot, Embedder, Chatter) are added without breaking the `Build` signature. Good API stability hygiene.
|
||||
- Cache write failure on `ResolveSymbol` is correctly non-fatal (one-line comment explains why).
|
||||
- `senderInfo` correctly refuses channel posts / inline queries to avoid `user:0` collision.
|
||||
- Defensive nil-map repair in `LoadPortfolio` is correct defence-in-depth.
|
||||
- Throttle implementation in cron is select-based on `ctx.Done` — cooperative cancellation is wired.
|
||||
- 24/24 packages green with `-race`; CI integration looks healthy.
|
||||
|
||||
---
|
||||
|
||||
## Recommended action order
|
||||
|
||||
1. **Fix C1** (gofmt) — 10s, unblocks CI.
|
||||
2. **Fix C2 + H1 together** by reordering handleSell to fetch price before lock (mirrors handleBuy). One change, two issues resolved.
|
||||
3. **Add H4** (From.ID == 0 check) — 3 lines.
|
||||
4. **Add M2** (ticker regex) — 5 lines + 1 test.
|
||||
5. **Rename M3** (`commingSoonMessage` → `comingSoonMessage`) — global replace.
|
||||
6. **Fix M4** (chatIDString dead code) — 2-line fix.
|
||||
7. **Defer rest** (M1 float→int64, M5 dead-chat unsub, L-series) to a follow-up PR.
|
||||
|
||||
After (1)–(6), the change is mergeable. (1)–(3) are mandatory before deploy.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Is the upstream JS `trading` module also using float64 for VND? If yes, M1 is parity (acceptable v1) — if no, this is a regression worth fixing now.
|
||||
2. What's the realistic peak `lolschedule` subscriber count? If <300 ever, H2 is non-blocking; if growth is plausible, the decision in H2 (a/b/c) needs choosing before Phase 05 EventBridge ships.
|
||||
3. Should the handleSell rollback path also restore `Meta.Invested` symmetry? Currently Buy doesn't touch Invested and Sell doesn't either — Invested only moves on `trade_topup`. This makes "Invested" mean "total deposits", not "cost basis", which deviates from typical brokerage semantics. Confirm intent matches JS source.
|
||||
4. Is `Phase 05 EventBridge` going to land before public release? If yes, the daily-push code is exercised on first deploy. If no, it's dead-but-tested code. Either is fine — just confirm.
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
**Summary:** Code is well-structured, hermetic-tested, race-clean. Two real correctness issues (C1 gofmt blocker, C2 silent rollback save) and one architectural smell (H1 lock-held HTTP call) need fixing before deploy. Trading module is a credible peer of wordle/loldle in shape and discipline; lolschedule cron is testable and correctly authenticated.
|
||||
**Concerns:** C1 will fail CI. C2 + H1 are data-integrity (low probability, but not negligible at production scale). H4 is defense-in-depth. M-tier are quality-of-life. Phase 05 EventBridge schedule is deferred-by-design — verify README doesn't over-promise active push.
|
||||
-55
@@ -1,55 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Source inventory and migration policy"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: []
|
||||
completed: 2026-05-16
|
||||
---
|
||||
|
||||
# Phase 01: Source inventory and migration policy
|
||||
|
||||
## Overview
|
||||
Identify the exact Cloudflare KV namespaces and D1 tables still carrying production data, then lock a per-key policy: migrate, skip, or archive. The goal is to prevent a noisy "copy everything" migration that drags stale caches, retired modules, or incompatible schemas into DynamoDB.
|
||||
|
||||
## Requirements
|
||||
- Functional: produce a concrete inventory of live CF data sources, active key prefixes, D1 tables, and the AWS target shape for each kept dataset.
|
||||
- Non-functional: decisions are explicit, reversible, and tied to current code paths — not guesses from old plans.
|
||||
|
||||
## Architecture
|
||||
- Inspect current AWS consumers first: `wordle stats:*`, `loldle stats:*` / `config:*`, `twentyq stats:*`, `lolschedule subscribers`, `misc:last_ping`, `trading user:*` portfolios.
|
||||
- Inspect legacy CF sources second: KV namespace(s), D1 trading tables, and any retired-module prefixes still present.
|
||||
- Lock default policy:
|
||||
- **Migrate:** long-lived, user-visible state.
|
||||
- **Skip:** `game:*`, `matches:*`, `sym:*`, other caches.
|
||||
- **Archive-only:** retired modules and optional historical trade rows not consumed by current AWS runtime.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Modify: `plans/260510-0114-aws-port/phase-07-cutover.md`
|
||||
- Read only: `internal/modules/wordle/state.go`, `internal/modules/loldle/state.go`, `internal/modules/twentyq/state.go`, `internal/modules/lolschedule/subscribers.go`, `internal/modules/trading/portfolio.go`
|
||||
|
||||
## Implementation Steps
|
||||
1. Enumerate current AWS key shapes from live code.
|
||||
2. Pull a source inventory from Cloudflare KV and D1 using operator credentials.
|
||||
3. Build a migration matrix: source dataset → target DynamoDB key → action (`migrate|skip|archive`).
|
||||
4. Lock the exact D1 source tables/columns for `Portfolio.Meta.CreatedAt` and `Portfolio.Meta.Invested`.
|
||||
5. Mark retired namespaces explicitly so they are not silently reintroduced.
|
||||
6. Update the AWS cutover phase to say final webhook flip is gated on this migration matrix.
|
||||
|
||||
## Success Criteria
|
||||
- [x] Every live CF dataset is classified as migrate, skip, or archive. (matrix in `docs/cf-to-aws-migration-runbook.md`)
|
||||
- [x] Every migrated dataset has an explicit AWS target key shape.
|
||||
- [x] Trading `meta.createdAt` and `meta.invested` have authoritative source fields. (KV `trading:user:<id>` — flat JSON snapshot, no D1 derivation)
|
||||
- [x] Retired-module data is excluded by policy. (`doantu`, `loldle-ability`, `loldle-emoji`, `semantle` stats keys skipped; no archive)
|
||||
- [x] The cutover plan references this migration gate. (`plans/260510-0114-aws-port/phase-07-cutover.md` lines 13, 20, 42, 72)
|
||||
|
||||
## Outcome notes (2026-05-16)
|
||||
- Live inventory taken via wrangler against prod CF account `miti99` (D1 `miti99bot-db`, KV `f7f190fcb2fa42eb84a05542911334b0`).
|
||||
- D1: only `trading_trades` exists (11 rows, 1 user). No `users` / `holdings` tables.
|
||||
- KV: 21 keys total. 9 durable, 7 cache, 5 retired-module stats. `misc:last_ping` does not exist upstream.
|
||||
- Trading transform branch invalidated — JS Worker already stores final `Portfolio` JSON in KV. Phase 03 rewritten to flat KV copy (effort 4-6h → ~2h).
|
||||
|
||||
## Risk Assessment
|
||||
Main risk is misclassifying a dataset as disposable when users still care about it. Mitigation: classify by current runtime consumers first, then validate Cloudflare inventory against those exact consumers before any tooling is written.
|
||||
-61
@@ -1,61 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Backfill toolchain and safety rails"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "3-4h"
|
||||
dependencies: [1]
|
||||
completed: 2026-05-16
|
||||
---
|
||||
|
||||
# Phase 02: Backfill toolchain and safety rails
|
||||
|
||||
## Overview
|
||||
Build operator-run migration tooling in Go that can read legacy Cloudflare data, write DynamoDB records idempotently, and support dry-runs plus checkpoints. This phase is about controlled mechanics, not the actual production import yet.
|
||||
|
||||
## Requirements
|
||||
- Functional: provide commands for KV export/import, trading import, and parity verification inputs.
|
||||
- Non-functional: idempotent writes, dry-run mode, resumable progress, zero admin HTTP surface, and no dependency on the running AWS bot process.
|
||||
|
||||
## Architecture
|
||||
- Keep tooling inside this repo and language stack, but keep it small:
|
||||
- `cmd/migrate_cf_data/` for inventory + KV import + trading import modes
|
||||
- `cmd/verify_cf_aws_parity/` for verification only
|
||||
- Shared logic lives under `internal/migration/` for Cloudflare REST reads, DynamoDB writes, and report formatting.
|
||||
- D1 source extraction stays simple: operator uses `wrangler d1 execute ... --json --remote` to create local JSON exports; Go import code consumes those files instead of re-implementing remote SQL access.
|
||||
- Checkpoint/resume is conditional: add it only if Phase 01 proves the keyspace is large enough to justify it.
|
||||
- No writes happen during `--dry-run`; output is a machine-readable summary plus human-readable progress logs.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/migrate_cf_data/main.go`
|
||||
- Create: `cmd/verify_cf_aws_parity/main.go`
|
||||
- Create: `internal/migration/cloudflare_kv_client.go`
|
||||
- Create: `internal/migration/dynamodb_writer.go`
|
||||
- Create: `internal/migration/report.go`
|
||||
- Optional create: `internal/migration/checkpoint_store.go`
|
||||
- Modify: `go.mod`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. Define CLI flags and env contract for Cloudflare and AWS credentials.
|
||||
2. Implement KV list/get readers against Cloudflare REST with pagination support.
|
||||
3. Implement DynamoDB writers against the live runtime shape: `pk = moduleName`, `sk = caller key`.
|
||||
4. Add checkpoint files only if Phase 01 proves resume support is worth the extra surface area.
|
||||
5. Add dry-run and report output before any real import path is allowed.
|
||||
6. Document the exact operator workflow in the runbook.
|
||||
|
||||
## Success Criteria
|
||||
- [x] Tooling reads CF KV metadata and values without touching app code paths. (`internal/migration/cloudflare_kv_client.go`)
|
||||
- [x] ~~Trading import mode accepts local D1 JSON exports.~~ → invalidated by Phase 01. Trading is a flat KV copy; D1 is audit-only via `trading-audit-dump --out=<jsonl>`.
|
||||
- [x] Every command supports `--dry-run`. (kv-import has --dry-run; inventory and trading-audit-dump are read-only by construction so a dry-run flag is redundant)
|
||||
- [x] Import path is idempotent or safely merge-based. (`attribute_not_exists(pk)` guard; `--overwrite` is explicit opt-in)
|
||||
- [x] Checkpoint/resume behavior is intentionally omitted: Phase 01 inventory shows 21 keys total (well below the threshold where resume earns its complexity cost).
|
||||
|
||||
## Outcome notes (2026-05-16)
|
||||
- Files created: `cmd/migrate_cf_data/main.go`, `internal/migration/policy.go`, `cloudflare_kv_client.go`, `cloudflare_d1_client.go`, `dynamodb_writer.go`, `report.go` + four `*_test.go` files.
|
||||
- Verify command (`cmd/verify_cf_aws_parity/`) intentionally moved to Phase 04 to remove the cross-phase ownership collision.
|
||||
- Toolchain smoke-tested against prod CF: `inventory` → 22 keys observed (one cache key drift since Phase 01 inventory); `kv-import --dry-run` → 9 durable keys map to runtime `(pk, sk)` exactly.
|
||||
- All `go test ./...` pass; `go vet ./...` clean.
|
||||
|
||||
## Risk Assessment
|
||||
The main risk is embedding too much migration logic into one giant binary. Mitigation: split command entrypoints and keep shared logic in small `internal/migration/` helpers so each command stays reviewable and under the repo's file-size guidance.
|
||||
@@ -1,59 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Durable KV import (trading included)"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "2h"
|
||||
dependencies: [1, 2]
|
||||
---
|
||||
|
||||
# Phase 03: Durable KV import (trading included)
|
||||
|
||||
## Overview
|
||||
Copy the 9 durable Cloudflare KV records into DynamoDB under the live runtime key shape. Trading is included as a flat KV copy — the JS Worker already snapshots `Portfolio` JSON into `trading:user:*`, so no D1 transform is required (locked in Phase 01).
|
||||
|
||||
## Requirements
|
||||
- Functional: each migrated KV key lands in DynamoDB at `(pk=moduleName, sk=callerKey)` with the original CF KV value placed in the `value` attribute, byte-for-byte where possible.
|
||||
- Non-functional: idempotent — rerun must not duplicate or corrupt; skipped/failed records must be reported, not silently dropped.
|
||||
|
||||
## Architecture
|
||||
- Single import mode: KV → DynamoDB. No D1 transform path.
|
||||
- Durable key set (locked in Phase 01):
|
||||
- `wordle:stats:*`
|
||||
- `loldle:stats:*`
|
||||
- `loldle:config:*`
|
||||
- `twentyq:stats:*`
|
||||
- `lolschedule:subscribers`
|
||||
- `trading:user:*`
|
||||
- Skip set: `trading:sym:*` (cache), retired modules (`doantu`, `loldle-ability`, `loldle-emoji`, `semantle` stats keys), `misc:last_ping` (never written upstream).
|
||||
- Optional sub-task: dump `trading_trades` (D1) to JSONL for cold audit. Not an import input. Operator-elective.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `cmd/migrate_cf_data/main.go` — wire up the KV-copy run mode
|
||||
- Create: `internal/migration/kv_filter.go` — durable/skip allowlist driven by the Phase 01 matrix
|
||||
- Create: `internal/migration/import_report.go` — counts for imported / skipped / failed
|
||||
- Create: `internal/migration/trading_audit_dump.go` — optional D1 → JSONL exporter (operator-elective)
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md` — append import command + report layout
|
||||
- Read only: `internal/storage/dynamodb_kv.go`, `internal/modules/trading/portfolio.go`
|
||||
|
||||
## Implementation Steps
|
||||
1. Wire the KV allowlist filter into the import binary so only Phase 01 durable keys are read.
|
||||
2. For each durable KV key, read the raw value and write to DynamoDB at the matching `(pk, sk)`. Preserve original bytes.
|
||||
3. Implement idempotency via conditional `PutItem` or `attribute_not_exists` guard, fall back to overwrite with operator flag.
|
||||
4. Emit an import report (stdout + JSON file) with per-prefix counts: imported, skipped, failed.
|
||||
5. Add the optional `--trading-audit-dump <path>` flag that streams `trading_trades` rows to a local JSONL file. Default off.
|
||||
6. Update the runbook with the exact command operators run, and the expected report layout.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] All 9 durable keys land in DynamoDB at the runtime-expected `(pk, sk)`.
|
||||
- [ ] `trading:user:<id>` round-trips through the Go runtime's `Portfolio` JSON unmarshal without modification (parity check).
|
||||
- [ ] Re-running the import without flags is a no-op (no duplicates, no corruption).
|
||||
- [ ] Skipped-by-policy keys (cache, retired modules) are listed in the report, not silently dropped.
|
||||
- [ ] Optional `trading_trades` audit dump produces JSONL with one row per trade when flag is passed.
|
||||
|
||||
## Risk Assessment
|
||||
- KV value encoding drift: CF KV may return values as strings while DynamoDB attribute typing prefers `B`/`S`. Mitigation: round-trip a known portfolio record and assert byte parity before bulk import.
|
||||
- Idempotency: an overwrite-by-default rerun could silently revert post-cutover writes. Mitigation: default to `attribute_not_exists` guard and require explicit `--overwrite` flag.
|
||||
|
||||
## Notes
|
||||
- Phase 03 used to assume a D1 → Portfolio transform. That was wrong: KV already holds the final shape. Earlier `trading_transform.go` work is dropped.
|
||||
-50
@@ -1,50 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Parity verification and rehearsal"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: [2, 3]
|
||||
---
|
||||
|
||||
# Phase 04: Parity verification and rehearsal
|
||||
|
||||
## Overview
|
||||
Prove the imported AWS data matches the Cloudflare source closely enough to trust a real cutover. This phase turns migration from a one-off script run into a repeatable, auditable procedure with a staging-table rehearsal.
|
||||
|
||||
## Requirements
|
||||
- Functional: verify counts, sample payload parity, and trading portfolio correctness between CF exports and DynamoDB.
|
||||
- Non-functional: produce a saved report, support reruns, and define rollback steps before the production webhook is moved.
|
||||
|
||||
## Architecture
|
||||
- Verifier compares source exports against the AWS target table using the same module/key selectors from Phase 01.
|
||||
- Checks by dataset type:
|
||||
- KV durable records: count parity + payload/hash comparisons
|
||||
- trading portfolios: count parity + deep field comparison on currency, assets, and invested metadata
|
||||
- Rehearsal happens against a staging DynamoDB table only. No destructive rerun path is allowed against the live table.
|
||||
- Final output is a migration report under `plans/reports/` plus runbook updates.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `cmd/verify_cf_aws_parity/main.go`
|
||||
- Create: `internal/migration/parity_checks.go`
|
||||
- Create: `internal/migration/rollback_scope.go`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Create during execution: `plans/reports/migration-260515-2250-cf-data-to-aws-parity.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. Implement count and payload verification per migrated dataset.
|
||||
2. Add trading-specific deep checks against the current `Portfolio` shape.
|
||||
3. Save the verifier result as a markdown report under `plans/reports/`.
|
||||
4. Rehearse import + verify against a staging DynamoDB table.
|
||||
5. Promote the exact same procedure to the live table only after staging is green.
|
||||
6. Mark the migration runbook ready only after a green verifier report.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Verifier reports pass for all migrated datasets.
|
||||
- [ ] Trading portfolios match expected balances and holdings on spot checks.
|
||||
- [ ] A staging-table rehearsal completes successfully without touching the live table.
|
||||
- [ ] The migration report is saved and linked from the runbook.
|
||||
- [ ] The cutover checklist now depends on a green parity report.
|
||||
|
||||
## Risk Assessment
|
||||
The main risks are false confidence from count-only checks and accidental destructive rehearsal against production storage. Mitigation: include dataset-specific deep comparisons, especially for trading portfolios, and require a saved report plus staging-table-only rehearsal before the cutover phase can begin.
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Cutover integration and Cloudflare decommission"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: [1, 2, 3, 4]
|
||||
---
|
||||
|
||||
# Phase 05: Cutover integration and Cloudflare decommission
|
||||
|
||||
## Overview
|
||||
Fold the verified migration into the AWS cutover path, then decommission Cloudflare resources only after a successful freeze-window migration and AWS soak. This phase closes the data-consistency gap left by the original AWS port plan.
|
||||
|
||||
## Requirements
|
||||
- Functional: production cutover moves webhook ownership to AWS without losing durable writes from the old Cloudflare stack.
|
||||
- Non-functional: rollback is fast only before the first AWS-served write; after that, the plan is forward-fix only unless reverse sync is built later. Cloudflare resources are deleted only after verification, and operator steps are explicit.
|
||||
|
||||
## Architecture
|
||||
- There is no legacy dual-write path, so final cutover uses a short **write-freeze window**:
|
||||
1. disable/pause Cloudflare cron triggers
|
||||
2. stop Cloudflare webhook intake so no new writes land there
|
||||
3. run final delta export/import + parity verify
|
||||
4. point Telegram webhook to AWS
|
||||
5. begin AWS soak
|
||||
- Before the first AWS-served write, rollback is still a webhook restore.
|
||||
- After the first AWS-served write, rollback is not symmetry; it becomes forward-fix only unless a reverse-sync mechanism exists.
|
||||
- This keeps migration correctness simple and avoids inventing temporary cross-runtime replication.
|
||||
- Cloudflare teardown is a separate final step after the AWS soak, not part of the initial webhook flip.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `plans/260510-0114-aws-port/plan.md`
|
||||
- Modify: `plans/260510-0114-aws-port/phase-07-cutover.md`
|
||||
- Modify: `docs/deploy-aws.md`
|
||||
- Modify: `docs/cf-to-aws-migration-runbook.md`
|
||||
- Optional create: `docs/cf-decommission-checklist.md`
|
||||
|
||||
## Implementation Steps
|
||||
1. Update the AWS cutover phase to depend on a green migration report.
|
||||
2. Add the freeze-window sequence and pre-flip vs post-flip rollback semantics to the runbook.
|
||||
3. Define the final delta import and verification command set.
|
||||
4. Flip the Telegram webhook only after the final delta verify succeeds.
|
||||
5. Add post-flip smoke checks for a migrated trading account, an existing lolschedule subscriber, and `/mstats` if `last_ping` is kept.
|
||||
6. Soak on AWS, then remove CF Worker/KV/D1 only when rollback is no longer needed.
|
||||
7. Archive or document any intentionally skipped legacy datasets before teardown.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] AWS cutover docs explicitly require a green migration report.
|
||||
- [ ] Freeze-window steps are documented end-to-end.
|
||||
- [ ] Pre-flip rollback and post-flip forward-fix semantics are documented explicitly.
|
||||
- [ ] Final delta import and rollback commands are ready before webhook flip.
|
||||
- [ ] Cloudflare resources are not deleted during the initial cutover window.
|
||||
- [ ] After soak, CF teardown is documented and low-risk.
|
||||
|
||||
## Risk Assessment
|
||||
The biggest risk is write drift between an early backfill and the final webhook flip. Mitigation: use a short freeze window for the last delta import instead of trying to add temporary dual-write behavior to a legacy system that lives outside this repo.
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
title: "Migrate Cloudflare data to AWS DynamoDB"
|
||||
description: "Export durable data from the legacy Cloudflare Worker stack, import it into the live AWS DynamoDB store, verify parity, and gate final cutover on a proven migration runbook."
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 1-2d
|
||||
branch: main
|
||||
tags: [migration, cloudflare, aws, dynamodb, cutover, data]
|
||||
created: 2026-05-15
|
||||
blockedBy: []
|
||||
blocks: [260510-0114-aws-port]
|
||||
---
|
||||
|
||||
# Plan: Cloudflare data → AWS DynamoDB
|
||||
|
||||
This plan adds the missing data-migration leg to the in-progress AWS port. AWS runtime + DynamoDB already exist; the gap is getting durable user data out of the legacy Cloudflare KV/D1 stack before final decommission.
|
||||
|
||||
## Why separate plan
|
||||
- `plans/260510-0114-aws-port/` covers runtime + deploy cutover.
|
||||
- This plan covers source-data inventory, export/import tooling, parity verification, and rollback.
|
||||
- `aws-port` should not be considered done until this plan passes.
|
||||
|
||||
## Locked decisions
|
||||
- Migrate **durable user-visible data only**.
|
||||
- Skip ephemeral or disposable data: in-flight game state, schedule caches, stale price caches.
|
||||
- ~~Keep `misc:last_ping`~~ → skip; KV inventory shows it was never written by the JS Worker (`/mstats` resets on AWS).
|
||||
- No admin HTTP routes. Migration runs as operator-invoked one-shot tooling.
|
||||
- **Pre-cutover bulk import may target the live table directly** while the Telegram webhook still points to Cloudflare and the AWS bot has served zero writes (amended 2026-05-16). Rationale: until webhook flip, the AWS DynamoDB table is empty and no real user traffic depends on it, so a separate staging table adds setup cost without de-risking anything. **After webhook cutover, any re-import must use a staging table first.** The `attribute_not_exists` idempotency guard is the only safe write path at any time — no wipe-and-rerun flow is allowed against the live table.
|
||||
- ~~Trading import is a transform~~ → **flat KV copy**. CF KV `trading:user:<id>` already holds the final `Portfolio` JSON shape; no D1 derivation needed. (Phase 01 inventory, 2026-05-16.)
|
||||
- After the first AWS-served write, rollback is forward-fix only unless a reverse-sync path is built later.
|
||||
- Retired module namespaces from the old CF stack are skipped (no archive — operator decision 2026-05-16).
|
||||
|
||||
## Source data classified in Phase 01 (closed 2026-05-16)
|
||||
- **Migrate (9 keys):** `wordle:stats:*` (1), `loldle:stats:*` (4), `loldle:config:*` (1), `twentyq:stats:*` (1), `lolschedule:subscribers` (1), `trading:user:*` (1).
|
||||
- **Skip (cache + missing + retired):** `trading:sym:*` (7), `misc:last_ping` (0), `doantu:stats:*` (2), `loldle-ability:stats:*` (1), `loldle-emoji:stats:*` (1), `semantle:stats:*` (1).
|
||||
- **Archive-only (optional, operator-elective):** D1 `trading_trades` (11 rows, 1 user) — audit dump, not import input.
|
||||
|
||||
Full matrix lives in `docs/cf-to-aws-migration-runbook.md`.
|
||||
|
||||
## Related current code
|
||||
- `cmd/server/main.go:167` — runtime storage backend selection (`dynamodb|firestore|memory`)
|
||||
- `internal/storage/dynamodb_provider.go:7` — live DynamoDB partitioning (`pk = moduleName`)
|
||||
- `internal/storage/dynamodb_kv.go:24` — live DynamoDB sort-key contract (`sk = caller key`)
|
||||
- `internal/modules/wordle/state.go:50` — `game:*` + `stats:*`
|
||||
- `internal/modules/loldle/state.go:48` — `game:*`, `stats:*`, `config:*`
|
||||
- `internal/modules/twentyq/state.go:37` — `game:*` + `stats:*`
|
||||
- `internal/modules/lolschedule/subscribers.go:14` — `subscribers`
|
||||
- `internal/modules/trading/portfolio.go:39` — current AWS target shape: per-user KV portfolio JSON
|
||||
- `plans/260508-2222-go-port-cloud-run/phase-12-cutover.md:37` — prior CF→Go cutover notes (trading-only import assumption)
|
||||
- `plans/260510-0114-aws-port/phase-07-cutover.md:13` — current AWS final cutover phase
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Key deliverable |
|
||||
|---|-------|--------|--------|-----------------|
|
||||
| 01 | [Source inventory and migration policy](phase-01-source-inventory-and-migration-policy.md) | completed | 2-3h | exact CF namespaces/tables mapped to migrate vs skip vs archive |
|
||||
| 02 | [Backfill toolchain and safety rails](phase-02-backfill-toolchain-and-safety-rails.md) | completed | 3-4h | operator-run `cmd/migrate_cf_data` binary (inventory, kv-import, trading-audit-dump) + dry-run + idempotency |
|
||||
| 03 | [Durable KV import (trading included)](phase-03-trading-and-durable-kv-import.md) | pending | 2h | flat KV→DynamoDB copy for 9 durable keys + optional D1 audit dump |
|
||||
| 04 | [Parity verification and rehearsal](phase-04-parity-verification-and-rehearsal.md) | pending | 2-3h | repeatable verifier, mismatch report, rollback drill |
|
||||
| 05 | [Cutover integration and Cloudflare decommission](phase-05-cutover-integration-and-cloudflare-decommission.md) | pending | 2-3h | AWS cutover checklist updated; CF teardown gated on verified migration |
|
||||
|
||||
## Key dependencies
|
||||
- Blocks: `plans/260510-0114-aws-port/phase-07-cutover.md`
|
||||
- Uses the already-live AWS target from `plans/260510-0114-aws-port/`
|
||||
- Should finish before deleting CF Worker/KV/D1 resources referenced in `plans/260508-2222-go-port-cloud-run/phase-12-cutover.md`
|
||||
|
||||
## Success bar
|
||||
- Durable CF data imported into DynamoDB with counts + sampled payload parity.
|
||||
- Trading balances/holdings and required portfolio metadata match the old system byte-for-byte (flat KV copy).
|
||||
- Cutover runbook explicitly distinguishes pre-flip rollback from post-flip forward-fix semantics.
|
||||
- CF resources are not deleted until parity report is green.
|
||||
@@ -1,138 +0,0 @@
|
||||
# Phase 01 — Add `/trongtruonghop` to misc module
|
||||
|
||||
**Status:** Planned
|
||||
**Priority:** Low (additive, no migration, no infra change)
|
||||
**Mode:** fast
|
||||
|
||||
## Context links
|
||||
|
||||
- Module under change: `internal/modules/misc/misc.go`
|
||||
- Helper API used: `internal/modules/util/chathelper/chathelper.go` (`ArgAfterCommand`, `ReplyHTML`)
|
||||
- Visibility / validation rules: `internal/modules/module.go`, `internal/modules/validate.go`
|
||||
- Forum-topic reply-routing fix that mandates `chathelper.Reply*` over raw `SendMessage`: commit 3a12615
|
||||
|
||||
## Overview
|
||||
|
||||
Stateless command. Two interpolation points (`<text>`, `@<sender>` × 2) into a fixed Vietnamese template. No KV. No new dependency. No new helper.
|
||||
|
||||
## Key insights
|
||||
|
||||
- `chathelper.ArgAfterCommand` already strips command + `@botname` correctly — covers `/trongtruonghop arg`, `/trongtruonghop@miti99bot arg`, etc.
|
||||
- `chathelper.ReplyHTML` already forwards `MessageThreadID` (forum-topic safe). Do NOT bypass.
|
||||
- Telegram's HTML parser accepts `@username` literally (it's not a tag) and resolves the mention server-side. Mixing `@username` with `<a href="tg://user?id=…">Name</a>` in the same message is allowed and standard.
|
||||
- `From.Username` can be empty (account never set one). `From.FirstName` is also optional (deleted accounts). Both can be empty simultaneously — handle.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
|
||||
- Command name: `trongtruonghop`, Visibility: `VisibilityPublic`, Description: `"Phát biểu disclaimer cho thành viên hiện tại"` (Vietnamese — keep short, fits `/help`).
|
||||
- On `/trongtruonghop [text]`:
|
||||
1. `arg := strings.TrimSpace(chathelper.ArgAfterCommand(msg.Text))`
|
||||
2. If `arg == ""` → `arg = defaultTarget` (= `"VNG"`).
|
||||
3. Resolve sender mention from `msg.From` (see algorithm below).
|
||||
4. Send single HTML message via `chathelper.ReplyHTML`.
|
||||
- If `msg == nil` or `msg.From == nil` → return `nil` (silent skip).
|
||||
|
||||
### Sender-mention algorithm
|
||||
|
||||
```go
|
||||
func senderMention(u *models.User) string {
|
||||
if u.Username != "" {
|
||||
return "@" + u.Username // safe verbatim; charset is [A-Za-z0-9_]
|
||||
}
|
||||
name := strings.TrimSpace(u.FirstName + " " + u.LastName)
|
||||
if name == "" {
|
||||
name = "thành viên"
|
||||
}
|
||||
return fmt.Sprintf(`<a href="tg://user?id=%d">%s</a>`, u.ID, html.EscapeString(name))
|
||||
}
|
||||
```
|
||||
|
||||
### Template
|
||||
|
||||
Package-level `const`:
|
||||
|
||||
```go
|
||||
const trongTruongHopTemplate = "Trong trường hợp nhóm này bị điều tra bởi %s, %s khẳng định không liên quan tới nhóm hoặc những cá nhân khác trong nhóm này. %s không rõ tại sao lại có mặt ở đây vào thời điểm này, có lẽ tài khoản đã được thêm bởi một bên thứ ba."
|
||||
const defaultTarget = "VNG"
|
||||
```
|
||||
|
||||
Render: `fmt.Sprintf(trongTruongHopTemplate, html.EscapeString(arg), mention, mention)`.
|
||||
|
||||
## Architecture
|
||||
|
||||
No new types, no state, no new files. Single command added to `New(deps modules.Deps)` in `misc.go`. `deps.KV` not used by this command but `New` already receives it for the other two — no signature change.
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify:**
|
||||
- `internal/modules/misc/misc.go`
|
||||
- `internal/modules/misc/misc_test.go`
|
||||
- `internal/modules/misc/handlers_test.go`
|
||||
- `README.md` (misc-row description only)
|
||||
|
||||
**Create:** none.
|
||||
**Delete:** none.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **misc.go**
|
||||
- Add imports: `fmt`, `html`, `strings` (only those not already imported).
|
||||
- Add `const trongTruongHopTemplate` and `const defaultTarget` near the existing `const lastPingKey`.
|
||||
- Add private function `senderMention(*models.User) string` (algorithm above).
|
||||
- Add `trongTruongHopCommand() modules.Command` (no `deps` needed — stateless).
|
||||
- Append it to the `Commands` slice in `New`.
|
||||
|
||||
2. **misc_test.go**
|
||||
- Extend the `want` map in `TestNew_RegistersExpectedCommands` with `"trongtruonghop": modules.VisibilityPublic`. The existing length check then implicitly verifies registration.
|
||||
|
||||
3. **handlers_test.go** — new test cases (reuse existing `installMisc`):
|
||||
- `TestTrongTruongHop_DefaultArgUsesVNG`: send `/trongtruonghop` from user 999 with username `boss`. Assert reply contains `"VNG"` and `"@boss"` (occurring twice).
|
||||
- `TestTrongTruongHop_CustomArg`: send `/trongtruonghop Acme Corp`. Assert reply contains `"Acme Corp"` and not `"VNG"`.
|
||||
- `TestTrongTruongHop_HTMLEscapesArg`: send `/trongtruonghop <script>`. Assert reply contains `<script>` and not the literal `<script>`.
|
||||
- `TestTrongTruongHop_NoUsernameFallsBackToLink`: build a custom update where `From.Username == ""`, `FirstName == "Anh"`. Assert reply contains `<a href="tg://user?id=42">Anh</a>` (twice).
|
||||
- `TestTrongTruongHop_EmptyDisplayNameFallsBackToThanhVien`: `Username == ""`, `FirstName == ""`, `LastName == ""`. Assert reply contains `>thành viên</a>`.
|
||||
|
||||
Note: existing `testutil.NewPrivateMessage` sets `FirstName: "Test"` and no `Username` — for username-bearing cases, build the update inline (see `NewPrivateMessage` source for the shape, ~10 lines).
|
||||
|
||||
4. **README.md** — change misc table row from `Coin flip, dice, RNG utilities` to `Coin flip, dice, RNG utilities, /trongtruonghop disclaimer`.
|
||||
|
||||
5. Compile + lint + test:
|
||||
- `go vet ./...`
|
||||
- `make test`
|
||||
- `golangci-lint run ./...`
|
||||
|
||||
## Todo list
|
||||
|
||||
- [ ] Add constants + helper + command in `misc.go`
|
||||
- [ ] Register command in `New`
|
||||
- [ ] Update `misc_test.go` `want` map
|
||||
- [ ] Add 5 handler-level tests in `handlers_test.go`
|
||||
- [ ] Update README misc row
|
||||
- [ ] `go vet`, `make test`, lint clean
|
||||
- [ ] Smoke-test in a real Telegram group post-deploy (manual)
|
||||
|
||||
## Success criteria
|
||||
|
||||
- All new tests pass; existing tests still pass.
|
||||
- `/help` automatically lists the command (registry-driven, no extra work).
|
||||
- Sending `/trongtruonghop` in any chat (private / group / supergroup / forum-topic) produces exactly one reply that mentions the sender twice and the target once, all routed to the originating topic.
|
||||
|
||||
## Risk assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| HTML parse error if escape is forgotten | Centralise escape in the `Sprintf` call; covered by `TestTrongTruongHop_HTMLEscapesArg`. |
|
||||
| `@<sender>` for username-bearing user breaks if username contains unexpected chars | Telegram enforces `[A-Za-z0-9_]{5,32}` server-side — no escaping required. Documented in handler comment. |
|
||||
| Test that constructs a custom `Update` drifts from `testutil.NewPrivateMessage` shape | Keep the custom builder local to the test file; only override `From` fields. Don't add a public helper for one caller. |
|
||||
|
||||
## Security considerations
|
||||
|
||||
- Auth: none — public command.
|
||||
- Input handling: user-supplied `<text>` is HTML-escaped before interpolation. No SQL / KV / shell surface.
|
||||
- Output: HTML mode. Mention link uses Telegram-internal `tg://user?id=` scheme, which the client resolves locally — no external network call.
|
||||
|
||||
## Next steps
|
||||
|
||||
After this phase merges + deploys, no follow-up. Command is self-contained.
|
||||
@@ -1,79 +0,0 @@
|
||||
# /trongtruonghop — disclaimer one-liner in misc module
|
||||
|
||||
**Date:** 2026-05-16
|
||||
**Slug:** `260516-1409-trongtruonghop-command`
|
||||
**Status:** Planned
|
||||
**Mode:** fast (single phase, well-scoped add-on to existing module)
|
||||
|
||||
## Goal
|
||||
|
||||
Add a public `/trongtruonghop` command to the `misc` module. When invoked it
|
||||
replies with a fixed Vietnamese disclaimer template, interpolating:
|
||||
|
||||
- `<text>` — argument after the command. Empty → `VNG`.
|
||||
- `@<sender>` — mention of the user who sent the command. Resolved from
|
||||
`update.Message.From` (Telegram guarantees this on group/private text
|
||||
messages).
|
||||
|
||||
Output (single message):
|
||||
|
||||
```
|
||||
Trong trường hợp nhóm này bị điều tra bởi <text>, @<sender> khẳng định không liên quan tới nhóm hoặc những cá nhân khác trong nhóm này. @<sender> không rõ tại sao lại có mặt ở đây vào thời điểm này, có lẽ tài khoản đã được thêm bởi một bên thứ ba.
|
||||
```
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | File |
|
||||
|---|-------|------|
|
||||
| 01 | Add `trongtruonghop` command to misc module | `phase-01-add-trongtruonghop-command.md` |
|
||||
|
||||
## Files
|
||||
|
||||
- `internal/modules/misc/misc.go` — register new `trongtruonghopCommand()` in `New`; implement handler.
|
||||
- `internal/modules/misc/handlers_test.go` — add coverage for default arg, custom arg, username vs no-username sender, HTML-escape of arg.
|
||||
- `internal/modules/misc/misc_test.go` — extend `TestNew_RegistersExpectedCommands` map with the new command (VisibilityPublic).
|
||||
- `README.md` — bump `misc` row description (single line) to mention the new command.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No KV interaction (this command is stateless).
|
||||
- No new helpers in `chathelper` — `ArgAfterCommand` + `ReplyHTML` already cover everything we need.
|
||||
- No localization framework — template is Vietnamese-only and inlined as a constant.
|
||||
- No rate limiting beyond what the dispatcher already provides (it doesn't, and this is fine; output is a single short message).
|
||||
|
||||
## Decisions (locked)
|
||||
|
||||
| Question | Decision | Reason |
|
||||
|---|---|---|
|
||||
| Reply mode (plain vs HTML)? | **HTML** via `chathelper.ReplyHTML` | We need to mention users who lack a `@username` — only Telegram HTML `<a href="tg://user?id=…">` works for that. Plain `@username` for users with one is rendered verbatim by HTML mode and Telegram still resolves it. Single code path for both cases. |
|
||||
| `@<sender>` formatting | If `From.Username != ""` → `@<username>` (literal, no HTML wrap). Else → `<a href="tg://user?id=<ID>">First Last</a>`. | Mirrors how Telegram itself renders mentions; `@username` is a native entity. `tg://user?id=` link is the documented fallback for username-less accounts. |
|
||||
| Display name when no username | `strings.TrimSpace(FirstName + " " + LastName)`; if still empty → `"thành viên"` | Defensive — Telegram allows accounts with no first name (rare; deleted accounts). |
|
||||
| Default `<text>` when arg empty | `"VNG"` (user spec) | Stored as a package-level const `defaultTarget` for visibility. |
|
||||
| Visibility | `VisibilityPublic` | Joke/disclaimer command — usable by anyone in any chat the bot is in. |
|
||||
| `<text>` sanitisation | `html.EscapeString` on the arg before interpolating | Arg is user-controlled. Without escaping, `<text>` containing `<` breaks the HTML parser and Telegram rejects the send (400). |
|
||||
| Mention sanitisation | `@<username>` is `[A-Za-z0-9_]{5,32}` — safe verbatim. Display name path → `html.EscapeString` on the trimmed name. | First/last names can legitimately contain `<` / `&`. |
|
||||
| `update.Message.From == nil` guard | Return nil (no reply) | Matches existing misc handlers' defensive shape. Channel posts (no `From`) are the only realistic path; we don't want to spam them. |
|
||||
| Forum-topic routing | Use `chathelper.ReplyHTML(ctx, b, msg, …)` — it already forwards `MessageThreadID` | Locked by 3a12615 — every new reply MUST go through these helpers. |
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. `/trongtruonghop` (no arg) in a private chat → reply contains `VNG` literally and `@<sender>` twice.
|
||||
2. `/trongtruonghop SomeCompany` → reply contains `SomeCompany` and `@<sender>` twice.
|
||||
3. `/trongtruonghop <script>` → reply renders `<script>` (verify via the recording bot's captured `Text`).
|
||||
4. Sender without `Username` → reply contains `<a href="tg://user?id=…">FirstName</a>` instead of `@username`.
|
||||
5. Sender with `Username` → reply contains `@username` (no `<a>` tag).
|
||||
6. `make vet` + `make test` clean; `golangci-lint run ./...` clean.
|
||||
7. `/help` lists `trongtruonghop` under `misc` (auto — registry-driven, no help-template change needed).
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Telegram rejects HTML on malformed entity (e.g. unclosed `<a>`) | Build the mention via a single small helper that returns a closed tag; unit-test the helper directly. |
|
||||
| User pastes very long text → message > 4096-char Telegram limit | The template is ~250 chars; arg would need to be ~3.6k to overflow. Acceptable risk for a joke command. Telegram returns 400 which `bot.SendMessage` propagates as an error; the dispatcher logs it. No silent failure. |
|
||||
| Vietnamese diacritics in the template encoding | Source files are UTF-8 (verified in existing `wordle`/`loldle` strings). Inline the string verbatim — no escape sequences. |
|
||||
| Command name `trongtruonghop` is unfamiliar | 14 chars, lowercase, alphanumeric — passes `validateCommand` regex per `internal/modules/validate_test.go:32`. |
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
None.
|
||||
@@ -1,130 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Narrow OIDC trust (F2)"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "30m"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Narrow OIDC trust (F2)
|
||||
|
||||
## Overview
|
||||
|
||||
Remove the `pull_request` claim (and `refs/heads/dev` if unused) from the OIDC trust policy on `github-deploy-miti99bot`. After this lands, only pushes to `main` can assume the deploy role.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- The `Condition.StringLike.token.actions.githubusercontent.com:sub` array on the role's trust policy must contain only branch refs that are actually used to deploy.
|
||||
- Current set: `refs/heads/main`, `refs/heads/dev`, `pull_request`.
|
||||
- Target set: `refs/heads/main` (plus `refs/heads/dev` ONLY if it's used; default-drop otherwise).
|
||||
|
||||
**Non-functional**
|
||||
- Apply out-of-band via maintainer's local `admin` AWS profile — not through the workflow being modified.
|
||||
- Rollback: re-apply the previous `iam-github-oidc-trust.json` snapshot via the same `aws iam update-assume-role-policy` call.
|
||||
|
||||
## Architecture
|
||||
|
||||
Single JSON file (`aws/iam-github-oidc-trust.json`) is the source of truth committed to the repo; AWS-side trust policy is updated via `aws iam update-assume-role-policy`. No CloudFormation involvement.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `aws/iam-github-oidc-trust.json` (drop 1-2 lines from `sub` allowlist)
|
||||
- Read-only: `.github/workflows/deploy.yml` (confirms only `main` is on the `push` trigger)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
0. **Verify admin profile is reachable** (RT-4). `aws/README.md:119` recommends deleting admin keys as hardening posture, so before proceeding confirm the operator can authenticate:
|
||||
```sh
|
||||
aws sts get-caller-identity --profile admin
|
||||
```
|
||||
If this fails with `InvalidClientTokenId` / `Unable to locate credentials`: recreate admin access keys via console (root → IAM → Users → admin → Security credentials → Create access key), or perform every step in this phase via the AWS Console fallback below.
|
||||
|
||||
**Console fallback for Phase 1:** IAM → Roles → `github-deploy-miti99bot` → Trust relationships → Edit trust policy → paste the JSON from step 3 → Update policy.
|
||||
|
||||
1. **Confirm `dev` is not used.** Inspect `.github/workflows/deploy.yml:5` — `on.push.branches` is `[main]`. Search every workflow file:
|
||||
```sh
|
||||
rg -l 'dev|id-token' .github/workflows/
|
||||
```
|
||||
Also check for any workflow with `permissions: id-token: write` (only OIDC-capable workflows matter). Current state (verified 2026-05-18): only `deploy.yml` has `id-token: write`. `ci.yml` has `permissions: contents: read` only — physically cannot mint OIDC. So `refs/heads/dev` is dormant; drop it. The procedure to re-add for a future preview env is documented in Phase 5 (RT-15).
|
||||
|
||||
2. **No /tmp snapshot needed** (RT-8). `aws/iam-github-oidc-trust.json` is git-tracked. Rollback = `git show HEAD:aws/iam-github-oidc-trust.json | aws iam update-assume-role-policy --role-name github-deploy-miti99bot --policy-document file:///dev/stdin --profile admin`. Capture the pre-edit `HEAD` commit hash for explicit rollback:
|
||||
```sh
|
||||
git rev-parse HEAD # save this — recovery uses it
|
||||
```
|
||||
|
||||
3. **Edit + commit FIRST, then apply** (V-4 decision). The repo file is the source of truth. Commit the edit before invoking `aws iam update-assume-role-policy` so:
|
||||
- `git show HEAD^:aws/iam-github-oidc-trust.json` always recovers the previous state.
|
||||
- If the apply fails / hangs, the repo file matches the intended target — no AWS/repo drift.
|
||||
|
||||
Edit `aws/iam-github-oidc-trust.json` to its final shape:
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Effect": "Allow",
|
||||
"Principal": {"Federated": "arn:aws:iam::225603493174:oidc-provider/token.actions.githubusercontent.com"},
|
||||
"Action": "sts:AssumeRoleWithWebIdentity",
|
||||
"Condition": {
|
||||
"StringEquals": {"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"},
|
||||
"StringLike": {"token.actions.githubusercontent.com:sub": [
|
||||
"repo:tiennm99/miti99bot:ref:refs/heads/main"
|
||||
]}
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
4. **Commit the edit**, then **apply** out-of-band (V-4 — commit-first):
|
||||
```sh
|
||||
git add aws/iam-github-oidc-trust.json
|
||||
git commit -m "fix(security): narrow OIDC trust to main only (F2)"
|
||||
|
||||
aws iam update-assume-role-policy \
|
||||
--role-name github-deploy-miti99bot \
|
||||
--policy-document file://aws/iam-github-oidc-trust.json \
|
||||
--profile admin
|
||||
```
|
||||
If the `aws iam` call fails, the commit is harmless on its own (workflows still use the live AWS-side trust). Reverse with `git revert HEAD` if abandoning the change.
|
||||
|
||||
5. **Smoke test (positive path):** trigger `workflow_dispatch` from GitHub Actions on `main`. Expect the `configure-aws-credentials` step to succeed and the deploy to proceed exactly as before.
|
||||
|
||||
6. **Smoke test (positive verifies narrowing):** the trust narrowing is enforced by AWS IAM at `sts:AssumeRoleWithWebIdentity` time, not by workflow trigger configuration. Step 5's successful `workflow_dispatch` on main proves the trust still permits the intended caller. No PR-triggered workflow currently has `id-token: write`, so a negative-path test would require provisioning a throwaway workflow — out of scope here. If you want belt-and-braces, see Phase 5's "Trust policy invariants" subsection for how to add a synthetic OIDC token verification step.
|
||||
|
||||
7. **Already committed in step 4** (V-4). Nothing to do here.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] **Step 0:** Verify `aws sts get-caller-identity --profile admin` succeeds (RT-4)
|
||||
- [ ] Verify `dev` branch and OIDC `id-token: write` usage with `rg -l 'dev|id-token' .github/workflows/`
|
||||
- [ ] Capture pre-edit HEAD via `git rev-parse HEAD` (RT-8 — git is the snapshot)
|
||||
- [ ] Edit `aws/iam-github-oidc-trust.json`
|
||||
- [ ] Apply via `aws iam update-assume-role-policy`
|
||||
- [ ] workflow_dispatch deploy succeeds on main
|
||||
- [ ] Commit JSON edit
|
||||
- [ ] Mark phase complete via `ck plan check 1`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `aws iam get-role --role-name github-deploy-miti99bot --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition.StringLike."token.actions.githubusercontent.com:sub"'` returns only `["repo:tiennm99/miti99bot:ref:refs/heads/main"]`.
|
||||
- [ ] `workflow_dispatch` deploy on `main` succeeds end-to-end.
|
||||
- [ ] Commit `aws/iam-github-oidc-trust.json` is on `main`.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Rollback needed (e.g. forgot dev branch is used by something) | Low | `git show <pre-edit-HEAD>:aws/iam-github-oidc-trust.json \| aws iam update-assume-role-policy --policy-document file:///dev/stdin --profile admin` — git is the snapshot (RT-8). |
|
||||
| OIDC-claim format typo locks out all deploys | Very Low | JSON file is small + reviewable; AWS rejects malformed `sub` patterns at update time. |
|
||||
| User has no `admin` profile / local AWS creds | Med | Step 0 verifies admin reachability before edits. Console fallback documented inline in step 0 (RT-4). |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- This phase REDUCES attack surface; no new privileges introduced.
|
||||
- After this lands, a leaked GitHub PR-context OIDC token cannot assume this role even if other findings remain unfixed.
|
||||
- Pairs with Phase 4 (F1 cutover) — together they reduce blast radius from "any PR = account takeover" to "any push to main = scoped deploy only".
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 1 is standalone. Phase 2 (Discover required actions) can start in parallel or after — no dependency.
|
||||
@@ -1,193 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Discover required actions"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "1-2h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 2: Discover required actions
|
||||
|
||||
## Overview
|
||||
|
||||
Enumerate every IAM action `sam deploy` invokes for the miti99bot stack, scoped to the specific resources in `template.yaml`. Output is a documented action × resource table that Phase 3 translates into JSON policy statements.
|
||||
|
||||
A missing action here = pipeline broken on next push (per F1 risk highlight). A too-broad action = doesn't satisfy least-privilege intent. Bias toward enumerating real call-sites over generic SAM-deploy guides.
|
||||
|
||||
**Inventory MUST be fully populated before Phase 3 begins** (RT-1). The "Action Inventory" section below cannot ship with `TBD` placeholders. Phase 3's blocking dependency on this phase requires a complete table.
|
||||
|
||||
### Categories that are easy to miss — explicit mandatory checks (RT-10, RT-13)
|
||||
|
||||
For each AWS service in `template.yaml`, you MUST enumerate:
|
||||
|
||||
1. **Lifecycle:** Create / Update / Delete / Get / List actions on each resource ARN.
|
||||
2. **Tagging:** `*:TagResource` / `*:UntagResource` / `*:ListTagsForResource` (or service-specific equivalents — `dynamodb:TagResource`, `lambda:TagResource`, `logs:TagLogGroup`/`TagResource`, `sqs:TagQueue`, `scheduler:TagResource`, `iam:TagRole`, `iam:UntagRole`, `iam:ListRoleTags`). CFN applies tags on every CREATE and many UPDATE paths — missing one = guaranteed UPDATE failure.
|
||||
3. **Sub-resources** for Lambda specifically: `lambda:CreateFunctionUrlConfig`, `lambda:UpdateFunctionUrlConfig`, `lambda:DeleteFunctionUrlConfig`, `lambda:GetFunctionUrlConfig`, `lambda:AddPermission`, `lambda:RemovePermission`, `lambda:GetPolicy`.
|
||||
4. **Rollback path:** `cloudformation:ContinueUpdateRollback`, `cloudformation:CancelUpdateStack`, `cloudformation:RollbackStack` (RT-3 — without these, a stuck stack cannot be recovered without re-attaching FullAccess).
|
||||
5. **SAM-managed S3 bucket bootstrap** (RT-13): `s3:CreateBucket`, `s3:GetBucketLocation`, `s3:GetBucketVersioning`, `s3:PutBucketVersioning`, `s3:GetEncryptionConfiguration`, `s3:PutEncryptionConfiguration`, `s3:PutBucketPolicy`, `s3:ListBucket`, `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`. SAM creates the bucket on first deploy when `resolve_s3 = true` (`samconfig.toml:13`).
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Cover every AWS API call the deploy workflow makes from start (`actions/checkout`) to end (`setMyCommands`).
|
||||
- Group calls by service. For each: action name, resource ARN pattern, optional Conditions.
|
||||
- Cover both happy-path (CREATE) and update-path (UPDATE_IN_PROGRESS → COMPLETE) and rollback (UPDATE_ROLLBACK_*) — IAM checks all three on a failed deploy.
|
||||
|
||||
**Non-functional**
|
||||
- Output lives in the phase file's "Action inventory" section so Phase 3 reads directly from here.
|
||||
- No AWS calls in this phase — pure code reading.
|
||||
|
||||
## Architecture
|
||||
|
||||
Read `template.yaml` resource-by-resource; for each `Type: AWS::*::*`, look up which IAM actions CFN issues. Cross-reference with the workflow's explicit `aws` CLI calls (`aws ssm get-parameter`, `aws cloudformation describe-stacks`).
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Read: `template.yaml` (every Resources entry + Outputs)
|
||||
- Read: `.github/workflows/deploy.yml` (steps post-`configure-aws-credentials` that issue AWS calls)
|
||||
- Read: `samconfig.toml` (`resolve_s3 = true` → SAM manages an artifact bucket)
|
||||
- No files modified in this phase. Output is appended to this phase doc.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Service inventory.** From `template.yaml` resource types:
|
||||
- `AWS::DynamoDB::Table`
|
||||
- `AWS::Logs::LogGroup`, `AWS::Logs::MetricFilter`
|
||||
- `AWS::Serverless::Function` (expands to `AWS::Lambda::Function` + `AWS::IAM::Role` + `AWS::Lambda::Url` + permissions)
|
||||
- `AWS::SQS::Queue`
|
||||
- `AWS::IAM::Role`
|
||||
- `AWS::Scheduler::Schedule`
|
||||
- `AWS::Budgets::Budget` (Conditional)
|
||||
- Plus framework: CloudFormation (changesets), S3 (artifact bucket), STS (caller identity).
|
||||
|
||||
2. **For each resource, enumerate the IAM actions CFN calls on UPDATE+ROLLBACK paths.** Sources: AWS CFN per-resource documentation (the IAM permissions table at the top of each page). Don't trust memory — verify against docs.
|
||||
|
||||
3. **Workflow-explicit calls:**
|
||||
- `aws cloudformation describe-stacks` → `cloudformation:DescribeStacks`
|
||||
- `aws ssm get-parameter --with-decryption` → `ssm:GetParameter` + the SSM service-managed KMS key has implicit access (no extra IAM action required for the AWS-owned key path)
|
||||
- SAM internal: `cloudformation:CreateChangeSet` / `DescribeChangeSet` / `ExecuteChangeSet` / `DeleteChangeSet`, `cloudformation:DescribeStackEvents` / `ListStackResources` / `GetTemplateSummary`, S3 multipart upload, `sts:GetCallerIdentity` (SAM probes account/region on startup).
|
||||
|
||||
4. **iam:PassRole identification.** SAM creates an execution role for `BotFunction` and an inline role for the SchedulerExecutionRole. Both need `iam:PassRole` so CFN can attach them to the Lambda / Scheduler. Scope: roles whose path or name match `miti99bot*`.
|
||||
|
||||
5. **Resource ARN patterns.** Use `miti99bot*` (not `miti99bot`) on stack-scoped resources so a future `miti99bot-dev` parallel stack works (RT-14). For each action, write the tightest ARN pattern still passing on a fresh deploy:
|
||||
- Stack: `arn:aws:cloudformation:ap-southeast-1:225603493174:stack/miti99bot*/*`
|
||||
- ChangeSet: `arn:aws:cloudformation:ap-southeast-1:225603493174:changeSet/*/miti99bot*/*`
|
||||
- DynamoDB: `arn:aws:dynamodb:ap-southeast-1:225603493174:table/miti99bot*` (covers `miti99bot-data` + future dev `miti99bot-dev-data`)
|
||||
- Lambda function: `arn:aws:lambda:ap-southeast-1:225603493174:function:miti99bot*`
|
||||
- Lambda layer (read-only ref to AWSLabs adapter): `arn:aws:lambda:ap-southeast-1:753240598075:layer:LambdaAdapterLayerArm64:*`
|
||||
- SQS queue: `arn:aws:sqs:ap-southeast-1:225603493174:miti99bot*`
|
||||
- IAM roles created by stack: `arn:aws:iam::225603493174:role/miti99bot*` (NOTE: the deploy role itself is `github-deploy-miti99bot` — does NOT match because IAM globs are left-anchored)
|
||||
- Log group: `arn:aws:logs:ap-southeast-1:225603493174:log-group:/aws/lambda/miti99bot*`
|
||||
- SSM parameter: `arn:aws:ssm:ap-southeast-1:225603493174:parameter/miti99bot/*/*` (`/prod/*` AND `/dev/*` — covers both envs without widening to other apps)
|
||||
- Budget: `arn:aws:budgets::225603493174:budget/miti99bot*`
|
||||
- Scheduler: `arn:aws:scheduler:ap-southeast-1:225603493174:schedule/*/miti99bot*` (wildcard the GroupName segment — `template.yaml:217-228` does not set `GroupName`; EventBridge defaults to `default` today but pinning to `default` is undocumented contract → RT-11)
|
||||
- SAM S3 bucket: **discover at execution time** — see step 5a below (RT-13)
|
||||
|
||||
5a. **Discover the actual SAM artifact bucket** (RT-13). Do not assume the `aws-sam-cli-managed-default-samclisourcebucket-*` convention is stable. Run:
|
||||
```sh
|
||||
aws s3 ls --profile admin | grep -E 'sam|miti99bot' || \
|
||||
aws cloudformation describe-stacks --stack-name aws-sam-cli-managed-default \
|
||||
--query "Stacks[0].Outputs[?OutputKey=='SourceBucket'].OutputValue" --output text --profile admin
|
||||
```
|
||||
Use the discovered name verbatim in the Phase 3 ARN list. If empty (fresh account), the policy must include `s3:CreateBucket` on `arn:aws:s3:::aws-sam-cli-managed-default-*` so SAM can create it on first deploy.
|
||||
|
||||
6. **Wildcard-required actions.** Some IAM actions have no resource-level support (must use `Resource: "*"`). Document each:
|
||||
- `sts:GetCallerIdentity` — always `*`
|
||||
- `cloudformation:ListStacks` (used by SAM during a deploy if it scans) — `*`
|
||||
- `s3:ListAllMyBuckets` (SAM uses this to find the managed bucket if not configured) — `*` (low risk: read-only across all buckets, no data exposed)
|
||||
|
||||
7. **Output: Action × Resource inventory.** Append a table to this phase doc with columns: Service · Action · Resource ARN · Notes. Phase 3 consumes this verbatim.
|
||||
|
||||
## Action Inventory
|
||||
|
||||
> Filled during execution. Phase 3 reads from here. **No TBD allowed** — Phase 3 is blocked until every category below has actual actions listed (RT-1).
|
||||
|
||||
Each category MUST include: lifecycle actions, tagging actions, sub-resource actions (if applicable), rollback-path actions (if applicable). See "Categories that are easy to miss" in Overview above.
|
||||
|
||||
### CloudFormation
|
||||
- Stack lifecycle: `cloudformation:CreateStack`, `UpdateStack`, `DeleteStack`, `DescribeStacks`, `DescribeStackEvents`, `DescribeStackResources`, `ListStackResources`, `GetTemplate`, `GetTemplateSummary`, `ValidateTemplate`
|
||||
- ChangeSet: `cloudformation:CreateChangeSet`, `ExecuteChangeSet`, `DescribeChangeSet`, `DeleteChangeSet`, `ListChangeSets`
|
||||
- Rollback (RT-3): `cloudformation:ContinueUpdateRollback`, `CancelUpdateStack`, `RollbackStack`
|
||||
- Tagging: `cloudformation:TagResource`, `UntagResource`, `ListStackResources`
|
||||
- Global read: `cloudformation:ListStacks` (no resource-level support)
|
||||
|
||||
### S3 (SAM artifact bucket)
|
||||
- Bucket lifecycle (RT-13): `s3:CreateBucket`, `GetBucketLocation`, `GetBucketVersioning`, `PutBucketVersioning`, `GetEncryptionConfiguration`, `PutEncryptionConfiguration`, `GetBucketPolicy`, `PutBucketPolicy`
|
||||
- Objects: `s3:ListBucket`, `PutObject`, `GetObject`, `DeleteObject`, `PutObjectTagging`
|
||||
- Global read (justified): `s3:ListAllMyBuckets` (no resource-level support; needed by SAM CLI to find the managed bucket on first run)
|
||||
|
||||
### IAM
|
||||
- Role lifecycle: `iam:CreateRole`, `DeleteRole`, `GetRole`, `ListRoles`, `PutRolePolicy`, `DeleteRolePolicy`, `GetRolePolicy`, `ListRolePolicies`, `AttachRolePolicy`, `DetachRolePolicy`, `ListAttachedRolePolicies`
|
||||
- **NOT included** (RT-2): `iam:UpdateAssumeRolePolicy` — CFN never invokes this on stack-managed roles (trust changes go via Delete+Create). Including it enables trust-rewrite escalation.
|
||||
- PassRole: `iam:PassRole` (with `iam:PassedToService` Condition — see Phase 3, RT-7)
|
||||
- AttachRolePolicy Condition (RT-6): scope `iam:PolicyARN` to AWS-managed policies the stack actually attaches (currently none — Lambda execution role uses SAM macros that inline policies, not attach managed; if SAM ever changes, add specific ARNs). Best path: omit `AttachRolePolicy` entirely until proven necessary.
|
||||
- Tagging: `iam:TagRole`, `UntagRole`, `ListRoleTags`
|
||||
|
||||
### Lambda
|
||||
- Function lifecycle: `lambda:CreateFunction`, `UpdateFunctionCode`, `UpdateFunctionConfiguration`, `GetFunction`, `GetFunctionConfiguration`, `DeleteFunction`, `PublishVersion`, `ListVersionsByFunction`
|
||||
- Function URL sub-resource (RT-10): `lambda:CreateFunctionUrlConfig`, `UpdateFunctionUrlConfig`, `DeleteFunctionUrlConfig`, `GetFunctionUrlConfig`
|
||||
- Resource-based policy: `lambda:AddPermission`, `RemovePermission`, `GetPolicy`
|
||||
- Layer read (cross-account, AWSLabs): `lambda:GetLayerVersion`
|
||||
- Tagging: `lambda:TagResource`, `UntagResource`, `ListTags`
|
||||
|
||||
### DynamoDB
|
||||
- Table lifecycle: `dynamodb:CreateTable`, `UpdateTable`, `DescribeTable`, `DeleteTable`, `ListTables`
|
||||
- Tagging: `dynamodb:TagResource`, `UntagResource`, `ListTagsOfResource`
|
||||
- (No data-plane actions for deploy role; Lambda execution role has those separately.)
|
||||
|
||||
### EventBridge Scheduler
|
||||
- Schedule lifecycle: `scheduler:CreateSchedule`, `UpdateSchedule`, `GetSchedule`, `DeleteSchedule`, `ListSchedules`
|
||||
- Tagging: `scheduler:TagResource`, `UntagResource`, `ListTagsForResource`
|
||||
|
||||
### SQS
|
||||
- Queue lifecycle: `sqs:CreateQueue`, `DeleteQueue`, `GetQueueAttributes`, `SetQueueAttributes`, `GetQueueUrl`, `ListQueues`
|
||||
- Tagging: `sqs:TagQueue`, `UntagQueue`, `ListQueueTags`
|
||||
|
||||
### CloudWatch Logs
|
||||
- Log group lifecycle: `logs:CreateLogGroup`, `DeleteLogGroup`, `DescribeLogGroups`, `PutRetentionPolicy`, `DeleteRetentionPolicy`
|
||||
- Metric filter: `logs:PutMetricFilter`, `DeleteMetricFilter`, `DescribeMetricFilters`
|
||||
- Tagging: `logs:TagResource`, `UntagResource`, `ListTagsForResource`
|
||||
|
||||
### Budgets
|
||||
- Budget lifecycle: `budgets:CreateBudget`, `ModifyBudget`, `DescribeBudget`, `DeleteBudget`
|
||||
- Notification: `budgets:CreateNotification`, `DeleteNotification`, `DescribeNotificationsForBudget`, `CreateSubscriber`, `DeleteSubscriber`
|
||||
|
||||
### SSM (workflow-explicit)
|
||||
- `ssm:GetParameter`, `ssm:GetParameters` — used by `.github/workflows/deploy.yml:53,76,80,84,108` (cron secret + telegram token + webhook secret fetches) and by Lambda cold start; scope to `parameter/miti99bot/*/*`
|
||||
|
||||
### STS
|
||||
- `sts:GetCallerIdentity` — used by SAM at deploy start (no resource-level support; wildcard required)
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Service-by-service walk of template.yaml
|
||||
- [ ] Cross-reference each AWS::* type against CFN per-resource IAM requirements
|
||||
- [ ] Enumerate workflow-explicit `aws` CLI calls
|
||||
- [ ] Identify `iam:PassRole` targets
|
||||
- [ ] Build action × resource × ARN-pattern table in this doc
|
||||
- [ ] Identify wildcard-required actions and justify each
|
||||
- [ ] Mark phase complete via `ck plan check 2`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Every CFN resource type in `template.yaml` has an entry in the action inventory.
|
||||
- [ ] Every wildcard `Resource: "*"` has a 1-line "why not scoped" justification.
|
||||
- [ ] Phase 3 can write the policy file by transcribing the inventory; no further AWS docs lookup needed in Phase 3.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Miss a CFN action (e.g. tagging permission, drift detection) | Med | Test by attaching the eventual policy in dual-mode (Phase 4) before detaching FullAccess. CFN failures surface in CloudFormation events; map to missing action and re-add. |
|
||||
| AWS adds new required actions after this work | Low | Documented in `aws/README.md` (Phase 5): on `sam deploy` UPDATE failure with AccessDenied, check CloudTrail event → identify missing action → patch policy. |
|
||||
| Over-tight ARN pattern (e.g. forgot `/index/*` for a future GSI) | Low | Phase 3 will use globs (`miti99bot*` not `miti99bot`) where the stack might extend. |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- This phase is read-only — no security implications.
|
||||
- Output drives Phase 3's security boundary.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 3 starts after this phase's action inventory is complete.
|
||||
@@ -1,137 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Draft custom policy"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "1h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 3: Draft custom policy
|
||||
|
||||
## Overview
|
||||
|
||||
Translate Phase 2's action inventory into a single JSON IAM policy document committed to the repo at `aws/iam-github-deploy-policy.json`. Validate JSON syntax and run AWS IAM Policy Simulator dry-run.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Single `Version: "2012-10-17"` policy with multiple `Statement` entries, grouped by service.
|
||||
- Each statement: `Effect: Allow`, action list, resource ARN list (with `${AWS::AccountId}` / `${AWS::Region}` interpolated to the actual account `225603493174` and region `ap-southeast-1`).
|
||||
- `iam:PassRole` statement scoped via `Condition.ForAllValues:StringEquals.iam:PassedToService` to `lambda.amazonaws.com` and `scheduler.amazonaws.com` only (RT-7). Adding a new service later = explicit policy update via documented procedure in Phase 5.
|
||||
- `iam:UpdateAssumeRolePolicy` deliberately EXCLUDED (RT-2) — CFN does not invoke it on stack-managed roles; including it enables trust-rewrite escalation.
|
||||
- `iam:AttachRolePolicy` either omitted entirely OR scoped via `Condition.ArnEquals.iam:PolicyARN` to a documented allowlist (RT-6). Current SAM macros use inline `PutRolePolicy` only — start with omission, add only if a deploy fails AccessDenied on this action.
|
||||
- Wildcard `Resource: "*"` only where the action has no resource-level support (Phase 2 enumerated these).
|
||||
|
||||
**Empirical verification (RT-7) before applying:**
|
||||
- After Phase 2 inventory complete, before Phase 4 starts, run `aws iam simulate-principal-policy` against the draft policy with each Phase-2-enumerated action × target ARN. Pay special attention to `iam:PassRole` on each stack-managed role: the simulator's "ImplicitDeny" result for `iam:PassedToService` mismatches surfaces here, not at deploy time.
|
||||
|
||||
**Non-functional**
|
||||
- Total policy size must stay under 6,144 chars (AWS managed-policy hard limit) OR be split into two inline policies on the same role.
|
||||
- File committed to repo so future bootstrap reads from version control.
|
||||
- JSON formatted with 2-space indent; trailing newline; sorted statements by service alphabetically for diff readability.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
aws/
|
||||
├── iam-github-deploy-policy.json ← NEW (this phase)
|
||||
├── iam-github-oidc-trust.json ← Phase 1 narrowed this
|
||||
└── README.md ← Phase 5 updates to reference new file
|
||||
```
|
||||
|
||||
Single artifact, version-controlled, applied via `aws iam put-role-policy --policy-name miti99bot-deploy --role-name github-deploy-miti99bot --policy-document file://aws/iam-github-deploy-policy.json` (inline policy, not managed — keeps the policy with the role lifecycle).
|
||||
|
||||
Inline vs managed:
|
||||
- Inline: scoped to role lifecycle, no separate ARN, deleted with role.
|
||||
- Managed: separate ARN, reusable across roles, has 6,144-char hard limit (same as inline) + 10-policies-per-role limit.
|
||||
- Choice: **inline.** Single role, no reuse needed, simpler lifecycle.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `aws/iam-github-deploy-policy.json`
|
||||
- Read-only: `plans/260518-1019-iam-least-privilege/phase-02-discover-required-actions.md` (source of truth for actions × resources)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Build the JSON from the Phase 2 inventory.** No `/* ... */` placeholders — every Action array fully populated by transcribing Phase 2 (RT-1). Statement skeleton (Sids + ARNs ready; transcribe full action lists from Phase 2):
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{"Sid": "Budgets", "Effect": "Allow", "Action": ["<from Phase 2 Budgets>"], "Resource": "arn:aws:budgets::225603493174:budget/miti99bot*"},
|
||||
{"Sid": "CloudFormation", "Effect": "Allow", "Action": ["<from Phase 2 CFN incl. ContinueUpdateRollback, CancelUpdateStack, *TagResource>"], "Resource": ["arn:aws:cloudformation:ap-southeast-1:225603493174:stack/miti99bot*/*", "arn:aws:cloudformation:ap-southeast-1:225603493174:changeSet/*/miti99bot*/*"]},
|
||||
{"Sid": "CloudFormationGlobalRead", "Effect": "Allow", "Action": ["cloudformation:ListStacks", "cloudformation:ValidateTemplate"], "Resource": "*"},
|
||||
{"Sid": "DynamoDB", "Effect": "Allow", "Action": ["<from Phase 2 DynamoDB incl. TagResource, UntagResource, ListTagsOfResource>"], "Resource": "arn:aws:dynamodb:ap-southeast-1:225603493174:table/miti99bot*"},
|
||||
{"Sid": "EventBridge", "Effect": "Allow", "Action": ["<from Phase 2 Scheduler incl. TagResource>"], "Resource": "arn:aws:scheduler:ap-southeast-1:225603493174:schedule/*/miti99bot*"},
|
||||
{"Sid": "IAMRolesScoped", "Effect": "Allow", "Action": ["iam:CreateRole","iam:DeleteRole","iam:GetRole","iam:ListRoles","iam:PutRolePolicy","iam:DeleteRolePolicy","iam:GetRolePolicy","iam:ListRolePolicies","iam:ListAttachedRolePolicies","iam:TagRole","iam:UntagRole","iam:ListRoleTags"], "Resource": "arn:aws:iam::225603493174:role/miti99bot*"},
|
||||
{"Sid": "IAMPassRole", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::225603493174:role/miti99bot*", "Condition": {"ForAllValues:StringEquals": {"iam:PassedToService": ["lambda.amazonaws.com", "scheduler.amazonaws.com"]}}},
|
||||
{"Sid": "Lambda", "Effect": "Allow", "Action": ["<from Phase 2 Lambda incl. *FunctionUrlConfig, AddPermission, RemovePermission, GetPolicy, TagResource>"], "Resource": "arn:aws:lambda:ap-southeast-1:225603493174:function:miti99bot*"},
|
||||
{"Sid": "LambdaLayerRead","Effect": "Allow", "Action": "lambda:GetLayerVersion", "Resource": "arn:aws:lambda:ap-southeast-1:753240598075:layer:LambdaAdapterLayerArm64:*"},
|
||||
{"Sid": "Logs", "Effect": "Allow", "Action": ["<from Phase 2 Logs incl. PutMetricFilter, *TagResource>"], "Resource": ["arn:aws:logs:ap-southeast-1:225603493174:log-group:/aws/lambda/miti99bot*", "arn:aws:logs:ap-southeast-1:225603493174:log-group:/aws/lambda/miti99bot*:*"]},
|
||||
{"Sid": "S3SamArtifacts", "Effect": "Allow", "Action": ["<from Phase 2 S3 incl. CreateBucket, GetBucketLocation, GetEncryptionConfiguration etc.>"], "Resource": ["arn:aws:s3:::aws-sam-cli-managed-default-*", "arn:aws:s3:::aws-sam-cli-managed-default-*/*"]},
|
||||
{"Sid": "S3GlobalList", "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*"},
|
||||
{"Sid": "SQS", "Effect": "Allow", "Action": ["<from Phase 2 SQS incl. TagQueue, UntagQueue, ListQueueTags>"], "Resource": "arn:aws:sqs:ap-southeast-1:225603493174:miti99bot*"},
|
||||
{"Sid": "SSMRead", "Effect": "Allow", "Action": ["ssm:GetParameter","ssm:GetParameters"], "Resource": "arn:aws:ssm:ap-southeast-1:225603493174:parameter/miti99bot/*/*"},
|
||||
{"Sid": "STS", "Effect": "Allow", "Action": "sts:GetCallerIdentity", "Resource": "*"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**NOT in the policy** (intentional, RT-2 + RT-6):
|
||||
- `iam:UpdateAssumeRolePolicy` — CFN doesn't need it; including it enables trust-rewrite escalation.
|
||||
- `iam:AttachRolePolicy` / `iam:DetachRolePolicy` — current SAM transforms use inline `PutRolePolicy` only. Add later with `Condition.ArnEquals.iam:PolicyARN` to a specific allowlist IF a real deploy fails on it; do not add prophylactically.
|
||||
|
||||
2. **Fill action lists** from Phase 2 inventory verbatim. Replace every `<from Phase 2 …>` placeholder with the actual action array. Sort alphabetically within each `Action` array.
|
||||
|
||||
3. **Validate JSON syntax:**
|
||||
```sh
|
||||
jq . aws/iam-github-deploy-policy.json > /dev/null && echo OK
|
||||
```
|
||||
|
||||
4. **Verify byte count** under 6,144:
|
||||
```sh
|
||||
wc -c aws/iam-github-deploy-policy.json
|
||||
```
|
||||
If over: split S3 + Lambda + IAM statements into a second inline policy `miti99bot-deploy-2`.
|
||||
|
||||
5. **IAM Policy Simulator dry-run** (optional but recommended — free). Use AWS Console: IAM → Policies → "Simulate" → paste the JSON → select each Phase-2 action with its target ARN → confirm "Allowed" for every legitimate operation. Note any "Implicit Deny" results and patch.
|
||||
|
||||
6. **Commit** the JSON file. Do NOT yet apply to the role (Phase 4).
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Build JSON skeleton with all statement Sids
|
||||
- [ ] Fill each Action array from Phase 2 inventory
|
||||
- [ ] Apply alphabetical sort within Actions for diff readability
|
||||
- [ ] `jq .` validates
|
||||
- [ ] Byte count under 6,144 (split if not)
|
||||
- [ ] (Optional) IAM Policy Simulator dry-run passes for every Phase-2 action
|
||||
- [ ] Commit `aws/iam-github-deploy-policy.json`
|
||||
- [ ] Mark phase complete via `ck plan check 3`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `aws/iam-github-deploy-policy.json` exists, valid JSON, under 6,144 bytes (single-policy form).
|
||||
- [ ] Every action enumerated in Phase 2 appears in exactly one statement.
|
||||
- [ ] `iam:PassRole` constrained by `iam:PassedToService` to the two services that need it.
|
||||
- [ ] Every `Resource: "*"` has a justification comment outside the JSON (in Phase 2 inventory).
|
||||
- [ ] Commit on `main`. Role NOT yet modified (Phase 4 applies).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Exceed 6,144 char policy limit | Low | Split into 2 inline policies (miti99bot-deploy-cfn-lambda + miti99bot-deploy-data-iam). |
|
||||
| Typo in action name | Low | `jq .` catches JSON syntax; IAM Policy Simulator catches unknown action names. |
|
||||
| Forgot a CFN resource-tagging action | Med | Phase 4 dual-attach validate catches at first deploy; iterate. |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- No AWS state changes in this phase — policy is on disk only.
|
||||
- File contains no secrets — safe to commit.
|
||||
- ARN patterns hardcode account ID `225603493174` and region `ap-southeast-1`; documented as project constants; rotating either invalidates the file but the project's `aws/README.md` already pins them.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 4 (cutover) consumes this file. Do not start Phase 4 until commit lands.
|
||||
@@ -1,233 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Cutover + validate"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "1-2h"
|
||||
dependencies: [3]
|
||||
---
|
||||
|
||||
# Phase 4: Cutover + validate
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the 10× `*FullAccess` managed policies on `github-deploy-miti99bot` with the inline custom policy from Phase 3. Validate via `workflow_dispatch`. This is the highest-risk phase — a wrong policy locks the deploy pipeline.
|
||||
|
||||
**Revised strategy after red-team (RT-3):** two-stage cutover with a dual-attach trial deploy first.
|
||||
|
||||
- **Stage 4a (Trial):** attach the new inline policy ALONGSIDE the existing FullAccess set. Trigger a deploy. The deploy succeeds because IAM evaluates the UNION — but CloudTrail records which policy authorized each action. This surfaces *missing actions* in the new policy WITHOUT the deploy actually failing. We don't claim sufficiency from this; we use it as a syntax-and-coverage smoke test before risking the cutover.
|
||||
- **Stage 4b (Cutover):** disable the deploy workflow, detach the 10 FullAccess policies, re-enable the workflow, trigger validation deploy. Rollback path: re-attach FullAccess from a committed script.
|
||||
|
||||
The "AccessDenied during ROLLBACK" risk requires `cloudformation:ContinueUpdateRollback` (already in Phase 3 policy per RT-3). Also requires the deploy workflow itself to be DISABLED during 4b so a concurrent push doesn't run mid-cutover (RT-9).
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Custom policy attached as inline policy `miti99bot-deploy` on role.
|
||||
- All 10 `*FullAccess` managed policies detached from role.
|
||||
- Subsequent `workflow_dispatch` deploy succeeds end-to-end (includes the smoke test + Telegram webhook setup steps already in the workflow).
|
||||
- On AccessDenied during deploy: instant rollback re-attaches all 10 FullAccess policies; iterate on the inline policy.
|
||||
|
||||
**Non-functional**
|
||||
- All IAM mutations applied out-of-band via maintainer's local `admin` profile (chicken-and-egg per F1).
|
||||
- Rollback script prepared and dry-tested BEFORE cutover starts.
|
||||
- Maintain a < 15-min window where deploys can be re-enabled if cutover fails.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Before: During: After:
|
||||
[10× *FullAccess managed] → [10× FullAccess] → [inline: miti99bot-deploy]
|
||||
[inline: miti99bot-deploy]
|
||||
^^^ never the steady state ^^^
|
||||
```
|
||||
|
||||
The middle "both attached" state exists only as a transient — used for the snapshot moment. We do not validate from there; we validate after the FullAccess detach.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Read: `aws/iam-github-deploy-policy.json` (from Phase 3)
|
||||
- Read: `.github/workflows/deploy.yml` (target for workflow_dispatch)
|
||||
- No code changes in this phase — only AWS state changes.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Pre-flight gates (RT-4, RT-8)
|
||||
|
||||
0. **Verify admin profile reachable.** `aws/README.md:119` recommends deleting admin keys as hardening posture. Before any cutover:
|
||||
```sh
|
||||
aws sts get-caller-identity --profile admin
|
||||
```
|
||||
If fails: recreate admin access keys via console (root login → IAM → Users → admin → Security credentials → Create access key). DO NOT proceed until this succeeds. Console-only path is documented but unwieldy for the 11+ IAM calls below.
|
||||
|
||||
1. **Commit the rollback script to the repo** (RT-8) at `aws/iam-rollback-fullaccess.sh`. Per-call retry on throttling + post-loop verification:
|
||||
```sh
|
||||
#!/bin/sh
|
||||
# Re-attaches the 10 FullAccess managed policies to github-deploy-miti99bot.
|
||||
# Idempotent: attach-role-policy succeeds even if policy already attached.
|
||||
ROLE=github-deploy-miti99bot
|
||||
POLICIES="
|
||||
arn:aws:iam::aws:policy/AWSCloudFormationFullAccess
|
||||
arn:aws:iam::aws:policy/AWSLambda_FullAccess
|
||||
arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
|
||||
arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess
|
||||
arn:aws:iam::aws:policy/AmazonSQSFullAccess
|
||||
arn:aws:iam::aws:policy/AmazonSSMFullAccess
|
||||
arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
|
||||
arn:aws:iam::aws:policy/AWSBudgetsActionsWithAWSResourceControlAccess
|
||||
arn:aws:iam::aws:policy/IAMFullAccess
|
||||
arn:aws:iam::aws:policy/AmazonS3FullAccess
|
||||
"
|
||||
for arn in $POLICIES; do
|
||||
for try in 1 2 3 4 5; do
|
||||
if aws iam attach-role-policy --role-name "$ROLE" --policy-arn "$arn" --profile admin 2>&1; then
|
||||
break
|
||||
fi
|
||||
echo "retry $try for $arn after throttle…"; sleep $((try * 2))
|
||||
done
|
||||
done
|
||||
# Verify final state — exit non-zero if anything is missing
|
||||
ATTACHED=$(aws iam list-attached-role-policies --role-name "$ROLE" --profile admin --query 'AttachedPolicies[].PolicyArn' --output text)
|
||||
MISSING=0
|
||||
for arn in $POLICIES; do
|
||||
echo "$ATTACHED" | grep -q "$arn" || { echo "MISSING: $arn"; MISSING=1; }
|
||||
done
|
||||
[ "$MISSING" = 0 ] && echo "Rollback complete — all 10 FullAccess policies attached." || { echo "Rollback INCOMPLETE — see MISSING lines above. Re-run or attach via console."; exit 1; }
|
||||
```
|
||||
`chmod +x aws/iam-rollback-fullaccess.sh`. Commit alongside `aws/iam-github-deploy-policy.json`. Any teammate can recover, not only the operator.
|
||||
|
||||
2. **Verify Phase 3 prerequisites:**
|
||||
- `aws/iam-github-deploy-policy.json` exists on `main`; `jq .` validates.
|
||||
- No deploy currently in progress (`aws cloudformation describe-stacks --stack-name miti99bot --query 'Stacks[0].StackStatus' --profile admin` returns `*_COMPLETE`).
|
||||
- Coordinate with collaborators: announce a deploy freeze in the team channel for the cutover window (~30 min).
|
||||
|
||||
### Stage 4a — Dual-attach trial (RT-3)
|
||||
|
||||
3a. **Attach the new inline policy alongside existing FullAccess** (does not detach anything yet):
|
||||
```sh
|
||||
aws iam put-role-policy \
|
||||
--role-name github-deploy-miti99bot \
|
||||
--policy-name miti99bot-deploy \
|
||||
--policy-document file://aws/iam-github-deploy-policy.json \
|
||||
--profile admin
|
||||
```
|
||||
|
||||
3b. **Trial deploy:** trigger `workflow_dispatch` on `main`. With BOTH policy sets attached, the deploy MUST succeed (FullAccess covers any gap in the new policy). Confirm success of every workflow step including smoke test + Telegram webhook + Telegram commands.
|
||||
|
||||
3c. **CloudTrail sanity check (optional but recommended):** for the trial-deploy invocation, query CloudTrail for `userIdentity.arn` matching the deploy role and look at the `requestParameters` — events authorized only by the FullAccess managed policies (and not by the inline policy) signal a coverage gap in `miti99bot-deploy`. Patch the inline policy + redo step 3a before proceeding to 4b. (This step uses console UI; CLI access not required.)
|
||||
|
||||
### Stage 4b — Cutover (the actual narrowing)
|
||||
|
||||
4. **Disable the deploy workflow** to prevent concurrent runs (RT-9):
|
||||
```sh
|
||||
gh workflow disable deploy-aws.yml
|
||||
```
|
||||
Or in GitHub UI: Actions → deploy-aws → "Disable workflow". Re-enabled in step 7.
|
||||
|
||||
5. **Detach the 10 FullAccess policies with retry-on-throttle:**
|
||||
```sh
|
||||
ROLE=github-deploy-miti99bot
|
||||
for arn in \
|
||||
arn:aws:iam::aws:policy/AWSCloudFormationFullAccess \
|
||||
arn:aws:iam::aws:policy/AWSLambda_FullAccess \
|
||||
arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess \
|
||||
arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess \
|
||||
arn:aws:iam::aws:policy/AmazonSQSFullAccess \
|
||||
arn:aws:iam::aws:policy/AmazonSSMFullAccess \
|
||||
arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \
|
||||
arn:aws:iam::aws:policy/AWSBudgetsActionsWithAWSResourceControlAccess \
|
||||
arn:aws:iam::aws:policy/IAMFullAccess \
|
||||
arn:aws:iam::aws:policy/AmazonS3FullAccess; do
|
||||
for try in 1 2 3 4 5; do
|
||||
aws iam detach-role-policy --role-name "$ROLE" --policy-arn "$arn" --profile admin && break
|
||||
sleep $((try * 2))
|
||||
done
|
||||
done
|
||||
```
|
||||
|
||||
6. **Verify role state:**
|
||||
```sh
|
||||
aws iam list-attached-role-policies --role-name github-deploy-miti99bot --profile admin
|
||||
# Expect: empty AttachedPolicies list
|
||||
aws iam list-role-policies --role-name github-deploy-miti99bot --profile admin
|
||||
# Expect: ["miti99bot-deploy"]
|
||||
```
|
||||
|
||||
7. **Re-enable the workflow:**
|
||||
```sh
|
||||
gh workflow enable deploy-aws.yml
|
||||
```
|
||||
|
||||
8. **Trigger validation deploy:** `workflow_dispatch` on `main` from the GitHub Actions UI. This is the FIRST deploy with ONLY the new inline policy. If it succeeds end-to-end → cutover complete. If AccessDenied appears anywhere:
|
||||
- **Immediately run** `bash aws/iam-rollback-fullaccess.sh --profile admin`. The script handles throttling + verifies all 10 re-attached.
|
||||
- **If the stack ended in `UPDATE_ROLLBACK_FAILED`** (RT-3): after re-attaching FullAccess, run:
|
||||
```sh
|
||||
aws cloudformation continue-update-rollback --stack-name miti99bot --profile admin
|
||||
```
|
||||
Wait for `UPDATE_ROLLBACK_COMPLETE`. Then push a fresh build (or `workflow_dispatch`) to re-establish baseline.
|
||||
- Capture the failing action from CloudTrail. Update `aws/iam-github-deploy-policy.json`, commit, and re-attempt from step 3a (trial again before cutover).
|
||||
|
||||
9. **Smoke test post-deploy:**
|
||||
- Workflow's built-in steps (`Smoke test`, `Register Telegram webhook`, `Register Telegram command menu`) must all succeed.
|
||||
- Manually trigger the EventBridge Scheduler "Run now" from the AWS Console to confirm the cron pathway still functions end-to-end with the new role.
|
||||
|
||||
10. **Final state:** role has only `miti99bot-deploy` inline policy. Document the cutover commit hash + timestamp in the "Cutover Record" section below.
|
||||
|
||||
## Cutover Record
|
||||
|
||||
> Filled in during execution.
|
||||
|
||||
- Cutover started: `YYYY-MM-DD HH:MM:SS UTC`
|
||||
- Cutover finished: `YYYY-MM-DD HH:MM:SS UTC`
|
||||
- Validating deploy run ID: `<GHA run URL>`
|
||||
- Final role policies: `["miti99bot-deploy"]`
|
||||
- Iterations needed: `<count>`
|
||||
- Missing actions found mid-cutover: `<list, if any>`
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] **Step 0:** `aws sts get-caller-identity --profile admin` succeeds (RT-4)
|
||||
- [ ] **Step 1:** `aws/iam-rollback-fullaccess.sh` committed with retry+verify logic (RT-8)
|
||||
- [ ] **Step 2:** Phase 3 prerequisites verified; deploy freeze announced
|
||||
- [ ] **Stage 4a step 3a:** new inline policy attached alongside FullAccess
|
||||
- [ ] **Stage 4a step 3b:** trial `workflow_dispatch` deploy succeeds end-to-end
|
||||
- [ ] **Stage 4a step 3c:** (optional) CloudTrail confirms no actions authorized solely by FullAccess
|
||||
- [ ] **Stage 4b step 4:** `gh workflow disable deploy-aws.yml` (RT-9)
|
||||
- [ ] **Stage 4b step 5:** All 10 FullAccess policies detached (retry-on-throttle)
|
||||
- [ ] **Stage 4b step 6:** Role state verified (only `miti99bot-deploy` policy listed)
|
||||
- [ ] **Stage 4b step 7:** `gh workflow enable deploy-aws.yml`
|
||||
- [ ] **Stage 4b step 8:** Validation `workflow_dispatch` deploy succeeds end-to-end with ONLY the new inline policy
|
||||
- [ ] **Stage 4b step 9:** EventBridge Scheduler "Run now" succeeds
|
||||
- [ ] Cutover record filled in
|
||||
- [ ] Mark phase complete via `ck plan check 4`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `aws iam list-attached-role-policies --role-name github-deploy-miti99bot` returns empty.
|
||||
- [ ] `aws iam list-role-policies --role-name github-deploy-miti99bot` returns `["miti99bot-deploy"]`.
|
||||
- [ ] **Stage 4a trial deploy** succeeds with both policy sets attached (proves new policy syntax is valid and doesn't break anything).
|
||||
- [ ] **Stage 4b validation deploy** succeeds with ONLY the new inline policy — zero rollbacks needed during cutover.
|
||||
- [ ] EventBridge Scheduler manual fire succeeds within 60s of "Run now".
|
||||
- [ ] Cutover record section above is filled in.
|
||||
- [ ] `aws/iam-rollback-fullaccess.sh` is committed to the repo (RT-8).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Severity | Mitigation |
|
||||
|---|---|---|---|
|
||||
| Missing IAM action — deploy fails mid-CFN-update | Med | High | Rollback script ready (step 1). Iteration loop documented in step 7. CFN UPDATE_ROLLBACK_COMPLETE state is recoverable — re-attach FullAccess, the next deploy fixes any drift. |
|
||||
| Missing action AFTER CFN_COMPLETE (e.g. Telegram webhook step uses ssm:GetParameter on a parameter not in scope) | Med | Med | Run rollback. The CFN state is fine; only the post-deploy steps failed. Patch policy and re-run workflow_dispatch — no CFN churn. |
|
||||
| AccessDenied during ROLLBACK path (worst case) | Low | Critical | If CFN can't roll back due to missing IAM action, run rollback script immediately and let CFN retry with FullAccess. Then patch the missing action and re-run. |
|
||||
| Maintainer loses local `admin` creds mid-cutover | Low | High | All steps idempotent — re-running from any point produces the same end state. AWS Console works as alternate path for every step. |
|
||||
| Concurrent push to main during cutover window | Low | Med | `concurrency: deploy-prod` group in workflow prevents overlapping runs. Cutover window <15min; coordinate with anyone else on the repo. |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- During the cutover window, the deploy role's permissions ARE temporarily over-broad (both old and new attached). This is < 1 minute.
|
||||
- After cutover: blast radius reduced from "10× FullAccess incl. account takeover via IAMFullAccess" to "stack-scoped CRUD on `miti99bot*` resources only". Paired with Phase 1 narrowing OIDC trust, the combined reduction is what F1+F2 set out to achieve.
|
||||
- `iam:PassRole` Condition keeps the role from being able to pass arbitrary roles to Lambda/Scheduler — only `miti99bot-*` roles.
|
||||
- The new inline policy is committed to git (Phase 3) — auditable, drift-detectable by comparing `aws iam get-role-policy` output to `aws/iam-github-deploy-policy.json`.
|
||||
|
||||
## Next Steps
|
||||
|
||||
After this phase: Phase 5 updates `aws/README.md` to reflect the new bootstrap. The plan is complete after Phase 5.
|
||||
@@ -1,148 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Update bootstrap docs"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "30m"
|
||||
dependencies: [4]
|
||||
---
|
||||
|
||||
# Phase 5: Update bootstrap docs
|
||||
|
||||
## Overview
|
||||
|
||||
Update `aws/README.md` step 4 to reflect the new least-privilege bootstrap: a single `aws iam put-role-policy` call from a committed JSON file instead of attaching 10× `*FullAccess` managed policies. Add a "drift detection" note + rollback procedure.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- `aws/README.md` step 4 replaced with new bootstrap flow.
|
||||
- New step references the committed `aws/iam-github-deploy-policy.json`.
|
||||
- Section 7 ("Tighten — optional but recommended") updated: bullet 3 (replacing broad managed policies) is now redundant — mark as DONE or remove.
|
||||
- Add a brief "Updating the deploy policy" subsection explaining: edit the JSON in repo → `aws iam put-role-policy` from `admin` profile (NOT through the workflow) → commit.
|
||||
|
||||
**Non-functional**
|
||||
- Docs explain WHY (link to F1 finding + this plan dir) so future maintainers don't reattach FullAccess for convenience.
|
||||
- Keep README concise — defer rationale to plan + audit report.
|
||||
|
||||
## Architecture
|
||||
|
||||
No architecture change. Pure documentation.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `aws/README.md` (steps 4, 7)
|
||||
- Read-only: `aws/iam-github-deploy-policy.json` (referenced from README)
|
||||
- Read-only: `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` (link target)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Read current `aws/README.md`** to confirm section anchors (especially step 4 and section 7).
|
||||
|
||||
2. **Rewrite step 4** ("Deploy IAM role for GitHub Actions"):
|
||||
- Keep the `aws iam create-role` call (trust policy unchanged from Phase 1 narrowing).
|
||||
- Replace the `for arn in ... do attach ... done` loop with:
|
||||
```sh
|
||||
aws iam put-role-policy \
|
||||
--role-name github-deploy-miti99bot \
|
||||
--policy-name miti99bot-deploy \
|
||||
--policy-document file://aws/iam-github-deploy-policy.json \
|
||||
--profile admin
|
||||
```
|
||||
- Add 1-line note: "Scoped to stacks/resources named `miti99bot*`. See [security audit](../plans/reports/code-reviewer-260518-1019-security-aws-infra.md) F1 for rationale and [plan](../plans/260518-1019-iam-least-privilege/) for the cutover record."
|
||||
|
||||
3. **Update section 7** ("Tighten — optional but recommended"):
|
||||
- Bullet 3 ("Replace the broad managed policies on `github-deploy-miti99bot` with stack-scoped custom policies") is now done — remove or mark `[done in 2026-05]`.
|
||||
- Bullet 1 (rotate admin keys) and bullet 2 (workflow_dispatch confirmation) remain.
|
||||
|
||||
4. **Add new subsection "Updating the deploy policy"** at the end of section 4:
|
||||
```md
|
||||
### Updating the deploy policy
|
||||
|
||||
When `template.yaml` adds a new resource type, the deploy role may need new IAM
|
||||
actions. Workflow:
|
||||
|
||||
1. Edit `aws/iam-github-deploy-policy.json` — add the action(s) + ARN pattern.
|
||||
2. Apply out-of-band from a maintainer's `admin` profile (NOT via the workflow):
|
||||
```sh
|
||||
aws iam put-role-policy --role-name github-deploy-miti99bot \
|
||||
--policy-name miti99bot-deploy \
|
||||
--policy-document file://aws/iam-github-deploy-policy.json --profile admin
|
||||
```
|
||||
3. Commit the JSON. Next deploy uses the new permissions.
|
||||
|
||||
Drift check — structural compare, not byte-diff (RT-12). `aws iam get-role-policy` returns JSON that differs from the local file in key ordering / whitespace but may be semantically identical. Compare normalized:
|
||||
```sh
|
||||
diff <(aws iam get-role-policy --role-name github-deploy-miti99bot \
|
||||
--policy-name miti99bot-deploy --profile admin --query PolicyDocument | jq -S .) \
|
||||
<(jq -S . aws/iam-github-deploy-policy.json)
|
||||
```
|
||||
Non-empty output = INVESTIGATE before reapplying. AWS-side may have been intentionally patched during an outage; blindly re-applying overwrites that fix.
|
||||
```
|
||||
|
||||
### Trust policy invariants (RT-15)
|
||||
|
||||
`aws/iam-github-oidc-trust.json` constrains which GitHub Actions contexts can
|
||||
assume `github-deploy-miti99bot`. The current allowlist is intentionally
|
||||
narrow: only pushes to `main` can deploy.
|
||||
|
||||
**To add a new branch / context** (e.g., a future `dev` preview deploy):
|
||||
|
||||
1. Edit `aws/iam-github-oidc-trust.json` — add the new `sub` claim to the
|
||||
`StringLike` array. Examples:
|
||||
- `repo:tiennm99/miti99bot:ref:refs/heads/dev` — pushes to `dev` branch
|
||||
- `repo:tiennm99/miti99bot:environment:preview` — workflows scoped to a
|
||||
GitHub Environment named `preview` (requires `permissions: id-token: write`)
|
||||
2. Apply out-of-band:
|
||||
```sh
|
||||
aws iam update-assume-role-policy --role-name github-deploy-miti99bot \
|
||||
--policy-document file://aws/iam-github-oidc-trust.json --profile admin
|
||||
```
|
||||
3. Commit. Test by triggering the new workflow path.
|
||||
|
||||
**Reasons `pull_request` is NOT in the allowlist** (do not re-add without
|
||||
reviewing): PR-context OIDC tokens are derivable from any contributor's
|
||||
PR. Granting the deploy role to PRs is equivalent to granting deploy access
|
||||
to every contributor. Combined with the inline policy's IAM/Lambda/DynamoDB
|
||||
actions, an attacker-controlled PR could exfiltrate or alter prod state.
|
||||
|
||||
5. **Commit** the README edit + the policy JSON file (if not already committed in Phase 4) on the same branch / PR.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Read current `aws/README.md` to map anchors
|
||||
- [ ] Rewrite step 4 with `put-role-policy` flow + link to `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` (RT-5 — file now exists)
|
||||
- [ ] Add "Updating the deploy policy" subsection with `jq -S` structural-diff drift check (RT-12)
|
||||
- [ ] Add "Trust policy invariants" subsection documenting how to re-add sub claims safely (RT-15)
|
||||
- [ ] Update section 7 — mark broad-policy-replacement as done
|
||||
- [ ] Add "Updating the deploy policy" subsection with drift-check command
|
||||
- [ ] Commit `aws/README.md` change
|
||||
- [ ] Mark phase complete via `ck plan check 5`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `aws/README.md` step 4 no longer references `*FullAccess` managed policies.
|
||||
- [ ] `aws/README.md` references `aws/iam-github-deploy-policy.json` as the canonical bootstrap source.
|
||||
- [ ] `aws/README.md` links to `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` for F1/F2 rationale.
|
||||
- [ ] Section 7 no longer lists "tighten policies" as a TODO.
|
||||
- [ ] New "Updating the deploy policy" subsection includes the `jq -S` structural-diff drift command (not plain `diff`).
|
||||
- [ ] New "Trust policy invariants" subsection documents the procedure to re-add a `sub` claim and explains why `pull_request` is excluded.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| README drifts from actual role state | Med | "Updating the deploy policy" subsection includes a drift-check command. Future maintainers run it before assuming the README is accurate. |
|
||||
| Future maintainer re-attaches FullAccess "just to ship a hotfix" | Med | README explicitly references the security audit finding F1 — explanation of why this is bad. Plan dir `260518-1019-iam-least-privilege/` provides full history. |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Docs-only phase; no AWS state changes.
|
||||
- Preserves the security work done in phases 1-4 by making the new bootstrap discoverable to future maintainers.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Plan complete. After Phase 5 lands:
|
||||
- `/ck:journal` — write a session journal entry recording the cutover lesson (in particular, the dual-attach-strategy investigation in Phase 4).
|
||||
- Archive the plan with `/ck:plan archive`.
|
||||
- Address remaining audit findings (F3, F4, F5-F16) — out of scope here.
|
||||
@@ -1,142 +0,0 @@
|
||||
---
|
||||
title: "IAM least-privilege + OIDC trust narrowing (F1, F2)"
|
||||
description: "Replace 10× *FullAccess managed policies on github-deploy-miti99bot with a single stack-scoped custom inline policy; remove pull_request claim from OIDC trust."
|
||||
status: pending
|
||||
priority: P1
|
||||
branch: "main"
|
||||
tags: [security, iam, deploy]
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: "2026-05-18T09:06:20.846Z"
|
||||
createdBy: "ck:plan"
|
||||
source: skill
|
||||
---
|
||||
|
||||
# IAM least-privilege + OIDC trust narrowing (F1, F2)
|
||||
|
||||
## Overview
|
||||
|
||||
Two HIGH-severity findings from the 2026-05-18 security audit. Both target the
|
||||
GitHub Actions OIDC deploy role `github-deploy-miti99bot`.
|
||||
|
||||
Defence-in-depth framing (revised after red-team review):
|
||||
- F2 today is dormant — no PR-trigger workflow currently has `id-token: write` (verified: `.github/workflows/ci.yml` has `permissions: contents: read` only; `deploy.yml` is the only OIDC consumer and triggers on `push: main` + `workflow_dispatch`). A future workflow addition would make it live. Removing the claim closes the latent path.
|
||||
- F1 is the bigger lever: combined with the dormant F2 path, the 10× `*FullAccess` set (incl. `IAMFullAccess`) means any future OIDC-loosening + workflow compromise = account takeover.
|
||||
|
||||
- **F2 (trivial, 1-line):** drop `repo:tiennm99/miti99bot:pull_request` (and `:ref:refs/heads/dev` if unused) from the OIDC trust `sub` allowlist.
|
||||
- **F1 (careful):** replace 10× `*FullAccess` managed policies with one stack-scoped inline custom policy. Must enumerate every IAM action `sam deploy` actually invokes for every CFN resource in `template.yaml` — missing one = pipeline broken on next push.
|
||||
|
||||
## References
|
||||
|
||||
- `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` — finding details (F1, F2)
|
||||
- `aws/iam-github-oidc-trust.json` — current trust policy
|
||||
- `aws/README.md` step 4 — current broad-policy provisioning loop (to be replaced)
|
||||
- `.github/workflows/deploy.yml` — ground truth for what the role needs
|
||||
- `template.yaml` — every CFN resource sam deploy manages
|
||||
- `docs/deploy-aws-free-tier-guide.md:11-37` — accepted security trade-off envelope
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status | Risk |
|
||||
|-------|------|--------|------|
|
||||
| 1 | [Narrow OIDC trust (F2)](./phase-01-narrow-oidc-trust-f2.md) | Pending | Low — JSON edit + 1 aws-iam call; rollback via `git show HEAD^:aws/iam-github-oidc-trust.json` (RT-8) |
|
||||
| 2 | [Discover required actions](./phase-02-discover-required-actions.md) | Pending | Low — read-only enumeration; output is a documented action × resource table |
|
||||
| 3 | [Draft custom policy](./phase-03-draft-custom-policy.md) | Pending | Low — file creation + JSON validation; no AWS calls |
|
||||
| 4 | [Cutover + validate](./phase-04-cutover-validate.md) | Pending | **High** — wrong policy = pipeline locked. Two-stage (4a dual-attach trial + 4b cutover) with committed rollback script + workflow-disable gate + `ContinueUpdateRollback` recovery (RT-3, RT-8, RT-9). |
|
||||
| 5 | [Update bootstrap docs](./phase-05-update-bootstrap-docs.md) | Pending | Low — `aws/README.md` only |
|
||||
|
||||
Phase 1 is independent of 2-5 and can land standalone.
|
||||
Phases 2 → 3 → 4 → 5 are strictly sequential.
|
||||
|
||||
## Constraints (locked from project memory)
|
||||
|
||||
- **Free tier hard:** no Secrets Manager, no KMS CMK, no Config rules, no IAM Access Analyzer (paid features). Use IAM Policy Simulator only (free).
|
||||
- **Security envelope:** secret-in-logs / secret-in-Input acceptable; designed public surface = Function URL only; documented at `docs/deploy-aws-free-tier-guide.md:11-37`.
|
||||
- **Bootstrap chicken-and-egg:** F1 + F2 modify the very role the pipeline uses. All IAM mutations must be applied out-of-band (maintainer local creds with the original `admin` profile or AWS Console), NOT via the workflow being modified.
|
||||
|
||||
## Dependencies
|
||||
|
||||
None — both findings are repo-internal. F2 has no upstream / downstream dependency.
|
||||
|
||||
## Red Team Review
|
||||
|
||||
### Session — 2026-05-18
|
||||
|
||||
**Findings:** 15 of 30 surviving deduplication (10 accepted-applied, 5 cut as duplicate-of-fix or speculative)
|
||||
**Severity breakdown:** 4 Critical · 8 High · 3 Medium
|
||||
**Reviewers:** Security Adversary · Failure Mode Analyst · Assumption Destroyer
|
||||
|
||||
| # | Sev | Finding | Disposition | Applied To |
|
||||
|---|---|---|---|---|
|
||||
| 1 | CRIT | Phase 2 inventory placeholders + Phase 3 designs against TBD | Accept | Phase 2, Phase 3 (rewrites) |
|
||||
| 2 | CRIT | `iam:UpdateAssumeRolePolicy` enables trust-rewrite escalation | Accept | Phase 3 (action dropped) |
|
||||
| 3 | CRIT | UPDATE_ROLLBACK_FAILED unrecoverable; `cloudformation:ContinueUpdateRollback` missing; dual-attach trial rejected too early | Accept | Phase 3 + Phase 4 (re-architect) |
|
||||
| 4 | CRIT | `--profile admin` everywhere conflicts with `aws/README.md:119` "delete admin keys" | Accept | Phase 1 + 4 + 5 (admin-gate + console fallback) |
|
||||
| 5 | HIGH | Audit report file `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` did not exist | Accept | Audit file written; references valid |
|
||||
| 6 | HIGH | `iam:AttachRolePolicy` without `iam:PolicyARN` Condition → admin-policy attach escalation | Accept | Phase 3 (Condition added or action dropped) |
|
||||
| 7 | HIGH | `iam:PassedToService` `StringEquals` brittle; CFN-internals + future services not covered | Accept | Phase 3 (empirical verification + extensibility note) |
|
||||
| 8 | HIGH | Rollback script in `/tmp` not repo; `set -e` aborts mid-loop on throttle | Accept | Phase 4 (script committed at `aws/iam-rollback-fullaccess.sh` with retry + verify) |
|
||||
| 9 | HIGH | `concurrency: deploy-prod` does not gate external IAM mutations | Accept | Phase 4 (workflow-disable during cutover) |
|
||||
| 10 | HIGH | `*:TagResource` / `*:UntagResource` / `*:ListTagsForResource` not enumerated | Accept | Phase 2 (mandatory categories) + Phase 3 (added) |
|
||||
| 11 | HIGH | Schedule ARN `schedule/default/miti99bot-*` depends on undocumented default-group folklore | Accept | Phase 3 (`schedule/*/miti99bot-*`) |
|
||||
| 12 | HIGH | Drift `diff` produces false positives — AWS normalizes JSON server-side | Accept | Phase 5 (`jq -S` structural compare) |
|
||||
| 13 | MED | SAM bucket prefix is convention not contract; bucket-bootstrap actions missing | Accept | Phase 2 (discovery step) + Phase 3 (broader S3 actions) |
|
||||
| 14 | MED | Stack ARN hardcodes `miti99bot` literal — future `miti99bot-dev` locked out | Accept | Phase 3 (`miti99bot*` globs) |
|
||||
| 15 | MED | Dropping `refs/heads/dev` without re-add procedure | Accept | Phase 5 ("Trust policy invariants" section) |
|
||||
|
||||
**Cut as duplicate-of-fix or speculative:**
|
||||
- Function URL config actions (subsumed by Finding 1 inventory rewrite)
|
||||
- F2 threat narrative inflation (addressed by Overview reframe above)
|
||||
- Cross-account layer assumption (low actionable impact)
|
||||
- `CAPABILITY_NAMED_IAM` future need (speculative forward-look)
|
||||
|
||||
**Reports written:**
|
||||
- `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` (was missing; produced from inline audit content)
|
||||
|
||||
### Whole-Plan Consistency Sweep — 2026-05-18
|
||||
|
||||
Re-read `plan.md` + 5 phase files after edits. Reconciled:
|
||||
|
||||
- ✅ ARN patterns match across Phase 2 (discovery) and Phase 3 (policy skeleton): `stack/miti99bot*/*`, `table/miti99bot*`, `function:miti99bot*`, `schedule/*/miti99bot*`, `role/miti99bot*`, `parameter/miti99bot/*/*`.
|
||||
- ✅ Phase 1 step 0 (admin pre-flight, RT-4) reflected in Todo List + Risk Assessment.
|
||||
- ✅ Phase 4 two-stage restructure (4a trial + 4b cutover) reflected in Implementation Steps, Todo List, Success Criteria, plan.md risk column.
|
||||
- ✅ Audit report `plans/reports/code-reviewer-260518-1019-security-aws-infra.md` exists; Phase 5 Success Criteria references it.
|
||||
- ✅ `aws/iam-rollback-fullaccess.sh` (committed script, RT-8) referenced consistently across Phase 4 step 1, Phase 4 step 8, Success Criteria.
|
||||
- ✅ `jq -S` structural-diff (RT-12) replaces `diff` in both phase doc and Success Criteria.
|
||||
- ✅ `iam:UpdateAssumeRolePolicy` (RT-2) and `iam:AttachRolePolicy` (RT-6) marked as deliberately-excluded in Phase 2 + Phase 3, with rationale for re-adding.
|
||||
- ✅ "F2 dormant today, defence-in-depth fix" framing (RT cuts) in plan.md Overview matches Phase 1 step 6 rewrite.
|
||||
|
||||
**Unresolved contradictions:** none. Plan ready for implementation.
|
||||
|
||||
## Validation Log
|
||||
|
||||
### Session — 2026-05-18 (post-red-team)
|
||||
|
||||
Critical-questions interview after red-team. 4 questions; 4 decisions recorded.
|
||||
|
||||
| # | Question | Decision | Applied To |
|
||||
|---|---|---|---|
|
||||
| V-1 | Phase 1 dev-branch: does user push to `dev` from local? | **No, never push to dev** → drop `refs/heads/dev` from OIDC trust as Phase 1 already proposes. Decisive narrowing. | Phase 1 (already reflected) |
|
||||
| V-2 | Phase 4 admin pre-flight: add multi-item preconditions checklist? | **No** — "make this workflow simple, just work first, then we will solve problems later." Existing single Step 0 (`aws sts get-caller-identity`) is enough. Don't add MFA/network/console-access checklist. | Phase 4 (no change — minimal step 0 retained) |
|
||||
| V-3 | Phase 4a trial: make CloudTrail coverage check mandatory? | **No** — same simplicity preference as V-2. Stays optional. Dual-attach trial succeeding + cutover deploy succeeding are the two coverage signals. | Phase 4 step 3c (no change — stays "optional but recommended") |
|
||||
| V-4 | Phase 1: commit `iam-github-oidc-trust.json` edit BEFORE or AFTER `aws iam update-assume-role-policy`? | **Commit FIRST, then apply.** Repo is source of truth; `git show HEAD^:...` always recovers previous state. AWS/repo drift avoided. | Phase 1 step 3 + 4 + 7 (commit step folded into apply step) |
|
||||
|
||||
**Memory captured:** [[simplicity-over-defensive-checklists]] — durable preference for minimum workflow on ops/deploy plans for this project.
|
||||
|
||||
### Whole-Plan Consistency Sweep (validation)
|
||||
|
||||
After V-4 edit:
|
||||
- ✅ Phase 1 step 3 now includes commit-before-edit guidance + rationale.
|
||||
- ✅ Phase 1 step 4 includes both `git commit` and `aws iam update-assume-role-policy` together.
|
||||
- ✅ Phase 1 step 7 redirected to step 4 (no double-commit).
|
||||
- ✅ Todo List unchanged — "Commit JSON edit" item still maps to step 4 (just consolidated, not removed).
|
||||
- ✅ Plan.md risk column for Phase 1 still accurate: "rollback via `git show HEAD^:aws/iam-github-oidc-trust.json`" works because the commit is on `HEAD`.
|
||||
|
||||
**Unresolved contradictions:** none. Plan ready for implementation.
|
||||
|
||||
## Non-goals (explicit cuts)
|
||||
|
||||
- F3 (CORS), F4 (root handler audit), F5-F16 — captured in audit report, separate fixes.
|
||||
- Moving secrets to Secrets Manager — violates free-tier rule.
|
||||
- Adding `govulncheck` to CI — separate hygiene work.
|
||||
- Rotating the existing CronSharedSecret — out of scope.
|
||||
-141
@@ -1,141 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: Implement deploynotify package + main.go hook
|
||||
status: completed
|
||||
priority: P3
|
||||
effort: 1h
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Implement deploynotify package + main.go hook
|
||||
|
||||
## Overview
|
||||
|
||||
Create `internal/deploynotify/` — a small package that compares the
|
||||
baked-in `gitSHA` against a `last_notified_sha` value in KV and DMs the
|
||||
bot owner if (and only if) it changed. Wire it from `cmd/server/main.go`
|
||||
right after `modules.Install`.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Send exactly one Telegram DM to `BOT_OWNER_ID` per *new* gitSHA observed.
|
||||
- On subsequent cold starts with the same SHA, send nothing.
|
||||
- Skip silently when: `gitSHA` empty (local build), `BOT_OWNER_ID == 0`,
|
||||
or KV operation fails.
|
||||
|
||||
**Non-functional**
|
||||
- Never panic; never return an error that aborts startup.
|
||||
- ≤3s wall time on the happy path (network DM + 1 KV read + 1 KV write).
|
||||
- No new env vars, no new IAM permissions (DynamoDB read/write already
|
||||
granted to the partition).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
cmd/server/main.go
|
||||
│
|
||||
│ after modules.Install(b, reg, auth):
|
||||
├─→ deploynotify.Run(ctx, deploynotify.Config{
|
||||
│ Bot: b,
|
||||
│ KV: provider.For("deploynotify"),
|
||||
│ OwnerID: cfg.BotOwnerID,
|
||||
│ GitSHA: gitSHA, // package-level var, ldflags-injected
|
||||
│ Timeout: 3 * time.Second,
|
||||
│ })
|
||||
│ │
|
||||
│ ├─ skipReason() short-circuit (no SHA / no owner)
|
||||
│ ├─ kv.GetJSON("last_notified_sha", &prev)
|
||||
│ ├─ if prev.SHA == gitSHA → return (silent)
|
||||
│ ├─ bot.SendMessage(owner, "🚀 miti99bot deployed: <code>SHA</code>")
|
||||
│ └─ kv.PutJSON("last_notified_sha", {SHA: gitSHA, At: now})
|
||||
```
|
||||
|
||||
KV namespace: `deploynotify` (new partition, isolated from module data).
|
||||
Key: `last_notified_sha`.
|
||||
Value shape:
|
||||
```go
|
||||
type notifyRecord struct {
|
||||
SHA string `json:"sha"`
|
||||
At int64 `json:"at"` // ms-since-epoch, for debug only
|
||||
}
|
||||
```
|
||||
|
||||
Telegram message (plain text — no parse_mode dependency on formatting
|
||||
edge cases):
|
||||
```
|
||||
🚀 miti99bot deployed: <SHORT_SHA>
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/deploynotify/deploy_notify.go` (snake_case per Go conventions)
|
||||
- Create: `internal/deploynotify/deploy_notify_test.go`
|
||||
- Modify: `cmd/server/main.go` — declare `var gitSHA string`, add
|
||||
`deploynotify.Run(...)` call after `modules.Install(b, reg, auth)` and
|
||||
before the `go func() { srv.ListenAndServe() }()` block.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Create `internal/deploynotify/deploy_notify.go`** with:
|
||||
- `type Config struct { Bot *bot.Bot; KV storage.KVStore; OwnerID int64; GitSHA, Timeout }`.
|
||||
- `func Run(ctx context.Context, cfg Config)` — fire-and-forget, no
|
||||
error return; all failures logged via `internal/log`.
|
||||
- Internal helper `shouldNotify(ctx, kv, sha) (bool, error)` so dedup
|
||||
is unit-testable without a real Telegram bot.
|
||||
- Internal helper `markNotified(ctx, kv, sha) error`.
|
||||
- Internal `renderMessage(sha string) string` — single line, easy to test.
|
||||
|
||||
2. **Wire into `cmd/server/main.go`**:
|
||||
- Add `var gitSHA string` at package level (alongside `factories()`).
|
||||
- After `modules.Install(b, reg, auth)` and the existing `log.Info("modules loaded", ...)`:
|
||||
```go
|
||||
deploynotify.Run(rootCtx, deploynotify.Config{
|
||||
Bot: b,
|
||||
KV: provider.For("deploynotify"),
|
||||
OwnerID: cfg.BotOwnerID,
|
||||
GitSHA: gitSHA,
|
||||
Timeout: 3 * time.Second,
|
||||
})
|
||||
```
|
||||
- Import: `"github.com/tiennm99/miti99bot/internal/deploynotify"`.
|
||||
|
||||
3. **Tests** (`deploy_notify_test.go`):
|
||||
- `TestShouldNotify_FirstRun` — empty KV → returns true.
|
||||
- `TestShouldNotify_SameSHA` — KV holds current SHA → returns false.
|
||||
- `TestShouldNotify_DifferentSHA` — KV holds old SHA → returns true.
|
||||
- `TestRun_SkipsWhenSHAEmpty` — gitSHA="" → no KV access, no send.
|
||||
- `TestRun_SkipsWhenNoOwner` — OwnerID=0 → no KV access, no send.
|
||||
- `TestRenderMessage_ContainsSHA` — output includes the SHA.
|
||||
- Use `storage.NewMemoryKVStore()` for KV.
|
||||
- For the Telegram send path: skip end-to-end Telegram tests — the
|
||||
existing `testutil.RecordingBot` pattern is heavier than needed.
|
||||
Cover send via an indirection: `Config.sender` field of type
|
||||
`func(ctx, chatID, text) error` defaulting to `b.SendMessage`
|
||||
wrapper. Tests inject a recorder.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `go build ./...` succeeds.
|
||||
- [ ] `go test ./internal/deploynotify/...` passes.
|
||||
- [ ] `go test ./...` passes (no regressions).
|
||||
- [ ] `cmd/server/main.go` still under reasonable size; deploynotify call
|
||||
adds ≤6 LOC.
|
||||
- [ ] Manual review: a `Run` invocation with empty SHA touches neither KV
|
||||
nor Telegram (traceable in code, not just behaviour).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Cold-start storm sends duplicate DMs (2+ instances boot concurrently after deploy) | Documented in plan Out-of-Scope; race window narrow; failure mode is annoyance not corruption. |
|
||||
| KV write succeeds, Telegram send fails | Order is reversed: send first, write only on send success. So a failed send doesn't permanently silence retries. |
|
||||
| KV read fails (DynamoDB throttle) | Treat as "not notified yet" → fall through to send. Worst case: extra DM. |
|
||||
| Test for Run-with-bot leaks a goroutine | Run is synchronous (uses Timeout via context.WithTimeout). No goroutine spawned. |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Owner ID is already in env; no new secret material.
|
||||
- Telegram message body contains only the short git SHA — public
|
||||
information (the repo is public). No env/secrets leak risk.
|
||||
- KV partition `deploynotify` is read/write under existing IAM scope.
|
||||
@@ -1,98 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: Wire git SHA into build
|
||||
status: completed
|
||||
priority: P3
|
||||
effort: 20m
|
||||
dependencies:
|
||||
- 1
|
||||
---
|
||||
|
||||
# Phase 2: Wire git SHA into build
|
||||
|
||||
## Overview
|
||||
|
||||
Inject the short git SHA into the binary at link time via
|
||||
`-X main.gitSHA=…`. Without this, Phase 1's `Run` short-circuits and the
|
||||
feature is dormant. Touches Makefile only; GitHub Actions already runs
|
||||
`make build-lambda`, so no workflow change is required.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- `make build-lambda` produces a binary whose `main.gitSHA` equals
|
||||
`git rev-parse --short HEAD` at build time.
|
||||
- `make build` (local host binary) does the same — useful for dogfooding.
|
||||
- Both targets degrade gracefully if `git` is unavailable: empty SHA →
|
||||
Phase 1's `Run` silently skips.
|
||||
|
||||
**Non-functional**
|
||||
- No new tools or actions added to CI.
|
||||
- `actions/checkout@v6` default depth (shallow) must support
|
||||
`git rev-parse --short HEAD` — it does, HEAD is always present.
|
||||
|
||||
## Architecture
|
||||
|
||||
```makefile
|
||||
GIT_SHA := $(shell git rev-parse --short HEAD 2>/dev/null)
|
||||
|
||||
LDFLAGS := -s -w -X main.gitSHA=$(GIT_SHA)
|
||||
|
||||
build:
|
||||
CGO_ENABLED=0 go build -ldflags="$(LDFLAGS)" -o ./bin/server ./cmd/server
|
||||
|
||||
build-lambda:
|
||||
@mkdir -p $(dir $(LAMBDA_OUT))
|
||||
CGO_ENABLED=0 GOOS=$(LAMBDA_GOOS) GOARCH=$(LAMBDA_GOARCH) \
|
||||
go build -tags lambda.norpc -ldflags="$(LDFLAGS)" \
|
||||
-o $(LAMBDA_OUT) ./cmd/server
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `Makefile` — add `GIT_SHA` + `LDFLAGS` vars, swap inline
|
||||
`-ldflags="-s -w"` for `-ldflags="$(LDFLAGS)"` in both `build` and
|
||||
`build-lambda` targets.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Add Makefile variables** near the top (after existing `LAMBDA_OUT` etc):
|
||||
```makefile
|
||||
GIT_SHA := $(shell git rev-parse --short HEAD 2>/dev/null)
|
||||
LDFLAGS := -s -w -X main.gitSHA=$(GIT_SHA)
|
||||
```
|
||||
|
||||
2. **Update `build` target** (`Makefile:51-52`):
|
||||
- Change `-ldflags="-s -w"` → `-ldflags="$(LDFLAGS)"`.
|
||||
|
||||
3. **Update `build-lambda` target** (`Makefile:54-60`):
|
||||
- Change `-ldflags="-s -w"` → `-ldflags="$(LDFLAGS)"`.
|
||||
|
||||
4. **Verify**:
|
||||
- `make build` then `strings ./bin/server | grep -E '^[0-9a-f]{7,}$'`
|
||||
should reveal the SHA.
|
||||
- Or: add a no-op `--version`-style log line behind a build flag —
|
||||
skip for now (YAGNI).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `make build` builds successfully.
|
||||
- [ ] `make build-lambda` builds successfully.
|
||||
- [ ] Resulting binary has `main.gitSHA` populated (verified once via
|
||||
`strings` grep or a temporary debug log — not a permanent test).
|
||||
- [ ] No change to `.github/workflows/deploy.yml` — `make build-lambda`
|
||||
in CI picks up the new ldflags automatically.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Dockerfile builds (CI `docker build -t miti99bot`) bypass Makefile and won't inject SHA | Out of scope — Lambda deploy uses `make build-lambda`, not Docker. CI's `docker build` is just a smoke check. Document the gap; revisit only if Cloud Run path is reactivated. |
|
||||
| Local `make build` in a tarball download with no `.git/` → SHA empty | Acceptable — feature silently disables on non-git builds. |
|
||||
| `git rev-parse` outputs spaces / unexpected chars → breaks ldflags | `git rev-parse --short HEAD` output is `[0-9a-f]{7,}` only. Safe. |
|
||||
| Reproducible builds tooling complains about non-deterministic SHA | Not relevant for this project. |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Short git SHA is published on every GitHub commit page — not sensitive.
|
||||
- No build-time secrets touched.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
title: Deploy notification to bot owner with git SHA
|
||||
description: >-
|
||||
On startup the bot DMs BOT_OWNER_ID with the deployed git SHA. Dedup via
|
||||
DynamoDB so only real new versions notify, not every Lambda cold start.
|
||||
status: completed
|
||||
priority: P3
|
||||
branch: main
|
||||
tags:
|
||||
- ops
|
||||
- observability
|
||||
- telegram
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-05-22T04:10:07.522Z'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Deploy notification to bot owner with git SHA
|
||||
|
||||
## Overview
|
||||
|
||||
Operator awareness of deploys. After `make build-lambda` + `sam deploy`, the
|
||||
owner currently has no in-Telegram signal that the new code is running on
|
||||
Lambda (only the GitHub Actions log). Add a startup hook that DMs the owner
|
||||
with the baked-in short git SHA, deduped by KV so subsequent cold starts of
|
||||
the same version stay silent.
|
||||
|
||||
Confirmation that the **new code is running**, not just that the deploy
|
||||
script finished — that's why this lives in the bot binary, not in the deploy
|
||||
workflow.
|
||||
|
||||
## Design Decisions (locked)
|
||||
|
||||
- **Dedup**: DynamoDB KV stores `last_notified_sha`. Send only when baked
|
||||
`gitSHA != stored`, then write. One `GetItem` per cold start (~free tier).
|
||||
- **Code placement**: new `internal/deploynotify/` package — testable in
|
||||
isolation, ~50 LOC, main.go just calls `deploynotify.Run(...)`.
|
||||
- **Build wiring**: `-ldflags "-X main.gitSHA=<short-sha>"` in Makefile.
|
||||
Empty `gitSHA` (local non-make build) → silently skip.
|
||||
- **Failure policy**: log + continue. Never block server startup. KV error,
|
||||
Telegram error, missing owner — all non-fatal.
|
||||
- **Timing**: synchronous, ≤3s timeout, runs after `modules.Install` and
|
||||
before `srv.ListenAndServe()`. Lambda init phase has 10s headroom.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Implement deploynotify package + main.go hook](./phase-01-implement-deploynotify-package-main-go-hook.md) | Completed |
|
||||
| 2 | [Wire git SHA into build](./phase-02-wire-git-sha-into-build.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
None.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Multi-environment routing (prod vs staging) — single owner, single env today.
|
||||
- Notify on rollback — same dedup mechanism naturally handles it; rollback to
|
||||
a previously-notified SHA just resends because stored value moved forward.
|
||||
Acceptable.
|
||||
- Conditional KV write to prevent concurrent-cold-start dupes — KV interface
|
||||
has no CAS today; the race window is narrow and the failure mode is
|
||||
"two identical DMs", not data corruption.
|
||||
-62
@@ -1,62 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: Hook signature + per-user write path
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 1h
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Hook signature + per-user write path
|
||||
|
||||
## Overview
|
||||
|
||||
Pass the Telegram update into the `CommandHook` so the stats module can attribute invocations to users, and extend `stats.counter.Inc` to write the new per-user and per-pair keys.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: every authorized command invocation increments `count:<cmd>`, `user:<userID>`, and `pair:<cmd>:<userID>`. Username field on `user:<userID>` is refreshed each call.
|
||||
- Functional: when `From.Username` is empty, skip the `user:`/`pair:` writes; `count:<cmd>` still increments.
|
||||
- Non-functional: hook still runs detached from request context (2s timeout, see `internal/modules/dispatcher.go:73`). Existing race on read-modify-write is accepted.
|
||||
- Non-functional: write fan-out stays within the hook's 2s budget. Three writes execute concurrently.
|
||||
|
||||
## Architecture
|
||||
|
||||
`CommandHook` is the only cross-module hook that needs to evolve. Today's signature `func(ctx context.Context, name string)` drops sender info before stats sees it. New signature `func(ctx context.Context, name string, update *models.Update)` is minimal, idiomatic, and only one current implementation (stats) needs to adapt.
|
||||
|
||||
Write fan-out lives inside `counter.Inc` and uses a `sync.WaitGroup` of three goroutines mirroring the existing render-side pattern at `internal/modules/stats/stats.go:94-109`. Errors are logged and swallowed (best-effort, same as today).
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/module.go` — update `CommandHook` type.
|
||||
- Modify: `internal/modules/dispatcher.go` — pass `update` to `RunCommandHooks`.
|
||||
- Modify: `internal/modules/registry.go` — adjust `RunCommandHooks` signature and storage.
|
||||
- Modify: `internal/modules/stats/stats.go` — update `Inc` signature, add per-user write fan-out, change `countEntry` and add `userEntry`.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. In `module.go`, change `CommandHook` type to `func(ctx context.Context, name string, update *models.Update)`. Update the field comment.
|
||||
2. In `registry.go`, locate `RunCommandHooks`. Add `update *models.Update` parameter and forward to each registered hook.
|
||||
3. In `dispatcher.go:73-76`, pass `update` into `reg.RunCommandHooks(hookCtx, cmdCopy.Name, update)`. The goroutine already captures `update` in scope.
|
||||
4. In `stats.go`:
|
||||
- Define `userEntry struct { Username string \`json:"username"\`; N int64 \`json:"n"\` }`.
|
||||
- Add helpers `userKey(id int64) string` returning `"user:" + strconv.FormatInt(id, 10)` and `pairKey(cmd string, id int64) string` returning `"pair:" + cmd + ":" + strconv.FormatInt(id, 10)`.
|
||||
- Change `Inc` signature to `func (c *counter) Inc(ctx context.Context, name string, update *models.Update)`.
|
||||
- Inside `Inc`: keep the existing `count:<cmd>` increment. If `update != nil && update.Message != nil && update.Message.From != nil && update.Message.From.Username != ""`, fan out two more increments under a single `sync.WaitGroup`:
|
||||
- `user:<id>` — GetJSON, set `Username` to current value, `++N`, PutJSON.
|
||||
- `pair:<cmd>:<id>` — GetJSON, `++N`, PutJSON.
|
||||
- Wrap the `count:` increment in the same fan-out so all three writes run in parallel.
|
||||
5. Run `make vet test` from repo root. No new test deps in this phase (tests in Phase 3).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `make vet` clean.
|
||||
- [ ] `make test` passes (existing stats tests will need a minor signature update — fix as part of this phase or defer to Phase 3 if straightforward).
|
||||
- [ ] `go build ./...` produces no errors.
|
||||
- [ ] Manual trace: handler invocation in dispatcher passes update through to `stats.Inc`; debug log line in `Inc` confirms user attribution path.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** Existing `internal/modules/stats/stats_test.go` constructs the counter directly and may call `Inc(ctx, "name")`. Fix call sites in the same phase.
|
||||
- **Risk:** Other modules might register a `CommandHook` in the future and break. Mitigation: search the codebase for `CommandHook:` assignments before merging — currently stats is the only user.
|
||||
- **Risk:** Concurrent invocations of the same command by the same user lose updates (read-modify-write race). Accepted, documented at `stats.go:35-37`.
|
||||
-68
@@ -1,68 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: Subcommand parser + new /stats views
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 2h
|
||||
dependencies:
|
||||
- 1
|
||||
---
|
||||
|
||||
# Phase 2: Subcommand parser + new /stats views
|
||||
|
||||
## Overview
|
||||
|
||||
Extend the existing `/stats` handler to dispatch on the first whitespace-separated token after the command (e.g. `/stats users`). Implement three new views: `users`, `user <username>`, `cmd <name>`. Bare `/stats` keeps its current behaviour (top commands).
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `/stats` (no args) → top 20 commands by count. Unchanged.
|
||||
- Functional: `/stats users` → top 20 users by total invocations. Lines `@<username>: <n>`.
|
||||
- Functional: `/stats user <username>` → top 20 commands invoked by that user. Accept with or without leading `@`. Reply `User @<username> not found.` if unknown.
|
||||
- Functional: `/stats cmd <name>` → top 20 users who invoked `<name>`. Reply `Command <name> not found or has no users.` if no `pair:<name>:*` rows.
|
||||
- Functional: unknown subcommand → reply `Usage:\n/stats\n/stats users\n/stats user <username>\n/stats cmd <name>`.
|
||||
- Non-functional: every view fans out GetItem reads (existing pattern at `stats.go:94-109`). No sequential per-key reads.
|
||||
- Non-functional: 4000-char truncation kept (`stats.go:129-137`). Top-K cap = 20 entries.
|
||||
|
||||
## Architecture
|
||||
|
||||
The handler reads `update.Message.Text`, strips the command entity using the same `@botname`-aware logic used in `dispatcher.matchCommand`, and trims leading whitespace. The remainder is `subargs`. `strings.Fields(subargs)` gives the subcommand token + tail.
|
||||
|
||||
Renders share one helper that takes `[]row` (name + n) and returns the truncated reply string — DRYs the four views.
|
||||
|
||||
Username → userID resolution: List the `user:` prefix, fan-out GetItem, linear scan for matching `Username`. Worst case is `len(users)` GetItems — same cost as `/stats users`, well under the 10s webhook deadline for any realistic free-tier user count.
|
||||
|
||||
`/stats cmd <name>` lists `pair:<name>:`. Each row's sort key is `pair:<name>:<id>`; userID extracted by trimming the prefix. For each such userID, fetch `user:<id>` in parallel to resolve username. Skip rows whose `user:<id>` lookup returns ErrNotFound or empty Username (defensive: pair without user record means race or pre-existing data).
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/stats/stats.go` — extend `statsCommand` handler, add view helpers.
|
||||
|
||||
No new files. Total addition expected ~150 lines, keeping `stats.go` under the 200-line guideline becomes tight — if it crosses, extract `stats.go` views into `internal/modules/stats/views.go`.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add helper `parseSubargs(update *models.Update, cmdName string) string` that mirrors `dispatcher.matchCommand`'s entity-stripping. Returns the trimmed remainder after the command token.
|
||||
2. Add helper `renderTopN(title string, rows []row, maxLen int) string` — folds the existing render+truncate logic. `row` gains an explicit `display string` field so users (`@username`) and commands (`/help`) render with the right prefix.
|
||||
3. Replace the body of the `statsCommand` Handler with a switch over the first token of `parseSubargs`:
|
||||
- `""` → existing top-commands path.
|
||||
- `"users"` → fetch `user:` rows, fan-out, sort desc by N, render with `display = "@" + username`. Skip rows with empty Username.
|
||||
- `"user"` → require one arg; resolve username → userID via `user:` listing; if not found, reply not-found. Otherwise list `pair:`, filter by `:<userID>` suffix, fan-out GetItem, sort desc, render with `display = "/" + cmd`.
|
||||
- `"cmd"` → require one arg; list `pair:<arg>:`, fan-out, resolve usernames via parallel `user:<id>` reads, sort desc, render with `display = "@" + username`.
|
||||
- Anything else → reply usage string.
|
||||
4. If `stats.go` exceeds ~200 lines, extract per-view helpers into `internal/modules/stats/views.go` (same package).
|
||||
5. Run `make vet test` and a local end-to-end smoke against an in-memory KV (`MODULES=stats go run ./cmd/server` against ngrok or a manual fake update fixture if available).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `/stats` returns the existing top-commands view byte-for-byte (regression check via test fixture).
|
||||
- [ ] `/stats users`, `/stats user <name>`, `/stats cmd <name>` each return correctly sorted, truncated output.
|
||||
- [ ] Unknown subcommand returns the usage string.
|
||||
- [ ] `/stats user <unknown>` and `/stats cmd <unknown>` return clear not-found messages.
|
||||
- [ ] Handler completes within 2s wall-clock on local in-memory KV; budget for DynamoDB is generous given fan-out.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** `List("pair:")` is unused — `cmd` view only lists `pair:<name>:` so prefix is bounded. `user` view derives userID then filters; if `pair:` grows large, this becomes O(commands × users). Mitigation: top-K is 20 per view; sort handles the cap. If perf degrades, add a `byuser:<id>:<cmd>` mirror key in a follow-up.
|
||||
- **Risk:** Race between Phase 1 writes and Phase 2 reads (eventually-consistent reads default in DDB). `DynamoDBKVStore.Get` uses `ConsistentRead: true` (see `dynamodb_kv.go:49`), so reads are strong. No risk.
|
||||
- **Risk:** A user changes their Telegram username between the `Inc` write and the `/stats users` render — stale display. Acceptable; refresh on next invocation.
|
||||
@@ -1,62 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: Tests
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 1h
|
||||
dependencies:
|
||||
- 1
|
||||
- 2
|
||||
---
|
||||
|
||||
# Phase 3: Tests
|
||||
|
||||
## Overview
|
||||
|
||||
Add unit tests for the new Inc fan-out and each subcommand view, using the in-memory KV store. Keep tests deterministic; no DynamoDB Local required for module-level tests.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Cover `Inc` writing all three keys when username is non-empty.
|
||||
- Cover `Inc` skipping `user:`/`pair:` writes when username is empty but still writing `count:`.
|
||||
- Cover each subcommand view: bare, users, user, cmd, unknown, not-found.
|
||||
- Cover the 4000-char truncation path with a synthetic dataset.
|
||||
- Tests must pass with `make test` (no DynamoDB needed); `make test-dynamodb` continues to pass for storage layer.
|
||||
|
||||
## Architecture
|
||||
|
||||
`internal/storage/memory_kv.go` provides `*MemoryKV` implementing `KVStore`. Tests instantiate a `counter{kv: NewMemoryKV()}` and a fake `*models.Update`. Use `t.Run` subtests, one per view.
|
||||
|
||||
For subcommand parsing, prefer driving the public Handler with a constructed `*models.Update` rather than testing the internal parser in isolation — the parser interaction with entity offsets is the riskier surface.
|
||||
|
||||
The Telegram bot send-message side effect is hard to assert directly. Two options:
|
||||
- (a) Refactor the Handler to return the rendered string, with `chathelper.Reply` as a thin wrapper. Lower-risk.
|
||||
- (b) Mock `bot.Bot.SendMessage`. Heavier.
|
||||
|
||||
Pick (a) if it's a small change; otherwise mock or skip the send and assert on the input data passed to `renderTopN`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/stats/stats_test.go` — add subtests.
|
||||
- (Optional) Modify: `internal/modules/stats/stats.go` — extract a pure `renderStatsReply(view, args, ...) string` if needed to make output assertable without touching `bot.Bot`.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Read current `internal/modules/stats/stats_test.go` to learn the existing fixture conventions.
|
||||
2. Add `TestCounterIncWritesAllThreeKeys` — invoke `Inc` with username, assert all three KV entries materialise with N==1; invoke twice, assert N==2.
|
||||
3. Add `TestCounterIncSkipsUserKeysWhenUsernameEmpty` — invoke with empty username, assert only `count:<cmd>` exists.
|
||||
4. Add a table-driven `TestStatsViews` with cases for: bare, `users`, `user <known>`, `user <unknown>`, `cmd <known>`, `cmd <unknown>`, `bogus`. Seed the KV with a known fixture, drive the Handler (or `renderStatsReply` per Architecture above), assert substring matches.
|
||||
5. Add `TestStatsViewTruncates` — seed 200 commands/users to push past 4000 chars; assert the reply ends with `…(truncated)`.
|
||||
6. Run `make vet test`. Iterate until green.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `make vet` clean.
|
||||
- [ ] `make test` passes including the new cases.
|
||||
- [ ] No flaky tests (re-run 3×).
|
||||
- [ ] Coverage on `stats.go` ≥ existing baseline.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** Refactoring `Reply` out of the handler to make output assertable touches the existing bare-stats path. Mitigation: keep `chathelper.Reply` call at the leaf; only the string-building moves.
|
||||
- **Risk:** `*models.Update` is non-trivial to construct. Mitigation: copy an existing fixture from `dispatcher_test.go` if one exists; otherwise build the minimal subset (Message.From.ID/Username, Message.Text, Message.Entities).
|
||||
@@ -1,49 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: Command menu
|
||||
status: completed
|
||||
priority: P3
|
||||
effort: 10m
|
||||
dependencies:
|
||||
- 2
|
||||
---
|
||||
|
||||
# Phase 4: Command menu
|
||||
|
||||
## Overview
|
||||
|
||||
Telegram's `setMyCommands` registers the command auto-complete shown in the input field. `/stats` is already registered (commit `db8ee9c`). Subcommands are not — clients won't auto-suggest `/stats users` etc. Decide whether to (a) leave the menu alone (subcommands are discoverable via the usage string), or (b) add the four forms.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `/stats` remains in `aws/telegram-commands.json`.
|
||||
- Optional: add `stats_users`, `stats_user`, `stats_cmd` *or* document subcommands in the existing `/stats` description.
|
||||
|
||||
## Architecture
|
||||
|
||||
Two paths:
|
||||
|
||||
1. **Single entry, updated description.** Edit the existing `/stats` description in `aws/telegram-commands.json` to read `Stats: /stats, /stats users, /stats user <name>, /stats cmd <name>`. Cheap, discoverable via the menu hover.
|
||||
2. **Multiple entries.** Add aliases that route into the same handler. Requires either (a) registering separate Telegram menu entries that point to the same `/stats *` invocation (Telegram doesn't enforce uniqueness; description is the only hint), or (b) creating real command aliases in code. (b) clutters `/help`.
|
||||
|
||||
Default: option 1.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `aws/telegram-commands.json` — update `/stats` description.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Read `aws/telegram-commands.json`.
|
||||
2. Update the `/stats` description field. New text suggestion: `Show stats. Try: /stats users, /stats user <name>, /stats cmd <name>`.
|
||||
3. The `Register Telegram command menu` step in `.github/workflows/deploy.yml:108-124` will push the update on next deploy. No code change required.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] JSON is valid (`jq . aws/telegram-commands.json` succeeds).
|
||||
- [ ] Description fits Telegram's 256-char limit per command.
|
||||
- [ ] On the next deploy, the bot's command menu hover text shows the subcommand hint.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk:** Description length limit. Mitigation: keep under 100 chars; the proposed text is ~70.
|
||||
@@ -1,48 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: Docs
|
||||
status: completed
|
||||
priority: P3
|
||||
effort: 15m
|
||||
dependencies:
|
||||
- 2
|
||||
---
|
||||
|
||||
# Phase 5: Docs
|
||||
|
||||
## Overview
|
||||
|
||||
Update repo docs so future readers understand the new schema and views without reading the diff.
|
||||
|
||||
## Requirements
|
||||
|
||||
- README module table mentions the extended `/stats` capabilities.
|
||||
- `docs/system-architecture.md` (or codebase summary) describes the per-user key schema.
|
||||
- `docs/project-changelog.md` records the feature.
|
||||
|
||||
## Architecture
|
||||
|
||||
Touch only the minimum doc surface; do not create new files. Per CLAUDE.md, docs live under `./docs/` and are kept current.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `README.md` — extend the `util`/`stats` module row.
|
||||
- Modify: `docs/system-architecture.md` (or closest equivalent — check actual file list).
|
||||
- Modify: `docs/project-changelog.md` — add an entry under today's date.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. `ls docs/` to confirm which of system-architecture / codebase-summary actually exist.
|
||||
2. README: extend the `stats` row in the modules table (currently doesn't appear in README — add a one-liner: `stats: /stats, /stats users, /stats user <name>, /stats cmd <name>`).
|
||||
3. system-architecture (or codebase-summary): add a short subsection under the Stats module describing the three sort-key shapes (`count:`, `user:`, `pair:`).
|
||||
4. project-changelog: add `## 2026-05-22` (or appropriate date) entry with feature summary.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] README documents the new subcommands.
|
||||
- [ ] One of the architecture docs lists the three storage keys.
|
||||
- [ ] Changelog has a dated entry.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Low. Doc-only.
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
title: Stats per-user analytics
|
||||
description: >-
|
||||
Extend /stats with per-user breakdowns: top users overall, per-user command
|
||||
history, per-command user ranking. Username-only display, all public,
|
||||
free-tier safe.
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: main
|
||||
tags:
|
||||
- stats
|
||||
- telegram
|
||||
- dynamodb
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-05-22T10:42:15.385Z'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Stats per-user analytics
|
||||
|
||||
## Overview
|
||||
|
||||
`/stats` today tracks `count:<cmd>` only — a single counter per command. Extend to capture per-user counts so the owner can see who uses the bot most, which command a given user runs most, and who uses a given command most. All views remain public per user decision (no admin gating). Display uses `@username` only; first_name is not stored. Users without a Telegram username are excluded from per-user listings but still increment the global command total.
|
||||
|
||||
## Scope decisions (user-confirmed)
|
||||
|
||||
- All `/stats *` subcommands are `VisibilityPublic`. No admin gating.
|
||||
- Store `username` only; do not store `first_name`.
|
||||
- If `update.Message.From.Username == ""`, skip `user:` / `pair:` writes for that invocation (still increments `count:<cmd>`). Acknowledged trade-off: unnamed users are unattributed.
|
||||
- Top-K cap = 20 entries per view (existing 4000-char truncation kept as belt-and-braces).
|
||||
|
||||
## Storage schema
|
||||
|
||||
Single DynamoDB partition `pk = "stats"`. Three sort-key shapes:
|
||||
|
||||
| Sort key | Value JSON | Purpose |
|
||||
|---|---|---|
|
||||
| `count:<cmd>` | `{"n":<int>}` | Existing. Total per command. |
|
||||
| `user:<userID>` | `{"username":"<str>","n":<int>}` | New. Per-user total + cached display name. |
|
||||
| `pair:<cmd>:<userID>` | `{"n":<int>}` | New. Per (command, user) pair. |
|
||||
|
||||
`<userID>` is decimal stringified `update.Message.From.ID`. `<cmd>` is the registered command name (no `/`).
|
||||
|
||||
## Query patterns
|
||||
|
||||
| View | Method |
|
||||
|---|---|
|
||||
| `/stats` (existing) | `List("count:")` → fan-out GetItem → sort desc by n |
|
||||
| `/stats users` | `List("user:")` → fan-out GetItem → sort desc by n |
|
||||
| `/stats user @foo` | `List("user:")` → fan-out GetItem → find matching username → derive userID → `List("pair:*:userID")` is not supported (sort-key wildcard); instead `List("pair:")` + filter by `:<userID>` suffix |
|
||||
| `/stats cmd help` | `List("pair:help:")` → fan-out GetItem; for each, GetItem `user:<id>` to resolve username |
|
||||
|
||||
`List("pair:")` worst case = N_users × N_commands rows. At scale, replace with a `byuser:<userID>:<cmd>` mirror key to enable prefix scan. **Deferred to a future phase** (see Risks); current free-tier traffic does not justify the extra write.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Hook signature + per-user write path](./phase-01-hook-signature-per-user-write-path.md) | Completed |
|
||||
| 2 | [Subcommand parser + new /stats views](./phase-02-subcommand-parser-new-stats-views.md) | Completed |
|
||||
| 3 | [Tests](./phase-03-tests.md) | Completed |
|
||||
| 4 | [Command menu](./phase-04-command-menu.md) | Completed |
|
||||
| 5 | [Docs](./phase-05-docs.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
No cross-plan dependencies. Builds directly on the already-deployed stats module.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Atomic increments (DynamoDB UpdateItem ADD). Existing race is documented and accepted.
|
||||
- Per-chat stats. Bot is single-owner; chat-scope breakdown not requested.
|
||||
- Backfill. Per-user attribution starts from deploy time. Existing `count:` keys carry over unchanged.
|
||||
- Privacy gating, anonymization, first_name capture. User declined.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
None.
|
||||
@@ -1,37 +0,0 @@
|
||||
---
|
||||
title: "Trade Income Events Command"
|
||||
status: completed
|
||||
created: 2026-06-05
|
||||
---
|
||||
|
||||
# Trade Income Events Command
|
||||
|
||||
## Context
|
||||
- Trading commands are registered in `internal/modules/trading/trading.go`.
|
||||
- Trading state is per-user KV portfolio in `internal/modules/trading/portfolio.go`.
|
||||
- Current market data client style is stdlib HTTP with injectable URL/client.
|
||||
- FireAnt is the default income-event provider; the bot calls `/symbols/{symbol}/timescale-marks` with `startDate` and `endDate`.
|
||||
|
||||
## Requirements
|
||||
- Add `/trade_income_events [TICKER]`.
|
||||
- With ticker: show recent income/right events for that stock.
|
||||
- Without ticker: check all non-zero stock holdings in current user's portfolio.
|
||||
- Recent means last 30 days by FireAnt mark date.
|
||||
- Do not mutate portfolio.
|
||||
|
||||
## Implementation
|
||||
1. Add FireAnt timescale-mark client and renderer.
|
||||
2. Register command in trading module.
|
||||
3. Add read-only handler.
|
||||
4. Add focused unit tests.
|
||||
5. Run `gofmt` and `go test ./internal/modules/trading`.
|
||||
|
||||
## Status
|
||||
- [x] Scout existing trading module.
|
||||
- [x] Select FireAnt configuration path.
|
||||
- [x] Implement command.
|
||||
- [x] Test command support.
|
||||
- [x] Review changes.
|
||||
|
||||
## Unresolved Questions
|
||||
- None.
|
||||
-65
@@ -1,65 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: Research and existing trade pattern
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: 1h
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Research and existing trade pattern
|
||||
|
||||
## Overview
|
||||
|
||||
Lock the exact behavior to mirror from the existing trading module and verify the external price path with a small manual smoke test before writing code.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: identify which trading behaviors apply directly to gold: topup, buy, sell, stats, per-user lock, per-user KV state, reply style.
|
||||
- Functional: verify default command syntax: `/gold_topup <amount>`, `/gold_buy <luong>`, `/gold_sell <luong>`, `/gold_stats`.
|
||||
- Non-functional: avoid coupling gold state to trading state; no shared mutable state between modules.
|
||||
- Non-functional: keep new code files under 200 lines where practical by splitting price, portfolio, handlers, format, and factory files.
|
||||
|
||||
## Architecture
|
||||
|
||||
Gold should copy the trading module's workflow, not import trading handlers. The shared pattern is conceptual:
|
||||
|
||||
1. Parse command args.
|
||||
2. Fetch current price if needed.
|
||||
3. Acquire `keylock.Map` by Telegram user ID.
|
||||
4. Load module-local portfolio from KV.
|
||||
5. Mutate portfolio.
|
||||
6. Save portfolio.
|
||||
7. Reply via `chathelper`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Read: `internal/modules/trading/trading.go`
|
||||
- Read: `internal/modules/trading/handlers.go`
|
||||
- Read: `internal/modules/trading/portfolio.go`
|
||||
- Read: `internal/modules/trading/prices.go`
|
||||
- Read: `internal/modules/trading/format.go`
|
||||
- Read: `cmd/server/main.go`
|
||||
- Read: `template.yaml`
|
||||
- Modify later: none in this phase
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Re-read trading command tests to copy expected style for parser and recording bot assertions.
|
||||
2. Smoke-test GoldPrice.org JSON shape with `curl https://data-asg.goldprice.org/dbXRates/USD`.
|
||||
3. Smoke-test USD/VND conversion source with `curl https://open.er-api.com/v6/latest/USD` and confirm `rates.VND` exists.
|
||||
4. Decide provider fallback behavior:
|
||||
- If GoldPrice.org fails: return user-facing "Could not fetch gold price. Try again later."
|
||||
- If FX conversion fails: same error; do not trade on stale unknown conversion.
|
||||
5. Record actual response fields used by code: `items[0].xauPrice`, `items[0].curr`, `ts`.
|
||||
6. Confirm all command names pass existing command validation regex.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] Existing trading workflow documented enough to implement without changing trading files.
|
||||
- [x] Price source response fields verified against live endpoint or a captured fixture.
|
||||
- [x] Decision recorded that v1 is spot gold converted to VND, not SJC retail price.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
GoldPrice.org endpoint is not formal API docs. Mitigation: isolate behind `GoldPriceClient`, keep tests fixture-based, and make endpoint overrideable by env/test injection so provider can be swapped without rewriting handlers.
|
||||
@@ -1,87 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: Gold price client
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 2h
|
||||
dependencies:
|
||||
- 1
|
||||
---
|
||||
|
||||
# Phase 2: Gold price client
|
||||
|
||||
## Overview
|
||||
|
||||
Implement a small HTTP client that returns VND price per `luong`, with injectable endpoints/HTTP client for tests and runtime endpoint overrides for operational fallback.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: fetch spot XAU price in USD per troy ounce.
|
||||
- Functional: fetch USD to VND exchange rate.
|
||||
- Functional: convert to VND per `luong`.
|
||||
- Functional: expose one method, `FetchLuongPrice(ctx) (float64, error)`.
|
||||
- Functional: support runtime endpoint overrides for gold and FX URLs.
|
||||
- Functional: cache FX response until `time_next_update_unix` when available; otherwise use a bounded fallback TTL.
|
||||
- Non-functional: no API key required in default path; timeout bounded for Lambda.
|
||||
- Non-functional: no global mutable client per request; reuse `http.Client` like trading does.
|
||||
- Non-functional: reject non-HTTPS override URLs except localhost/127.0.0.1 test servers.
|
||||
|
||||
## Architecture
|
||||
|
||||
Create `internal/modules/gold/prices.go`:
|
||||
|
||||
```go
|
||||
const gramsPerLuong = 37.5
|
||||
const gramsPerTroyOunce = 31.1034768
|
||||
|
||||
priceVNDPerLuong := xauUSDPerTroyOunce * usdToVND * (gramsPerLuong / gramsPerTroyOunce)
|
||||
```
|
||||
|
||||
Use two HTTP calls in v1. Keep each response struct minimal and defensive. FX can cache because ExchangeRate-API updates once daily; GoldPrice remains uncached in v1 unless latency proves painful.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/gold/prices.go`
|
||||
- Create: `internal/modules/gold/prices_test.go`
|
||||
- Modify: `cmd/server/main.go` if endpoint env vars are wired through config
|
||||
- Modify: `template.yaml` only for optional env var pass-through, not default module enablement
|
||||
- Read: `internal/modules/trading/prices.go`
|
||||
- Read: `internal/modules/trading/income_events.go` for HTTPS URL validation pattern
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add `GoldPriceClient` with `HTTP`, `GoldURL`, `FXURL`, `defaultOnce`, and `defaultClient`.
|
||||
2. Add default URLs:
|
||||
- `https://data-asg.goldprice.org/dbXRates/USD`
|
||||
- `https://open.er-api.com/v6/latest/USD`
|
||||
3. Add optional env/config plumbing for override URLs:
|
||||
- `GOLD_PRICE_API_URL`
|
||||
- `GOLD_FX_API_URL`
|
||||
4. Validate override URLs:
|
||||
- remote URLs must be `https`
|
||||
- `http://localhost`, `http://127.0.0.1`, and `http://[::1]` allowed for tests/local dev only
|
||||
5. Add bounded timeout, likely 10s to match trading.
|
||||
6. Decode GoldPrice.org response:
|
||||
- require non-empty `items`
|
||||
- require `curr == "USD"` if present
|
||||
- require `xauPrice > 0`
|
||||
7. Decode FX response:
|
||||
- require success result when field exists
|
||||
- require `rates.VND > 0`
|
||||
- read `time_next_update_unix` when present and cache until then
|
||||
- treat HTTP 429 as retryable upstream failure, not no-price
|
||||
8. Return a domain error `ErrNoGoldPrice` for empty/invalid upstream data.
|
||||
9. Wrap network/decode errors with `gold:` prefix.
|
||||
10. Unit-test success conversion, cache behavior, 429 handling, HTTPS validation, localhost exception, and invalid response paths with `httptest.Server`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] `FetchLuongPrice` returns expected VND/luong value from fixtures.
|
||||
- [x] Non-2xx, 429, malformed JSON, missing XAU, missing VND, and zero prices are covered.
|
||||
- [x] FX cache uses `time_next_update_unix` when present and avoids repeated FX calls inside that window.
|
||||
- [x] Runtime override URLs are validated and test-local HTTP URLs still work.
|
||||
- [x] No API key or secret is required for default client construction.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Two upstream calls increase latency and failure rate. Mitigation: cache FX by provider metadata, keep GoldPrice isolated behind a small client, and make provider URLs overrideable without source changes.
|
||||
@@ -1,89 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: Gold portfolio commands
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 3h
|
||||
dependencies:
|
||||
- 2
|
||||
---
|
||||
|
||||
# Phase 3: Gold portfolio commands
|
||||
|
||||
## Overview
|
||||
|
||||
Build the gold module's state, formatting, and user-facing command handlers.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `/gold_topup <amount>` credits VND and increments invested amount.
|
||||
- Functional: `/gold_buy <luong>` deducts VND at current VND/luong price and adds gold holding.
|
||||
- Functional: `/gold_sell <luong>` deducts gold holding and credits VND.
|
||||
- Functional: `/gold_stats` renders VND, gold luong, current price, gold value, total value, invested, P&L.
|
||||
- Functional: full sell after fractional buys must leave exact zero after dust normalization.
|
||||
- Non-functional: no command accepts a currency, ticker, or unit in v1.
|
||||
- Non-functional: floating quantities must reject NaN, Inf, overflow, zero, and negative values.
|
||||
|
||||
## Architecture
|
||||
|
||||
Use module-local state:
|
||||
|
||||
```go
|
||||
type Portfolio struct {
|
||||
VND float64 `json:"vnd"`
|
||||
Luong float64 `json:"luong"`
|
||||
Meta PortfolioMeta `json:"meta"`
|
||||
}
|
||||
```
|
||||
|
||||
This is simpler than trading's `Currency` and `Assets` maps because v1 gold has exactly one cash currency and one asset. Storage key remains `user:<telegramID>` inside the gold module namespace. Arithmetic uses a concrete dust threshold: after each balance mutation, values whose absolute value is `< 1e-9` are set to zero.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/gold/gold.go`
|
||||
- Create: `internal/modules/gold/handlers.go`
|
||||
- Create: `internal/modules/gold/portfolio.go`
|
||||
- Create: `internal/modules/gold/format.go`
|
||||
- Create: `internal/modules/gold/handlers_test.go`
|
||||
- Create: `internal/modules/gold/portfolio_test.go`
|
||||
- Read: `internal/modules/trading/handlers.go`
|
||||
- Read: `internal/modules/trading/portfolio.go`
|
||||
- Read: `internal/modules/trading/format.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add `state` with `kv`, `prices`, `locks`, and `nowFn`.
|
||||
2. Copy `senderInfo` and `argsAfterCommand` locally or extract only if another module already has a shared helper. Do not refactor trading unless necessary.
|
||||
3. Add a shared local parser for positive finite floats. It must reject `NaN`, `Inf`, `+Inf`, `-Inf`, overflow, zero, and negative values.
|
||||
4. Implement `LoadPortfolio`, `SavePortfolio`, `AddVND`, `DeductVND`, `AddLuong`, and `DeductLuong`.
|
||||
5. Apply dust cleanup after each mutation using `const goldDustEpsilon = 1e-9`.
|
||||
6. Implement `FormatLuong`, keeping up to 4 decimals and trimming trailing zeros.
|
||||
7. Implement `handleTopup`:
|
||||
- usage: `Usage: /gold_topup <amount>`
|
||||
- parse amount as positive finite float
|
||||
- add VND, increment invested
|
||||
8. Implement `handleBuy`:
|
||||
- usage: `Usage: /gold_buy <luong>`
|
||||
- fetch VND/luong before lock
|
||||
- cost = qty * price
|
||||
- deduct VND, add luong
|
||||
9. Implement `handleSell`:
|
||||
- fetch price before lock
|
||||
- deduct luong, add VND
|
||||
10. Implement `handleStats`:
|
||||
- fetch price; if unavailable, show holdings with `(no price)` and cash balance
|
||||
- include total value and P&L when price exists
|
||||
11. Keep replies plain text unless future Telegram formatting needs HTML.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] Fresh user can top up, buy, sell, and view stats.
|
||||
- [x] Insufficient VND and insufficient gold return clear messages.
|
||||
- [x] Price errors do not mutate portfolio.
|
||||
- [x] Portfolio load repairs zero-value/missing fields safely.
|
||||
- [x] Fractional buy/sell round trips leave no dust above `1e-9`.
|
||||
- [x] Special float strings and overflow inputs are rejected before mutation.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Using float for `luong` can produce tiny rounding artifacts. Mitigation: use one explicit epsilon, normalize dust to zero after arithmetic, test full fractional sell scenarios, and keep display precision bounded.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: Module registration and docs
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: 1h
|
||||
dependencies:
|
||||
- 3
|
||||
---
|
||||
|
||||
# Phase 4: Module registration and docs
|
||||
|
||||
## Overview
|
||||
|
||||
Wire the gold module into the bot catalog and docs as an opt-in module for first deploy.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `gold` is a first-class module selectable by `MODULES`.
|
||||
- Functional: first implementation keeps `gold` opt-in; do not add it to the default `template.yaml` `ModulesCSV` until the operator explicitly promotes the spot-priced module after opt-in smoke.
|
||||
- Functional: optional gold price endpoint env vars are documented if implemented.
|
||||
- Non-functional: docs must state price source limitation clearly.
|
||||
|
||||
## Architecture
|
||||
|
||||
Registration follows the current composition-root pattern:
|
||||
|
||||
- import `internal/modules/gold` in `cmd/server/main.go`
|
||||
- add `"gold": gold.New` to `factories()`
|
||||
- leave `template.yaml` default `ModulesCSV` unchanged for first deploy
|
||||
- optionally pass `GOLD_PRICE_API_URL` and `GOLD_FX_API_URL` through Lambda env if runtime overrides are implemented through config
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `cmd/server/main.go`
|
||||
- Modify: `README.md`
|
||||
- Modify: `docs/deploy-aws.md`
|
||||
- Maybe modify: `template.yaml` only for override env pass-through, not default module enablement
|
||||
- Maybe modify: `cmd/server/main_test.go` or equivalent to test real catalog wiring
|
||||
- Maybe modify: `internal/modules/registry_test.go` only if existing tests assert full known list
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add `gold` import and factory entry.
|
||||
2. Keep `gold` opt-in for first deploy:
|
||||
- do not add `gold` to default `ModulesCSV`
|
||||
- document `MODULES=...,gold` enablement
|
||||
3. Update README module table with `gold` and mark it opt-in if defaults remain unchanged.
|
||||
4. Add docs section:
|
||||
- default commands
|
||||
- price source: world spot XAU converted to VND, not SJC local retail
|
||||
- no secrets required for default source
|
||||
- endpoint override env vars and HTTPS validation rules if implemented
|
||||
- ExchangeRate-API attribution note if displayed/required
|
||||
5. Add or update tests for real composition-root wiring:
|
||||
- `factories()["gold"]` exists
|
||||
- `modules.Build([]string{"gold"}, factories(), ...)` succeeds
|
||||
6. Add namespace-isolation coverage for `trading` and `gold` both using `user:<id>` under different module prefixes.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] `MODULES=gold` starts without unknown-module error through the real `cmd/server` factory catalog.
|
||||
- [x] `/help` lists gold commands when module is enabled.
|
||||
- [x] README and deploy docs accurately describe gold's price source, opt-in status, and default units.
|
||||
- [x] `template.yaml` default modules remain unchanged unless user explicitly accepts default production enablement.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Adding `gold` to default modules would expose a spot-price product decision immediately after deploy. Mitigation: keep it opt-in for first deploy and promote to default only after a separate operator decision.
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: Tests and verification
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 1.5h
|
||||
dependencies:
|
||||
- 4
|
||||
---
|
||||
|
||||
# Phase 5: Tests and verification
|
||||
|
||||
## Overview
|
||||
|
||||
Add focused coverage and run compile/test commands required for a safe module addition.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: test all user-visible command paths.
|
||||
- Functional: test real module catalog wiring and module namespace isolation.
|
||||
- Non-functional: no live network dependency in unit tests.
|
||||
- Non-functional: no syntax errors; Go tests pass.
|
||||
|
||||
## Architecture
|
||||
|
||||
Tests should mirror `internal/modules/trading/*_test.go` and use:
|
||||
|
||||
- `httptest.Server` for price and FX clients.
|
||||
- `internal/storage.NewMemoryKV()` or equivalent existing memory KV.
|
||||
- `internal/testutil/recording_bot.go` for Telegram replies.
|
||||
- Injected `nowFn` for deterministic metadata.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create/modify: `internal/modules/gold/*_test.go`
|
||||
- Modify: `cmd/server/*_test.go` if needed for real `factories()` coverage
|
||||
- Maybe modify: `internal/modules/validate_test.go` only if command validation expectations list commands explicitly
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Test price conversion and invalid upstream responses.
|
||||
2. Test FX cache behavior, 429 handling, HTTPS override validation, and localhost override exception.
|
||||
3. Test parser rejection for `NaN`, `Inf`, `+Inf`, `-Inf`, overflow inputs like `1e9999`, zero, and negatives.
|
||||
4. Test portfolio first-load defaults, add/deduct, insufficient balance, dust cleanup, and save/load round trip.
|
||||
5. Test full fractional sell after fractional buys leaves zero after dust normalization.
|
||||
6. Test handlers:
|
||||
- topup usage and success
|
||||
- buy usage, success, insufficient VND, price failure
|
||||
- sell usage, success, insufficient luong, price failure
|
||||
- stats with holdings and stats with no price
|
||||
7. Test module factory registers exact commands:
|
||||
- `gold_topup`
|
||||
- `gold_buy`
|
||||
- `gold_sell`
|
||||
- `gold_stats`
|
||||
8. Test real composition-root wiring:
|
||||
- `factories()["gold"]` exists
|
||||
- `modules.Build([]string{"gold"}, factories(), ...)` succeeds
|
||||
9. Test namespace isolation by enabling both `trading` and `gold` and verifying their `user:<id>` portfolio keys do not collide under module-prefixed KV storage.
|
||||
10. Run:
|
||||
- `gofmt` on new/modified Go files
|
||||
- `go test ./internal/modules/gold`
|
||||
- `go test ./internal/modules ./cmd/server`
|
||||
- `go test ./...` before push
|
||||
11. Do a self-review for file size; split handlers if any new code file exceeds 200 lines and logical extraction is clean.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] All gold unit tests pass without network.
|
||||
- [x] Existing module registry and server catalog tests pass.
|
||||
- [x] Parser tests reject special float and overflow inputs.
|
||||
- [x] Namespace isolation test proves trading and gold portfolios do not collide.
|
||||
- [x] `go test ./...` passes locally.
|
||||
- [x] Manual smoke syntax documented: `/gold_topup 10000000`, `/gold_buy 1`, `/gold_stats`, `/gold_sell 0.5`.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Stats depends on external price source at runtime. Tests must verify graceful degradation so users still see cash/holding state if upstream is temporarily unavailable.
|
||||
@@ -1,111 +0,0 @@
|
||||
---
|
||||
title: Gold module matching trading workflow
|
||||
description: >-
|
||||
Add a standalone gold paper-trading module that mirrors trading UX, defaults
|
||||
topups to VND, defaults buys/sells to Vietnamese luong, and uses a free/no-key
|
||||
spot price source for v1.
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: main
|
||||
tags:
|
||||
- gold
|
||||
- trading
|
||||
- telegram
|
||||
- price-api
|
||||
- free-tier
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-06-11T07:35:05.803Z'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Gold module matching trading workflow
|
||||
|
||||
## Overview
|
||||
|
||||
Add `internal/modules/gold` as a separate module, not an extension of `trading`. Keep user behavior parallel to `/trade_*` commands, but gold-only:
|
||||
|
||||
- `/gold_topup <amount>` credits VND only. No currency argument.
|
||||
- `/gold_buy <luong>` buys gold in `luong` by default. No symbol or unit argument.
|
||||
- `/gold_sell <luong>` sells gold in `luong` by default. No symbol or unit argument.
|
||||
- `/gold_stats` shows VND balance, gold holding, current price, total value, invested amount, and P&L.
|
||||
|
||||
V1 pricing is explicitly **world spot XAU converted to VND per `luong`**, not Vietnamese SJC retail buy/sell price. Default price path: no-key GoldPrice.org spot XAU USD JSON plus no-key ExchangeRate-API USD to VND conversion, converted to VND per `luong` (`1 luong = 37.5g = 37.5 / 31.1034768 troy oz`). This is free-tier friendly but must be isolated behind a provider interface because GoldPrice.org JSON is undocumented. Exact Vietnamese SJC retail pricing is out of v1 unless a separate, higher-maintenance source is approved.
|
||||
|
||||
## Current Code Context
|
||||
|
||||
- `internal/modules/trading/trading.go` registers `trade_topup`, `trade_buy`, `trade_sell`, `trade_stats`, plus income helpers.
|
||||
- `internal/modules/trading/handlers.go` already has the target workflow: parse command args, fetch price outside the per-user lock, mutate KV portfolio under `keylock.Map`, reply through `chathelper`.
|
||||
- `internal/modules/trading/portfolio.go` stores per-user `Currency`, `Assets`, and `Meta.Invested` under `user:<id>`.
|
||||
- `cmd/server/main.go` owns the module catalog; adding a module requires import + `"gold": gold.New`.
|
||||
- `template.yaml` default `MODULES` currently includes `trading` but not `gold`.
|
||||
|
||||
## Price API Research
|
||||
|
||||
| Candidate | Free shape | Fit | Decision |
|
||||
|---|---|---|---|
|
||||
| GoldPrice.org `https://data-asg.goldprice.org/dbXRates/USD` | No key; current JSON has `items[0].xauPrice`, `curr`, and timestamp fields. Undocumented endpoint, no stability/SLA claim. | Best zero-secret v1 source for spot XAU if treated as best-effort. | Use as default provider for v1 behind an isolated client and runtime URL override. |
|
||||
| ExchangeRate-API open endpoint `https://open.er-api.com/v6/latest/USD` | No key; docs require attribution, allow caching, note rate limiting, update once daily, and include `rates.VND`. | Good USD to VND conversion companion. | Use for USD/VND conversion; cache until `time_next_update_unix` when available and handle 429 explicitly. |
|
||||
| Frankfurter | No-key FX API. | Possible FX fallback if VND support is verified during implementation. | Fallback only. |
|
||||
| SJC official site | HTML price table, no public JSON API found. | Exact Vietnam local retail price would require scraping or another higher-maintenance source. | Out of v1. Do not plan default SJC JSON integration. |
|
||||
| API Ninjas `/v1/goldprice` | Requires `X-Api-Key`; free users receive delayed data, and current product pages gate some endpoints. | Less aligned with no-secret free-tier. | Do not default. Keep as optional future provider. |
|
||||
| Metals-API | Requires API key; current product is key-based and not a strict no-secret default. | Not free-tier enough for this bot. | Do not default. |
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- Standalone module/package named `gold`, commands prefixed `gold_`.
|
||||
- Separate KV namespace from `trading`; no cross-portfolio mixing.
|
||||
- Keep holdings as `float64` luong, with a concrete dust rule: balances whose absolute value is `< 1e-9` are normalized to zero after arithmetic.
|
||||
- Use VND as only cash currency. Do not accept `USD`, `VND`, symbols, or units in v1 commands.
|
||||
- Fetch price before locking user state, same as trading, to keep lock scope short.
|
||||
- Keep `gold` opt-in for first deploy. Do not add it to default `template.yaml` `MODULES` until the operator explicitly promotes the spot-priced module after opt-in smoke.
|
||||
- No real order execution, no SJC spread, no fees, no cron refresh in v1.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Research and existing trade pattern](./phase-01-research-and-existing-trade-pattern.md) | Completed |
|
||||
| 2 | [Gold price client](./phase-02-gold-price-client.md) | Completed |
|
||||
| 3 | [Gold portfolio commands](./phase-03-gold-portfolio-commands.md) | Completed |
|
||||
| 4 | [Module registration and docs](./phase-04-module-registration-and-docs.md) | Completed |
|
||||
| 5 | [Tests and verification](./phase-05-tests-and-verification.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
No blocking unfinished plan. Related prior plans are complete or broader deploy work:
|
||||
|
||||
- `plans/260510-0234-pre-deploy-wrapup/` completed the trading module pattern this plan mirrors.
|
||||
- `plans/260605-0256-trade-income-events-command/` completed recent trading command additions; useful for test style only.
|
||||
|
||||
## Review Notes
|
||||
|
||||
Three read-only ClaudeKit agents reviewed this plan on 2026-06-11. Accepted changes:
|
||||
|
||||
- Resolved v1 pricing as world spot XAU converted to VND; SJC retail is out of scope.
|
||||
- Changed rollout stance from default-enabled to opt-in first deploy.
|
||||
- Added FX caching/rate-limit handling and GoldPrice best-effort caveat.
|
||||
- Added concrete fractional `luong` dust behavior and special-float tests.
|
||||
- Added real composition-root and cross-module namespace test requirements.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Gold module compiles and registers when enabled in `MODULES`.
|
||||
- `/gold_topup`, `/gold_buy`, `/gold_sell`, `/gold_stats` match trading behavior where applicable.
|
||||
- Buy/sell quantities are interpreted as `luong` by default without a unit argument.
|
||||
- Topup always credits VND without a currency argument.
|
||||
- Unit tests cover parsing, insufficient funds/holdings, price failures, stats rendering, module registration, and namespace isolation.
|
||||
- `go test ./internal/modules/gold ./internal/modules ./cmd/server` passes; run broader `go test ./...` before push.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Physical gold dealer/SJC buy/sell spread.
|
||||
- Multiple gold units in commands (`gram`, `chi`, `oz`).
|
||||
- Cross-module transfers between trading and gold.
|
||||
- Historical price charts, alerts, leaderboards, or cron refresh.
|
||||
- Default production enablement in `template.yaml` before spot-price semantics are accepted.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None for implementation. Product caveat: v1 uses world spot converted to VND, not Vietnam SJC retail price.
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
---
|
||||
title: "Gold module plan review"
|
||||
date: 2026-06-11
|
||||
status: completed
|
||||
reviewers: [planner, researcher, codebase-fit]
|
||||
---
|
||||
|
||||
# Gold module plan review
|
||||
|
||||
## Summary
|
||||
|
||||
Three ClaudeKit sub-agents reviewed `plans/260611-0735-gold-module-trading-parity/`. All completed with `DONE_WITH_CONCERNS`; no blocker, but plan needed tightening before implementation.
|
||||
|
||||
## Accepted Findings
|
||||
|
||||
- Pricing decision was inconsistent: plan used spot XAU but left spot-vs-SJC unresolved while considering default enablement.
|
||||
- GoldPrice.org endpoint works today but is undocumented; treat as best-effort soft dependency.
|
||||
- ExchangeRate-API open endpoint needs attribution awareness, cache handling, and 429 handling.
|
||||
- Free/no-key SJC JSON source is not credible for v1; exact SJC retail price is out of scope.
|
||||
- Fractional `luong` arithmetic needed explicit epsilon/dust behavior.
|
||||
- Parser tests must cover `NaN`, `Inf`, `+Inf`, `-Inf`, and overflow values accepted by `strconv.ParseFloat`.
|
||||
- Real `cmd/server` factory wiring and trading/gold namespace isolation needed tests.
|
||||
|
||||
## Plan Changes Applied
|
||||
|
||||
- V1 pricing locked to world spot XAU converted to VND per `luong`.
|
||||
- `gold` kept opt-in for first deploy; default `template.yaml` enablement deferred.
|
||||
- Runtime provider override and HTTPS/localhost validation added to price-client phase.
|
||||
- FX cache and 429 handling added.
|
||||
- Dust rule added: absolute balance below `1e-9` normalizes to zero.
|
||||
- Special-float, catalog wiring, and namespace-isolation tests added.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None.
|
||||
-48
@@ -1,48 +0,0 @@
|
||||
---
|
||||
title: "Gold module completion report"
|
||||
date: 2026-06-11
|
||||
status: completed
|
||||
---
|
||||
|
||||
# Gold module completion report
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented opt-in `gold` module per reviewed plan. Module mirrors trading workflow for a gold-only paper account: VND topup, buy/sell in `luong`, stats with P&L.
|
||||
|
||||
## Files Changed
|
||||
|
||||
| Area | Files |
|
||||
|---|---|
|
||||
| Gold module | `internal/modules/gold/*.go` |
|
||||
| Server wiring | `cmd/server/main.go`, `cmd/server/main_test.go` |
|
||||
| Deploy config | `template.yaml` |
|
||||
| Docs | `README.md`, `docs/deploy-aws.md` |
|
||||
| Plan/report | `plans/260611-0735-gold-module-trading-parity/` |
|
||||
|
||||
## Verification
|
||||
|
||||
- `go test -count=1 ./internal/modules/gold ./internal/modules ./cmd/server` passed.
|
||||
- `go test -count=1 ./...` passed.
|
||||
- `go test -race -count=1 ./internal/modules/gold` passed.
|
||||
- `go test -cover ./internal/modules/gold` passed at 83.4% statement coverage.
|
||||
- `make vet` passed.
|
||||
- `git diff --check` passed.
|
||||
- `sam validate` could not run because `sam` is not installed in this environment.
|
||||
- Tester subagent re-check: DONE, no blockers.
|
||||
- Reviewer subagent re-check: DONE, no blockers.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] Gold module compiles and registers when enabled in `MODULES`.
|
||||
- [x] `/gold_topup` credits VND only.
|
||||
- [x] `/gold_buy` and `/gold_sell` require exactly one `luong` argument.
|
||||
- [x] Price client computes VND/luong from XAU USD and USD/VND.
|
||||
- [x] FX cache, 429 handling, URL validation, and localhost test exception covered.
|
||||
- [x] Special float, overflow, and too-large finite transaction inputs rejected.
|
||||
- [x] Trading/gold storage namespace isolation tested.
|
||||
- [x] `template.yaml` default `ModulesCSV` remains unchanged; `gold` is opt-in.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None.
|
||||
@@ -1,79 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Implement gold_price command"
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: "30min"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Implement gold_price command
|
||||
|
||||
## Overview
|
||||
|
||||
Add `/gold_price` — a read-only command that fetches and displays spot XAU in USD/oz and VND/luong. No arguments, no portfolio state, no lock needed.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `/gold_price` with no arguments displays current gold price in USD per troy ounce and VND per luong.
|
||||
- Functional: on price fetch failure, show user-friendly error (same pattern as `replyPriceError`).
|
||||
- Non-functional: no portfolio mutation, no keylock acquisition.
|
||||
- Non-functional: reject extra arguments (`len(args) != 0`).
|
||||
|
||||
## Architecture
|
||||
|
||||
Add a `GoldPrice` struct and `FetchPrice(ctx)` method to `GoldPriceClient` that returns all three components (XAU USD, USD/VND rate, VND/luong) in one call. Reuses existing `fetchXAUUSD` and `fetchUSDVND` internals. The `priceFetcher` interface gains `FetchPrice` so tests can mock it.
|
||||
|
||||
Output format:
|
||||
```
|
||||
Gold Spot Price
|
||||
XAU: $3,285.50 USD/oz
|
||||
Rate: 25,000 VND/USD
|
||||
VND: 99,450,000 VND/luong
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/gold/prices.go` — add `GoldPrice` struct + `FetchPrice` method
|
||||
- Modify: `internal/modules/gold/helpers.go` — update `priceFetcher` interface
|
||||
- Modify: `internal/modules/gold/handlers.go` — add `handlePrice` handler
|
||||
- Modify: `internal/modules/gold/gold.go` — register `gold_price` command
|
||||
- Modify: `internal/modules/gold/format.go` — add `FormatUSD` helper
|
||||
- Modify: `internal/modules/gold/handlers_test.go` — add handler test
|
||||
- Modify: `internal/modules/gold/prices_test.go` — add `FetchPrice` test
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add `GoldPrice` struct to `prices.go`:
|
||||
```go
|
||||
type GoldPrice struct {
|
||||
XAUUSD float64 // USD per troy ounce
|
||||
USDVND float64 // VND per USD
|
||||
VNDPerLuong float64 // VND per luong
|
||||
}
|
||||
```
|
||||
2. Add `FetchPrice(ctx context.Context) (GoldPrice, error)` to `GoldPriceClient` — calls `fetchXAUUSD` + `fetchUSDVND`, computes VNDPerLuong, returns struct.
|
||||
3. Update `priceFetcher` interface in `helpers.go` to include `FetchPrice`.
|
||||
4. Update `fakePriceFetcher` in `handlers_test.go` to implement `FetchPrice`.
|
||||
5. Add `FormatUSD(n float64) string` to `format.go` — e.g. `$3,285.50`.
|
||||
6. Add `handlePrice` to `handlers.go`:
|
||||
- Reject if `len(args) != 0` → usage message.
|
||||
- Call `s.prices.FetchPrice(ctx)`.
|
||||
- On error → `replyPriceError`.
|
||||
- Format and reply with 3-line price summary.
|
||||
7. Register `gold_price` command in `gold.go` with description `"Show current gold spot price (USD & VND)"`.
|
||||
8. Add tests:
|
||||
- `TestHandlePrice` — verify output contains USD and VND lines.
|
||||
- `TestHandlePriceRejectsArgs` — verify extra args rejected.
|
||||
- `TestHandlePriceFetchError` — verify error path.
|
||||
- `TestGoldPriceClient_FetchPrice` — verify struct fields from httptest server.
|
||||
9. Run `go test -count=1 ./internal/modules/gold/...` and `go vet ./...`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `/gold_price` returns USD/oz and VND/luong prices.
|
||||
- [ ] Extra arguments rejected with usage message.
|
||||
- [ ] Price fetch errors handled gracefully.
|
||||
- [ ] `priceFetcher` interface updated; `fakePriceFetcher` implements both methods.
|
||||
- [ ] All existing + new tests pass.
|
||||
- [ ] `go vet` clean.
|
||||
@@ -1,33 +0,0 @@
|
||||
---
|
||||
title: "Gold price command"
|
||||
description: "Add /gold_price command showing spot XAU in USD/oz and VND/luong"
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: "main"
|
||||
tags: [gold]
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: "2026-06-11T10:44:35.588Z"
|
||||
createdBy: "ck:plan"
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Gold price command
|
||||
|
||||
## Overview
|
||||
|
||||
Add `/gold_price` to the gold module. Displays current spot gold price in both USD per troy ounce and VND per luong. No arguments, no portfolio mutation — read-only price lookup.
|
||||
|
||||
## Context
|
||||
|
||||
`GoldPriceClient` already fetches XAU USD/oz and USD/VND internally via unexported `fetchXAUUSD` and `fetchUSDVND`. Currently only `FetchLuongPrice` (combined VND/luong) is exposed. Need a new method returning all components.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Implement gold_price command](./phase-01-implement-gold-price-command.md) | Pending |
|
||||
|
||||
## Dependencies
|
||||
|
||||
None. Builds on committed gold module (`2304657`).
|
||||
@@ -1,114 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: Price provider chain
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 3-4h
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Price provider chain
|
||||
|
||||
## Context Links
|
||||
|
||||
- Research: `plans/reports/260612-0948-coin-module-research-report.md`
|
||||
- Reference price clients: `internal/modules/gold/prices.go`, `internal/modules/trading/prices.go`
|
||||
|
||||
## Overview
|
||||
|
||||
Build the crypto USD price lookup layer with fixed fallback order: Binance, Coinbase, CoinGecko. This phase should be independent from Telegram handlers and portfolio mutation.
|
||||
|
||||
## Key Insights
|
||||
|
||||
- Binance `/api/v3/ticker/price` is best first for major `USDT` pairs but pair coverage is not universal.
|
||||
- Coinbase `/v2/exchange-rates` requires no auth and returns direct USD rates for supported base currencies.
|
||||
- CoinGecko `/simple/price` has broad coin-ID coverage but public/demo rate limit is around 30 calls/min and variable.
|
||||
- All provider failures must return typed errors and allow fallback where safe.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: fetch USD price for whitelisted symbol.
|
||||
- Functional: try Binance `SYMBOLUSDT`, then optionally `SYMBOLUSD`, then Coinbase, then CoinGecko.
|
||||
- Functional: return `CoinPrice{Symbol, USD, Source}` from first valid provider.
|
||||
- Functional: expose env URL overrides: `COIN_BINANCE_API_URL`, `COIN_COINBASE_API_URL`, `COIN_COINGECKO_API_URL`.
|
||||
- Non-functional: 10s HTTP timeout, injected HTTP client for tests, no network calls in tests.
|
||||
- Non-functional: 15-30s in-memory cache; cache only valid positive prices.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
handlers/stats
|
||||
-> PriceClient.FetchUSD(symbol)
|
||||
-> cache lookup
|
||||
-> BinanceProvider.FetchUSD(symbol)
|
||||
-> CoinbaseProvider.FetchUSD(symbol)
|
||||
-> CoinGeckoProvider.FetchUSD(symbol)
|
||||
-> ErrNoCoinPrice
|
||||
```
|
||||
|
||||
Provider-specific structs stay inside `internal/modules/coin`. Do not create shared price infrastructure until another module needs it.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/coin/prices.go`
|
||||
- Create: `internal/modules/coin/price_providers.go`
|
||||
- Create: `internal/modules/coin/symbols.go`
|
||||
- Create: `internal/modules/coin/prices_test.go`
|
||||
- Read: `internal/modules/gold/prices.go`
|
||||
- Read: `internal/modules/gold/price_providers.go`
|
||||
- Read: `internal/modules/trading/prices.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Define `CoinPrice`, `PriceProvider`, `PriceClient`, `ErrNoCoinPrice`, and provider error handling.
|
||||
2. Add supported symbol mapping:
|
||||
- `BTC -> bitcoin`
|
||||
- `ETH -> ethereum`
|
||||
- `SOL -> solana`
|
||||
- `BNB -> binancecoin`
|
||||
- `XRP -> ripple`
|
||||
- `ADA -> cardano`
|
||||
- `DOGE -> dogecoin`
|
||||
- `TON -> the-open-network`
|
||||
3. Implement Binance provider:
|
||||
- request `?symbol={SYMBOL}USDT` first
|
||||
- if no price, request `?symbol={SYMBOL}USD`
|
||||
- parse JSON string `price`
|
||||
- treat non-2xx, 429, invalid/zero price as fallback-safe errors
|
||||
4. Implement Coinbase provider:
|
||||
- request `?currency={SYMBOL}`
|
||||
- parse `data.rates.USD` string
|
||||
5. Implement CoinGecko provider:
|
||||
- request `?ids={coinID}&vs_currencies=usd&include_last_updated_at=true`
|
||||
- parse `{coinID}.usd`
|
||||
6. Implement cache keyed by symbol with source and expiry.
|
||||
7. Add URL validation if following gold's endpoint validation pattern. At minimum reject blank/malformed override URLs.
|
||||
8. Wrap errors with `coin:` prefix but keep typed `ErrNoCoinPrice` detectable.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [x] Price client and provider interfaces added.
|
||||
- [x] Binance provider implemented.
|
||||
- [x] Coinbase provider implemented.
|
||||
- [x] CoinGecko provider implemented.
|
||||
- [x] Symbol whitelist and CoinGecko ID mapping added.
|
||||
- [x] Cache implemented and tested.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] First valid provider wins and returns source name.
|
||||
- [x] 429/non-2xx/decode/no-price failures fall through to next provider.
|
||||
- [x] Unsupported local symbols fail before network calls.
|
||||
- [x] Unit tests cover each provider and fallback path.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Main risk: accidental provider spam from `/coin_stats`. Mitigation: cache and no retry loops. Binance has IP-ban risk after repeated 429 abuse; on 429, immediately fall through and do not re-call within cache/backoff window.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
No API keys required. Do not add secrets. Do not accept arbitrary user-provided URLs; env overrides only. Reject untrusted symbols before request building.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 2 consumes `PriceClient` via a small interface so handlers can use fakes in tests.
|
||||
@@ -1,128 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: Portfolio and commands
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 4-5h
|
||||
dependencies:
|
||||
- 1
|
||||
---
|
||||
|
||||
# Phase 2: Portfolio and commands
|
||||
|
||||
## Context Links
|
||||
|
||||
- Price client phase: `phase-01-price-provider-chain.md`
|
||||
- Gold handler pattern: `internal/modules/gold/handlers.go`, `internal/modules/gold/portfolio.go`
|
||||
- Trading handler pattern: `internal/modules/trading/handlers.go`, `internal/modules/trading/portfolio.go`
|
||||
|
||||
## Overview
|
||||
|
||||
Implement the standalone `coin` module state, portfolio model, formatting helpers, and Telegram commands for USD paper trading.
|
||||
|
||||
## Key Insights
|
||||
|
||||
- Coin balances must be fractional `float64`, like gold holdings.
|
||||
- Portfolio state must stay module-scoped via `kv.For("coin")`; storage key can remain `user:<telegramID>` inside namespace.
|
||||
- Fetch price before acquiring per-user lock or CAS update.
|
||||
- Use `UpdatePortfolio` CAS pattern from `gold` if storage supports `CompareAndSwapStore`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `/coin_price <COIN>` shows USD price and provider source.
|
||||
- Functional: `/coin_topup <usd_amount>` credits fake USD and increments invested.
|
||||
- Functional: `/coin_buy <usd_amount> <COIN>` deducts USD and credits `usd_amount / price` units.
|
||||
- Functional: `/coin_sell <qty> <COIN>` deducts coin units and credits `qty * price` USD.
|
||||
- Functional: `/coin_stats` shows USD, holdings, price source, market value, total, invested, P&L.
|
||||
- Non-functional: reject invalid sender, unsupported coin, non-finite amount, insufficient balance, and too-large trade values.
|
||||
|
||||
## Architecture
|
||||
|
||||
```go
|
||||
type Portfolio struct {
|
||||
USD float64 `json:"usd"`
|
||||
Assets map[string]float64 `json:"assets"`
|
||||
Meta PortfolioMeta `json:"meta"`
|
||||
}
|
||||
|
||||
type PortfolioMeta struct {
|
||||
Invested float64 `json:"invested"`
|
||||
CreatedAt int64 `json:"createdAt"`
|
||||
}
|
||||
```
|
||||
|
||||
Handlers depend on a `priceFetcher` interface returning `CoinPrice`, not concrete providers. This keeps tests deterministic.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/coin/coin.go`
|
||||
- Create: `internal/modules/coin/helpers.go`
|
||||
- Create: `internal/modules/coin/handlers.go`
|
||||
- Create: `internal/modules/coin/portfolio.go`
|
||||
- Create: `internal/modules/coin/format.go`
|
||||
- Create: `internal/modules/coin/handlers_test.go`
|
||||
- Create: `internal/modules/coin/portfolio_test.go`
|
||||
- Read: `internal/modules/gold/*`
|
||||
- Read: `internal/modules/trading/*`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Create `coin.go` with public commands:
|
||||
- `coin_price`
|
||||
- `coin_topup`
|
||||
- `coin_buy`
|
||||
- `coin_sell`
|
||||
- `coin_stats`
|
||||
2. Create `helpers.go` with local copies/patterns for `senderInfo`, `argsAfterCommand`, finite positive parsing, and safe USD checks.
|
||||
3. Create `portfolio.go` using gold's `UpdatePortfolio` retry/CAS approach:
|
||||
- initialize `Assets` map on load
|
||||
- normalize NaN/Inf and dust below `1e-9`
|
||||
- delete asset key when holding hits zero
|
||||
4. Create formatting helpers:
|
||||
- `FormatUSD`
|
||||
- `FormatCoinQty`
|
||||
- `FormatPnLUSD`
|
||||
5. Implement `/coin_price <COIN>` read-only path.
|
||||
6. Implement `/coin_topup <usd_amount>` with no provider call.
|
||||
7. Implement `/coin_buy <usd_amount> <COIN>`:
|
||||
- validate amount and symbol
|
||||
- fetch price
|
||||
- compute qty
|
||||
- CAS update deduct USD/add asset
|
||||
8. Implement `/coin_sell <qty> <COIN>`:
|
||||
- validate qty and symbol
|
||||
- fetch price
|
||||
- CAS update deduct asset/add USD
|
||||
9. Implement `/coin_stats`:
|
||||
- load portfolio
|
||||
- fetch prices for held assets through cached client
|
||||
- degrade gracefully if some prices fail
|
||||
10. Keep replies concise and include source, e.g. `Price: $67,321.42 (Binance)`.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [x] Module factory created.
|
||||
- [x] Portfolio state and update helpers created.
|
||||
- [x] Command handlers implemented.
|
||||
- [x] Formatting helpers created.
|
||||
- [x] Handler tests use fake price fetcher.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] All commands return usage messages for bad argument count.
|
||||
- [x] Topup changes USD and invested only.
|
||||
- [x] Buy/sell mutate state only after price success.
|
||||
- [x] Stats works with empty portfolio and with held assets.
|
||||
- [x] Source name visible in price-dependent replies.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Main risk: floating-point dust or accidental negative balances. Mitigation: finite validation, dust normalization, safe range checks, and table-driven tests around exact insufficient-balance paths.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
This is fake money. Still treat state updates as user-owned data: key by Telegram user ID, reject senderless updates, and avoid logging user balances unnecessarily.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 3 wires the package into startup and deployment docs.
|
||||
-102
@@ -1,102 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: Module registration and docs
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: 1.5-2h
|
||||
dependencies:
|
||||
- 2
|
||||
---
|
||||
|
||||
# Phase 3: Module registration and docs
|
||||
|
||||
## Context Links
|
||||
|
||||
- Composition root: `cmd/server/main.go`
|
||||
- Module docs: `README.md`
|
||||
- Deploy config: `template.yaml`
|
||||
- Gold registration reference: `plans/260611-0735-gold-module-trading-parity/phase-04-module-registration-and-docs.md`
|
||||
|
||||
## Overview
|
||||
|
||||
Register `coin` as a first-class module, enable it in the deployed AWS module default, and document optional provider URL overrides.
|
||||
|
||||
## Key Insights
|
||||
|
||||
- `cmd/server/main.go` owns the factory map to avoid import cycles.
|
||||
- `template.yaml` controls deployed `MODULES` default via `ModulesCSV`.
|
||||
- User explicitly requested AWS registration; `coin` is included in deployed `ModulesCSV` default.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: deployed `MODULES` default includes `coin`; `MODULES=coin` also starts and registers all coin commands.
|
||||
- Functional: optional env overrides are passed through startup config if needed by `NewCoinPriceClientFromEnv`.
|
||||
- Functional: README module table includes `coin` and command summary.
|
||||
- Non-functional: keep deployment free-tier; no SSM secret required.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
cmd/server/main.go factories()
|
||||
"coin": coin.New
|
||||
|
||||
loadConfig()
|
||||
CoinBinanceAPIURL
|
||||
CoinCoinbaseAPIURL
|
||||
CoinCoinGeckoAPIURL
|
||||
|
||||
main()
|
||||
exportOptionalEnv("COIN_BINANCE_API_URL", cfg.CoinBinanceAPIURL)
|
||||
exportOptionalEnv("COIN_COINBASE_API_URL", cfg.CoinCoinbaseAPIURL)
|
||||
exportOptionalEnv("COIN_COINGECKO_API_URL", cfg.CoinCoinGeckoAPIURL)
|
||||
```
|
||||
|
||||
If the coin price client reads env directly, config additions are optional. Prefer explicit config additions only if matching the gold pattern is worth the extra lines.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `cmd/server/main.go`
|
||||
- Modify: `README.md`
|
||||
- Modify: `template.yaml`
|
||||
- Modify/create: `cmd/server/*_test.go` if factory/config tests exist or are needed
|
||||
- Read: `internal/modules/registry_test.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Import `internal/modules/coin` in `cmd/server/main.go`.
|
||||
2. Add `"coin": coin.New` to `factories()`.
|
||||
3. Decide default deployment behavior:
|
||||
- add `coin` to `ModulesCSV` default so AWS deploy enables it
|
||||
4. Add non-secret template parameters only if URL overrides must be deploy-configurable:
|
||||
- `CoinBinanceAPIURL`
|
||||
- `CoinCoinbaseAPIURL`
|
||||
- `CoinCoinGeckoAPIURL`
|
||||
5. Wire env variables into Lambda environment if parameters are added.
|
||||
6. Update README module table with `coin` and provider fallback summary.
|
||||
7. Add command usage examples in README only if existing docs style supports it; otherwise keep docs minimal.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [x] Factory import and map entry added.
|
||||
- [x] README module table updated.
|
||||
- [x] `template.yaml` updated with `coin` in deployed `ModulesCSV` default.
|
||||
- [x] Optional env overrides documented.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] `modules.Build([]string{"coin"}, factories(), ...)` succeeds in test or local verification.
|
||||
- [x] `/help` can list coin commands when module enabled.
|
||||
- [x] No new secrets or paid services required.
|
||||
- [x] Docs do not overpromise real trading.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Adding `coin` to default deployed modules exposes new public commands immediately. User explicitly chose AWS registration; commands are public paper-trading only and require no secrets.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Do not add API keys or Parameter Store secrets. Provider URL overrides are non-secret config only.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 4 verifies unit behavior and full module integration.
|
||||
@@ -1,118 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: Tests and verification
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 2-3h
|
||||
dependencies:
|
||||
- 1
|
||||
- 2
|
||||
- 3
|
||||
---
|
||||
|
||||
# Phase 4: Tests and verification
|
||||
|
||||
## Context Links
|
||||
|
||||
- Existing tests: `internal/modules/gold/*_test.go`, `internal/modules/trading/*_test.go`
|
||||
- Module registry tests: `internal/modules/registry_test.go`, `cmd/server` tests if present
|
||||
- Commands: `make test`, `make vet`
|
||||
|
||||
## Overview
|
||||
|
||||
Add focused unit tests and run compile/test gates. Tests must not call real Binance, Coinbase, or CoinGecko.
|
||||
|
||||
## Key Insights
|
||||
|
||||
- Provider tests should use `httptest.Server` or fake `RoundTripper` like existing modules.
|
||||
- Handler tests should inject fake price fetchers, not rely on provider chain.
|
||||
- Fallback behavior is the highest-risk logic; test it explicitly.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: unit tests cover provider decoding, fallback, cache, portfolio mutation, handlers, and registration.
|
||||
- Non-functional: no network in tests, deterministic time where needed, no flaky provider timing.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
coin tests
|
||||
provider fixtures -> prices.go / price_providers.go
|
||||
portfolio tests -> portfolio.go
|
||||
handler tests -> fake priceFetcher + memory KV
|
||||
registration -> factories/build if practical
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/coin/prices_test.go`
|
||||
- Create: `internal/modules/coin/portfolio_test.go`
|
||||
- Create: `internal/modules/coin/handlers_test.go`
|
||||
- Modify/create: `cmd/server/main_test.go` if factory coverage is missing
|
||||
- Modify: existing docs only if verification changes behavior
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add provider tests:
|
||||
- Binance success parses `price` string.
|
||||
- Binance non-2xx/429 falls back.
|
||||
- Coinbase success parses `data.rates.USD`.
|
||||
- CoinGecko success parses `{id}.usd`.
|
||||
- all providers fail -> `ErrNoCoinPrice`.
|
||||
2. Add cache tests:
|
||||
- repeated fetch within TTL avoids second provider call.
|
||||
- expired cache refetches.
|
||||
3. Add portfolio tests:
|
||||
- new user initializes USD 0 and empty assets.
|
||||
- topup increments USD and invested.
|
||||
- buy/sell math and dust cleanup.
|
||||
- insufficient USD/asset returns current balance.
|
||||
4. Add handler tests:
|
||||
- usage errors.
|
||||
- unknown coin.
|
||||
- topup success.
|
||||
- buy success and insufficient USD.
|
||||
- sell success and insufficient coin.
|
||||
- stats empty and stats with price failure partial display.
|
||||
5. Add registration test if current test structure supports it.
|
||||
6. Run verification commands:
|
||||
- `go test ./internal/modules/coin`
|
||||
- `go test ./internal/modules/... ./cmd/server`
|
||||
- `go test ./...`
|
||||
- `go vet ./...` or `make vet`
|
||||
|
||||
## Todo List
|
||||
|
||||
- [x] Provider tests added.
|
||||
- [x] Cache tests added.
|
||||
- [x] Portfolio tests added.
|
||||
- [x] Handler tests added.
|
||||
- [x] Registration/docs verification added.
|
||||
- [x] Compile/test commands run and results recorded.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] All coin tests pass without network.
|
||||
- [x] Provider fallback test proves order Binance -> Coinbase -> CoinGecko.
|
||||
- [x] Handler tests prove portfolio not mutated on price failure.
|
||||
- [x] Broader module/server tests pass.
|
||||
- [x] README/template docs match actual default module behavior.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
Main risk: passing unit tests with fake providers while real endpoints have shape drift. Mitigation: keep provider decoders strict but small, include env URL overrides for quick hotfix, and make runtime error messages graceful.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Tests must not require real API keys or real network. Do not add dotenv or credentials. Avoid fixtures with sensitive data.
|
||||
|
||||
## Next Steps
|
||||
|
||||
After this phase, implementation is ready for code review and optional release decision on default enablement.
|
||||
|
||||
## Verification Results
|
||||
|
||||
- `go test ./internal/modules/coin` passed.
|
||||
- `go test ./...` passed.
|
||||
- `go vet ./...` passed.
|
||||
- `sam validate` not run: SAM CLI is not installed in this environment.
|
||||
@@ -1,83 +0,0 @@
|
||||
---
|
||||
title: Coin module with crypto price fallback
|
||||
description: >-
|
||||
Add a standalone coin paper-trading module with USD balances and Binance ->
|
||||
Coinbase -> CoinGecko price fallback.
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: 10-14h
|
||||
branch: main
|
||||
tags:
|
||||
- feature
|
||||
- backend
|
||||
- api
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-06-12'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Coin module with crypto price fallback
|
||||
|
||||
## Overview
|
||||
|
||||
Add `internal/modules/coin` as a standalone crypto paper-trading module. Users top up fake USD, buy/sell supported coins at market price, and view portfolio stats. Price lookup uses a best-effort provider chain: Binance first, Coinbase second, CoinGecko third.
|
||||
|
||||
## Scope Challenge
|
||||
|
||||
- Existing code: `gold` already has the closest fractional-asset portfolio, CAS update, command, price-client, and docs pattern. `trading` has useful asset-map and command naming patterns.
|
||||
- Minimum changes: new `coin` package, factory registration, optional env URL config, README/template docs, tests. No shared trading refactor required.
|
||||
- Complexity: expected 10-12 touched files. New abstractions limited to `PriceProvider` interface and provider structs inside `coin` package.
|
||||
- Selected mode: HOLD SCOPE. Deliver robust MVP, defer nonessential crypto features.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- Paper trading only. No wallets, private keys, deposits, withdrawals, real exchange orders, leverage, charts, or tax logic.
|
||||
- USD-only cash balance for v1.
|
||||
- Supported coin whitelist first; no arbitrary symbols.
|
||||
- Provider order fixed for v1: Binance -> Coinbase -> CoinGecko.
|
||||
- Include price source in user replies so provider differences are visible.
|
||||
- Enable `coin` in the deployed `ModulesCSV` default so AWS registration includes the module.
|
||||
|
||||
## References
|
||||
|
||||
- Research: `plans/reports/260612-0948-coin-module-research-report.md`
|
||||
- Patterns: `internal/modules/gold`, `internal/modules/trading`
|
||||
- Composition root: `cmd/server/main.go`
|
||||
- Deployment config: `template.yaml`
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Price provider chain](./phase-01-price-provider-chain.md) | Completed |
|
||||
| 2 | [Portfolio and commands](./phase-02-portfolio-and-commands.md) | Completed |
|
||||
| 3 | [Module registration and docs](./phase-03-module-registration-and-docs.md) | Completed |
|
||||
| 4 | [Tests and verification](./phase-04-tests-and-verification.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- No blocking active plan detected.
|
||||
- Completed gold plan is reference only: `plans/260611-0735-gold-module-trading-parity/plan.md`.
|
||||
- Existing unresolved migration/deploy plans touch `trading`/infra, not `coin`; no bidirectional dependency needed.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- `/coin_price`, `/coin_topup`, `/coin_buy`, `/coin_sell`, `/coin_stats` work when `coin` is enabled.
|
||||
- Price client falls back Binance -> Coinbase -> CoinGecko and never mutates portfolio when all providers fail.
|
||||
- Tests cover portfolio math, handler validation, provider decoding, provider fallback, and module registration.
|
||||
- `go test ./internal/modules/coin ./cmd/server ./internal/modules/...` passes; run `go test ./...` before push.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- Real exchange trading.
|
||||
- Wallet/on-chain integration.
|
||||
- Cross-module transfers between `trading`, `gold`, and `coin`.
|
||||
- Limit orders, recurring buys, alerts, charts, or leaderboards.
|
||||
- Dynamic coin discovery from public APIs.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
- Should `/coin_buy` remain USD amount only, or support quantity mode too?
|
||||
- Initial whitelist: top 8 from research enough, or include more now?
|
||||
@@ -1,63 +0,0 @@
|
||||
---
|
||||
title: Fix VNAppMob gold price key parsing
|
||||
description: >-
|
||||
The /api/request_api_key endpoint returns the JWT inside a JSON object
|
||||
{"results":"<jwt>"}, but the client was sending the whole JSON object as the
|
||||
Bearer token, causing SJC requests to fail with 403 "Invalid header padding".
|
||||
status: completed
|
||||
priority: P1
|
||||
branch: main
|
||||
tags:
|
||||
- gold
|
||||
- vnappmob
|
||||
- bugfix
|
||||
- api-key
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-06-15T04:45:00Z'
|
||||
createdBy: opencode
|
||||
---
|
||||
|
||||
# Fix VNAppMob gold price key parsing
|
||||
|
||||
## Root cause
|
||||
|
||||
`VNAppMobClient.refreshKeyLocked` reads the response body from
|
||||
`GET /api/request_api_key?scope=gold` and tries to handle either a raw JWT or a
|
||||
JSON-quoted string. The live endpoint actually returns:
|
||||
|
||||
```json
|
||||
{"results":"eyJhbGciOiJIUzI1NiIs..."}
|
||||
```
|
||||
|
||||
Because `json.Unmarshal` into a plain `string` fails for an object, the code
|
||||
falls back to using the entire JSON object (including braces and quotes) as the
|
||||
token. The subsequent `Authorization: Bearer {"results":...}` header is rejected
|
||||
by `/api/v2/gold/sjc` with `403 - Forbidden: Auth Error: Invalid header padding`,
|
||||
so all `/gold_price` requests fail with "Could not fetch gold price".
|
||||
|
||||
## Fix
|
||||
|
||||
1. Parse the refresh response as `struct{ Results string }` first.
|
||||
2. Fall back to raw body / JSON-quoted string for backward compatibility.
|
||||
3. Update tests to exercise the JSON-object response format.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | Parse JSON object wrapper in refresh response | Completed |
|
||||
| 2 | Add unit tests for object-format token | Completed |
|
||||
| 3 | Run unit tests and live smoke test | Completed |
|
||||
| 4 | Code review and docs update | Completed |
|
||||
|
||||
## Files
|
||||
|
||||
- `internal/modules/gold/vnappmob_client.go`
|
||||
- `internal/modules/gold/vnappmob_client_test.go`
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `/gold_price` returns live SJC buy/sell prices.
|
||||
- Existing unit tests still pass.
|
||||
- New test covers the `{"results":"<jwt>"}` response shape.
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: Update sell message and tests
|
||||
status: completed
|
||||
priority: P2
|
||||
dependencies: []
|
||||
effort: 30-45m
|
||||
---
|
||||
|
||||
# Phase 1: Update sell message and tests
|
||||
|
||||
## Overview
|
||||
|
||||
Improve `/coin_sell` insufficient-holdings replies so users see the failed sell amount and available-to-sell value in USD, with coin quantity and current price as supporting context.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: If user has some holdings but less than requested USD sell amount, reply:
|
||||
- `Not enough <COIN> to sell <requested USD>.`
|
||||
- `Available to sell: <available USD> (<held qty> <COIN> @ <price USD>).`
|
||||
- `Try <available USD> or less.`
|
||||
- Functional: If user has zero holdings, reply:
|
||||
- `No <COIN> available to sell.`
|
||||
- `Try /coin_buy <COIN> <usd_amount> first.`
|
||||
- Functional: Keep successful sell response unchanged.
|
||||
- Non-functional: Preserve current portfolio mutation semantics and command argument contract.
|
||||
|
||||
## Architecture
|
||||
|
||||
`handleSell` already resolves the USD amount, fetches price, computes required coin quantity, and receives held quantity from `Portfolio.DeductAsset` on failure. Reuse that held quantity to format a user-facing reply through a small helper or localized branch inside `handleSell`.
|
||||
|
||||
No new storage, price, command registration, or module boundary changes.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/coin/handlers.go` — format improved insufficient sell reply.
|
||||
- Modify: `internal/modules/coin/handlers_test.go` — assert partial-holding and zero-holding wording.
|
||||
- Delete: none.
|
||||
- Create: none.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add or inline a small formatter for insufficient sell replies.
|
||||
2. In `handleSell`, keep `held` from `DeductAsset`.
|
||||
3. If `held` normalizes to zero, return the zero-holdings message.
|
||||
4. Otherwise compute `availableUSD := held * price.USD` and return the three-line available-to-sell message.
|
||||
5. Update `TestHandleSellInsufficientCoin` to assert the zero-holdings copy.
|
||||
6. Update `TestHandleSellInsufficientCoinWithHoldings` to assert requested USD, available USD, coin quantity, price, and retry hint; keep mutation assertion.
|
||||
7. Run focused and full tests.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] Partial-holdings insufficient sell message uses requested USD as primary value.
|
||||
- [x] Partial-holdings insufficient sell message includes available-to-sell USD, coin quantity, and price.
|
||||
- [x] Zero-holdings insufficient sell message does not show `$0.00 (0 ETH @ price)`.
|
||||
- [x] Failed sell leaves portfolio unchanged.
|
||||
- [x] `go test ./internal/modules/coin -count=1` passes.
|
||||
- [x] `go test ./... -count=1` passes.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: message includes USD cash-like wording and remains ambiguous. Mitigation: use `Available to sell`, not `have`.
|
||||
- Risk: floating point precision leaks into text. Mitigation: use existing `FormatUSD` and `FormatCoinQty`.
|
||||
- Risk: tests become too brittle around full text. Mitigation: assert meaningful substrings, not every newline.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
title: Improve coin sell insufficient message
|
||||
description: >-
|
||||
Make /coin_sell insufficient-holdings replies explain available-to-sell USD
|
||||
value, with coin quantity and price as context.
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: main
|
||||
tags:
|
||||
- bugfix
|
||||
- backend
|
||||
- ux
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-06-20T02:12:55.097Z'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Improve coin sell insufficient message
|
||||
|
||||
## Overview
|
||||
|
||||
`/coin_sell <COIN> <usd_amount>` sells a USD amount of crypto. When holdings are insufficient, the reply must stay in the user's USD framing while making clear the limiting balance is coin holdings, not USD cash.
|
||||
|
||||
Recommended UX:
|
||||
|
||||
```text
|
||||
Not enough BTC to sell $600.00.
|
||||
Available to sell: $500.00 (0.01 BTC @ $50,000.00).
|
||||
Try $500.00 or less.
|
||||
```
|
||||
|
||||
Zero holdings should use a clearer special case:
|
||||
|
||||
```text
|
||||
No ETH available to sell.
|
||||
Try /coin_buy ETH <usd_amount> first.
|
||||
```
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Update sell message and tests](./phase-01-update-sell-message-and-tests.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- No blocking active plan. Existing coin module plan `plans/260612-1005-coin-module-price-fallback/plan.md` is completed and only provides context.
|
||||
|
||||
## Scope
|
||||
|
||||
- In scope: sell insufficient-holdings message in `internal/modules/coin/handlers.go`; regression tests in `internal/modules/coin/handlers_test.go`.
|
||||
- Out of scope: changing command arguments, portfolio math, price providers, command registration, storage schema, buy wording, docs.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- `/coin_sell 600 BTC` after holding only `$500` worth of BTC says not enough BTC to sell `$600`, shows `$500` available, includes `0.01 BTC @ $50,000.00`, and suggests `$500.00 or less`.
|
||||
- `/coin_sell 10 ETH` with zero ETH says no ETH available to sell and suggests buying ETH first.
|
||||
- Failed sell still does not mutate portfolio.
|
||||
- Existing successful sell response remains unchanged.
|
||||
- `go test ./internal/modules/coin -count=1` and `go test ./... -count=1` pass.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None.
|
||||
@@ -1,84 +0,0 @@
|
||||
# Fix: stats handlers starve Telegram reply of context budget
|
||||
|
||||
Status: DONE (2026-06-25) — implemented, `make vet` + `make test -race` green, code-review DONE (no critical/high).
|
||||
|
||||
## Post-deploy correction (2026-06-25)
|
||||
|
||||
First deploy delivered the reply (original bug fixed) but every ticker showed "(no price)".
|
||||
Cause: parallelizing fetches opened N simultaneous TLS handshakes into an empty connection
|
||||
pool; on the 256MB Lambda (~0.15 vCPU) the CPU-bound handshakes thrashed and each exceeded
|
||||
the 3s timeout. Sequential fetches reuse the pooled KBS/provider connection (one handshake),
|
||||
which is why the original code was sequential by design.
|
||||
|
||||
Fix: reverted stock + coin loops to **sequential** (kept `FetchContext` reply reserve and the
|
||||
3s per-fetch timeout); dropped errgroup (back to indirect); added per-fetch error logging
|
||||
(`stock_fetch_price` / `coin_fetch_price`) to close the swallowed-error diagnostic gap.
|
||||
|
||||
## Problem
|
||||
|
||||
Update handler runs under one 10s ctx (`telegram/webhook.go:81`). Stats handlers reuse
|
||||
that same ctx for both upstream price fetches and the final `sendMessage`. Per-upstream
|
||||
HTTP timeout is also 10s (== handler budget), so one slow upstream drains the deadline and
|
||||
the reply fails: `context deadline exceeded`. Confirmed in CloudWatch for `/stock_stats`
|
||||
(dispatch→error exactly 10.0s; coin/gold succeeded same window — KBS was slow).
|
||||
|
||||
## Remedies (all approved)
|
||||
|
||||
1. **Reserve reply budget** — fetches run under a derived sub-ctx that leaves a reserve for
|
||||
the reply; the reply itself uses the original handler ctx.
|
||||
2. **Lower per-fetch HTTP timeout** 10s → 3s, so one hung upstream can't eat the budget.
|
||||
3. **Parallelize** the per-symbol fetch loops (stock, coin) so total ≈ slowest single fetch.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `/stock_stats`, `/coin_stats` with N holdings: total fetch time bounded by slowest single
|
||||
fetch (~3s), not N×. Reply always delivered if any upstream responds within budget.
|
||||
- A single unresponsive upstream → that line shows "(no price)" / "(price unavailable)";
|
||||
the summary still sends. No more whole-reply `context deadline exceeded`.
|
||||
- `make vet` + `make test` green. No public-contract changes. Output text/order unchanged.
|
||||
|
||||
## Scope
|
||||
|
||||
IN: stock/coin/gold stats handlers; gold price handler; the 4 HTTP-timeout consts.
|
||||
OUT: buy/sell single-fetch handlers (the 3s timeout alone leaves ~7s reply headroom);
|
||||
`incomeEventsHTTPTimeout` (separate command, not a stats loop); webhook 10s budget itself.
|
||||
|
||||
## Changes
|
||||
|
||||
### New shared helper — `internal/modules/util/chathelper/chathelper.go`
|
||||
- `const replyReserve = 3 * time.Second`
|
||||
- `func FetchContext(ctx) (context.Context, context.CancelFunc)` — derive child ctx leaving
|
||||
`replyReserve` before the parent deadline (floor 1s); pass-through if parent has no deadline.
|
||||
|
||||
### `internal/modules/stock/prices.go`
|
||||
- `kbsHTTPTimeout` 10s → 3s.
|
||||
|
||||
### `internal/modules/stock/handlers.go` (`handleStats`)
|
||||
- Build `heldList`, fetch prices concurrently via `errgroup` (SetLimit 8) into an indexed
|
||||
results slice (preserves order), using `chathelper.FetchContext(ctx)`. Reply on original ctx.
|
||||
- Fix the stale "in parallel" comment (now actually parallel).
|
||||
|
||||
### `internal/modules/coin/prices.go` + `internal/modules/coin/views.go` (`handleStats`)
|
||||
- `coinHTTPTimeout` 10s → 3s.
|
||||
- Parallelize the `sortedAssetSymbols` loop the same way; reply on original ctx.
|
||||
|
||||
### `internal/modules/gold/prices.go`, `vnappmob_client.go`, `handlers.go`
|
||||
- `goldHTTPTimeout` + `vnappmobHTTPTimeout` 10s → 3s.
|
||||
- `handleStats` + `handlePrice`: fetch under `chathelper.FetchContext(ctx)`, reply on original ctx.
|
||||
(Single fetch but composite tries providers sequentially → reserve guarantees reply headroom.)
|
||||
|
||||
### `go.mod`
|
||||
- `errgroup` (golang.org/x/sync) moves indirect → direct via `go mod tidy`.
|
||||
|
||||
## Risks / rollback
|
||||
|
||||
- Concurrency: `PriceClient`/coin client HTTP clients are safe for concurrent reuse (shared
|
||||
pool via sync.Once). Result slice written by index → no shared-write race.
|
||||
- 3s may be tight on Lambda cold-start TLS to a slow upstream; mitigated by reserve+parallel
|
||||
and graceful "(no price)" fallback. Revert = restore constants to 10s.
|
||||
- Tests inject HTTP client + URL; lowering real-client timeout doesn't affect injected ones.
|
||||
|
||||
## Validation
|
||||
|
||||
`make vet`, `make test`. Add focused test: stats handler with a hanging upstream returns a
|
||||
reply (not a deadline error) within budget.
|
||||
@@ -1,83 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "MongoDB Storage Provider"
|
||||
status: done
|
||||
priority: P1
|
||||
dependencies: []
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 1: MongoDB Storage Provider
|
||||
|
||||
## Overview
|
||||
|
||||
Add `mongodb` as a 4th `KVProvider`, modeled exactly on the existing `firestore` provider (collection-per-module isolation). Make it the **default backend when `MONGO_URL` is set** — no `KV_PROVIDER` needed. Config: `MONGO_URL` + `MONGO_DATABASE`. `KV_PROVIDER` stays as an optional override (e.g. `memory` for local dev). No module code changes — the `KVStore` interface is the only contract.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: implement `KVStore` (Get/GetJSON/Put/PutJSON/Delete/List) + `CompareAndSwapStore` (CompareAndSwap) backed by MongoDB.
|
||||
- Functional: one Mongo collection per module (mirrors `FirestoreProvider`); document `_id` = key, field `value` = raw bytes, field `updatedAt` = timestamp.
|
||||
- Functional: `List(prefix)` = range/regex query on `_id` with `begins_with` semantics; empty prefix = whole collection.
|
||||
- Non-functional: behavior parity with firestore/dynamodb — same `ErrNotFound`, `ErrConflict`, key validation (reuse `validateKey`/`validatePrefix`), same `collectionNameRe` module-name guard via `invalidStore`.
|
||||
- Non-functional: bounded startup connect timeout (match `dynamodbInitTimeout = 5s` style), graceful `Close()`.
|
||||
|
||||
## Architecture
|
||||
|
||||
Reuse the shared helpers already in `internal/storage`:
|
||||
- `validateKey` / `validatePrefix` (in `firestore_kv.go`) — key constraints. Mongo `_id` has no `/` restriction, but reusing keeps cross-backend parity and is harmless.
|
||||
- `collectionNameRe` (in `firestore_provider.go`) — module-name alphabet.
|
||||
- `invalidStore` (in `invalid_store.go`) — returned for bad module names.
|
||||
|
||||
Document shape:
|
||||
```
|
||||
{ "_id": "<key>", "value": "<JSON string>", "updatedAt": <int64 nanos> }
|
||||
```
|
||||
> **Superseded 2026-06-28** by plan `260628-1113-mongo-native-value-documents`: `value` is now a **native BSON document** + a `version` field (version-based CAS), not a string. The string note below is historical.
|
||||
|
||||
Store `value` as a **BSON string** (decided 2026-06-28) so it is directly readable in the Atlas/Compass UI — mirroring DynamoDB's String storage (`dynamodb_kv.go:88`). Every caller writes JSON (UTF-8 safe); non-UTF-8 callers must encode upstream (e.g. base64), the same constraint DynamoDB carries. On read, accept both string and binary (the binary case is a backward-compat fallback for any docs written by the original binary build). **Store `updatedAt` as int64 unix-nanos (NOT BSON datetime)** — matches DynamoDB exactly (`dynamodb_kv.go:101`), keeps migration faithful, and avoids ms-truncation if a future TTL/sort ever reads it. The migrator (Phase 4) and the provider MUST share one encoding — see Phase 4 (migrator writes through `MongoKVStore.Put`, not raw `UpdateOne`).
|
||||
|
||||
`CompareAndSwap` mapping — the `expected == nil` branch is a LIVE path (first write of every new coin/gold portfolio, `coin/portfolio.go:81-83`, `gold/portfolio.go:62-80`), not an edge case. Map it to a plain **`InsertOne`** and rely SOLELY on the unique `_id` index for the conflict:
|
||||
- `expected == nil` → `InsertOne({_id, value, updatedAt})`; `mongo.IsDuplicateKeyError(err)` → `ErrConflict`. Do NOT use a `{value:{$exists:false}}` upsert filter (it can false-conflict on a value-less doc and muddies the contract).
|
||||
- `expected != nil` → `UpdateOne({_id, value: expected}, {$set:{value,updatedAt}})`; `MatchedCount == 0` → `ErrConflict`.
|
||||
`_id` is unique by default, so the absent-insert race is linearizable: exactly one `InsertOne` wins, losers get duplicate-key → `ErrConflict` → caller retry loop reloads the winner's state. This must be proven by a **blocking** concurrent-writer test (see Success Criteria), not just asserted.
|
||||
|
||||
`List(prefix)`: `Find({_id: {$gte: prefix, $lt: prefixSuccessor(prefix)}}, projection={_id:1})` — reuse the existing `prefixSuccessor` helper from `firestore_kv.go`. Empty prefix → `Find({})`. Avoids regex injection and uses the `_id` index.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/storage/mongodb_client.go` — `NewMongoClient(ctx, uri) (*mongo.Client, error)` with connect+ping timeout; `NewMongoDatabase`. Mirror `dynamodb_client.go`.
|
||||
- Create: `internal/storage/mongodb_provider.go` — `MongoProvider{ db *mongo.Database }`, `For(module)` returns `invalidStore` on bad name else `NewMongoKVStore`. Mirror `firestore_provider.go`.
|
||||
- Create: `internal/storage/mongodb_kv.go` — `MongoKVStore`, all methods. Mirror `firestore_kv.go`.
|
||||
- Create: `internal/storage/mongodb_kv_test.go` + `mongodb_provider_test.go` — parity tests, gated on `MONGODB_TEST_URL` (skip when unset), mirroring `dynamodb_kv_test.go` gating on `DYNAMODB_LOCAL_URL`.
|
||||
- Modify: `cmd/server/main.go` — add `mongodb` case to `buildProvider`; add `MongoURL`, `MongoDatabase` to `config` + `loadConfig` (`MONGO_URL`, `MONGO_DATABASE`). Change the auto-detect default: when `KV_PROVIDER` is empty, pick `mongodb` if `MONGO_URL` is set, else `memory` (the old `AWS_LAMBDA_FUNCTION_NAME → dynamodb` auto-detect is removed — AWS is gone; `dynamodb`/`firestore` remain reachable only via explicit `KV_PROVIDER`, kept for the migrator/tests).
|
||||
- Modify: `go.mod` / `go.sum` — add `go.mongodb.org/mongo-driver/v2`.
|
||||
- Modify: `Makefile` — add `mongo-local` (docker `mongo:7`) + `test-mongo` target gated by `MONGODB_TEST_URL`, mirroring `dynamodb-local`/`test-dynamodb`.
|
||||
- Modify: `README.md` — add `mongodb` to the storage backend list + local-run snippet.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. `go get go.mongodb.org/mongo-driver/v2/mongo` (and `/bson`).
|
||||
2. Write `mongodb_client.go`: connect with `options.Client().ApplyURI(uri)`, `client.Ping` under a 5s context, return client; helper to get `*mongo.Database` from `MONGO_DATABASE`.
|
||||
3. Write `mongodb_kv.go`: implement methods per Architecture; reuse `validateKey`, `validatePrefix`, `prefixSuccessor`; constants `mongoValueField="value"`, `mongoUpdatedAtField="updatedAt"`.
|
||||
4. Write `mongodb_provider.go`: `For` guards with `collectionNameRe`, returns `db.Collection(module)`-backed store.
|
||||
5. Wire `buildProvider`: `case "mongodb"`: require `MONGO_URL` + `MONGO_DATABASE` (error if missing, mirror dynamodb's `DYNAMODB_TABLE` check); construct client under timeout; closer calls `client.Disconnect`. **The startup `log.Info("storage backend", …)` line MUST log only non-secret fields — `"backend","mongodb","database",cfg.MongoDatabase`. NEVER log `MONGO_URL`** (it is `mongodb+srv://user:pass@…`; the firestore/dynamodb cases at `main.go:234-253` log a benign identifier, but the mongo equivalent is a credential). If a host is wanted for diagnostics, parse and log only the host, never the userinfo.
|
||||
6. Add config fields + env reads. Set the auto-detect default to `mongodb` when `MONGO_URL` is present (else `memory`); `KV_PROVIDER` overrides.
|
||||
7. Tests: replicate the firestore/dynamodb test bodies against a real Mongo (`mongo-local`), covering Get/Put/Delete/List/prefix/CAS-absent/CAS-match/CAS-conflict/ErrNotFound + cross-module isolation.
|
||||
8. `make vet && make test && MONGODB_TEST_URL=mongodb://localhost:27017 make test-mongo`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `internal/storage` exposes `MongoProvider`/`MongoKVStore` passing the same test matrix as `DynamoDBKVStore`.
|
||||
- [ ] Setting `MONGO_URL` + `MONGO_DATABASE` (no `KV_PROVIDER`) boots as mongodb; `MONGO_URL` set but `MONGO_DATABASE` missing errors clearly; no env → memory.
|
||||
- [ ] Cross-module isolation verified (collection-per-module).
|
||||
- [ ] CompareAndSwap returns `ErrConflict` on stale expected + on absent-with-non-nil-expected; succeeds on nil-expected insert.
|
||||
- [ ] **(blocking)** Concurrent-writer CAS test: N goroutines race a nil-expected insert + a stale-update on the same key against a real Mongo; assert exactly one winner, losers get `ErrConflict`. Plus a "doc exists without value field" edge case.
|
||||
- [ ] Startup log shows `backend=mongodb database=<db>` and does NOT contain the connection string / any credential.
|
||||
- [ ] `make vet` and `make test` pass; AWS/firestore paths untouched.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **CAS semantics drift**: Mongo upsert race differs from DynamoDB conditional put. Mitigation: unique `_id` + `IsDuplicateKeyError` for the absent case; assert with a concurrent-writer test.
|
||||
- **Value type on read**: driver may decode as binary or string. Mitigation: dual-type switch like firestore's.
|
||||
- **TLS to Atlas**: `mongodb+srv://` URIs need DNS SRV + TLS. Mitigation: driver handles it; document that `MONGO_URL` is the full Atlas SRV connection string incl. credentials.
|
||||
- **Driver version**: v2 API differs from v1 (`mongo.Connect` signature). Mitigation: pin v2, follow current docs.
|
||||
@@ -1,91 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "In-Process Cron Scheduler"
|
||||
status: done
|
||||
priority: P1
|
||||
dependencies: []
|
||||
effort: "S"
|
||||
---
|
||||
|
||||
# Phase 2: In-Process Cron Scheduler
|
||||
|
||||
## Overview
|
||||
|
||||
Off AWS there is no EventBridge Scheduler to hit `/cron/{name}`. The long-lived container runs an in-process scheduler **by default** (no env toggle) that reads each registered cron's existing `Cron.Schedule` field and fires `Cron.Handler` on time. AWS/EventBridge is gone, so there is no "external" mode to preserve — the scheduler simply runs.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: at startup, parse every `reg.Crons()[i].Schedule` and invoke its handler on schedule, in UTC (match the old EventBridge `ScheduleExpressionTimezone: UTC`). No `CRON_MODE` env.
|
||||
- Functional: cutover safety comes from ordering (EventBridge schedule disabled before the container starts, Phase 4) + the idempotency guard below — not from a gate. Locally (`go run`, memory KV) the scheduler also runs; harmless (no subscribers).
|
||||
- Functional: the existing `/cron/{name}` HTTP route stays as-is (still usable for manual/curl triggers and as the sidecar fallback).
|
||||
- Non-functional: each tick runs under the same 60s budget as the HTTP path (`defaultCronTimeout`); a panicking handler is recovered and logged, scheduler keeps running.
|
||||
- Non-functional: scheduler stops cleanly on `rootCtx` cancellation (graceful shutdown).
|
||||
|
||||
## Architecture
|
||||
|
||||
The `Cron` struct already carries `Schedule string` (currently "documentation only", e.g. lolschedule `"0 1 * * *"`). Promote it to the real trigger source for self-host.
|
||||
|
||||
Use `github.com/robfig/cron/v3` (standard, well-maintained 5-field cron parser) with `cron.New(cron.WithLocation(time.UTC))`. For each registered cron, `c.AddFunc(schedule, fn)` where `fn` dispatches the handler through the existing cron dispatcher path so logging/metrics/timeout/panic-recovery are identical to the HTTP route.
|
||||
|
||||
Dispatch: the existing exported helper is **`modules.DispatchScheduled(ctx, name, reg)`** (`cron_dispatcher.go:19` — note arg order: name BEFORE reg). It does ONLY a registry lookup + `cron.Handler(ctx, deps)` — it has **no timeout, no panic-recovery, no structured logging**. That wrapping lives in the server-package `cronHandler` (`internal/server/router.go`), NOT in the dispatcher. So the scheduler must add its own:
|
||||
- wrap each fire in `context.WithTimeout(ctx, 60s)` (the `defaultCronTimeout` constant is in `internal/server/timeouts.go`; do not import `internal/server` into the scheduler — define a local `cronTimeout = 60*time.Second` or lift the constant to a shared package to avoid a layering inversion),
|
||||
- `recover()` around the handler call, logging the cron name on panic so one bad cron doesn't kill the scheduler,
|
||||
- structured `log.Info("cron triggered", …)` / `log.Error("cron failed", …)` mirroring the HTTP path.
|
||||
Call `modules.DispatchScheduled(ctx, name, reg)` inside that wrapper.
|
||||
|
||||
New file `internal/cron/scheduler.go` (new small package, mirrors `internal/metrics` lifecycle style):
|
||||
```
|
||||
func Run(ctx context.Context, reg *modules.Registry) (stop func(), err error)
|
||||
```
|
||||
- Skips crons with empty `Schedule` (logs a warning — a self-hosted cron with no schedule never fires).
|
||||
- Validates schedule strings at startup; a bad expression is fatal (fail fast, like other config errors).
|
||||
|
||||
Wire in `cmd/server/main.go` after `modules.Install` (unconditional):
|
||||
```
|
||||
stop, err := cron.Run(rootCtx, reg)
|
||||
if err != nil { log.Fatal("cron scheduler init failed", "err", err) }
|
||||
defer stop()
|
||||
log.Info("cron scheduler started", "crons", len(reg.Crons()))
|
||||
```
|
||||
|
||||
No `CronMode` config / `CRON_MODE` env.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/cron/scheduler.go` — scheduler lifecycle.
|
||||
- Create: `internal/cron/scheduler_test.go` — fake registry with a fast schedule (`@every 1s` or injected clock) asserts handler fires; bad-schedule errors; ctx-cancel stops.
|
||||
- Modify: `cmd/server/main.go` — unconditional `cron.Run(rootCtx, reg)` after `modules.Install`.
|
||||
- Modify: `internal/modules/module.go` doc comments — update `CronHandler`/`Cron.Schedule` text that currently says "real schedule lives in EventBridge" to note the in-process scheduler drives it. (The scheduler calls the existing `modules.DispatchScheduled`; if the 60s-timeout/panic-recover wrapper is worth sharing with the HTTP `cronHandler`, lift it to a shared helper — optional.)
|
||||
- Modify: `go.mod` / `go.sum` — add `github.com/robfig/cron/v3`.
|
||||
- Modify: `README.md` — note the container runs an in-process scheduler for module crons (no config).
|
||||
- Modify: `internal/modules/lolschedule/cron.go` — add an idempotency guard: the daily-push handler reads/writes a KV "last push UTC date" key and no-ops if already pushed today (defends against all double-fire windows). This also makes the existing EventBridge path safe during cutover overlap.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. `go get github.com/robfig/cron/v3`.
|
||||
2. Call `modules.DispatchScheduled(ctx, name, reg)` (`cron_dispatcher.go:19`) from inside a scheduler-local wrapper that adds the 60s timeout + `recover()` + logging (the dispatcher provides none of these).
|
||||
3. Write `internal/cron/scheduler.go`: build `cron.New(cron.WithLocation(time.UTC))`, register each non-empty schedule, `c.Start()`, return a `stop` that calls `c.Stop()` and waits for the context done.
|
||||
4. Wire gated startup in `main.go`; add `CronMode` config field + env read.
|
||||
5. Update the now-stale "documentation only / EventBridge owns timing" comments on `Cron.Schedule` and `CronHandler`.
|
||||
6. Tests + `make vet && make test`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] lolschedule daily push fires at 01:00 UTC; observable in logs (`cron triggered`).
|
||||
- [ ] Scheduler starts unconditionally at container boot (`cron scheduler started` logged).
|
||||
- [ ] Bad schedule string fails startup with a clear error.
|
||||
- [ ] Handler panic is recovered (scheduler-local `recover()`); scheduler survives and fires next tick.
|
||||
- [ ] Each fire runs under a 60s timeout (scheduler-local, not imported from `internal/server`).
|
||||
- [ ] Daily push is idempotent per UTC date: invoking the handler twice on the same date sends subscribers exactly one digest.
|
||||
- [ ] Scheduler stops within shutdown grace period on SIGTERM.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Double-fire (Critical — the daily push is NOT idempotent)**: `lolschedule` `runDailyPush` (`cron.go:128-191`) fans out to every subscriber unconditionally — no "already sent today" marker. So ANY double-fire = every subscriber DM'd twice. Three concrete double-fire windows the opt-in flag does NOT cover:
|
||||
1. **Cutover overlap**: EventBridge `AWS::Scheduler::Schedule` (`template.yaml:271-289`) invokes the Lambda DIRECTLY, independent of the webhook URL — re-pointing the webhook does NOT stop it. It must be **disabled/deleted before** the Coolify container starts (its scheduler runs by default) — ordered prerequisite in Phase 4, not "N days later" cleanup.
|
||||
2. **Rolling deploy**: Coolify/compose may run old+new containers briefly; both run the in-process scheduler. A redeploy near 01:00 UTC double-fires.
|
||||
3. Operator misconfig (`internal` set on a second instance).
|
||||
**Primary mitigation (covers all three cheaply)**: add a KV "last push date" guard in `lolschedule` — the handler records the date it pushed and no-ops if already pushed for that UTC date. This makes the push idempotent regardless of trigger count. Secondary: prefer stop-first redeploy in Coolify; disable the EventBridge schedule before first container start.
|
||||
- **5-field vs 6-field cron**: `"0 1 * * *"` is 5-field standard. Mitigation: use robfig `cron/v3` default 5-field parser (not the seconds-enabled one).
|
||||
- **Single-instance assumption**: if Coolify scales the service to >1 replica, crons fire per replica. Mitigation: document "run exactly 1 replica" (the bot must be single-instance anyway — Telegram allows only one long-polling consumer per token, Phase 3); revisit with a DB lock only if scaling is ever needed (YAGNI now).
|
||||
- **Missed fire while container restarts**: a deploy at 01:00 UTC could skip that day's push. Mitigation: accept (same risk class as Lambda cold-start miss); not data-loss.
|
||||
-123
@@ -1,123 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Long-Polling Runtime + Containerize + Coolify Deploy"
|
||||
status: code-complete-operator-pending
|
||||
priority: P2
|
||||
dependencies: [1, 2]
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 3: Long-Polling Runtime + Containerize + Coolify Deploy
|
||||
|
||||
## Overview
|
||||
|
||||
Two coupled changes: (1) switch the Telegram transport from webhook to **long polling** so the self-hosted container needs NO public inbound ingress, and (2) ship the existing distroless image via a docker-compose stack on Coolify. MongoDB Atlas is external/managed, so compose runs only the bot service. Config is supplied as Coolify env vars.
|
||||
|
||||
Long polling is the key self-host simplification: the bot opens an OUTBOUND connection to `api.telegram.org` and pulls updates (`getUpdates`), instead of Telegram POSTing to a public URL. That removes the public HTTPS domain, TLS/Traefik routing for inbound, the `/webhook` route, the webhook secret, and the entire unauthenticated-ingress attack surface. The same library (`go-telegram/bot v1.20.0`) does this via `b.Start(ctx)` — no library change, handlers unchanged.
|
||||
|
||||
**Polling is the ONLY transport — no toggle.** Webhook mode existed solely for Lambda, which is being decommissioned (Phase 5), so the webhook code path is removed entirely (YAGNI). The currently-deployed Lambda keeps running its own already-built webhook code until `sam delete`, so the short rollback window in Phase 4 is unaffected by deleting webhook code from this branch.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `docker compose up` runs the bot with persistent storage in Atlas — mongodb is auto-selected from `MONGO_URL` and the cron scheduler runs by default (no `KV_PROVIDER`/`CRON_MODE`).
|
||||
- Functional: bot consumes updates via long polling (`b.Start(ctx)`) as the sole transport; no `/webhook` route, no webhook secret, no public domain. Only OUTBOUND HTTPS to Telegram is needed.
|
||||
- Functional: the webhook code path (`internal/telegram/webhook.go`, the `/webhook` route, `TELEGRAM_WEBHOOK_SECRET` config + its startup fatal) is **deleted**, not gated.
|
||||
- Non-functional: container restart policy `unless-stopped`; **exactly one replica** (Telegram allows only ONE polling consumer per bot — a second poller gets HTTP 409; also cron correctness, Phase 2).
|
||||
- Non-functional: health server stays available internally (`GET /` → `text/plain` `miti99bot ok`) for Coolify's container healthcheck, but is NOT publicly routed.
|
||||
- Non-functional: no secrets committed — all via Coolify env / `.env` (gitignored). Provide `.env.example`.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Telegram transport: long polling (only mode)
|
||||
`go-telegram/bot` runs long polling when you call `b.Start(ctx)` (its `getUpdates` loop). In `cmd/server/main.go`, after `modules.Install`, run `b.Start(rootCtx)` in a goroutine; it returns when `rootCtx` is cancelled (graceful shutdown). Remove the webhook wiring:
|
||||
- Delete `internal/telegram/webhook.go` + `webhook_test.go`, the `/webhook` route in `internal/server/router.go`, and the `WebhookSecret` config field + its `if cfg.WebhookSecret == ""` fatal (`main.go:85-88`).
|
||||
- The HTTP server still runs for the `/` health route only (Coolify healthcheck) — bind it, do not publicly route it. (`/cron` stays mountable but is unused in polling+internal-cron mode and isn't exposed.)
|
||||
- A one-time `deleteWebhook` is required before first poll (else `getUpdates` 409s if a webhook is still set — Phase 4 cutover). Only ONE polling process per bot token (second → 409) → single replica.
|
||||
- `WithSkipGetMe()` (currently set for fast Lambda cold start) is harmless to keep. `WithNotAsyncHandlers` rationale was webhook-specific — re-evaluate, but handlers take `ctx` (not `r.Context()`) so leaving them as-is is safe.
|
||||
|
||||
### Container + Coolify
|
||||
The existing `Dockerfile` (golang:1.25-alpine builder → distroless static nonroot, `ENTRYPOINT ["/server"]`, `EXPOSE 8080`) builds with `-ldflags="-s -w"` only — it does **not** inject `gitSHA`. Only the Makefile injects it (`Makefile:13`). `deploynotify` runs unconditionally at startup (`main.go:148`) and stays silent when `gitSHA` is empty (`main.go:40-41`), so as-is the "new version" owner DM is **silently dead on self-host** — a behavior regression from Lambda. Required (not optional): add `ARG GIT_SHA` + `-ldflags "-s -w -X main.gitSHA=$GIT_SHA"` to the Dockerfile and pass `--build-arg GIT_SHA=$(git rev-parse --short HEAD)` (Coolify exposes the commit SHA as a build var). If deploynotify parity is explicitly unwanted, instead document in acceptance criteria that the feature is disabled on self-host — do not leave it as undocumented breakage.
|
||||
|
||||
`compose.yml` (committed; Coolify consumes it):
|
||||
```yaml
|
||||
services:
|
||||
bot:
|
||||
build: . # or image: ghcr.io/tiennm99/miti99bot:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MONGO_URL: ${MONGO_URL}
|
||||
MONGO_DATABASE: ${MONGO_DATABASE}
|
||||
MODULES: ${MODULES}
|
||||
OWNER_ID: ${OWNER_ID}
|
||||
ADMIN_IDS: ${ADMIN_IDS}
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
|
||||
GEMINI_API_KEY: ${GEMINI_API_KEY}
|
||||
# Storage auto-selects mongodb because MONGO_URL is set (no KV_PROVIDER needed).
|
||||
# The in-process cron scheduler runs by default (no CRON_MODE needed).
|
||||
# PORT defaults to 8080 (health server) — omit unless overriding.
|
||||
# Long polling = no TELEGRAM_WEBHOOK_SECRET, no /webhook, no CRON_SHARED_SECRET.
|
||||
# Do NOT set any *_PARAMETER_NAME vars (see Secrets below).
|
||||
# No stock/coin/gold *_API_URL overrides — modules use their coded default
|
||||
# providers (stock: SSI/FireAnt; coin: Binance→Coinbase→CoinGecko; gold: VNAppMob→spot).
|
||||
# No published ports / domain: polling is outbound-only, nothing inbound to route.
|
||||
# `expose` (internal-only) is enough for Coolify's container healthcheck on /.
|
||||
expose: ["8080"]
|
||||
# No compose healthcheck: distroless has no shell/curl AND cmd/server has no
|
||||
# flag parsing (no -healthcheck flag exists). Use Coolify's container/HTTP monitor
|
||||
# against GET / (returns text/plain "miti99bot ok", NOT JSON).
|
||||
```
|
||||
|
||||
Healthcheck: distroless has no shell/curl/wget, and `cmd/server` has no flag parsing — `/server -healthcheck` does NOT exist and would just start the server. Use **Coolify's native HTTP monitor against `/`** (returns `text/plain` body `miti99bot ok`, per `internal/server/health.go` — not JSON). Only if a compose-level healthcheck is mandatory, first implement a real `-healthcheck` flag in `cmd/server` (localhost GET `/`, exit 0/1) — otherwise omit it. Note: a plain `/` check does NOT verify Mongo connectivity (see Risk Assessment).
|
||||
|
||||
Secrets: because `*_PARAMETER_NAME` env vars are NOT set, `cmd/server` reads `TELEGRAM_BOT_TOKEN`, `GEMINI_API_KEY` directly (verified: `resolveSSMSecrets` returns early when no param names are set, `main.go:355-366`). `TELEGRAM_WEBHOOK_SECRET` is removed with the webhook code (no longer read), so the `if cfg.WebhookSecret == ""` fatal (`main.go:85-88`) is deleted — the bot boots without it. `CRON_SHARED_SECRET` is unused (internal cron) and left unset. **Hard requirement: `.env.example` must list all six `*_PARAMETER_NAME` vars explicitly as "leave UNSET for self-host."** `resolveSSMSecrets` is called unconditionally (`main.go:79`); if even one `*_PARAMETER_NAME` is set with an unset target, it calls `awsconfig.LoadDefaultConfig` + `ssm.GetParameters` (`main.go:371-378`) which fails with no AWS creds → `log.Fatal` (`main.go:80`), bricking startup. With all six unset, no AWS credentials are needed in the container.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `cmd/server/main.go` — `go b.Start(rootCtx)` after `modules.Install`; delete the `WebhookSecret` config field, its load, and the `if cfg.WebhookSecret == ""` fatal (`main.go:85-88`). Rename the env keys read in `loadConfig`: `BOT_OWNER_ID` → `OWNER_ID`, `ADMIN_USER_IDS` → `ADMIN_IDS` (no backward-compat — AWS template that set the old names is being deleted).
|
||||
- Delete: `internal/telegram/webhook.go` + `internal/telegram/webhook_test.go`.
|
||||
- Modify: `internal/server/router.go` — remove the `/webhook` route; keep `/` health (and `/cron`, unused/unexposed).
|
||||
- Modify: `internal/telegram/client.go` — update the "configured for webhook mode" doc to polling; keep `WithSkipGetMe`, re-evaluate `WithNotAsyncHandlers`.
|
||||
- Create: `compose.yml` — bot service as above.
|
||||
- Create: `.env.example` — every env var with placeholder values + comments; real `.env` gitignored.
|
||||
- Modify: `.gitignore` — ensure `.env` ignored (verify; add if missing).
|
||||
- Modify (optional): `Dockerfile` — `ARG GIT_SHA` + `-ldflags "-X main.gitSHA=$GIT_SHA"` for deploynotify parity.
|
||||
- Modify (optional): `cmd/server/main.go` — `-healthcheck` flag (only if compose healthcheck chosen over Coolify HTTP monitor).
|
||||
- Create: `docs/deploy-coolify-selfhosted.md` — full onboarding: create Atlas M0 cluster, get SRV URL, create a least-privilege DB user (`readWrite` on one DB, strong unique password), set network access to `0.0.0.0/0` (accepted trade-off — record it), Coolify new resource from compose/Git, set env vars, deploy (no public domain needed — polling is outbound-only), one-time `deleteWebhook`, register the command menu via `setMyCommands`.
|
||||
- Modify: `README.md` — add "Self-host (Coolify + MongoDB Atlas)" deploy option alongside the AWS path; drop the stock income-events row from the module table.
|
||||
- Delete: `internal/modules/stock/income_events.go` + `income_events_test.go`; remove the `stock_income_events` command registration (`stock/stock.go:45-48`), the `incomeEvents *IncomeEventClient` field + `NewIncomeEventClientFromEnv` wiring (`stock/handlers.go:25,43`), `handleIncomeEvents`, and the `STOCK_INCOME_EVENTS_API_URL`/`_TOKEN` config + `exportOptionalEnv` lines in `main.go`. Keep `stock_income_stock` / `stock_income_vnd`.
|
||||
- Modify: `aws/telegram-commands.json` (the `setMyCommands` source) — remove the `stock_income_events` entry so the command menu matches.
|
||||
- No change for gold: `gold/vnappmob_client.go` already auto-fetches + caches the VNAppMob key in KV (`vnappmob:api_key`); just leave `GOLD_VNAPP_API_KEY` unset.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Write `compose.yml` + `.env.example`; verify `.env` gitignored.
|
||||
2. Decide healthcheck approach (Coolify HTTP monitor preferred); implement `-healthcheck` flag only if needed.
|
||||
1b. Switch `cmd/server/main.go` to `b.Start(rootCtx)`; delete the webhook handler, `/webhook` route, and `WebhookSecret` config + fatal.
|
||||
3. Local validation: `MONGO_URL=… MONGO_DATABASE=… docker compose up --build`; confirm boot logs show `storage backend backend=mongodb database=…` (NO connection string), `internal cron scheduler started`, and the polling loop started (getUpdates); `curl localhost:8080/` returns `miti99bot ok`. (Requires the bot's webhook to be unset — see Phase 4 / `deleteWebhook`, else getUpdates 409s.)
|
||||
4. Atlas setup (M0): cluster, DB user, network access, copy `mongodb+srv://…` SRV URL.
|
||||
5. Coolify: create resource (Git repo + compose, or prebuilt image), set env vars, deploy. No public domain/route needed (outbound-only); ensure exactly 1 replica.
|
||||
6. Command menu only: `setMyCommands` (still an HTTPS POST to Telegram). `make telegram-commands` currently pulls the token from SSM — for self-host, add an env-var/token-arg variant or document the direct curl. No webhook registration step exists in polling mode.
|
||||
7. Smoke test: send `/ping`, `/help`; confirm a persisted command (e.g. stock paper trade) survives a container restart.
|
||||
8. Write `docs/deploy-coolify-selfhosted.md`; update README.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `docker compose up --build` boots the bot with mongo + internal cron + polling locally; logs show the getUpdates loop running.
|
||||
- [ ] Coolify deployment runs with NO public domain/port (outbound-only); `/` health passes internally; restarts `unless-stopped`; exactly 1 replica.
|
||||
- [ ] Bot receives messages via long polling (no webhook set); live commands respond.
|
||||
- [ ] Webhook code removed (`webhook.go`, `/webhook` route, `WebhookSecret` config/fatal gone); bot boots with no `TELEGRAM_WEBHOOK_SECRET`.
|
||||
- [ ] No secret committed; `.env.example` documents every variable, lists all six `*_PARAMETER_NAME` as "leave UNSET", and omits `CRON_SHARED_SECRET` by default.
|
||||
- [ ] Container boots with zero `*_PARAMETER_NAME` set (no AWS creds present) and never attempts an SSM/AWS call.
|
||||
- [ ] `deploynotify` parity decided: either `gitSHA` injected via Dockerfile build arg (owner DM works) or the feature is documented as disabled on self-host.
|
||||
- [ ] `docs/deploy-coolify-selfhosted.md` is complete enough to redo from scratch.
|
||||
- [ ] No stock/coin/gold `*_API_URL` overrides set; modules work on coded defaults. `stock_income_events` command + FireAnt client removed; command catalog updated; `make vet`/`make test` green after removal.
|
||||
- [ ] Gold works with `GOLD_VNAPP_API_KEY` unset (auto-fetches + caches the VNAppMob key to Mongo at `vnappmob:api_key`).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Distroless healthcheck**: no shell, and no `-healthcheck` flag exists. Mitigation: use Coolify's HTTP monitor against `/` (text body `miti99bot ok`), or implement the flag first. Do not ship the bogus `["CMD","/server","-healthcheck"]` line.
|
||||
- **Atlas network access — `0.0.0.0/0` accepted (validated decision)**: the Coolify host has no stable egress IP, so the Atlas IP access list is `0.0.0.0/0` (internet-reachable). This widens the surface beyond the project's documented "no non-designed-surface public resource" boundary (`docs/deploy-aws-free-tier-guide.md:25,35`) — under DynamoDB the DB was IAM-gated and never internet-reachable — and is knowingly accepted for self-host. **Mandatory compensating controls (hard requirements):** (1) a strong unique DB password, (2) a least-privilege Atlas DB user scoped to `readWrite` on the single app database (never Atlas admin / cluster-wide), (3) the connection string is treated as a secret and never logged (see Phase 1 / Finding-3 constraint). Record the acceptance in `docs/deploy-coolify-selfhosted.md`. <!-- Updated: Validation Session 1 - 0.0.0.0/0 accepted; least-priv user + strong password required -->
|
||||
- **No public ingress (resolved by polling)**: long polling is outbound-only, so there is NO public domain or `/webhook` to flood — the entire unauthenticated-ingress attack surface that webhook mode created is gone. (This supersedes the earlier "public ingress is the bot's own process" risk.) Residual: the `/` health server should stay internal (Coolify `expose`, not `ports`); never publish it.
|
||||
- **Single polling consumer (409 conflict)**: Telegram permits only ONE `getUpdates` consumer per bot token. Two replicas, OR a leftover webhook still set, OR an overlapping old instance during redeploy → HTTP 409 / lost updates. Mitigation: exactly 1 replica; `deleteWebhook` before first poll (Phase 4); prefer stop-first redeploy (also covers the Phase 2 cron-overlap concern).
|
||||
- **Mongo runtime connection loss (Medium)**: unlike the per-call short-lived DynamoDB client (`dynamodb_client.go`), self-host holds one long-lived Mongo client for days; Atlas M0 idles/fails over. The v2 driver auto-reconnects on the next op (confirm against current driver docs) — set sane pool + server-selection timeout options. A plain `/` healthcheck reports healthy even when Mongo is unreachable, so Coolify won't restart a DB-wedged bot; either make the healthcheck DB-aware (lightweight `Ping`) or explicitly accept "stays up reporting healthy during DB outage" as the chosen trade-off.
|
||||
- **Telegram tooling assumes SSM**: `make telegram-commands` reads the token from SSM. Mitigation: document an env-var/token-arg variant for self-host (only `setMyCommands` is needed in polling mode; no webhook registration).
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Data Migration and Cutover"
|
||||
status: code-complete-operator-pending
|
||||
priority: P1
|
||||
dependencies: [1, 3]
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 4: Data Migration and Cutover
|
||||
|
||||
## Overview
|
||||
|
||||
Copy all existing items from the prod DynamoDB table (`miti99bot-data`) into MongoDB Atlas using the finalized Phase 1 document schema, verify parity, then switch Telegram from the Lambda webhook to the long-polling container (Phase 3). Idempotent and re-runnable.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: every DynamoDB item → one Mongo document in collection `pk`, `_id = sk`, `value` = decoded value, `updatedAt` preserved.
|
||||
- Functional: idempotent (re-run overwrites by `_id`, no duplicates); resumable on failure.
|
||||
- Functional: verification step compares per-module (per-`pk`) counts DynamoDB vs Mongo and spot-checks values.
|
||||
- Non-functional: read-only on DynamoDB (Scan only); never mutates source.
|
||||
- Non-functional: dry-run mode that reports counts without writing.
|
||||
|
||||
## Architecture
|
||||
|
||||
Mapping (confirmed from `dynamodb_kv.go` ↔ `firestore_kv.go`):
|
||||
|
||||
| DynamoDB | Mongo |
|
||||
|---|---|
|
||||
| `pk` (S) = module name | collection name |
|
||||
| `sk` (S) = user key | document `_id` |
|
||||
| `value` (S) = JSON string | `value` field (BSON binary, same bytes) |
|
||||
| `updatedAt` (N) = unix nanos | `updatedAt` (int64 nanos, verbatim) |
|
||||
|
||||
A full-table `Scan` (table is small KV) yields all items across all partitions. Group by `pk` → write into the matching collection. Read with `storage.NewDynamoDBClient`; **write through the Phase 1 `MongoKVStore.Put` (NOT raw `UpdateOne`)** so `value`/`updatedAt` encoding is byte-identical to what the live app writes — otherwise the migrator could store `value` as a BSON string while the app writes binary, and a later CAS (`{value: expected}`) never matches → spurious `ErrConflict` and the "idempotent re-run" criterion silently breaks.
|
||||
|
||||
New one-off CLI `cmd/migrate-dynamo-to-mongo/main.go`:
|
||||
- Flags/env: `--dynamodb-table` (default `miti99bot-data`), `MONGO_URL`, `MONGO_DATABASE`, `--dry-run`, `--verify`.
|
||||
- Uses AWS default cred chain via a **dedicated read-only profile/role** (see IAM below) — NOT an admin profile.
|
||||
- For each item: **`validateKey(sk)` before writing**; fail loudly on any `/`-containing or otherwise-invalid `sk`. The app's read path runs `validateKey` first (firestore/dynamodb pattern), so a key the migrator writes but `validateKey` rejects would be silently unreadable. No `/` keys exist today (all modules use `:` separators), so this is a guard, not a transform.
|
||||
- Writes via `MongoKVStore.Put`; `Put` overwrites by `_id`, so re-runs are idempotent and produce no duplicates.
|
||||
- `--verify`: per-`pk` count on DynamoDB via a **Scan tally** (so a single `dynamodb:Scan` permission suffices — do NOT use Query, which would need `dynamodb:Query` and over-grant) vs `CountDocuments` per collection; print a table + mismatches; exit non-zero on mismatch.
|
||||
|
||||
IAM (least privilege): exactly `dynamodb:Scan` on the specific table ARN (`arn:…:table/miti99bot-data`), nothing else. No write actions on the source — enforces the read-only requirement and removes the destructive-credential foot-gun.
|
||||
|
||||
Cutover sequence (documented runbook — zero-loss):
|
||||
1. Deploy the Coolify container (Phase 3, polling — the only transport), but keep it **stopped/scaled-to-0** for now (a running poller would 409 against the live Lambda webhook, and would start serving before data is migrated, and its always-on scheduler would overlap EventBridge). Fresh/empty Atlas DB.
|
||||
2. **Disable/delete the EventBridge `LolscheduleDailyPushSchedule`** (`template.yaml:271-289`) — it invokes the Lambda directly, independent of transport, so it must be stopped before any internal scheduler runs. (The Phase 2 last-push-date guard is the belt-and-braces backup.)
|
||||
3. `deleteWebhook` (MANDATORY) — does double duty: (a) Telegram now buffers incoming updates up to 24h, making the cut lossless (do NOT take an "accept the gap" path — a `/buy` in the gap would write to DynamoDB only and be lost, `coin/portfolio.go:64-90`); (b) it releases the webhook so the polling container's `getUpdates` won't 409. After this the Lambda stops receiving updates.
|
||||
4. Run migrator `--dry-run` → review counts. Then run for real; run `--verify` (counts equal per `pk`, exit 0). Keep this window short (target minutes).
|
||||
5. Start the Coolify container (its in-process scheduler runs by default — safe now that EventBridge is disabled). Exactly 1 replica (single polling consumer).
|
||||
6. The container's `getUpdates` loop drains Telegram's buffered queue automatically — no `setWebhook`, no public URL. Confirm in logs that updates are being received.
|
||||
7. Verify `getWebhookInfo` shows `url` empty (webhook cleared) and `pending_update_count` draining toward 0 as the poller consumes the backlog (same queue signal as `docs/deploy-aws.md:77`).
|
||||
8. Smoke `/ping`, `/stats`, a coin/stock balance command — confirm migrated state is visible from Atlas.
|
||||
9. **Tear down AWS** (validated decision): after `--verify` passes and smoke tests are green, `sam delete` the stack (Lambda, DynamoDB, EventBridge, SQS, etc.) and disable the GitHub Actions `deploy.yml` workflow. The user coordinates users to avoid activity during the brief cutover window, so the simple cut is lossless; no reverse migrator is built. <!-- Updated: Validation Session 1 - tear down AWS after verify; no reverse migrator -->
|
||||
10. **Last clean revert point:** rollback to AWS is only possible BEFORE `sam delete`. To revert during the observation window: stop the polling container, then re-`setWebhook` to the Lambda Function URL — the still-deployed Lambda runs its own already-built webhook code (this branch's webhook removal doesn't touch the live function until `sam delete`). Lossless only until the first post-cutover Mongo write. After `sam delete`, MongoDB/Coolify is the sole system of record. <!-- Updated: Validation Session 2 - polling cutover -->
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `cmd/migrate-dynamo-to-mongo/main.go` — the migrator.
|
||||
- Create: `cmd/migrate-dynamo-to-mongo/README.md` (or section in deploy doc) — usage, required IAM (`dynamodb:Scan` on the table), env vars, dry-run/verify, cutover runbook.
|
||||
- Modify: `Makefile` — `migrate-dynamo-to-mongo` + `migrate-verify` targets wrapping the CLI with sensible defaults.
|
||||
- Modify: `docs/deploy-coolify-selfhosted.md` — link the cutover runbook.
|
||||
- Reference: prior `plans/260515-2250-cf-data-to-aws-migration/` — same migration shape (Firestore→DynamoDB); reuse its verification approach if present.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Implement the migrator using existing storage clients; default table `miti99bot-data`.
|
||||
2. Implement `--dry-run` (scan + group + report counts, no writes) and `--verify`.
|
||||
3. Test locally: seed DynamoDB Local with a few items across 2+ modules, run against Mongo Local, assert documents match via the Phase 1 `MongoKVStore.Get`.
|
||||
4. Add Make targets.
|
||||
5. Dry-run against prod DynamoDB; review counts per module.
|
||||
6. Execute migration + verify; record the count table in the cutover doc.
|
||||
7. `deleteWebhook`, start the polling container, smoke-test, monitor.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Dry-run reports per-module item counts without writing.
|
||||
- [ ] Real run copies all items; `--verify` shows DynamoDB and Mongo counts equal for every `pk`, exit 0.
|
||||
- [ ] Re-running the migrator produces no duplicates and no value changes (idempotent).
|
||||
- [ ] Migrator runs `validateKey` per `sk` and writes through `MongoKVStore.Put`; a `Get` round-trip spot check returns byte-identical values.
|
||||
- [ ] After cutover (polling live, webhook cleared), a previously-stored value (e.g. a user's paper-trade balance) is returned by the live bot from Atlas.
|
||||
- [ ] EventBridge schedule disabled before the container starts; daily push fires exactly once on cutover day.
|
||||
- [ ] `getWebhookInfo` shows the webhook URL empty and the polling container is the sole update consumer (no 409).
|
||||
- [ ] Rollback truth documented: reverting to the Lambda webhook is lossless only before the first post-cutover Mongo write (RPO stated explicitly).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Rollback bound — accepted (validated decision)**: AWS is torn down after verify, so there is no long-term fallback by design and no reverse migrator is built. The user coordinates users to pause around the brief cutover window, so the simple cut loses no writes. The only clean-revert opportunity is the short observation window BEFORE `sam delete` (stop the poller, re-`setWebhook` to Lambda), and even then it is lossless only until the first post-cutover Mongo write. This is an accepted RPO for a personal paper-trading bot, not an open risk. <!-- Updated: Validation Session 1 - accepted; teardown after verify -->
|
||||
- **Migration window write-loss**: the user pauses user activity during cutover (coordinated), and the mandatory `deleteWebhook` buffers any stray updates (Telegram retains ~24h). The "accept the gap" option is removed. Keep the window short and confirm `pending_update_count` drains post-flip.
|
||||
- **Cron double-fire during cutover**: EventBridge schedule is disabled BEFORE the container starts (runbook step 2); the Phase 2 last-push-date guard is the backup. Without both, subscribers get the daily push twice (`lolschedule/cron.go:128-191` is non-idempotent).
|
||||
- **`updatedAt` type**: store as int64 unix-nanos in Mongo (matches DynamoDB `dynamodb_kv.go:101`) — exact parity, no truncation. No code reads `updatedAt` today (write-only), so this is cheap insurance against a future TTL/sort.
|
||||
- **Value/encoding mismatch**: migrator writes through `MongoKVStore.Put` (same encoding as the app), not raw `UpdateOne` — guarantees byte-identical `value` and keeps re-runs + CAS correct. Spot-check with a `Get` round-trip.
|
||||
- **Key validity asymmetry**: migrator runs `validateKey(sk)` before writing and fails loud on any rejected key, so it never writes data the app's read path can't load.
|
||||
- **IAM for Scan**: dedicated read-only profile/role with exactly `dynamodb:Scan` on the table ARN. No admin profile, no write actions on the source.
|
||||
- **Large value / 16MB BSON cap**: KV values are tiny JSON; far under limit. No action.
|
||||
@@ -1,139 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "AWS Full Decommission"
|
||||
status: code-complete-operator-pending
|
||||
priority: P1
|
||||
dependencies: [4]
|
||||
effort: "S"
|
||||
---
|
||||
|
||||
# Phase 5: AWS Full Decommission
|
||||
|
||||
## Overview
|
||||
|
||||
After migration + cutover verify (Phase 4), delete **everything** miti99bot ever deployed to AWS, plus verify/clean the legacy pre-AWS services (Cloudflare, GCP — section D). The trap: `sam delete` only removes CloudFormation-managed resources — the GitHub OIDC IAM role, the SSM Parameter Store secrets, and SAM's own S3 deploy bucket were created **manually outside CloudFormation** (`aws/README.md` steps 2-4) and must be deleted separately or they linger (and the secrets keep your bot token/Gemini key sitting in the cloud). GitHub itself is already clean (verified: no stored secrets/keys — AWS deploy used OIDC only).
|
||||
|
||||
This phase is informed by a git-history scan of every `template.yaml` version + the `aws/` setup docs, **verified against the live account on 2026-06-27 (admin profile)**. Account `225603493174`, region `ap-southeast-1`.
|
||||
|
||||
**Live verification (2026-06-27, read-only):** stack `miti99bot` = `UPDATE_COMPLETE`. The account has exactly TWO CloudFormation stacks (`miti99bot` + `aws-sam-cli-managed-default`) and ONE OIDC provider — so **miti99bot is the sole SAM project AND the sole OIDC consumer**, which makes the shared-resource deletes (OIDC provider, SAM bucket) safe and unconditional. SSM holds exactly the 4 known secrets (no FireAnt/VNAppMob extras). SAM deploy bucket: `aws-sam-cli-managed-default-samclisourcebucket-ctwpsmoxnwvm`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: zero miti99bot resources remain in AWS after this phase; Cost Explorer trends to $0.
|
||||
- Functional: run ONLY after Phase 4 `--verify` passes and the bot is confirmed live on Coolify (DynamoDB holds the only copy of prod data until migrated).
|
||||
- Non-functional: do NOT delete account-shared resources (OIDC provider, SAM bucket) if other stacks/repos use them — verify first.
|
||||
- Non-functional: executed by the user with the `admin` profile (this session has no AWS credentials); plan provides the exact commands.
|
||||
|
||||
## Architecture — Complete Resource Inventory
|
||||
|
||||
### A. CloudFormation-managed → removed by `sam delete --stack-name miti99bot`
|
||||
From the current `template.yaml`:
|
||||
- `AWS::DynamoDB::Table` — `miti99bot-data` (**destroyed — must be migrated first**)
|
||||
- `AWS::Serverless::Function` — `miti99bot` + its Function URL + the LWA layer reference
|
||||
- `AWS::Logs::LogGroup` — `/aws/lambda/miti99bot` (+ `AWS::Logs::MetricFilter` ColdStartInitDuration)
|
||||
- `AWS::SQS::Queue` — `miti99bot-cron-dlq`
|
||||
- `AWS::IAM::Role` — `SchedulerExecutionRole` AND the SAM-auto-generated Lambda execution role `miti99bot-BotFunctionRole-*` (both CFN-managed; `sam delete` removes both — no separate action) + the public Function-URL invoke `AWS::Lambda::Permission`s
|
||||
- `AWS::Scheduler::Schedule` — `miti99bot-lolschedule-daily-push`
|
||||
- `AWS::Budgets::Budget` — `miti99bot-monthly` (only if AlertEmail was set)
|
||||
|
||||
Historical (git history shows earlier `template.yaml` used these, since replaced by Scheduler): `AWS::Events::Rule`, `AWS::Events::Connection`, `AWS::Events::ApiDestination`. CloudFormation deleted them when the template changed, so they are NOT orphans — but a post-delete tag sweep (step 6) confirms.
|
||||
|
||||
### B. Created manually OUTSIDE CloudFormation → `sam delete` does NOT touch these
|
||||
From `aws/README.md`:
|
||||
- **SSM Parameter Store SecureStrings** (`aws/README.md:23-39`, verified — exactly these 4, no extras): `/miti99bot/prod/telegram-bot-token`, `/miti99bot/prod/telegram-webhook-secret`, `/miti99bot/prod/gemini-api-key`, `/miti99bot/prod/cron-shared-secret`. **These hold live secrets — deleting them is the security-relevant step.**
|
||||
- **IAM role** `github-deploy-miti99bot` + inline policy `miti99bot-deploy` (`aws/README.md:57-73`).
|
||||
- **IAM OIDC identity provider** `token.actions.githubusercontent.com` (`aws/README.md:43-53`) — verified as the account's ONLY OIDC provider and used solely by miti99bot → **safe to delete.**
|
||||
- **SAM managed S3 deploy bucket** `aws-sam-cli-managed-default-samclisourcebucket-ctwpsmoxnwvm` + its bootstrap stack `aws-sam-cli-managed-default` — verified miti99bot is the sole SAM project → **safe to delete the bucket + bootstrap stack.**
|
||||
- **Bootstrap `admin` IAM user + access keys** (`aws/README.md:15`) — user's discretion; out of scope unless they want a full account wipe.
|
||||
|
||||
### C. Not deletable / self-expiring (no action)
|
||||
- Custom CloudWatch metric namespace `miti99bot` (metrics age out; not billable, not deletable).
|
||||
|
||||
### D. Legacy services from the pre-AWS lineage (verify + clean — not AWS)
|
||||
The bot's history is **Cloudflare Worker (original) → [Cloud Run port, abandoned] → AWS → Coolify (this plan)**. Two legacy footprints to confirm:
|
||||
- **Cloudflare (verified live 2026-06-27, account `miti99` / `7774466151858e13a3c482af5f9ccd6b`):**
|
||||
- **KV namespaces**: only `claude-status` remains — NO miti99bot KV → the legacy bot KV namespace was already deleted. ✅ no action.
|
||||
- **D1 databases**: none → legacy `trading_trades` D1 already deleted. ✅ no action.
|
||||
- **Workers**: 4 exist — `claude-status-webhook`, `rplace`, `miti-loki` (Grafana Loki log-shipper), and `miti-telegram`. All confirmed (user, 2026-06-27) as separate active projects, NOT the legacy miti99bot. `miti-telegram` is a different single-chat notifier the user still uses.
|
||||
- **Conclusion: Cloudflare is fully clean — NO action.** The legacy miti99bot Worker was already deleted during the May 2026 migration; only its data stores (KV/D1) were ever in scope and both are gone.
|
||||
- **GCP project `miti99bot-prod`** — almost certainly never created (Cloud Run port superseded before phase-01 ran; no bootstrap artifacts ever existed). Optional certainty check: `gcloud projects list | grep miti99bot` → if a project exists, `gcloud projects delete miti99bot-prod`. Low probability.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `docs/aws-decommission-runbook.md` — the ordered teardown commands below (so it's repeatable + auditable).
|
||||
- Modify: `.github/workflows/deploy.yml` — **delete or disable** (the AWS deploy path is retired; otherwise a push to `main` re-creates the stack). On the `feature/selfhosted` branch this file is replaced by the Coolify flow.
|
||||
- Modify: `README.md` / `docs/deploy-aws*.md` — mark the AWS deployment path as retired, point to the Coolify guide.
|
||||
- Keep (do not delete): `aws/` dir + `template.yaml` in git history — useful if AWS is ever revisited; they cost nothing as files.
|
||||
|
||||
## Implementation Steps (runbook — user runs with `admin` profile)
|
||||
|
||||
Precondition: Phase 4 done — data migrated, `--verify` green, bot live on Coolify, Telegram webhook pointed at Coolify, EventBridge schedule already disabled at cutover.
|
||||
|
||||
```sh
|
||||
AWS_PROFILE=admin; REGION=ap-southeast-1; ACCT=225603493174
|
||||
|
||||
# 1. Final safety check — confirm bot is NOT serving from AWS anymore
|
||||
aws --profile $AWS_PROFILE cloudformation describe-stacks --stack-name miti99bot \
|
||||
--query "Stacks[0].StackStatus" # exists → about to be deleted
|
||||
|
||||
# 2. Delete the CloudFormation stack (DynamoDB, Lambda, FunctionUrl, Logs, SQS, Scheduler, IAM role, Budget)
|
||||
aws --profile $AWS_PROFILE sam delete --stack-name miti99bot --region $REGION --no-prompts
|
||||
# (or: aws cloudformation delete-stack --stack-name miti99bot ; then wait)
|
||||
aws --profile $AWS_PROFILE cloudformation wait stack-delete-complete --stack-name miti99bot
|
||||
|
||||
# 3. Delete SSM secrets (NOT managed by CFN)
|
||||
aws --profile $AWS_PROFILE ssm get-parameters-by-path --path /miti99bot --recursive \
|
||||
--query "Parameters[].Name" --output text # list what exists first
|
||||
for P in telegram-bot-token telegram-webhook-secret gemini-api-key cron-shared-secret; do
|
||||
aws --profile $AWS_PROFILE ssm delete-parameter --name /miti99bot/prod/$P
|
||||
done
|
||||
# delete any extra /miti99bot/* the list in step 3 revealed
|
||||
|
||||
# 4. Delete the GitHub deploy IAM role (inline policy first, then role)
|
||||
aws --profile $AWS_PROFILE iam delete-role-policy \
|
||||
--role-name github-deploy-miti99bot --policy-name miti99bot-deploy
|
||||
aws --profile $AWS_PROFILE iam delete-role --role-name github-deploy-miti99bot
|
||||
|
||||
# 5. OIDC provider — verified sole consumer = miti99bot, safe to delete.
|
||||
# (Re-confirm it's still the only one before deleting, in case the account changed.)
|
||||
aws --profile $AWS_PROFILE iam list-open-id-connect-providers
|
||||
aws --profile $AWS_PROFILE iam delete-open-id-connect-provider \
|
||||
--open-id-connect-provider-arn arn:aws:iam::$ACCT:oidc-provider/token.actions.githubusercontent.com
|
||||
|
||||
# 6. SAM S3 deploy bucket + bootstrap stack — verified miti99bot is the sole SAM project.
|
||||
# (Re-confirm only miti99bot + aws-sam-cli-managed-default stacks exist first.)
|
||||
aws --profile $AWS_PROFILE cloudformation list-stacks \
|
||||
--query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].StackName" --output text
|
||||
aws --profile $AWS_PROFILE s3 rb \
|
||||
s3://aws-sam-cli-managed-default-samclisourcebucket-ctwpsmoxnwvm --force
|
||||
aws --profile $AWS_PROFILE cloudformation delete-stack --stack-name aws-sam-cli-managed-default
|
||||
|
||||
# 7. Verify nothing tagged app=miti99bot remains
|
||||
aws --profile $AWS_PROFILE resourcegroupstaggingapi get-resources \
|
||||
--tag-filters Key=app,Values=miti99bot --region $REGION
|
||||
aws --profile $AWS_PROFILE cloudformation list-stacks \
|
||||
--query "StackSummaries[?contains(StackName,'miti99bot')].[StackName,StackStatus]" --output table
|
||||
```
|
||||
|
||||
Then in the repo (on `feature/selfhosted`): remove/disable `.github/workflows/deploy.yml` and mark AWS docs retired.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `cloudformation describe-stacks --stack-name miti99bot` → does not exist (DELETE_COMPLETE).
|
||||
- [ ] No `/miti99bot/*` SSM parameters remain (secrets purged from cloud).
|
||||
- [ ] `github-deploy-miti99bot` role gone; OIDC provider gone OR confirmed still needed by another project.
|
||||
- [ ] SAM bucket emptied/deleted OR `miti99bot/` prefix cleared (and rationale recorded).
|
||||
- [ ] `resourcegroupstaggingapi get-resources` for `app=miti99bot` returns empty.
|
||||
- [ ] `.github/workflows/deploy.yml` removed/disabled so `main` pushes no longer recreate the stack.
|
||||
- [ ] Cost Explorer shows $0 for the following billing period.
|
||||
- [x] Cloudflare verified clean (2026-06-27): KV + D1 already gone; all 4 Workers confirmed as separate active projects (no legacy bot remnant). No action.
|
||||
- [ ] GCP checked: no `miti99bot-prod` project exists (or deleted).
|
||||
- [ ] GitHub confirmed clean: no AWS secrets/keys stored; `deploy-aws.yml` removed (only OIDC was used, nothing to revoke beyond the role deleted above).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Deleting before migration completes (Critical)**: `sam delete` destroys `miti99bot-data` (DynamoDB). Mitigation: hard dependency on Phase 4 `--verify`; runbook step 1 is an explicit precondition check. Never run Phase 5 standalone.
|
||||
- **Deleting shared account resources**: verified on 2026-06-27 that miti99bot is the sole OIDC consumer and sole SAM project, so steps 5-6 are safe. Mitigation: the runbook re-confirms (list-first) before each delete in case the account changes before you run it; if a new stack/provider appears, leave the shared resource (it costs ~nothing).
|
||||
- **Re-creation by CI**: a `main` push with `deploy.yml` still active would rebuild the whole stack post-teardown. Mitigation: disable/delete the workflow as part of this phase (and it's already superseded on `feature/selfhosted`).
|
||||
- **Lost rollback**: per the Phase 4 validated decision, teardown ends the AWS rollback option. Mitigation: that was explicitly accepted; optionally keep the stack a few days before running step 2.
|
||||
- **Secret hygiene**: rotate the Telegram bot token + Gemini key after teardown if you want defense-in-depth (they lived in SSM and CloudWatch under the accepted trade-offs). Optional.
|
||||
@@ -1,151 +0,0 @@
|
||||
---
|
||||
title: "Self-host miti99bot on Coolify with MongoDB Atlas"
|
||||
description: "Add a MongoDB Atlas storage backend + in-process cron, containerize for Coolify docker-compose, migrate existing DynamoDB data."
|
||||
status: in-progress
|
||||
priority: P2
|
||||
branch: "feature/selfhosted"
|
||||
tags: [selfhost, coolify, mongodb, migration]
|
||||
blockedBy: []
|
||||
blocks: [260628-1113-mongo-native-value-documents, 260628-1310-flatten-mongo-value-documents]
|
||||
created: "2026-06-27T12:01:53.894Z"
|
||||
createdBy: "ck:plan"
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Self-host miti99bot on Coolify with MongoDB Atlas
|
||||
|
||||
## Overview
|
||||
|
||||
Run miti99bot as a long-lived container on Coolify (docker-compose) instead of AWS Lambda, using MongoDB Atlas (`MONGO_URL` + `MONGO_DATABASE`) instead of DynamoDB. Existing DynamoDB data is migrated into Atlas. At the **code** level this is additive — a 4th KV backend + a self-host run mode; the DynamoDB/Lambda code path is NOT ripped out (kept for portability and to run the migrator). At the **infrastructure** level, the deployed AWS stack is fully decommissioned after a verified cutover (Phase 5, validated decision).
|
||||
|
||||
Why this is low-risk: storage is already a pluggable `KVProvider` interface with 3 backends (`memory`, `firestore`, `dynamodb`); adding `mongodb` follows the exact `firestore` collection-per-module pattern. Secrets already fall back to plain env vars when `*_PARAMETER_NAME` is unset, so Coolify env vars need no code change. Two gaps to close, both with minimal code: (1) **cron** — EventBridge Scheduler triggers `/cron/{name}` today, replaced by an in-process scheduler (Phase 2); (2) **transport** — the bot runs webhook-only today, switched to **long polling** (Phase 3) via the same `go-telegram/bot` library's built-in polling mode, which removes the need for any public inbound ingress on the self-hosted box.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [MongoDB Storage Provider](./phase-01-mongodb-storage-provider.md) | Done (code + tests) |
|
||||
| 2 | [In-Process Cron Scheduler](./phase-02-in-process-cron-scheduler.md) | Done (code + tests) |
|
||||
| 3 | [Long-Polling Runtime + Containerize + Coolify Deploy](./phase-03-containerize-and-coolify-deploy.md) | Done (code + compose + docs); operator runs Atlas/Coolify setup |
|
||||
| 4 | [Data Migration and Cutover](./phase-04-data-migration-and-cutover.md) | Migrator done (code + e2e test); operator runs the live cutover |
|
||||
| 5 | [AWS Full Decommission](./phase-05-aws-decommission.md) | Runbook delivered + deploy.yml disabled; operator runs teardown (admin creds) |
|
||||
|
||||
## Implementation Log
|
||||
|
||||
### Session — 2026-06-28 (/cook full code until deployable)
|
||||
All code for Phases 1-4 implemented + Phase 5 runbook delivered. `go build`, `go vet`, full `go test ./...` green. Storage Mongo parity tests (incl. the blocking concurrent-CAS linearizability gate), DynamoDB parity tests, and the migrator e2e (migrate→idempotent re-run→verify→byte-identical round-trip) all PASS against real MongoDB 7 + DynamoDB Local (run in-container; `-race` skipped only because the alpine image lacks cgo/gcc). Code review (code-reviewer subagent): 0 Critical/High, all 8 acceptance criteria verified; one Low cosmetic (stale `BOT_OWNER_ID` log string) fixed.
|
||||
|
||||
**Implementation decision (within plan intent):** the container calls `DeleteWebhook(drop_pending_updates=false)` on startup before `b.Start`, making the "deleteWebhook before first poll" requirement automatic + idempotent (the manual `make telegram-deletewebhook-selfhost` target remains for the explicit cutover step). Reviewer confirmed safe vs the manual-cutover framing.
|
||||
|
||||
**Remaining (operator-run, need live creds/infra, out of code scope):** create Atlas M0 + DB user + network access; create the Coolify resource + env vars + deploy; run the live cutover (disable EventBridge → deleteWebhook → migrate → verify → start container); execute the AWS teardown runbook with the admin profile.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Phase 2 is independent of Phase 1 (cron touches no storage).
|
||||
- Phase 3 depends on 1 + 2 (container must boot with mongo + internal cron).
|
||||
- Phase 4 depends on Phase 1 (Mongo document schema must be final before copying data) and Phase 3 (the polling container must exist before cutover); it is the last step before switching Telegram to the polling container.
|
||||
- Phase 5 (AWS decommission) depends on Phase 4 `--verify` passing — it destroys DynamoDB, so it runs only after data is migrated and the bot is confirmed live on Coolify.
|
||||
|
||||
Suggested order: 1 → 2 → 3 → 4 → 5. Phases 1 and 2 can be done in parallel by separate developers (disjoint files).
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
```
|
||||
BEFORE (AWS) AFTER (Coolify self-host)
|
||||
Telegram ──webhook──> Lambda Function URL Telegram <──long poll── container (outbound only)
|
||||
EventBridge Scheduler ──> /cron/{name} in-process scheduler ──> cron.Handler
|
||||
DynamoDB (pk=module, sk=key) MongoDB Atlas (db / collection-per-module)
|
||||
SSM Parameter Store secrets Coolify env vars (plain)
|
||||
public HTTPS ingress (Function URL) NO public ingress (polling = outbound only)
|
||||
```
|
||||
|
||||
Same Go binary (`cmd/server`), same HTTP server on `:8080`, same module framework. Backend + cron + secret source are all selected by env vars at startup.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `MONGO_URL=… MONGO_DATABASE=…` (mongodb auto-selected, no `KV_PROVIDER`) boots, runs the cron scheduler by default, and consumes updates via long polling (the only transport) with persistent storage; no `/webhook` route, no webhook secret.
|
||||
- [ ] `make test` green; new mongo provider has parity tests with the firestore/dynamodb suites.
|
||||
- [ ] Self-hosted container fires the lolschedule daily push at 01:00 UTC (08:00 ICT) without EventBridge.
|
||||
- [ ] `docker compose up` (Coolify) brings the bot live via long polling with NO public domain/ingress (outbound-only); `/` health check passes internally; exactly 1 replica.
|
||||
- [ ] All existing prod DynamoDB items present in Atlas with identical values; migration is idempotent + verifiable by per-module counts.
|
||||
- [ ] After cutover, ALL miti99bot AWS resources are deleted — CloudFormation stack AND the manually-created SSM secrets, GitHub OIDC role, and SAM S3 bucket; `app=miti99bot` tag sweep is empty and Cost Explorer trends to $0.
|
||||
- [ ] `.github/workflows/deploy.yml` is removed/disabled so `main` pushes no longer recreate the AWS stack.
|
||||
|
||||
## Red Team Review
|
||||
|
||||
### Session — 2026-06-27
|
||||
**Findings:** 15 (15 accepted, 0 rejected) — 3 reviewers (security/secrets, assumptions, failure-modes), all findings carried `file:line` evidence.
|
||||
**Severity breakdown:** 2 Critical, 6 High, 7 Medium.
|
||||
|
||||
| # | Finding | Severity | Disposition | Applied To |
|
||||
|---|---------|----------|-------------|------------|
|
||||
| 1 | nil-expected CAS is a live first-write path; `$exists:false` upsert is wrong primitive → use `InsertOne` + unique-`_id` + blocking concurrent test | Critical | Accept | Phase 1 |
|
||||
| 2 | Rollback loses ALL post-cutover writes; no reverse path → state true RPO / reverse migrator | Critical | Accept | Phase 4 |
|
||||
| 3 | `buildProvider` log line would leak `MONGO_URL` creds → log only `database`, never URL | High | Accept | Phase 1 |
|
||||
| 4 | Cron dispatcher symbol misdescribed (`DispatchScheduled(ctx,name,reg)`, no timeout/recover/log) → scheduler adds own | High | Accept | Phase 2 |
|
||||
| 5 | Dockerfile omits `gitSHA` → `deploynotify` silently dead → inject build arg or document disabled | High | Accept | Phase 3 |
|
||||
| 6 | Cron double-fire (EventBridge-live + rolling deploy; push non-idempotent) → disable schedule before internal cron + last-push-date guard | High | Accept | Phase 2, Phase 4 |
|
||||
| 7 | Atlas `0.0.0.0/0` = public DB surface vs project boundary → egress-IP allowlist + least-priv user default | High | Accept | Phase 3 |
|
||||
| 8 | Cutover write-loss gap; ambiguous webhook-unset → mandatory `deleteWebhook`→migrate→verify→`setWebhook` + `pending_update_count` | High | Accept | Phase 4 |
|
||||
| 9 | Migrator raw `UpdateOne` diverges from `Put` encoding → write through `Put`; store `updatedAt` int64 | Medium | Accept | Phase 1, Phase 4 |
|
||||
| 10 | Stray `*_PARAMETER_NAME` fatal off-AWS → `.env.example` lists all six "leave UNSET" + boot criterion | Medium | Accept | Phase 3 |
|
||||
| 11 | `/` is plain text not JSON; `-healthcheck` flag doesn't exist → Coolify HTTP monitor, drop bogus CMD | Medium | Accept | Phase 3 |
|
||||
| 12 | `/cron/` redundant double-trigger when internal → leave `CRON_SHARED_SECRET` unset (route 404) | Medium | Accept | Phase 3 |
|
||||
| 13 | Migrator IAM under-specified / admin profile → exact least-priv `Scan`-only + Scan-based verify, read-only profile | Medium | Accept | Phase 4 |
|
||||
| 14 | No Mongo reconnect/health story → document auto-reconnect + pool opts; DB-aware healthcheck or accept trade-off | Medium | Accept | Phase 3 |
|
||||
| 15 | `validateKey` rejects `/` but migrator bypasses it → migrator `validateKey` each `sk`, fail loud | Medium | Accept | Phase 4 |
|
||||
|
||||
Verified non-issues (no change needed): `List()` `$gte/$lt` range avoids regex injection (sound, mirrors firestore); secrets fallback to plain env is verified correct; `.env` already gitignored.
|
||||
|
||||
### Whole-Plan Consistency Sweep
|
||||
Re-read all phase files after applying findings. Reconciled:
|
||||
- `updatedAt` storage type now consistently **int64 nanos** in Phase 1 doc-shape and Phase 4 migrator/risk (was "BSON datetime").
|
||||
- CAS absent-case is **`InsertOne`** everywhere (Phase 1 architecture + Phase 4 encoding note); the `$exists:false` upsert is removed.
|
||||
- Migrator writes **through `MongoKVStore.Put`** in Phase 4 architecture, risk, and success criteria (no raw `UpdateOne`).
|
||||
- Health endpoint described as **plain text `miti99bot ok`** in Phase 3 requirements, architecture, and risk (was "JSON"); bogus `-healthcheck` CMD removed from the compose snippet.
|
||||
- Cron dispatcher named **`DispatchScheduled`** in Phase 2 with scheduler-local timeout/recover.
|
||||
- Cutover ordering (disable EventBridge → `deleteWebhook` → migrate → start polling container, whose scheduler runs by default) consistent across Phase 2 risk and Phase 4 runbook. <!-- Session 2: polling cutover supersedes setWebhook -->
|
||||
No unresolved contradictions remain.
|
||||
|
||||
## Validation Log
|
||||
|
||||
### Session — 2026-06-27
|
||||
Verification pass skipped: `## Red Team Review` already carries full `file:line` evidence and no `[UNVERIFIED]` tags remain. Interview resolved all 8 open questions.
|
||||
|
||||
| # | Question | Decision | Affects |
|
||||
|---|----------|----------|---------|
|
||||
| 1 | Rollback RPO | **Simple short-window cutover; NO reverse migrator.** User coordinates users to pause around cutover, so no writes occur mid-migration. | Phase 4 |
|
||||
| 2 | Decommission AWS | **Tear down the SAM stack after `--verify` passes.** No long-term fallback. EventBridge schedule still disabled as an explicit pre-cutover step. Stop the GitHub Actions deploy workflow. | Phase 4 |
|
||||
| 3 | Atlas network access | **`0.0.0.0/0` (no stable Coolify egress IP).** Accepted trade-off: mandatory strong unique password + least-privilege DB user (`readWrite` on one DB, not admin). Documented as a knowing widening vs DynamoDB's IAM-gated posture. | Phase 3 |
|
||||
| 4 | deploynotify | **Keep it** — inject `gitSHA` via Dockerfile `ARG GIT_SHA` + Coolify build-arg. | Phase 3 |
|
||||
| 5 | `/cron/` HTTP route | **Disable in prod** — leave `CRON_SHARED_SECRET` unset (route 404s); internal scheduler is the sole trigger. | Phase 2, Phase 3 |
|
||||
| 6 | Mongo driver | `go.mongodb.org/mongo-driver/v2` (current stable); `robfig/cron/v3` for the scheduler. | Phase 1, Phase 2 |
|
||||
| 7 | Atlas tier | Free **M0** (512 MB) — sufficient for the tiny paper-trading KV. | Phase 3 |
|
||||
| 8 | `updatedAt` reader | Confirmed write-only today; store as int64 for cheap parity. No TTL/sort planned. | Phase 1, Phase 4 |
|
||||
|
||||
### Session 2 — 2026-06-27 (Telegram transport)
|
||||
User directive: use long polling, **as the only mode — no env toggle**. Decision: keep the existing `go-telegram/bot` library (native polling via `b.Start(ctx)` — no library swap, handlers unchanged) and **delete the webhook code path entirely** (webhook existed only for Lambda, which is being decommissioned — YAGNI). Removed: `internal/telegram/webhook.go` + test, the `/webhook` route, and the `WebhookSecret` config + its startup fatal. Consequences propagated to Phase 3 (no public ingress/domain/TLS/webhook secret; health server internal-only; single replica = single polling consumer) and Phase 4 (cutover = `deleteWebhook` → start polling container, which drains the buffered queue; no `setWebhook`). Rollback during the pre-`sam delete` window still works because the live Lambda keeps its own already-built webhook code until teardown. This eliminated the earlier "public ingress DDoS" risk entirely.
|
||||
|
||||
### Session 3 — 2026-06-27 (minimize env surface)
|
||||
User directive: don't require `KV_PROVIDER`/`CRON_MODE`; mongo + in-process cron are defaults; remove unneeded system envs. Decisions: (1) `buildProvider` auto-selects `mongodb` when `MONGO_URL` is set (else `memory`); `KV_PROVIDER` is an optional override; the old `AWS_LAMBDA_FUNCTION_NAME → dynamodb` auto-detect is removed. (2) The cron scheduler runs unconditionally at container start — `CRON_MODE` env removed (cutover safety is from ordering + the idempotency guard, not a gate). (3) Dropped from required env: `KV_PROVIDER`, `CRON_MODE`, `PORT` (defaults 8080). Final required env = `TELEGRAM_BOT_TOKEN`, `MONGO_URL`, `MONGO_DATABASE`; operational = `MODULES`, `OWNER_ID`, `ADMIN_IDS`; optional = `GEMINI_API_KEY`. Propagated to Phases 1-4.
|
||||
|
||||
### Session 4 — 2026-06-27 (drop per-module API URL overrides + two credentials)
|
||||
User directive: remove the stock/coin/gold URL envs; use the providers configured in code. Decisions:
|
||||
- **URL overrides dropped** — self-host sets NONE of `STOCK_INCOME_EVENTS_API_URL`, `GOLD_PRICE_API_URL`, `GOLD_FX_API_URL`, `GOLD_VNAPP_API_URL`, `COIN_BINANCE_API_URL`, `COIN_COINBASE_API_URL`, `COIN_COINGECKO_API_URL`; modules use coded default endpoints. Override plumbing stays (dormant) unless later cleaned.
|
||||
- **`GOLD_VNAPP_API_KEY` dropped** — already unnecessary: `gold/vnappmob_client.go` auto-fetches a VNAppMob API key via the refresh endpoint and caches it in KV (`vnappmob:api_key`, 24h refresh buffer). With Mongo as the KV the key auto-caches to the DB. The env var was only an optional override. No code change — just don't set it.
|
||||
- **`STOCK_INCOME_EVENTS_API_TOKEN` dropped AND the income-events feature removed** — delete the `stock_income_events` command + its FireAnt `IncomeEventClient` (`internal/modules/stock/income_events.go` + test), the token env, and any user-facing notice/description. The other stock commands (`stock_income_stock`, `stock_income_vnd`) stay. Removing a public command requires updating the command catalog (`aws/telegram-commands.json` → the self-host `setMyCommands` source). Code task added to Phase 3.
|
||||
|
||||
Final optional env reduces to `GEMINI_API_KEY` only.
|
||||
|
||||
### Session 5 — 2026-06-27 (rename owner/admin envs)
|
||||
User directive: cleaner names. `BOT_OWNER_ID` → `OWNER_ID`, `ADMIN_USER_IDS` → `ADMIN_IDS`. Rename the env keys in `cmd/server/main.go` `loadConfig`; no backward-compat (the AWS template + workflow that set the old names are being deleted). The Go config field names + the `Auth{BotOwnerID, AdminUserIDs}` struct can keep their internal names — only the env keys change.
|
||||
|
||||
### Whole-Plan Consistency Sweep (post-validation)
|
||||
- Phase 4 rollback/teardown rewritten: AWS torn down after verify; reverse-migrator option removed; rollback framed as "coordinate users, short window" not "keep Lambda N days."
|
||||
- Phase 3 Atlas networking: `0.0.0.0/0` is now the chosen path (was "egress-IP default") with password + least-priv user as hard requirements.
|
||||
- deploynotify `gitSHA` injection and `/cron/` disabled (`CRON_SHARED_SECRET` unset) were already the recommended defaults in Phases 2/3 — now confirmed, no contradiction.
|
||||
- No unresolved contradictions remain.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None — all resolved in the Validation Log above.
|
||||
-142
@@ -1,142 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: Version-Field Optimistic Locking
|
||||
status: completed
|
||||
priority: P1
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Version-Field Optimistic Locking
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the value-bytes `CompareAndSwap` with a version-field optimistic lock,
|
||||
across all backends + the 3 callers, while values are still stored as strings.
|
||||
Representation-neutral and independently shippable; unblocks Phase 2 (native
|
||||
storage can't use byte-exact CAS).
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: a versioned read+swap contract — read returns value + a version
|
||||
token; write succeeds only if the stored version is unchanged, else
|
||||
`ErrConflict`. Absent key = version 0; first write must be create-only.
|
||||
- Functional: concurrent writers → exactly one winner, losers `ErrConflict`
|
||||
(same guarantee as today's CAS).
|
||||
- Non-functional: `KVStore` bulk interface (Get/Put/PutJSON/GetJSON/Delete/List)
|
||||
unchanged. <!-- Updated: Validation Session 1 - versioned CAS only on memory + mongodb; firestore backend dropped; dynamodb keeps base KVStore for the migrator (no CAS). -->
|
||||
Only memory + mongodb implement the versioned lock (the live backends).
|
||||
dynamodb keeps only the base `KVStore` (migrator uses Scan/Put, never CAS).
|
||||
The firestore backend is **removed entirely** (legacy, unused since self-host).
|
||||
|
||||
## Architecture
|
||||
|
||||
Replace `CompareAndSwapStore` with a versioned contract in
|
||||
`internal/storage/kv_store.go`:
|
||||
|
||||
```go
|
||||
type VersionedStore interface {
|
||||
// GetVersioned returns the value and its current version; ErrNotFound if absent.
|
||||
GetVersioned(ctx context.Context, key string) (val []byte, version int64, err error)
|
||||
// PutVersioned writes val iff the stored version still equals expectedVersion.
|
||||
// expectedVersion == 0 means "must not exist yet". ErrConflict on mismatch.
|
||||
PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error
|
||||
}
|
||||
```
|
||||
|
||||
Per-backend version source:
|
||||
- **memory** (`memory_kv.go`): per-key `version int64` alongside the bytes;
|
||||
`PutVersioned` checks+increments under the existing mutex. `prefixedStore`
|
||||
(`prefix.go`) forwards both methods with the key prefix.
|
||||
- **mongodb** (`mongodb_kv.go`): add a `version` int field to the doc.
|
||||
`GetVersioned` reads value+version; **a doc with no `version` field (written
|
||||
by the pre-refactor build) reports version 0 but still "exists"**.
|
||||
`PutVersioned` with `expectedVersion==0` → `UpdateOne(filter={_id, version:
|
||||
{$in:[0,null]} OR $exists:false}, {$set:{value,version:1,updatedAt}}, upsert:true)`
|
||||
so it upserts a new key AND adopts a legacy version-less doc without a
|
||||
spurious duplicate-key conflict; `expectedVersion>0` →
|
||||
`UpdateOne(filter={_id, version:expectedVersion}, {$set:{value,updatedAt}, $inc:{version:1}})`,
|
||||
`MatchedCount==0` → `ErrConflict`. Keeps the linearizable single-winner
|
||||
property. <!-- Updated: Validation Session 1 - legacy version-less docs treated as v0 + match missing-or-equal so existing users' updates don't fail during rollout. -->
|
||||
- **dynamodb** (`dynamodb_kv.go`): **drop `CompareAndSwap`** — the migrator only
|
||||
Scans/Puts, never locks. Keeps the base `KVStore` only.
|
||||
- **firestore**: **delete the backend** (`firestore_client.go`,
|
||||
`firestore_provider.go`, `firestore_kv.go` + tests), the `firestore` case in
|
||||
`buildProvider`, and the firestore go.mod deps. Legacy, unused since self-host.
|
||||
Keep the shared helpers it currently hosts (`validateKey`/`validatePrefix`/
|
||||
`prefixSuccessor`/`collectionNameRe`) by relocating them to a backend-neutral
|
||||
storage file so memory/mongodb/dynamodb keep compiling.
|
||||
|
||||
Callers (load → mutate → swap-by-version, retry on conflict):
|
||||
- `coin/portfolio.go` `UpdatePortfolio` + `loadPortfolioForUpdate`: capture
|
||||
`version` instead of `expected []byte`; call `PutVersioned`.
|
||||
- `gold/portfolio.go`: same shape.
|
||||
- `lolschedule/cron.go` `claimDailyPush`: `GetVersioned` the date key, claim via
|
||||
`PutVersioned(key, version, today)`; conflict → already claimed.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/storage/kv_store.go` — replace `CompareAndSwapStore` with `VersionedStore`.
|
||||
- Modify: `internal/storage/memory_kv.go`, `prefix.go`, `mongodb_kv.go` —
|
||||
implement versioned methods; drop old CAS.
|
||||
- Modify: `internal/storage/dynamodb_kv.go` — drop `CompareAndSwap` (migrate-only).
|
||||
- Create: `internal/storage/keys.go` (or similar) — relocate shared helpers
|
||||
`validateKey`/`validatePrefix` (from `firestore_kv.go`), `prefixSuccessor`
|
||||
(from `firestore_kv.go`), `collectionNameRe` (from `firestore_provider.go`)
|
||||
to a backend-neutral file so they survive the firestore deletion.
|
||||
- Delete: `internal/storage/firestore_client.go`, `firestore_provider.go`,
|
||||
`firestore_kv.go`, `firestore_kv_test.go`, `firestore_provider_test.go`.
|
||||
- Modify: `cmd/server/main.go` — remove the `firestore` case + `FirestoreProject`/
|
||||
`FirestoreEmulatorHost` config and the firestore init-timeout const.
|
||||
- Modify: `go.mod`/`go.sum` — `go mod tidy` drops `cloud.google.com/go/firestore`
|
||||
+ now-unused google.golang.org/api deps.
|
||||
- Modify: `internal/modules/coin/portfolio.go`, `internal/modules/gold/portfolio.go`,
|
||||
`internal/modules/lolschedule/cron.go` — switch to version flow.
|
||||
- Modify tests: `memory_kv_test.go`, `mongodb_kv_test.go`, `dynamodb_kv_test.go`,
|
||||
`coin/portfolio_test.go`, `gold/portfolio_test.go`, `lolschedule/cron_test.go`.
|
||||
- Check: `Makefile` (drop `firestore-emulator`/`test-emulator` targets),
|
||||
`.github/workflows/ci.yml` (firestore-emulator note), `README.md` storage list.
|
||||
|
||||
## Implementation Steps (TDD)
|
||||
|
||||
1. **Tests first — lock current behavior, expressed in the new contract.** Before
|
||||
changing impls, write/convert backend tests asserting: create-only on
|
||||
version 0, conflict on stale version, success on current version, and the
|
||||
N-goroutine concurrent single-winner test (memory + mongodb). Convert the
|
||||
coin/gold concurrent-update tests to assert the same observable outcome
|
||||
(exactly one mutation wins; balances never double-applied). These fail to
|
||||
compile/pass until the impl lands — that's the red state.
|
||||
2. Relocate shared helpers (`validateKey`/`validatePrefix`/`prefixSuccessor`/
|
||||
`collectionNameRe`) to a backend-neutral file; delete the firestore backend +
|
||||
its `buildProvider` case + config; `go mod tidy`. Confirm build still green.
|
||||
3. Define `VersionedStore`; implement in memory + `prefixedStore`.
|
||||
4. Implement in mongodb (`version` field; legacy version-less doc = v0 via
|
||||
match-missing-or-equal + upsert).
|
||||
5. Drop `CompareAndSwap` from dynamodb (keeps base `KVStore`).
|
||||
6. Migrate the 3 callers to load-version → mutate → `PutVersioned`.
|
||||
7. Make red tests green: `make test`; `make test-mongo`; `make test-dynamodb`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `VersionedStore` implemented + tested on memory + mongodb.
|
||||
- [ ] Concurrent version-CAS test: exactly one winner, losers `ErrConflict` (memory + real Mongo).
|
||||
- [ ] A legacy doc with no `version` field is updated successfully (treated as v0); test seeds one and asserts no spurious conflict.
|
||||
- [ ] dynamodb no longer implements CAS; migrator (Scan/Put) + its e2e test still pass.
|
||||
- [ ] firestore backend deleted; shared helpers relocated; `go mod tidy` drops firestore deps; build green.
|
||||
- [ ] coin/gold portfolio updates + lolschedule daily-push claim use version flow; their tests pass.
|
||||
- [ ] Values still stored as strings (representation unchanged this phase).
|
||||
- [ ] `make vet` + full `go test ./...` green; 12 non-CAS consumers untouched.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Legacy version-less docs (HIGH for live rollout)** — docs from the deployed
|
||||
build have no `version` field; naive create-only CAS would conflict and fail
|
||||
existing users' coin/gold updates. Mitigation: treat absent version as v0 and
|
||||
match missing-or-equal (upsert) — explicit seeded test.
|
||||
- **Helper relocation regression** — `validateKey`/`prefixSuccessor`/
|
||||
`collectionNameRe` currently live in firestore files; deleting firestore
|
||||
without relocating them breaks memory/mongodb/dynamodb. Mitigation: relocate
|
||||
first (step 2), build before proceeding.
|
||||
- **Contract ripple** — interface + 3 callers + memory/mongodb. Mitigation:
|
||||
tests-first per backend; phase is representation-neutral, ships independently.
|
||||
- **Conflict-loop regression** — a bug in version compare could exhaust the
|
||||
bounded retry. Mitigation: explicit conflict + success unit tests before wiring callers.
|
||||
-108
@@ -1,108 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: Native BSON Value Representation
|
||||
status: completed
|
||||
priority: P1
|
||||
dependencies:
|
||||
- 1
|
||||
---
|
||||
|
||||
# Phase 2: Native BSON Value Representation
|
||||
|
||||
## Overview
|
||||
|
||||
Store the Mongo `value` field as a native BSON document (object/array) instead
|
||||
of a stringified-JSON blob, with a string fallback for non-JSON values, so
|
||||
values are expandable and queryable in Atlas/Compass. Depends on Phase 1's
|
||||
version CAS (byte-exact compare is gone, so native storage is now safe).
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: JSON object → native BSON object; JSON array → native array;
|
||||
non-JSON / bare scalar (e.g. lolschedule `daily_push:last_date`) → BSON string.
|
||||
- Functional: `Get`/`GetJSON` reconstruct the caller's `[]byte`/struct from the
|
||||
native form with **no numeric type/precision loss**.
|
||||
- Functional: dual-read — documents written by the prior string/binary build
|
||||
still read correctly.
|
||||
- Non-functional: `KVStore` + `VersionedStore` signatures unchanged; the 12
|
||||
PutJSON/GetJSON/List consumers untouched. Conversion is confined to
|
||||
`mongodb_kv.go`.
|
||||
|
||||
## Architecture
|
||||
|
||||
### JSON ↔ BSON conversion (the fidelity-critical part)
|
||||
- **Write** (`Put`/`PutVersioned` receive JSON `[]byte`): if `json.Valid` and
|
||||
the top level is `{` or `[`, decode with `json.Decoder` + `UseNumber()` into a
|
||||
generic value, then a recursive `jsonToBSON` converts `json.Number` →
|
||||
**int64 when integral and representable, else float64** (so int64 stays int64,
|
||||
not coerced to double). Store the result under `value`. Otherwise store
|
||||
`string(val)`.
|
||||
- **Read** (`decodeValue`): inspect the decoded `value`:
|
||||
- `bson.M` / `bson.D` / `bson.A` (object/array) → recursive `bsonToJSON` →
|
||||
`json.Marshal`-compatible bytes (int64 → integer, double → number).
|
||||
- `string` → `[]byte(v)`.
|
||||
- `bson.Binary` / `[]byte` → legacy fallback (pre-refactor docs).
|
||||
- Map-key ordering is NOT preserved on object re-serialization; acceptable —
|
||||
no byte-exact consumer remains (Phase 1 removed value-bytes CAS; all readers
|
||||
are `GetJSON`/unmarshal or the bare-string `last_date`).
|
||||
|
||||
### Codec decision (RESOLVED — Validation Session 1)
|
||||
<!-- Updated: Validation Session 1 - int64-preserving json.Number codec chosen. -->
|
||||
**Use the int64-preserving `json.Number` codec.** Verification confirmed no
|
||||
persisted integer exceeds 2^53 today (all timestamps are `UnixMilli` ~1.7e12;
|
||||
`CreatedAt int64` = UnixMilli; lolschedule cache `Ts` = UnixMilli), so float64
|
||||
coercion would be safe *now* — but the `json.Number` codec is chosen anyway to
|
||||
future-proof against any later large-int field, for trivial extra cost. Decode
|
||||
with `json.Decoder.UseNumber()`; `jsonToBSON` maps `json.Number` → BSON int64
|
||||
when integral and representable, else double.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/storage/mongodb_kv.go` — `jsonToBSON`/`bsonToJSON` helpers;
|
||||
`doc()` stores native-or-string; `decodeValue` handles object/array/string/
|
||||
binary; `GetVersioned` re-serializes value.
|
||||
- Modify: `internal/storage/mongodb_kv_test.go` — fidelity + native-shape tests.
|
||||
- Modify: `cmd/migrate-dynamo-to-mongo/` — no code change (writes through `Put`,
|
||||
now native); update `main_test.go` assertions if they inspect raw value type.
|
||||
- Modify: `docs/deploy-coolify-selfhosted.md` + the self-host plan's
|
||||
`phase-01` note — value now native BSON, re-migration note.
|
||||
|
||||
## Implementation Steps (TDD)
|
||||
|
||||
1. Codec already chosen (int64-preserving `json.Number`; audit done in
|
||||
validation — no field >2^53). Proceed straight to tests.
|
||||
2. **Tests first** — per-struct round-trip fidelity tests through
|
||||
`PutJSON`→Mongo→`GetJSON` for: coin/gold/stock Portfolio (balances, int qty,
|
||||
timestamps), wordle/loldle/twentyq state, stats counters, lolschedule
|
||||
subscribers (array) + `last_date` (bare string). Assert struct equality AND
|
||||
that the raw stored `value` is a native object/array (not a string) for the
|
||||
JSON cases, and a string for `last_date`. Add a dual-read test seeding a
|
||||
legacy string-value doc. These are red until the impl lands.
|
||||
3. Implement `jsonToBSON`/`bsonToJSON` + wire into `doc()`/`decodeValue`/`GetVersioned`.
|
||||
4. Make red tests green: `make test-mongo`; full `go test ./...`.
|
||||
5. Re-run migrator e2e (DynamoDB Local → Mongo) — values land native, `--verify`
|
||||
counts match, byte round-trip via `Get` still decodes.
|
||||
6. Docs: native representation + re-migration note.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Raw stored `value` is a native object/array for JSON values, a string for
|
||||
`last_date`; verified by reading the raw doc in a test.
|
||||
- [ ] Every persisted struct round-trips through `PutJSON`/`GetJSON` with no
|
||||
type/precision loss (fidelity tests green).
|
||||
- [ ] Dual-read test: a seeded legacy string-value doc still decodes.
|
||||
- [ ] Migrator e2e green; values native; `--verify` matches.
|
||||
- [ ] `make vet` + full `go test ./...` (incl. Mongo + DynamoDB) green; 12
|
||||
non-CAS consumers untouched.
|
||||
- [ ] In Atlas, a portfolio document is visibly expandable (manual confirm).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Int/double fidelity (HIGH)** — see codec decision. Mitigation: int64-
|
||||
preserving `json.Number` codec + per-struct fidelity tests as the gate.
|
||||
- **Map-key reordering on read** — harmless for unmarshal consumers; called out
|
||||
so no future code assumes byte-stable `Get` on objects.
|
||||
- **Legacy docs** — dual-read fallback covers string/binary docs from the prior
|
||||
build; explicit test.
|
||||
- **Re-migration** — if prod data was already migrated as strings, re-run the
|
||||
migrator (idempotent; overwrites to native). Atlas currently fresh → low.
|
||||
@@ -1,128 +0,0 @@
|
||||
---
|
||||
title: Mongo-native value documents + version-field CAS
|
||||
description: >-
|
||||
Store Mongo KV values as native BSON documents (object/array, string fallback)
|
||||
and switch CompareAndSwap to a version-field optimistic lock, keeping the
|
||||
KVStore interface.
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: feature/selfhosted
|
||||
tags:
|
||||
- storage
|
||||
- mongodb
|
||||
- refactor
|
||||
- cas
|
||||
- tdd
|
||||
blockedBy:
|
||||
- 260627-1849-selfhost-coolify-mongodb
|
||||
blocks: []
|
||||
created: '2026-06-28T04:51:16.659Z'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Mongo-native value documents + version-field CAS
|
||||
|
||||
## Overview
|
||||
|
||||
Make MongoDB store each KV value as a **native BSON document** (object/array
|
||||
natively; string fallback for non-JSON like the lolschedule date guard) instead
|
||||
of a stringified-JSON blob, so values are expandable and queryable in
|
||||
Atlas/Compass. This requires replacing the value-bytes `CompareAndSwap` with a
|
||||
**version-field optimistic lock** first — byte-exact compare is the only thing
|
||||
that makes native storage impossible. The generic `KVStore` interface
|
||||
(Get/Put/PutJSON/GetJSON/Delete/List) is preserved, so the 12 PutJSON/GetJSON/
|
||||
List consumers stay unchanged; only the 3 CAS callers (coin, gold, lolschedule)
|
||||
and the storage layer change.
|
||||
|
||||
Approved design: `plans/reports/brainstorm-260628-1113-mongo-native-documents-report.md` (Option A).
|
||||
|
||||
Sequencing rationale: Phase 1 (version CAS) is representation-neutral and
|
||||
independently shippable — it de-risks the concurrency change while values are
|
||||
still strings. Phase 2 flips the representation to native on top of the
|
||||
already-safe version lock.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Version-Field Optimistic Locking](./phase-01-version-field-optimistic-locking.md) | Completed |
|
||||
| 2 | [Native BSON Value Representation](./phase-02-native-bson-value-representation.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **blockedBy `260627-1849-selfhost-coolify-mongodb`** — that plan created the
|
||||
`mongodb` KVProvider, the migrator, and the deployed self-host runtime this
|
||||
refactor modifies. Its code is already committed on `feature/selfhosted`.
|
||||
- Phase 2 depends on Phase 1 (native storage requires version CAS, not
|
||||
value-bytes CAS).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] In Atlas/Compass, a coin/gold/stock portfolio and a game-state value
|
||||
render as **native, expandable BSON documents** (not a quoted JSON string).
|
||||
- [ ] Non-JSON values (lolschedule `daily_push:last_date`) stored as a plain
|
||||
BSON string; round-trip byte-exact.
|
||||
- [ ] CAS is version-based: concurrent updates yield exactly one winner, losers
|
||||
get `ErrConflict` (proven by the existing-style concurrent test against real
|
||||
Mongo, plus memory).
|
||||
- [ ] coin/gold/stock portfolios + each game-state struct round-trip through
|
||||
`PutJSON`/`GetJSON` with **no numeric precision/type loss** (TDD fidelity
|
||||
tests; explicit check that no int field exceeds 2^53 or the codec preserves
|
||||
int64).
|
||||
- [ ] The 12 PutJSON/GetJSON/List-only consumers compile and pass unchanged.
|
||||
- [ ] memory backend + hermetic `go test ./...` (no DB) still pass; full suite
|
||||
(incl. Mongo + DynamoDB integration) green.
|
||||
- [ ] Migrator writes native documents; `--verify` still matches per-module
|
||||
counts; re-migration path documented (Atlas fresh → low impact now).
|
||||
- [ ] Dual-read fallback: documents written by the previous string/binary build
|
||||
still read correctly.
|
||||
|
||||
## Risks (carried into phases)
|
||||
|
||||
1. **JSON↔BSON int/double fidelity (HIGH)** — `json.Unmarshal` to a generic
|
||||
coerces all numbers to float64 → BSON double; large int64 (>2^53) would lose
|
||||
precision. Phase 2 step 1 audits struct numeric fields and picks the codec
|
||||
(accept float64 if all safe, else a `json.Number`-preserving decode). Gated
|
||||
by TDD fidelity tests.
|
||||
2. **CAS contract change ripple** — `CompareAndSwapStore` → `VersionedStore`
|
||||
(memory + mongodb implement; dynamodb drops CAS; firestore backend deleted)
|
||||
+ 3 callers + tests. Phase 1 isolates this.
|
||||
3. **Re-migration** — migrator output representation changes; document re-run
|
||||
(Atlas currently fresh).
|
||||
4. **Legacy version-less docs (HIGH, live rollout)** — docs from the deployed
|
||||
build have no `version` field; Phase 1 treats absent version as v0 +
|
||||
match-missing-or-equal so existing users' updates don't fail.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None — all resolved in the Validation Log below.
|
||||
|
||||
## Validation Log
|
||||
|
||||
### Session 1 — 2026-06-28
|
||||
|
||||
**Verification pass (Light tier, Fact Checker, 2 phases):** Claims checked
|
||||
against code. Key result — **no persisted integer exceeds 2^53**: all timestamps
|
||||
are `UnixMilli` (~1.7e12) / `Unix()` seconds; `coin|gold.Portfolio.Meta.CreatedAt int64`
|
||||
= UnixMilli; lolschedule cache `Ts` = UnixMilli. CAS interface
|
||||
(`CompareAndSwapStore`), its 3 callers, and the backend files confirmed present.
|
||||
Failures: 0.
|
||||
|
||||
| # | Question | Decision | Affects |
|
||||
|---|----------|----------|---------|
|
||||
| 1 | JSON↔BSON codec | **int64-preserving `json.Number`.** float64 is safe today (no int >2^53) but json.Number future-proofs cheaply. | Completed |
|
||||
| 2 | Versioned CAS backend scope | **memory + mongodb only.** Drop the **firestore backend entirely** (legacy/unused); **dynamodb** keeps base `KVStore` only (migrate-only; drop its CAS). | Completed |
|
||||
| 3 | Pre-existing version-less docs | **Treat absent `version` as v0, match missing-or-equal (upsert).** No backfill; live docs keep working through the Phase 1 deploy. | Phase 1 |
|
||||
|
||||
Consequence surfaced: dropping firestore requires relocating the shared helpers
|
||||
it hosts (`validateKey`/`validatePrefix`/`prefixSuccessor`/`collectionNameRe`)
|
||||
to a backend-neutral file — added to Phase 1.
|
||||
|
||||
### Whole-Plan Consistency Sweep
|
||||
Re-read `plan.md` + both phase files after propagation. Reconciled: backend
|
||||
scope is now "memory + mongodb implement versioned CAS; dynamodb base-only;
|
||||
firestore deleted" consistently in plan risks, Phase 1 architecture/files/steps/
|
||||
criteria. Codec is "int64-preserving json.Number" in plan + Phase 2. Version-less
|
||||
doc handling consistent (Phase 1 architecture + risk + criteria). No unresolved
|
||||
contradictions.
|
||||
-136
@@ -1,136 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Schema Contract and Regression Tests"
|
||||
status: pending
|
||||
priority: P2
|
||||
dependencies: []
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 1: Schema Contract and Regression Tests
|
||||
|
||||
## Overview
|
||||
|
||||
Define the exact Mongo document contract and lock it with failing tests before
|
||||
changing the codec. This phase prevents a vague "flatten value" implementation
|
||||
from preserving stale fields, losing type fidelity, or silently breaking legacy
|
||||
reads.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: specify new raw document shapes for JSON object, array, scalar,
|
||||
non-JSON string, reserved-key collision, and legacy `value` docs.
|
||||
- Functional: preserve current `KVStore` and `VersionedStore` method contracts.
|
||||
- Functional: prove `updatedAt` is BSON Date / Go `time.Time`.
|
||||
- Non-functional: tests must be explicit enough to fail against the current
|
||||
`{ _id, value, version, updatedAt int64 }` shape.
|
||||
|
||||
## Architecture
|
||||
|
||||
Target document examples:
|
||||
|
||||
```javascript
|
||||
// Normal JSON object payload, flattened.
|
||||
{
|
||||
_id: "user:7",
|
||||
usd: 1000.25,
|
||||
assets: { BTC: 1 },
|
||||
meta: { createdAt: 1782604800000 },
|
||||
version: 4,
|
||||
updatedAt: ISODate("2026-06-28T06:10:29Z"),
|
||||
schemaVersion: 2
|
||||
}
|
||||
|
||||
// Non-object fallback. Mongo root must be a document, so array/scalar payloads
|
||||
// need a reserved payload field. This is not the old generic `value` field.
|
||||
{
|
||||
_id: "daily_push:last_date",
|
||||
_payload: "2026-06-28",
|
||||
payloadKind: "string",
|
||||
version: 1,
|
||||
updatedAt: ISODate("2026-06-28T06:10:29Z"),
|
||||
schemaVersion: 2
|
||||
}
|
||||
```
|
||||
|
||||
Reserved fields: `_id`, `value`, `version`, `updatedAt`, `schemaVersion`,
|
||||
`payloadKind`, `_payload`. If an object payload contains one of these keys, the
|
||||
codec must preserve data by storing the whole object in `_payload` with
|
||||
`payloadKind: "object"` instead of flattening. Object payloads with keys that
|
||||
contain null, contain `.`, or start with `$` also use `_payload`; MongoDB
|
||||
permits some of these in modern versions but documents restrictions, so fallback
|
||||
keeps query/index behavior predictable. <!-- Updated: Validation Session 1 - reserve legacy value + unsafe Mongo field names -->
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/storage/mongodb_kv_test.go` — raw-shape, legacy-read,
|
||||
stale-field, updatedAt, and CAS tests.
|
||||
- Modify: `internal/storage/mongodb_value_codec.go` — only if test helpers need
|
||||
exported/unexported constants clarified later.
|
||||
- Read: `internal/storage/mongodb_kv.go`
|
||||
- Read: `internal/storage/kv_store.go`
|
||||
- Read: `plans/reports/research-260628-0605-mongodb-document-design-standards.md`
|
||||
|
||||
## Tests Before
|
||||
|
||||
Add/adjust Mongo integration tests gated by `MONGODB_TEST_URL`:
|
||||
|
||||
1. `TestMongoKVStore_RootObjectRepresentation`
|
||||
- `PutJSON` a coin-like portfolio.
|
||||
- Raw Mongo doc has root `usd`, `assets`, `meta`.
|
||||
- Raw Mongo doc has no `value`.
|
||||
- `updatedAt` decodes as `time.Time`.
|
||||
2. `TestMongoKVStore_RootObjectOverwriteRemovesStaleFields`
|
||||
- Write `{a:1,b:2}` then overwrite `{a:3}`.
|
||||
- Raw doc has `a`, no stale `b`.
|
||||
3. `TestMongoKVStore_NonObjectPayloadFallback`
|
||||
- Write bare string and JSON array.
|
||||
- Raw docs use `_payload` + `payloadKind`, no `value`.
|
||||
- `Get` round-trips.
|
||||
4. `TestMongoKVStore_ReservedRootFieldCollision`
|
||||
- Write object with `value`, `version`, or `updatedAt` in payload.
|
||||
- Raw doc preserves payload under `_payload`; `GetJSON` round-trips.
|
||||
5. `TestMongoKVStore_UnsafeMongoFieldNameFallback`
|
||||
- Write object payloads with keys containing null, containing `.`, and
|
||||
starting with `$`.
|
||||
- Raw docs preserve payload under `_payload`; `GetJSON` round-trips.
|
||||
6. `TestMongoKVStore_LegacyValueDocsStillRead`
|
||||
- Seed old docs directly with `value` as string, `bson.Binary`, object,
|
||||
and array.
|
||||
- `Get`/`GetJSON` decode all.
|
||||
7. Keep/strengthen `TestMongoKVStore_PutVersioned_ConcurrentCreate`.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add named constants in tests for reserved fields expected in the new shape.
|
||||
2. Add raw Mongo assertions before changing implementation.
|
||||
3. Audit current persisted structs for reserved top-level key collisions and
|
||||
document the result in test comments.
|
||||
4. Confirm all new tests fail for the intended reason against current code.
|
||||
5. Keep existing int64-fidelity and legacy-version tests in place.
|
||||
|
||||
## Tests After
|
||||
|
||||
- Re-run the new Mongo storage tests after Phase 2 implementation.
|
||||
- Ensure old native-`value` tests are either rewritten to the new contract or
|
||||
retained only as legacy-read tests.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] New raw-shape tests exist and fail before implementation.
|
||||
- [ ] Legacy-read coverage includes the old `value` field.
|
||||
- [ ] Reserved-field coverage includes payload key `value`.
|
||||
- [ ] Unsafe Mongo field-name coverage includes null, `.`, and `$` cases.
|
||||
- [ ] Stale-field removal has a dedicated regression test.
|
||||
- [ ] `updatedAt` BSON Date behavior has a dedicated assertion.
|
||||
- [ ] Existing version-CAS tests remain in the suite.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: tests assert driver-specific decoded types too tightly. Mitigation:
|
||||
assert behavior and accepted BSON-decoded Go types, not exact internal map
|
||||
ordering.
|
||||
- Risk: reserved-field collision behavior gets forgotten because current structs
|
||||
do not collide. Mitigation: synthetic collision test is required.
|
||||
- Risk: stale-field issue appears only on overwrite. Mitigation: explicit
|
||||
overwrite test before implementation.
|
||||
-139
@@ -1,139 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Root Document Storage Encoding"
|
||||
status: pending
|
||||
priority: P2
|
||||
dependencies: [1]
|
||||
effort: "L"
|
||||
---
|
||||
|
||||
# Phase 2: Root Document Storage Encoding
|
||||
|
||||
## Overview
|
||||
|
||||
Implement the new root-payload Mongo encoding in `MongoKVStore` while preserving
|
||||
all caller-facing storage interfaces. Writes should stop producing `value`,
|
||||
reads should support both new root docs and legacy `value` docs.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: `Put`, `PutJSON`, `PutVersioned`, `Get`, `GetJSON`,
|
||||
`GetVersioned`, `Delete`, and `List` signatures remain unchanged.
|
||||
- Functional: JSON object values flatten to root unless they collide with
|
||||
reserved metadata fields, the legacy `value` field, or unsafe Mongo field
|
||||
names.
|
||||
- Functional: arrays/scalars/non-JSON values use `_payload` and `payloadKind`,
|
||||
never the old `value` field.
|
||||
- Functional: `updatedAt` is stored as `time.Time`.
|
||||
- Functional: `version` increments on plain and versioned writes.
|
||||
- Non-functional: no stale payload fields after overwrite.
|
||||
|
||||
## Architecture
|
||||
|
||||
Split the codec into document-level helpers:
|
||||
|
||||
```go
|
||||
type payloadKind string
|
||||
|
||||
const (
|
||||
payloadKindObject payloadKind = "object"
|
||||
payloadKindArray payloadKind = "array"
|
||||
payloadKindString payloadKind = "string"
|
||||
)
|
||||
|
||||
func encodeRootDocument(key string, val []byte, version int64, now time.Time) (bson.M, error)
|
||||
func decodeRootDocument(key string, doc bson.M) ([]byte, error)
|
||||
func payloadFieldsFromRoot(doc bson.M) bson.M
|
||||
```
|
||||
|
||||
Decode order:
|
||||
|
||||
1. Legacy `value` field exists -> current `decodeValue` compatibility path.
|
||||
2. `_payload` exists -> decode `_payload` by `payloadKind`.
|
||||
3. Else -> collect all root fields except reserved metadata and marshal as JSON.
|
||||
|
||||
Write strategy:
|
||||
|
||||
- `PutVersioned(expectedVersion > 0)`: `ReplaceOne({_id:key, version:expected}, replacement(version=expected+1))`.
|
||||
- `PutVersioned(expectedVersion == 0)`: replace/upsert with filter matching
|
||||
`{_id:key, version missing or 0}`, duplicate key -> `ErrConflict`.
|
||||
- `Put`: implement as a bounded loop over `GetVersioned` + `PutVersioned` so it
|
||||
overwrites unconditionally while still removing stale fields and bumping
|
||||
version. Use existing retry style (`portfolioUpdateAttempts` is 5) or a
|
||||
small storage-local constant.
|
||||
|
||||
Use `ReplaceOne`, not partial `$set`, for root-payload writes. Partial updates
|
||||
are the foot-gun: `{a:1,b:2}` overwritten by `{a:3}` would otherwise leave `b`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/storage/mongodb_kv.go` — write/read methods and version
|
||||
replacement logic.
|
||||
- Modify: `internal/storage/mongodb_value_codec.go` — convert byte payload to
|
||||
flattened root docs and reconstruct JSON.
|
||||
- Modify: `internal/storage/mongodb_kv_test.go` — make Phase 1 tests pass.
|
||||
- Read: `internal/storage/memory_kv.go` — keep interface behavior aligned.
|
||||
- Read: `internal/storage/dynamodb_kv.go` — ensure migrate-only DynamoDB stays
|
||||
unaffected.
|
||||
|
||||
## Tests Before
|
||||
|
||||
- Run Phase 1 Mongo tests and confirm failure before implementation.
|
||||
- Run hermetic storage tests to ensure no memory/DynamoDB API changes are
|
||||
required.
|
||||
|
||||
## Refactor
|
||||
|
||||
1. Rename legacy `decodeValue` to make compatibility explicit, e.g.
|
||||
`decodeLegacyValueField`.
|
||||
2. Add reserved-field constants and helper `isMongoRootMetadataField`.
|
||||
3. Add root-document encoder:
|
||||
- trim/decode JSON with `json.Decoder.UseNumber`;
|
||||
- object -> root fields if no reserved/legacy `value` collision and no
|
||||
unsafe key containing null, containing `.`, or starting with `$`;
|
||||
- array -> `_payload` + `payloadKind:"array"`;
|
||||
- scalar/non-JSON -> `_payload` + `payloadKind:"string"` or scalar kind;
|
||||
- always add `_id`, `version`, `updatedAt`, `schemaVersion`.
|
||||
<!-- Updated: Validation Session 1 - reserve legacy value + unsafe Mongo field names -->
|
||||
4. Add root-document decoder:
|
||||
- legacy `value` first;
|
||||
- `_payload` fallback second;
|
||||
- root object reconstruction third.
|
||||
5. Replace `$set`/`$inc` writes with whole-document replacement in
|
||||
`PutVersioned`.
|
||||
6. Rework `Put` to bump versions and replace whole docs without exposing
|
||||
conflicts to ordinary callers unless retries are exhausted.
|
||||
7. Keep `Delete` and `List` unchanged.
|
||||
8. Tighten error messages so malformed root docs name the module/key and
|
||||
missing payload reason.
|
||||
|
||||
## Tests After
|
||||
|
||||
- `go test ./internal/storage`
|
||||
- `MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage -run 'MongoKVStore'`
|
||||
- `go test ./internal/modules/coin ./internal/modules/gold ./internal/modules/lolschedule`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All Phase 1 tests pass.
|
||||
- [ ] No new write path emits `value`.
|
||||
- [ ] Payload objects with key `value` use `_payload` fallback, not root
|
||||
flattening.
|
||||
- [ ] Payload objects with null/`.`/`$` field names use `_payload` fallback.
|
||||
- [ ] Legacy `value` docs continue to read.
|
||||
- [ ] Plain `Put` and `PutVersioned` both bump `version`.
|
||||
- [ ] Concurrent create/update tests keep exactly-one-winner semantics.
|
||||
- [ ] Stale payload fields are removed on overwrite.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: bounded `Put` loop can conflict under extreme contention. Mitigation:
|
||||
keep retries small but adequate; if tests show trouble, switch to a
|
||||
single-write aggregation pipeline with `$replaceRoot` and `$literal` payload
|
||||
expressions.
|
||||
- Risk: root reconstruction accidentally includes metadata fields. Mitigation:
|
||||
centralize reserved field list and test it.
|
||||
- Risk: `time.Time` changes raw `updatedAt` type. Mitigation: no app reader uses
|
||||
it today; integration tests assert new date behavior.
|
||||
- Risk: malformed existing docs without `value` or root payload become harder
|
||||
to diagnose. Mitigation: descriptive errors in decoder.
|
||||
-126
@@ -1,126 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Migration and Documentation"
|
||||
status: pending
|
||||
priority: P2
|
||||
dependencies: [2]
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 3: Migration and Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
Add an operator-safe path to rewrite existing Mongo docs from the old `value`
|
||||
envelope into the new flattened root shape, and update docs that currently
|
||||
describe the old layout.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: existing Atlas documents can be migrated in place without relying
|
||||
on DynamoDB still existing.
|
||||
- Functional: migration is idempotent and reports per-collection counts.
|
||||
- Functional: migration preserves logical `Get`/`GetJSON` values and bumps or
|
||||
normalizes `version` consistently.
|
||||
- Non-functional: migration must not read or print secrets; `MONGO_URL` remains
|
||||
secret.
|
||||
- Non-functional: docs must stop calling int64 `updatedAt` TTL-ready.
|
||||
|
||||
## Architecture
|
||||
|
||||
Add a small Mongo-to-Mongo schema migration command rather than overloading the
|
||||
DynamoDB migrator:
|
||||
|
||||
```text
|
||||
cmd/migrate-mongo-schema/
|
||||
main.go
|
||||
main_test.go
|
||||
```
|
||||
|
||||
Tool behavior:
|
||||
|
||||
1. Connect using `MONGO_URL` + `MONGO_DATABASE`.
|
||||
2. Iterate collections or an allow-list flag.
|
||||
3. For each document:
|
||||
- skip if it already has `schemaVersion >= 2` and no `value`;
|
||||
- read through `MongoKVStore.GetVersioned` so legacy decode paths are used;
|
||||
- rewrite through `PutVersioned` so root encoding is shared with app writes;
|
||||
- on `ErrConflict`, re-read the document: skip it if a live app write
|
||||
already migrated it, otherwise retry up to a small bounded limit;
|
||||
- count migrated/skipped/errors.
|
||||
4. `--dry-run` prints counts without writes.
|
||||
5. `--verify` confirms no documents with `value` remain, except explicitly
|
||||
allowed malformed docs if any are reported.
|
||||
|
||||
Do not drop or rename collections. Keep one collection per module.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `cmd/migrate-mongo-schema/main.go`
|
||||
- Create: `cmd/migrate-mongo-schema/main_test.go`
|
||||
- Modify: `Makefile` — add a migration target, e.g. `migrate-mongo-schema`.
|
||||
- Modify: `README.md` — update storage layout summary.
|
||||
- Modify: `docs/deploy-coolify-selfhosted.md` — update Atlas storage layout
|
||||
and migration note.
|
||||
- Modify: `cmd/migrate-dynamo-to-mongo/README.md` — update destination shape.
|
||||
- Modify: `cmd/migrate-dynamo-to-mongo/main.go` comments only if stale.
|
||||
- Read: `cmd/migrate-dynamo-to-mongo/main.go` — reuse connection/config style.
|
||||
|
||||
## Tests Before
|
||||
|
||||
- Add migrator e2e test that seeds old `value` docs directly, runs migration,
|
||||
then asserts:
|
||||
- logical values round-trip;
|
||||
- raw docs have no `value`;
|
||||
- `updatedAt` is `time.Time`;
|
||||
- counts match.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Implement CLI flags:
|
||||
- `--dry-run`
|
||||
- `--verify`
|
||||
- `--collection` optional repeat/single collection selector.
|
||||
2. Reuse `storage.NewMongoClient`, `storage.NewMongoDatabase`, and
|
||||
`storage.NewMongoProvider`.
|
||||
3. Add collection listing using the Mongo driver, excluding system collections.
|
||||
4. Rewrite docs through the storage layer, not raw BSON mutation.
|
||||
5. Print a concise table: collection, total, migrated, skipped, errors.
|
||||
6. Add Makefile target with env examples.
|
||||
7. Update docs and remove stale wording:
|
||||
- no old `value` envelope in new writes;
|
||||
- `updatedAt` is BSON Date;
|
||||
- legacy `value` docs dual-read until migrated.
|
||||
8. Add migration conflict tests: simulate a version change between read and
|
||||
write; assert the migrator re-reads and either skips already-migrated docs or
|
||||
retries legacy docs without clobbering the concurrent update.
|
||||
<!-- Updated: Validation Session 1 - migration must handle live-write conflicts -->
|
||||
|
||||
## Tests After
|
||||
|
||||
- `go test ./cmd/migrate-mongo-schema`
|
||||
- `MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./cmd/migrate-mongo-schema ./internal/storage`
|
||||
- `make test`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] In-place migration rewrites old `value` docs without changing logical app
|
||||
values.
|
||||
- [ ] Migration is idempotent.
|
||||
- [ ] `--dry-run` performs no writes.
|
||||
- [ ] `--verify` exits non-zero when old `value` docs remain.
|
||||
- [ ] Migration handles version conflicts by re-reading and never clobbers a
|
||||
concurrent app write.
|
||||
- [ ] README and Coolify deploy docs show the new flattened layout.
|
||||
- [ ] DynamoDB-to-Mongo migrator docs no longer mention BSON binary/string as
|
||||
the destination `value` representation.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: migration touches live data. Mitigation: dry-run + verify + local e2e
|
||||
test; recommend backup/export before production run.
|
||||
- Risk: old DynamoDB source may be gone. Mitigation: in-place Mongo migration is
|
||||
independent of DynamoDB.
|
||||
- Risk: malformed documents stop migration. Mitigation: report collection/key
|
||||
and continue only if an explicit `--continue-on-error` is later accepted;
|
||||
default should fail loud.
|
||||
@@ -1,121 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Verification and Rollout"
|
||||
status: pending
|
||||
priority: P2
|
||||
dependencies: [3]
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 4: Verification and Rollout
|
||||
|
||||
## Overview
|
||||
|
||||
Run the full validation suite, check raw Mongo documents, and define the safe
|
||||
production rollout order. This phase is the gate before deploying a storage
|
||||
schema change.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: app behavior remains unchanged for all Telegram commands.
|
||||
- Functional: Mongo raw shape is verified for representative module documents.
|
||||
- Functional: legacy docs remain readable before migration and absent after
|
||||
migration verify.
|
||||
- Non-functional: rollout includes backup, deploy, migration, verify, and
|
||||
rollback notes.
|
||||
|
||||
## Architecture
|
||||
|
||||
Rollout order:
|
||||
|
||||
1. Backup/export current Atlas database or snapshot if available.
|
||||
2. Deploy app version with dual-read + new-write root encoding.
|
||||
3. Let app run; new/touched docs rewrite naturally.
|
||||
4. Run `migrate-mongo-schema --dry-run`.
|
||||
5. Run `migrate-mongo-schema`.
|
||||
6. Run `migrate-mongo-schema --verify`.
|
||||
7. Inspect sample docs in Atlas/Compass:
|
||||
- coin/gold/stock portfolio root fields;
|
||||
- wordle/loldle/twentyq game state root fields;
|
||||
- lolschedule scalar date guard under `_payload`;
|
||||
- no old `value` on migrated docs.
|
||||
|
||||
Rollback:
|
||||
|
||||
- If the new app fails before it handles writes, roll back binary; old `value`
|
||||
docs are still readable by the previous build.
|
||||
- Once the new app has handled writes, rollback to the previous binary is unsafe
|
||||
for any touched flattened docs. Prefer fixing forward, or restore the backup.
|
||||
- If migration ran, previous build may not read flattened docs at all. Rollback
|
||||
after migration requires restoring backup or a reverse migrator. Prefer fixing
|
||||
forward unless the migration corrupted data.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: tests only if verification gaps are found.
|
||||
- Read: `Makefile`
|
||||
- Read: `README.md`
|
||||
- Read: `docs/deploy-coolify-selfhosted.md`
|
||||
- Read: `cmd/migrate-mongo-schema/main.go`
|
||||
- Read: `internal/storage/mongodb_kv_test.go`
|
||||
|
||||
## Tests Before
|
||||
|
||||
- Confirm Phase 1-3 tests run locally.
|
||||
- Start local Mongo with `make mongo-local` for integration tests.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Run focused storage tests.
|
||||
2. Run module tests that rely on versioned storage flows:
|
||||
- coin portfolio;
|
||||
- gold portfolio;
|
||||
- lolschedule daily-push claim.
|
||||
3. Run full hermetic suite.
|
||||
4. Run Mongo integration suite.
|
||||
5. Run DynamoDB tests only if the DynamoDB migrator or shared storage contracts
|
||||
changed.
|
||||
6. Manually inspect raw docs or add a small test assertion report for:
|
||||
- no `value`;
|
||||
- expected root fields;
|
||||
- Date `updatedAt`;
|
||||
- stable `version`.
|
||||
7. Document production run commands in the final completion report.
|
||||
|
||||
## Tests After
|
||||
|
||||
Required gates:
|
||||
|
||||
```sh
|
||||
make vet
|
||||
make test
|
||||
MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage ./cmd/migrate-mongo-schema
|
||||
```
|
||||
|
||||
Conditional gate if DynamoDB migrator comments/tests changed:
|
||||
|
||||
```sh
|
||||
make test-dynamodb
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `make vet` passes.
|
||||
- [ ] `make test` passes.
|
||||
- [ ] Mongo integration tests pass against local Mongo.
|
||||
- [ ] New migrator e2e test passes.
|
||||
- [ ] Raw docs verified no longer use `value` after migration.
|
||||
- [ ] Production rollout notes include backup and rollback constraints.
|
||||
- [ ] No user-visible Telegram command behavior changes.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: previous binary cannot read newly-written or post-migration flattened
|
||||
docs. Mitigation: deploy dual-read binary first; backup before rollout;
|
||||
rollback old binary only before new writes happen; fix forward preferred.
|
||||
- Risk: Atlas M0 resource limits during migration. Mitigation: tiny dataset
|
||||
expected; still scan collection-by-collection and report progress.
|
||||
- Risk: hidden module stores non-object values. Mitigation: fallback `_payload`
|
||||
contract and migration tests for scalar/array docs.
|
||||
- Risk: docs drift. Mitigation: grep for old `{ _id, value, version, updatedAt }`
|
||||
wording before finalizing.
|
||||
@@ -1,221 +0,0 @@
|
||||
---
|
||||
title: "Flatten Mongo value documents"
|
||||
description: "Remove Mongo's generic value envelope for JSON object values, keep KVStore callers stable, and store Mongo metadata in queryable root fields."
|
||||
status: superseded
|
||||
supersededBy: 260628-1318-mongo-native-typed-stores
|
||||
priority: P2
|
||||
branch: "feature/selfhosted"
|
||||
tags: [database, mongodb, refactor, storage, tdd]
|
||||
blockedBy: [260627-1849-selfhost-coolify-mongodb]
|
||||
blocks: []
|
||||
created: "2026-06-28T06:10:29.184Z"
|
||||
createdBy: "ck:plan"
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Flatten Mongo value documents
|
||||
|
||||
> **Superseded (2026-06-28)** by
|
||||
> [`260628-1318-mongo-native-typed-stores`](../260628-1318-mongo-native-typed-stores/plan.md).
|
||||
> User chose the full Mongo-native direction this plan deferred: delete the
|
||||
> `KVStore` abstraction in favor of typed stores, MongoDB-only runtime (memory
|
||||
> kept for tests), and drop the legacy `value` dual-read + in-place Mongo
|
||||
> migrator (Atlas is empty until cutover, so the DynamoDB→Mongo migrator writes
|
||||
> the final flattened shape directly). Kept for history; do not implement.
|
||||
|
||||
## Overview
|
||||
|
||||
Change MongoDB's on-disk KV document shape from a generic envelope:
|
||||
|
||||
```javascript
|
||||
{ _id, value: { ... }, version, updatedAt }
|
||||
```
|
||||
|
||||
to a more Mongo-native root-document shape for JSON object values:
|
||||
|
||||
```javascript
|
||||
{ _id, ...payloadFields, version, updatedAt: ISODate(...), schemaVersion }
|
||||
```
|
||||
|
||||
Keep the public `KVStore` / `VersionedStore` interfaces stable so modules keep
|
||||
using `GetJSON`, `PutJSON`, `GetVersioned`, and `PutVersioned`. This plan is a
|
||||
storage-representation refactor, not a full rewrite to typed repositories.
|
||||
|
||||
Non-object values cannot be Mongo root documents. New writes for arrays/scalars
|
||||
use a small reserved `_payload` fallback with `payloadKind`, but **new writes
|
||||
must not use the old `value` field**. Legacy docs with `value` remain readable
|
||||
and are rewritten to the new shape by normal updates or by the in-place
|
||||
migration tool.
|
||||
|
||||
## Scope Challenge
|
||||
|
||||
- Existing code: `MongoKVStore` already owns all Mongo encoding/decoding and
|
||||
version CAS; `mongodb_value_codec.go` already preserves `int64` via
|
||||
`json.Number`; modules do not need direct Mongo access.
|
||||
- Minimum changes: storage codec + raw-shape tests + migrator/docs. Defer
|
||||
per-module typed repositories until there is a real query/report need.
|
||||
- Complexity: touches storage, one migration command, docs, and tests. Avoid
|
||||
changing module handlers or public storage interfaces.
|
||||
- Selected mode: HOLD SCOPE. Refactor the persisted Mongo shape; do not expand
|
||||
into a full domain repository rewrite.
|
||||
|
||||
## Design Decision
|
||||
|
||||
Chosen approach: **root-payload KV documents**.
|
||||
|
||||
Why not full typed repositories now: it would touch most modules, duplicate
|
||||
test plumbing, and remove the memory backend value before there is a query need.
|
||||
Why not keep `value`: it defeats the user's Mongo document-design goal and
|
||||
keeps domain fields one level away from simple Compass/query/index use.
|
||||
|
||||
Reserved root fields:
|
||||
|
||||
| Field | Purpose |
|
||||
|---|---|
|
||||
| `_id` | Module-local KV key, unchanged |
|
||||
| `value` | Legacy envelope field; forbidden in new root payloads |
|
||||
| `version` | Optimistic-lock token, unchanged semantics |
|
||||
| `updatedAt` | BSON Date (`time.Time`), not unix nanos |
|
||||
| `schemaVersion` | Storage document shape version |
|
||||
| `payloadKind` | Only needed for non-object fallback or collision fallback |
|
||||
| `_payload` | Array/scalar payload fallback; never used for ordinary JSON objects |
|
||||
|
||||
Mongo field-name guard: object payloads also fall back to `_payload` if any
|
||||
flattened key contains the null character, contains `.`, or starts with `$`.
|
||||
MongoDB permits `.` and `$` in modern versions but documents restrictions around
|
||||
them; fallback keeps normal query/index/schema behavior predictable.
|
||||
Reference: https://www.mongodb.com/docs/manual/core/document/#field-names
|
||||
|
||||
Decoder rules:
|
||||
|
||||
1. If old `value` exists, decode using the legacy path.
|
||||
2. Else if `_payload` exists, decode `_payload` according to `payloadKind`.
|
||||
3. Else reconstruct the JSON object from all non-reserved root fields.
|
||||
|
||||
Writer rules:
|
||||
|
||||
1. JSON object without reserved-field collision writes payload fields at root.
|
||||
2. JSON array, scalar, invalid JSON, or object with reserved-field collision
|
||||
or unsafe Mongo field names writes `_payload` + `payloadKind`.
|
||||
3. Writes replace the whole document (except the `_id` key value is preserved in
|
||||
the replacement) so stale old payload fields cannot survive an overwrite.
|
||||
4. `PutVersioned` keeps single-document CAS semantics by filtering on `_id` and
|
||||
`version`.
|
||||
5. Plain `Put` must still bump `version`; implement as a bounded versioned
|
||||
write loop or a carefully tested aggregation update pipeline. Prefer the
|
||||
simpler loop unless tests prove unacceptable contention.
|
||||
|
||||
## Cross-Plan Dependencies
|
||||
|
||||
| Relationship | Plan | Status | Rationale |
|
||||
|---|---|---|---|
|
||||
| Blocked by | `260627-1849-selfhost-coolify-mongodb` | in-progress | Created the Mongo provider, migrator, and self-host runtime this refactor modifies. |
|
||||
| Supersedes storage choice from | `260628-1113-mongo-native-value-documents` | completed | Keeps its version CAS + native BSON goal, but replaces the `value` envelope with root payload fields. |
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Schema Contract and Regression Tests](./phase-01-schema-contract-and-regression-tests.md) | Pending |
|
||||
| 2 | [Root Document Storage Encoding](./phase-02-root-document-storage-encoding.md) | Pending |
|
||||
| 3 | [Migration and Documentation](./phase-03-migration-and-documentation.md) | Pending |
|
||||
| 4 | [Verification and Rollout](./phase-04-verification-and-rollout.md) | Pending |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Existing Mongo provider and version-CAS code from
|
||||
`plans/260628-1113-mongo-native-value-documents/`.
|
||||
- Research report:
|
||||
`plans/reports/research-260628-0605-mongodb-document-design-standards.md`.
|
||||
- MongoDB official guidance used in the research report:
|
||||
- https://www.mongodb.com/docs/manual/data-modeling/
|
||||
- https://www.mongodb.com/docs/manual/data-modeling/best-practices/
|
||||
- https://www.mongodb.com/docs/manual/core/write-operations-atomicity/
|
||||
- https://www.mongodb.com/docs/manual/core/index-ttl/
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] New JSON object writes contain no top-level `value` field; payload fields
|
||||
live at root with `_id`, `version`, `updatedAt`, and `schemaVersion`.
|
||||
- [ ] Payload objects with top-level `value`, reserved metadata keys, null
|
||||
characters, `.`, or `$`-prefixed keys use `_payload` fallback and round-trip.
|
||||
- [ ] New array/scalar/non-JSON writes contain no old `value` field; they use
|
||||
`_payload` + `payloadKind` and round-trip byte-equivalent where required.
|
||||
- [ ] `updatedAt` stores as BSON Date / Go `time.Time`, not int64 nanos.
|
||||
- [ ] Legacy docs with `value` as string, binary, object, or array still read.
|
||||
- [ ] Overwriting a document removes stale payload fields from prior values.
|
||||
- [ ] Versioned writes remain single-winner under concurrent create/update.
|
||||
- [ ] A Mongo in-place migration can rewrite existing `value` docs to the new
|
||||
shape without changing logical values or document counts.
|
||||
- [ ] `make test`, `make vet`, and Mongo integration tests pass.
|
||||
|
||||
## Not In Scope
|
||||
|
||||
- Replacing every module's `KVStore` usage with typed Mongo repositories.
|
||||
- Adding Mongo schema validation/indexes for every module.
|
||||
- Deleting DynamoDB migrator support.
|
||||
- Changing Telegram bot behavior or command outputs.
|
||||
|
||||
## Validation Log
|
||||
|
||||
### Session 1 — 2026-06-28
|
||||
**Trigger:** `/ck:plan validate /config/workspace/tiennm99/miti99bot/plans/260628-1310-flatten-mongo-value-documents/plan.md`
|
||||
**Questions asked:** 0 — validation found technical plan gaps with clear repo/doc evidence; no user-facing trade-off needed.
|
||||
|
||||
#### Plan Recap
|
||||
|
||||
- Refactor Mongo on-disk KV documents from `{ _id, value, version, updatedAt }`
|
||||
to root payload fields.
|
||||
- Keep `KVStore` / `VersionedStore` caller contracts stable.
|
||||
- Keep legacy `value` docs readable.
|
||||
- Use `_payload` only for non-object or unsafe/colliding object payloads.
|
||||
- Store `updatedAt` as BSON Date.
|
||||
- Add an in-place Mongo schema migration and rollout guardrails.
|
||||
|
||||
#### Verification Results
|
||||
|
||||
- **Tier:** Standard (4 phases)
|
||||
- **Claims checked:** 32
|
||||
- **Verified:** 29 | **Failed:** 3 | **Unverified:** 0
|
||||
|
||||
Failures fixed in this validation:
|
||||
|
||||
1. [Contract Verifier] Reserved fields omitted `value`. Evidence:
|
||||
`internal/storage/mongodb_kv.go:19` uses `mongoValueField = "value"` and
|
||||
`decodeValue` treats any top-level `value` as legacy payload
|
||||
(`internal/storage/mongodb_kv.go:46`). A flattened payload containing
|
||||
top-level `value` would violate the no-`value` contract and decode wrong.
|
||||
2. [Fact Checker] Field-name guard omitted null/`.`/`$` cases. Evidence:
|
||||
MongoDB documents forbid null in field names and document restrictions for
|
||||
`.`/`$` field names: https://www.mongodb.com/docs/manual/core/document/#field-names.
|
||||
The plan now falls back to `_payload` for those keys.
|
||||
3. [Contract Verifier] Migration rewrite via `PutVersioned` lacked conflict
|
||||
retry handling. Evidence: `PutVersioned` returns `ErrConflict` on version
|
||||
mismatch (`internal/storage/kv_store.go:35`) and concurrent app writes are
|
||||
possible after deploying the dual-read/new-write binary. Phase 3 now requires
|
||||
re-read/skip/retry behavior.
|
||||
|
||||
#### Confirmed Decisions
|
||||
|
||||
- Keep `value` reserved forever for backward-compatible legacy decode.
|
||||
- Use `_payload` fallback for reserved keys and Mongo-awkward field names.
|
||||
- Keep migration writes through storage encoding and handle conflicts by
|
||||
re-reading, skipping already-migrated docs, or retrying.
|
||||
|
||||
#### Impact on Phases
|
||||
|
||||
- Phase 1: add tests for payload key `value`, null/`.`/`$` field names, and
|
||||
migration conflict setup.
|
||||
- Phase 2: update reserved-field and unsafe-field encoder rules.
|
||||
- Phase 3: add conflict retry semantics to the migration command.
|
||||
|
||||
### Whole-Plan Consistency Sweep
|
||||
|
||||
- Files reread: `plan.md`, all four `phase-*.md` files.
|
||||
- Decision deltas checked: 3 (`value` reserved, unsafe field fallback, migration conflict retry).
|
||||
- Reconciled stale references: 6.
|
||||
- Unresolved contradictions: 0.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None.
|
||||
-107
@@ -1,107 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Typed DocStore contract + memory backend"
|
||||
status: done
|
||||
priority: P2
|
||||
dependencies: []
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 1: Typed DocStore contract + memory backend
|
||||
|
||||
## Overview
|
||||
|
||||
Define the new typed storage contract and a memory implementation first, with
|
||||
tests, before touching Mongo or any module. This nails the interface every
|
||||
module and the Mongo backend will target.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `DocStore[T]` generic interface: `Get`, `Put`, `PutVersioned`, `Delete`, `List`.
|
||||
- `Provider.Collection(module)` returns an opaque per-module `Collection`.
|
||||
- `Typed[T](Collection) DocStore[T]` free function (methods cannot be generic).
|
||||
- Memory implementation: hermetic, concurrency-safe, version-CAS correct.
|
||||
- Reserved-field name guard: payload BSON tags must not collide with `_id`,
|
||||
`version`, `updatedAt`.
|
||||
- `ErrNotFound`, `ErrConflict` reused (move to this package if `kv_store.go` is deleted later).
|
||||
|
||||
## Architecture
|
||||
|
||||
```go
|
||||
// doc_store.go
|
||||
type DocStore[T any] interface {
|
||||
Get(ctx, id) (T, int64, error)
|
||||
Put(ctx, id, T) error
|
||||
PutVersioned(ctx, id, expectedVersion int64, T) error
|
||||
Delete(ctx, id) error
|
||||
List(ctx, prefix string) ([]string, error)
|
||||
}
|
||||
|
||||
type Provider interface{ Collection(module string) Collection }
|
||||
type Collection interface{ collection() } // sealed marker
|
||||
|
||||
func Typed[T any](c Collection) DocStore[T] // switch concrete type → mongo|memory store
|
||||
```
|
||||
|
||||
Memory backend:
|
||||
|
||||
```go
|
||||
// memory_provider.go (replaces kv_provider.go's MemoryProvider)
|
||||
type MemoryProvider struct{ mu sync.Mutex; cols map[string]*memoryCollection }
|
||||
type memoryCollection struct{ mu sync.Mutex; rows map[string]memoryRow }
|
||||
type memoryRow struct{ version int64; val any } // val holds the typed value
|
||||
|
||||
type MemoryDocStore[T any] struct{ c *memoryCollection }
|
||||
```
|
||||
|
||||
- `Get`: type-assert `row.val.(T)`; missing → `ErrNotFound`.
|
||||
- `Put`: overwrite, `version++`.
|
||||
- `PutVersioned`: compare stored version to expected; mismatch → `ErrConflict`;
|
||||
`expected==0` requires absent/zero-version row.
|
||||
- `Delete`: idempotent.
|
||||
- `List(prefix)`: sorted keys with prefix (reuse `validatePrefix`).
|
||||
|
||||
Keys: keep `validateKey`/`validatePrefix` from `keys.go`/`prefix.go`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/storage/doc_store.go` (interface, Provider, Collection, `Typed`, errors).
|
||||
- Create: `internal/storage/memory_provider.go` (MemoryProvider + memoryCollection + MemoryDocStore).
|
||||
- Create: `internal/storage/memory_doc_store_test.go`.
|
||||
- Create: `internal/storage/reserved_fields.go` + test (reserved-name guard helper).
|
||||
- Read: `internal/storage/keys.go`, `prefix.go`, `kv_store.go` (for error vars to migrate).
|
||||
|
||||
## Tests Before
|
||||
|
||||
Write failing tests first:
|
||||
|
||||
1. `TestMemoryDocStore_PutGetRoundTrip` — typed struct round-trips; version starts at 1.
|
||||
2. `TestMemoryDocStore_PutBumpsVersion` — repeated Put increments version.
|
||||
3. `TestMemoryDocStore_PutVersioned_CAS` — stale expected → `ErrConflict`; fresh succeeds.
|
||||
4. `TestMemoryDocStore_PutVersioned_CreateZero` — `expected==0` creates; second create → `ErrConflict`.
|
||||
5. `TestMemoryDocStore_Delete_Idempotent`.
|
||||
6. `TestMemoryDocStore_List_Prefix` — disjoint key prefixes isolate types in one collection.
|
||||
7. `TestReservedFields_RejectsCollision` — a payload type tagged `version` is flagged.
|
||||
8. `TestMemoryProvider_CollectionIsolation` — two modules' collections don't share keys.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add `doc_store.go` with interface, sealed `Collection`, `Typed[T]`, and the
|
||||
`ErrNotFound`/`ErrConflict` vars (kept here so `kv_store.go` can be deleted).
|
||||
2. Implement memory provider + store.
|
||||
3. Implement reserved-field reflection guard (`bson` tag scan).
|
||||
4. Make all Phase 1 tests pass.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `DocStore[T]`, `Provider`, `Collection`, `Typed` compile and are documented.
|
||||
- [ ] Memory store passes CAS/version/list/delete tests.
|
||||
- [ ] Reserved-name guard catches `_id`/`version`/`updatedAt` collisions.
|
||||
- [ ] No dependency on the soon-to-be-deleted byte `KVStore`.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: generic free-function `Typed[T]` is unusual. Mitigation: document why
|
||||
(Go methods can't be generic); single switch point, well tested.
|
||||
- Risk: memory `val any` type mismatch panics. Mitigation: `Get` returns a typed
|
||||
error on failed assertion, not a panic.
|
||||
@@ -1,116 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Mongo typed store + integration tests"
|
||||
status: done
|
||||
priority: P2
|
||||
dependencies: [1]
|
||||
effort: "L"
|
||||
---
|
||||
|
||||
# Phase 2: Mongo typed store + integration tests
|
||||
|
||||
## Overview
|
||||
|
||||
Implement `MongoDocStore[T]` and a `MongoProvider` that satisfies the Phase 1
|
||||
contract, persisting typed payloads as flattened native root documents. Lock the
|
||||
raw shape and CAS semantics with Mongo integration tests.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `MongoProvider.Collection(module)` returns a Mongo-backed `Collection`.
|
||||
- `Typed[T]` over a Mongo collection yields `MongoDocStore[T]`.
|
||||
- Write shape: `{ _id, ...payload, version, updatedAt(Date) }`; no `value`, no `_payload`.
|
||||
- Whole-document `ReplaceOne` (no `$set`) so overwrites drop stale fields.
|
||||
- Version CAS identical in behavior to the current `MongoKVStore`.
|
||||
- `updatedAt` stored as `time.Time`.
|
||||
- Plain `Put` bumps version via a bounded versioned-write retry loop.
|
||||
|
||||
## Architecture
|
||||
|
||||
```go
|
||||
type storedDoc[T any] struct {
|
||||
ID string `bson:"_id"`
|
||||
Version int64 `bson:"version"`
|
||||
UpdatedAt time.Time `bson:"updatedAt"`
|
||||
Payload T `bson:",inline"`
|
||||
}
|
||||
|
||||
type MongoDocStore[T any] struct{ coll *mongo.Collection }
|
||||
```
|
||||
|
||||
- `Get`: `FindOne({_id})` → decode `storedDoc[T]`; `ErrNoDocuments`→`ErrNotFound`;
|
||||
return `Payload`, `Version`.
|
||||
- `PutVersioned(expected>0)`: `ReplaceOne({_id, version:expected}, storedDoc{version:expected+1, now, payload})`;
|
||||
`MatchedCount==0`→`ErrConflict`.
|
||||
- `PutVersioned(0)`: upsert filter `{_id, $or:[version absent, version:0]}` with
|
||||
replacement `version:1`; duplicate-key→`ErrConflict`. (Port existing logic.)
|
||||
- `Put`: loop `Get`→`PutVersioned(version)` up to N attempts (reuse the small
|
||||
retry constant style already in coin/gold); first write uses `expected=0`.
|
||||
- `Delete`: `DeleteOne` idempotent.
|
||||
- `List(prefix)`: half-open `_id` range scan via `prefixSuccessor`, project `_id`
|
||||
only. (Port from current `MongoKVStore.List`.)
|
||||
- Error messages name module + key + reason.
|
||||
|
||||
`MongoProvider` wraps `*mongo.Database`; `Collection(module)` validates the
|
||||
module name (reuse existing validation) and returns `mongoCollection{db.Collection(module)}`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/storage/mongo_doc_store.go` (MongoDocStore + storedDoc).
|
||||
- Modify: `internal/storage/mongodb_provider.go` (implement new `Provider`/`Collection`).
|
||||
- Create: `internal/storage/mongo_doc_store_test.go` (integration, gated by `MONGODB_TEST_URL`).
|
||||
- Delete (this phase): `mongodb_kv.go`, `mongodb_value_codec.go`, `mongodb_kv_test.go`.
|
||||
- Read: current `mongodb_kv.go` (port CAS + List), `mongodb_provider.go`.
|
||||
|
||||
## Tests Before
|
||||
|
||||
Gated by `MONGODB_TEST_URL`:
|
||||
|
||||
1. `TestMongoDocStore_RootShape` — Put a portfolio-like struct; raw `bson.M` has
|
||||
root payload fields, `version`, `updatedAt`; **no** `value`, **no** `_payload`;
|
||||
`updatedAt` decodes as `time.Time`.
|
||||
2. `TestMongoDocStore_OverwriteRemovesStaleFields` — write struct A with field
|
||||
set X, overwrite with struct B lacking some of X; raw doc has no stale field.
|
||||
3. `TestMongoDocStore_PutVersioned_ConcurrentCreate` — N goroutines create same
|
||||
`_id`; exactly one wins, rest get `ErrConflict`.
|
||||
4. `TestMongoDocStore_PutVersioned_Update` — stale expected → `ErrConflict`.
|
||||
5. `TestMongoDocStore_Put_BumpsVersion` — plain Put increments version, survives
|
||||
a concurrent bump (retry loop succeeds or conflicts deterministically).
|
||||
6. `TestMongoDocStore_List_Prefix`.
|
||||
7. `TestMongoDocStore_WrappedScalar` — a `struct{ Date string }` payload stores
|
||||
`date` at root (proves the lolschedule wrapping works).
|
||||
8. `TestMongoDocStore_WrappedArray` — a `struct{ Subscribers []sub }` payload
|
||||
stores `subscribers` array at root.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Implement `MongoDocStore[T]` (port CAS + List from `MongoKVStore`, typed).
|
||||
2. Reshape `MongoProvider` to the new `Provider`/`Collection`; wire `Typed[T]`.
|
||||
3. Delete `mongodb_kv.go`, `mongodb_value_codec.go` and their tests.
|
||||
4. `go build ./internal/storage` — expect module/server breakage deferred to
|
||||
Phase 3 (storage package itself must compile).
|
||||
5. Make Phase 2 integration tests pass against `make mongo-local`.
|
||||
|
||||
## Tests After
|
||||
|
||||
```sh
|
||||
go test ./internal/storage
|
||||
MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage -run 'MongoDocStore'
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Raw Mongo doc is flattened root payload + `version` + Date `updatedAt`; no `value`/`_payload`.
|
||||
- [ ] Overwrite removes stale fields.
|
||||
- [ ] Concurrent create/update keep single-winner CAS.
|
||||
- [ ] Wrapped scalar/array payloads store as named root fields.
|
||||
- [ ] Old `MongoKVStore`/codec deleted; storage package builds.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: `bson:",inline"` edge cases (pointer payloads, embedded maps). Mitigation:
|
||||
require payload `T` to be a concrete struct; test scalar/array wrappers.
|
||||
- Risk: bounded `Put` loop contention. Mitigation: small retry count; CAS users
|
||||
call `PutVersioned` directly anyway.
|
||||
- Risk: driver decodes `version` as int32. Mitigation: `storedDoc.Version int64`
|
||||
+ decode test.
|
||||
@@ -1,115 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Migrate modules + wiring; delete KVStore"
|
||||
status: done
|
||||
priority: P2
|
||||
dependencies: [2]
|
||||
effort: "XL"
|
||||
---
|
||||
|
||||
# Phase 3: Migrate modules + wiring; delete KVStore
|
||||
|
||||
## Overview
|
||||
|
||||
Switch every module from the byte `KVStore` to typed `DocStore[T]`, rewire
|
||||
`Deps`/registry/server, and delete the old byte abstraction and DynamoDB runtime
|
||||
backend. This is the largest phase; do it module-by-module with tests green
|
||||
after each.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `Deps.KV storage.KVStore` → `Deps.Store storage.Collection`.
|
||||
- `registry.Build` takes `storage.Provider` instead of `storage.KVProvider`.
|
||||
- Each module builds typed stores via `storage.Typed[T](deps.Store)`.
|
||||
- CAS behavior preserved for coin, gold, lolschedule last-push.
|
||||
- `lolschedule` subscribers + last-push wrapped in named structs.
|
||||
- No remaining references to `KVStore`, `VersionedStore`, `PutJSON`, `GetJSON`,
|
||||
`kv.Put`/`kv.Get` byte calls, or the DynamoDB runtime backend.
|
||||
- `cmd/server/main.go` `buildProvider` returns a Mongo `Provider` (or memory for
|
||||
`MODULES=`/no-`MONGO_URL` local runs); the `dynamodb` runtime branch is removed.
|
||||
|
||||
## Per-module migration map
|
||||
|
||||
| Module | Keys / payload types | Store ops | Notes |
|
||||
|---|---|---|---|
|
||||
| coin | `Portfolio` (per user) | Get/PutVersioned (CAS loop) | typed CAS via `DocStore[Portfolio]` |
|
||||
| gold | `Portfolio` (per user) | Get/PutVersioned (CAS loop) | same |
|
||||
| stock | `Portfolio` (per user) | Get/Put | no CAS today; keep plain Put |
|
||||
| wordle | `GameState`, `Stats` | Get/Put | two typed views, disjoint key prefixes |
|
||||
| loldle | `gameState`, `stats`, `roundConfig` | Get/Put/Delete | three typed views |
|
||||
| twentyq | `GameState`, `Stats` | Get/Put/Delete | two typed views |
|
||||
| lolschedule | `subscribers` (wrap `[]Subscriber`), `lastPush` (wrap date string) | List/Put + CAS on last-push | **wrap array+scalar in named structs** |
|
||||
| stats | counter entry struct(s) | Get/Put/List | List-heavy (`views.go`) |
|
||||
| misc | `lastPing` struct | Get/Put | |
|
||||
| util | none | — | no storage |
|
||||
|
||||
`lolschedule` wrappers:
|
||||
|
||||
```go
|
||||
type subscribersDoc struct { Subscribers []Subscriber `bson:"subscribers" json:"subscribers"` }
|
||||
type lastPushDoc struct { Date string `bson:"date" json:"date"` }
|
||||
```
|
||||
|
||||
`claimDailyPush` uses `DocStore[lastPushDoc].Get`/`PutVersioned` for the
|
||||
single-winner daily claim (replaces the current `VersionedStore` byte path).
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/module.go` (Deps.Store), `registry.go` (Build signature, `Collection(name)`).
|
||||
- Modify: every module's storage-touching file (see map) + their factories
|
||||
(`coin.go`, `gold.go`, … `New(deps)` build typed stores).
|
||||
- Modify: `cmd/server/main.go` `buildProvider` (Mongo|memory only).
|
||||
- Modify: all module `*_test.go` that construct stores (use `MemoryProvider`/`Typed`).
|
||||
- Delete: `kv_store.go`, `kv_provider.go`, `memory_kv.go`, `memory_kv_test.go`,
|
||||
`dynamodb_kv.go`, `dynamodb_provider.go`, `dynamodb_kv_test.go`,
|
||||
`dynamodb_provider_test.go`, `invalid_store.go`.
|
||||
|
||||
## Tests Before / After (per module)
|
||||
|
||||
For each module: update its tests to build a `MemoryProvider`, get
|
||||
`deps.Store = provider.Collection(name)`, then run existing behavior assertions.
|
||||
Run `go test ./internal/modules/<m>/...` green before moving on. Order:
|
||||
misc → stock → wordle → loldle → twentyq → coin → gold → stats → lolschedule
|
||||
(simplest first; CAS + List modules last).
|
||||
|
||||
After all modules:
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go test ./internal/modules/...
|
||||
grep -rn "KVStore\|VersionedStore\|PutJSON\|GetJSON\|\.For(" internal/ cmd/ # expect: none (except migrator handled in Phase 4)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add `Deps.Store storage.Collection`; change `registry.Build` to take
|
||||
`storage.Provider` and set `Deps.Store = provider.Collection(name)`.
|
||||
2. Migrate modules one at a time per the order above; keep the suite green.
|
||||
3. For coin/gold, replace the `kv.(VersionedStore)` assertion + byte CAS loop
|
||||
with `DocStore[Portfolio]` Get/PutVersioned (same retry count).
|
||||
4. For lolschedule, introduce the wrapper structs and migrate subscribers (List +
|
||||
Put) and last-push (CAS).
|
||||
5. Update `cmd/server/main.go`: drop the `dynamodb` runtime branch; auto-detect =
|
||||
Mongo when `MONGO_URL` set, else memory (warn). Keep secret-safe logging.
|
||||
6. Delete the dead files listed above; ensure `go build ./...` is clean.
|
||||
7. Run full module + storage suites.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All modules compile and pass tests using `DocStore[T]`.
|
||||
- [ ] No `KVStore`/`VersionedStore`/`PutJSON`/`GetJSON` references outside the migrator.
|
||||
- [ ] DynamoDB runtime backend deleted; `dynamodb_client.go` retained for migrator.
|
||||
- [ ] coin/gold/lolschedule CAS behavior unchanged (tests prove single-winner).
|
||||
- [ ] lolschedule subscribers/last-push persist as named root fields.
|
||||
- [ ] `cmd/server` builds a Mongo-only runtime (memory fallback for local/no-DB).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: large blast radius. Mitigation: strict module-by-module order, suite
|
||||
green between each; no behavior changes, only the store type.
|
||||
- Risk: a module silently relied on byte-identical round-trip. Mitigation: typed
|
||||
structs already define the JSON contract; assert decoded values, not bytes.
|
||||
- Risk: stats List semantics differ. Mitigation: `List(prefix)` ported verbatim;
|
||||
reuse stats' existing key prefixes.
|
||||
- Risk: cron Deps scoping regresses. Mitigation: keep registry's per-module Deps
|
||||
cloning; `Deps.Store` is already per-module.
|
||||
@@ -1,115 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "DynamoDB→Mongo migrator + docs"
|
||||
status: done
|
||||
priority: P2
|
||||
dependencies: [3]
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 4: DynamoDB→Mongo migrator + docs
|
||||
|
||||
## Overview
|
||||
|
||||
Keep `cmd/migrate-dynamo-to-mongo` working, but make it write the new flattened
|
||||
native shape directly. The migrator is schema-agnostic over modules, so it
|
||||
cannot use the typed `DocStore[T]` (it has no compile-time `T` per row). It
|
||||
writes documents with a small generic JSON→root-document encoder plus a tiny
|
||||
explicit table for the only two non-object values.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Migrator still Scans DynamoDB (read-only, `dynamodb:Scan` only) — unchanged.
|
||||
- It writes `{ _id: sk, ...payloadFields, version:1, updatedAt(Date) }` per item.
|
||||
- JSON object values flatten to root.
|
||||
- The two known non-object DynamoDB values are wrapped to match the module's
|
||||
typed shape:
|
||||
- `lolschedule` subscribers (JSON array) → `{ subscribers: [...] }`
|
||||
- `lolschedule` last-push date (JSON/scalar string) → `{ date: "..." }`
|
||||
- Idempotent (re-run overwrites by `_id`, never duplicates).
|
||||
- `--dry-run` (counts only) and `--verify` (per-collection count parity) kept.
|
||||
- No secrets read/printed; `MONGO_URL` stays secret.
|
||||
|
||||
## Architecture
|
||||
|
||||
Add a migrator-local encoder (NOT in `internal/storage`, to keep the runtime
|
||||
typed and the generic JSON path confined to migration):
|
||||
|
||||
```go
|
||||
// cmd/migrate-dynamo-to-mongo/encode.go
|
||||
func rootDocForItem(module, key string, value []byte, now time.Time) (bson.M, error)
|
||||
```
|
||||
|
||||
- Decode `value` with `json.Decoder.UseNumber` (int64 fidelity).
|
||||
- If a `(module,key-prefix)` wrap rule matches → `{ wrapField: decoded }`.
|
||||
- Else if decoded is a JSON object → spread its fields at root (reject keys
|
||||
colliding with `_id`/`version`/`updatedAt`; none expected — fail loud if seen).
|
||||
- Else → error (an unexpected non-object value means a new wrap rule is needed;
|
||||
fail loud rather than guess).
|
||||
- Always set `_id`, `version:int64(1)`, `updatedAt: now`.
|
||||
|
||||
Wrap rules table (explicit, documented):
|
||||
|
||||
```go
|
||||
var wrapRules = []struct{ module, keyPrefix, field string }{
|
||||
{"lolschedule", "subscribers", "subscribers"},
|
||||
{"lolschedule", "last_push_date", "date"},
|
||||
}
|
||||
```
|
||||
|
||||
Write with `Collection(module).ReplaceOne({_id:key}, doc, upsert=true)` directly
|
||||
via the driver (the migrator already holds `*mongo.Database`).
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `cmd/migrate-dynamo-to-mongo/main.go` — replace `provider.For(pk).Put`
|
||||
with `rootDocForItem` + `ReplaceOne` upsert; validate module/key names inline
|
||||
(the old `For` returned an erroring store for bad names — replicate that guard).
|
||||
- Create: `cmd/migrate-dynamo-to-mongo/encode.go` + `encode_test.go`.
|
||||
- Modify: `cmd/migrate-dynamo-to-mongo/main_test.go` — assert flattened shape +
|
||||
wrapped lolschedule docs.
|
||||
- Modify: `cmd/migrate-dynamo-to-mongo/README.md` — new destination shape.
|
||||
- Modify: `README.md` (storage layout summary), `docs/deploy-coolify-selfhosted.md`
|
||||
(Atlas layout + cutover note), `Makefile` if migrator target wording changed.
|
||||
|
||||
## Tests Before
|
||||
|
||||
1. `TestRootDocForItem_Object` — object JSON → root fields + meta; no `value`.
|
||||
2. `TestRootDocForItem_Int64Fidelity` — large integral number stays int64.
|
||||
3. `TestRootDocForItem_LolscheduleSubscribers` — array → `{subscribers:[...]}`.
|
||||
4. `TestRootDocForItem_LolscheduleLastPush` — date string → `{date:"..."}`.
|
||||
5. `TestRootDocForItem_UnknownScalar_Errors` — bare scalar w/o wrap rule fails loud.
|
||||
6. Migrator e2e (`MONGODB_TEST_URL` + DynamoDB Local): seed DynamoDB rows for a
|
||||
few modules incl. lolschedule array+scalar → migrate → assert raw Mongo docs
|
||||
are flattened/wrapped, no `value`; re-run idempotent; `--verify` counts match;
|
||||
values readable by the module's `DocStore[T]`.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Implement `rootDocForItem` + wrap rules.
|
||||
2. Rewrite `runMigrate` to encode + `ReplaceOne` upsert; keep `--dry-run`.
|
||||
3. Keep `runVerify` (count parity) — unchanged logic.
|
||||
4. Add inline module/key validation (reuse `storage.ValidateKey` if exported, or
|
||||
replicate the rule) so bad rows fail loud.
|
||||
5. Make migrator tests pass.
|
||||
6. Update README/deploy docs to the flattened layout; remove BSON-string/binary
|
||||
destination wording.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Migrator writes flattened native docs (no `value`) for all modules.
|
||||
- [ ] lolschedule array + scalar land as `subscribers`/`date` root fields.
|
||||
- [ ] Migrated docs are read back correctly by the typed module stores.
|
||||
- [ ] Idempotent; `--dry-run` writes nothing; `--verify` exits non-zero on mismatch.
|
||||
- [ ] Unknown non-object value fails loud (no silent guess).
|
||||
- [ ] README + Coolify deploy docs show the flattened layout.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: a module other than lolschedule stores a non-object (missed in survey).
|
||||
Mitigation: encoder fails loud on un-wrapped scalars/arrays; e2e covers the
|
||||
known set; the error names module/key so a new wrap rule is a one-liner.
|
||||
- Risk: re-introducing a generic JSON codec invites runtime reuse. Mitigation:
|
||||
it lives only in `cmd/migrate-dynamo-to-mongo`, not `internal/storage`.
|
||||
- Risk: DynamoDB source already gone at run time. Mitigation: migrator is the
|
||||
cutover step; if DynamoDB is empty/absent the operator skips it (documented).
|
||||
@@ -1,89 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Verification + Mongo-only rollout"
|
||||
status: done
|
||||
priority: P2
|
||||
dependencies: [4]
|
||||
effort: "M"
|
||||
---
|
||||
|
||||
# Phase 5: Verification + Mongo-only rollout
|
||||
|
||||
## Overview
|
||||
|
||||
Full validation and a simple cutover. Because Atlas is empty until cutover and
|
||||
there is no legacy Mongo data, rollout is one-way and clean: migrate from
|
||||
DynamoDB, then run Mongo-only. No in-place Mongo migration, no dual-read.
|
||||
|
||||
## Requirements
|
||||
|
||||
- All test gates pass (`make vet`, `make test`, Mongo integration, migrator e2e).
|
||||
- Raw Mongo docs verified flattened for representative modules.
|
||||
- App behavior unchanged for all Telegram commands.
|
||||
- Cutover + rollback notes recorded.
|
||||
|
||||
## Validation gates
|
||||
|
||||
```sh
|
||||
make vet
|
||||
make test
|
||||
MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage ./cmd/migrate-dynamo-to-mongo
|
||||
# migrator e2e additionally needs DynamoDB Local (DYNAMODB_LOCAL_URL)
|
||||
```
|
||||
|
||||
Manual raw-doc inspection (Atlas/Compass or a small test assertion) for:
|
||||
|
||||
- coin/gold/stock portfolio: root fields, `version`, Date `updatedAt`, no `value`.
|
||||
- wordle/loldle/twentyq game + stats: root fields.
|
||||
- lolschedule: `subscribers` array field; last-push `date` field.
|
||||
- stats counters: root fields + List works.
|
||||
|
||||
## Cutover order (operator)
|
||||
|
||||
1. Backup/export current DynamoDB table (source of truth) before cutover.
|
||||
2. Disable EventBridge / stop the old AWS app so DynamoDB writes stop.
|
||||
3. `migrate-dynamo-to-mongo --dry-run` → review per-module counts.
|
||||
4. `migrate-dynamo-to-mongo` → writes flattened native docs to Atlas.
|
||||
5. `migrate-dynamo-to-mongo --verify` → per-collection count parity (exit 0).
|
||||
6. Start the Mongo-only container (`MONGO_URL`+`MONGO_DATABASE`); confirm health.
|
||||
7. Spot-check a few commands (coin balance, wordle state, lolschedule subscribe).
|
||||
|
||||
## Rollback
|
||||
|
||||
- Before step 6 (no Mongo writes yet): re-enable the old AWS app on DynamoDB.
|
||||
DynamoDB is untouched (migrator is Scan-only), so this is safe.
|
||||
- After step 6 (Mongo has taken live writes): forward-fix preferred. To revert to
|
||||
DynamoDB you would need a reverse Mongo→DynamoDB export of anything written
|
||||
after cutover. Keep the DynamoDB backup from step 1 as the floor.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Read: `Makefile`, `README.md`, `docs/deploy-coolify-selfhosted.md`,
|
||||
`docs/aws-decommission-runbook.md`.
|
||||
- Modify: tests only if verification finds gaps.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Run all gates above; fix regressions (do not weaken tests).
|
||||
2. Run the migrator e2e against local Mongo + DynamoDB Local.
|
||||
3. Inspect raw docs per the list; optionally add a small assertion test.
|
||||
4. Record the cutover commands + rollback constraints in the completion report
|
||||
under `plans/reports/`.
|
||||
5. Grep docs for stale `{ _id, value, version, updatedAt }` wording; fix.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `make vet`, `make test` pass.
|
||||
- [ ] Mongo integration + migrator e2e pass.
|
||||
- [ ] Raw docs verified flattened (no `value`/`_payload`) for representative modules.
|
||||
- [ ] No Telegram command behavior change.
|
||||
- [ ] Cutover + rollback documented in the completion report.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- Risk: a module's data didn't round-trip through migration. Mitigation: e2e
|
||||
reads migrated docs back through the typed store; spot-check live commands.
|
||||
- Risk: Atlas M0 limits during migrate. Mitigation: tiny dataset; collection-by-
|
||||
collection counts in `--verify`.
|
||||
- Risk: one-way cutover. Mitigation: DynamoDB backup + Scan-only migrator means
|
||||
pre-write rollback is safe; post-write rollback documented as forward-fix.
|
||||
@@ -1,223 +0,0 @@
|
||||
---
|
||||
title: "MongoDB-native typed stores (replace KVStore)"
|
||||
description: "Delete the generic byte-oriented KVStore abstraction; give every module a typed, Mongo-native versioned store. MongoDB is the only runtime backend (memory kept for tests). The DynamoDB→Mongo migrator stays and writes the new flattened shape directly."
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: "feature/selfhosted"
|
||||
tags: [database, mongodb, refactor, storage, typed-repositories, tdd]
|
||||
blockedBy: [260627-1849-selfhost-coolify-mongodb]
|
||||
blocks: []
|
||||
supersedes: [260628-1113-mongo-native-value-documents, 260628-1310-flatten-mongo-value-documents]
|
||||
created: "2026-06-28T13:18:00.000Z"
|
||||
createdBy: "manual"
|
||||
source: user-direction
|
||||
---
|
||||
|
||||
# MongoDB-native typed stores (replace KVStore)
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the generic, byte-oriented `KVStore` / `VersionedStore` abstraction with
|
||||
a small **typed versioned store** that persists each module's value as a native
|
||||
MongoDB root document:
|
||||
|
||||
```javascript
|
||||
{ _id, ...payloadFields, version, updatedAt: ISODate(...), schemaVersion }
|
||||
```
|
||||
|
||||
There is no `value` envelope, no JSON-bytes round-trip, and no `_payload` /
|
||||
`payloadKind` fallback. Module payload structs map straight to BSON via the
|
||||
driver; the two non-object values (`lolschedule` subscribers array and last-push
|
||||
date) are wrapped in named structs so they too become ordinary root fields.
|
||||
|
||||
This is the **full Mongo-native** direction the earlier flatten plan
|
||||
deliberately deferred. Per user direction (2026-06-28):
|
||||
|
||||
- **Drop `KVStore`.** Modules use typed stores; the generic byte interface is removed.
|
||||
- **MongoDB is the only runtime backend.** The DynamoDB *runtime* store and
|
||||
provider are deleted. The memory store is kept **only** as a test/local double.
|
||||
- **Keep the DynamoDB→Mongo migrator.** It still Scans DynamoDB and now writes
|
||||
the flattened native shape directly. No in-place Mongo schema migrator and no
|
||||
legacy `value` dual-read — Atlas is empty until cutover, so neither is needed.
|
||||
|
||||
## Why this is now correct (and was not before)
|
||||
|
||||
The blocking plan `260627-1849-selfhost-coolify-mongodb` Phase 4 is
|
||||
`code-complete-operator-pending`: the live cutover has **not** run, so MongoDB
|
||||
Atlas holds **no production data**. The prior flatten plan's legacy-`value`
|
||||
dual-read and standalone in-place migrator existed only to convert pre-existing
|
||||
Mongo docs — docs that do not exist. The real and only migration is
|
||||
DynamoDB → Mongo, which writes the final shape in one pass.
|
||||
|
||||
## Design Decision
|
||||
|
||||
Chosen approach: **one generic typed store, two backends.**
|
||||
|
||||
A single generic `DocStore[T]` (Mongo + memory implementations) instead of
|
||||
hand-written per-module repositories. This honors "typed Mongo repos" without
|
||||
duplicating store/CAS/list logic ten times (DRY/KISS).
|
||||
|
||||
```go
|
||||
// internal/storage/doc_store.go
|
||||
type DocStore[T any] interface {
|
||||
Get(ctx context.Context, id string) (val T, version int64, err error) // ErrNotFound
|
||||
Put(ctx context.Context, id string, val T) error // unconditional; bumps version
|
||||
PutVersioned(ctx context.Context, id string, expectedVersion int64, val T) error // CAS; ErrConflict
|
||||
Delete(ctx context.Context, id string) error
|
||||
List(ctx context.Context, prefix string) ([]string, error)
|
||||
}
|
||||
```
|
||||
|
||||
Backend wiring stays backend-agnostic for modules. A provider hands each module
|
||||
an opaque per-module `Collection` handle; a package-level generic constructor
|
||||
turns it into a typed store:
|
||||
|
||||
```go
|
||||
type Provider interface{ Collection(module string) Collection }
|
||||
type Collection interface{ /* opaque: mongo or memory */ }
|
||||
func Typed[T any](c Collection) DocStore[T] // type-switches on the concrete Collection
|
||||
```
|
||||
|
||||
Module factory usage:
|
||||
|
||||
```go
|
||||
portfolios := storage.Typed[Portfolio](deps.Store) // deps.Store = provider.Collection("coin")
|
||||
games := storage.Typed[GameState](deps.Store) // same collection, disjoint key prefixes
|
||||
```
|
||||
|
||||
Go methods cannot be generic, so the typed view is a free function, not a
|
||||
`Provider` method. Multiple types per module share one collection but use
|
||||
disjoint `_id` key prefixes (existing `gameKey`/`statsKey`/… helpers), so reads
|
||||
never decode a doc into the wrong type.
|
||||
|
||||
### Stored document encoding (Mongo)
|
||||
|
||||
```go
|
||||
type storedDoc[T any] struct {
|
||||
ID string `bson:"_id"`
|
||||
Version int64 `bson:"version"`
|
||||
UpdatedAt time.Time `bson:"updatedAt"`
|
||||
Payload T `bson:",inline"` // payload fields hoisted to root
|
||||
}
|
||||
```
|
||||
|
||||
- `bson:",inline"` requires `T` to be a struct (or map). Module payloads already
|
||||
are, except the two `lolschedule` values, which Phase 3 wraps in named structs.
|
||||
- Writes use whole-document `ReplaceOne` (never `$set`), so overwriting a value
|
||||
cannot leave stale fields from a previous value.
|
||||
- `version` CAS: `PutVersioned(expected>0)` filters `{_id, version:expected}`;
|
||||
`MatchedCount==0` → `ErrConflict`. `expected==0` upserts on `{_id, version
|
||||
absent/0}`; duplicate-key → `ErrConflict`. (Same proven semantics as today's
|
||||
`MongoKVStore`, just typed.)
|
||||
- Plain `Put` overwrites unconditionally and bumps `version` via a small bounded
|
||||
get-version → put-versioned retry loop (reuse existing retry-count style).
|
||||
- `updatedAt` is `time.Time` (BSON Date), not int64 nanos.
|
||||
- `schemaVersion` is omitted for now (single shape). Add later only if a real
|
||||
migration need appears — YAGNI.
|
||||
|
||||
### Reserved root field names
|
||||
|
||||
`_id`, `version`, `updatedAt` are reserved. A payload struct must not define BSON
|
||||
tags colliding with these. This is enforced once, at compile/review time, per
|
||||
payload type — not at runtime — because types are known. Phase 1 adds a
|
||||
reserved-name check helper + test; no runtime `_payload` fallback exists.
|
||||
|
||||
## Scope Challenge
|
||||
|
||||
- This is intentionally a **large** refactor (user-selected maximal scope):
|
||||
storage layer rewrite + all ~10 modules + all module tests + server wiring +
|
||||
migrator + docs.
|
||||
- Held back from going further: no `schemaVersion` machinery, no per-module
|
||||
bespoke repositories, no Mongo schema validators/indexes beyond the existing
|
||||
`_id` index, no change to Telegram behavior or command output.
|
||||
|
||||
## Backends after this change
|
||||
|
||||
| Concern | Before | After |
|
||||
|---|---|---|
|
||||
| Runtime store | memory \| dynamodb \| mongodb | **mongodb only** |
|
||||
| Test / local-no-DB store | memory | memory (typed double) |
|
||||
| DynamoDB | runtime backend + migrator | **migrator only** (Scan + write) |
|
||||
| Generic byte `KVStore` | yes | **deleted** |
|
||||
|
||||
## Files removed
|
||||
|
||||
- `internal/storage/kv_store.go` (KVStore, VersionedStore interfaces)
|
||||
- `internal/storage/kv_provider.go` (old KVProvider/MemoryProvider) — replaced
|
||||
- `internal/storage/memory_kv.go`
|
||||
- `internal/storage/mongodb_kv.go`, `mongodb_value_codec.go`
|
||||
- `internal/storage/dynamodb_kv.go`, `dynamodb_provider.go`, `dynamodb_provider_test.go`, `dynamodb_kv_test.go`
|
||||
- `internal/storage/invalid_store.go`
|
||||
- `internal/storage/mongodb_kv_test.go`, `memory_kv_test.go` (rewritten as doc-store tests)
|
||||
|
||||
## Files kept / reused
|
||||
|
||||
- `internal/storage/dynamodb_client.go` (`NewDynamoDBClient`, `DynamoDBEndpointFromEnv`) — migrator only
|
||||
- `internal/storage/mongodb_client.go`, `mongodb_provider.go` (reshaped to the new Provider)
|
||||
- `internal/storage/keys.go`, `prefix.go` (key/prefix validation + range scan reused)
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Typed DocStore contract + memory backend](./phase-01-typed-docstore-and-memory-backend.md) | Done |
|
||||
| 2 | [Mongo typed store + integration tests](./phase-02-mongo-typed-store.md) | Done |
|
||||
| 3 | [Migrate modules + wiring; delete KVStore](./phase-03-migrate-modules-and-wiring.md) | Done |
|
||||
| 4 | [DynamoDB→Mongo migrator + docs](./phase-04-dynamo-migrator-and-docs.md) | Done |
|
||||
| 5 | [Verification + Mongo-only rollout](./phase-05-verification-and-rollout.md) | Done (code; operator runs live cutover) |
|
||||
|
||||
## Implementation Log
|
||||
|
||||
### Session — 2026-06-28 (/cook)
|
||||
All 5 phases implemented. Deleted byte `KVStore`/`VersionedStore` + DynamoDB/memory
|
||||
KV impls; added generic typed `DocStore[T]` (`Provider`/`Collection`/`Typed`) with
|
||||
Mongo (flattened native `storedDoc[T]` via `bson:",inline"`) + memory backends.
|
||||
Migrated all 10 modules + `internal/deploynotify` to typed stores; every persisted
|
||||
struct (incl. nested `lolschedule` `ScheduleEvent` tree and `wordle` `LetterScore`)
|
||||
carries `bson` tags == `json` names. lolschedule wraps its array/scalar values
|
||||
(`subscribers`, `daily_push:last_date`) in named structs. MongoDB is the only
|
||||
runtime backend (memory kept for tests/local); DynamoDB survives only as the
|
||||
migrator source. Migrator rewritten to write the flattened shape via
|
||||
`storage.Typed[bson.M]` + wrap rules (`cmd/migrate-dynamo-to-mongo/encode.go`).
|
||||
|
||||
Verification: `go vet`/`go build` clean; full `go test ./...` green hermetically
|
||||
**and** in-container against real Mongo 7 + DynamoDB Local (storage integration +
|
||||
migrator e2e `TestMigrateAndVerify`). code-reviewer: CHANGES_REQUESTED →
|
||||
addressed (H1 nested bson tags, L1 LetterScore tags, L2 stale comment, M1
|
||||
camelCase-fidelity test); CAS/parity/migrator/security verified correct.
|
||||
|
||||
Operator-pending (out of code scope): the live DynamoDB→Mongo cutover
|
||||
(backup → migrate → verify → start Mongo-only container).
|
||||
|
||||
## Cross-Plan Dependencies
|
||||
|
||||
| Relationship | Plan | Status | Rationale |
|
||||
|---|---|---|---|
|
||||
| Blocked by | `260627-1849-selfhost-coolify-mongodb` | in-progress (cutover pending) | Provides Mongo provider, dynamo migrator, self-host runtime. Cutover-pending state is *why* legacy dual-read is unnecessary. |
|
||||
| Supersedes | `260628-1113-mongo-native-value-documents` | completed | Keeps its version-CAS goal; replaces byte-codec values with typed root docs. |
|
||||
| Supersedes | `260628-1310-flatten-mongo-value-documents` | pending | Same flatten goal; takes the full typed-repo path instead of preserving `KVStore`. |
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `internal/storage` exposes a typed `DocStore[T]` with Mongo + memory impls; no `KVStore`/`VersionedStore` remain.
|
||||
- [ ] All modules persist/read through `DocStore[T]`; no module imports a byte KV API.
|
||||
- [ ] New Mongo docs are `{ _id, ...payloadFields, version, updatedAt(Date), }` with no `value`, no `_payload`.
|
||||
- [ ] `lolschedule` subscribers + last-push are named-struct root fields (`subscribers`, `date`), not bare array/scalar.
|
||||
- [ ] Overwriting a value removes stale prior fields (ReplaceOne).
|
||||
- [ ] Versioned writes remain exactly-one-winner under concurrent create/update (coin/gold/lolschedule).
|
||||
- [ ] MongoDB is the only runtime backend; memory store works for hermetic tests + `MODULES=` local run.
|
||||
- [ ] DynamoDB→Mongo migrator writes the new flattened shape, is idempotent, supports `--dry-run`/`--verify`.
|
||||
- [ ] `make vet`, `make test`, and Mongo integration tests pass; migrator e2e passes.
|
||||
|
||||
## Not In Scope
|
||||
|
||||
- Deleting the DynamoDB→Mongo migrator or its dynamo Scan path.
|
||||
- Mongo schema validators / per-field indexes beyond the existing `_id` index.
|
||||
- `schemaVersion` evolution machinery.
|
||||
- Any change to Telegram command behavior or output.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. (Scope confirmed by user 2026-06-28: delete KVStore → typed repos; drop
|
||||
legacy dual-read + in-place migrator; Mongo-only runtime, memory for tests;
|
||||
keep dynamo migrate.)
|
||||
@@ -1,50 +0,0 @@
|
||||
---
|
||||
status: complete
|
||||
created: 2026-06-29
|
||||
topic: world-cup-schedule-module
|
||||
---
|
||||
|
||||
# World Cup Schedule Module Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Add a `wc` module based on `lolschedule` for World Cup 2026 schedule lookup and
|
||||
daily Telegram digest subscriptions.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Commands: `/wc [date]`, `/wc_today`, `/wc_week`, `/wc_subscribe`,
|
||||
`/wc_unsubscribe`.
|
||||
- Daily push: 08:00 ICT via in-process cron, separate from `lolschedule`.
|
||||
- Provider: football-data.org with `WC_FOOTBALL_DATA_TOKEN`.
|
||||
- Live score: best-effort from provider `status` and `score`; commands fetch
|
||||
live provider data and use cache only when provider fails.
|
||||
- Storage: module-local typed `DocStore` records, flattened Mongo documents.
|
||||
|
||||
## Files
|
||||
|
||||
- Create `internal/modules/wc/*`.
|
||||
- Update `cmd/server/main.go`.
|
||||
- Update `telegram-commands.json`.
|
||||
- Update `README.md`, `.env.example`, and `docs/deploy-coolify-selfhosted.md`.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `go test ./internal/modules/wc` passes.
|
||||
- `go test ./cmd/server ./internal/modules` passes after registration updates.
|
||||
- `go test ./...` passes.
|
||||
- `go vet ./...` passes.
|
||||
- Missing token returns a friendly command error instead of panicking.
|
||||
- Live fetch happens on each command/cron call when upstream is available.
|
||||
- Cached/stale matches work only when upstream fails.
|
||||
- `/wc_subscribe` uses its own subscriber list and preserves forum topic IDs.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- Paid API-Football fallback.
|
||||
- Static no-key fallback.
|
||||
- High-frequency live score polling.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None.
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: VNAppMob API research and key refresh design
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 1h
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: VNAppMob API research and key refresh design
|
||||
|
||||
## Overview
|
||||
|
||||
Confirm the exact request/response contracts for API key refresh and SJC price fetch, decide where the key is persisted, and define the expiry detection strategy.
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Functional**: Document how to refresh the free API key, how to call `GET /api/v2/gold/sjc`, and the JSON response shape.
|
||||
- **Non-functional**: Use only built-in Go packages for JWT claim extraction if possible; avoid new dependencies.
|
||||
|
||||
## Architecture
|
||||
|
||||
1. **Key refresh endpoint**: `POST https://api.vnappmob.com/api/request_api_key?scope=gold`
|
||||
- Returns a raw JWT string.
|
||||
- JWT payload contains `exp` (Unix seconds), `scope`, `permission`.
|
||||
2. **SJC price endpoint**: `GET https://api.vnappmob.com/api/v2/gold/sjc`
|
||||
- Header: `Authorization: Bearer <jwt>`
|
||||
- Response: `{"results":[{"buy_1l":<float>,"sell_1l":<float>, ...}]}`
|
||||
- Use the first element's `buy_1l`/`sell_1l` as the spot price.
|
||||
3. **Key storage**:
|
||||
- KV key: `vnappmob:api_key`
|
||||
- Value JSON: `{"token":"<jwt>","exp":<unix>}`
|
||||
- Stored under the gold module's prefixed KV (already `gold:`).
|
||||
4. **Expiry detection**:
|
||||
- Parse JWT payload (middle segment), base64-url decode, JSON-decode `exp`.
|
||||
- Refresh when `exp - now < refreshBuffer` (buffer e.g. 1h or 1 day).
|
||||
- If parsing fails, force refresh.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Read: `internal/modules/gold/prices.go`
|
||||
- Read: `internal/modules/gold/price_providers.go`
|
||||
- Read: `internal/modules/gold/price_urls.go`
|
||||
- Read: `internal/storage/kv_store.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Re-fetch `https://api.vnappmob.com/api/request_api_key?scope=gold` to confirm response format and inspect a fresh JWT via `jwt.io` or a small Go snippet.
|
||||
2. Decode JWT payload and verify fields (`exp`, `iat`, `scope`, `permission`).
|
||||
3. Document the response sample in this phase file.
|
||||
4. Choose buffer: 24h (refresh one day before expiry, avoiding midnight edge cases).
|
||||
5. Decide concurrency strategy: CAS lock key `vnappmob:refresh_lock` or accept last-write-wins with 1-min TTL.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] API refresh endpoint verified to return a JWT string.
|
||||
- [x] JWT payload fields and expiry semantics documented.
|
||||
- [x] SJC endpoint response shape documented with sample values.
|
||||
- [x] KV key names and refresh buffer decided.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk**: Refresh endpoint returns non-JWT or changes shape. **Mitigation**: treat non-JWT as an error and fall back to existing providers.
|
||||
- **Risk**: Clock skew causes premature expiry. **Mitigation**: 24h buffer absorbs skew; refresh on any 403 even if expiry appears valid.
|
||||
@@ -1,76 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: Implement SJC price client with API key management
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 3h
|
||||
dependencies:
|
||||
- 1
|
||||
---
|
||||
|
||||
# Phase 2: Implement SJC price client with API key management
|
||||
|
||||
## Overview
|
||||
|
||||
Create `internal/modules/gold/vnappmob_client.go` with a client that refreshes, caches, and uses the VNAppMob API key, and exposes a method that returns a VND/lượng price.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Self-contained client: refresh key on demand, cache in KV, parse expiry from JWT payload.
|
||||
- Env overrides for base URL and token.
|
||||
- 403 from SJC endpoint triggers one refresh retry.
|
||||
- No new external dependencies beyond standard library.
|
||||
|
||||
## Architecture
|
||||
|
||||
```go
|
||||
type VNAppMobClient struct {
|
||||
HTTP *http.Client
|
||||
BaseURL string // default https://api.vnappmob.com
|
||||
Token string // optional env override GOLD_VNAPP_API_KEY
|
||||
KV storage.KVStore // module KV
|
||||
nowFn func() time.Time
|
||||
}
|
||||
|
||||
func NewVNAppMobClientFromEnv(kv storage.KVStore) *VNAppMobClient
|
||||
|
||||
func (c *VNAppMobClient) FetchSJCPrice(ctx context.Context) (buy, sell float64, err error)
|
||||
```
|
||||
|
||||
- Key is stored under `"vnappmob:api_key"`.
|
||||
- Helper `getKey(ctx)` returns the current valid key, refreshing if needed.
|
||||
- Helper `refreshKey(ctx)` calls `POST {BaseURL}/api/request_api_key?scope=gold`, validates JWT, stores JSON.
|
||||
- Helper `jwtExp(token)` extracts middle segment, base64-url decodes, JSON-parses `exp`.
|
||||
- Helper `isExpired(token)` returns true if `exp - now < 24h` or parse fails.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/gold/vnappmob_client.go`
|
||||
- Reference: `internal/modules/gold/price_urls.go`
|
||||
- Reference: `internal/modules/stock/income_events.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Add constants:
|
||||
```go
|
||||
vnappmobDefaultURL = "https://api.vnappmob.com"
|
||||
vnappmobKeyCacheKey = "vnappmob:api_key"
|
||||
vnappmobRefreshBuffer = 24 * time.Hour
|
||||
vnappmobHTTPTimeout = 10 * time.Second
|
||||
```
|
||||
2. Implement `NewVNAppMobClientFromEnv(kv)` reading `GOLD_VNAPP_API_URL` and `GOLD_VNAPP_API_KEY`.
|
||||
3. Implement `getKey`/`refreshKey`/`jwtExp`/`isExpired`.
|
||||
4. Implement `FetchSJCPrice`: build request, attach Bearer token, decode `{"results":[...]}`, validate first result, return buy/sell.
|
||||
5. On 403, call `refreshKey` once and retry.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] `VNAppMobClient` compiles and passes go vet.
|
||||
- [x] Unit tests for JWT expiry parsing cover valid, malformed, and missing `exp`.
|
||||
- [x] Mock server test verifies 403 triggers refresh and retry.
|
||||
- [x] `FetchSJCPrice` returns `ErrNoGoldPrice` when response has no results or invalid values.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk**: KV not available in local `memory` provider. **Mitigation**: KVStore is always passed; memory provider implements the interface.
|
||||
- **Risk**: Race between concurrent refreshes. **Mitigation**: simple last-write-wins is acceptable; key validity is the same for all callers.
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Wire client into gold module and handlers"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "2h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 3: Wire client into gold module and handlers
|
||||
|
||||
## Overview
|
||||
|
||||
Integrate the VNAppMob SJC client as the primary gold price provider while keeping the existing XAU/USD chain as fallback. Update the `priceFetcher` interface and handlers so `/gold_price` shows SJC data and trading commands use SJC-derived prices.
|
||||
|
||||
## Requirements
|
||||
|
||||
- VNAppMob SJC is the first price source.
|
||||
- On VNAppMob failure, fall back to existing `GoldPriceClient.FetchLuongPrice`.
|
||||
- `/gold_price` output shows SJC buy/sell in VND/lượng.
|
||||
- Portfolio commands (`gold_buy`, `gold_sell`, `gold_stats`) use a representative price (e.g. mid of buy/sell or sell price).
|
||||
|
||||
## Architecture
|
||||
|
||||
Introduce a composite fetcher in `internal/modules/gold/prices.go`:
|
||||
|
||||
```go
|
||||
type compositePriceFetcher struct {
|
||||
vnappmob *VNAppMobClient
|
||||
fallback *GoldPriceClient
|
||||
}
|
||||
|
||||
func (f *compositePriceFetcher) FetchLuongPrice(ctx context.Context) (float64, error) {
|
||||
buy, sell, err := f.vnappmob.FetchSJCPrice(ctx)
|
||||
if err == nil {
|
||||
return (buy + sell) / 2, nil
|
||||
}
|
||||
log.Warn("vnappmob_sjc_failed", "err", err)
|
||||
return f.fallback.FetchLuongPrice(ctx)
|
||||
}
|
||||
|
||||
func (f *compositePriceFetcher) FetchPrice(ctx context.Context) (GoldPrice, error) {
|
||||
buy, sell, err := f.vnappmob.FetchSJCPrice(ctx)
|
||||
if err == nil {
|
||||
mid := (buy + sell) / 2
|
||||
return GoldPrice{XAUUSD: 0, USDVND: 0, VNDPerLuong: mid}, nil
|
||||
}
|
||||
return f.fallback.FetchPrice(ctx)
|
||||
}
|
||||
```
|
||||
|
||||
Update `newState(kv)` to build the composite fetcher. `helpers.go` already defines the `priceFetcher` interface.
|
||||
|
||||
Update `handlePrice` in `handlers.go`:
|
||||
- If `FetchPrice` returned from SJC (detect via new flag or by checking `XAUUSD == 0`), show SJC-specific output.
|
||||
- Otherwise keep existing spot-price output for fallback.
|
||||
|
||||
Add a field to `GoldPrice` to indicate the source, e.g.:
|
||||
```go
|
||||
type GoldPrice struct {
|
||||
XAUUSD float64
|
||||
USDVND float64
|
||||
VNDPerLuong float64
|
||||
Source string // "vnappmob-sjc" or "xau-fallback"
|
||||
SJC *SJCPrice // optional
|
||||
}
|
||||
|
||||
type SJCPrice struct {
|
||||
Buy float64
|
||||
Sell float64
|
||||
}
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `internal/modules/gold/prices.go`
|
||||
- Modify: `internal/modules/gold/helpers.go`
|
||||
- Modify: `internal/modules/gold/handlers.go`
|
||||
- Reference: `internal/modules/gold/gold.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Extend `GoldPrice` struct with `Source string` and `SJC *SJCPrice`.
|
||||
2. Implement `compositePriceFetcher` in `prices.go` (or new `composite_prices.go`).
|
||||
3. Change `newState(kv)` to use composite fetcher.
|
||||
4. Update `handlePrice` to render SJC-specific lines when `Source == "vnappmob-sjc"`.
|
||||
5. Ensure `handleBuy`, `handleSell`, `handleStats` continue to work via `FetchLuongPrice`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] `/gold_price` shows SJC buy/sell when VNAppMob succeeds.
|
||||
- [x] `/gold_price` falls back to old output when VNAppMob fails.
|
||||
- [x] `gold_buy` and `gold_sell` use SJC mid price for cost/revenue.
|
||||
- [x] `go vet` passes.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk**: SJC response has only one of `buy_1l`/`sell_1l`. **Mitigation**: if one is missing, use the other; if both missing, return error so fallback kicks in.
|
||||
- **Risk**: Existing tests assume `GoldPrice` shape. **Mitigation**: add fields without removing old ones.
|
||||
@@ -1,55 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Update IaC and env handling"
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: "1h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 4: Update IaC and env handling
|
||||
|
||||
## Overview
|
||||
|
||||
Add optional env vars for VNAppMob configuration, export them in `cmd/server/main.go`, and expose CloudFormation parameters in `template.yaml`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Allow manual API key override (`GOLD_VNAPP_API_KEY`).
|
||||
- Allow base URL override (`GOLD_VNAPP_API_URL`) for testing.
|
||||
- Support SSM Parameter Store injection via `GOLD_VNAPP_API_KEY_PARAMETER_NAME`.
|
||||
|
||||
## Architecture
|
||||
|
||||
In `cmd/server/main.go`:
|
||||
- Add fields to `config`: `GoldVNAppAPIURL`, `GoldVNAppAPIKey`, `GoldVNAppAPIKeyParam`.
|
||||
- Read env vars `GOLD_VNAPP_API_URL`, `GOLD_VNAPP_API_KEY`, `GOLD_VNAPP_API_KEY_PARAMETER_NAME`.
|
||||
- Add binding to `resolveSSMSecrets`.
|
||||
- Export via `exportOptionalEnv("GOLD_VNAPP_API_URL", ...)` and `GOLD_VNAPP_API_KEY`.
|
||||
|
||||
In `template.yaml`:
|
||||
- Add parameters `GoldVNAppAPIURL`, `GoldVNAppAPIKeyParameterName`.
|
||||
- Add env vars under `BotFunction.Environment.Variables`.
|
||||
- IAM policy already allows SSM fetch for `/miti99bot/${StackEnv}/*`, so no new policy needed.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `cmd/server/main.go`
|
||||
- Modify: `template.yaml`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Extend `config` struct and `loadConfig`.
|
||||
2. Add SSM binding and optional env export.
|
||||
3. Add CFN parameters and pass to Lambda env.
|
||||
4. Verify SSM path pattern matches existing IAM wildcard.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] `GOLD_VNAPP_API_KEY` env var reaches the gold module.
|
||||
- [x] `GOLD_VNAPP_API_KEY_PARAMETER_NAME` is fetched from SSM at startup.
|
||||
- [x] `template.yaml` deploys without syntax errors (`sam validate`).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk**: Forgetting to export env var means `os.Getenv` in module sees nothing. **Mitigation**: mirror existing `exportOptionalEnv` calls exactly.
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Tests and verification"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "2h"
|
||||
dependencies: [3, 4]
|
||||
---
|
||||
|
||||
# Phase 5: Tests and verification
|
||||
|
||||
## Overview
|
||||
|
||||
Add unit tests for the new client and integration smoke tests for the composite fetcher, then run the full suite.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Test VNAppMob key refresh, storage, expiry parsing, 403 retry, and SJC price parsing.
|
||||
- Test composite fetcher fallback behavior.
|
||||
- Ensure existing gold module tests still pass.
|
||||
|
||||
## Architecture
|
||||
|
||||
Create `internal/modules/gold/vnappmob_client_test.go`:
|
||||
- `TestJWTExp` — valid, expired, malformed tokens.
|
||||
- `TestRefreshKey` — mock refresh endpoint, verify KV storage.
|
||||
- `TestFetchSJCPrice` — mock SJC endpoint, verify buy/sell.
|
||||
- `TestFetchSJCPrice_403Refreshes` — first 403, refresh, second 200.
|
||||
- `TestFetchSJCPrice_FallbackError` — on total failure, return `ErrNoGoldPrice`.
|
||||
|
||||
Create/update `internal/modules/gold/prices_test.go` or `composite_prices_test.go`:
|
||||
- `TestCompositeFetcher_PrefersVNAppMob`
|
||||
- `TestCompositeFetcher_FallsBack`
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `internal/modules/gold/vnappmob_client_test.go`
|
||||
- Modify: existing test files in `internal/modules/gold/`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Write `vnappmob_client_test.go` using `httptest.Server`.
|
||||
2. Add composite fetcher tests with stub `priceFetcher` implementations.
|
||||
3. Run `make vet` and `make test`.
|
||||
4. Run local server with `MODULES=gold` and hit `/gold_price` via Telegram or curl if possible.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] All new tests pass.
|
||||
- [x] `make test` passes.
|
||||
- [x] `make vet` passes.
|
||||
- [ ] Manual local smoke test returns SJC price.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Risk**: External API tests are flaky. **Mitigation**: use `httptest` for all unit tests; external calls only in manual smoke test.
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
title: Integrate VNAppMob SJC gold price with auto-refresh API key
|
||||
description: >-
|
||||
Add a VNAppMob SJC price provider to the gold module. The provider
|
||||
self-manages a free JWT API key by refreshing it when missing or expired,
|
||||
stores it in KV, and uses it to call the SJC endpoint.
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: main
|
||||
tags:
|
||||
- gold
|
||||
- vnappmob
|
||||
- sjc
|
||||
- api-key
|
||||
- kv
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: '2026-06-15T02:15:10.444Z'
|
||||
createdBy: 'ck:plan'
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Integrate VNAppMob SJC gold price with auto-refresh API key
|
||||
|
||||
## Overview
|
||||
|
||||
Replace/add the gold spot-price source with VNAppMob's Vietnam SJC price feed (`api.vnappmob.com/api/v2/gold/sjc`). The feed returns VND/lượng directly, removing the XAU/USD + FX conversion step. It requires a free `api_key` JWT that expires in ~14 days, so the implementation must refresh and persist the key automatically.
|
||||
|
||||
The existing `GoldPriceClient` provider chain is extended: VNAppMob SJC becomes the new primary provider; the old XAU/USD chain becomes the fallback. A new `VNAppMobClient` handles key refresh via `POST /api/request_api_key?scope=gold`, stores the key under KV (`vnappmob:api_key`), and uses it as `Authorization: Bearer <jwt>` for `GET /api/v2/gold/sjc`.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [VNAppMob API research and key refresh design](./phase-01-vnappmob-api-research-and-key-refresh-design.md) | Completed |
|
||||
| 2 | [Implement SJC price client with API key management](./phase-02-implement-sjc-price-client-with-api-key-management.md) | Completed |
|
||||
| 3 | [Wire client into gold module and handlers](./phase-03-wire-client-into-gold-module-and-handlers.md) | Completed |
|
||||
| 4 | [Update IaC and env handling](./phase-04-update-iac-and-env-handling.md) | Completed |
|
||||
| 5 | [Tests and verification](./phase-05-tests-and-verification.md) | Completed |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- No blocking plans. This touches only `internal/modules/gold`, `cmd/server/main.go`, and `template.yaml`.
|
||||
|
||||
## Risks
|
||||
|
||||
- VNAppMob key refresh endpoint may change or rate-limit. Mitigation: env override + fallback to existing XAU/USD chain.
|
||||
- JWT parsing for expiry must not require a JWT library if possible (base64 + JSON). Mitigation: implement minimal JWT claim extraction; treat parse failure as "refresh needed".
|
||||
- Concurrent Lambda containers may race to refresh the key. Mitigation: CAS-based single-flight or last-write-wins with short TTL.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- `/gold_price` returns SJC buy/sell prices in VND/lượng when VNAppMob is healthy.
|
||||
- Missing/expired key triggers a refresh transparently without user error.
|
||||
- If VNAppMob fails, the bot falls back to the existing XAU/USD-derived price.
|
||||
- Env var `GOLD_VNAPP_API_KEY` can bypass auto-fetch for local dev or SSM injection.
|
||||
@@ -1,431 +0,0 @@
|
||||
# Research Report: Coin Module For USD Topup And Crypto Paper Trading
|
||||
|
||||
---
|
||||
type: report
|
||||
topic: coin-module
|
||||
created_at: 2026-06-12 09:48 UTC
|
||||
status: complete
|
||||
---
|
||||
|
||||
## Table Of Contents
|
||||
|
||||
- [Executive Summary](#executive-summary)
|
||||
- [Research Methodology](#research-methodology)
|
||||
- [Key Findings](#key-findings)
|
||||
- [Comparative Analysis](#comparative-analysis)
|
||||
- [Implementation Recommendations](#implementation-recommendations)
|
||||
- [Resources And References](#resources-and-references)
|
||||
- [Next Steps](#next-steps)
|
||||
- [Unresolved Questions](#unresolved-questions)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Add a separate `coin` module, not an extension of `trading`. Existing `trading` is VN-stock/VND oriented. Existing `gold` is the better structural reference for fractional assets, standalone KV namespace, price client, topup/buy/sell/stats commands, and optional env URL overrides.
|
||||
|
||||
Recommended free price strategy for MVP: provider chain with Binance, Coinbase, and CoinGecko. Try Binance first for exchange-listed USD/USDT pairs, then Coinbase public exchange rates, then CoinGecko simple price. If one provider fails, rate limits, returns no price, or does not support the coin, fall back to the next provider without mutating portfolio state.
|
||||
|
||||
Keep scope tight: paper trading only, USD cash balance only, market-price buy/sell only, whitelist common coins first. No deposits, withdrawals, live orders, wallets, tax, charts, limit orders, leverage, or on-chain tokens.
|
||||
|
||||
## Research Methodology
|
||||
|
||||
- Sources consulted: 4 official docs plus local repo source.
|
||||
- Date range: live docs checked on 2026-06-12.
|
||||
- Key search terms used: `CoinGecko simple price free API`, `Binance ticker price endpoint`, `Coinbase exchange rates unauthenticated`, `crypto price API rate limit`.
|
||||
- Local references checked:
|
||||
- `README.md`
|
||||
- `internal/modules/trading/*`
|
||||
- `internal/modules/gold/*`
|
||||
- `internal/modules/registry.go`
|
||||
- `template.yaml`
|
||||
|
||||
## Key Findings
|
||||
|
||||
### 1. Repo Architecture
|
||||
|
||||
The bot loads modules by name from `MODULES`. Each module gets module-scoped KV through `kv.For(name)`, so a new `coin` module can own independent per-user state with key `user:{telegram_id}`.
|
||||
|
||||
Existing trading flow:
|
||||
|
||||
```text
|
||||
Telegram command
|
||||
-> parse sender and args
|
||||
-> fetch/resolve price before lock
|
||||
-> acquire per-user lock or CAS update
|
||||
-> load portfolio
|
||||
-> mutate cash/assets
|
||||
-> save portfolio
|
||||
-> reply
|
||||
```
|
||||
|
||||
Use same flow. Network calls should stay outside mutation critical path.
|
||||
|
||||
### 2. Current Trade References
|
||||
|
||||
`trading` strengths:
|
||||
|
||||
- command registration style is clear.
|
||||
- `senderInfo` and `argsAfterCommand` are reusable patterns.
|
||||
- price client uses `http.Client` timeout and test injection.
|
||||
- portfolio methods keep mutation readable.
|
||||
|
||||
`gold` strengths:
|
||||
|
||||
- standalone module for one asset class.
|
||||
- fractional quantity model.
|
||||
- normalize invalid floats.
|
||||
- CAS `UpdatePortfolio` avoids lost updates when storage supports it.
|
||||
- env URL overrides for price APIs.
|
||||
|
||||
For `coin`, use `gold` as the closer reference, with `trading` command names and asset map style.
|
||||
|
||||
### 3. API Recommendation
|
||||
|
||||
Recommended primary architecture: three-provider failover.
|
||||
|
||||
```text
|
||||
Fetch coin USD price
|
||||
-> Binance ticker price: SYMBOLUSDT, then SYMBOLUSD when supported
|
||||
-> Coinbase exchange rates: currency=SYMBOL, read rates.USD
|
||||
-> CoinGecko simple price: local symbol -> CoinGecko coin ID, read usd
|
||||
-> return ErrNoCoinPrice if all fail
|
||||
```
|
||||
|
||||
Provider order:
|
||||
|
||||
1. Binance: best first source for highly traded pairs, simple ticker endpoint, live exchange quote.
|
||||
2. Coinbase: broad public exchange-rate endpoint, no auth, direct USD quote.
|
||||
3. CoinGecko: broadest metadata-backed fallback, useful when exchanges miss a symbol.
|
||||
|
||||
Binance API:
|
||||
|
||||
```http
|
||||
GET https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT
|
||||
```
|
||||
|
||||
Expected shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"symbol": "BTCUSDT",
|
||||
"price": "67321.42"
|
||||
}
|
||||
```
|
||||
|
||||
Coinbase API:
|
||||
|
||||
```http
|
||||
GET https://api.coinbase.com/v2/exchange-rates?currency=BTC
|
||||
```
|
||||
|
||||
Expected shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"currency": "BTC",
|
||||
"rates": {
|
||||
"USD": "67321.42"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
CoinGecko API:
|
||||
|
||||
```http
|
||||
GET https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_last_updated_at=true
|
||||
```
|
||||
|
||||
Expected shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"bitcoin": {
|
||||
"usd": 67321.42,
|
||||
"last_updated_at": 1711356300
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Pros:
|
||||
|
||||
- resilient to provider-specific outages.
|
||||
- Binance gives exchange-like current pair quote for major coins.
|
||||
- Coinbase gives direct USD rates without API key.
|
||||
- CoinGecko gives broad ID-based coverage and optional freshness metadata.
|
||||
- no secret key required for MVP if using public/demo-free paths.
|
||||
|
||||
Cons:
|
||||
|
||||
- more code than one provider.
|
||||
- Binance is pair-based; some coins will not have `USD`/`USDT` pairs.
|
||||
- Coinbase quote availability depends on Coinbase supported currencies.
|
||||
- CoinGecko public/demo rate limits vary by traffic; avoid chatty stats calls.
|
||||
- provider prices can differ. Reply should include source used.
|
||||
|
||||
### 4. Provider Notes
|
||||
|
||||
Binance:
|
||||
|
||||
- `/api/v3/ticker/price?symbol=BTCUSDT` returns latest exchange-pair price.
|
||||
- Very simple and high-quality for listed USDT pairs.
|
||||
- Good first provider for `BTCUSDT`, `ETHUSDT`, etc.
|
||||
- Requires strict 429 backoff because repeat abuse can lead to temporary IP ban.
|
||||
|
||||
Coinbase:
|
||||
|
||||
- `/v2/exchange-rates?currency=BTC` returns rates for one base currency.
|
||||
- No authentication required.
|
||||
- Good second provider because it provides direct USD quote and simple JSON.
|
||||
- Rate limits are less explicit in the opened exchange-rate page, so still cache.
|
||||
|
||||
CoinGecko:
|
||||
|
||||
- `/simple/price` supports coin IDs/symbols/names, `vs_currencies`, market cap, volume, 24h change, and `last_updated_at`.
|
||||
- Docs warn public/demo usage is around 30 calls/minute and varies by traffic.
|
||||
- Use as third provider because it has broad coverage and stable coin IDs.
|
||||
- Maintain local symbol-to-ID mapping to avoid ambiguous symbol lookup.
|
||||
|
||||
CoinCap:
|
||||
|
||||
- Current docs redirect to pro API docs.
|
||||
- Not recommended for no-key MVP unless verified in implementation.
|
||||
|
||||
### 5. Security Considerations
|
||||
|
||||
- This is paper trading. User balances are self-declared topups, not real money.
|
||||
- Do not integrate wallets, private keys, exchange accounts, deposits, withdrawals, or API trade credentials.
|
||||
- Whitelist supported coins to avoid phishing-style fake tickers and ambiguous symbols.
|
||||
- Validate finite positive amounts and quantities. Reject NaN, Inf, zero, negative.
|
||||
- Use Telegram user ID for state, same as trading/gold.
|
||||
- Rate-limit failure must not mutate portfolio.
|
||||
- Never log full Telegram message text if it could include user-entered values beyond command diagnostics.
|
||||
|
||||
### 6. Performance And Reliability
|
||||
|
||||
- Use 10s HTTP timeout, same as existing price clients.
|
||||
- Cache prices in memory for short TTL, recommended 15-30 seconds, to reduce API calls during `/coin_stats`.
|
||||
- Fetch price before portfolio update.
|
||||
- For stats, use cache first. If cache misses, CoinGecko can batch via `ids=bitcoin,ethereum&vs_currencies=usd`; Binance can fetch multiple symbols through the `symbols` parameter, but only for listed pairs. Keep MVP simple: per-symbol provider chain with 15-30s cache and cap displayed holdings if needed.
|
||||
- Surface `429` as "price API rate limited, try later"; do not retry in tight loop.
|
||||
|
||||
## Comparative Analysis
|
||||
|
||||
| Provider | Auth | Best For | Weakness | MVP Role |
|
||||
|---|---:|---|---|---|
|
||||
| Binance ticker price | No for market data | listed exchange pairs | USDT/USD-pair only, IP ban on abuse | First |
|
||||
| Coinbase exchange rates | No | direct crypto-to-USD quote | unclear explicit rate quota on page | Second |
|
||||
| CoinGecko simple price | Demo/pro key preferred; public limits vary | broad coin IDs, batch, freshness | rate limits, key/root URL confusion | Third |
|
||||
| CoinCap | unclear/current pro docs | asset market data | free no-key path unclear | Avoid now |
|
||||
|
||||
## Implementation Recommendations
|
||||
|
||||
### Module Shape
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/modules/coin/
|
||||
coin.go
|
||||
handlers.go
|
||||
portfolio.go
|
||||
prices.go
|
||||
format.go
|
||||
symbols.go
|
||||
*_test.go
|
||||
```
|
||||
|
||||
Register in composition root where current module factories live. Add `coin` to `template.yaml` `ModulesCSV` only if this module should be enabled by default.
|
||||
|
||||
### Commands
|
||||
|
||||
Use public commands:
|
||||
|
||||
```text
|
||||
/coin_price <COIN>
|
||||
/coin_topup <usd_amount>
|
||||
/coin_buy <usd_amount> <COIN>
|
||||
/coin_sell <qty> <COIN>
|
||||
/coin_stats
|
||||
```
|
||||
|
||||
Reasoning:
|
||||
|
||||
- Buy by USD amount is easier for users than fractional coin quantity.
|
||||
- Sell by quantity is explicit and avoids accidental "sell all" behavior.
|
||||
- Add `/coin_sell_usd <usd_amount> <COIN>` later only if needed.
|
||||
|
||||
### Supported Coins
|
||||
|
||||
Start with local whitelist:
|
||||
|
||||
```go
|
||||
var supportedCoins = map[string]string{
|
||||
"BTC": "BTC",
|
||||
"ETH": "ETH",
|
||||
"SOL": "SOL",
|
||||
"BNB": "BNB",
|
||||
"XRP": "XRP",
|
||||
"ADA": "ADA",
|
||||
"DOGE": "DOGE",
|
||||
"TON": "TON",
|
||||
}
|
||||
```
|
||||
|
||||
Keep symbols uppercase. Do not accept arbitrary names at first.
|
||||
|
||||
### Portfolio Model
|
||||
|
||||
Use USD cash plus fractional holdings:
|
||||
|
||||
```go
|
||||
type Portfolio struct {
|
||||
USD float64 `json:"usd"`
|
||||
Assets map[string]float64 `json:"assets"`
|
||||
Meta PortfolioMeta `json:"meta"`
|
||||
}
|
||||
|
||||
type PortfolioMeta struct {
|
||||
Invested float64 `json:"invested"`
|
||||
CreatedAt int64 `json:"createdAt"`
|
||||
}
|
||||
```
|
||||
|
||||
This mirrors `gold` more than `trading`, because coins are fractional.
|
||||
|
||||
### Price Client
|
||||
|
||||
MVP client should be a small provider chain, not one hardcoded API:
|
||||
|
||||
```go
|
||||
type PriceClient struct {
|
||||
HTTP *http.Client
|
||||
Providers []PriceProvider
|
||||
CacheTTL time.Duration
|
||||
}
|
||||
|
||||
type PriceProvider interface {
|
||||
FetchUSD(ctx context.Context, symbol string) (CoinPrice, error)
|
||||
}
|
||||
|
||||
type CoinPrice struct {
|
||||
Symbol string
|
||||
USD float64
|
||||
Source string
|
||||
}
|
||||
```
|
||||
|
||||
Default provider URLs:
|
||||
|
||||
```text
|
||||
Binance: https://api.binance.com/api/v3/ticker/price
|
||||
Coinbase: https://api.coinbase.com/v2/exchange-rates
|
||||
CoinGecko: https://api.coingecko.com/api/v3/simple/price
|
||||
```
|
||||
|
||||
Env overrides:
|
||||
|
||||
```text
|
||||
COIN_BINANCE_API_URL
|
||||
COIN_COINBASE_API_URL
|
||||
COIN_COINGECKO_API_URL
|
||||
```
|
||||
|
||||
Failover rules:
|
||||
|
||||
```text
|
||||
for provider in providers:
|
||||
price, err := provider.FetchUSD(symbol)
|
||||
if err == nil && price.USD > 0:
|
||||
return price
|
||||
if err is rate-limit/network/no-price:
|
||||
continue
|
||||
return ErrNoCoinPrice
|
||||
```
|
||||
|
||||
Do not fallback after invalid user input or unsupported local symbol; fail before provider calls.
|
||||
|
||||
### Mutation Rules
|
||||
|
||||
- Topup: add USD, increment `Meta.Invested`.
|
||||
- Buy: deduct USD amount, add `usd_amount / price` units.
|
||||
- Sell: deduct units, add `qty * price` USD.
|
||||
- Stats: show USD balance, each coin holding, market value, total value, simple P/L vs `Meta.Invested`.
|
||||
|
||||
### Validation
|
||||
|
||||
Reject:
|
||||
|
||||
- unsupported coin.
|
||||
- amount <= 0.
|
||||
- qty <= 0.
|
||||
- NaN/Inf.
|
||||
- price <= 0.
|
||||
- insufficient USD or coin balance.
|
||||
|
||||
Normalize dust below `1e-9` to zero.
|
||||
|
||||
### Tests
|
||||
|
||||
Minimum test set:
|
||||
|
||||
- portfolio load new user.
|
||||
- topup increments USD and invested.
|
||||
- buy deducts USD and credits fractional coin.
|
||||
- sell deducts coin and credits USD.
|
||||
- insufficient USD.
|
||||
- insufficient coin.
|
||||
- unsupported coin.
|
||||
- Binance price decode success.
|
||||
- Coinbase price decode success.
|
||||
- CoinGecko price decode success.
|
||||
- provider chain falls back from Binance to Coinbase.
|
||||
- provider chain falls back from Coinbase to CoinGecko.
|
||||
- provider chain returns no price after all providers fail.
|
||||
- price no USD rate.
|
||||
- API 429 / non-2xx error.
|
||||
- stats with cached/fake price client.
|
||||
|
||||
## Quick Start Guide
|
||||
|
||||
1. Copy structure from `internal/modules/gold`.
|
||||
2. Replace VND/Luong with USD/assets map.
|
||||
3. Implement provider-chain price client with injected HTTP client and URL overrides.
|
||||
4. Register commands in `coin.go`.
|
||||
5. Register factory in server composition root.
|
||||
6. Add module to `MODULES` for local testing.
|
||||
7. Run `go test ./internal/modules/coin ./internal/modules/...`.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- Do not store real exchange credentials. Out of scope.
|
||||
- Do not use arbitrary ticker lookup. Whitelist first.
|
||||
- Do not mutate portfolio before price fetch succeeds.
|
||||
- Do not assume all symbols have Binance pairs, Coinbase USD rates, or CoinGecko IDs.
|
||||
- Do not hide price source; include source in `/coin_price`, buy, sell, and stats replies.
|
||||
- Do not fan out unlimited price calls in stats. Add cache or cap holdings.
|
||||
- Do not use `int64` for coin holdings; crypto needs fractional units.
|
||||
|
||||
## Resources And References
|
||||
|
||||
- Coinbase Exchange Rates API: https://docs.cdp.coinbase.com/coinbase-app/track-apis/exchange-rates
|
||||
- CoinGecko Simple Price API: https://docs.coingecko.com/reference/simple-price
|
||||
- CoinGecko common errors and rate limits: https://docs.coingecko.com/docs/common-errors-rate-limit
|
||||
- Binance symbol price ticker: https://developers.binance.com/docs/binance-spot-api-docs/rest-api/market-data-endpoints
|
||||
- Binance limits: https://developers.binance.com/docs/binance-spot-api-docs/rest-api/limits
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Implement `internal/modules/coin` using `gold` as base pattern.
|
||||
2. Use Binance -> Coinbase -> CoinGecko price provider chain with per-provider URL overrides.
|
||||
3. Add short TTL cache in price client.
|
||||
4. Add unit tests for portfolio, handlers, and price decode.
|
||||
5. Update `README.md` module table after implementation.
|
||||
6. Decide whether `coin` is enabled by default in `template.yaml`.
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
- Enable `coin` by default in deployed `ModulesCSV`, or keep opt-in like `gold`?
|
||||
- Should `/coin_buy` accept USD amount only, or also support quantity mode?
|
||||
- Which initial whitelist: top 8 above enough, or include more from day one?
|
||||
- Should provider order be configurable by env, or fixed as Binance -> Coinbase -> CoinGecko?
|
||||
@@ -1,292 +0,0 @@
|
||||
---
|
||||
type: research-report
|
||||
topic: stable-vn-stock-price-provider
|
||||
created: 2026-06-25 09:36 UTC
|
||||
status: done
|
||||
---
|
||||
|
||||
# Research Report: Stable Free VN Stock Price Provider
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Best fit: **EODHD free plan as primary EOD provider**, with DynamoDB daily cache. It is the only provider found with explicit Vietnam exchange support (`VN` / MIC `XSTC`) and a real free API-key plan. It is not real-time on free tier, but this bot is paper trading, so previous/official-ish EOD close is acceptable if rendered as EOD/last close.
|
||||
|
||||
Do not depend on SSI/KBS broker-app endpoints as primary. Live evidence from Lambda: SSI returns Cloudflare security pages, KBS times out from AWS egress. Yahoo works today, but it is not a stable contracted API for VN stocks. Keep Yahoo/SSI/KBS only as emergency fallback.
|
||||
|
||||
If user wants intraday/current price with a stable contract, no truly free option found. The free stable path is EOD only. Paid floor appears around $19.99/mo for EODHD all-world EOD; intraday/live is higher.
|
||||
|
||||
## Methodology
|
||||
|
||||
- Date: 2026-06-25.
|
||||
- Scope: Vietnam listed equities for `stock` module: `/stock_stats`, `/stock_buy`, `/stock_sell`.
|
||||
- Criteria: explicit VN coverage, free API key, documented REST API, low-volume Lambda-safe, legal/terms risk, implementation effort.
|
||||
- Sources consulted: EODHD, Marketstack, Twelve Data, FMP, Alpha Vantage, existing production probes.
|
||||
- Direct probes:
|
||||
- Twelve Data `/stocks?country=Vietnam`, `/stocks?country=Viet%20Nam`, `/stocks?exchange=HOSE` returned empty arrays.
|
||||
- Twelve Data `/stocks?symbol=FPT` resolved Thailand `FPT`, not Vietnam FPT Corp.
|
||||
- EODHD demo token returned `403 Forbidden` for `exchange-symbol-list/VN`; expected, demo token not real free key.
|
||||
- Marketstack without key returned `missing_access_key`; expected.
|
||||
|
||||
## Key Findings
|
||||
|
||||
### 1. EODHD
|
||||
|
||||
EODHD explicitly lists **Vietnam Stocks - VN**, country Vietnam, MIC `XSTC`, timezone `Asia/Ho_Chi_Minh`, and 593 active tickers. Example listed format is `AAA.VN`, `ACB.VN`, etc. Source: https://eodhd.com/exchange/VN
|
||||
|
||||
Free plan:
|
||||
- $0/mo.
|
||||
- 20 calls/day.
|
||||
- EOD data only, past-year depth.
|
||||
- Personal use.
|
||||
- Requires registration/API key.
|
||||
|
||||
Docs/pricing sources:
|
||||
- https://eodhd.com/financial-apis/api-for-historical-data-and-volumes
|
||||
- https://eodhd.com/pricing
|
||||
- https://eodhd.com/list-of-stock-markets
|
||||
|
||||
Pros:
|
||||
- Explicit VN coverage.
|
||||
- Documented REST API.
|
||||
- Free key allowed.
|
||||
- Enough for bot if we cache daily and request only held/watchlist symbols.
|
||||
- Symbol format simple: `FPT.VN`, `TCB.VN`, etc.
|
||||
|
||||
Cons:
|
||||
- Free quota low: 20/day.
|
||||
- EOD, not live current price.
|
||||
- Terms disclaim data may not be real-time/accurate and is indicative.
|
||||
- Commercial/display use may require paid/commercial agreement.
|
||||
|
||||
Verdict: **Primary recommendation**.
|
||||
|
||||
### 2. Marketstack
|
||||
|
||||
Marketstack has a documented free plan and stable API infra, but no confirmed Vietnam symbol coverage from public unauthenticated probe. It advertises global EOD data, stock tickers info, and exchange info.
|
||||
|
||||
Free plan:
|
||||
- 100 requests/month.
|
||||
- EOD data and up to 12 months history.
|
||||
- API key required.
|
||||
|
||||
Sources:
|
||||
- https://marketstack.com/
|
||||
- https://marketstack.com/pricing
|
||||
- https://docs.apilayer.com/marketstack/docs/api-endpoints-v1
|
||||
|
||||
Pros:
|
||||
- Documented.
|
||||
- Stable SaaS provider.
|
||||
- Supports multi-symbol EOD requests via `symbols` parameter according to docs/search result.
|
||||
|
||||
Cons:
|
||||
- Free quota worse than EODHD for daily bot use unless multi-symbol request covers all holdings.
|
||||
- VN coverage not verified without real key.
|
||||
- Intraday/real-time mostly paid/US-oriented.
|
||||
|
||||
Verdict: **Candidate backup only after registering a free key and proving `FPT`, `TCB`, `HPG`, `MSN`, `MWG`, `SSI`, `VND` coverage**.
|
||||
|
||||
### 3. Twelve Data
|
||||
|
||||
Twelve Data has good free quota reputation and strong docs, but public reference data did not show Vietnam equities in probes.
|
||||
|
||||
Source:
|
||||
- https://support.twelvedata.com/en/articles/5620513-how-to-find-all-available-symbols-at-twelve-data
|
||||
- https://twelvedata.com/
|
||||
|
||||
Probe results:
|
||||
- `https://api.twelvedata.com/stocks?country=Vietnam` -> `[]`
|
||||
- `https://api.twelvedata.com/stocks?country=Viet%20Nam` -> `[]`
|
||||
- `https://api.twelvedata.com/stocks?exchange=HOSE` -> `[]`
|
||||
- `https://api.twelvedata.com/stocks?symbol=FPT` -> Thailand `FPT`, not Vietnam.
|
||||
|
||||
Verdict: **Reject for now** unless support confirms VN coverage on a plan/add-on.
|
||||
|
||||
### 4. Financial Modeling Prep
|
||||
|
||||
FMP free plan is attractive on paper: 250 calls/day and EOD/reference data. But the public docs/pricing suggest global coverage only at higher tiers, and demo API key did not allow exchange/symbol coverage validation.
|
||||
|
||||
Sources:
|
||||
- https://site.financialmodelingprep.com/developer/docs
|
||||
- https://site.financialmodelingprep.com/pricing-plans
|
||||
|
||||
Pros:
|
||||
- 250 calls/day free.
|
||||
- Strong docs/API shape.
|
||||
|
||||
Cons:
|
||||
- Vietnam coverage not verified.
|
||||
- Pricing page implies Basic/free has limited symbols and EOD only; global coverage appears in Ultimate paid tier.
|
||||
- Display/redistribution needs licensing agreement.
|
||||
|
||||
Verdict: **Reject until a real free key confirms VN symbols**.
|
||||
|
||||
### 5. Alpha Vantage
|
||||
|
||||
Alpha Vantage is reputable and free-key based, but no evidence found for Vietnam exchange coverage. It is more useful for US/global large markets than VN equities.
|
||||
|
||||
Source:
|
||||
- https://www.alphavantage.co/
|
||||
|
||||
Verdict: **Reject for VN stocks**.
|
||||
|
||||
### 6. Broker/App Internal Endpoints
|
||||
|
||||
Current/existing providers:
|
||||
- SSI direct quote: `iboard-query.ssi.com.vn`.
|
||||
- SSI chart history: `iboard-api.ssi.com.vn`.
|
||||
- KBS data-day: `kbbuddywts.kbsec.com.vn`.
|
||||
- Yahoo chart: `query1.finance.yahoo.com`.
|
||||
|
||||
Findings:
|
||||
- SSI now returns 403 Cloudflare security page from this workspace and Lambda.
|
||||
- KBS returns locally but times out from Lambda.
|
||||
- Yahoo works from Lambda today, but is not a contracted API.
|
||||
|
||||
Verdict: **fallback only, never primary**.
|
||||
|
||||
## Comparative Analysis
|
||||
|
||||
| Provider | VN coverage proven | Free key | Free quota | Current price | Stable API | Fit |
|
||||
|---|---:|---:|---:|---:|---:|---|
|
||||
| EODHD | yes | yes | 20/day | EOD only | yes | best |
|
||||
| Marketstack | unknown | yes | 100/month | EOD free | yes | backup candidate |
|
||||
| Twelve Data | no in probes | yes | likely generous | unknown | yes | reject |
|
||||
| FMP | unknown | yes | 250/day | EOD free | yes | reject until proven |
|
||||
| Alpha Vantage | no evidence | yes | free | unknown | yes | reject |
|
||||
| SSI/KBS/Yahoo | yes-ish | no | unlimited-ish | yes | no | fallback only |
|
||||
|
||||
## Implementation Recommendation
|
||||
|
||||
### Provider Order
|
||||
|
||||
```text
|
||||
EODHD EOD cache -> Yahoo emergency -> SSI emergency -> KBS emergency -> no price
|
||||
```
|
||||
|
||||
Do not put Yahoo before EODHD once EODHD key exists. Yahoo is useful operationally but not stable.
|
||||
|
||||
### Data Model
|
||||
|
||||
```go
|
||||
type StockQuote struct {
|
||||
Symbol string
|
||||
PriceVND float64
|
||||
Source string // eodhd, yahoo, ssi, kbs
|
||||
AsOfDate string // YYYY-MM-DD
|
||||
RetrievedAt int64
|
||||
Stale bool
|
||||
}
|
||||
```
|
||||
|
||||
Cache keys:
|
||||
|
||||
```text
|
||||
stock-price:FPT:2026-06-25
|
||||
stock-price:TCB:2026-06-25
|
||||
```
|
||||
|
||||
### API Shape
|
||||
|
||||
EODHD EOD endpoint:
|
||||
|
||||
```text
|
||||
GET https://eodhd.com/api/eod/FPT.VN?api_token=$EODHD_API_KEY&fmt=json&period=d&from=YYYY-MM-DD&to=YYYY-MM-DD
|
||||
```
|
||||
|
||||
Use latest returned row:
|
||||
|
||||
```json
|
||||
{
|
||||
"date": "2026-06-25",
|
||||
"open": 71000,
|
||||
"high": 71700,
|
||||
"low": 70800,
|
||||
"close": 71000,
|
||||
"adjusted_close": 71000,
|
||||
"volume": 6592100
|
||||
}
|
||||
```
|
||||
|
||||
For portfolio valuation, use `close`. For paper buy/sell, either:
|
||||
- Use same `close` and label trade as "last close", or
|
||||
- Keep Yahoo as live-ish fallback only for trade commands.
|
||||
|
||||
### Quota Strategy
|
||||
|
||||
Free EODHD quota is 20/day. Therefore:
|
||||
|
||||
1. Cache per symbol per trading date.
|
||||
2. Fetch only missing cache entries.
|
||||
3. For `/stock_stats`, batch unique held symbols but call EODHD sequentially under a short timeout.
|
||||
4. Do not refresh more than once/day per symbol.
|
||||
5. If quota exceeded, use stale cache and label `(stale YYYY-MM-DD)`.
|
||||
|
||||
With current portfolio of 7 symbols, daily refresh costs 7 calls/day. Fits.
|
||||
|
||||
### Env Config
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
STOCK_EODHD_API_KEY_PARAMETER_NAME=/miti99bot/prod/eodhd-api-key
|
||||
STOCK_EODHD_API_URL=https://eodhd.com/api
|
||||
```
|
||||
|
||||
Store key in SSM SecureString, same pattern as Telegram/Gemini secrets.
|
||||
|
||||
### User-Facing Copy
|
||||
|
||||
When using EODHD:
|
||||
|
||||
```text
|
||||
FPT x2300 @ 71.000 VND (EOD 2026-06-25) = ...
|
||||
```
|
||||
|
||||
When fallback/stale:
|
||||
|
||||
```text
|
||||
FPT x2300 @ 71.000 VND (stale EOD 2026-06-24) = ...
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- Calling EODHD on every `/stock_stats`: burns quota fast.
|
||||
- Treating EOD price as live market price without label.
|
||||
- Assuming Marketstack/Twelve/FMP support Vietnam because marketing says "global".
|
||||
- Keeping broker-app endpoints as primary.
|
||||
- No stale cache. The bot should degrade to cached prices, not zero portfolio value.
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
1. Register EODHD free key.
|
||||
2. Manually verify these URLs with real key:
|
||||
- `FPT.VN`
|
||||
- `HPG.VN`
|
||||
- `MSN.VN`
|
||||
- `MWG.VN`
|
||||
- `SSI.VN`
|
||||
- `TCB.VN`
|
||||
- `VND.VN`
|
||||
3. If all seven work, implement `EODHDPriceProvider` + daily cache.
|
||||
4. Change `/stock_stats` labels to show source/date.
|
||||
5. Keep Yahoo/SSI/KBS emergency fallbacks after EODHD/cache.
|
||||
|
||||
## Resources
|
||||
|
||||
- EODHD Vietnam exchange: https://eodhd.com/exchange/VN
|
||||
- EODHD supported exchanges: https://eodhd.com/list-of-stock-markets
|
||||
- EODHD EOD API: https://eodhd.com/financial-apis/api-for-historical-data-and-volumes
|
||||
- EODHD pricing: https://eodhd.com/pricing
|
||||
- Marketstack pricing: https://marketstack.com/pricing
|
||||
- Marketstack docs: https://docs.apilayer.com/marketstack/docs/api-endpoints-v1
|
||||
- Twelve Data symbol reference: https://support.twelvedata.com/en/articles/5620513-how-to-find-all-available-symbols-at-twelve-data
|
||||
- FMP docs: https://site.financialmodelingprep.com/developer/docs
|
||||
- FMP pricing: https://site.financialmodelingprep.com/pricing-plans
|
||||
- Alpha Vantage: https://www.alphavantage.co/
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
- Does EODHD free key return all current holdings (`FPT.VN`, `HPG.VN`, `MSN.VN`, `MWG.VN`, `SSI.VN`, `TCB.VN`, `VND.VN`) via API, not just website pages?
|
||||
- Is "last close" acceptable for `/stock_buy` and `/stock_sell`, or should those commands keep live-ish Yahoo fallback?
|
||||
- Is bot use personal/non-commercial under EODHD terms, or does Telegram display to group chats require paid/commercial terms?
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user