docs(selfhost): plan Coolify + MongoDB Atlas self-host with AWS decommission

Add 5-phase plan to self-host on Coolify (docker-compose) with a MongoDB
Atlas backend, an in-process cron scheduler, DynamoDB->Atlas migration,
and full AWS teardown. Red-teamed and validated; decommission scope
verified against live AWS/Cloudflare accounts. Include free-tier audit
and S3-elimination research reports. Ignore wrangler local cache.
This commit is contained in:
2026-06-27 20:46:43 +07:00
parent 50f1566494
commit 0a66fb15bc
9 changed files with 934 additions and 0 deletions
+3
View File
@@ -43,3 +43,6 @@ bin/
# Per-developer SAM overrides (samconfig.toml IS checked in)
samconfig.local.toml
# wrangler local cache
.wrangler/
@@ -0,0 +1,81 @@
---
phase: 1
title: "MongoDB Storage Provider"
status: pending
priority: P1
dependencies: []
effort: "M"
---
# Phase 1: MongoDB Storage Provider
## Overview
Add `mongodb` as a 4th `KVProvider`, modeled exactly on the existing `firestore` provider (collection-per-module isolation). Wire it into `buildProvider` and config via `KV_PROVIDER=mongodb`, `MONGO_URL`, `MONGO_DATABASE`. No module code changes — the `KVStore` interface is the only contract.
## Requirements
- Functional: implement `KVStore` (Get/GetJSON/Put/PutJSON/Delete/List) + `CompareAndSwapStore` (CompareAndSwap) backed by MongoDB.
- Functional: one Mongo collection per module (mirrors `FirestoreProvider`); document `_id` = key, field `value` = raw bytes, field `updatedAt` = timestamp.
- Functional: `List(prefix)` = range/regex query on `_id` with `begins_with` semantics; empty prefix = whole collection.
- Non-functional: behavior parity with firestore/dynamodb — same `ErrNotFound`, `ErrConflict`, key validation (reuse `validateKey`/`validatePrefix`), same `collectionNameRe` module-name guard via `invalidStore`.
- Non-functional: bounded startup connect timeout (match `dynamodbInitTimeout = 5s` style), graceful `Close()`.
## Architecture
Reuse the shared helpers already in `internal/storage`:
- `validateKey` / `validatePrefix` (in `firestore_kv.go`) — key constraints. Mongo `_id` has no `/` restriction, but reusing keeps cross-backend parity and is harmless.
- `collectionNameRe` (in `firestore_provider.go`) — module-name alphabet.
- `invalidStore` (in `invalid_store.go`) — returned for bad module names.
Document shape (parity with firestore `value`/`updatedAt`):
```
{ "_id": "<key>", "value": <BinData>, "updatedAt": <int64 nanos> }
```
Store `value` as BSON binary (`bson.Binary`) so non-UTF-8 round-trips; on read accept both binary and string (firestore does the same dual-type handling). **Store `updatedAt` as int64 unix-nanos (NOT BSON datetime)** — matches DynamoDB exactly (`dynamodb_kv.go:101`), keeps migration byte-faithful, and avoids ms-truncation if a future TTL/sort ever reads it. The migrator (Phase 4) and the provider MUST share one encoding — see Phase 4 (migrator writes through `MongoKVStore.Put`, not raw `UpdateOne`).
`CompareAndSwap` mapping — the `expected == nil` branch is a LIVE path (first write of every new coin/gold portfolio, `coin/portfolio.go:81-83`, `gold/portfolio.go:62-80`), not an edge case. Map it to a plain **`InsertOne`** and rely SOLELY on the unique `_id` index for the conflict:
- `expected == nil``InsertOne({_id, value, updatedAt})`; `mongo.IsDuplicateKeyError(err)``ErrConflict`. Do NOT use a `{value:{$exists:false}}` upsert filter (it can false-conflict on a value-less doc and muddies the contract).
- `expected != nil``UpdateOne({_id, value: expected}, {$set:{value,updatedAt}})`; `MatchedCount == 0``ErrConflict`.
`_id` is unique by default, so the absent-insert race is linearizable: exactly one `InsertOne` wins, losers get duplicate-key → `ErrConflict` → caller retry loop reloads the winner's state. This must be proven by a **blocking** concurrent-writer test (see Success Criteria), not just asserted.
`List(prefix)`: `Find({_id: {$gte: prefix, $lt: prefixSuccessor(prefix)}}, projection={_id:1})` — reuse the existing `prefixSuccessor` helper from `firestore_kv.go`. Empty prefix → `Find({})`. Avoids regex injection and uses the `_id` index.
## Related Code Files
- Create: `internal/storage/mongodb_client.go``NewMongoClient(ctx, uri) (*mongo.Client, error)` with connect+ping timeout; `NewMongoDatabase`. Mirror `dynamodb_client.go`.
- Create: `internal/storage/mongodb_provider.go``MongoProvider{ db *mongo.Database }`, `For(module)` returns `invalidStore` on bad name else `NewMongoKVStore`. Mirror `firestore_provider.go`.
- Create: `internal/storage/mongodb_kv.go``MongoKVStore`, all methods. Mirror `firestore_kv.go`.
- Create: `internal/storage/mongodb_kv_test.go` + `mongodb_provider_test.go` — parity tests, gated on `MONGODB_TEST_URL` (skip when unset), mirroring `dynamodb_kv_test.go` gating on `DYNAMODB_LOCAL_URL`.
- Modify: `cmd/server/main.go` — add `mongodb` case to `buildProvider`; add `MongoURL`, `MongoDatabase` to `config` + `loadConfig` (`MONGO_URL`, `MONGO_DATABASE`); update `buildProvider` doc comment + auto-detect note (mongo is explicit-only).
- Modify: `go.mod` / `go.sum` — add `go.mongodb.org/mongo-driver/v2`.
- Modify: `Makefile` — add `mongo-local` (docker `mongo:7`) + `test-mongo` target gated by `MONGODB_TEST_URL`, mirroring `dynamodb-local`/`test-dynamodb`.
- Modify: `README.md` — add `mongodb` to the storage backend list + local-run snippet.
## Implementation Steps
1. `go get go.mongodb.org/mongo-driver/v2/mongo` (and `/bson`).
2. Write `mongodb_client.go`: connect with `options.Client().ApplyURI(uri)`, `client.Ping` under a 5s context, return client; helper to get `*mongo.Database` from `MONGO_DATABASE`.
3. Write `mongodb_kv.go`: implement methods per Architecture; reuse `validateKey`, `validatePrefix`, `prefixSuccessor`; constants `mongoValueField="value"`, `mongoUpdatedAtField="updatedAt"`.
4. Write `mongodb_provider.go`: `For` guards with `collectionNameRe`, returns `db.Collection(module)`-backed store.
5. Wire `buildProvider`: `case "mongodb"`: require `MONGO_URL` + `MONGO_DATABASE` (error if missing, mirror dynamodb's `DYNAMODB_TABLE` check); construct client under timeout; closer calls `client.Disconnect`. **The startup `log.Info("storage backend", …)` line MUST log only non-secret fields — `"backend","mongodb","database",cfg.MongoDatabase`. NEVER log `MONGO_URL`** (it is `mongodb+srv://user:pass@…`; the firestore/dynamodb cases at `main.go:234-253` log a benign identifier, but the mongo equivalent is a credential). If a host is wanted for diagnostics, parse and log only the host, never the userinfo.
6. Add config fields + env reads. Mongo is explicit-only (not in the auto-detect switch) to avoid surprising Lambda.
7. Tests: replicate the firestore/dynamodb test bodies against a real Mongo (`mongo-local`), covering Get/Put/Delete/List/prefix/CAS-absent/CAS-match/CAS-conflict/ErrNotFound + cross-module isolation.
8. `make vet && make test && MONGODB_TEST_URL=mongodb://localhost:27017 make test-mongo`.
## Success Criteria
- [ ] `internal/storage` exposes `MongoProvider`/`MongoKVStore` passing the same test matrix as `DynamoDBKVStore`.
- [ ] `KV_PROVIDER=mongodb` with `MONGO_URL`/`MONGO_DATABASE` boots; missing either errors clearly at startup.
- [ ] Cross-module isolation verified (collection-per-module).
- [ ] CompareAndSwap returns `ErrConflict` on stale expected + on absent-with-non-nil-expected; succeeds on nil-expected insert.
- [ ] **(blocking)** Concurrent-writer CAS test: N goroutines race a nil-expected insert + a stale-update on the same key against a real Mongo; assert exactly one winner, losers get `ErrConflict`. Plus a "doc exists without value field" edge case.
- [ ] Startup log shows `backend=mongodb database=<db>` and does NOT contain the connection string / any credential.
- [ ] `make vet` and `make test` pass; AWS/firestore paths untouched.
## Risk Assessment
- **CAS semantics drift**: Mongo upsert race differs from DynamoDB conditional put. Mitigation: unique `_id` + `IsDuplicateKeyError` for the absent case; assert with a concurrent-writer test.
- **Value type on read**: driver may decode as binary or string. Mitigation: dual-type switch like firestore's.
- **TLS to Atlas**: `mongodb+srv://` URIs need DNS SRV + TLS. Mitigation: driver handles it; document that `MONGO_URL` is the full Atlas SRV connection string incl. credentials.
- **Driver version**: v2 API differs from v1 (`mongo.Connect` signature). Mitigation: pin v2, follow current docs.
@@ -0,0 +1,93 @@
---
phase: 2
title: "In-Process Cron Scheduler"
status: pending
priority: P1
dependencies: []
effort: "S"
---
# Phase 2: In-Process Cron Scheduler
## Overview
Off AWS there is no EventBridge Scheduler to hit `/cron/{name}`. Add an opt-in in-process scheduler that reads each registered cron's existing `Cron.Schedule` field and fires `Cron.Handler` on time. Gated by `CRON_MODE=internal` so the Lambda/EventBridge path is unchanged by default.
## Requirements
- Functional: when `CRON_MODE=internal`, parse every `reg.Crons()[i].Schedule` and invoke its handler on schedule, in UTC (match EventBridge's `ScheduleExpressionTimezone: UTC`).
- Functional: default (`CRON_MODE` unset/`external`) starts NO scheduler — preserves current Lambda behavior where EventBridge owns timing.
- Functional: the existing `/cron/{name}` HTTP route stays as-is (still usable for manual/curl triggers and as the sidecar fallback).
- Non-functional: each tick runs under the same 60s budget as the HTTP path (`defaultCronTimeout`); a panicking handler is recovered and logged, scheduler keeps running.
- Non-functional: scheduler stops cleanly on `rootCtx` cancellation (graceful shutdown).
## Architecture
The `Cron` struct already carries `Schedule string` (currently "documentation only", e.g. lolschedule `"0 1 * * *"`). Promote it to the real trigger source for self-host.
Use `github.com/robfig/cron/v3` (standard, well-maintained 5-field cron parser) with `cron.New(cron.WithLocation(time.UTC))`. For each registered cron, `c.AddFunc(schedule, fn)` where `fn` dispatches the handler through the existing cron dispatcher path so logging/metrics/timeout/panic-recovery are identical to the HTTP route.
Dispatch: the existing exported helper is **`modules.DispatchScheduled(ctx, name, reg)`** (`cron_dispatcher.go:19` — note arg order: name BEFORE reg). It does ONLY a registry lookup + `cron.Handler(ctx, deps)` — it has **no timeout, no panic-recovery, no structured logging**. That wrapping lives in the server-package `cronHandler` (`internal/server/router.go`), NOT in the dispatcher. So the scheduler must add its own:
- wrap each fire in `context.WithTimeout(ctx, 60s)` (the `defaultCronTimeout` constant is in `internal/server/timeouts.go`; do not import `internal/server` into the scheduler — define a local `cronTimeout = 60*time.Second` or lift the constant to a shared package to avoid a layering inversion),
- `recover()` around the handler call, logging the cron name on panic so one bad cron doesn't kill the scheduler,
- structured `log.Info("cron triggered", …)` / `log.Error("cron failed", …)` mirroring the HTTP path.
Call `modules.DispatchScheduled(ctx, name, reg)` inside that wrapper.
New file `internal/cron/scheduler.go` (new small package, mirrors `internal/metrics` lifecycle style):
```
func Run(ctx context.Context, reg *modules.Registry) (stop func(), err error)
```
- Skips crons with empty `Schedule` (logs a warning — a self-hosted cron with no schedule never fires).
- Validates schedule strings at startup; a bad expression is fatal (fail fast, like other config errors).
Wire in `cmd/server/main.go` after `modules.Install`, gated:
```
if strings.EqualFold(cfg.CronMode, "internal") {
stop, err := cron.Run(rootCtx, reg)
if err != nil { log.Fatal("cron scheduler init failed", "err", err) }
defer stop()
log.Info("internal cron scheduler started", "crons", len(reg.Crons()))
}
```
Add `CronMode` to `config` from env `CRON_MODE`.
## Related Code Files
- Create: `internal/cron/scheduler.go` — scheduler lifecycle.
- Create: `internal/cron/scheduler_test.go` — fake registry with a fast schedule (`@every 1s` or injected clock) asserts handler fires; bad-schedule errors; ctx-cancel stops.
- Modify: `cmd/server/main.go``CronMode` config + gated `cron.Run`.
- Modify: `internal/modules/module.go` doc comments — update `CronHandler`/`Cron.Schedule` text that currently says "real schedule lives in EventBridge" to note the `CRON_MODE=internal` path. (The scheduler calls the existing `modules.DispatchScheduled`; if the 60s-timeout/panic-recover wrapper is worth sharing with the HTTP `cronHandler`, lift it to a shared helper — but that is optional, not required.)
- Modify: `go.mod` / `go.sum` — add `github.com/robfig/cron/v3`.
- Modify: `README.md` — document `CRON_MODE` (external default vs internal self-host).
- Modify: `internal/modules/lolschedule/cron.go` — add an idempotency guard: the daily-push handler reads/writes a KV "last push UTC date" key and no-ops if already pushed today (defends against all double-fire windows). This also makes the existing EventBridge path safe during cutover overlap.
## Implementation Steps
1. `go get github.com/robfig/cron/v3`.
2. Call `modules.DispatchScheduled(ctx, name, reg)` (`cron_dispatcher.go:19`) from inside a scheduler-local wrapper that adds the 60s timeout + `recover()` + logging (the dispatcher provides none of these).
3. Write `internal/cron/scheduler.go`: build `cron.New(cron.WithLocation(time.UTC))`, register each non-empty schedule, `c.Start()`, return a `stop` that calls `c.Stop()` and waits for the context done.
4. Wire gated startup in `main.go`; add `CronMode` config field + env read.
5. Update the now-stale "documentation only / EventBridge owns timing" comments on `Cron.Schedule` and `CronHandler`.
6. Tests + `make vet && make test`.
## Success Criteria
- [ ] With `CRON_MODE=internal`, lolschedule daily push fires at 01:00 UTC; observable in logs (`cron triggered`).
- [ ] With `CRON_MODE` unset, no scheduler starts (Lambda/EventBridge path byte-for-byte unchanged).
- [ ] Bad schedule string fails startup with a clear error.
- [ ] Handler panic is recovered (scheduler-local `recover()`); scheduler survives and fires next tick.
- [ ] Each fire runs under a 60s timeout (scheduler-local, not imported from `internal/server`).
- [ ] Daily push is idempotent per UTC date: invoking the handler twice on the same date sends subscribers exactly one digest.
- [ ] Scheduler stops within shutdown grace period on SIGTERM.
## Risk Assessment
- **Double-fire (Critical — the daily push is NOT idempotent)**: `lolschedule` `runDailyPush` (`cron.go:128-191`) fans out to every subscriber unconditionally — no "already sent today" marker. So ANY double-fire = every subscriber DM'd twice. Three concrete double-fire windows the opt-in flag does NOT cover:
1. **Cutover overlap**: EventBridge `AWS::Scheduler::Schedule` (`template.yaml:271-289`) invokes the Lambda DIRECTLY, independent of the webhook URL — re-pointing the webhook does NOT stop it. It must be **disabled/deleted before** the Coolify container runs `CRON_MODE=internal` (ordered prerequisite in Phase 4, not "N days later" cleanup).
2. **Rolling deploy**: Coolify/compose may run old+new containers briefly; both run the in-process scheduler. A redeploy near 01:00 UTC double-fires.
3. Operator misconfig (`internal` set on a second instance).
**Primary mitigation (covers all three cheaply)**: add a KV "last push date" guard in `lolschedule` — the handler records the date it pushed and no-ops if already pushed for that UTC date. This makes the push idempotent regardless of trigger count. Secondary: prefer stop-first redeploy in Coolify; keep `CRON_MODE` opt-in and never set `internal` on Lambda.
- **5-field vs 6-field cron**: `"0 1 * * *"` is 5-field standard. Mitigation: use robfig `cron/v3` default 5-field parser (not the seconds-enabled one).
- **Single-instance assumption**: if Coolify scales the service to >1 replica, crons fire per replica. Mitigation: document "run exactly 1 replica" (the bot is a single-instance webhook consumer anyway); revisit with a DB lock only if scaling is ever needed (YAGNI now).
- **Missed fire while container restarts**: a deploy at 01:00 UTC could skip that day's push. Mitigation: accept (same risk class as Lambda cold-start miss); not data-loss.
@@ -0,0 +1,100 @@
---
phase: 3
title: "Containerize and Coolify Deploy"
status: pending
priority: P2
dependencies: [1, 2]
effort: "M"
---
# Phase 3: Containerize and Coolify Deploy
## Overview
Ship the existing distroless image via a docker-compose stack on Coolify. MongoDB Atlas is external/managed, so compose runs only the bot service. Configuration (secrets + backend selection) is supplied as Coolify env vars. Document webhook registration against the Coolify public domain.
## Requirements
- Functional: `docker compose up` runs the bot on `:8080` with `KV_PROVIDER=mongodb` + `CRON_MODE=internal`, persistent storage in Atlas.
- Functional: Coolify exposes the service over HTTPS at a domain; `GET /` returns `text/plain` `miti99bot ok` (health, not JSON).
- Functional: Telegram webhook points at `https://<coolify-domain>/webhook` with the secret token.
- Non-functional: container healthcheck on `/`; restart policy `unless-stopped`; runs a single replica (cron correctness, see Phase 2).
- Non-functional: no secrets committed — all via Coolify env / `.env` (gitignored). Provide `.env.example`.
## Architecture
The existing `Dockerfile` (golang:1.25-alpine builder → distroless static nonroot, `ENTRYPOINT ["/server"]`, `EXPOSE 8080`) builds with `-ldflags="-s -w"` only — it does **not** inject `gitSHA`. Only the Makefile injects it (`Makefile:13`). `deploynotify` runs unconditionally at startup (`main.go:148`) and stays silent when `gitSHA` is empty (`main.go:40-41`), so as-is the "new version" owner DM is **silently dead on self-host** — a behavior regression from Lambda. Required (not optional): add `ARG GIT_SHA` + `-ldflags "-s -w -X main.gitSHA=$GIT_SHA"` to the Dockerfile and pass `--build-arg GIT_SHA=$(git rev-parse --short HEAD)` (Coolify exposes the commit SHA as a build var). If deploynotify parity is explicitly unwanted, instead document in acceptance criteria that the feature is disabled on self-host — do not leave it as undocumented breakage.
`docker-compose.yml` (committed; Coolify consumes it):
```yaml
services:
bot:
build: . # or image: ghcr.io/tiennm99/miti99bot:latest
restart: unless-stopped
environment:
PORT: "8080"
KV_PROVIDER: mongodb
MONGO_URL: ${MONGO_URL}
MONGO_DATABASE: ${MONGO_DATABASE}
CRON_MODE: internal
MODULES: ${MODULES}
BOT_OWNER_ID: ${BOT_OWNER_ID}
ADMIN_USER_IDS: ${ADMIN_USER_IDS}
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
TELEGRAM_WEBHOOK_SECRET: ${TELEGRAM_WEBHOOK_SECRET}
GEMINI_API_KEY: ${GEMINI_API_KEY}
# NOTE: CRON_SHARED_SECRET intentionally OMITTED — with CRON_MODE=internal
# the in-process scheduler owns cron timing, so leaving it unset makes the
# public /cron/{name} route 404 (router.go:59,66-69) and removes a redundant
# internet-reachable trigger surface. Set it only if you want manual curl triggers.
# NOTE: do NOT set any *_PARAMETER_NAME vars (see Secrets below).
# optional API overrides as needed
expose: ["8080"]
# No compose healthcheck: distroless has no shell/curl AND cmd/server has no
# flag parsing (no -healthcheck flag exists). Use Coolify's native HTTP monitor
# against GET / (returns text/plain "miti99bot ok", NOT JSON).
```
Healthcheck: distroless has no shell/curl/wget, and `cmd/server` has no flag parsing — `/server -healthcheck` does NOT exist and would just start the server. Use **Coolify's native HTTP monitor against `/`** (returns `text/plain` body `miti99bot ok`, per `internal/server/health.go` — not JSON). Only if a compose-level healthcheck is mandatory, first implement a real `-healthcheck` flag in `cmd/server` (localhost GET `/`, exit 0/1) — otherwise omit it. Note: a plain `/` check does NOT verify Mongo connectivity (see Risk Assessment).
Secrets: because `*_PARAMETER_NAME` env vars are NOT set, `cmd/server` reads `TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `CRON_SHARED_SECRET`, `GEMINI_API_KEY` directly (verified: `resolveSSMSecrets` returns early when no param names are set, `main.go:355-366`). **Hard requirement: `.env.example` must list all six `*_PARAMETER_NAME` vars explicitly as "leave UNSET for self-host."** `resolveSSMSecrets` is called unconditionally (`main.go:79`); if even one `*_PARAMETER_NAME` is set with an unset target, it calls `awsconfig.LoadDefaultConfig` + `ssm.GetParameters` (`main.go:371-378`) which fails with no AWS creds → `log.Fatal` (`main.go:80`), bricking startup. With all six unset, no AWS credentials are needed in the container.
## Related Code Files
- Create: `docker-compose.yml` — bot service as above.
- Create: `.env.example` — every env var with placeholder values + comments; real `.env` gitignored.
- Modify: `.gitignore` — ensure `.env` ignored (verify; add if missing).
- Modify (optional): `Dockerfile``ARG GIT_SHA` + `-ldflags "-X main.gitSHA=$GIT_SHA"` for deploynotify parity.
- Modify (optional): `cmd/server/main.go``-healthcheck` flag (only if compose healthcheck chosen over Coolify HTTP monitor).
- Create: `docs/deploy-coolify-selfhosted.md` — full onboarding: create Atlas M0 cluster, get SRV URL, create a least-privilege DB user (`readWrite` on one DB, strong unique password), set network access to `0.0.0.0/0` (accepted trade-off — record it), Coolify new resource from compose/Git, set env vars, deploy, grab public domain, register webhook + command menu.
- Modify: `README.md` — add "Self-host (Coolify + MongoDB Atlas)" deploy option alongside the AWS path.
## Implementation Steps
1. Write `docker-compose.yml` + `.env.example`; verify `.env` gitignored.
2. Decide healthcheck approach (Coolify HTTP monitor preferred); implement `-healthcheck` flag only if needed.
3. Local validation: `MONGO_URL=… MONGO_DATABASE=… docker compose up --build`; confirm boot logs show `storage backend backend=mongodb database=…` (NO connection string) and `internal cron scheduler started`; `curl localhost:8080/` returns `miti99bot ok` (plain text).
4. Atlas setup (M0): cluster, DB user, network access, copy `mongodb+srv://…` SRV URL.
5. Coolify: create resource (Git repo + compose, or prebuilt image), set env vars, deploy, note assigned HTTPS domain.
6. Register webhook + commands against the Coolify domain (reuse `make telegram-webhook`/`telegram-commands` with the URL, or the documented curl). Note: `make telegram-*` currently pulls token from SSM — for self-host, add env-var-based variants or document the direct curl.
7. Smoke test: send `/ping`, `/help`; confirm a persisted command (e.g. stock paper trade) survives a container restart.
8. Write `docs/deploy-coolify-selfhosted.md`; update README.
## Success Criteria
- [ ] `docker compose up --build` boots the bot with mongo + internal cron locally.
- [ ] Coolify deployment is reachable at its HTTPS domain; `/` health passes; restarts `unless-stopped`.
- [ ] Telegram webhook set to the Coolify domain; live commands respond.
- [ ] No secret committed; `.env.example` documents every variable, lists all six `*_PARAMETER_NAME` as "leave UNSET", and omits `CRON_SHARED_SECRET` by default.
- [ ] Container boots with zero `*_PARAMETER_NAME` set (no AWS creds present) and never attempts an SSM/AWS call.
- [ ] `deploynotify` parity decided: either `gitSHA` injected via Dockerfile build arg (owner DM works) or the feature is documented as disabled on self-host.
- [ ] `docs/deploy-coolify-selfhosted.md` is complete enough to redo from scratch.
## Risk Assessment
- **Distroless healthcheck**: no shell, and no `-healthcheck` flag exists. Mitigation: use Coolify's HTTP monitor against `/` (text body `miti99bot ok`), or implement the flag first. Do not ship the bogus `["CMD","/server","-healthcheck"]` line.
- **Atlas network access — `0.0.0.0/0` accepted (validated decision)**: the Coolify host has no stable egress IP, so the Atlas IP access list is `0.0.0.0/0` (internet-reachable). This widens the surface beyond the project's documented "no non-designed-surface public resource" boundary (`docs/deploy-aws-free-tier-guide.md:25,35`) — under DynamoDB the DB was IAM-gated and never internet-reachable — and is knowingly accepted for self-host. **Mandatory compensating controls (hard requirements):** (1) a strong unique DB password, (2) a least-privilege Atlas DB user scoped to `readWrite` on the single app database (never Atlas admin / cluster-wide), (3) the connection string is treated as a secret and never logged (see Phase 1 / Finding-3 constraint). Record the acceptance in `docs/deploy-coolify-selfhosted.md`. <!-- Updated: Validation Session 1 - 0.0.0.0/0 accepted; least-priv user + strong password required -->
- **Public ingress is now the bot's own process (Medium)**: a Function URL sat behind AWS edge (throttling, managed TLS, absorbed L3/4 floods); a Coolify domain exposes the single `:8080` Go process directly, and Phase 2 pins 1 replica. An unauthenticated flood saturates the box even though `/webhook` rejects via constant-time secret compare (`telegram/webhook.go:55-60`). Mitigation: front the service with Coolify's built-in Traefik rate-limit middleware (no extra cost); keep existing `ReadHeaderTimeout`/`ReadTimeout` (`main.go:165-166`).
- **Mongo runtime connection loss (Medium)**: unlike the per-call short-lived DynamoDB client (`dynamodb_client.go`), self-host holds one long-lived Mongo client for days; Atlas M0 idles/fails over. The v2 driver auto-reconnects on the next op (confirm against current driver docs) — set sane pool + server-selection timeout options. A plain `/` healthcheck reports healthy even when Mongo is unreachable, so Coolify won't restart a DB-wedged bot; either make the healthcheck DB-aware (lightweight `Ping`) or explicitly accept "stays up reporting healthy during DB outage" as the chosen trade-off.
- **Webhook tooling assumes SSM**: `make telegram-*` reads SSM. Mitigation: document env-var/curl variant for self-host; optionally add `telegram-webhook-url URL=…` Make target.
- **Single replica**: scaling >1 double-fires cron + duplicates webhook processing is fine but cron is not. Mitigation: pin 1 replica in Coolify; documented in Phase 2.
@@ -0,0 +1,95 @@
---
phase: 4
title: "Data Migration and Cutover"
status: pending
priority: P1
dependencies: [1, 3]
effort: "M"
---
# Phase 4: Data Migration and Cutover
## Overview
Copy all existing items from the prod DynamoDB table (`miti99bot-data`) into MongoDB Atlas using the finalized Phase 1 document schema, verify parity, then flip the Telegram webhook from the Lambda Function URL to the Coolify domain. Idempotent and re-runnable.
## Requirements
- Functional: every DynamoDB item → one Mongo document in collection `pk`, `_id = sk`, `value` = decoded value, `updatedAt` preserved.
- Functional: idempotent (re-run overwrites by `_id`, no duplicates); resumable on failure.
- Functional: verification step compares per-module (per-`pk`) counts DynamoDB vs Mongo and spot-checks values.
- Non-functional: read-only on DynamoDB (Scan only); never mutates source.
- Non-functional: dry-run mode that reports counts without writing.
## Architecture
Mapping (confirmed from `dynamodb_kv.go``firestore_kv.go`):
| DynamoDB | Mongo |
|---|---|
| `pk` (S) = module name | collection name |
| `sk` (S) = user key | document `_id` |
| `value` (S) = JSON string | `value` field (BSON binary, same bytes) |
| `updatedAt` (N) = unix nanos | `updatedAt` (int64 nanos, verbatim) |
A full-table `Scan` (table is small KV) yields all items across all partitions. Group by `pk` → write into the matching collection. Read with `storage.NewDynamoDBClient`; **write through the Phase 1 `MongoKVStore.Put` (NOT raw `UpdateOne`)** so `value`/`updatedAt` encoding is byte-identical to what the live app writes — otherwise the migrator could store `value` as a BSON string while the app writes binary, and a later CAS (`{value: expected}`) never matches → spurious `ErrConflict` and the "idempotent re-run" criterion silently breaks.
New one-off CLI `cmd/migrate-dynamo-to-mongo/main.go`:
- Flags/env: `--dynamodb-table` (default `miti99bot-data`), `MONGO_URL`, `MONGO_DATABASE`, `--dry-run`, `--verify`.
- Uses AWS default cred chain via a **dedicated read-only profile/role** (see IAM below) — NOT an admin profile.
- For each item: **`validateKey(sk)` before writing**; fail loudly on any `/`-containing or otherwise-invalid `sk`. The app's read path runs `validateKey` first (firestore/dynamodb pattern), so a key the migrator writes but `validateKey` rejects would be silently unreadable. No `/` keys exist today (all modules use `:` separators), so this is a guard, not a transform.
- Writes via `MongoKVStore.Put`; `Put` overwrites by `_id`, so re-runs are idempotent and produce no duplicates.
- `--verify`: per-`pk` count on DynamoDB via a **Scan tally** (so a single `dynamodb:Scan` permission suffices — do NOT use Query, which would need `dynamodb:Query` and over-grant) vs `CountDocuments` per collection; print a table + mismatches; exit non-zero on mismatch.
IAM (least privilege): exactly `dynamodb:Scan` on the specific table ARN (`arn:…:table/miti99bot-data`), nothing else. No write actions on the source — enforces the read-only requirement and removes the destructive-credential foot-gun.
Cutover sequence (documented runbook — zero-loss):
1. Deploy Coolify stack (Phase 3) with a fresh/empty Atlas DB. Do NOT set the Telegram webhook to it yet. Do NOT set `CRON_MODE=internal` yet (avoids cron overlap, below).
2. **Disable/delete the EventBridge `LolscheduleDailyPushSchedule`** (`template.yaml:271-289`) — it invokes the Lambda directly, independent of the webhook, so it must be stopped before any internal scheduler runs. (The Phase 2 last-push-date guard is the belt-and-braces backup.)
3. `deleteWebhook` (MANDATORY, not optional) — Telegram now buffers incoming updates up to 24h. This is what makes the cut lossless; do NOT take an "accept the gap" path (a `/buy` between snapshot and flip would write to DynamoDB only and be lost — `coin/portfolio.go:64-90`).
4. Run migrator `--dry-run` → review counts. Then run for real; run `--verify` (counts equal per `pk`, exit 0). Keep this window short (target minutes).
5. Set `CRON_MODE=internal` on the Coolify service (now safe — EventBridge disabled).
6. `setWebhook` → Coolify domain. Telegram delivers the buffered updates to the new host.
7. Verify `getWebhookInfo`: `pending_update_count` drains toward 0 and `last_error_date` is empty (same health signal as `docs/deploy-aws.md:77`).
8. Smoke `/ping`, `/stats`, a coin/stock balance command — confirm migrated state is visible from Atlas.
9. **Tear down AWS** (validated decision): after `--verify` passes and smoke tests are green, `sam delete` the stack (Lambda, DynamoDB, EventBridge, SQS, etc.) and disable the GitHub Actions `deploy.yml` workflow. The user coordinates users to avoid activity during the brief cutover window, so the simple cut is lossless; no reverse migrator is built. <!-- Updated: Validation Session 1 - tear down AWS after verify; no reverse migrator -->
10. **Last clean revert point:** rollback to AWS is only possible BEFORE `sam delete`. If you want a safety margin, defer step 9's `sam delete` by a short observation window (still keeping EventBridge disabled); re-pointing the webhook to the Function URL reverts with ~zero loss only until the first post-cutover Mongo write. After `sam delete`, MongoDB/Coolify is the sole system of record.
## Related Code Files
- Create: `cmd/migrate-dynamo-to-mongo/main.go` — the migrator.
- Create: `cmd/migrate-dynamo-to-mongo/README.md` (or section in deploy doc) — usage, required IAM (`dynamodb:Scan` on the table), env vars, dry-run/verify, cutover runbook.
- Modify: `Makefile``migrate-dynamo-to-mongo` + `migrate-verify` targets wrapping the CLI with sensible defaults.
- Modify: `docs/deploy-coolify-selfhosted.md` — link the cutover runbook.
- Reference: prior `plans/260515-2250-cf-data-to-aws-migration/` — same migration shape (Firestore→DynamoDB); reuse its verification approach if present.
## Implementation Steps
1. Implement the migrator using existing storage clients; default table `miti99bot-data`.
2. Implement `--dry-run` (scan + group + report counts, no writes) and `--verify`.
3. Test locally: seed DynamoDB Local with a few items across 2+ modules, run against Mongo Local, assert documents match via the Phase 1 `MongoKVStore.Get`.
4. Add Make targets.
5. Dry-run against prod DynamoDB; review counts per module.
6. Execute migration + verify; record the count table in the cutover doc.
7. Flip webhook; smoke-test; monitor.
## Success Criteria
- [ ] Dry-run reports per-module item counts without writing.
- [ ] Real run copies all items; `--verify` shows DynamoDB and Mongo counts equal for every `pk`, exit 0.
- [ ] Re-running the migrator produces no duplicates and no value changes (idempotent).
- [ ] Migrator runs `validateKey` per `sk` and writes through `MongoKVStore.Put`; a `Get` round-trip spot check returns byte-identical values.
- [ ] After webhook cutover, a previously-stored value (e.g. a user's paper-trade balance) is returned by the live bot from Atlas.
- [ ] EventBridge schedule disabled before `CRON_MODE=internal`; daily push fires exactly once on cutover day.
- [ ] Rollback truth documented: webhook re-point is lossless only before the first post-cutover Mongo write (RPO stated explicitly).
## Risk Assessment
- **Rollback bound — accepted (validated decision)**: AWS is torn down after verify, so there is no long-term fallback by design and no reverse migrator is built. The user coordinates users to pause around the brief cutover window, so the simple cut loses no writes. The only clean-revert opportunity is the short observation window BEFORE `sam delete`, and even then re-pointing the webhook is lossless only until the first post-cutover Mongo write. This is an accepted RPO for a personal paper-trading bot, not an open risk. <!-- Updated: Validation Session 1 - accepted; teardown after verify -->
- **Migration window write-loss**: the user pauses user activity during cutover (coordinated), and the mandatory `deleteWebhook` buffers any stray updates (Telegram retains ~24h). The "accept the gap" option is removed. Keep the window short and confirm `pending_update_count` drains post-flip.
- **Cron double-fire during cutover**: EventBridge schedule is disabled BEFORE `CRON_MODE=internal` (runbook step 2); the Phase 2 last-push-date guard is the backup. Without both, subscribers get the daily push twice (`lolschedule/cron.go:128-191` is non-idempotent).
- **`updatedAt` type**: store as int64 unix-nanos in Mongo (matches DynamoDB `dynamodb_kv.go:101`) — exact parity, no truncation. No code reads `updatedAt` today (write-only), so this is cheap insurance against a future TTL/sort.
- **Value/encoding mismatch**: migrator writes through `MongoKVStore.Put` (same encoding as the app), not raw `UpdateOne` — guarantees byte-identical `value` and keeps re-runs + CAS correct. Spot-check with a `Get` round-trip.
- **Key validity asymmetry**: migrator runs `validateKey(sk)` before writing and fails loud on any rejected key, so it never writes data the app's read path can't load.
- **IAM for Scan**: dedicated read-only profile/role with exactly `dynamodb:Scan` on the table ARN. No admin profile, no write actions on the source.
- **Large value / 16MB BSON cap**: KV values are tiny JSON; far under limit. No action.
@@ -0,0 +1,139 @@
---
phase: 5
title: "AWS Full Decommission"
status: pending
priority: P1
dependencies: [4]
effort: "S"
---
# Phase 5: AWS Full Decommission
## Overview
After migration + cutover verify (Phase 4), delete **everything** miti99bot ever deployed to AWS, plus verify/clean the legacy pre-AWS services (Cloudflare, GCP — section D). The trap: `sam delete` only removes CloudFormation-managed resources — the GitHub OIDC IAM role, the SSM Parameter Store secrets, and SAM's own S3 deploy bucket were created **manually outside CloudFormation** (`aws/README.md` steps 2-4) and must be deleted separately or they linger (and the secrets keep your bot token/Gemini key sitting in the cloud). GitHub itself is already clean (verified: no stored secrets/keys — AWS deploy used OIDC only).
This phase is informed by a git-history scan of every `template.yaml` version + the `aws/` setup docs, **verified against the live account on 2026-06-27 (admin profile)**. Account `225603493174`, region `ap-southeast-1`.
**Live verification (2026-06-27, read-only):** stack `miti99bot` = `UPDATE_COMPLETE`. The account has exactly TWO CloudFormation stacks (`miti99bot` + `aws-sam-cli-managed-default`) and ONE OIDC provider — so **miti99bot is the sole SAM project AND the sole OIDC consumer**, which makes the shared-resource deletes (OIDC provider, SAM bucket) safe and unconditional. SSM holds exactly the 4 known secrets (no FireAnt/VNAppMob extras). SAM deploy bucket: `aws-sam-cli-managed-default-samclisourcebucket-ctwpsmoxnwvm`.
## Requirements
- Functional: zero miti99bot resources remain in AWS after this phase; Cost Explorer trends to $0.
- Functional: run ONLY after Phase 4 `--verify` passes and the bot is confirmed live on Coolify (DynamoDB holds the only copy of prod data until migrated).
- Non-functional: do NOT delete account-shared resources (OIDC provider, SAM bucket) if other stacks/repos use them — verify first.
- Non-functional: executed by the user with the `admin` profile (this session has no AWS credentials); plan provides the exact commands.
## Architecture — Complete Resource Inventory
### A. CloudFormation-managed → removed by `sam delete --stack-name miti99bot`
From the current `template.yaml`:
- `AWS::DynamoDB::Table``miti99bot-data` (**destroyed — must be migrated first**)
- `AWS::Serverless::Function``miti99bot` + its Function URL + the LWA layer reference
- `AWS::Logs::LogGroup``/aws/lambda/miti99bot` (+ `AWS::Logs::MetricFilter` ColdStartInitDuration)
- `AWS::SQS::Queue``miti99bot-cron-dlq`
- `AWS::IAM::Role``SchedulerExecutionRole` AND the SAM-auto-generated Lambda execution role `miti99bot-BotFunctionRole-*` (both CFN-managed; `sam delete` removes both — no separate action) + the public Function-URL invoke `AWS::Lambda::Permission`s
- `AWS::Scheduler::Schedule``miti99bot-lolschedule-daily-push`
- `AWS::Budgets::Budget``miti99bot-monthly` (only if AlertEmail was set)
Historical (git history shows earlier `template.yaml` used these, since replaced by Scheduler): `AWS::Events::Rule`, `AWS::Events::Connection`, `AWS::Events::ApiDestination`. CloudFormation deleted them when the template changed, so they are NOT orphans — but a post-delete tag sweep (step 6) confirms.
### B. Created manually OUTSIDE CloudFormation → `sam delete` does NOT touch these
From `aws/README.md`:
- **SSM Parameter Store SecureStrings** (`aws/README.md:23-39`, verified — exactly these 4, no extras): `/miti99bot/prod/telegram-bot-token`, `/miti99bot/prod/telegram-webhook-secret`, `/miti99bot/prod/gemini-api-key`, `/miti99bot/prod/cron-shared-secret`. **These hold live secrets — deleting them is the security-relevant step.**
- **IAM role** `github-deploy-miti99bot` + inline policy `miti99bot-deploy` (`aws/README.md:57-73`).
- **IAM OIDC identity provider** `token.actions.githubusercontent.com` (`aws/README.md:43-53`) — verified as the account's ONLY OIDC provider and used solely by miti99bot → **safe to delete.**
- **SAM managed S3 deploy bucket** `aws-sam-cli-managed-default-samclisourcebucket-ctwpsmoxnwvm` + its bootstrap stack `aws-sam-cli-managed-default` — verified miti99bot is the sole SAM project → **safe to delete the bucket + bootstrap stack.**
- **Bootstrap `admin` IAM user + access keys** (`aws/README.md:15`) — user's discretion; out of scope unless they want a full account wipe.
### C. Not deletable / self-expiring (no action)
- Custom CloudWatch metric namespace `miti99bot` (metrics age out; not billable, not deletable).
### D. Legacy services from the pre-AWS lineage (verify + clean — not AWS)
The bot's history is **Cloudflare Worker (original) → [Cloud Run port, abandoned] → AWS → Coolify (this plan)**. Two legacy footprints to confirm:
- **Cloudflare (verified live 2026-06-27, account `miti99` / `7774466151858e13a3c482af5f9ccd6b`):**
- **KV namespaces**: only `claude-status` remains — NO miti99bot KV → the legacy bot KV namespace was already deleted. ✅ no action.
- **D1 databases**: none → legacy `trading_trades` D1 already deleted. ✅ no action.
- **Workers**: 4 exist — `claude-status-webhook`, `rplace`, `miti-loki` (Grafana Loki log-shipper), and `miti-telegram`. All confirmed (user, 2026-06-27) as separate active projects, NOT the legacy miti99bot. `miti-telegram` is a different single-chat notifier the user still uses.
- **Conclusion: Cloudflare is fully clean — NO action.** The legacy miti99bot Worker was already deleted during the May 2026 migration; only its data stores (KV/D1) were ever in scope and both are gone.
- **GCP project `miti99bot-prod`** — almost certainly never created (Cloud Run port superseded before phase-01 ran; no bootstrap artifacts ever existed). Optional certainty check: `gcloud projects list | grep miti99bot` → if a project exists, `gcloud projects delete miti99bot-prod`. Low probability.
## Related Code Files
- Create: `docs/aws-decommission-runbook.md` — the ordered teardown commands below (so it's repeatable + auditable).
- Modify: `.github/workflows/deploy.yml`**delete or disable** (the AWS deploy path is retired; otherwise a push to `main` re-creates the stack). On the `feature/selfhosted` branch this file is replaced by the Coolify flow.
- Modify: `README.md` / `docs/deploy-aws*.md` — mark the AWS deployment path as retired, point to the Coolify guide.
- Keep (do not delete): `aws/` dir + `template.yaml` in git history — useful if AWS is ever revisited; they cost nothing as files.
## Implementation Steps (runbook — user runs with `admin` profile)
Precondition: Phase 4 done — data migrated, `--verify` green, bot live on Coolify, Telegram webhook pointed at Coolify, EventBridge schedule already disabled at cutover.
```sh
AWS_PROFILE=admin; REGION=ap-southeast-1; ACCT=225603493174
# 1. Final safety check — confirm bot is NOT serving from AWS anymore
aws --profile $AWS_PROFILE cloudformation describe-stacks --stack-name miti99bot \
--query "Stacks[0].StackStatus" # exists → about to be deleted
# 2. Delete the CloudFormation stack (DynamoDB, Lambda, FunctionUrl, Logs, SQS, Scheduler, IAM role, Budget)
aws --profile $AWS_PROFILE sam delete --stack-name miti99bot --region $REGION --no-prompts
# (or: aws cloudformation delete-stack --stack-name miti99bot ; then wait)
aws --profile $AWS_PROFILE cloudformation wait stack-delete-complete --stack-name miti99bot
# 3. Delete SSM secrets (NOT managed by CFN)
aws --profile $AWS_PROFILE ssm get-parameters-by-path --path /miti99bot --recursive \
--query "Parameters[].Name" --output text # list what exists first
for P in telegram-bot-token telegram-webhook-secret gemini-api-key cron-shared-secret; do
aws --profile $AWS_PROFILE ssm delete-parameter --name /miti99bot/prod/$P
done
# delete any extra /miti99bot/* the list in step 3 revealed
# 4. Delete the GitHub deploy IAM role (inline policy first, then role)
aws --profile $AWS_PROFILE iam delete-role-policy \
--role-name github-deploy-miti99bot --policy-name miti99bot-deploy
aws --profile $AWS_PROFILE iam delete-role --role-name github-deploy-miti99bot
# 5. OIDC provider — verified sole consumer = miti99bot, safe to delete.
# (Re-confirm it's still the only one before deleting, in case the account changed.)
aws --profile $AWS_PROFILE iam list-open-id-connect-providers
aws --profile $AWS_PROFILE iam delete-open-id-connect-provider \
--open-id-connect-provider-arn arn:aws:iam::$ACCT:oidc-provider/token.actions.githubusercontent.com
# 6. SAM S3 deploy bucket + bootstrap stack — verified miti99bot is the sole SAM project.
# (Re-confirm only miti99bot + aws-sam-cli-managed-default stacks exist first.)
aws --profile $AWS_PROFILE cloudformation list-stacks \
--query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].StackName" --output text
aws --profile $AWS_PROFILE s3 rb \
s3://aws-sam-cli-managed-default-samclisourcebucket-ctwpsmoxnwvm --force
aws --profile $AWS_PROFILE cloudformation delete-stack --stack-name aws-sam-cli-managed-default
# 7. Verify nothing tagged app=miti99bot remains
aws --profile $AWS_PROFILE resourcegroupstaggingapi get-resources \
--tag-filters Key=app,Values=miti99bot --region $REGION
aws --profile $AWS_PROFILE cloudformation list-stacks \
--query "StackSummaries[?contains(StackName,'miti99bot')].[StackName,StackStatus]" --output table
```
Then in the repo (on `feature/selfhosted`): remove/disable `.github/workflows/deploy.yml` and mark AWS docs retired.
## Success Criteria
- [ ] `cloudformation describe-stacks --stack-name miti99bot` → does not exist (DELETE_COMPLETE).
- [ ] No `/miti99bot/*` SSM parameters remain (secrets purged from cloud).
- [ ] `github-deploy-miti99bot` role gone; OIDC provider gone OR confirmed still needed by another project.
- [ ] SAM bucket emptied/deleted OR `miti99bot/` prefix cleared (and rationale recorded).
- [ ] `resourcegroupstaggingapi get-resources` for `app=miti99bot` returns empty.
- [ ] `.github/workflows/deploy.yml` removed/disabled so `main` pushes no longer recreate the stack.
- [ ] Cost Explorer shows $0 for the following billing period.
- [x] Cloudflare verified clean (2026-06-27): KV + D1 already gone; all 4 Workers confirmed as separate active projects (no legacy bot remnant). No action.
- [ ] GCP checked: no `miti99bot-prod` project exists (or deleted).
- [ ] GitHub confirmed clean: no AWS secrets/keys stored; `deploy-aws.yml` removed (only OIDC was used, nothing to revoke beyond the role deleted above).
## Risk Assessment
- **Deleting before migration completes (Critical)**: `sam delete` destroys `miti99bot-data` (DynamoDB). Mitigation: hard dependency on Phase 4 `--verify`; runbook step 1 is an explicit precondition check. Never run Phase 5 standalone.
- **Deleting shared account resources**: verified on 2026-06-27 that miti99bot is the sole OIDC consumer and sole SAM project, so steps 5-6 are safe. Mitigation: the runbook re-confirms (list-first) before each delete in case the account changes before you run it; if a new stack/provider appears, leave the shared resource (it costs ~nothing).
- **Re-creation by CI**: a `main` push with `deploy.yml` still active would rebuild the whole stack post-teardown. Mitigation: disable/delete the workflow as part of this phase (and it's already superseded on `feature/selfhosted`).
- **Lost rollback**: per the Phase 4 validated decision, teardown ends the AWS rollback option. Mitigation: that was explicitly accepted; optionally keep the stack a few days before running step 2.
- **Secret hygiene**: rotate the Telegram bot token + Gemini key after teardown if you want defense-in-depth (they lived in SSM and CloudWatch under the accepted trade-offs). Optional.
@@ -0,0 +1,124 @@
---
title: "Self-host miti99bot on Coolify with MongoDB Atlas"
description: "Add a MongoDB Atlas storage backend + in-process cron, containerize for Coolify docker-compose, migrate existing DynamoDB data."
status: pending
priority: P2
branch: "feature/selfhosted"
tags: [selfhost, coolify, mongodb, migration]
blockedBy: []
blocks: []
created: "2026-06-27T12:01:53.894Z"
createdBy: "ck:plan"
source: skill
---
# Self-host miti99bot on Coolify with MongoDB Atlas
## Overview
Run miti99bot as a long-lived container on Coolify (docker-compose) instead of AWS Lambda, using MongoDB Atlas (`MONGO_URL` + `MONGO_DATABASE`) instead of DynamoDB. Existing DynamoDB data is migrated into Atlas. At the **code** level this is additive — a 4th KV backend + a self-host run mode; the DynamoDB/Lambda code path is NOT ripped out (kept for portability and to run the migrator). At the **infrastructure** level, the deployed AWS stack is fully decommissioned after a verified cutover (Phase 5, validated decision).
Why this is low-risk: storage is already a pluggable `KVProvider` interface with 3 backends (`memory`, `firestore`, `dynamodb`); adding `mongodb` follows the exact `firestore` collection-per-module pattern. Secrets already fall back to plain env vars when `*_PARAMETER_NAME` is unset, so Coolify env vars need no code change. The only genuine gap is cron: EventBridge Scheduler triggers `/cron/{name}` today, which does not exist off-AWS.
## Phases
| Phase | Name | Status |
|-------|------|--------|
| 1 | [MongoDB Storage Provider](./phase-01-mongodb-storage-provider.md) | Pending |
| 2 | [In-Process Cron Scheduler](./phase-02-in-process-cron-scheduler.md) | Pending |
| 3 | [Containerize and Coolify Deploy](./phase-03-containerize-and-coolify-deploy.md) | Pending |
| 4 | [Data Migration and Cutover](./phase-04-data-migration-and-cutover.md) | Pending |
| 5 | [AWS Full Decommission](./phase-05-aws-decommission.md) | Pending |
## Dependencies
- Phase 2 is independent of Phase 1 (cron touches no storage).
- Phase 3 depends on 1 + 2 (container must boot with mongo + internal cron).
- Phase 4 depends on Phase 1 (Mongo document schema must be final before copying data) and is the last step before flipping the Telegram webhook to the Coolify URL.
- Phase 5 (AWS decommission) depends on Phase 4 `--verify` passing — it destroys DynamoDB, so it runs only after data is migrated and the bot is confirmed live on Coolify.
Suggested order: 1 → 2 → 3 → 4 → 5. Phases 1 and 2 can be done in parallel by separate developers (disjoint files).
## Architecture Summary
```
BEFORE (AWS) AFTER (Coolify self-host)
Telegram ──webhook──> Lambda Function URL Telegram ──webhook──> Coolify domain
EventBridge Scheduler ──> /cron/{name} in-process scheduler ──> cron.Handler
DynamoDB (pk=module, sk=key) MongoDB Atlas (db / collection-per-module)
SSM Parameter Store secrets Coolify env vars (plain)
```
Same Go binary (`cmd/server`), same HTTP server on `:8080`, same module framework. Backend + cron + secret source are all selected by env vars at startup.
## Acceptance Criteria
- [ ] `KV_PROVIDER=mongodb MONGO_URL=… MONGO_DATABASE=…` boots and serves `/webhook` with persistent storage.
- [ ] `make test` green; new mongo provider has parity tests with the firestore/dynamodb suites.
- [ ] Self-hosted container fires the lolschedule daily push at 01:00 UTC (08:00 ICT) without EventBridge.
- [ ] `docker compose up` (Coolify) brings the bot live behind a public HTTPS domain; `/` health check passes.
- [ ] All existing prod DynamoDB items present in Atlas with identical values; migration is idempotent + verifiable by per-module counts.
- [ ] After cutover, ALL miti99bot AWS resources are deleted — CloudFormation stack AND the manually-created SSM secrets, GitHub OIDC role, and SAM S3 bucket; `app=miti99bot` tag sweep is empty and Cost Explorer trends to $0.
- [ ] `.github/workflows/deploy.yml` is removed/disabled so `main` pushes no longer recreate the AWS stack.
## Red Team Review
### Session — 2026-06-27
**Findings:** 15 (15 accepted, 0 rejected) — 3 reviewers (security/secrets, assumptions, failure-modes), all findings carried `file:line` evidence.
**Severity breakdown:** 2 Critical, 6 High, 7 Medium.
| # | Finding | Severity | Disposition | Applied To |
|---|---------|----------|-------------|------------|
| 1 | nil-expected CAS is a live first-write path; `$exists:false` upsert is wrong primitive → use `InsertOne` + unique-`_id` + blocking concurrent test | Critical | Accept | Phase 1 |
| 2 | Rollback loses ALL post-cutover writes; no reverse path → state true RPO / reverse migrator | Critical | Accept | Phase 4 |
| 3 | `buildProvider` log line would leak `MONGO_URL` creds → log only `database`, never URL | High | Accept | Phase 1 |
| 4 | Cron dispatcher symbol misdescribed (`DispatchScheduled(ctx,name,reg)`, no timeout/recover/log) → scheduler adds own | High | Accept | Phase 2 |
| 5 | Dockerfile omits `gitSHA``deploynotify` silently dead → inject build arg or document disabled | High | Accept | Phase 3 |
| 6 | Cron double-fire (EventBridge-live + rolling deploy; push non-idempotent) → disable schedule before internal cron + last-push-date guard | High | Accept | Phase 2, Phase 4 |
| 7 | Atlas `0.0.0.0/0` = public DB surface vs project boundary → egress-IP allowlist + least-priv user default | High | Accept | Phase 3 |
| 8 | Cutover write-loss gap; ambiguous webhook-unset → mandatory `deleteWebhook`→migrate→verify→`setWebhook` + `pending_update_count` | High | Accept | Phase 4 |
| 9 | Migrator raw `UpdateOne` diverges from `Put` encoding → write through `Put`; store `updatedAt` int64 | Medium | Accept | Phase 1, Phase 4 |
| 10 | Stray `*_PARAMETER_NAME` fatal off-AWS → `.env.example` lists all six "leave UNSET" + boot criterion | Medium | Accept | Phase 3 |
| 11 | `/` is plain text not JSON; `-healthcheck` flag doesn't exist → Coolify HTTP monitor, drop bogus CMD | Medium | Accept | Phase 3 |
| 12 | `/cron/` redundant double-trigger when internal → leave `CRON_SHARED_SECRET` unset (route 404) | Medium | Accept | Phase 3 |
| 13 | Migrator IAM under-specified / admin profile → exact least-priv `Scan`-only + Scan-based verify, read-only profile | Medium | Accept | Phase 4 |
| 14 | No Mongo reconnect/health story → document auto-reconnect + pool opts; DB-aware healthcheck or accept trade-off | Medium | Accept | Phase 3 |
| 15 | `validateKey` rejects `/` but migrator bypasses it → migrator `validateKey` each `sk`, fail loud | Medium | Accept | Phase 4 |
Verified non-issues (no change needed): `List()` `$gte/$lt` range avoids regex injection (sound, mirrors firestore); secrets fallback to plain env is verified correct; `.env` already gitignored.
### Whole-Plan Consistency Sweep
Re-read all phase files after applying findings. Reconciled:
- `updatedAt` storage type now consistently **int64 nanos** in Phase 1 doc-shape and Phase 4 migrator/risk (was "BSON datetime").
- CAS absent-case is **`InsertOne`** everywhere (Phase 1 architecture + Phase 4 encoding note); the `$exists:false` upsert is removed.
- Migrator writes **through `MongoKVStore.Put`** in Phase 4 architecture, risk, and success criteria (no raw `UpdateOne`).
- Health endpoint described as **plain text `miti99bot ok`** in Phase 3 requirements, architecture, and risk (was "JSON"); bogus `-healthcheck` CMD removed from the compose snippet.
- Cron dispatcher named **`DispatchScheduled`** in Phase 2 with scheduler-local timeout/recover.
- Cutover ordering (disable EventBridge → migrate → `CRON_MODE=internal``setWebhook`) consistent across Phase 2 risk and Phase 4 runbook.
No unresolved contradictions remain.
## Validation Log
### Session — 2026-06-27
Verification pass skipped: `## Red Team Review` already carries full `file:line` evidence and no `[UNVERIFIED]` tags remain. Interview resolved all 8 open questions.
| # | Question | Decision | Affects |
|---|----------|----------|---------|
| 1 | Rollback RPO | **Simple short-window cutover; NO reverse migrator.** User coordinates users to pause around cutover, so no writes occur mid-migration. | Phase 4 |
| 2 | Decommission AWS | **Tear down the SAM stack after `--verify` passes.** No long-term fallback. EventBridge schedule still disabled as an explicit pre-cutover step. Stop the GitHub Actions deploy workflow. | Phase 4 |
| 3 | Atlas network access | **`0.0.0.0/0` (no stable Coolify egress IP).** Accepted trade-off: mandatory strong unique password + least-privilege DB user (`readWrite` on one DB, not admin). Documented as a knowing widening vs DynamoDB's IAM-gated posture. | Phase 3 |
| 4 | deploynotify | **Keep it** — inject `gitSHA` via Dockerfile `ARG GIT_SHA` + Coolify build-arg. | Phase 3 |
| 5 | `/cron/` HTTP route | **Disable in prod** — leave `CRON_SHARED_SECRET` unset (route 404s); internal scheduler is the sole trigger. | Phase 2, Phase 3 |
| 6 | Mongo driver | `go.mongodb.org/mongo-driver/v2` (current stable); `robfig/cron/v3` for the scheduler. | Phase 1, Phase 2 |
| 7 | Atlas tier | Free **M0** (512 MB) — sufficient for the tiny paper-trading KV. | Phase 3 |
| 8 | `updatedAt` reader | Confirmed write-only today; store as int64 for cheap parity. No TTL/sort planned. | Phase 1, Phase 4 |
### Whole-Plan Consistency Sweep (post-validation)
- Phase 4 rollback/teardown rewritten: AWS torn down after verify; reverse-migrator option removed; rollback framed as "coordinate users, short window" not "keep Lambda N days."
- Phase 3 Atlas networking: `0.0.0.0/0` is now the chosen path (was "egress-IP default") with password + least-priv user as hard requirements.
- deploynotify `gitSHA` injection and `/cron/` disabled (`CRON_SHARED_SECRET` unset) were already the recommended defaults in Phases 2/3 — now confirmed, no contradiction.
- No unresolved contradictions remain.
## Open Questions
None — all resolved in the Validation Log above.
@@ -0,0 +1,166 @@
# Research Report: Skip S3 For miti99bot Deploys
Timestamp: 2026-06-27 11:43 UTC
## Executive Summary
Official AWS docs do not show a supported SAM/CloudFormation ZIP deploy path that avoids S3 for this Go Lambda. SAM deploy/package uploads local ZIP artifacts to S3; CloudFormation ZIP functions expect S3 object location, except inline Node.js/Python source.
Official no-S3 path exists only outside SAM/CloudFormation stack code management: `aws lambda update-function-code --zip-file fileb://...`. This repo's current ZIP is ~7.8 MB, below Lambda direct upload limit. Use it only for code-only deploys/hotfixes, or accept drift from CloudFormation.
Best zero-cost-risk answer: keep SAM for infra, but use direct Lambda code upload for routine code deploys if S3 must be eliminated. Run SAM only when infra/config changes, or replace SAM with imperative AWS CLI provisioning. No perfect official "SAM but no S3" option found.
## Codebase Context
- Project: Go 1.25 Lambda app, `provided.al2023`, ARM64.
- Deploy: `.github/workflows/deploy.yml` runs `make build-lambda` then `sam deploy --template-file template.yaml`.
- SAM config: `samconfig.toml` has `resolve_s3 = true` and `s3_prefix = "miti99bot"`.
- CloudFormation code: `template.yaml` uses `CodeUri: build/lambda/`.
- Current artifact: `build/lambda/function.zip` about 7.8 MB, direct upload compatible.
## Official Findings
### 1. SAM ZIP deploy requires S3 for local artifacts
AWS SAM package docs:
- `sam deploy` implicitly performs package.
- `--resolve-s3` creates an S3 bucket for packaging.
- If artifact > 51,200 bytes, `--s3-bucket` or `--resolve-s3` required.
Source: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-package.html
AWS SAM tutorial:
- SAM deploy uploads application files to S3.
- SAM creates an S3 bucket and uploads `.aws-sam` directory.
Source: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html
### 2. CloudFormation ZIP Lambda uses S3, except inline Node/Python
CloudFormation `AWS::Lambda::Function Code`:
- ZIP package can specify S3 object.
- Inline `ZipFile` only for Node.js and Python, max 4 MB.
- Container images use ECR.
Sources:
- https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-lambda-function-code.html
- https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-lambda-function.html
This repo is compiled Go (`provided.al2023`), so inline code is not viable.
### 3. Lambda direct ZIP upload skips S3
Lambda Go ZIP docs:
- For ZIP smaller than 50 MB, AWS CLI can upload from local file.
- Larger files must use S3.
AWS CLI docs:
- `aws lambda update-function-code --zip-file fileb://my-function.zip` updates function code directly.
Sources:
- https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html
- https://docs.aws.amazon.com/cli/latest/reference/lambda/update-function-code.html
- https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-deployment.html
### 4. ECR avoids S3 but is not better for zero-cost private deploys
SAM supports image package deployments via ECR. ECR private repository free tier is limited: 500 MB/month for first year for new ECR customers. Public repositories have larger always-free storage, but public bot images may not be acceptable.
Source: https://aws.amazon.com/ecr/pricing/
## Brainstormed Options
### Option A: Keep SAM + S3, add lifecycle cleanup
Pros:
- Official canonical SAM path.
- CloudFormation remains source of truth.
- Lowest operational risk.
Cons:
- Does not satisfy "skip S3".
- Still has S3 bucket, requests, storage, and possible artifact accumulation.
Use when:
- Accept S3 free tier / credits and control storage lifecycle.
### Option B: Direct Lambda code upload for code-only deploys
Pros:
- Official AWS Lambda path.
- No S3 artifact bucket needed for code updates.
- Current repo artifact fits 50 MB direct upload limit.
- Simple for this single-function app.
Cons:
- Bypasses CloudFormation code state.
- Next SAM deploy can overwrite direct code.
- Need separate logic for infra/config changes.
- Must keep env, role, Function URL, EventBridge, DynamoDB managed elsewhere.
Use when:
- Non-negotiable: no S3 deploy artifact bucket.
- App remains single Lambda and ZIP stays < 50 MB.
### Option C: Replace ZIP Lambda with container image in ECR
Pros:
- Official SAM/CloudFormation path without S3 ZIP artifacts.
- CloudFormation can track `ImageUri`.
Cons:
- Uses ECR, another billable storage service.
- Private ECR free tier is only first-year/limited; image layers can grow fast.
- More build complexity than current Go binary ZIP.
Use when:
- Container image needed for runtime/dependency reasons. Not true now.
### Option D: Fully imperative AWS CLI provisioning, no SAM
Pros:
- Can create/update Lambda with direct local ZIP upload.
- Avoids SAM-managed S3.
Cons:
- You reimplement infra orchestration: IAM roles, Function URL, DynamoDB, EventBridge, logs, budgets, permissions.
- Higher drift risk and maintenance burden.
- Less rollback safety.
Use when:
- Absolute no-S3 policy outranks IaC simplicity.
## Recommendation
No official recommended "SAM deploy ZIP without S3" path exists for this Go Lambda. If S3 must be zero, use:
1. SAM/CloudFormation only for first infra provisioning and infra changes.
2. Direct `aws lambda update-function-code --zip-file` for code-only deploys.
3. Never run SAM for routine code-only changes unless willing to use S3 again.
4. Add a guard/check that fails direct deploy if ZIP >= 50 MB.
This is a compromise, not pure IaC.
## Candidate Command
```sh
make build-lambda
zip -j build/lambda/function.zip build/lambda/bootstrap
aws lambda update-function-code \
--function-name miti99bot \
--zip-file fileb://build/lambda/function.zip
aws lambda wait function-updated --function-name miti99bot
```
## Kill Criteria
- ZIP reaches 50 MB.
- Need code + env/config changes in one atomic deploy.
- Need CloudFormation to be sole source of truth.
- Need Lambda versions/aliases/code signing workflow through IaC.
## Unresolved Questions
- Do you want zero S3 resources, or just zero S3 cost risk?
- Are you okay with CloudFormation drift for code-only deploys?
- Should direct upload become canonical CI deploy, or only emergency/manual path?
@@ -0,0 +1,133 @@
# Research Report: Whole-Project Free-Tier Audit + S3 Elimination
Timestamp: 2026-06-27 18:49 ICT
Supersedes/extends: `260627-1143-sam-no-s3-deploy-research.md` (S3-only research). This adds a full per-service free-tier audit and a verified 2026 free-tier picture.
## Executive Summary
S3 is the **only** AWS resource in this stack with no permanent free tier. Everything else (Lambda, DynamoDB on-demand, EventBridge Scheduler, SQS, CloudWatch Logs, X-Ray, SSM Standard, Budgets) is **always-free** within monthly caps the bot will never approach. The user's instinct is correct: keep SAM-on-every-push and you keep an S3 line item forever (microscopic, but nonzero).
For a compiled Go Lambda there is no IaC path that avoids both S3 and ECR — SAM/CloudFormation must stage the ZIP somewhere. The only zero-S3 option is to deliver code with `aws lambda update-function-code --zip-file fileb://...` and run SAM only when infra changes. Recommended below.
Magnitude check (honesty): the SAM bucket for this app costs ~$0.002/month storage + sub-cent request charges per deploy after the free window. Real, but trivial. The recommendation assumes the stated hard rule "zero cost line items" — if the rule is actually "stay within free tier / no meaningful cost," keeping S3 with a lifecycle rule is also fine.
## Free-Tier Audit (every resource in `template.yaml`)
| Resource | Service | Free-tier status | Verdict |
|---|---|---|---|
| `BotFunction` | Lambda (`provided.al2023`, arm64, 256MB) | Always-free: 1M req/mo + 400k GB-s/mo | ✅ Free forever |
| `BotTable` | DynamoDB `PAY_PER_REQUEST` | Always-free: 25GB + 2.5M RRU + 1M WRU/mo | ✅ Free forever |
| `LolscheduleDailyPushSchedule` | EventBridge Scheduler | Always-free: 14M invocations/mo | ✅ Free forever |
| `CronDLQ` | SQS | Always-free: 1M requests/mo | ✅ Free forever |
| `BotFunctionLogGroup` (7-day retention) | CloudWatch Logs | Always-free: 5GB ingest + 5GB store/mo | ✅ Free (retention caps growth) |
| `ColdStartMetricFilter``ColdStartInitDuration` | CloudWatch custom metric | Free: 10 custom metrics. This is 1. | ✅ Free (watch the count) |
| `Tracing: Active` | X-Ray | Always-free: 100k traces recorded/mo | ✅ Free (bot volume ≪ cap) |
| SSM `GetParameter`/`GetParameters` (cold-start secret load) | SSM Parameter Store | Standard tier + standard throughput = free | ✅ Free (do NOT switch to Advanced/High-throughput) |
| `MonthlyBudget` | AWS Budgets | First 2 budgets free/account. This is 1. | ✅ Free |
| **SAM deploy bucket** (`aws-sam-cli-managed-default-*`) | **S3** | **No always-free tier.** 12-mo (legacy) / 6-mo credits (new acct) only | ⚠️ **Only paid line item** |
| Lambda Function URL (`AuthType: NONE`) | Lambda | No extra charge (counts as Lambda req) | ✅ Free |
| Lambda Web Adapter layer | Lambda layer | No charge | ✅ Free |
External (off-AWS, no AWS bill): Telegram Bot API, FireAnt, GoldPrice.org, ExchangeRate-API, VNAppMob, Binance/Coinbase/CoinGecko, Gemini API. Gemini has its own free quota — out of scope for AWS cost.
### Watch-items (free now, could tip over if scaled)
- **X-Ray `Tracing: Active`** — free to 100k traces/mo. A personal bot is nowhere near. But it is *not* always-free above the cap. If you never plan to inspect traces, you can set `Tracing: PassThrough` (or remove) to drop the dependency entirely. Low priority.
- **Custom metrics** — only 1 of 10 free slots used. Adding ~10 more metric filters would start billing $0.30/metric/mo.
- **CloudWatch Logs** — 7-day retention is set, which is what keeps ingest/storage under 5GB. Don't remove it.
- **SSM throughput** — secrets load once per cold start at Standard throughput (free). Don't enable "higher throughput" (paid) or convert params to Advanced tier ($0.05/param/mo).
## S3: the one thing that isn't free
Verified 2026: AWS overhauled the Free Tier on 2025-07-15. S3 never had an always-free tier; the 5GB/12-month trial (legacy accounts) and the new $100$200 6-month credit plan (accounts created on/after 2025-07-15) are both **time-boxed**. After the window, S3 Standard = $0.023/GB-mo + per-request charges. There is no S3 storage allotment that survives the trial.
Why SAM forces S3 here:
- `sam deploy` packages the local ZIP and uploads it to S3 (`resolve_s3=true`, `s3_prefix="miti99bot"` in `samconfig.toml`). Artifacts > 51,200 bytes *must* go to S3; this binary is ~7.8MB.
- CloudFormation `AWS::Lambda::Function` ZIP code requires an S3 object. Inline code is Node/Python-only — not viable for a Go binary.
- ECR avoids S3 but is itself billable storage with only a time-boxed free tier — strictly worse for this single small ZIP.
So: with pure SAM/CFN, a compiled Go Lambda cannot avoid both S3 and ECR. To reach zero S3 you must move *code delivery* out of CloudFormation.
## Brainstormed Options (zero-S3)
### Option A — Keep SAM + S3, add 1-day lifecycle expiry
Add an S3 lifecycle rule so artifacts expire after 1 day; storage trends to ~0.
- Pros: zero code change to deploy flow; CFN stays sole source of truth; cost ≈ a few cents/year.
- Cons: still a nonzero S3 line item (PUT/GET per deploy). **Fails the strict "zero line items" rule.**
### Option B — Direct `aws lambda update-function-code` for code; SAM only for infra (RECOMMENDED)
Routine pushes update Lambda code directly from a local ZIP (no S3). Run SAM only when `template.yaml` changes.
- Pros: official Lambda path; zero S3 for the 99% code-only case; trivially simple for one function; ZIP (~7.8MB) is far under the 50MB direct-upload limit.
- Cons: code-only deploys bypass CloudFormation's view of code (benign drift for a single function); first-ever create and infra changes still touch S3 once.
- Fit: best match for this repo. Single Lambda, small ZIP, free-tier-hard requirement.
### Option C — Container image in ECR
- Rejected: trades S3 for ECR, another time-boxed-free storage service. More build complexity, no benefit here.
### Option D — Fully imperative provisioning, no SAM at all
- Rejected: you'd reimplement IAM, Function URL, DynamoDB, Scheduler, SQS, Logs, Budgets and lose rollback safety to delete one sub-penny S3 dependency. Not worth it.
## Recommendation
Adopt **Option B**. Concretely:
1. Keep `template.yaml` as the infra source of truth.
2. Change `.github/workflows/deploy.yml`: on push to `main`, deploy code only via direct upload (no S3). Run `sam deploy` only when `template.yaml` (or params) changed — detect via `git diff`.
3. First-time creation and infra changes use SAM (one-time S3 touch — accept it, or delete the managed bucket afterward).
4. Guard: fail the direct deploy if the ZIP ≥ 50MB (forces a rethink before silently breaking).
This yields zero S3 for every code-only push, which is the overwhelming majority of deploys.
### Candidate CI deploy step (code-only path)
```sh
make build-lambda
zip -j build/lambda/function.zip build/lambda/bootstrap
SIZE=$(stat -c%s build/lambda/function.zip)
[ "$SIZE" -lt 52428800 ] || { echo "ZIP >= 50MB; use S3/SAM"; exit 1; }
aws lambda update-function-code \
--function-name miti99bot \
--zip-file fileb://build/lambda/function.zip \
--publish
aws lambda wait function-updated --function-name miti99bot
```
Env vars / Function URL / Scheduler / DynamoDB / role stay managed by the SAM stack — direct code upload doesn't touch them.
### CI routing sketch (only run SAM when infra changed)
```sh
if git diff --quiet HEAD~1 -- template.yaml samconfig.toml; then
# code-only: direct upload (no S3)
else
# infra change: sam deploy (touches S3 once)
fi
```
Note: the OIDC deploy role (`github-deploy-miti99bot`) needs `lambda:UpdateFunctionCode` + `lambda:GetFunction` for the direct path; it likely already has broad Lambda perms via SAM, but verify before switching CI.
## Bottom line
- The project is **already free** on every service except S3, and S3's real cost is fractions of a cent — but it is a permanent nonzero line item once the free window ends.
- The clean way to hit truly $0 is Option B: direct Lambda code upload for routine deploys, SAM reserved for infra.
- No other resource needs changing to stay free; just don't add custom metrics past 10, keep log retention, and keep SSM at Standard tier.
## Unresolved Questions
1. Strict rule check: do you want **zero S3 resources** (→ Option B), or just **zero meaningful S3 cost** (→ Option A lifecycle rule is simpler)? Memory says hard "no line items," so Option B assumed.
2. Is the AWS account legacy (pre-2025-07-15, 12-mo trial) or new (6-mo credits)? Determines whether S3 is already billing today or still inside a free window.
3. OK with benign CloudFormation drift on Lambda code for code-only deploys?
4. Should direct-upload become the canonical CI path, or only a manual/emergency path with SAM staying primary?
## Sources
- [AWS Free Tier in 2026 — what changed / always-free services](https://infratally.com/articles/aws-free-tier-2026/)
- [What's New in AWS Free Tier (2025)](https://dev.to/aws-builders/whats-new-in-aws-free-tier-2025-2ba5)
- [AWS Lambda pricing / free tier](https://aws.amazon.com/pm/lambda/)
- [Amazon CloudWatch pricing (custom metrics, free tier)](https://aws.amazon.com/cloudwatch/pricing/)
- [AWS X-Ray pricing (100k traces/mo free)](https://aws.amazon.com/xray/pricing/)
- [Amazon S3 pricing (no always-free storage tier)](https://aws.amazon.com/s3/pricing/)
- [SAM package — S3 required for artifacts > 51,200 bytes](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-package.html)
- [CloudFormation Lambda Function Code — ZIP requires S3, inline is Node/Python only](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-lambda-function-code.html)
- [lambda update-function-code (direct ZIP upload)](https://docs.aws.amazon.com/cli/latest/reference/lambda/update-function-code.html)