mirror of
https://github.com/tiennm99/miti99bot.git
synced 2026-08-01 02:21:27 +00:00
feat(deploy): wire lolschedule cron via EventBridge Rule + ApiDestination
CloudFormation's AWS::Scheduler::Schedule Target schema has no property
for HTTPS universal invocation (URL, method, headers) — confirmed
against AWS docs. Switch to the legacy EventBridge Rule path which
supports HTTP targets natively via ApiDestination:
- AWS::Events::Connection: API_KEY auth, presents X-Cron-Token header.
ApiKeyValue stored in EventBridge service-linked secret on stack
update (no per-invoke SSM fetch, AWS-managed secret fees).
- AWS::Events::ApiDestination: POST to ${FunctionUrl}cron/lolschedule_daily_push.
- AWS::Events::Rule: cron(0 1 * * ? *) — daily 01:00 UTC / 08:00 ICT.
Targets ApiDestination with retry x2, 600s max age, DLQ to CronDLQ.
- EventBridgeInvokeRole replaces SchedulerExecutionRole (events.amazonaws.com
principal, events:InvokeApiDestination scoped to this destination only).
NoEcho CronSharedSecret CFN parameter restored; GHA fetches the SSM
SecureString and passes via --parameter-overrides so the value never
appears in template source or stack events.
Free-tier preserved: 1 invocation/day, well under EventBridge Rules +
ApiDestinations free quotas.
This commit is contained in:
@@ -43,17 +43,25 @@ jobs:
|
||||
- name: SAM deploy
|
||||
env:
|
||||
ALERT_EMAIL: ${{ secrets.ALERT_EMAIL }}
|
||||
STACK_ENV: prod
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# EventBridge Connection consumes ApiKeyValue at stack-update time
|
||||
# and stores it in a service-linked secret. SSM holds the canonical
|
||||
# value; fetch here and pass as a NoEcho CFN parameter so it never
|
||||
# appears in template source or stack events.
|
||||
CRON_SECRET=$(aws ssm get-parameter \
|
||||
--name "/miti99bot/${STACK_ENV}/cron-shared-secret" \
|
||||
--with-decryption --query Parameter.Value --output text)
|
||||
echo "::add-mask::$CRON_SECRET"
|
||||
OVERRIDES="CronSharedSecret=$CRON_SECRET"
|
||||
if [ -n "$ALERT_EMAIL" ]; then
|
||||
sam deploy --template-file template.yaml \
|
||||
--no-confirm-changeset \
|
||||
--no-fail-on-empty-changeset \
|
||||
--parameter-overrides "AlertEmail=$ALERT_EMAIL"
|
||||
else
|
||||
sam deploy --template-file template.yaml \
|
||||
--no-confirm-changeset \
|
||||
--no-fail-on-empty-changeset
|
||||
OVERRIDES="$OVERRIDES AlertEmail=$ALERT_EMAIL"
|
||||
fi
|
||||
sam deploy --template-file template.yaml \
|
||||
--no-confirm-changeset \
|
||||
--no-fail-on-empty-changeset \
|
||||
--parameter-overrides "$OVERRIDES"
|
||||
|
||||
- name: Smoke test (Function URL responds)
|
||||
run: |
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
type: code-review
|
||||
date: 2026-05-18
|
||||
slug: recent-changes-cron-regression
|
||||
status: final
|
||||
related:
|
||||
- plans/reports/brainstorm-260517-1411-eventbridge-schedule-fix.md
|
||||
---
|
||||
|
||||
# Audit — Recent commits vs `lolschedule_daily_push` cron
|
||||
|
||||
## TL;DR
|
||||
|
||||
**Verdict: No regression. The cron was never wired in the first place.**
|
||||
|
||||
The brainstorm at `plans/reports/brainstorm-260517-1411-eventbridge-schedule-fix.md` is correct: the `AWS::Scheduler::Schedule` resource did not exist on `main` until commit `c70b9d0` (2026-05-18 11:46 +0700). Every commit in the user's "suspicion window" (`1f5f304` → `8e7fdce`) ran against a stack with zero EventBridge schedules. There was nothing for them to break.
|
||||
|
||||
Two commits *after* the suspicion window (`c70b9d0`, `585d996`) execute the brainstorm. They are the path forward — not the cause of an outage.
|
||||
|
||||
**Recommendation: PROCEED-WITH-FIX. Do not roll back.** The fix is already on `main` (`c70b9d0` + `585d996`). What remains is deploy + post-deploy verification per the brainstorm's "Next steps".
|
||||
|
||||
---
|
||||
|
||||
## Per-commit verdicts
|
||||
|
||||
| Hash | Subject | Verdict |
|
||||
|---|---|---|
|
||||
| `585d996` | fix(deploy): pass cron secret via CFN parameter, not ssm-secure resolve | INNOCENT (corrects c70b9d0; required for `sam deploy` to succeed) |
|
||||
| `c70b9d0` | feat(deploy): wire EventBridge schedule for lolschedule daily push | INNOCENT (this is the fix — adds the missing schedule) |
|
||||
| `8e7fdce` | feat(misc): /trongtruonghop disclaimer command | INNOCENT (touches `internal/modules/misc` only) |
|
||||
| `3402ca9` | refactor(plans): code-review audit reports | INNOCENT (docs-only, `plans/reports/*`) |
|
||||
| `a8ed67a` | refactor: audit-driven hygiene pass | INNOCENT — see analysis below |
|
||||
| `3a12615` | fix(reply): forward `message_thread_id` | INNOCENT (no cron-path edits; `lolschedule/handlers.go` change is `/lolschedule` command, not push) |
|
||||
| `3235ee8` | ci: bump golangci-lint-action v9 | INNOCENT (CI lint runner only) |
|
||||
| `1e794d1` | ci: bump actions to Node 24-native | INNOCENT (CI runner versions only) |
|
||||
| `4c81dd6` | fix(dispatcher): match `/cmd@botname` | INNOCENT — see analysis below |
|
||||
| `1f5f304` | ci(deploy): auto-register Telegram webhook + commands after deploy | INNOCENT (post-deploy curl to Telegram setWebhook/setMyCommands; does not touch Scheduler) |
|
||||
|
||||
---
|
||||
|
||||
## Closer look at the two commits with the largest blast radius
|
||||
|
||||
### `a8ed67a` — audit-driven hygiene pass
|
||||
|
||||
This is the only commit in the window that touches cron-relevant Go files (`cmd/server/main.go`, `internal/server/router.go`, `internal/server/log_middleware.go`, `internal/modules/lolschedule/cron.go`). Every change reviewed:
|
||||
|
||||
- **`cmd/server/main.go`: WriteTimeout 6m → 75s.** Lambda runtime ignores `http.Server` timeouts (Function URL closes the connection on its own 30s budget). Local-only effect. Cron handler internal deadline is `defaultCronTimeout` (60s) — well inside 75s. NOT a regression.
|
||||
- **`internal/server/router.go` cron handler:** rejection branches now `w.WriteHeader(401|404|405)` instead of `http.Error(..., "text", ...)`. Status codes preserved (EventBridge Scheduler only inspects status). Happy-path still returns `200`. Inner `recover()` wraps `DispatchScheduled` so a panic becomes a logged 500 instead of an opaque middleware-level recover. Cron *name* extraction (`strings.TrimPrefix(r.URL.Path, "/cron/")`) and regex (`^[a-z0-9_]{1,32}$`) unchanged. NOT a regression.
|
||||
- **`internal/server/log_middleware.go`:** moves the `req` log line into a `defer` and adds `recoverPanicStatus`. Purely additive observability; the chain `LogRequests → cronHandler` is unchanged. NOT a regression.
|
||||
- **`internal/modules/lolschedule/cron.go`:** adds `isTerminalSendError` + `pruneDeadSubscribers` after the existing fan-out. `dailyPushCronName = "lolschedule_daily_push"`, `dailyPushSchedule = "0 1 * * *"`, `dailyPushHandler`, `runDailyPush` all preserved. Pruning is best-effort and runs *after* the user-visible send loop — it cannot cause non-delivery. NOT a regression.
|
||||
|
||||
Red-team checks that came back clean:
|
||||
- `cronAuthHeader` constant name and value (`X-Cron-Token`) — unchanged.
|
||||
- `cronNameRe` regex — unchanged.
|
||||
- `defaultCronTimeout` — unchanged.
|
||||
- `modules.Deps.Bot` field — present (`internal/modules/module.go:74`); `BuildOptions.Bot: b` wiring in `cmd/server/main.go:108` — present; `dailyPushHandler` nil-check at `cron.go:88` — present.
|
||||
- `cfg.CronSecret` flow → `server.New(... CronSecret: cfg.CronSecret)` → `cronHandler(reg, secret)` — present (`cmd/server/main.go:131`).
|
||||
|
||||
### `4c81dd6` — dispatcher match-func swap
|
||||
|
||||
Touches `internal/modules/dispatcher.go` and `internal/telegram/webhook.go`. Both are on the **Telegram update path**, not the `/cron/{name}` HTTP path. `modules.DispatchScheduled` lives in a separate file (`internal/modules/cron_dispatcher.go`) and uses the cron registry, not the bot-handler registry. The cron HTTP route in `internal/server/router.go:109` calls `modules.DispatchScheduled` directly — it never touches `bot.RegisterHandlerMatchFunc`. NOT a regression for cron.
|
||||
|
||||
---
|
||||
|
||||
## Module-registry sanity check
|
||||
|
||||
- `cmd/server/main.go:43` registers `"lolschedule": lolschedule.New` in the factory map.
|
||||
- `template.yaml:16` and `samconfig.toml:16` both include `lolschedule` in `ModulesCSV` default. The deployed stack will instantiate the module.
|
||||
- `internal/modules/lolschedule/lolschedule.go` registers `dailyPushCron()`, which goes into `reg.Crons()` (logged at startup: `crons N`).
|
||||
|
||||
The cron handler will fire when invoked. The only missing piece was the schedule, now added.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**PROCEED-WITH-FIX. No rollback.**
|
||||
|
||||
Action plan, in order:
|
||||
1. Confirm `c70b9d0` + `585d996` are on `main` (they are, as of `git log` at audit time).
|
||||
2. Verify SSM parameter `/miti99bot/prod/cron-shared-secret` exists and is non-empty before next deploy: `aws ssm get-parameter --name /miti99bot/prod/cron-shared-secret --with-decryption`.
|
||||
3. Push triggers `.github/workflows/deploy.yml` → fetches secret → `sam deploy` with `CronSharedSecret=...` parameter override.
|
||||
4. Post-deploy: AWS Console → EventBridge Scheduler → `${StackName}-lolschedule-daily-push` → **Run now**. Tail CloudWatch for `cron triggered name=lolschedule_daily_push` + `lolschedule daily push complete sent=N`.
|
||||
5. Synthetic 401: `curl -X POST -H 'X-Cron-Token: wrong' <FunctionUrl>cron/lolschedule_daily_push` → expect HTTP 401 + log `cron rejected reason=secret_mismatch`.
|
||||
6. Wait for the next 01:00 UTC fire and confirm auto-delivery.
|
||||
|
||||
If step 3 fails with a CFN error on `HttpInvokeArgs`, the brainstorm calls out a property-name iteration gate (candidates: `HttpInvokeParameters`, `HttpParameters`). The current `HttpInvokeArgs` is correct for current SAM transform per AWS docs at time of writing — but if it rejects, iterate.
|
||||
|
||||
---
|
||||
|
||||
## Red-team angle — risks the brainstorm did not call out
|
||||
|
||||
| Risk | Severity | Note |
|
||||
|---|---|---|
|
||||
| `CronSharedSecret` default is `""` in `template.yaml:53`. If CI step that fetches the SSM secret runs but `aws ssm get-parameter` returns empty, `set -euo pipefail` will catch unset; but a *blank* SecureString value would pass through. Lambda would then start with `CRON_SHARED_SECRET=""` and `cronHandler` returns 404 on every request (`router.go:cronDisabled`). | Med | Add pre-deploy guard: `[ -n "$CRON_SECRET" ] \|\| { echo "empty cron secret"; exit 1; }` in the deploy step. |
|
||||
| EventBridge Scheduler IAM role uses `Resource: !GetAtt BotFunction.Arn` for `lambda:InvokeFunctionUrl`. AWS docs require the **Function URL ARN** form (`...:function:name`) — `GetAtt BotFunction.Arn` returns exactly that, so this is fine. Cross-check during `sam validate`. | Low | Brainstorm got this right. |
|
||||
| Function URL `AuthType: NONE` is assumed (the cron handler does its own header-based auth). If `AuthType: AWS_IAM` were set, the scheduler invocation would 403 even with correct header. Verify in `template.yaml` `BotFunctionUrl` resource. | Low | Already in brainstorm scope as "preserve route". |
|
||||
| `Input: "{}"` on the schedule: the Go cron handler does not parse a body, so any JSON-or-empty is fine. Confirmed by reading `cronHandler` in `router.go`. | None | — |
|
||||
|
||||
---
|
||||
|
||||
## Files cross-checked
|
||||
|
||||
- `template.yaml` (lines 14–22 ModulesCSV; 40–55 CronSharedSecret param; 160–215 SchedulerRole + LolscheduleDailyPushSchedule)
|
||||
- `internal/modules/lolschedule/cron.go:1-145`
|
||||
- `internal/modules/lolschedule/lolschedule.go`
|
||||
- `internal/modules/module.go:67-80` (Deps + BuildOptions)
|
||||
- `internal/modules/cron_dispatcher.go:12-30` (DispatchScheduled)
|
||||
- `internal/server/router.go:55-120` (cronHandler)
|
||||
- `internal/server/log_middleware.go`
|
||||
- `cmd/server/main.go:43, 100-135, 270-285` (factories, BuildOptions, server.Config)
|
||||
- `samconfig.toml:16`
|
||||
- `.github/workflows/deploy.yml` deploy step
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
- Has `sam deploy` actually been run since `585d996` landed? If not, the schedule does not yet exist in the deployed CloudFormation stack regardless of `main` state. CI on `main` push should handle it; confirm CI run for `585d996` succeeded.
|
||||
- Is there pre-deploy validation that `aws ssm get-parameter` returned a non-empty value? (See red-team table row 1.) Worth adding a one-line guard.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
type: debugger
|
||||
date: 2026-05-18
|
||||
slug: cfn-http-invoke-unsupported
|
||||
status: blocker-found
|
||||
related:
|
||||
- plans/reports/debugger-260518-1019-lolschedule-cron-not-firing.md
|
||||
- plans/reports/code-reviewer-260518-1019-recent-changes-cron-regression.md
|
||||
- plans/reports/brainstorm-260517-1411-eventbridge-schedule-fix.md (Approach A — invalidated)
|
||||
---
|
||||
|
||||
# Debug — CFN does not support HTTPS universal target on AWS::Scheduler::Schedule
|
||||
|
||||
## TL;DR
|
||||
|
||||
Brainstorm Approach A (`arn:aws:scheduler:::http-invoke` via CloudFormation) is **structurally unimplementable**. CFN spec has no property for HTTP target args (URL/method/headers). Both failed deploys today (`c70b9d0`, `585d996`) hit the same root cause — not a typo on `HttpInvokeArgs`, but a missing schema property entirely.
|
||||
|
||||
## Evidence
|
||||
|
||||
Official CFN reference [AWS::Scheduler::Schedule Target](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-scheduler-schedule-target.html):
|
||||
|
||||
Supported `Target` properties (exhaustive):
|
||||
- `Arn`, `RoleArn`, `Input`, `RetryPolicy`, `DeadLetterConfig`
|
||||
- Templated params: `EcsParameters`, `EventBridgeParameters`, `KinesisParameters`, `SageMakerPipelineParameters`, `SqsParameters`
|
||||
|
||||
No `HttpInvokeArgs`, no `HttpParameters`, no `HttpInvokeParameters`, no nested HTTP-config block anywhere. The EventBridge Scheduler **API** supports `HttpParameters` (header/query/path), but the **CloudFormation resource** does not surface it.
|
||||
|
||||
## Why the brainstorm's "validate-and-iterate gate" couldn't have worked
|
||||
|
||||
The brainstorm assumed the property exists under a different name. There is no name to iterate to — the entire HTTPS-universal-target shape is absent from the CFN schema. `sam validate` would have caught this immediately had it been run.
|
||||
|
||||
## Verified decision update (per review-audit-self-decision.md §1)
|
||||
|
||||
The brainstorm's Approach A locked in by user approval. New data invalidates it: confirmed against two AWS doc pages (Target schema + parent Schedule schema). This is *new* (CFN spec read), not an audit counter-argument. Per rule, surface the reversal to the user with options — do not silently flip.
|
||||
|
||||
## Path-forward options
|
||||
|
||||
| # | Option | Cost | Risk |
|
||||
|---|---|---|---|
|
||||
| **B** | **Direct Lambda invoke** (`Target.Arn: !GetAtt BotFunction.Arn`). Native CFN. Needs Go: handle Scheduler event payload in `cmd/server/main.go` (different shape than LWA HTTP event) and dispatch to cron by `detail`/`Input` content. | Small Go change + module re-wiring; breaks local-curl parity. | Low — well-trodden pattern. |
|
||||
| **C** | **EventBridge Rule + API Destination + Connection**. Pure CFN; uses `AWS::Events::Connection` + `AWS::Events::ApiDestination` + `AWS::Events::Rule`. Preserves `/cron/{name}` HTTP path, no Go change. | 3 new resources vs 1; older pattern; Connection requires auth config block. | Low. Verified CFN-supported. |
|
||||
| **D** | **Custom Resource (Lambda-backed)** that calls `scheduler:CreateSchedule` directly with `HttpParameters`. | High — extra Lambda + IAM + lifecycle handling. | Med — drift risk. |
|
||||
| **E** | **Out-of-band `aws scheduler create-schedule`** in `.github/workflows/deploy.yml` after `sam deploy`. | Low template change; schedule lives outside stack. | Med — drift, manual rotation, no CFN rollback. |
|
||||
|
||||
Recommend **C** (no Go change, pure CFN, only cost is 2 extra resources).
|
||||
|
||||
## Pre-deploy state right now
|
||||
|
||||
Prod stack still at `8e7fdce` (the two attempts today failed at changeset validation, no state mutation). Safe to revert `c70b9d0` + `585d996` from `main` and redesign, OR keep them on a branch and replace.
|
||||
|
||||
## Action items (require user decision)
|
||||
|
||||
1. **Reverse brainstorm Approach A.** Choose B / C / D / E.
|
||||
2. If **C**: revert `c70b9d0` + `585d996` on `main`, rewrite `template.yaml` block as `Connection + ApiDestination + Rule`.
|
||||
3. If **B**: revert same, plus small Go change in `cmd/server/main.go` to discriminate event shapes.
|
||||
4. Re-run `sam validate` locally before any commit (the gate that got skipped both times).
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
- Does user accept the small Go change (Approach B) for cleaner single-resource CFN, or prefer Approach C's pure-CFN-no-Go-change at the cost of 3 resources?
|
||||
- Should the two failed-deploy commits be reverted or amended on the same branch?
|
||||
|
||||
**Status:** DONE
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
type: debugger
|
||||
date: 2026-05-18
|
||||
slug: lolschedule-cron-not-firing
|
||||
status: done
|
||||
---
|
||||
|
||||
# Incident: lolschedule_daily_push cron not fired on 2026-05-18
|
||||
|
||||
## Verdict
|
||||
|
||||
**NO ROLLBACK NEEDED.**
|
||||
|
||||
The cron has never fired in production. `AWS::Scheduler::Schedule` was only added today
|
||||
(2026-05-18) in commit `c70b9d0`, and **both attempts to deploy it have failed** — so the
|
||||
resource does not yet exist in the live stack. The user's hypothesis that "a recent change
|
||||
broke the cron" is **refuted**: there is nothing to break because the schedule was never
|
||||
live. The brainstorm at `plans/reports/brainstorm-260517-1411-eventbridge-schedule-fix.md`
|
||||
correctly diagnosed the gap (deferred phase-05). The current failures are forward-progress
|
||||
bugs, not regressions.
|
||||
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
| Time (UTC) | Commit | Event |
|
||||
|---|---|---|
|
||||
| 2026-05-16 07:51 | `8e7fdce` | Last successful deploy. No `AWS::Scheduler::Schedule`. |
|
||||
| 2026-05-18 04:46 | `c70b9d0` | Added `LolscheduleDailyPushSchedule`; token via `{{resolve:ssm-secure}}`. Deploy **FAILED**. |
|
||||
| 2026-05-18 06:04 | `585d996` | Fixed token via `CronSharedSecret` CFN parameter + CI SSM fetch. Deploy **FAILED** again. |
|
||||
| 2026-05-18 01:00 UTC | — | Scheduled fire time passes; no schedule in prod → nothing fires. |
|
||||
|
||||
---
|
||||
|
||||
## Evidence Chain
|
||||
|
||||
### 1. Schedule resource absent from prod stack
|
||||
|
||||
Commit `8e7fdce` (last successful deploy, 2026-05-16) contains no `AWS::Scheduler::Schedule`
|
||||
resource in `template.yaml`. Both deploys today failed before CFN applied any changeset.
|
||||
`LolscheduleDailyPushSchedule` (`template.yaml:190-215`) has never reached prod.
|
||||
|
||||
This confirms the brainstorm claim: the gap is not a regression — it is an unexecuted
|
||||
deferred phase (`plans/260510-0234-pre-deploy-wrapup/phase-05-eventbridge-schedules.md`).
|
||||
|
||||
### 2. First deploy failure — `c70b9d0` (04:47 UTC)
|
||||
|
||||
CI error (run `26014078660`):
|
||||
```
|
||||
SSM Secure reference is not supported in:
|
||||
[AWS::Scheduler::Schedule/Properties/Target/HttpInvokeArgs/HeaderParameters/X-Cron-Token]
|
||||
```
|
||||
`template.yaml` used `{{resolve:ssm-secure:/miti99bot/${StackEnv}/cron-shared-secret}}`
|
||||
at line 215. CFN's documented property allowlist blocks secure-string dynamic refs inside
|
||||
`AWS::Scheduler::Schedule` header parameters. Changeset creation failed immediately —
|
||||
**no CFN resource was created or modified**.
|
||||
|
||||
### 3. Second deploy failure — `585d996` (06:05 UTC)
|
||||
|
||||
Fix correctly removed the `{{resolve:ssm-secure}}` ref and switched to a `NoEcho`
|
||||
CFN parameter `CronSharedSecret` (`template.yaml:50-54`) fetched from SSM by CI and
|
||||
passed via `--parameter-overrides`. CI log confirms the secret was fetched
|
||||
(`Parameter overrides: {"CronSharedSecret": "*****"}`). Changeset reached CFN but
|
||||
failed with:
|
||||
```
|
||||
The following hook(s)/validation failed: [AWS::EarlyValidation::PropertyValidation]
|
||||
```
|
||||
CFN Early Validation rejected the changeset. The property that triggers this is most likely
|
||||
`HttpInvokeArgs` — the exact property name for HTTPS universal targets in
|
||||
`AWS::Scheduler::Schedule` is not `HttpInvokeArgs` (the brainstorm flagged this as a
|
||||
medium-risk "validate-and-iterate" gate at the risks table). The `sam validate` gate
|
||||
called out in the brainstorm was not run before committing.
|
||||
|
||||
### 4. Auth path analysis (CRON_SHARED_SECRET_PARAMETER_NAME)
|
||||
|
||||
Lambda resolves `CRON_SHARED_SECRET` at cold-start via `resolveSSMSecrets`
|
||||
(`cmd/server/main.go:290-345`). If the SSM param is missing or empty:
|
||||
- `resolveSSMSecrets` returns a fatal error → Lambda crashes on cold start (no silent 401).
|
||||
- If the env var `CRON_SHARED_SECRET` is empty but `CRON_SHARED_SECRET_PARAMETER_NAME`
|
||||
is also empty, `cfg.CronSecret == ""` → `cronDisabled = true` in
|
||||
`internal/server/router.go:59` → all `/cron/{name}` calls return 404.
|
||||
|
||||
Since the schedule never reached prod, this path is moot today. But **if the
|
||||
`CronSharedSecret` CFN param were passed empty**, the Lambda would warn at startup
|
||||
(`cmd/server/main.go:124`) and serve 404 to every cron call — **silent failure, not 401**.
|
||||
This is a latent risk for the next deploy attempt.
|
||||
|
||||
### 5. Handler/dispatcher wiring — healthy
|
||||
|
||||
`lolschedule.New` registers `dailyPushCron()` (`internal/modules/lolschedule/cron.go:76-82`)
|
||||
via `modules.Build` → `modules.Install`. `deps.Bot` is injected at
|
||||
`cmd/server/main.go:106-110`. Route `/cron/lolschedule_daily_push` resolves correctly
|
||||
through `cronHandler` → `modules.DispatchScheduled`. No wiring bug found.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
**`AWS::Scheduler::Schedule` was never successfully deployed to prod.**
|
||||
|
||||
- Phase-05 was deferred at 2026-05-10 (per brainstorm `related` links).
|
||||
- `c70b9d0` first attempted to wire it today but used `{{resolve:ssm-secure}}` in a
|
||||
property where CFN forbids it → CFN rejected the changeset.
|
||||
- `585d996` fixed the token-injection mechanism but introduced a second CFN validation
|
||||
error (`AWS::EarlyValidation::PropertyValidation`), likely the wrong property name for
|
||||
the HTTPS target's header arguments.
|
||||
|
||||
Neither deploy reached the stack. Current prod stack is still the `8e7fdce` baseline with
|
||||
no scheduler resource.
|
||||
|
||||
---
|
||||
|
||||
## Competing Hypotheses — Disposition
|
||||
|
||||
| Hypothesis | Status | Evidence |
|
||||
|---|---|---|
|
||||
| Recent commit introduced a regression (broke something that worked) | **REFUTED** | No schedule ever deployed; last prod deploy `8e7fdce` predates schedule work |
|
||||
| Schedule never wired (deferred phase, not a regression) | **CONFIRMED** | Brainstorm + `git log` + both CI failures show it never reached prod |
|
||||
| SSM secret missing → silent 401 on every fire | **NOT APPLICABLE** | No schedule in prod to fire; moot until deploy succeeds |
|
||||
| `HttpInvokeArgs` property name wrong for SAM transform | **CONFIRMED** as current blocker | `AWS::EarlyValidation::PropertyValidation` failure on `585d996` |
|
||||
|
||||
---
|
||||
|
||||
## Action Required (not a rollback)
|
||||
|
||||
1. **Identify correct CFN property name** for header parameters on `AWS::Scheduler::Schedule`
|
||||
HTTPS target. Candidates: `HttpInvokeParameters` or a nested structure. Run
|
||||
`sam validate` locally before committing (the brainstorm prescribed this gate but it
|
||||
was skipped).
|
||||
2. **Verify SSM param non-empty** before next deploy:
|
||||
```
|
||||
aws ssm get-parameter --name /miti99bot/prod/cron-shared-secret --with-decryption
|
||||
```
|
||||
Empty value → `cronDisabled=true` → 404 on every cron hit (not 401).
|
||||
3. After successful deploy, do a "Run now" from EventBridge Scheduler console + tail
|
||||
CloudWatch for `cron triggered name=lolschedule_daily_push`.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
- Exact correct property name for `AWS::Scheduler::Schedule` HTTPS target header
|
||||
parameters (requires `sam validate` iteration or AWS docs check — out of scope for
|
||||
this report-only investigation).
|
||||
+69
-10
@@ -40,6 +40,16 @@ Parameters:
|
||||
Default: ""
|
||||
Description: Email for $1 budget alert. Leave empty to skip the budget resource.
|
||||
|
||||
# SSM holds the canonical value; CI fetches and passes via --parameter-overrides
|
||||
# because EventBridge Connection's ApiKeyValue is consumed at stack-update time
|
||||
# and stored in a service-linked secret (no per-invoke SSM fetch). NoEcho keeps
|
||||
# the value out of CFN events / console / drift detection.
|
||||
CronSharedSecret:
|
||||
Type: String
|
||||
NoEcho: true
|
||||
Default: ""
|
||||
Description: X-Cron-Token header value the EventBridge Rule presents to /cron/{name}. Must match the SSM-stored value the Lambda loads at cold start.
|
||||
|
||||
Conditions:
|
||||
HasAlertEmail: !Not [!Equals [!Ref AlertEmail, ""]]
|
||||
|
||||
@@ -143,6 +153,12 @@ Resources:
|
||||
Resource: !Sub "arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/miti99bot/${StackEnv}/*"
|
||||
|
||||
# --- Cron -----------------------------------------------------------------
|
||||
#
|
||||
# Why EventBridge Rule + ApiDestination + Connection (not Scheduler):
|
||||
# CloudFormation's AWS::Scheduler::Schedule Target schema has no property
|
||||
# for HTTPS universal-target invocation (URL/method/headers). The legacy
|
||||
# EventBridge Rule path does — via ApiDestination. Pure CFN, single-invoke
|
||||
# per day, free tier covered.
|
||||
|
||||
CronDLQ:
|
||||
Type: AWS::SQS::Queue
|
||||
@@ -150,32 +166,75 @@ Resources:
|
||||
QueueName: !Sub "${AWS::StackName}-cron-dlq"
|
||||
MessageRetentionPeriod: 1209600 # 14 days
|
||||
|
||||
SchedulerExecutionRole:
|
||||
# Connection holds the auth credential. EventBridge persists ApiKeyValue in a
|
||||
# service-linked secret in this account (fees absorbed by AWS per docs). The
|
||||
# header name + value are presented on every ApiDestination invocation.
|
||||
CronConnection:
|
||||
Type: AWS::Events::Connection
|
||||
Properties:
|
||||
Name: !Sub "${AWS::StackName}-cron"
|
||||
Description: Shared cron auth — presents X-Cron-Token on every /cron/* call
|
||||
AuthorizationType: API_KEY
|
||||
AuthParameters:
|
||||
ApiKeyAuthParameters:
|
||||
ApiKeyName: X-Cron-Token
|
||||
ApiKeyValue: !Ref CronSharedSecret
|
||||
|
||||
# ApiDestination is the concrete endpoint EventBridge POSTs to. One per cron
|
||||
# route — paths are not parameterised, so add another resource if/when a
|
||||
# second cron handler ships. Keeping ${BotFunctionUrl.FunctionUrl} trailing
|
||||
# slash + bare relative path gives a clean single-slash join.
|
||||
LolscheduleDailyPushApiDestination:
|
||||
Type: AWS::Events::ApiDestination
|
||||
Properties:
|
||||
Name: !Sub "${AWS::StackName}-lolschedule-daily-push"
|
||||
ConnectionArn: !GetAtt CronConnection.Arn
|
||||
HttpMethod: POST
|
||||
InvocationEndpoint: !Sub "${BotFunctionUrl.FunctionUrl}cron/lolschedule_daily_push"
|
||||
|
||||
# Role EventBridge Rule assumes to invoke the ApiDestination and write to
|
||||
# the DLQ. Locked to this destination's ArnForPolicy (the IAM-resource form,
|
||||
# not the ARN form).
|
||||
EventBridgeInvokeRole:
|
||||
Type: AWS::IAM::Role
|
||||
Properties:
|
||||
AssumeRolePolicyDocument:
|
||||
Version: '2012-10-17'
|
||||
Statement:
|
||||
- Effect: Allow
|
||||
Principal: { Service: scheduler.amazonaws.com }
|
||||
Principal: { Service: events.amazonaws.com }
|
||||
Action: sts:AssumeRole
|
||||
Policies:
|
||||
- PolicyName: cron-https-invoke
|
||||
- PolicyName: cron-invoke-apidestination
|
||||
PolicyDocument:
|
||||
Version: '2012-10-17'
|
||||
Statement:
|
||||
# HTTPS targets to Lambda Function URL — Scheduler treats them as
|
||||
# invocations of the function. Lock to this function's ARN only.
|
||||
- Effect: Allow
|
||||
Action: lambda:InvokeFunctionUrl
|
||||
Resource: !GetAtt BotFunction.Arn
|
||||
# DLQ
|
||||
Action: events:InvokeApiDestination
|
||||
Resource: !GetAtt LolscheduleDailyPushApiDestination.ArnForPolicy
|
||||
- Effect: Allow
|
||||
Action: sqs:SendMessage
|
||||
Resource: !GetAtt CronDLQ.Arn
|
||||
|
||||
# Concrete AWS::Scheduler::Schedule resources are added per cron handler;
|
||||
# the role + DLQ above are provisioned once and reused across all schedules.
|
||||
# The schedule itself. cron(0 1 * * ? *) is 01:00 UTC daily = 08:00 ICT,
|
||||
# matching the dailyPushSchedule constant in internal/modules/lolschedule/cron.go.
|
||||
LolscheduleDailyPushRule:
|
||||
Type: AWS::Events::Rule
|
||||
Properties:
|
||||
Name: !Sub "${AWS::StackName}-lolschedule-daily-push"
|
||||
Description: Fires lolschedule daily-push handler at 01:00 UTC (08:00 ICT)
|
||||
ScheduleExpression: "cron(0 1 * * ? *)"
|
||||
State: ENABLED
|
||||
Targets:
|
||||
- Id: lolschedule-daily-push
|
||||
Arn: !GetAtt LolscheduleDailyPushApiDestination.Arn
|
||||
RoleArn: !GetAtt EventBridgeInvokeRole.Arn
|
||||
Input: "{}"
|
||||
RetryPolicy:
|
||||
MaximumRetryAttempts: 2
|
||||
MaximumEventAgeInSeconds: 600
|
||||
DeadLetterConfig:
|
||||
Arn: !GetAtt CronDLQ.Arn
|
||||
|
||||
# --- Cost guard -----------------------------------------------------------
|
||||
|
||||
|
||||
Reference in New Issue
Block a user