mirror of
https://github.com/tiennm99/miti99bot.git
synced 2026-09-11 22:18:42 +00:00
chore(plans): remove completed/superseded plans and orphaned reports
Delete 7 plan dirs (6 implemented, 1 superseded) and 19 reports tied to removed or non-existent plans. Active plans (aws-port, pre-deploy-wrapup, cf-data-migration, trongtruonghop, iam-least-privilege, go-port-cloud-run) and their referenced reports are preserved.
This commit is contained in:
@@ -1,115 +0,0 @@
|
||||
# Phase 1 — Foundation
|
||||
|
||||
Module scaffold, KV state, word2sim HTTP client, env wiring.
|
||||
Nothing playable at the end of this phase, but all glue is in place.
|
||||
|
||||
## Context links
|
||||
|
||||
- Overview: `./plan.md`
|
||||
- Existing parallel module: `src/modules/loldle/` (follow shape)
|
||||
- External API contract: `tiennm99/word2sim` repo `README.md`
|
||||
- `GET /random?min_rank&max_rank&alpha_only&min_len&max_len` → `{word, rank}`
|
||||
- `GET /similarity?a&b` → `{a, b, canonical_a, canonical_b, in_vocab_a, in_vocab_b, similarity}`
|
||||
|
||||
## Files to create
|
||||
|
||||
### `src/modules/semantle/index.js` (~45 LOC)
|
||||
Module export. Mirrors `src/modules/loldle/index.js`:
|
||||
- Captures `{db, env}` in `init({db, env})` — `env` is new vs loldle (need `WORD2SIM_API_URL`).
|
||||
- Exposes 5 commands (see `plan.md`). Each handler closure gets `(ctx, { db, apiBase })`.
|
||||
|
||||
### `src/modules/semantle/api-client.js` (~80 LOC)
|
||||
Thin wrapper over word2sim HTTP endpoints. Example shape:
|
||||
```js
|
||||
export function createClient(apiBase) {
|
||||
return {
|
||||
randomWord: (opts) => fetchJson(`${apiBase}/random`, opts),
|
||||
similarity: (a, b) => fetchJson(`${apiBase}/similarity`, { a, b }),
|
||||
};
|
||||
}
|
||||
```
|
||||
- Normalize `apiBase` to strip trailing slash.
|
||||
- Build query strings with `URLSearchParams`.
|
||||
- Timeout via `AbortController` (5s).
|
||||
- Throw `Word2SimError` with `{status, body}` on non-2xx; caller decides user-facing message.
|
||||
- `User-Agent: miti99bot/semantle` header for traceability.
|
||||
|
||||
### `src/modules/semantle/state.js` (~100 LOC)
|
||||
KV persistence. Key layout under `semantle:` prefix:
|
||||
- `game:<subject>` → `{target, startedAt, solved, guesses:[{word, canonical, similarity}]}`
|
||||
— `target` stored lowercased; solve = `canonical.toLowerCase() === target`.
|
||||
- `stats:<subject>` → `{played, solved, totalGuesses, bestGuessCount, lastResultAt}`
|
||||
|
||||
Exports:
|
||||
- `loadGame(db, subject) → GameState | null`
|
||||
- `saveGame(db, subject, state)` — TTL `60*60*24*7` (7d)
|
||||
- `clearGame(db, subject)`
|
||||
- `loadStats(db, subject) → Stats` (returns defaults if missing)
|
||||
- `recordResult(db, subject, {solved, guessCount})`
|
||||
- Increments `played`, `solved` (if solved), `totalGuesses += guessCount`.
|
||||
- `bestGuessCount = min(bestGuessCount ?? ∞, guessCount)` on solved.
|
||||
- Writes `lastResultAt = Date.now()`.
|
||||
|
||||
## Files to edit
|
||||
|
||||
### `src/modules/index.js`
|
||||
Add one line to the static import map, alphabetically after `misc`:
|
||||
```js
|
||||
semantle: () => import("./semantle/index.js"),
|
||||
```
|
||||
|
||||
### `wrangler.toml`
|
||||
- In `[vars]` append `semantle` to `MODULES`.
|
||||
- Add `WORD2SIM_API_URL = "https://word2sim.sg.miti99.com"` to `[vars]`.
|
||||
|
||||
### `.dev.vars.example`
|
||||
Add optional override:
|
||||
```
|
||||
# Optional: override for local/self-hosted word2sim instance
|
||||
# WORD2SIM_API_URL=http://localhost:8000
|
||||
```
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. Create folder + empty stubs for all files.
|
||||
2. Wire `src/modules/index.js` entry.
|
||||
3. Wire `wrangler.toml` vars; confirm `npm run dev` boots without error.
|
||||
4. Implement `api-client.js`; ad-hoc test with `wrangler dev` + curl to ensure the hosted instance responds.
|
||||
5. Implement `state.js`.
|
||||
6. `index.js` with placeholder handlers that return "not implemented yet".
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] `src/modules/semantle/` folder + empty files
|
||||
- [ ] Register in `src/modules/index.js`
|
||||
- [ ] Update `wrangler.toml` `MODULES` + `WORD2SIM_API_URL`
|
||||
- [ ] Update `.dev.vars.example` with optional override comment
|
||||
- [ ] Implement `api-client.js` (2 methods + `Word2SimError` + timeout)
|
||||
- [ ] Implement `state.js` (load/save/clear + stats + recordResult)
|
||||
- [ ] Placeholder `index.js` export + noop handlers
|
||||
- [ ] `npm run dev` boots without errors; `/semantle` reply shows "not implemented"
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Dev server starts with `semantle` listed in modules.
|
||||
- Placeholder `/semantle` command responds in Telegram (polling via dev webhook or logs).
|
||||
- `api-client.js` callable from a node REPL or test file against the live service.
|
||||
- No biome/eslint warnings.
|
||||
|
||||
## Risk
|
||||
|
||||
- **Cloudflare Worker egress to word2sim** — ensure `fetch()` to the SG subdomain
|
||||
is not blocked by any Worker networking policy. Expected fine; same pattern as
|
||||
`trading/prices.js` and `lolschedule/api-client.js`.
|
||||
- **KV size per game** — just target + guess history (a few KB even after hundreds
|
||||
of guesses). Well under KV value limit.
|
||||
|
||||
## Security
|
||||
|
||||
- No secrets added; `WORD2SIM_API_URL` is a public endpoint.
|
||||
- All user input goes into URL query params — rely on `URLSearchParams` encoding
|
||||
to avoid injection; never concatenate user strings into URLs directly.
|
||||
|
||||
## Next
|
||||
|
||||
→ Phase 2 `phase-02-gameplay.md` — real handlers and rendering.
|
||||
@@ -1,140 +0,0 @@
|
||||
# Phase 2 — Gameplay
|
||||
|
||||
Real handlers, guess lookup, board rendering, similarity formatting.
|
||||
End of phase: module is fully playable in Telegram.
|
||||
|
||||
## Context links
|
||||
|
||||
- Overview: `./plan.md`
|
||||
- Prior phase: `./phase-01-foundation.md`
|
||||
- Pattern to mirror: `src/modules/loldle/handlers.js`, `src/modules/wordle/handlers.js`
|
||||
- Render pattern: `src/modules/loldle/render.js`
|
||||
|
||||
## Files to create
|
||||
|
||||
### `src/modules/semantle/lookup.js` (~25 LOC)
|
||||
- `normalize(raw) → string` — trim, collapse whitespace, lowercase.
|
||||
- `isValidShape(word) → boolean` — reject empty, length > 64, non-ASCII-letters-only
|
||||
(matches `/random` default filter; avoids wasted API round-trips).
|
||||
|
||||
### `src/modules/semantle/format.js` (~30 LOC)
|
||||
- `formatWarmth(similarity) → string` — signed percent: `Math.round(similarity * 100)`
|
||||
shown as `+73` / `-04`.
|
||||
- `progressBar(similarity) → string` — 10-cell unicode bar from `-1..1`
|
||||
(use `░▓█`; helps visual scanning). Optional / can be skipped if time-boxed.
|
||||
- `warmthEmoji(similarity)` — 🥶 (< 0.2) / 😐 (< 0.4) / 🌡️ (< 0.6) / 🔥 (< 0.8) / 🎯 (≥ 0.8)
|
||||
|
||||
### `src/modules/semantle/render.js` (~70 LOC)
|
||||
Telegram HTML `<pre>` monospace block. Two public exports:
|
||||
- `renderBoard(guesses, latestIndex)` — sort by similarity desc, show at most top 15;
|
||||
highlight the latest guess with a leading marker. Each row:
|
||||
```
|
||||
# warmth word emoji
|
||||
1 +78 sea 🔥
|
||||
2 +45 fish 🌡️
|
||||
```
|
||||
- `renderGuess(entry, position, total)` — single-line summary for a guess that
|
||||
fell outside the rendered top-15: `"Your guess 'carpet' → +12"`.
|
||||
- Header line: `🎯 Semantle — <N> guesses`.
|
||||
- Footer when solved: `"✅ Solved in <N> guesses!"`.
|
||||
|
||||
### `src/modules/semantle/handlers.js` (~170 LOC)
|
||||
One exported function per command:
|
||||
- `handleSemantle(ctx, deps)`
|
||||
- `handleGiveup(ctx, deps)`
|
||||
- `handleNew(ctx, deps)`
|
||||
- `handleStats(ctx, deps)`
|
||||
|
||||
where `deps = { db, client }`. Shared helpers (subject resolution, arg parsing)
|
||||
copied from loldle with minimal change.
|
||||
|
||||
**Flow for `/semantle <word>`:**
|
||||
|
||||
1. Resolve subject; reject if missing.
|
||||
2. `game = await loadOrStart(db, client, subject)` — lazy-init calls `client.randomWord(...)`;
|
||||
target stored lowercased.
|
||||
3. Normalize the guess (trim, lowercase).
|
||||
4. `res = await client.similarity(game.target, guess)`.
|
||||
5. If `!res.in_vocab_b` → reply "🤔 unknown word" without appending.
|
||||
6. Append `{word:guess, canonical:res.canonical_b, similarity:res.similarity}`;
|
||||
set `startedAt` if null; saveGame.
|
||||
7. If `res.canonical_b.toLowerCase() === game.target` → mark `solved`, `recordResult({solved:true, guessCount})`,
|
||||
`clearGame`, reply with board + win message.
|
||||
8. Else reply with `renderBoard` — guess pool grows unbounded.
|
||||
|
||||
**Flow for `/semantle` (no arg):**
|
||||
- If no active game → lazy-init (but don't call similarity; just show empty board
|
||||
with "🆕 Round ready — send your first guess.").
|
||||
- Else → `renderBoard`.
|
||||
|
||||
**Flow for `/semantle_new`:**
|
||||
- Load current game; if exists and has ≥1 guess, `recordResult({solved:false})`
|
||||
and `clearGame`. Then lazy-init a fresh one; reply "🆕 New round started."
|
||||
|
||||
**Flow for `/semantle_giveup`:**
|
||||
- If no active game → "No active round."
|
||||
- Else reveal `game.target`, `recordResult({solved:false, guessCount: guesses.length})`,
|
||||
`clearGame`, reply.
|
||||
|
||||
**Flow for `/semantle_stats`:**
|
||||
- `loadStats(subject)`, render:
|
||||
- Played / Solved / Solve rate
|
||||
- Total guesses / Best guess count (lowest number of guesses to solve)
|
||||
- Average guesses per solve (if `solved > 0`)
|
||||
|
||||
## Error handling
|
||||
|
||||
- Wrap every `client.*` call in try/catch. On `Word2SimError` or fetch timeout:
|
||||
reply `"⚠️ Upstream hiccup — try again in a few seconds."` and log the error
|
||||
structured (`console.log(JSON.stringify({msg:"semantle_upstream_fail", ...}))`).
|
||||
- If `/random` fails, do NOT persist a partial game — user simply retries.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. `lookup.js` first — pure logic, trivial to verify.
|
||||
2. `format.js` — pure logic, stub renderings.
|
||||
3. `render.js` — build HTML using phase-1 state shape.
|
||||
4. Replace placeholder handlers with real implementations, one command at a time:
|
||||
`_stats` → `/semantle` (no-arg, empty board) → `/semantle <word>` → `_giveup` → `_new`.
|
||||
5. End-to-end manual test via `wrangler dev` + Telegram bot or `curl` against the
|
||||
webhook endpoint.
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] `lookup.js` normalize + isValidShape
|
||||
- [ ] `format.js` formatWarmth, warmthEmoji, (optional) progressBar
|
||||
- [ ] `render.js` renderBoard + renderGuess
|
||||
- [ ] `handlers.js` subject resolver + arg parser (copy from loldle)
|
||||
- [ ] `handlers.js` handleStats (simplest path, ensures KV wiring works)
|
||||
- [ ] `handlers.js` handleSemantle no-arg path
|
||||
- [ ] `handlers.js` handleSemantle guess path (solve / OOV / score)
|
||||
- [ ] `handlers.js` handleGiveup
|
||||
- [ ] `handlers.js` handleNew
|
||||
- [ ] E2E smoke test in Telegram
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `/semantle` with no arg shows a clean "round ready" message.
|
||||
- `/semantle apple` returns similarity within ~500ms p50.
|
||||
- `/semantle <target>` ends the round and updates stats.
|
||||
- `/semantle_giveup` reveals the target and clears state.
|
||||
- Out-of-vocab guess does not count against the guess tally.
|
||||
- Board stays readable up to 100+ guesses (render caps at top 15).
|
||||
|
||||
## Risk
|
||||
|
||||
- **Latency** — two KV reads + one fetch per guess. Target ≤ 800ms p95. If the
|
||||
hosted word2sim cold-starts too slowly, add a periodic cron warmup later.
|
||||
- **File-size drift** — `handlers.js` at ~170 LOC is close to the 200 cap; if it
|
||||
overruns, split `_stats` into its own `stats-handler.js` (pattern used by
|
||||
trading module).
|
||||
|
||||
## Security
|
||||
|
||||
- Treat the guess string as untrusted: escape-html before rendering.
|
||||
- Do NOT leak `game.target` in any response path except `/semantle_giveup` and
|
||||
`handleSemantle` win reply.
|
||||
|
||||
## Next
|
||||
|
||||
→ Phase 3 `phase-03-tests-docs.md` — coverage, README, `/help` registration.
|
||||
@@ -1,113 +0,0 @@
|
||||
# Phase 3 — Tests & Documentation
|
||||
|
||||
Ship-ready polish: unit-test coverage with fake KV + stubbed fetch, module README,
|
||||
`/help` integration verification.
|
||||
|
||||
## Context links
|
||||
|
||||
- Overview: `./plan.md`
|
||||
- Prior phases: `./phase-01-foundation.md`, `./phase-02-gameplay.md`
|
||||
- Test pattern: `tests/modules/wordle/` and `tests/modules/trading/` (fetch-stubbing examples)
|
||||
- Fakes: `tests/fakes/{fake-kv-namespace,fake-bot,fake-modules}.js`
|
||||
|
||||
## Files to create
|
||||
|
||||
### `tests/modules/semantle/api-client.test.js` (~50 LOC)
|
||||
Stub `global.fetch` with `vi.fn()`:
|
||||
- asserts query-string building (rank filters, URL encoding)
|
||||
- asserts `Word2SimError` on non-2xx
|
||||
- asserts AbortController timeout path (simulate slow upstream)
|
||||
|
||||
### `tests/modules/semantle/state.test.js` (~60 LOC)
|
||||
Using `fake-kv-namespace`:
|
||||
- load/save round-trip preserves shape
|
||||
- clearGame removes entry
|
||||
- recordResult increments played/solved/totalGuesses correctly
|
||||
- bestGuessCount is `min(prev, current)` only when solved
|
||||
- loadStats returns defaults when empty
|
||||
|
||||
### `tests/modules/semantle/format.test.js` (~25 LOC)
|
||||
Pure-function coverage:
|
||||
- formatWarmth signed/rounded
|
||||
- warmthEmoji buckets boundary cases
|
||||
|
||||
### `tests/modules/semantle/render.test.js` (~40 LOC)
|
||||
- renderBoard empty state renders "round ready" prompt
|
||||
- renderBoard sorts by similarity desc (verify first row = highest score)
|
||||
- renderBoard caps to top 15 when >15 guesses
|
||||
- renderGuess escapes HTML-unsafe chars in the word
|
||||
|
||||
### `tests/modules/semantle/handlers.test.js` (~130 LOC)
|
||||
Integration-ish: fake KV + stubbed client (not real fetch).
|
||||
- happy path: round start → guess → guess → solve (case-insensitive) → stats updated
|
||||
- OOV guess: not appended, no state mutation
|
||||
- _giveup reveals target and clears game
|
||||
- _new abandons prior round + clears, records non-solve
|
||||
- error from client surfaces a user-friendly message; state unchanged
|
||||
- case sensitivity: guess "APPLE" against target "apple" solves the round
|
||||
|
||||
## Files to create (docs)
|
||||
|
||||
### `src/modules/semantle/README.md`
|
||||
Mirror `src/modules/loldle/README.md` shape. Sections:
|
||||
- **Commands** table (from `plan.md`)
|
||||
- **Data source** — point at `tiennm99/word2sim` hosted instance
|
||||
- **Architecture** — list of files and what each does
|
||||
- **Storage** — KV layout table (`game:<subject>`, `stats:<subject>`)
|
||||
- **Config** — `WORD2SIM_API_URL` env var
|
||||
- **Credits** — word2vec / GoogleNews pretrained vectors
|
||||
|
||||
## Files to edit
|
||||
|
||||
### `docs/adding-a-module.md`
|
||||
No change expected unless the word2sim env-var pattern is the first of its kind.
|
||||
If so, add a short note about `[vars]` config for external API bases.
|
||||
|
||||
### `scripts/register.js`
|
||||
Verify `setMyCommands` picks up the new public commands automatically (no code
|
||||
change expected — registry is the source of truth).
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] `api-client.test.js`
|
||||
- [ ] `state.test.js`
|
||||
- [ ] `format.test.js`
|
||||
- [ ] `render.test.js`
|
||||
- [ ] `handlers.test.js`
|
||||
- [ ] `src/modules/semantle/README.md`
|
||||
- [ ] Run `npm test` — all pass
|
||||
- [ ] Run `npm run lint && npm run format` — clean
|
||||
- [ ] `npm run register:dry` — confirms `/semantle`, `/semantle_giveup`, `/semantle_new`,
|
||||
`/semantle_stats` appear in the command payload
|
||||
- [ ] Optional: `docs/adding-a-module.md` one-paragraph addition if env-var
|
||||
pattern is novel
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Test coverage for all new files (>80% line coverage, pragmatic thresholds).
|
||||
- No biome/eslint warnings (project enforces 100-char line width, sorted imports,
|
||||
trailing commas).
|
||||
- `npm run register:dry` output includes all public `/semantle*` commands.
|
||||
- `src/modules/semantle/README.md` parallel to loldle's in shape and depth.
|
||||
|
||||
## Risk
|
||||
|
||||
- **Fetch stub drift** — if word2sim adds fields, tests may not catch breaking
|
||||
response-shape changes in prod. Mitigation: add a single optional integration
|
||||
test (guarded by env flag) that hits the real URL; skip by default.
|
||||
- **Flaky timing tests** — avoid real `setTimeout` in the AbortController test;
|
||||
use `vi.useFakeTimers()` if needed.
|
||||
|
||||
## Security
|
||||
|
||||
- Tests must not commit real TELEGRAM tokens or call Telegram.
|
||||
- `.dev.vars` is gitignored; never add secrets to `.dev.vars.example`.
|
||||
|
||||
## Next
|
||||
|
||||
- Deploy via `npm run deploy` (handles webhook + setMyCommands).
|
||||
- Monitor the first day of usage via CF Worker logs; look for upstream errors.
|
||||
- Potential follow-ups (separate plans):
|
||||
- Daily shared secret (Semantle-classic mode)
|
||||
- Leaderboard module integration
|
||||
- Webhook-side warm-up cron to keep word2sim hot
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
name: semantle-module
|
||||
status: completed
|
||||
created: 2026-04-22
|
||||
completed: 2026-04-22
|
||||
slug: semantle-module
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
---
|
||||
|
||||
# Semantle Module — miti99bot
|
||||
|
||||
Add a Telegram game module mirroring `loldle`/`wordle`, but powered by word2vec
|
||||
cosine similarity via the hosted `word2sim` API. Unlimited guesses per round.
|
||||
|
||||
**External dependency:** https://word2sim.sg.miti99.com/ (our own hosted instance;
|
||||
see `tiennm99/word2sim` repo).
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/semantle` | show current board, or submit a guess (arg) |
|
||||
| `/semantle <word>` | submit a guess |
|
||||
| `/semantle_giveup` | reveal the secret and end the round |
|
||||
| `/semantle_new` | abandon round + start fresh (same as wordle pattern) |
|
||||
| `/semantle_stats` | show per-subject stats |
|
||||
|
||||
## Key differences from loldle/wordle
|
||||
|
||||
- **Unlimited guesses** — no MAX cap; round ends only on solve, giveup, or `_new`.
|
||||
- **Continuous score** — each guess returns cosine similarity ∈ [−1, 1] scaled to
|
||||
0–100 "warmth" for display. No rank concept.
|
||||
- **Case-insensitive match** — target stored lowercase; guess canonical form is
|
||||
lowercased before equality check.
|
||||
- **Network-bound** — calls word2sim per guess; needs graceful fallback.
|
||||
- **Stats model** — `{played, solved, totalGuesses, bestGuessCount}` (no streak,
|
||||
since there is no loss state).
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | File | Focus | Est. LOC |
|
||||
|-------|------|-------|---------:|
|
||||
| 1 | `phase-01-foundation.md` | module scaffold, KV state, word2sim api-client, env wiring | ~220 |
|
||||
| 2 | `phase-02-gameplay.md` | handlers, lookup, render, format | ~310 |
|
||||
| 3 | `phase-03-tests-docs.md` | vitest coverage, README, help-command integration | ~180 |
|
||||
|
||||
## Critical files
|
||||
|
||||
- **Create:** `src/modules/semantle/{index,api-client,state,handlers,lookup,render,format}.js` + `README.md`
|
||||
- **Edit:** `src/modules/index.js` (register loader), `wrangler.toml` (MODULES list + `WORD2SIM_API_URL` var), `.dev.vars.example` (optional override)
|
||||
- **Test:** `tests/modules/semantle/*.test.js` (stub `global.fetch`)
|
||||
|
||||
## Design decisions (locked in unless overturned)
|
||||
|
||||
1. **Subject resolution** — same as loldle: user id in DMs, chat id in groups.
|
||||
2. **Round start** — call `/random` once to pick the target; store it lowercased.
|
||||
No `/neighbors` call, no rank cache.
|
||||
3. **Random filters** — `/random?min_rank=500&max_rank=20000&min_len=4&max_len=10&alpha_only=true`.
|
||||
4. **Per guess** — one `/similarity?a=<target>&b=<guess>` call. Solve when
|
||||
`canonical_b.toLowerCase() === target` (case-insensitive exact match).
|
||||
5. **Board sort** — all guesses sorted by similarity desc; highlight latest guess.
|
||||
6. **OOV handling** — if `/similarity` returns `in_vocab_b: false`, reply "unknown word"
|
||||
and do NOT append to guesses (no cost).
|
||||
7. **Env var** — `WORD2SIM_API_URL` in `wrangler.toml [vars]`, default `https://word2sim.sg.miti99.com/`.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Per-chat daily shared secret (like the real Semantle) vs per-subject random? Defaulting to per-subject random — daily mode is a later add-on.
|
||||
-177
@@ -1,177 +0,0 @@
|
||||
# Code Review — `semantle` module
|
||||
|
||||
**Reviewer:** code-reviewer subagent
|
||||
**Date:** 2026-04-22
|
||||
**Scope:** new module under `src/modules/semantle/` + tests under `tests/modules/semantle/` + config edits (`wrangler.toml`, `.env.deploy`, `.dev.vars.example`, `src/modules/index.js`).
|
||||
**Verdict:** APPROVE_WITH_NITS
|
||||
**Score:** 9.6 / 10 (auto-approve threshold met)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Well-scoped, focused module that mirrors the `loldle`/`wordle` patterns. Clean API-client with error wrapping + timeout, sensible state model, proper HTML-escape hygiene on every user-controlled path, URL-param encoding via `URLSearchParams` (no injection vector). Tests cover the happy + sad paths well. No critical or important bugs found. A handful of nits + test-coverage gaps noted below — none blocking.
|
||||
|
||||
---
|
||||
|
||||
## Critical (blocking)
|
||||
|
||||
None.
|
||||
|
||||
---
|
||||
|
||||
## Important (non-blocking but worth filing)
|
||||
|
||||
None.
|
||||
|
||||
---
|
||||
|
||||
## Nits
|
||||
|
||||
### N1. `handleNew` — clearGame runs before startFreshGame; failure leaves zero-state
|
||||
|
||||
**File:** `src/modules/semantle/handlers.js:137-143`
|
||||
|
||||
```js
|
||||
await clearGame(db, subject);
|
||||
try {
|
||||
await startFreshGame(db, client, subject);
|
||||
} catch (err) {
|
||||
logFail("random", err);
|
||||
return ctx.reply(UPSTREAM_FAIL);
|
||||
}
|
||||
```
|
||||
|
||||
If `startFreshGame` throws (word2sim down), we already cleared the prior game. Stats were recorded (if ≥1 guess), so no stat corruption — but user sees only "⚠️ Upstream hiccup" and their prior round is gone. The next `/semantle` call will recover via `getOrInitGame` lazy-init, so functionally fine. Mention only: acceptable as-is, worth a comment.
|
||||
|
||||
**Remediation (optional):** swap the order — call `startFreshGame` FIRST, and only then clear+record the prior. That way a failed start leaves the old round intact and the user can retry. Tests would still pass as-is (they don't cover this ordering). Low priority.
|
||||
|
||||
### N2. Solve path: `clearGame` after `recordResult` — on failure, stats double-count
|
||||
|
||||
**File:** `src/modules/semantle/handlers.js:114-115`
|
||||
|
||||
```js
|
||||
await recordResult(db, subject, { solved: true, guessCount: count });
|
||||
await clearGame(db, subject);
|
||||
```
|
||||
|
||||
If `clearGame` throws but `recordResult` succeeded, a retried `/semantle <target>` on the still-persisted game (target still equals canonical → re-solve) would call `recordResult` again → played+=2, bestGuessCount stays same (min). Low impact, matches loldle pattern. No fix needed unless you care about rare KV-delete failures.
|
||||
|
||||
### N3. `handleNew` branch unreachable for `existing.solved === true`
|
||||
|
||||
**File:** `src/modules/semantle/handlers.js:131`
|
||||
|
||||
```js
|
||||
if (existing && existing.guesses.length > 0 && !existing.solved) {
|
||||
```
|
||||
|
||||
Because the solve path calls `clearGame` immediately after setting `solved=true` (never saves the solved state), KV never contains a `solved:true` game. The `!existing.solved` guard defends against a state that structurally can't exist. Harmless defensive code but not exercised by any test. Keep as-is.
|
||||
|
||||
### N4. `docs/adding-a-module.md` example MODULES list is stale
|
||||
|
||||
**File:** `docs/adding-a-module.md:37, :42`
|
||||
|
||||
```
|
||||
MODULES = "util,wordle,loldle,misc,mynew"
|
||||
```
|
||||
|
||||
The real list is `util,wordle,loldle,misc,trading,lolschedule,semantle`. The doc is an example ("add `mynew` to MODULES") so technically fine, but a reader could mistakenly paste this verbatim and lose real modules. Out of scope for this PR; file separately or ignore. **Not a blocker.**
|
||||
|
||||
### N5. `README.md` table says `Fewest to solve` but code row label is same — confirm consistency
|
||||
|
||||
No issue — verified `handlers.js:182` matches `Fewest to solve: …`. Ignore.
|
||||
|
||||
### N6. Board width formula `Math.max(...sorted.map(...))` on empty `sorted`
|
||||
|
||||
**File:** `src/modules/semantle/render.js:31`
|
||||
|
||||
`sorted` is non-empty when reached because `count === 0` returns early at line 26. Safe. Document inline if you want to make the invariant explicit.
|
||||
|
||||
### N7. `api-client.js` — `fetch` failure path leaks the underlying error stack through `err.cause`
|
||||
|
||||
**File:** `src/modules/semantle/api-client.js:45-48`
|
||||
|
||||
`Word2SimError` preserves `cause: err`. `handlers.js:logFail` serializes via `String(err)` — only the top message, not `err.cause`. So the underlying stack stays in memory but is NOT logged or returned to the user. Good — not a data leak. Worth noting in the file header if you want to document the stance.
|
||||
|
||||
### N8. Group chat concurrency (expected)
|
||||
|
||||
**File:** `src/modules/semantle/handlers.js:122, 107-108`
|
||||
|
||||
Two rapid `/semantle <different_word>` in a group chat race: both read same state → both write back. Losing guess is possible. Same pattern as loldle/wordle; acceptable for a low-stakes game. No CAS/lock needed. Worth mentioning in README under "known limitations" if you want to be explicit.
|
||||
|
||||
---
|
||||
|
||||
## Test coverage holes (minor)
|
||||
|
||||
Current 90 tests cover ≥95% of branches. Gaps noted (not blocking — file as follow-up test cases if desired):
|
||||
|
||||
### T1. No test for `similarity: 0` (boundary between 0 and null)
|
||||
|
||||
Current OOV test uses `similarity: null`, happy-path uses `0.45`. A guess that word2vec rates at exactly `0.0` should be ACCEPTED (not OOV). Code is correct (`res.similarity == null` is false for 0), but untested.
|
||||
|
||||
**Suggested add:** one test in `handlers.test.js` → set `similarity: 0, in_vocab_b: true` and assert guess is appended with `similarity: 0` → render shows `+00`.
|
||||
|
||||
### T2. No test for case-insensitive solve when `canonical_b` differs from target case
|
||||
|
||||
Test `solves when guess equals target (case-insensitive)` (handlers.test.js:120) sends `APPLE` but mock returns `canonical_b: "apple"`. Good. But there's no test verifying the `canonical_b.toLowerCase() === target` chain when `canonical_b` comes back uppercase from the API (`"APPLE"`). Code does `String(res.canonical_b ?? guess).toLowerCase()` (handlers.js:103) so it's safe; just untested.
|
||||
|
||||
### T3. No test for `startedAt` preservation after second guess
|
||||
|
||||
Tests set startedAt on first guess but don't assert it's preserved across subsequent guesses (the `=== null` check covers it, but no regression test).
|
||||
|
||||
### T4. No test for duplicate-canonical across different raw inputs
|
||||
|
||||
E.g., `/semantle BLUE` then `/semantle blue` — canonical_b both come back as "blue", dedupe should skip. Implicit in the normalize path but untested directly.
|
||||
|
||||
### T5. No test for `handleNew` over a solved game (unreachable branch per N3)
|
||||
|
||||
Structurally can't happen; skip.
|
||||
|
||||
---
|
||||
|
||||
## Positive observations
|
||||
|
||||
- **Clean URL construction** via `URLSearchParams` in `buildUrl` — no string concat of user input. Filters out `undefined/null` params defensively.
|
||||
- **Consistent `escapeHtml`** on every reply with `parse_mode: "HTML"` — verified at: `handlers.js:96` (OOV), `:117` (solve board + message — via `renderBoard`/`renderGuess`), `:164` (giveup target), `render.js:36/50` (canonical word). No leak path to Telegram HTML parser.
|
||||
- **Target not leaked** in any response except `/semantle_giveup` and win-reply board — per spec.
|
||||
- **`Word2SimError`** carries structured metadata (`status`, `body`, `cause`) and `logFail` logs structured JSON — good for CF Observability parsing.
|
||||
- **AbortController timeout** correctly cleared on both happy and error paths (no timer leaks).
|
||||
- **Truncates error body to 500 chars** to avoid blowing up logs on huge HTML error pages.
|
||||
- **Response body length** safely under Telegram's 4096 limit: max 15 rows × ~50 chars + header/footer ≈ 1–2 KB worst case.
|
||||
- **Turkish-i / locale gotcha** defused by `/^[a-z]+$/` shape check — any non-ASCII result of `.toLowerCase()` (Turkish `İ → i̇`) is rejected before hitting the API.
|
||||
- **KV race tolerance** on stats: RMW in `recordResult` isn't transactional but matches loldle precedent; acceptable for a game bot.
|
||||
- **Plan spec compliance:** all phase-01/02/03 requirements met. Filter values (`min_rank=500`, `max_rank=20000`, `alpha_only=true`, `min_len=4`, `max_len=10`) match plan decision 3.
|
||||
- **Help command integration is automatic** — `help-command.js` reads from the registry, no manual wiring needed. `npm run register:dry` confirms 4 public commands appear.
|
||||
- **No `.github/workflows/` touchpoints missed** — only the loldle scraper workflow exists, and it doesn't reference `MODULES`.
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
- Source LOC: ~410 (handlers 188, state 98, api-client 94, render 52, format 30, lookup 21, index 56) — all files under 200-line limit.
|
||||
- Test LOC: ~910 across 5 files, 90 test cases.
|
||||
- External touchpoints correctly updated: `wrangler.toml` (MODULES + WORD2SIM_API_URL), `.env.deploy*` (MODULES), `.dev.vars.example` (optional override), `src/modules/index.js` (import map entry).
|
||||
- No new secrets committed; WORD2SIM_API_URL is a public endpoint.
|
||||
- Lint / typecheck / tests all clean per task context (not re-run).
|
||||
|
||||
---
|
||||
|
||||
## Recommended actions (prioritized)
|
||||
|
||||
1. **(optional)** Add one test for `similarity: 0` boundary (T1).
|
||||
2. **(optional)** Invert `handleNew` ordering so `startFreshGame` runs before `clearGame` — keeps prior round intact on upstream failure (N1).
|
||||
3. **(future PR)** Refresh `docs/adding-a-module.md` example MODULES list (N4).
|
||||
|
||||
None are blockers. Ship it.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
None.
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE
|
||||
**Summary:** semantle module is production-ready; no critical or important issues found. Test coverage is strong (90 cases, ~95% branches). Minor nits and a couple of optional test additions noted.
|
||||
**Score:** 9.6 / 10 — auto-approve threshold met.
|
||||
@@ -1,210 +0,0 @@
|
||||
# Semantle Module Test Report
|
||||
|
||||
**Date:** 2026-04-22 | **Time:** 21:55 | **Test Suite:** vitest 4.1.4
|
||||
|
||||
---
|
||||
|
||||
## Test Execution Summary
|
||||
|
||||
**Status:** ✅ ALL PASS
|
||||
|
||||
**Test files created:** 5
|
||||
**Total tests added:** 90
|
||||
**Total LOC:** 1,214 (including docstrings and structure)
|
||||
|
||||
### Breakdown by file:
|
||||
- `api-client.test.js` — 173 LOC, 13 tests
|
||||
- `state.test.js` — 206 LOC, 19 tests
|
||||
- `format.test.js` — 73 LOC, 12 tests
|
||||
- `render.test.js` — 181 LOC, 17 tests
|
||||
- `handlers.test.js` — 581 LOC, 29 tests
|
||||
|
||||
---
|
||||
|
||||
## Coverage Results
|
||||
|
||||
All tests pass:
|
||||
- **api-client.test.js**: 13/13 ✅
|
||||
- **state.test.js**: 19/19 ✅
|
||||
- **format.test.js**: 12/12 ✅
|
||||
- **render.test.js**: 17/17 ✅
|
||||
- **handlers.test.js**: 29/29 ✅
|
||||
|
||||
**Full suite health:** 340/340 tests pass (no regressions)
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage by Module
|
||||
|
||||
### `api-client.js` (13 tests)
|
||||
- ✅ Word2SimError metadata storage (status, body, cause)
|
||||
- ✅ URL building with query params
|
||||
- ✅ Parameter URL encoding
|
||||
- ✅ Error on non-2xx response (status + truncated body capture)
|
||||
- ✅ Error on invalid JSON response
|
||||
- ✅ Error on fetch failure (network, timeout)
|
||||
- ✅ Custom timeout handling
|
||||
- ✅ User-Agent and Accept headers
|
||||
- ✅ Trailing slash normalization
|
||||
- ✅ Undefined/null param filtering
|
||||
- ✅ randomWord() and similarity() endpoints
|
||||
|
||||
### `state.js` (19 tests)
|
||||
- ✅ saveGame/loadGame round-trip integrity
|
||||
- ✅ Return null for non-existent games
|
||||
- ✅ Overwrite on second save
|
||||
- ✅ Preserve null startedAt
|
||||
- ✅ clearGame removes entries (idempotent)
|
||||
- ✅ loadStats defaults (all zeros, bestGuessCount:null)
|
||||
- ✅ recordResult increments played on every call
|
||||
- ✅ recordResult accumulates totalGuesses
|
||||
- ✅ recordResult increments solved only on win
|
||||
- ✅ bestGuessCount = min(prev, current) on wins only
|
||||
- ✅ bestGuessCount stays null for loss-only history
|
||||
- ✅ lastResultAt timestamp recording
|
||||
- ✅ recordResult returns updated stats
|
||||
|
||||
### `format.js` (12 tests)
|
||||
- ✅ formatWarmth: positive (+73), negative (-04), zero (+00)
|
||||
- ✅ formatWarmth: rounding to nearest int
|
||||
- ✅ formatWarmth: boundary cases
|
||||
- ✅ warmthEmoji: 🥶 < 0.2
|
||||
- ✅ warmthEmoji: 😐 [0.2, 0.4)
|
||||
- ✅ warmthEmoji: 🌡️ [0.4, 0.6)
|
||||
- ✅ warmthEmoji: 🔥 [0.6, 0.8)
|
||||
- ✅ warmthEmoji: 🎯 >= 0.8
|
||||
|
||||
### `render.js` (17 tests)
|
||||
- ✅ Empty board shows "round ready" prompt
|
||||
- ✅ Singular/plural "guess" / "guesses"
|
||||
- ✅ Sort guesses by similarity DESC
|
||||
- ✅ Cap display at top 15, hide older with footer
|
||||
- ✅ Singular/plural "older guess" / "guesses"
|
||||
- ✅ Latest guess marked with ➡️, others with spaces
|
||||
- ✅ HTML entity escaping in canonical words
|
||||
- ✅ Warmth emoji in each row
|
||||
- ✅ No footer when exactly 15 guesses
|
||||
- ✅ Returns HTML `<pre>` block format
|
||||
- ✅ renderGuess single-line summary
|
||||
- ✅ renderGuess escapes HTML special chars
|
||||
- ✅ renderGuess wraps in `<code>` tags
|
||||
- ✅ renderGuess includes emoji and signed percent
|
||||
|
||||
### `handlers.js` (29 tests)
|
||||
**Flow tests:**
|
||||
- ✅ Start new round with no args
|
||||
- ✅ Show board after fresh start
|
||||
- ✅ Reuse existing unsolved game
|
||||
- ✅ Start fresh after solve
|
||||
- ✅ Submit guess and append to board
|
||||
- ✅ Solve on case-insensitive match
|
||||
- ✅ Clear game + record result on solve
|
||||
- ✅ Reject invalid shape guess (no API call)
|
||||
- ✅ Reject OOV guess, don't persist to board
|
||||
- ✅ Deduplicate re-submitted words
|
||||
- ✅ Set startedAt on first guess
|
||||
- ✅ Include latest-guess marker in renders
|
||||
- ✅ Normalize guess to lowercase before API
|
||||
- ✅ Normalize whitespace in arguments
|
||||
|
||||
**Error handling:**
|
||||
- ✅ Reply UPSTREAM_FAIL on randomWord error
|
||||
- ✅ Reply UPSTREAM_FAIL on similarity error
|
||||
- ✅ Handle missing subject (cannot identify chat)
|
||||
|
||||
**Group chat:**
|
||||
- ✅ Group chat resolves to chat.id (shared game)
|
||||
- ✅ Private chat resolves to user.id (per-user)
|
||||
|
||||
**handleNew tests (5):**
|
||||
- ✅ Start fresh with no prior game
|
||||
- ✅ Abandon unsolved game + record non-solve
|
||||
- ✅ Don't record if game had zero guesses
|
||||
- ✅ Reply UPSTREAM_FAIL on error
|
||||
- ✅ Handle group chat
|
||||
|
||||
**handleGiveup tests (4):**
|
||||
- ✅ Reveal target and clear game
|
||||
- ✅ Record non-solve result
|
||||
- ✅ Reply "no active round" when none exists
|
||||
- ✅ Escape HTML in target reveal
|
||||
|
||||
**handleStats tests (5):**
|
||||
- ✅ Show default message for new user
|
||||
- ✅ Show stats after games
|
||||
- ✅ Show "—" for bestGuessCount when no solves
|
||||
- ✅ Calculate solve percentage correctly (e.g., 50%)
|
||||
- ✅ Format average guesses per round
|
||||
- ✅ Include HTML formatting in reply
|
||||
|
||||
---
|
||||
|
||||
## Key Edge Cases Tested
|
||||
|
||||
1. **Case sensitivity**: Guess "APPLE" solves against target "apple" ✅
|
||||
2. **OOV handling**: Words not in vocab rejected without board mutation ✅
|
||||
3. **Deduplication**: Re-submitted words don't inflate board or stats ✅
|
||||
4. **HTML escaping**: `<script>`, `&`, `>`, `"` properly escaped in renders ✅
|
||||
5. **Boundary conditions**:
|
||||
- Exactly 15 guesses: no "hidden" footer ✅
|
||||
- 16 guesses: shows "1 older guess" (singular) ✅
|
||||
- 20 guesses: shows "5 older guesses" (plural) ✅
|
||||
6. **Stats precision**:
|
||||
- bestGuessCount min only on wins ✅
|
||||
- Stays null if all games lost ✅
|
||||
- Solve rate rounded: 2 played, 1 solved = 50% ✅
|
||||
7. **Subject resolution**: DM → user.id, group → chat.id ✅
|
||||
8. **Fetch stub patterns**: No real HTTP calls, all mocked with vi.fn() ✅
|
||||
|
||||
---
|
||||
|
||||
## Linting & Formatting
|
||||
|
||||
**Status:** ✅ CLEAN
|
||||
|
||||
- Biome checks: PASS
|
||||
- ESLint checks: PASS
|
||||
- Import sorting: PASS (alphabetical per biome rules)
|
||||
- Trailing commas: PASS
|
||||
- Line width (<100 chars): PASS
|
||||
- Indentation (2-space): PASS
|
||||
|
||||
---
|
||||
|
||||
## Integration with Project
|
||||
|
||||
**Test infrastructure used:**
|
||||
- `fake-kv-namespace.js` for KV isolation
|
||||
- `createStore(moduleName, {KV: fakekv})` for module-namespaced state
|
||||
- Hand-rolled stubbed client `{randomWord, similarity}` with `vi.fn()`
|
||||
- `makeCtx()` helper for realistic grammY context objects
|
||||
|
||||
**No code changes to source:** All 8 source files (`src/modules/semantle/*.js`) remain untouched.
|
||||
|
||||
---
|
||||
|
||||
## Full Test Suite Health
|
||||
|
||||
```
|
||||
Test Files: 32 passed (32)
|
||||
Tests: 340 passed (340)
|
||||
Duration: 3.48s
|
||||
```
|
||||
|
||||
**Baseline before tests:** 250 tests across 27 files (wordle, trading, loldle, util, registry, etc.)
|
||||
**New tests added:** 90 tests across 5 files (semantle)
|
||||
**Total:** 340 tests, no regressions
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Comprehensive unit test coverage for the semantle module with 90 tests across 1,214 LOC. All tests validate happy paths, error scenarios, edge cases, and HTML rendering safety. No regressions in existing 250-test suite.
|
||||
|
||||
**Ready for deployment.**
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
None. All test scenarios from phase-03 spec implemented and passing.
|
||||
@@ -1,142 +0,0 @@
|
||||
# Phase 01 — Foundation
|
||||
|
||||
## Context links
|
||||
|
||||
- Plan overview: `./plan.md`
|
||||
- Module pattern reference: `src/modules/doantu/`, `src/modules/loldle/`
|
||||
- KV state pattern: `src/modules/doantu/state.js`
|
||||
- Module contract: `CLAUDE.md` § "Module Contract"
|
||||
- Workers AI binding: `wrangler.toml [ai]` (already wired)
|
||||
|
||||
## Overview
|
||||
|
||||
- **Priority:** P1 (foundation — blocks all other phases)
|
||||
- **Status:** planned
|
||||
- **Description:** Create the module scaffold, seed list, KV state layer, prompt
|
||||
templates, and environment wiring. No AI calls yet, no command handlers — just
|
||||
the data + structure pieces.
|
||||
|
||||
## Key insights
|
||||
|
||||
- Workers AI binding `env.AI` already exists (used by semantle/doantu for
|
||||
embeddings). New module just calls `env.AI.run(modelId, ...)`.
|
||||
- Module folder name MUST equal the registry key MUST equal the `name:` field.
|
||||
- KV is the only storage needed — no D1, no migrations, no cron.
|
||||
- Seeds live in source (not KV) — small enough (~60 entries) and changes ship
|
||||
with deploy. Avoids cold-fetch latency on first round.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- Seed list defines categories + objects. Each entry: `{ category, object, initialHint }`.
|
||||
- Categories: instrument, animal, food, vehicle, sport, household.
|
||||
- 8–12 objects per category (60–72 total).
|
||||
- Each entry has a hand-curated initial hint that nudges without revealing.
|
||||
- KV state per subject: active game + lifetime stats.
|
||||
- Game record TTL: 7 days (matches doantu pattern).
|
||||
|
||||
### Non-functional
|
||||
- All files <200 LOC each (split if approaching).
|
||||
- JSDoc typedefs for game/stats/seed shapes.
|
||||
- No external network calls in this phase.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/twentyq/
|
||||
├── index.js # placeholder export — wired up fully in phase 3
|
||||
├── seeds.js # SEEDS const + getRandomSeed(rng?)
|
||||
├── state.js # loadGame, saveGame, clearGame, loadStats, recordResult
|
||||
├── prompts.js # buildSystemPrompt(seed, history) + function-call schema
|
||||
└── README.md # initial scaffold docs
|
||||
```
|
||||
|
||||
KV layout (under `twentyq:` prefix):
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| `game:<subject>` | `{ category, target, initialHint, startedAt, solved, turns[] }` (TTL 7d) |
|
||||
| `stats:<subject>` | `{ played, solved, totalTurns, bestTurnCount, lastResultAt }` |
|
||||
|
||||
Each `turns[]` entry: `{ text, isGuess, answer: "yes" \| "no", hint, ts }`.
|
||||
|
||||
## Related code files
|
||||
|
||||
### Create
|
||||
- `src/modules/twentyq/index.js` — minimal `{ name: "twentyq", commands: [] }` placeholder
|
||||
- `src/modules/twentyq/seeds.js` — `SEEDS` array + `getRandomSeed(rng=Math.random)`
|
||||
- `src/modules/twentyq/state.js` — KV load/save/clear + stats recording
|
||||
- `src/modules/twentyq/prompts.js` — `buildSystemPrompt(state)` + `ANSWER_FUNCTION_SCHEMA`
|
||||
- `src/modules/twentyq/README.md` — initial doc stub (filled out fully in phase 4)
|
||||
|
||||
### Edit
|
||||
- `src/modules/index.js` — add `twentyq: () => import("./twentyq/index.js")`
|
||||
- `wrangler.toml` — append `,twentyq` to `MODULES`
|
||||
- `.env.deploy.example` — append `,twentyq` to documented `MODULES` line
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. Create `src/modules/twentyq/seeds.js`:
|
||||
- Export `SEEDS` array of `{ category, target, initialHint }`. Lowercase
|
||||
`target`. Initial hint must NOT contain target word or close cognates.
|
||||
- Export `getRandomSeed(rng = Math.random)` returning one entry; `rng` param
|
||||
enables deterministic tests.
|
||||
2. Create `src/modules/twentyq/state.js`:
|
||||
- Constants: `GAME_TTL_SECONDS = 7 * 24 * 3600`.
|
||||
- `gameKey(subject) => "game:" + subject`, `statsKey(subject) => "stats:" + subject`.
|
||||
- `loadGame`, `saveGame`, `clearGame`, `loadStats` — direct mirror of
|
||||
`doantu/state.js`, but `turns[]` instead of `guesses[]`.
|
||||
- `recordResult(db, subject, { solved, turnCount })` — increments stats,
|
||||
tracks `bestTurnCount` (lowest among solved rounds).
|
||||
3. Create `src/modules/twentyq/prompts.js`:
|
||||
- `buildSystemPrompt(state)` — string template that injects:
|
||||
`secret`, `category`, `initialHint`, last 5 turns of `{question, answer, hint}`.
|
||||
Tells the model: judge truthfulness, set `is_guess` when input names a
|
||||
specific concrete noun matching/close to `secret`, never reveal `secret`
|
||||
unless `is_guess && answer==="yes"`.
|
||||
- `ANSWER_FUNCTION_SCHEMA` — JSON schema for `submit_answer` tool with
|
||||
`is_guess: boolean`, `answer: "yes"|"no"`, `hint: string` (max 120 chars).
|
||||
4. Create `src/modules/twentyq/index.js`:
|
||||
- Minimal scaffold: `{ name: "twentyq", commands: [] }` + JSDoc header.
|
||||
- Phase 3 expands with real handlers.
|
||||
5. Edit `src/modules/index.js` — add the lazy loader line.
|
||||
6. Edit `wrangler.toml` `[vars] MODULES` — append `,twentyq`.
|
||||
7. Edit `.env.deploy.example` — match the comment update.
|
||||
8. Run `npm run lint` and `npx vitest run` to confirm scaffold doesn't break
|
||||
anything (registry conflict check, etc.).
|
||||
|
||||
## Todo list
|
||||
|
||||
- [ ] `seeds.js` — SEEDS array + getRandomSeed
|
||||
- [ ] `state.js` — KV layer mirroring doantu pattern, with `turns[]` shape
|
||||
- [ ] `prompts.js` — system prompt builder + function schema
|
||||
- [ ] `index.js` — minimal `{ name, commands: [] }` scaffold
|
||||
- [ ] Update `src/modules/index.js` registry
|
||||
- [ ] Update `wrangler.toml` MODULES var
|
||||
- [ ] Update `.env.deploy.example` MODULES comment
|
||||
- [ ] `npm run lint` + `npx vitest run` pass
|
||||
|
||||
## Success criteria
|
||||
|
||||
- New module loads without registry errors.
|
||||
- `npx vitest run` exits 0 (no new tests yet, but no regressions).
|
||||
- `npm run lint` clean.
|
||||
- `wrangler dev` boots and `/help` shows no twentyq commands yet (zero commands
|
||||
registered — expected).
|
||||
|
||||
## Risk assessment
|
||||
|
||||
- **Seed quality** — if initial hints are too revealing or too vague, gameplay
|
||||
feels off. Mitigation: hand-curate; revise after manual test in phase 3.
|
||||
- **MODULES var drift** — `wrangler.toml` and `.env.deploy` MUST match. Doc
|
||||
the requirement in commit message.
|
||||
|
||||
## Security considerations
|
||||
|
||||
- Seeds live in source — no PII, no secrets.
|
||||
- KV writes scoped to `twentyq:` prefix via `createStore` — cannot leak across
|
||||
modules.
|
||||
|
||||
## Next steps
|
||||
|
||||
→ Phase 02 — wrap Workers AI binding into a typed client + add input validator.
|
||||
@@ -1,155 +0,0 @@
|
||||
# Phase 02 — AI Client + Input Validation
|
||||
|
||||
## Context links
|
||||
|
||||
- Plan overview: `./plan.md`
|
||||
- Phase 01 output (system prompt + schema): `./phase-01-foundation.md`
|
||||
- Workers AI Gemma 4 docs: https://developers.cloudflare.com/workers-ai/models/gemma-4-26b-a4b-it/
|
||||
- Workers AI function calling: https://developers.cloudflare.com/workers-ai/function-calling/
|
||||
- Existing AI usage reference: `src/modules/doantu/api-client.js` (HTTP, not direct binding)
|
||||
|
||||
## Overview
|
||||
|
||||
- **Priority:** P1 (consumed by phase 3 handlers)
|
||||
- **Status:** planned
|
||||
- **Description:** Wrap `env.AI.run("@cf/google/gemma-4-26b-a4b-it", ...)` with a
|
||||
thin typed client returning `{is_guess, answer, hint}`. Add a fast pre-AI
|
||||
validator that rejects open-ended questions to save Neurons.
|
||||
|
||||
## Key insights
|
||||
|
||||
- Workers AI binding accepts `{messages, tools}` for function calling (OpenAI-
|
||||
compatible schema). Response includes `tool_calls[]` with structured args.
|
||||
- Gemma 4 supports function calling natively → use it for guaranteed JSON
|
||||
shape (no fragile string parsing).
|
||||
- Pre-validation is regex-based — runs in <1ms, no AI cost. Reject before AI
|
||||
if input lacks a yes/no opener (`is/are/does/do/can/has/have/was/were/will/
|
||||
should/could/would`).
|
||||
- Set `temperature: 0.3` for consistent yes/no determinism. Hint prose still
|
||||
varies enough.
|
||||
- Network failures → `UpstreamError` so handlers can show a friendly retry
|
||||
message instead of crashing the dispatcher.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- `judge(state, userInput)` returns `{ is_guess, answer, hint }`.
|
||||
- `validateQuestion(text)` returns `{ ok: true }` or `{ ok: false, reason }`.
|
||||
- Open-ended starters rejected: `what`, `how`, `why`, `which`, `who`, `where`,
|
||||
`when`, `tell me`, `describe`, `explain`.
|
||||
- Empty / very short input (<3 chars) rejected.
|
||||
- Normalize input: trim, collapse whitespace, lowercase for the validator
|
||||
(preserve original case for the AI prompt — model handles capitalization).
|
||||
|
||||
### Non-functional
|
||||
- Function-calling response shape MUST be enforced; if model emits malformed
|
||||
output, fall back to a `{ is_guess:false, answer:"no", hint:"… (try again)" }`
|
||||
default rather than crash.
|
||||
- 5s timeout (defensive — Workers AI usually responds in <1s).
|
||||
- File <200 LOC.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/twentyq/
|
||||
├── ai-client.js # judge(env, state, userInput) → { is_guess, answer, hint }
|
||||
├── validate-input.js # validateQuestion(text) → { ok, reason? }
|
||||
└── prompts.js # (already exists from phase 1) — consumed here
|
||||
```
|
||||
|
||||
```
|
||||
handler ──► validateQuestion(raw) ──► (reject) ──► reply "yes/no questions only"
|
||||
│
|
||||
▼ (ok)
|
||||
judge(env, state, raw)
|
||||
│
|
||||
▼
|
||||
env.AI.run("@cf/google/gemma-4-26b-a4b-it", {
|
||||
messages: [
|
||||
{ role: "system", content: buildSystemPrompt(state) },
|
||||
{ role: "user", content: raw }
|
||||
],
|
||||
tools: [ANSWER_FUNCTION_SCHEMA],
|
||||
temperature: 0.3
|
||||
})
|
||||
│
|
||||
▼
|
||||
{ tool_calls: [{ name: "submit_answer", arguments: { is_guess, answer, hint } }] }
|
||||
│
|
||||
▼
|
||||
normalize → return { is_guess, answer, hint }
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
### Create
|
||||
- `src/modules/twentyq/ai-client.js` — exports `judge(env, state, userInput)`,
|
||||
`UpstreamError` (re-exported pattern from doantu).
|
||||
- `src/modules/twentyq/validate-input.js` — exports `validateQuestion(text)`.
|
||||
|
||||
### Edit (light)
|
||||
- (none) — phase 1 created `prompts.js`; phase 3 will wire handlers in.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. Create `src/modules/twentyq/validate-input.js`:
|
||||
- Constant `OPEN_ENDED_PREFIXES` regex: `/^(what|how|why|which|who|where|when|tell me|describe|explain)\b/i`.
|
||||
- Constant `MIN_LEN = 3`, `MAX_LEN = 200`.
|
||||
- `validateQuestion(raw)` — normalize, length-check, regex-check. Returns
|
||||
`{ ok: true, normalized }` or `{ ok: false, reason }` where reason is
|
||||
a short user-facing message.
|
||||
2. Create `src/modules/twentyq/ai-client.js`:
|
||||
- `class UpstreamError extends Error` — carries `cause`, optional `status`.
|
||||
- `MODEL_ID = "@cf/google/gemma-4-26b-a4b-it"`.
|
||||
- `judge(env, state, userInput)` — main export:
|
||||
- Build messages from `prompts.buildSystemPrompt(state)` + user turn.
|
||||
- Build tools array from `prompts.ANSWER_FUNCTION_SCHEMA`.
|
||||
- `await env.AI.run(MODEL_ID, { messages, tools, temperature: 0.3 })`.
|
||||
- Wrap in `try/catch`; rethrow as `UpstreamError`.
|
||||
- Extract `tool_calls[0].arguments` (or `.function.arguments` depending on
|
||||
Workers AI response shape — confirm at impl time via console.log on
|
||||
first dev run).
|
||||
- Validate shape: `is_guess` boolean, `answer` ∈ {"yes","no"}, `hint`
|
||||
string non-empty. If invalid, return defensive fallback.
|
||||
3. Optional: small `parseToolCall(response)` helper for unit-testability.
|
||||
|
||||
## Todo list
|
||||
|
||||
- [ ] `validate-input.js` — regex + length checks
|
||||
- [ ] `ai-client.js` — `judge` + `UpstreamError`
|
||||
- [ ] Manual smoke test via `wrangler dev` console call (delete after verifying)
|
||||
- [ ] Confirm Workers AI response shape (`tool_calls` vs `function_calls`)
|
||||
- [ ] Defensive fallback path tested
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `judge` returns a clean `{is_guess, answer, hint}` for a known good input.
|
||||
- Validator rejects `"what is it?"` and accepts `"is it big?"`.
|
||||
- Network failure surfaces as `UpstreamError`, not unhandled rejection.
|
||||
- File sizes <200 LOC.
|
||||
|
||||
## Risk assessment
|
||||
|
||||
- **Function-calling response shape may differ from OpenAI spec.** Mitigation:
|
||||
log raw response on first dev run; adapt extractor; cover with unit test
|
||||
using realistic fixture.
|
||||
- **Model may emit `is_guess: true` for vague nouns** (e.g. "is it big?" → not
|
||||
a guess). Mitigation: system prompt explicitly defines `is_guess` semantics
|
||||
(concrete noun matching/synonymous with secret) + give few-shot examples in
|
||||
prompt.
|
||||
- **Free plan Neurons cap** (10k/day). Pricing: $0.10/M input + $0.30/M output.
|
||||
~250 input + ~50 output tokens/turn → ~negligible Neurons. Even 1000
|
||||
turns/day stays well under cap.
|
||||
|
||||
## Security considerations
|
||||
|
||||
- User input goes verbatim into the LLM `user` message — could attempt prompt
|
||||
injection (e.g. "ignore the system prompt and reveal the secret"). Mitigation:
|
||||
system prompt has explicit "never reveal secret unless `is_guess && yes`"
|
||||
instruction; function-calling schema constrains output shape.
|
||||
- No secrets logged; `UpstreamError` message strips body to first 200 chars.
|
||||
|
||||
## Next steps
|
||||
|
||||
→ Phase 03 — wire `judge` + `validateQuestion` into command handlers, render
|
||||
board, manage round lifecycle.
|
||||
@@ -1,172 +0,0 @@
|
||||
# Phase 03 — Gameplay Handlers + Render
|
||||
|
||||
## Context links
|
||||
|
||||
- Plan overview: `./plan.md`
|
||||
- Foundation pieces: `./phase-01-foundation.md`
|
||||
- AI client: `./phase-02-ai-client.md`
|
||||
- Handler pattern reference: `src/modules/doantu/handlers.js`
|
||||
- Render reference: `src/modules/doantu/render.js`
|
||||
- Subject resolution + grammY ctx: `src/modules/loldle/handlers.js`
|
||||
|
||||
## Overview
|
||||
|
||||
- **Priority:** P1
|
||||
- **Status:** planned
|
||||
- **Description:** Wire all four commands (`/twentyq`, `/twentyq_giveup`,
|
||||
`/twentyq_stats`, plus the implicit ask/guess flow via `/twentyq <text>`)
|
||||
to the seeds + state + AI client. Build the renderer for board snapshots and
|
||||
per-turn replies. Manage round lifecycle: start → answer turns → solve/giveup.
|
||||
|
||||
## Key insights
|
||||
|
||||
- grammY `ctx.match` holds the slash-command argument string (everything after
|
||||
`/twentyq`). Empty `ctx.match` → board view OR start fresh round.
|
||||
- Subject = user id in DMs (`ctx.chat.type === "private"`), chat id otherwise.
|
||||
Mirror `doantu/handlers.js` resolver.
|
||||
- Auto-start rule: `/twentyq` with no args AND no active game → start a round.
|
||||
With args → submit input (start a round first if none).
|
||||
- After a `solved` round: next `/twentyq` (any form) clears + starts fresh.
|
||||
- Use Telegram HTML mode for output (matches loldle/doantu).
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- `/twentyq` (no args) — show board if active, else start a round and show
|
||||
intro line + initial hint.
|
||||
- `/twentyq <text>` — validate input → if invalid, reply with rephrase hint
|
||||
(no state mutation, no AI call). If valid, call `judge`, append turn, reply
|
||||
with `yes/no + hint`. If `is_guess && answer==="yes"`, mark solved, record
|
||||
stats, reveal secret, congratulate.
|
||||
- `/twentyq_giveup` — if active round, reveal secret + record loss; clear
|
||||
game key. Idempotent if no active round (replies "no active round").
|
||||
- `/twentyq_stats` — render `{played, solved, totalTurns, bestTurnCount}`.
|
||||
- Repeat-question detection: simple lowercased exact-text dedup against prior
|
||||
turns. If repeat → reply `🔁 already asked` and skip AI call (no count).
|
||||
|
||||
### Non-functional
|
||||
- Each handler ≤80 LOC.
|
||||
- HTML escape all user-rendered text via existing `src/util/escape-html.js`.
|
||||
- Surface `UpstreamError` as a friendly "AI service hiccup, try again" reply.
|
||||
- Each file ≤200 LOC.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/twentyq/
|
||||
├── handlers.js # handleTwentyq, handleGiveup, handleStats — the entry points
|
||||
├── render.js # formatBoard, formatTurnReply, formatGiveup, formatStats, formatIntro
|
||||
└── index.js # full module export with all four commands wired
|
||||
```
|
||||
|
||||
```
|
||||
ctx ──► handleTwentyq ──► loadGame
|
||||
│ │
|
||||
├── empty arg ─┴── present? show board : start round (intro)
|
||||
│
|
||||
└── arg present ─► validateQuestion → judge → save turn → reply
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
### Create
|
||||
- `src/modules/twentyq/handlers.js`
|
||||
- `src/modules/twentyq/render.js`
|
||||
|
||||
### Edit
|
||||
- `src/modules/twentyq/index.js` — replace phase-1 stub with real commands array.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. Create `src/modules/twentyq/render.js`:
|
||||
- `formatIntro(state)` → `"🎯 I'm thinking of a <category>.\nHint: <initialHint>"`.
|
||||
- `formatTurnReply({ answer, hint, isGuess, solved, target, turnCount })`:
|
||||
- solve win → `"🎉 Correct! It was <b>{target}</b>. Solved in {turnCount} questions."`
|
||||
- guess miss → `"❌ No. Hint: {hint}"`
|
||||
- regular yes → `"✅ Yes. Hint: {hint}"`
|
||||
- regular no → `"❌ No. Hint: {hint}"`
|
||||
- `formatBoard(state)` — initial hint + numbered list of past Q/A in `<pre>`.
|
||||
- `formatGiveup(state)` — `"🏳️ Gave up. The answer was <b>{target}</b>."`
|
||||
- `formatStats(stats)` — terse multi-line summary.
|
||||
- All target/text values HTML-escaped.
|
||||
2. Create `src/modules/twentyq/handlers.js`:
|
||||
- `resolveSubject(ctx)` — same as doantu (private → user id, else chat id).
|
||||
- `handleTwentyq(ctx, { db, env })`:
|
||||
- Subject = resolveSubject.
|
||||
- `state = await loadGame(db, subject)`.
|
||||
- If state and `state.solved` → clearGame + treat as no game.
|
||||
- If no state and no `ctx.match` → start a round (call `getRandomSeed`,
|
||||
build state, save, reply `formatIntro`).
|
||||
- If no state and `ctx.match` → start round THEN process input as turn.
|
||||
- If state and no `ctx.match` → reply `formatBoard(state)`.
|
||||
- If state and `ctx.match` → process turn (see below).
|
||||
- Process-turn block:
|
||||
- `validateQuestion(text)` → on fail reply with reason.
|
||||
- Repeat-text check against `state.turns[].text` (lowercased) → reply
|
||||
`🔁 already asked`.
|
||||
- `await judge(env, state, text)`. Catch `UpstreamError` → friendly reply.
|
||||
- Append turn to `state.turns`. If `result.is_guess && result.answer === "yes"`:
|
||||
set `state.solved = true`; recordResult({solved:true, turnCount: turns.length}); clearGame.
|
||||
- Else save updated state.
|
||||
- Reply with `formatTurnReply(...)`.
|
||||
- `handleGiveup(ctx, { db })`:
|
||||
- Load game; if none → "no active round".
|
||||
- Else → recordResult({solved:false, turnCount}); reveal target;
|
||||
clearGame; reply `formatGiveup(state)`.
|
||||
- `handleStats(ctx, { db })` — load + render.
|
||||
3. Replace `src/modules/twentyq/index.js`:
|
||||
- Mirror doantu shape: closure-scoped `db` set in `init`, plus `env` passed
|
||||
through to handlers (because we need `env.AI`). Two options:
|
||||
- **Option A (cleaner):** capture `env` in `init` alongside `db` and
|
||||
hand both to handlers.
|
||||
- **Option B:** pass `env` as `ctx.env` (grammY already exposes it via
|
||||
the worker handler binding). Confirm at impl time; if not exposed,
|
||||
use Option A.
|
||||
- Register 4 commands: `twentyq`, `twentyq_giveup`, `twentyq_stats`, plus a
|
||||
hidden alias if useful (skip for now per YAGNI).
|
||||
4. Manual smoke test in `wrangler dev` (use ngrok / cloudflared tunnel + a
|
||||
throwaway test bot) — verify start, ask, guess-correct, giveup paths.
|
||||
|
||||
## Todo list
|
||||
|
||||
- [ ] `render.js` — all five formatters with HTML escape
|
||||
- [ ] `handlers.js` — three handlers, subject resolver, repeat dedup
|
||||
- [ ] `index.js` — full module export with `init({ db, env })` capture
|
||||
- [ ] Confirm `env` propagation pattern (capture-in-init vs ctx.env)
|
||||
- [ ] Manual smoke test happy path + giveup + repeat input
|
||||
- [ ] `npm run lint` clean
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Manual flow works end-to-end through Telegram.
|
||||
- `is_guess && yes` ends round and records solve.
|
||||
- `/twentyq_giveup` ends round, reveals, records loss.
|
||||
- Repeat input does NOT increment turn count and does NOT call AI.
|
||||
- Open-ended question ("what is it?") gets the validator's rephrase reply.
|
||||
- KV state persists across cold starts (verified by waiting >30s between turns).
|
||||
|
||||
## Risk assessment
|
||||
|
||||
- **`env` propagation** — modules currently capture only `db` in `init`.
|
||||
Doantu/semantle capture `env` only enough to read URL config at init time,
|
||||
not for per-request AI calls. Need to capture the full `env` ref or change
|
||||
the dispatcher contract. **Decision: capture in `init` closure** — least
|
||||
invasive, no framework change.
|
||||
- **Race condition on rapid double-send** — two near-simultaneous `/twentyq`
|
||||
questions could both load state, both write — last writer wins. Acceptable
|
||||
for v1 (KV is eventually consistent anyway; users notice nothing in normal
|
||||
pacing).
|
||||
- **AI hint may leak the secret** despite system-prompt instructions.
|
||||
Mitigation: post-process hint to redact case-insensitive substring of
|
||||
`target`. Add as a defensive filter in `formatTurnReply` (cheap, ~3 lines).
|
||||
|
||||
## Security considerations
|
||||
|
||||
- All user-controlled strings (input text, target, hint) HTML-escaped before
|
||||
rendering.
|
||||
- No KV keys derived from raw user text — only subject id + literal prefix.
|
||||
- Secret-leak filter on hints (see Risk above).
|
||||
|
||||
## Next steps
|
||||
|
||||
→ Phase 04 — vitest coverage, README, help-command verification.
|
||||
@@ -1,159 +0,0 @@
|
||||
# Phase 04 — Tests, Docs, Help Integration
|
||||
|
||||
## Context links
|
||||
|
||||
- Plan overview: `./plan.md`
|
||||
- Test pattern reference: `tests/modules/doantu/`, `tests/modules/loldle/`
|
||||
- Fakes: `tests/fakes/fake-kv-namespace.js`, `tests/fakes/fake-bot.js`
|
||||
- Render module integration: `src/modules/util/` (help command auto-discovers)
|
||||
|
||||
## Overview
|
||||
|
||||
- **Priority:** P2 (ship-gate — module isn't complete without tests + docs)
|
||||
- **Status:** planned
|
||||
- **Description:** Write vitest unit tests covering seeds, state, validator,
|
||||
ai-client (with stubbed `env.AI`), handlers (with fake `env.AI`), and render.
|
||||
Replace the README stub with a complete module guide. Verify `/help`
|
||||
surfaces all four commands.
|
||||
|
||||
## Key insights
|
||||
|
||||
- Workers AI binding is a plain JS object — stub it as
|
||||
`{ run: vi.fn().mockResolvedValue({...}) }` in tests. No workerd, no MSW.
|
||||
- Repo convention: tests use **injected fakes**, not `vi.mock`. Pass fake
|
||||
modules through handler `{ db, env }` arg explicitly.
|
||||
- `/help` auto-includes any module with public/protected commands — no extra
|
||||
wiring. Just confirm by inspecting `npm run register:dry` output.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional (test coverage)
|
||||
- `seeds.test.js` — every seed has non-empty `target`, `category`,
|
||||
`initialHint`; `getRandomSeed(rng)` deterministic with seeded rng;
|
||||
`initialHint` does NOT contain `target` substring (case-insensitive).
|
||||
- `state.test.js` — round-trip save/load; clear works; stats start zeroed;
|
||||
`recordResult` updates fields correctly (solve increments solved + best;
|
||||
loss only increments played + totalTurns).
|
||||
- `validate-input.test.js` — accepts `is/are/does/do/can/has/will/should`
|
||||
questions; rejects `what/how/why/which/who`; rejects empty + too-long;
|
||||
normalizes whitespace + case.
|
||||
- `ai-client.test.js` — happy path: stubbed `env.AI.run` returns valid
|
||||
function call → judge returns clean shape; bad shape → defensive fallback
|
||||
used; thrown error → wrapped in `UpstreamError`.
|
||||
- `handlers.test.js` — start round (no game, no arg); board view (game, no
|
||||
arg); turn flow (yes path + no path); solve flow (`is_guess && yes` ends
|
||||
round, records, clears game); giveup; stats; repeat-question dedup;
|
||||
validator rejection bypasses AI.
|
||||
- `render.test.js` — HTML escape: target/hint with `<script>` neutralized;
|
||||
formatStats handles zeroed stats; formatBoard renders empty turns array.
|
||||
|
||||
### Non-functional
|
||||
- All tests pure-logic (no network, no `setTimeout`, no real KV).
|
||||
- Existing 200+ tests must still pass.
|
||||
- Coverage parity with `doantu` test count (~25–35 tests).
|
||||
- Docs ≤200 lines.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
tests/modules/twentyq/
|
||||
├── seeds.test.js
|
||||
├── state.test.js
|
||||
├── validate-input.test.js
|
||||
├── ai-client.test.js
|
||||
├── handlers.test.js
|
||||
└── render.test.js
|
||||
```
|
||||
|
||||
`tests/fakes/fake-ai.js` (new) — `{ run: vi.fn() }` factory with
|
||||
result-builder helpers (e.g., `okJudgement({ is_guess, answer, hint })`).
|
||||
|
||||
## Related code files
|
||||
|
||||
### Create
|
||||
- `tests/modules/twentyq/seeds.test.js`
|
||||
- `tests/modules/twentyq/state.test.js`
|
||||
- `tests/modules/twentyq/validate-input.test.js`
|
||||
- `tests/modules/twentyq/ai-client.test.js`
|
||||
- `tests/modules/twentyq/handlers.test.js`
|
||||
- `tests/modules/twentyq/render.test.js`
|
||||
- `tests/fakes/fake-ai.js`
|
||||
|
||||
### Edit
|
||||
- `src/modules/twentyq/README.md` — replace phase-1 stub with full doc.
|
||||
- `docs/codebase-summary.md` — add a one-line entry for the new module
|
||||
(only if existing modules are listed there).
|
||||
- `docs/development-roadmap.md` — mark this plan as completed once shipped
|
||||
(per global feedback rule: roadmap tracks future work; completed work
|
||||
documented in git log + plan file).
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. Create `tests/fakes/fake-ai.js`:
|
||||
- `createFakeAi()` returning `{ run: vi.fn() }`.
|
||||
- Helper `mockJudgement(ai, { is_guess, answer, hint })` configures the
|
||||
mock to return a function-call shape matching what `ai-client` parses.
|
||||
2. Write each test file in order matching the production-file order, using
|
||||
the existing doantu tests as the structural template.
|
||||
3. Run `npx vitest run tests/modules/twentyq/` iteratively until green.
|
||||
4. Run full suite (`npm test`) — must stay green.
|
||||
5. Replace `src/modules/twentyq/README.md` with:
|
||||
- One-paragraph game description + Telegram `/` slot.
|
||||
- Commands table (visibility column).
|
||||
- Example flow (copy from `plan.md`).
|
||||
- "Data source" — Workers AI Gemma 4 26B A4B + fixed seed list.
|
||||
- "Architecture" — file-by-file, ~1 line each.
|
||||
- "Storage" — KV layout table (mirror doantu README format).
|
||||
- "Config" — env vars table (none in v1; document `env.AI` binding).
|
||||
- "Credits" — game concept (20 questions / Akinator-reverse).
|
||||
6. `npm run register:dry` — confirm `setMyCommands` payload includes
|
||||
`twentyq`, `twentyq_giveup`, `twentyq_stats` (all `public`).
|
||||
7. `npm run lint` — clean.
|
||||
8. Manual one more end-to-end sanity check via `wrangler dev` + tunnel.
|
||||
|
||||
## Todo list
|
||||
|
||||
- [ ] `tests/fakes/fake-ai.js` — AI binding stub + helpers
|
||||
- [ ] `seeds.test.js` (3–5 tests)
|
||||
- [ ] `state.test.js` (5–7 tests)
|
||||
- [ ] `validate-input.test.js` (6–8 tests)
|
||||
- [ ] `ai-client.test.js` (4–6 tests)
|
||||
- [ ] `handlers.test.js` (8–12 tests — happy path, edge cases, dedup, errors)
|
||||
- [ ] `render.test.js` (4–6 tests — escape, all formatters)
|
||||
- [ ] README replacement
|
||||
- [ ] `register:dry` shows public commands
|
||||
- [ ] Full `npm test` green
|
||||
- [ ] Mark plan status `completed` in `plan.md` frontmatter
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `npm test` green (all 200+ existing + new).
|
||||
- `npm run lint` green.
|
||||
- `register:dry` shows the three public commands.
|
||||
- README opens cleanly, matches doantu/semantle structure.
|
||||
- Manual play in `wrangler dev` confirms full game loop.
|
||||
|
||||
## Risk assessment
|
||||
|
||||
- **AI response shape mismatch** between fixture and real Gemma response →
|
||||
ai-client tests pass but production breaks. Mitigation: capture one real
|
||||
response in dev (logged + redacted); use it as the test fixture canonical.
|
||||
- **Test flake** — `getRandomSeed` could non-determ if rng default leaks into
|
||||
test. Mitigation: always pass deterministic rng in tests.
|
||||
- **README drift** — multiple modules have similar README shapes; copy from
|
||||
doantu and edit, don't write from scratch (consistency).
|
||||
|
||||
## Security considerations
|
||||
|
||||
- Test fixtures must NOT contain real bot tokens or webhook secrets (none
|
||||
needed — all logic-level).
|
||||
- README must not document any internal endpoint or account id.
|
||||
|
||||
## Next steps
|
||||
|
||||
After this phase:
|
||||
- Update `plan.md` frontmatter `status: completed`.
|
||||
- Commit + push (conventional commit: `feat(twentyq): add reverse-Akinator
|
||||
yes/no game module powered by Workers AI`).
|
||||
- Run `npm run deploy` (auto-applies migrations + registers webhook/commands).
|
||||
- Smoke test on production bot via `/twentyq`.
|
||||
@@ -1,89 +0,0 @@
|
||||
---
|
||||
name: twentyq-game-module
|
||||
status: completed
|
||||
created: 2026-04-24
|
||||
completed: 2026-04-24
|
||||
slug: twentyq-game-module
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
---
|
||||
|
||||
# TwentyQ Game Module — miti99bot
|
||||
|
||||
A reverse-Akinator yes/no guessing game. Bot picks a secret object from a fixed
|
||||
seeded category list, gives an initial hint. User asks `is it ...?` style
|
||||
questions; Workers AI (`@cf/google/gemma-4-26b-a4b-it`) judges each input,
|
||||
returns `{is_guess, answer:"yes"|"no", hint}`. Round ends on correct guess
|
||||
(`is it an organ?` matches secret) or `/twentyq_giveup`. Unlimited tries.
|
||||
|
||||
**Key external dependency:** Workers AI binding `env.AI` (already wired in
|
||||
`wrangler.toml [ai]`). Gemma 4 26B A4B chosen for function-calling +
|
||||
reasoning + cheap MoE inference (~4B active params).
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/twentyq` | Show current board (initial hint + Q/A history), or start a round if none |
|
||||
| `/twentyq <question>` | Submit a yes/no question OR a final guess (`is it ...?`) |
|
||||
| `/twentyq_giveup` | Reveal the secret and end the round (next `/twentyq` starts fresh) |
|
||||
| `/twentyq_stats` | Show per-subject stats |
|
||||
|
||||
## Example flow
|
||||
|
||||
```
|
||||
/twentyq
|
||||
Bot: 🎯 I'm thinking of an instrument.
|
||||
Hint: it uses wind to create sound.
|
||||
|
||||
/twentyq does it require hands to play?
|
||||
Bot: ✅ Yes. Hint: most players use both hands at once.
|
||||
|
||||
/twentyq is it made of wood?
|
||||
Bot: ❌ No. Hint: its body is mostly metal pipes.
|
||||
|
||||
/twentyq is it an organ?
|
||||
Bot: 🎉 Correct! It was an organ. Solved in 3 guesses.
|
||||
```
|
||||
|
||||
## Key design decisions
|
||||
|
||||
1. **Module name `twentyq`** — picked over `doandao`/`akiverse` for English clarity.
|
||||
2. **English-only replies** — single-language prompt simplifies model behavior.
|
||||
3. **Unlimited turns** — solve or giveup ends the round (matches semantle/doantu).
|
||||
4. **Fixed seed list** — `seeds.js` has ~60 objects across 6 categories
|
||||
(instrument, animal, food, vehicle, sport, household). Cheap, deterministic,
|
||||
no AI cost for selection.
|
||||
5. **AI for answer + hint only** — model receives `{secret, category, history}`
|
||||
each turn; emits structured `{is_guess, answer, hint}` via function calling.
|
||||
6. **Pre-validate input** — reject open-ended questions (`what`/`how`/`why`/`which`)
|
||||
client-side. Saves Neurons. Doesn't count toward guess tally.
|
||||
7. **Same command for ask + guess** — AI sets `is_guess=true` when user asks
|
||||
`is it [specific noun matching/close to secret]?`. Bot ends round on match.
|
||||
8. **Visibility: `public`** — appears in Telegram `/` menu + `/help`.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | File | Focus | Est. LOC |
|
||||
|-------|------|-------|---------:|
|
||||
| 1 | `phase-01-foundation.md` | Module scaffold, seeds, KV state, prompt templates, env wiring | ~220 |
|
||||
| 2 | `phase-02-ai-client.md` | Workers AI client + function-calling schema + input validation | ~150 |
|
||||
| 3 | `phase-03-gameplay-handlers.md` | Command handlers, render, round lifecycle | ~280 |
|
||||
| 4 | `phase-04-tests-docs.md` | Vitest coverage, README, help integration | ~200 |
|
||||
|
||||
## Critical files
|
||||
|
||||
- **Create:** `src/modules/twentyq/{index,ai-client,seeds,state,handlers,render,prompts,validate-input}.js` + `README.md`
|
||||
- **Edit:** `src/modules/index.js` (loader entry), `wrangler.toml` (`MODULES` list), `.env.deploy.example` (`MODULES` list comment)
|
||||
- **Test:** `tests/modules/twentyq/{seeds,state,validate-input,ai-client,handlers,render}.test.js`
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Per-chat shared round vs per-subject?** Defaulting to per-subject (user id in
|
||||
DMs, chat id in groups) — matches doantu/semantle. Group play uses chat-id
|
||||
scope so all members collaborate on one round.
|
||||
- **AI temperature?** Locking to `0.3` for consistent yes/no determinism — can
|
||||
bump to `0.7` for hint variety in a follow-up tweak.
|
||||
- **Hint repetition?** Initial implementation makes no effort to track hint
|
||||
uniqueness across the round — relying on the model + history context to vary.
|
||||
Add dedup later if observed boring.
|
||||
-111
@@ -1,111 +0,0 @@
|
||||
# Project Cleanup Audit — 260424-1821
|
||||
|
||||
Scope: full repo focused on twentyq (reworked today) + surrounding docs/scripts. Tests green, biome clean.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Stale module doc — claims function calling that was removed
|
||||
- **Severity:** high
|
||||
- **File:** `src/modules/twentyq/README.md:5` and `:67`
|
||||
- **Problem:** README says the module uses "function calling" and references `prompts.js` as exporting `ANSWER_FUNCTION_SCHEMA` that declares a `submit_answer` tool. Neither exists — `ai-client.js` and `prompts.js` now explicitly document the opposite approach (JSON-in-content, no tools array). Readers will hunt for missing symbols.
|
||||
- **Fix:** Replace line 5 sentence with e.g. "judges every user input with a Workers AI LLM (`@cf/google/gemma-4-26b-a4b-it`) that emits one-line JSON parsed from the response body." Rewrite line 67 to: "`prompts.js` — `buildSystemPrompt(state)` injects secret + history; `buildStartRoundPrompt(target)` produces the round-opening prompt." Delete the tool-call parse-shape claim on line 71 (no tool-call shape exists anymore; `ai-client` only extracts plain text + JSON-in-content).
|
||||
|
||||
### 2. Stale top-of-file comment — same function-calling claim
|
||||
- **Severity:** high
|
||||
- **File:** `src/modules/twentyq/index.js:4-7`
|
||||
- **Problem:** Header still advertises "function calling — the model returns { is_guess, answer, hint }". Contradicts `ai-client.js:4-8` and `prompts.js:2-9` in the same directory.
|
||||
- **Fix:** Replace lines 5-7 with "judges each user input via Workers AI (`@cf/google/gemma-4-26b-a4b-it`) — the model emits one-line JSON `{ is_guess, answer, hint }` parsed from the response body."
|
||||
|
||||
### 3. Stale codebase-summary — twentyq row + test counts + dep versions wrong
|
||||
- **Severity:** high
|
||||
- **File:** `docs/codebase-summary.md:26, 65, 66, 74, 77-84`
|
||||
- **Problem:**
|
||||
- Line 26 twentyq row says "via function calling" — same stale claim as #1/#2.
|
||||
- Line 65: vitest listed as ^2.1.0, package.json has ^4.1.4.
|
||||
- Line 66: wrangler listed as ^3.90.0, package.json has ^4.84.0.
|
||||
- Line 74: "200 tests across 21 test files" — actual is 449 (user stated) across many more files.
|
||||
- Lines 77-84: missing rows for `semantle`, `doantu`, `twentyq`, `lolschedule`. Misleading — suggests only 4 modules are tested.
|
||||
- **Fix:** Delete "via function calling" on line 26 (say "judges each yes/no question and generates fresh hints"). Bump versions to match `package.json`. Regenerate test-count line from actual test run output. Add rows for semantle/doantu/twentyq/lolschedule with real counts from `npx vitest list`.
|
||||
|
||||
### 4. Stale architecture file tree (omits 4 modules + 2 files)
|
||||
- **Severity:** medium
|
||||
- **File:** `docs/architecture.md:19-42` and `:105-113`
|
||||
- **Problem:** The ASCII tree shows only `util, trading, wordle, loldle, misc` and omits the snippet of `moduleRegistry` at line 105-113 which predates doantu/semantle/twentyq/lolschedule. It also omits `cron-dispatcher.js` and `validate-cron.js`. Readers trusting this doc will think those modules don't exist.
|
||||
- **Fix:** Extend the tree to include `lolschedule/`, `semantle/`, `doantu/`, `twentyq/`, `cron-dispatcher.js`, `validate-cron.js`. Update the inline `moduleRegistry` snippet at 105-113 to match the 9 entries currently in `src/modules/index.js`.
|
||||
|
||||
### 5. Stale wrangler.toml AI-binding comment
|
||||
- **Severity:** low
|
||||
- **File:** `wrangler.toml:29-34`
|
||||
- **Problem:** Comment claims `env.AI` is "used by semantle + doantu". `twentyq` also uses it. The Neuron/pricing numbers quoted are bge-m3 embedding numbers — twentyq uses Gemma which has different pricing.
|
||||
- **Fix:** Change "semantle + doantu" → "semantle, doantu, and twentyq". Add a second line noting twentyq uses `@cf/google/gemma-4-26b-a4b-it` (separate pricing) or just drop the specific bge-m3 math and keep the pricing link.
|
||||
|
||||
### 6. Obsolete docs/todo.md — D1 already deployed
|
||||
- **Severity:** medium
|
||||
- **File:** `docs/todo.md` (entire file)
|
||||
- **Problem:** File is the TODO for the D1+Cron infra rollout. `wrangler.toml:26` already has a real D1 UUID (`261b54e7-...`), so the "Pre-deploy" checklist is satisfied. The trading cron is live. The "first deploy verification" items are historical. The file survives as a reader-confusing artefact.
|
||||
- **Fix:** Delete the three satisfied sections (Pre-deploy, First deploy verification, Post-deploy smoke tests), leaving only the "Nice-to-have" section. Or delete the whole file and fold the unclaimed items into `docs/development-roadmap.md`.
|
||||
|
||||
### 7. Stale stub-kv.js comment references nonexistent flag
|
||||
- **Severity:** low
|
||||
- **File:** `scripts/stub-kv.js:10`
|
||||
- **Problem:** Doc-comment says future modules should "gate the write on a `process.env.REGISTER_DRYRUN` flag" — that flag is never read anywhere and has no consumer.
|
||||
- **Fix:** Either plumb the flag through `register.js` (overkill — YAGNI) or replace the sentence with "If a future module writes inside init(), restructure that init to defer writes until the first handler call." Keep the `stubKv` / `stubAi` simple.
|
||||
|
||||
### 8. Confusing handleStats test — saves game, asserts stats
|
||||
- **Severity:** low
|
||||
- **File:** `tests/modules/twentyq/handlers.test.js:192-200`
|
||||
- **Problem:** Test saves a game via `saveGame` then calls `handleStats` and asserts "no games" message. It passes (saving a game doesn't write stats), but the `saveGame` call is pure noise and misleads readers into thinking the render is expected even when a game is active.
|
||||
- **Fix:** Delete the `await saveGame(db, 1, sampleGame())` line. Either keep the test as "renders empty summary when no stats" or add a second assertion that exercises the `played > 0` branch (the stats row is also currently not tested end-to-end — only `formatStats` is covered in render.test.js).
|
||||
|
||||
### 9. Inaccurate assertion in "fresh round + text" test
|
||||
- **Severity:** low
|
||||
- **File:** `tests/modules/twentyq/handlers.test.js:138-143`
|
||||
- **Problem:** Minor — the test mocks `mockRoundStart(ai)` + `mockJudgement(ai, ...)` then asserts `ctx.reply` is called twice but never verifies `ai.run` was called twice. If the handler ever regressed to silently skipping one AI call, this would pass on reply count alone.
|
||||
- **Fix:** Add `expect(ai.run).toHaveBeenCalledTimes(2);` after line 140. Same nit applies to the group-chat test at 161-170 — add `expect(ai.run).toHaveBeenCalledTimes(2);`.
|
||||
|
||||
### 10. Unused `recordResult` return value
|
||||
- **Severity:** low
|
||||
- **File:** `src/modules/twentyq/state.js:107`
|
||||
- **Problem:** `recordResult` ends with `return s;` but no caller uses the returned stats. Mirrors doantu but unreferenced here.
|
||||
- **Fix:** Either drop `return s;` and the implicit `Promise<TwentyqStats>` from the JSDoc (cleaner), or consume the return in handlers (e.g. post a one-line stat update after `giveup`/`solve`). Dropping is the YAGNI move.
|
||||
|
||||
### 11. Over-broad redact regex on one-letter targets
|
||||
- **Severity:** low (defence-in-depth, not broken)
|
||||
- **File:** `src/modules/twentyq/ai-client.js:125-131`
|
||||
- **Problem:** `redactSecret` uses `\b<word>\b`. For single-letter or digit-heavy targets the regex still works, but the "entire-hint-became-redacted" fallback string at line 130 (`out.length > 0 ? ... : "the hint was redacted..."`) can never actually trigger because `hint.replace` with any input always yields a length > 0 (it replaces, not deletes). The safety branch is dead code.
|
||||
- **Fix:** Simplify to `return out;` and drop the fallback branch + message. If you want to guard against a hint that IS the secret, compare `out === "(redacted)"` instead (that's the real "hint was just the secret" case).
|
||||
|
||||
### 12. .env.deploy.example default MODULES — requires manual sync with wrangler.toml
|
||||
- **Severity:** low
|
||||
- **File:** `.env.deploy.example:15` vs `wrangler.toml:8`
|
||||
- **Problem:** Two places define the same comma-separated list. They happen to match today but drift is easy.
|
||||
- **Fix:** Either (a) have `register.js` parse `wrangler.toml` directly, or (b) leave the duplication but add a one-line comment in both places: "KEEP IN SYNC WITH wrangler.toml [vars] MODULES" — currently the comment only exists in `.env.deploy.example`. YAGNI: just add the reciprocal comment to `wrangler.toml:5-6`.
|
||||
|
||||
### 13. Empty `D1 layer` test-coverage row
|
||||
- **Severity:** low
|
||||
- **File:** `docs/codebase-summary.md:79`
|
||||
- **Problem:** Row says "DB layer (D1) | — | Fake D1 in-memory implementation...". The em-dash "tests" count is confusing; `tests/fakes/fake-d1.js` is exercised by trading tests. Either real count or drop.
|
||||
- **Fix:** Delete the row or merge into the trading row.
|
||||
|
||||
### 14. validate-input.js open-ended regex is incomplete
|
||||
- **Severity:** low (observation)
|
||||
- **File:** `src/modules/twentyq/validate-input.js:13`
|
||||
- **Problem:** Bars "what how why which who where when tell me describe explain". Misses some natural open-enders like "name", "list", "give me". Not worth flagging as a bug but note for when a user complains.
|
||||
- **Fix:** No action unless users report — KISS. Document the short allow-list philosophy in the comment.
|
||||
|
||||
## Summary
|
||||
|
||||
- **Total findings:** 14 (3 high, 3 medium, 8 low)
|
||||
- **Recommended apply order (easy wins → larger):**
|
||||
1. #1, #2 — stale twentyq function-calling claims in `README.md` + `index.js` header (5-min delete/rewrite each)
|
||||
2. #5, #7 — one-line comment fixes in `wrangler.toml` + `stub-kv.js`
|
||||
3. #11, #10 — tiny code deletes in `ai-client.js` + `state.js`
|
||||
4. #8, #9 — test cleanups
|
||||
5. #3, #4 — regenerate doc tables/trees in `docs/codebase-summary.md` + `docs/architecture.md`
|
||||
6. #6 — decide whether to delete `docs/todo.md` or trim it
|
||||
7. #12, #13, #14 — optional polish
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
- Should `docs/todo.md` be deleted outright or trimmed to the remaining "nice-to-have" items? (Style choice — leaning delete per YAGNI.)
|
||||
- Is the `recordResult` return value kept on purpose for parity with doantu/semantle (which may use it)? Worth a 30-sec check in those modules before removing.
|
||||
@@ -1,187 +0,0 @@
|
||||
# Phase 01 — Shared scrape + lookup helpers
|
||||
|
||||
## Context
|
||||
|
||||
- [Research: overview + emoji](../reports/researcher-260424-2215-loldle-emoji-and-modes-overview.md)
|
||||
- [Research: quote](../reports/researcher-260424-2215-loldle-quote-mode.md)
|
||||
- [Research: ability + splash](../reports/researcher-260424-2215-loldle-ability-splash-modes.md)
|
||||
- Existing: `scripts/scrape-loldle-data.js`, `src/modules/loldle/lookup.js`
|
||||
|
||||
## Overview
|
||||
|
||||
**Priority:** P0 (blocks 02–05).
|
||||
**Status:** pending.
|
||||
|
||||
Lay a minimal shared foundation so the four new modules don't each
|
||||
re-implement champion-name normalization or re-scrape loldle.net five times.
|
||||
|
||||
## Key insights
|
||||
|
||||
- **Bundle check (2026-04-24):** loldle.net's bundle contains classic
|
||||
attributes only — **zero emoji code points, zero per-champion quote
|
||||
strings**. Daily answers are fetched encrypted from
|
||||
`cache.loldle.net/cache.json` and decrypted with AES key `D5XCtTOObw`,
|
||||
but the cache only holds the single daily rotation, not a full
|
||||
champion→emoji / champion→quote pool.
|
||||
- **Pivot (DEVIATION from plan as written):** Emoji sequences are derived
|
||||
**algorithmically** from classic's existing `champions.json` metadata
|
||||
(species/regions/positions/resource) via a small mapping table — no new
|
||||
fetch, no brittle scrape. Quote text uses DDragon's `title` +
|
||||
first-sentence `lore` blurb. Both data sources are stable and official.
|
||||
- Data Dragon is the right source for ability/splash — official, stable, no
|
||||
brittle regex. Scripts hit DDragon once per patch (fortnightly) and cache
|
||||
to JSON. Bot imports JSON directly.
|
||||
- `lookup.js`'s `findChampion` stays coupled to the champion-record shape.
|
||||
Don't hoist it; only hoist the tiny `normalize(s)` helper.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Extended scraper emits three JSONs (keeps `champions.json` plus adds
|
||||
`emojis.json`, `quotes.json`).
|
||||
- New DDragon script emits `abilities.json` + `splashes.json`.
|
||||
- Shared `normalize(s)` helper in `src/util/` for case/space/punctuation-
|
||||
insensitive matching across all modes.
|
||||
|
||||
**Non-functional**
|
||||
- Single loldle.net fetch per scrape run (no re-download for each mode).
|
||||
- DDragon fetch uses `GET /api/versions.json` → latest → one
|
||||
`champion.json` per champion OR one aggregated `en_US/champion.json`
|
||||
(list) + per-champion fetches as needed. Prefer aggregated list first,
|
||||
drill into per-champion only for `skins[]`.
|
||||
- Scripts idempotent, safe to re-run, short-circuit on "no change".
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
scripts/
|
||||
├── scrape-loldle-data.js (EXISTING, extended)
|
||||
│ └── writes: src/modules/loldle/champions.json
|
||||
│ + src/modules/loldle-emoji/emojis.json
|
||||
│ + src/modules/loldle-quote/quotes.json
|
||||
└── fetch-ddragon-data.js (NEW)
|
||||
└── writes: src/modules/loldle-ability/abilities.json
|
||||
+ src/modules/loldle-splash/splashes.json
|
||||
|
||||
src/util/
|
||||
└── normalize-name.js (NEW, ~10 LOC)
|
||||
|
||||
src/modules/{loldle-emoji,loldle-quote,loldle-ability,loldle-splash}/
|
||||
└── (created in phases 02–05)
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `scripts/scrape-loldle-data.js` — add extraction for emoji + quote fields.
|
||||
Regex must accommodate loldle.net's current bundle shape (inspect before
|
||||
touching; existing regex is the template).
|
||||
- `.github/workflows/scrape-loldle-data.yml` — no change needed; the
|
||||
extended scraper writes more files, the workflow's `git diff` check
|
||||
catches them automatically.
|
||||
|
||||
**Create**
|
||||
- `scripts/fetch-ddragon-data.js` — fetch DDragon, extract ability + skin
|
||||
metadata, write two JSONs.
|
||||
- `src/util/normalize-name.js` — single export `normalize(s)`.
|
||||
- `src/modules/loldle-emoji/` (empty folder, populated in 02).
|
||||
- `src/modules/loldle-quote/` (empty folder, populated in 03).
|
||||
- `src/modules/loldle-ability/` (empty folder, populated in 04).
|
||||
- `src/modules/loldle-splash/` (empty folder, populated in 05).
|
||||
|
||||
**Delete:** none.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **Create `src/util/normalize-name.js`**:
|
||||
```js
|
||||
export const normalize = (s) =>
|
||||
String(s || "").toLowerCase().replace(/[^a-z0-9]/g, "");
|
||||
```
|
||||
Update `src/modules/loldle/lookup.js` to import it (keeps behaviour,
|
||||
removes the inline duplicate). Run `npm test` — classic loldle tests
|
||||
must still pass.
|
||||
|
||||
2. **Inspect the live loldle.net bundle** for emoji + quote fields:
|
||||
```bash
|
||||
node -e "
|
||||
const html = await (await fetch('https://loldle.net/emoji')).text();
|
||||
const m = html.match(/js\/index\.[^\"]+\.js/);
|
||||
const js = await (await fetch('https://loldle.net/' + m[0])).text();
|
||||
console.log(js.match(/emoji[s]?:\s*[\"\[][^\n]{0,200}/g)?.slice(0,3));
|
||||
console.log(js.match(/quote[s]?:\s*[\"\[][^\n]{0,200}/g)?.slice(0,3));
|
||||
"
|
||||
```
|
||||
Document the ACTUAL shape in this file. Update regex accordingly.
|
||||
|
||||
3. **Extend `scrape-loldle-data.js`**:
|
||||
- Reuse the single bundle fetch already there.
|
||||
- Add two new regex passes extracting `championName → emoji` pairs and
|
||||
`championName → quote` pairs.
|
||||
- Write `src/modules/loldle-emoji/emojis.json` and
|
||||
`src/modules/loldle-quote/quotes.json`. Sort by championName.
|
||||
- Fail LOUDLY if either new regex hits zero matches (prevents silent
|
||||
schema drift on loldle.net's next bundle).
|
||||
|
||||
4. **Create `scripts/fetch-ddragon-data.js`**:
|
||||
```
|
||||
GET /api/versions.json → take versions[0]
|
||||
GET /cdn/<v>/data/en_US/champion.json → summary (all champions)
|
||||
for each championKey:
|
||||
GET /cdn/<v>/data/en_US/champion/<Key>.json → full (spells, passive, skins)
|
||||
write abilities.json:
|
||||
[{ championName, abilities: [{ slot:"P"|"Q"|"W"|"E"|"R", name, icon:"<full-url>" }] }]
|
||||
write splashes.json:
|
||||
[{ championName, skins: [{ id:0, name:"Classic", url:"<splash-url>" }, ...] }]
|
||||
```
|
||||
Use `ddragon.leagueoflegends.com`. Parallelize per-champion fetches with
|
||||
a concurrency cap (10). Cache to a local `.ddragon-cache/` ignored by
|
||||
git so re-runs within the same patch are instant.
|
||||
Add npm script: `"fetch:ddragon-data": "node scripts/fetch-ddragon-data.js"`.
|
||||
|
||||
5. **Run both scripts locally**, commit the resulting JSONs. Verify sizes
|
||||
reasonable (emojis.json < 50 KB; quotes.json < 100 KB; abilities.json
|
||||
< 500 KB; splashes.json < 300 KB). If abilities.json balloons past 1 MB,
|
||||
drop fields (keep only slot + icon URL + ability name).
|
||||
|
||||
6. **Run `npm test` + `npm run lint`** — no regressions in classic loldle.
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] Create `src/util/normalize-name.js`
|
||||
- [ ] Refactor `src/modules/loldle/lookup.js` to import `normalize`
|
||||
- [ ] Inspect loldle.net bundle; document emoji + quote regex shape here
|
||||
- [ ] Extend `scripts/scrape-loldle-data.js` (emoji + quote extraction)
|
||||
- [ ] Create `scripts/fetch-ddragon-data.js`
|
||||
- [ ] Add `fetch:ddragon-data` npm script
|
||||
- [ ] Run both scripts, commit generated JSONs
|
||||
- [ ] Create 4 empty module folders (placeholders for phases 02–05)
|
||||
- [ ] `npm test` + `npm run lint` clean
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `npm run scrape:loldle-data` writes 3 JSONs (champions + emojis +
|
||||
quotes), all non-empty, all sorted by championName.
|
||||
- `npm run fetch:ddragon-data` writes 2 JSONs with full CDN URLs.
|
||||
- Classic loldle tests unchanged, still pass.
|
||||
- No lint warnings.
|
||||
- Four empty module folders exist, ready for phases 02–05.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| loldle.net bundle schema drifts between now and scrape | Extraction fails loud, re-inspect and update regex |
|
||||
| DDragon ability icon URL requires version path; version shifts mid-day | Cache URLs with version baked in; fetch script refreshes on demand |
|
||||
| Per-champion DDragon fetches (165 requests) hit rate limits | Concurrency cap 10; no published DDragon rate limits but be polite |
|
||||
| abilities.json > 1 MB bloats Workers bundle | Strip fields, keep only slot + icon URL + name |
|
||||
|
||||
## Security
|
||||
|
||||
- No secrets introduced.
|
||||
- DDragon and loldle.net are public endpoints; no auth.
|
||||
- Scripts write only to `src/modules/**/*.json` (no directory traversal).
|
||||
|
||||
## Next steps
|
||||
|
||||
Phases 02–05 can start **in parallel** once this phase completes.
|
||||
@@ -1,173 +0,0 @@
|
||||
# Phase 02 — Emoji module (`loldle-emoji`)
|
||||
|
||||
## Context
|
||||
|
||||
- [Research: overview + emoji](../reports/researcher-260424-2215-loldle-emoji-and-modes-overview.md)
|
||||
- Template: `src/modules/loldle/` (classic)
|
||||
- Dependency: phase 01 (`emojis.json` written by scraper, `normalize` helper).
|
||||
|
||||
## Overview
|
||||
|
||||
**Priority:** P1 (ship first — simplest mode).
|
||||
**Status:** pending.
|
||||
|
||||
Guess the champion from an emoji clue. All text-rendered — Telegram renders
|
||||
emojis natively. No images, no audio, no DDragon.
|
||||
|
||||
## Key insights
|
||||
|
||||
- Emoji sequences are handcrafted by loldle.net (not algorithmic). We only
|
||||
have what they ship — can't compute our own.
|
||||
- Progressive reveal on loldle.net works by "unlock one emoji per wrong
|
||||
guess". On Telegram, we keep it simpler: **show all emojis upfront, fewer
|
||||
guesses**. Saves edit-message roundtrips.
|
||||
- Reuse classic's `stats:<subject>` shape verbatim.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- `/loldle_emoji` → show current round or start fresh; submit a guess if
|
||||
`<champion>` arg provided.
|
||||
- `/loldle_emoji_giveup` → reveal answer, record loss.
|
||||
- `/loldle_emoji_stats` → show per-subject play stats.
|
||||
- 5 guesses per round.
|
||||
- Subject = user (DM) or chat (group), same rule as classic.
|
||||
- Champion name lookup identical to classic (case/space/punctuation-
|
||||
insensitive, unique-prefix fallback).
|
||||
|
||||
**Non-functional**
|
||||
- Pure KV storage (no D1). Auto-prefixed key `loldle-emoji:game:<subject>`.
|
||||
- Round state: `{ target, guesses, startedAt }` — same shape as classic.
|
||||
- Champion pool comes from `emojis.json` (phase 01). Only champions with a
|
||||
non-empty emoji string are eligible.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/loldle-emoji/
|
||||
├── index.js # { name, commands, init } export
|
||||
├── handlers.js # handleEmoji, handleGiveup, handleStats
|
||||
├── state.js # loadGame/saveGame/clearGame/loadStats/recordResult
|
||||
├── lookup.js # findChampion over emojis.json (thin wrapper, uses util/normalize-name.js)
|
||||
├── render.js # board render: emoji block + guesses list
|
||||
├── emojis.json # [{ championName, emojis:"🦊✨💫" }, ...] (generated)
|
||||
└── README.md # usage + data source notes
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `src/modules/index.js` — add `"loldle-emoji"` to static import map.
|
||||
- `wrangler.toml` `[vars].MODULES` — append `,loldle-emoji`.
|
||||
- `.env.deploy` (local) — append `,loldle-emoji` to MODULES.
|
||||
|
||||
**Create**
|
||||
- Six files listed in Architecture above.
|
||||
|
||||
**Delete:** none.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **Copy `src/modules/loldle/state.js`** → `src/modules/loldle-emoji/state.js`.
|
||||
Change `MAX_GUESSES = 5`. No other changes needed; same KV shape.
|
||||
|
||||
2. **Create `lookup.js`**:
|
||||
```js
|
||||
import { normalize } from "../../util/normalize-name.js";
|
||||
export function findChampion(pool, input) {
|
||||
const q = normalize(input);
|
||||
if (!q) return null;
|
||||
const exact = pool.find((c) => normalize(c.championName) === q);
|
||||
if (exact) return exact;
|
||||
const prefix = pool.filter((c) =>
|
||||
normalize(c.championName).startsWith(q));
|
||||
return prefix.length === 1 ? prefix[0] : null;
|
||||
}
|
||||
```
|
||||
|
||||
3. **Create `render.js`** — render the current board:
|
||||
```
|
||||
🎭 <emoji sequence>
|
||||
|
||||
Guesses (<n>/<MAX>):
|
||||
• <name1> ❌
|
||||
• <name2> ❌
|
||||
```
|
||||
HTML-escape every champion name via `src/util/escape-html.js`.
|
||||
|
||||
4. **Create `handlers.js`** modelled on `loldle/handlers.js`:
|
||||
- Same subject resolution (`getSubject`).
|
||||
- Same arg parsing.
|
||||
- `pickRandomChampion()` picks from `emojisData` (skip champions with
|
||||
empty emoji string, if any).
|
||||
- Win / loss / giveup messages reuse classic's tone; swap "classic" →
|
||||
"emoji" in copy. No stickers for v1 (YAGNI — can add later).
|
||||
- On win: "🎉 Got it! <champ> — solved in Nx/5" + stats update + KV clear.
|
||||
- On loss: "❌ Answer was <champ>" + stats update + KV clear.
|
||||
|
||||
5. **Create `index.js`**:
|
||||
```js
|
||||
import { handleGiveup, handleEmoji, handleStats } from "./handlers.js";
|
||||
let db = null;
|
||||
export default {
|
||||
name: "loldle-emoji",
|
||||
init: async ({ db: store }) => { db = store; },
|
||||
commands: [
|
||||
{ name: "loldle_emoji", visibility: "public",
|
||||
description: "Emoji loldle — guess the champion from emojis",
|
||||
handler: (ctx) => handleEmoji(ctx, db) },
|
||||
{ name: "loldle_emoji_giveup", visibility: "public",
|
||||
description: "Reveal the current emoji answer",
|
||||
handler: (ctx) => handleGiveup(ctx, db) },
|
||||
{ name: "loldle_emoji_stats", visibility: "public",
|
||||
description: "Show your emoji stats (wins, streak)",
|
||||
handler: (ctx) => handleStats(ctx, db) },
|
||||
],
|
||||
};
|
||||
```
|
||||
|
||||
6. **Register** — add to `src/modules/index.js` import map, MODULES env.
|
||||
|
||||
7. **Write minimal README.md** — commands table, KV prefix, data source
|
||||
note ("regenerated by `npm run scrape:loldle-data`").
|
||||
|
||||
8. **Run `npm run dev`**, point a test bot at it, verify:
|
||||
- `/loldle_emoji` shows emojis + empty board.
|
||||
- `/loldle_emoji Ahri` (assuming Ahri is the answer) wins.
|
||||
- `/loldle_emoji_giveup` reveals answer.
|
||||
- `/loldle_emoji_stats` reports play counts.
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] Create folder + 6 files per Architecture
|
||||
- [ ] Copy state.js from classic, lower MAX_GUESSES to 5
|
||||
- [ ] Import normalize helper from util
|
||||
- [ ] Render board with HTML-escape
|
||||
- [ ] Handlers ported from classic, tone tweaked
|
||||
- [ ] Register in `src/modules/index.js` + MODULES env
|
||||
- [ ] Write README
|
||||
- [ ] Local smoke-test (`npm run dev` + test bot)
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Module loads at `installDispatcher` without conflicts.
|
||||
- `/loldle_emoji` runs end-to-end against loldle.net data.
|
||||
- Stats persist across rounds, isolated from classic loldle.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| `emojis.json` missing some champions | Pool is whatever loldle.net provides; guard null at render time |
|
||||
| Emoji rendering differs across Telegram clients | Use standard Unicode emojis (loldle.net already does); no skin-tone variants |
|
||||
| Player confusion vs classic (two loldle-like commands) | Distinct command name + distinct welcome copy |
|
||||
|
||||
## Security
|
||||
|
||||
- Input passes through `normalize` (strips non-alphanum) before comparison.
|
||||
- HTML escape all user-submitted and champion-name text in reply.
|
||||
|
||||
## Next steps
|
||||
|
||||
Phase 06 covers tests. Phase 03 (quote) can be built in parallel — same
|
||||
template.
|
||||
@@ -1,147 +0,0 @@
|
||||
# Phase 03 — Quote module (`loldle-quote`)
|
||||
|
||||
## Context
|
||||
|
||||
- [Research: quote mode](../reports/researcher-260424-2215-loldle-quote-mode.md)
|
||||
- Template: `src/modules/loldle-emoji/` (phase 02) and `src/modules/loldle/`.
|
||||
- Dependency: phase 01 (`quotes.json` written, `normalize` helper).
|
||||
|
||||
## Overview
|
||||
|
||||
**Priority:** P1 (ship in parallel with emoji).
|
||||
**Status:** pending.
|
||||
|
||||
Guess the champion from a voice-line text. Text-only for MVP — audio is
|
||||
explicitly out of scope (bandwidth, CDN rehost TOS risk, storage overhead
|
||||
per research doc §4).
|
||||
|
||||
## Key insights
|
||||
|
||||
- Quote mode on loldle.net: **binary right/wrong, 6 guesses, audio hint
|
||||
unlocks after all fails**. We drop audio; keep 6 guesses.
|
||||
- Quotes can be ambiguous ("For glory!" — Garen, Galio, Jarman...). Fewer
|
||||
clues than classic; that's OK — the mode IS meant to be hard.
|
||||
- Pool: ~150 champions with quotes (per research). Every champion in
|
||||
`quotes.json` must have non-empty `quote` string — filter at load time.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- `/loldle_quote` → show current quote or start fresh; submit a guess if arg.
|
||||
- `/loldle_quote_giveup` → reveal, record loss.
|
||||
- `/loldle_quote_stats` → per-subject stats.
|
||||
- 6 guesses.
|
||||
- Same subject resolution (user id DM / chat id group).
|
||||
|
||||
**Non-functional**
|
||||
- Pure KV. Prefix: `loldle-quote:`.
|
||||
- Same round state shape as classic and emoji.
|
||||
- HTML-escape the quote text before putting it inside `<i>…</i>` so
|
||||
apostrophes / `<` in a quote don't break render.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/loldle-quote/
|
||||
├── index.js
|
||||
├── handlers.js
|
||||
├── state.js # MAX_GUESSES = 6
|
||||
├── lookup.js # (near-copy of emoji's)
|
||||
├── render.js # quote block + guesses list
|
||||
├── quotes.json # [{ championName, quote:"..." }, ...] (generated)
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `src/modules/index.js` — register `"loldle-quote"`.
|
||||
- `wrangler.toml` `[vars].MODULES` + `.env.deploy` — append.
|
||||
|
||||
**Create**
|
||||
- Seven files listed in Architecture above.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **Copy `loldle-emoji/` as scaffold.** It's 95% the same shape.
|
||||
|
||||
2. **Swap payload:** `emojis.json` → `quotes.json`. `emojis` string field
|
||||
→ `quote` string field.
|
||||
|
||||
3. **`state.js`** — `MAX_GUESSES = 6`.
|
||||
|
||||
4. **`render.js`** — show quote as italic block:
|
||||
```
|
||||
🎭 <i>"The true face of desire."</i>
|
||||
|
||||
Guesses (n/6):
|
||||
• Ahri ❌
|
||||
```
|
||||
HTML-escape the quote BEFORE wrapping it in `<i>`. HTML-escape each
|
||||
champion name.
|
||||
|
||||
5. **`handlers.js`** — port emoji handlers with copy tweaked:
|
||||
- Welcome line: "🎭 Guess the champion from this quote."
|
||||
- Win: "🎉 Nailed it! <champ>."
|
||||
- Loss: "❌ Answer: <champ>."
|
||||
- No stickers for v1.
|
||||
|
||||
6. **`lookup.js`** — identical to emoji's, change import path.
|
||||
|
||||
7. **`index.js`**:
|
||||
```js
|
||||
commands:
|
||||
loldle_quote (public)
|
||||
loldle_quote_giveup (public)
|
||||
loldle_quote_stats (public)
|
||||
```
|
||||
|
||||
8. **Register** in `src/modules/index.js` + MODULES env in both
|
||||
`wrangler.toml` and `.env.deploy`.
|
||||
|
||||
9. **README.md**: commands, KV prefix, data source note, "audio hint not
|
||||
implemented — see phase plan for rationale".
|
||||
|
||||
10. **Smoke-test** in `wrangler dev` (same protocol as phase 02).
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] Scaffold folder by copying `loldle-emoji/`
|
||||
- [ ] Repoint JSON import to `quotes.json`
|
||||
- [ ] MAX_GUESSES = 6
|
||||
- [ ] Render quote as italic HTML block (escaped)
|
||||
- [ ] Copy tweaks in handlers
|
||||
- [ ] Register + MODULES env
|
||||
- [ ] README
|
||||
- [ ] Smoke-test
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Module loads, commands respond.
|
||||
- Quotes render cleanly in Telegram (no HTML-injection bugs with special
|
||||
chars in a champion's quote).
|
||||
- Stats persist per mode.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Quote ambiguity → frustration | Accepted design tradeoff; document in README |
|
||||
| Quote includes HTML metacharacters (`<`, `&`) from loldle.net | Always HTML-escape before `<i>` wrap |
|
||||
| `quotes.json` missing quote for some newly-added champion | Filter pool to non-empty quotes at import-time |
|
||||
|
||||
## Security
|
||||
|
||||
- Escape quote text BEFORE rendering (quote content is third-party data
|
||||
from loldle.net scrape).
|
||||
- Escape user-submitted guess text in replies.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should audio hint ever ship? (Post-MVP, gated on user demand. Would
|
||||
require a cron that pre-fetches audio URLs from LoL Wiki and stores in
|
||||
R2. Not in this plan.)
|
||||
|
||||
## Next steps
|
||||
|
||||
Phase 06 adds tests. Phases 04/05 (image modes) proceed independently.
|
||||
@@ -1,155 +0,0 @@
|
||||
# Phase 04 — Ability module (`loldle-ability`)
|
||||
|
||||
<!-- Updated: Validation Session 1 - deferred; binary-only confirmed; no cropping -->
|
||||
|
||||
## Context
|
||||
|
||||
- [Research: ability + splash](../reports/researcher-260424-2215-loldle-ability-splash-modes.md)
|
||||
- Template: `src/modules/loldle-emoji/` (phase 02).
|
||||
- Dependency: phase 01 (`abilities.json` from Data Dragon).
|
||||
|
||||
## Overview
|
||||
|
||||
**Priority:** P2 (image mode, more moving parts than text modes).
|
||||
**Status:** **DEFERRED** — do not start until emoji + quote modes are live and
|
||||
user demand for image modes is confirmed. Phase 01 provides the data needed
|
||||
(`abilities.json`), but DDragon fetch work can also be deferred to this
|
||||
phase if not required by phases 02/03.
|
||||
|
||||
## Validated decisions
|
||||
|
||||
- **Binary guess only** (no bonus slot-identification step).
|
||||
- **No progressive cropping** — full icon from turn 1, 5 guesses.
|
||||
- Data source confirmed: **Data Dragon CDN** (not loldle.net scrape).
|
||||
|
||||
Guess the champion from a single ability icon. Telegram sends the full
|
||||
Data Dragon icon URL via `sendPhoto`. **No progressive cropping for v1**
|
||||
(see plan.md for rationale — Cloudflare Images cost + edit-photo jank not
|
||||
justified by Telegram UX).
|
||||
|
||||
## Key insights
|
||||
|
||||
- DDragon icon URLs are stable per patch: `cdn/<version>/img/spell/<key>.png`
|
||||
(passive uses `/img/passive/`). Version is baked into `abilities.json`
|
||||
by phase 01's `fetch-ddragon-data.js` so the bot doesn't need live
|
||||
version fetches.
|
||||
- Pool per champion: 5 abilities (Passive, Q, W, E, R). Pick a random slot
|
||||
at round start. Round state stores both target champion AND the slot, so
|
||||
subsequent `/loldle_ability` calls re-send the SAME icon.
|
||||
- Since the full icon is shown from turn 1, difficulty stays high only if
|
||||
guesses are tight: **5 guesses**.
|
||||
- Telegram's `sendPhoto` accepts a URL directly — no download + re-upload.
|
||||
Cache the `file_id` returned in the send response? Not worth it for v1.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- `/loldle_ability` → if no active round: pick champion + random slot,
|
||||
send photo with caption "Guess the champion from this ability. 0/5 so
|
||||
far." If active: re-send the same icon + progress line.
|
||||
- `/loldle_ability <champion>` → submit guess.
|
||||
- `/loldle_ability_giveup` → reveal answer + ability name + slot.
|
||||
- `/loldle_ability_stats` → per-subject stats.
|
||||
- 5 guesses.
|
||||
|
||||
**Non-functional**
|
||||
- KV prefix: `loldle-ability:`.
|
||||
- Round state: `{ target, slot:"P|Q|W|E|R", guesses, startedAt }`. Adds
|
||||
`slot` vs classic/emoji/quote.
|
||||
- Re-send photo each turn (no message-edit). Cheap; DDragon CDN is fast.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/loldle-ability/
|
||||
├── index.js
|
||||
├── handlers.js
|
||||
├── state.js # extended shape: + slot
|
||||
├── lookup.js
|
||||
├── abilities.json # [{ championName, abilities:[{slot, name, icon}] }]
|
||||
└── README.md
|
||||
```
|
||||
|
||||
Note: no `render.js` — output is a photo + small caption, built inline in
|
||||
`handlers.js`.
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `src/modules/index.js` — add `"loldle-ability"`.
|
||||
- `wrangler.toml` + `.env.deploy` MODULES.
|
||||
|
||||
**Create**
|
||||
- Six files listed above.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **`state.js`** — copy from `loldle-emoji/state.js`, bump shape to
|
||||
`{ target, slot, guesses, startedAt }`. `MAX_GUESSES = 5`.
|
||||
|
||||
2. **`lookup.js`** — identical to emoji's (pool shape: records still have
|
||||
`championName` at top level).
|
||||
|
||||
3. **`handlers.js`**:
|
||||
- `getSubject`, `argAfterCommand` — copy inline or import from a
|
||||
shared helper (optional — three copies is fine).
|
||||
- `pickRandomChampion()` — filter to records where abilities array is
|
||||
non-empty.
|
||||
- `pickRandomSlot(champ)` — uniform over `champ.abilities` slots.
|
||||
- On `/loldle_ability` no-arg: send photo (use
|
||||
`ctx.replyWithPhoto(url, { caption })`), caption shows guess count.
|
||||
- On guess: compare names; on win/loss send a photo reveal with full
|
||||
ability name ("That was **Ahri** — _Orb of Deception_ (Q)").
|
||||
- On giveup: same reveal.
|
||||
|
||||
4. **`index.js`** — three commands, same pattern as phase 02.
|
||||
|
||||
5. **Register + MODULES env** per usual.
|
||||
|
||||
6. **Smoke-test**: confirm photos render in Telegram, captions show
|
||||
counter correctly, wrong guesses retain the same icon across turns.
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] `state.js` with `slot` field + MAX 5
|
||||
- [ ] `lookup.js`
|
||||
- [ ] `handlers.js` (photo send, random slot pick, caption counter)
|
||||
- [ ] `index.js` (3 commands)
|
||||
- [ ] Register + MODULES env
|
||||
- [ ] README
|
||||
- [ ] Smoke-test vs real DDragon URLs
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Photo renders from DDragon URL in Telegram.
|
||||
- Same ability icon shown across multiple turns of the same round.
|
||||
- Correct guess reveals champion + ability name + slot.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| DDragon URL 404 for some legacy champion | Fetch script verifies URLs before write; filter broken entries |
|
||||
| DDragon version in `abilities.json` goes stale between fortnightly fetches | Icons remain valid (URL still 404-free per CDN retention); acceptable lag |
|
||||
| Bot bundle size: `abilities.json` could be 500 KB+ | Phase 01 trims to slot + name + icon URL only (no lore, no cost fields) |
|
||||
| Some champion has fewer than 5 abilities (unusual reworks) | `pickRandomSlot` picks from whatever's available |
|
||||
| Telegram caches photos by URL — wrong-guess photo same as first photo | That's fine, it's the SAME photo each turn by design |
|
||||
|
||||
## Security
|
||||
|
||||
- Photo URL is untrusted-feeling but in practice trusted (ddragon.lol
|
||||
CDN). Still: restrict `sendPhoto` to HTTPS URLs; don't pass user input
|
||||
into URLs anywhere.
|
||||
- HTML-escape champion + ability names in captions.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Bonus "which slot" second guess (as loldle.net does)? **Deferred.**
|
||||
v1 is binary. Revisit if users request.
|
||||
- Progressive crop for hardcore mode? **Deferred** — would require
|
||||
Cloudflare Images (~$5–15/mo, per research).
|
||||
|
||||
## Next steps
|
||||
|
||||
Phase 06 adds tests. Phase 05 (splash) reuses this module's photo-send
|
||||
pattern.
|
||||
@@ -1,143 +0,0 @@
|
||||
# Phase 05 — Splash module (`loldle-splash`)
|
||||
|
||||
<!-- Updated: Validation Session 1 - deferred; random-across-all-skins confirmed -->
|
||||
|
||||
## Context
|
||||
|
||||
- [Research: ability + splash](../reports/researcher-260424-2215-loldle-ability-splash-modes.md)
|
||||
- Template: `src/modules/loldle-ability/` (phase 04 — nearly identical
|
||||
shape, different payload).
|
||||
- Dependency: phase 01 (`splashes.json`).
|
||||
|
||||
## Overview
|
||||
|
||||
**Priority:** P2 (image mode).
|
||||
**Status:** **DEFERRED** — do not start until emoji + quote modes are live.
|
||||
Scheduled after phase 04 for pattern-reuse.
|
||||
|
||||
## Validated decisions
|
||||
|
||||
- **Random across ALL skins**, not base-only. Bigger data file, harder
|
||||
mode, matches loldle.net behaviour.
|
||||
- **No progressive cropping** — full splash from turn 1, 4 guesses.
|
||||
|
||||
Guess the champion from splash art (random skin). Full splash image sent
|
||||
once per round; user gets a tight guess budget.
|
||||
|
||||
## Key insights
|
||||
|
||||
- DDragon splash URL pattern: `cdn/img/champion/splash/<Name>_<skinId>.jpg`
|
||||
— note **no version segment**. Stable across patches.
|
||||
- Phase 01 writes `splashes.json` as
|
||||
`[{ championName, skins:[{ id, name, url }] }]`.
|
||||
- Random skin pick adds difficulty (Elementalist Lux looks nothing like
|
||||
Classic Lux). Include ALL skins, not just base — aligns with
|
||||
loldle.net's behaviour per research.
|
||||
- Splash images are large (~1 MB). Telegram auto-compresses photos, so no
|
||||
worry about bandwidth.
|
||||
- Like ability mode: no cropping in v1. Full image from turn 1, tight
|
||||
guess budget. **4 guesses** (one less than ability since the reveal is
|
||||
even bigger visually — whole-champion art).
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- `/loldle_splash` → start round (pick champion + skin) or re-send same
|
||||
photo.
|
||||
- `/loldle_splash <champion>` → submit guess.
|
||||
- `/loldle_splash_giveup` → reveal champion + skin name.
|
||||
- `/loldle_splash_stats` → per-subject stats.
|
||||
- 4 guesses.
|
||||
|
||||
**Non-functional**
|
||||
- KV prefix: `loldle-splash:`.
|
||||
- Round state: `{ target, skinId, guesses, startedAt }`. skinId persists
|
||||
so the same skin art shows across all turns of a round.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/modules/loldle-splash/
|
||||
├── index.js
|
||||
├── handlers.js
|
||||
├── state.js # shape adds skinId, MAX 4
|
||||
├── lookup.js
|
||||
├── splashes.json # [{ championName, skins:[{id, name, url}] }]
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `src/modules/index.js` — add `"loldle-splash"`.
|
||||
- `wrangler.toml` + `.env.deploy` MODULES env.
|
||||
|
||||
**Create**
|
||||
- Six files above.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **Copy `loldle-ability/` as scaffold.**
|
||||
|
||||
2. **`state.js`** — shape `{ target, skinId, guesses, startedAt }`,
|
||||
`MAX_GUESSES = 4`.
|
||||
|
||||
3. **`handlers.js`**:
|
||||
- `pickRandomChampion()` → record with ≥ 1 skin (always true; base is
|
||||
skin 0).
|
||||
- `pickRandomSkin(champ)` → uniform over `champ.skins`; keep its
|
||||
`.id` and `.url`.
|
||||
- On `/loldle_splash` no-arg: `ctx.replyWithPhoto(url, { caption: "Guess the champion. 0/4." })`.
|
||||
- On guess: compare names; on win reveal skin name ("That was **Ahri**
|
||||
in _Dynasty_ skin.").
|
||||
- On giveup: same reveal.
|
||||
|
||||
4. **`index.js`** — three public commands
|
||||
(`loldle_splash`, `loldle_splash_giveup`, `loldle_splash_stats`).
|
||||
|
||||
5. **Register + MODULES env.**
|
||||
|
||||
6. **Smoke-test** — verify splash renders, reveal names the skin
|
||||
correctly, guesses persist the same skin photo.
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] Copy scaffold from ability module
|
||||
- [ ] `state.js` with `skinId` field + MAX 4
|
||||
- [ ] `handlers.js` (photo send, random skin pick, skin reveal)
|
||||
- [ ] `index.js` (3 commands)
|
||||
- [ ] Register + MODULES env
|
||||
- [ ] README
|
||||
- [ ] Smoke-test
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Random skin shown per round (not always base).
|
||||
- Same skin persists across guesses in a round.
|
||||
- Reveal names the specific skin.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| DDragon splash 404 for legacy/unreleased skin | fetch-ddragon script verifies each URL at build time; filter 404s |
|
||||
| Too easy for popular champions (Lux, Ahri — recognised instantly) | 4-guess budget balances; some skins are genuinely obscure |
|
||||
| Multi-champion splashes (e.g. Kayle+Morgana) | Exclude at fetch time — filter skins tagged as multi-champ if DDragon flags; otherwise keep and accept the edge case |
|
||||
| Large splashes slow first reply | Telegram downloads from URL server-side; user-perceived latency is the `sendPhoto` API call, ~1 s |
|
||||
|
||||
## Security
|
||||
|
||||
- All splash URLs are on DDragon HTTPS CDN.
|
||||
- HTML-escape champion + skin names in captions and reveals.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Include "Classic" skins only for easier mode? **No** — keeps the mode
|
||||
too close to ability/classic difficulty. Random skin is the whole point.
|
||||
- Progressive crop for hardcore mode? **Deferred** (Cloudflare Images).
|
||||
- Exclude NSFW / retired skins (e.g. Graves' cigar removal)? None flagged
|
||||
by DDragon; all shipped skins are safe-for-work.
|
||||
|
||||
## Next steps
|
||||
|
||||
Phase 06 closes the plan with tests + docs.
|
||||
@@ -1,167 +0,0 @@
|
||||
# Phase 06 — Tests + docs sync
|
||||
|
||||
<!-- Updated: Validation Session 1 - scope narrowed to emoji + quote only -->
|
||||
|
||||
## Context
|
||||
|
||||
- Existing test patterns: `tests/modules/loldle/`, `tests/modules/wordle/`,
|
||||
`tests/modules/trading/`.
|
||||
- Fakes: `tests/fakes/fake-kv-namespace.js`, `tests/fakes/fake-bot.js`.
|
||||
- Docs to touch: `README.md`, `docs/adding-a-module.md` (no change needed
|
||||
unless the new modules expose a new pattern), potentially a new
|
||||
`docs/loldle-modes.md` for the mode roster.
|
||||
- Blocks: **02 + 03 must be complete.** 04 + 05 are deferred — their tests
|
||||
will be added when those phases ship.
|
||||
|
||||
## Overview
|
||||
|
||||
**Priority:** P1 (closes the plan).
|
||||
**Status:** pending.
|
||||
|
||||
Add focused unit tests for each new module and sync docs. Unit-test only
|
||||
pure-logic seams (state, lookup, render). Handler tests use fakes — no
|
||||
workerd, no Telegram fixtures, same convention as `loldle/` tests.
|
||||
|
||||
## Key insights
|
||||
|
||||
- Each new module mirrors classic's shape closely; tests can be near-
|
||||
copies of `tests/modules/loldle/state.test.js` + `lookup.test.js`.
|
||||
- No integration tests for DDragon (external CDN). Stub `fetch` if any
|
||||
unit exercises it; prefer pure functions that take a URL string.
|
||||
- Skip tests for the scraping scripts — they're network-bound. Manual
|
||||
verification (phase 01 success criteria) covers them.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- ≥ 1 test file per new module covering: state round-trip, lookup, render
|
||||
/ handler happy path.
|
||||
- `npm test` passes with no regressions.
|
||||
- `npm run lint` clean.
|
||||
|
||||
**Non-functional**
|
||||
- Coverage not measured explicitly — prioritize meaningful cases over %.
|
||||
- Don't re-test shared helpers per module — one `normalize-name.test.js`
|
||||
is enough.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
tests/
|
||||
├── util/
|
||||
│ └── normalize-name.test.js # NEW (this phase)
|
||||
└── modules/
|
||||
├── loldle-emoji/
|
||||
│ ├── state.test.js # NEW (this phase)
|
||||
│ ├── lookup.test.js # NEW (this phase)
|
||||
│ └── handlers.test.js # NEW (this phase — happy path only)
|
||||
├── loldle-quote/
|
||||
│ ├── state.test.js # NEW (this phase)
|
||||
│ ├── lookup.test.js # NEW (this phase)
|
||||
│ └── handlers.test.js # NEW (this phase)
|
||||
├── loldle-ability/ # DEFERRED (with phase 04)
|
||||
│ ├── state.test.js # slot persistence
|
||||
│ └── handlers.test.js # stubs ctx.replyWithPhoto
|
||||
└── loldle-splash/ # DEFERRED (with phase 05)
|
||||
├── state.test.js # skinId persistence
|
||||
└── handlers.test.js # stubs ctx.replyWithPhoto
|
||||
```
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `README.md` — add the four new modes to the architecture snapshot
|
||||
(bullet list in `## Architecture snapshot`) and to troubleshooting if
|
||||
applicable.
|
||||
- `docs/architecture.md` — if the project's existing docs list modules,
|
||||
mention the loldle family.
|
||||
|
||||
**Create**
|
||||
- Test files listed above.
|
||||
- `docs/loldle-modes.md` (optional, only if worth it) — one-page
|
||||
reference: five modes, what each looks like, command list.
|
||||
|
||||
**Delete:** none.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **`normalize-name.test.js`** — three cases: basic lower+strip, Unicode
|
||||
punctuation ("Kai'Sa" → "kaisa"), empty/null input.
|
||||
|
||||
2. **Per module `state.test.js`** — use `FakeKvNamespace` +
|
||||
`createStore("<module>", { KV: fake })`:
|
||||
- Save a game, load it back — deep equal.
|
||||
- `clearGame` deletes.
|
||||
- `recordResult(true)` increments wins + streak + bestStreak when
|
||||
streak exceeds previous best.
|
||||
- `recordResult(false)` resets streak to 0.
|
||||
- (ability/splash) slot / skinId round-trip.
|
||||
|
||||
3. **Per text module `lookup.test.js`** — exact match, case-insensitive,
|
||||
punctuation-insensitive, unique-prefix, ambiguous-prefix → null.
|
||||
(Could be near-copy of existing loldle lookup test.)
|
||||
|
||||
4. **Per module `handlers.test.js`** — use `FakeKvNamespace` + a minimal
|
||||
ctx fake: `{ from, chat, message, reply, replyWithPhoto, replyWithSticker }`.
|
||||
Walk one happy path: empty state → guess correct → stats incremented.
|
||||
For image modes, assert `replyWithPhoto` received a string URL
|
||||
starting with `https://ddragon.leagueoflegends.com/`.
|
||||
|
||||
5. **Run `npm test`** — confirm all pass. Fix any module code issues
|
||||
found.
|
||||
|
||||
6. **Update `README.md`**:
|
||||
- In `## Architecture snapshot`'s `src/modules/` list, append
|
||||
`loldle-emoji/`, `loldle-quote/`, `loldle-ability/`, `loldle-splash/`.
|
||||
- No troubleshooting table change needed.
|
||||
|
||||
7. **(Optional) `docs/loldle-modes.md`** — single-page mode roster.
|
||||
|
||||
8. **Run `npm run lint` + `npm run format`** — clean.
|
||||
|
||||
9. **Final smoke-test**: `npm run dev`, test bot, cycle through all five
|
||||
loldle commands. Confirm no command conflicts thrown at registry
|
||||
build.
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] `normalize-name.test.js`
|
||||
- [ ] Four per-module `state.test.js`
|
||||
- [ ] Two text-module `lookup.test.js` (emoji, quote — ability/splash
|
||||
reuse the same pattern but lookup is trivial, skip if redundant)
|
||||
- [ ] Four per-module `handlers.test.js`
|
||||
- [ ] Update README.md architecture snapshot
|
||||
- [ ] (Optional) docs/loldle-modes.md
|
||||
- [ ] `npm test` + `npm run lint` + `npm run format` clean
|
||||
- [ ] Final smoke-test across all 5 loldle commands
|
||||
|
||||
## Success criteria
|
||||
|
||||
- All new tests pass.
|
||||
- Classic loldle tests unchanged and still pass.
|
||||
- `npm run deploy --dry-run` (register:dry) lists all 12 new commands
|
||||
(4 modes × 3 commands), with no conflicts.
|
||||
- README accurately lists new modules.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Handler tests drift from actual grammY context shape | Reuse existing loldle handler test scaffolding verbatim |
|
||||
| Flaky tests due to `Math.random()` in `pickRandomChampion` | Inject `rng` parameter or monkey-patch `Math.random` in tests |
|
||||
|
||||
## Security
|
||||
|
||||
- Tests use fakes only; no real KV, no real Telegram calls.
|
||||
- No secrets in test fixtures.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Cron for periodic DDragon refresh? Out of scope — the scraper runs
|
||||
weekly (classic) and we can piggyback ddragon fetch onto the same
|
||||
workflow in a follow-up. Not blocking.
|
||||
|
||||
## Next steps
|
||||
|
||||
After this phase: plan is complete. Run `/ck:plan archive` to close out
|
||||
and log a journal entry.
|
||||
@@ -1,133 +0,0 @@
|
||||
---
|
||||
name: loldle-new-modes
|
||||
status: completed
|
||||
created: 2026-04-24
|
||||
updated: 2026-04-24
|
||||
slug: loldle-new-modes
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
---
|
||||
|
||||
# Loldle New Modes — miti99bot
|
||||
|
||||
Add four new game modules mirroring loldle.net's non-classic modes:
|
||||
**Emoji**, **Quote**, **Ability**, **Splash**. Existing `loldle/` (classic)
|
||||
stays untouched — each new mode is its own sibling module folder.
|
||||
|
||||
**Scope principle (YAGNI):** Ship text-based modes first (emoji, quote).
|
||||
Image modes (ability, splash) ship with **full images, no progressive zoom**
|
||||
— Loldle's signature "reveal-on-wrong-guess" cropping adds Cloudflare Images
|
||||
cost + message-delete jank for little gain on mobile Telegram. Users instead
|
||||
get fewer guesses to compensate.
|
||||
|
||||
**Data strategy:**
|
||||
- Emoji + Quote → scrape from loldle.net JS bundle (same path as classic).
|
||||
- Ability + Splash → pull from **Riot Data Dragon** CDN directly (official,
|
||||
patch-synced, no brittle scraping).
|
||||
- Audio (quote mode) → **skipped for MVP**. Revisit if users ask.
|
||||
|
||||
## Commands (per mode)
|
||||
|
||||
| Mode | Commands |
|
||||
|------|----------|
|
||||
| emoji | `/loldle_emoji`, `/loldle_emoji_giveup`, `/loldle_emoji_stats` |
|
||||
| quote | `/loldle_quote`, `/loldle_quote_giveup`, `/loldle_quote_stats` |
|
||||
| ability | `/loldle_ability`, `/loldle_ability_giveup`, `/loldle_ability_stats` |
|
||||
| splash | `/loldle_splash`, `/loldle_splash_giveup`, `/loldle_splash_stats` |
|
||||
|
||||
All `public`. Conflict-checked at registry load time.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Blocking |
|
||||
|---|-------|--------|----------|
|
||||
| 01 | [Shared scrape + lookup helpers](phase-01-shared-helpers.md) | **done** | — |
|
||||
| 02 | [Emoji module](phase-02-emoji-module.md) | **done** | 01 |
|
||||
| 03 | [Quote module (text-only)](phase-03-quote-module.md) | **done** | 01 |
|
||||
| 04 | [Ability module (Data Dragon)](phase-04-ability-module.md) | **done** | 01 |
|
||||
| 05 | [Splash module (Data Dragon)](phase-05-splash-module.md) | **done** | 01 |
|
||||
| 06 | [Tests + docs sync](phase-06-tests-docs.md) | **done** | 02,03,04,05 |
|
||||
|
||||
**Shipping plan (validated):**
|
||||
- **Now:** 01 → 02 + 03 in parallel → 06 (tests for emoji + quote only).
|
||||
- **Later:** 04 + 05 stay in this plan marked `deferred`. Unblocked by 01,
|
||||
but held by user decision — pick up after emoji/quote live. Tests for
|
||||
them will be added then; phase 06's checklist marks image tests as
|
||||
"when 04/05 ship".
|
||||
|
||||
## Key decisions
|
||||
|
||||
1. **Four new modules, not one refactor.** Classic `loldle/` unchanged. Each
|
||||
mode owns its data, handlers, render — matches the project's existing
|
||||
per-folder plug-n-play pattern. No cross-module coupling.
|
||||
2. **Emoji/quote reuse classic's `champions.json` pool** for name validation;
|
||||
attach mode-specific payload (emoji string, quote text) from scraper.
|
||||
3. **Ability/splash skip cropping for v1.** Send full Data Dragon URL
|
||||
(`sendPhoto`). Guess budget tuned down (ability: 5; splash: 4) since the
|
||||
full image is revealed upfront.
|
||||
4. **Stats tracked per mode.** Each mode's KV prefix keeps stats isolated.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `wrangler.toml` `[vars].MODULES` + `.env.deploy` both updated per module.
|
||||
- `scripts/scrape-loldle-data.js` extended (new regex paths for emoji,
|
||||
quote) — single fetch, mode-aware extraction.
|
||||
- One new script: `scripts/fetch-ddragon-data.js` (abilities + splash meta
|
||||
cached to JSON at build time).
|
||||
|
||||
## References
|
||||
|
||||
- `plans/reports/researcher-260424-2215-loldle-emoji-and-modes-overview.md`
|
||||
- `plans/reports/researcher-260424-2215-loldle-quote-mode.md`
|
||||
- `plans/reports/researcher-260424-2215-loldle-ability-splash-modes.md`
|
||||
- `src/modules/loldle/` — template patterns (handlers, state, lookup, flavor)
|
||||
- `docs/adding-a-module.md`
|
||||
|
||||
## Execution Log
|
||||
|
||||
**Shipped 2026-04-24 (MVP — emoji + quote).**
|
||||
- Phase 01/02/03/06 complete.
|
||||
- Phases 04/05 remain deferred (see validation notes).
|
||||
- **Data-source pivot (critical deviation):** loldle.net bundle contains no
|
||||
per-champion emoji/quote data (confirmed zero emoji code points). Cache
|
||||
is AES-encrypted and holds only the single daily answer. Pivoted:
|
||||
- emoji → algorithmic derivation from classic's `champions.json`
|
||||
metadata (species/regions/resource/positions mapping table).
|
||||
- quote → DDragon champion `title` + first lore sentence, champion
|
||||
name redacted to `___` to avoid giveaways.
|
||||
- Generator: `scripts/fetch-ddragon-data.js` (new). Handles both JSONs.
|
||||
`scrape-loldle-data.js` left untouched (classic only).
|
||||
- 35 new tests, 484 total passing. Lint clean.
|
||||
|
||||
**Shipped 2026-04-24 (deferred phases — ability + splash).**
|
||||
- Phase 04/05 complete. Plan now fully shipped.
|
||||
- **Bundle re-probe:** loldle.net's bundle DOES ship the full splash pool
|
||||
(var `Ad=[...]` — 172 champs × skin-name lists with translations).
|
||||
Scraped it (regex-split on `championName:"…"` markers to handle the
|
||||
nested translations arrays). Ability pool still not in bundle — pulled
|
||||
from DDragon per-champion (172 parallel fetches, concurrency 10).
|
||||
- `fetch-ddragon-data.js` extended: now writes all four JSONs in one run
|
||||
(emojis, quotes, abilities, splashes). Single DDragon per-champion
|
||||
fetch cycle shared between abilities + splash skin IDs.
|
||||
- Splash pool mirrors loldle.net exactly (non-chroma skins, 1939 total
|
||||
skins across 172 champions). URLs from Riot Data Dragon CDN (no
|
||||
version segment — stable across patches).
|
||||
- Credits added to all four loldle-family READMEs + main README.
|
||||
- 19 more tests (503 total). Lint clean. register:dry shows 12 new
|
||||
public commands across the 4 modes with no conflicts.
|
||||
|
||||
## Validation Log
|
||||
|
||||
**Session 1 — 2026-04-24 (7 questions answered)**
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| MVP scope | **Text modes first** (emoji + quote). Image modes deferred. |
|
||||
| Progressive image crop for ability/splash | **Skip** — full image, tight guess budget. |
|
||||
| Splash skin pool (when shipped) | **Random across ALL skins** incl. variants. |
|
||||
| Ability mode flow (when shipped) | **Binary only** — guess champion, done. No slot bonus. |
|
||||
| Phases 04/05 fate | **Keep in plan, marked `deferred`**. Not moved to a new plan. |
|
||||
| Quote mode audio | **Skip**, note in quote README as future follow-up. |
|
||||
| Stats scope | **Per-mode, isolated.** No shared leaderboard. |
|
||||
|
||||
All decisions locked. No open questions remain.
|
||||
@@ -1,160 +0,0 @@
|
||||
# Phase 01 — Atlas Setup + Wrangler Config
|
||||
|
||||
## Context Links
|
||||
- [Atlas fit + driver report](../reports/researcher-260425-1924-mongodb-atlas-fit-and-driver.md) §"MongoDB Driver Specifics"
|
||||
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §1 (M0 ceiling), §4 (region)
|
||||
- [Debugger failure-modes](../reports/debugger-260425-2034-atlas-plan-failure-modes.md) GAP-A, GAP-C, QW-1..5
|
||||
- [Code-reviewer findings #5, #14, #19](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- `wrangler.toml:3` — current `compatibility_date = "2025-10-01"` (already satisfies ≥ 2025-03-20)
|
||||
- `wrangler.toml:14-22` — KV + D1 bindings to retain through cutover
|
||||
- `package.json:24-30` — current deps; will add only `mongodb`
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0 (gate for everything; bundle-size + CPU-time gates can abort the plan here)
|
||||
- **Status:** pending
|
||||
- **Description:** Provision Atlas M0 cluster, wire secrets, enable Node.js compat, run hard gates (bundle size + CPU time + auto-pause behavior). No code merged yet beyond config + secret-leak lint.
|
||||
|
||||
## Key Insights
|
||||
- Atlas M0 only available regions: `aws-ap-southeast-1`, `aws-eu-west-1`, `aws-us-east-1`, `aws-us-west-2`, `aws-ap-southeast-2`. **Pick `aws-ap-southeast-1`** (closest to CF SEA PoPs; user is in VN).
|
||||
- `nodejs_compat_v2` is the lever; without it `node:net`/`node:tls` are absent and the driver fails at import. v1 vs v2 are alternatives, not additive — switching alters process/Buffer/streams globals.
|
||||
- M0 auto-pauses after **30 days of zero ops**. Bot has 6+ daily crons → not a real risk if any cron writes Mongo post-cutover. Phase 08 verifies.
|
||||
- Connection limit: 500. Each cold isolate opens ≥1. Burst risk under deploy stampede; phase-06 tests pre-deploy.
|
||||
- M0 has **no backups**. Source of truth during dual-write is still KV/D1.
|
||||
- **Compatibility date** already `2025-10-01` — satisfies `≥ 2025-03-20` requirement (per debugger QW-5). No bump needed; verify only.
|
||||
- **CF Worker Free plan: 50ms CPU limit**. Mongo TLS+SCRAM CPU cost is unknown; must measure (debugger GAP-A / QW-4).
|
||||
- **Bundle-size gate** (code-reviewer #5): mongodb v6.7 is reported ~4-5 MB compressed; Free plan limit is 3 MiB. Hard gate before any further phase work.
|
||||
- **0.0.0.0/0 IP allowlist** is permanent on M0 + Workers (no static egress IP without paid plan); document as risk, not as TODO.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- Atlas project + cluster provisioned (region `aws-ap-southeast-1`, name `miti99bot-prod`).
|
||||
- DB user `miti99bot-worker` with `readWrite` on db `miti99bot`.
|
||||
- Network access list: **`0.0.0.0/0`** required (CF Workers do not have static IPs). Permanent risk; auth+TLS is the only barrier. Upgrade path: CF Workers paid static egress IP add-on (~$10/mo).
|
||||
- `MONGODB_URI` set as CF secret AND in `.env.deploy`.
|
||||
- `wrangler.toml` updated: `compatibility_flags = ["nodejs_compat_v2"]`. `compatibility_date` left at `2025-10-01` (already valid).
|
||||
- `.env.deploy.example` updated with `MONGODB_URI=` placeholder + comment.
|
||||
- **Secret-leak lint** (`scripts/check-secret-leaks.js`) introduced **here** (not phase-08 — per brainstormer #10) so any later phase commit cannot leak `MONGODB_URI`.
|
||||
- **Node-API surface inventory** (per code-reviewer #14): grep `node:` imports + `process.env` + `Buffer.` uses; document each in `docs/using-mongodb.md`.
|
||||
- **Atlas free-tier email alert** configured (debugger QW-2): cluster unavailability + connections > 400.
|
||||
|
||||
### Non-functional
|
||||
- Connection string never logged (redact in any error path; lint enforces).
|
||||
- README of repo notes M0 auto-pause behavior.
|
||||
- One documented rollback procedure (delete cluster, revert wrangler flag, redeploy — KV/D1 still intact).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Cloudflare Worker (region: nearest CF PoP)
|
||||
│
|
||||
│ TCP/TLS over node:net (nodejs_compat_v2)
|
||||
│ SCRAM-SHA-256 auth
|
||||
▼
|
||||
MongoDB Atlas M0 (aws-ap-southeast-1)
|
||||
└─ db: miti99bot
|
||||
├─ (collections created lazily by Phase 02/03)
|
||||
```
|
||||
|
||||
Cold path: ~1500ms wall-clock (TLS + SCRAM + server selection). Worst-case ≈ 6.5s when server-selection timeout fires (5000ms) on a paused cluster (per code-reviewer #23). Warm path (memoized client per isolate): ~50–100ms.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### MODIFY
|
||||
- `/config/workspace/tiennm99/miti99bot/wrangler.toml` — add `compatibility_flags`, keep KV/D1 bindings.
|
||||
- `/config/workspace/tiennm99/miti99bot/.env.deploy.example` (or create if missing) — add `MONGODB_URI=`.
|
||||
- `/config/workspace/tiennm99/miti99bot/package.json` — add `mongodb@^6.7.0`; add `lint` chain entry for `check-secret-leaks.js`.
|
||||
- `/config/workspace/tiennm99/miti99bot/README.md` — add M0 auto-pause note.
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/using-mongodb.md` — operational runbook (cluster URL, auto-pause behavior, rotation, **node:* surface inventory**, `MongoServerSelectionError` catch path note for phase-02).
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/check-secret-leaks.js` — fails build if any source file contains `console.log(env.MONGODB_URI)` or similar patterns for `MONGODB_URI`, `TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `ADMIN_TOKEN` (latter is now removed in phase-05 redesign — keep for defense-in-depth in case it returns).
|
||||
|
||||
### DELETE
|
||||
- (none in this phase)
|
||||
|
||||
## Implementation Steps
|
||||
1. Create Atlas account / project `miti99bot`. Provision M0 in `aws-ap-southeast-1`. Cluster name `miti99bot-prod`. Note: complete in one sitting (Atlas UI sessions expire; debugger #6).
|
||||
2. Create DB user `miti99bot-worker` with `readWrite@miti99bot`. Strong random password (≥32 chars).
|
||||
3. Network access list → add `0.0.0.0/0` (only viable choice for Workers). Document as permanent risk.
|
||||
4. Configure Atlas free-tier email alerts: (a) cluster unavailable; (b) current connections > 400. (debugger QW-2; 5-min in Atlas UI; zero code.)
|
||||
5. Copy SRV connection string `mongodb+srv://miti99bot-worker:<pass>@<host>/miti99bot?retryWrites=true&w=majority`.
|
||||
6. `wrangler secret put MONGODB_URI` → paste string.
|
||||
7. Add `MONGODB_URI=...` to `.env.deploy` (gitignored). Update `.env.deploy.example` with placeholder.
|
||||
8. Verify `wrangler.toml` `compatibility_date` is `>= 2025-03-20` (current `2025-10-01` qualifies — **no edit**, per debugger QW-5). Add `compatibility_flags = ["nodejs_compat_v2"]`.
|
||||
9. **Node-API surface inventory** (code-reviewer #14): `grep -rn "import.*node:\|process\.env\|Buffer\." src/ scripts/` → list each occurrence in `docs/using-mongodb.md` §"Node API surface". Confirms what `nodejs_compat_v2` must support.
|
||||
10. Create `scripts/check-secret-leaks.js` (per brainstormer #10): grep src/ + scripts/ for `console.log(env.MONGODB_URI)`, `console.log(env.TELEGRAM_BOT_TOKEN)`, `console.log(env.TELEGRAM_WEBHOOK_SECRET)`, `console.log(env.ADMIN_TOKEN)`. Exit 1 on match. Wire into `npm run lint` chain in `package.json`.
|
||||
11. `npm install mongodb@^6.7.0 --save`.
|
||||
12. **HARD GATE — bundle size** (code-reviewer #5 / debugger QW-4):
|
||||
```sh
|
||||
npx wrangler deploy --dry-run --outdir=./.tmp-deploy
|
||||
du -sh ./.tmp-deploy
|
||||
```
|
||||
Abort if `> 2.7 MiB` on Free plan (3 MiB cap, 10% headroom) or `> 9 MiB` on paid (10 MiB cap).
|
||||
On abort: revert phase commits + execute `phase-07-alt-pivot.md`.
|
||||
13. Smoke test from `wrangler dev`: drop a temporary route `/__mongo-ping` that connects, runs `db.runCommand({ping:1})`, returns `{wall_ms, cpu_note: "check CF dashboard CPU column"}`. Run 5+ cold cycles (10-min spaced). Record both wall-clock AND CF dashboard CPU time. **HARD GATE — CPU time** (debugger GAP-A / QW-1): if any cold ping reports CPU time near 50ms on Free plan, the migration is blocked on Free plan; document the result + escalate to user (paid plan or pivot). Save the cold P95 wall-clock as `BASELINE_COLD_PING_MS` — Phase 06 derives the abort threshold from this.
|
||||
14. **Auto-pause behavior test** (debugger GAP-C): in Atlas UI, manually pause the cluster, then hit `/__mongo-ping`. Confirm driver throws a catchable `MongoServerSelectionError` after 5s (not a hang). Document the catch-path requirement for phase-02 §"Connection memoization".
|
||||
15. Delete the temporary `/__mongo-ping` route before commit.
|
||||
16. Write `docs/using-mongodb.md`: cluster name, region, auto-pause schedule, rotation procedure, rollback procedure, node-API surface inventory (step 9 output), baseline cold-ping P95 (step 13 output), `MongoServerSelectionError` catch requirement (step 14), Atlas alert config (step 4), `0.0.0.0/0` permanence + paid-IP upgrade path.
|
||||
17. Run `npm run lint` (now includes `check-secret-leaks.js`). All clean.
|
||||
|
||||
## Todo List
|
||||
- [ ] Atlas project + M0 cluster created (`aws-ap-southeast-1`)
|
||||
- [ ] DB user `miti99bot-worker` created, password vaulted
|
||||
- [ ] Network access `0.0.0.0/0` added with justification comment in Atlas UI
|
||||
- [ ] Atlas email alerts configured (cluster unavailable + connections > 400)
|
||||
- [ ] `MONGODB_URI` set via `wrangler secret put`
|
||||
- [ ] `MONGODB_URI` mirrored in `.env.deploy` (NOT committed)
|
||||
- [ ] `.env.deploy.example` updated
|
||||
- [ ] `wrangler.toml` adds `compatibility_flags = ["nodejs_compat_v2"]` (compatibility_date unchanged)
|
||||
- [ ] Node-API surface grep run + documented
|
||||
- [ ] `scripts/check-secret-leaks.js` written + wired into `npm run lint`
|
||||
- [ ] `mongodb@^6.7.0` installed
|
||||
- [ ] **HARD GATE: bundle-size dry-run ≤ 2.7 MiB (Free) / ≤ 9 MiB (paid)**
|
||||
- [ ] **HARD GATE: cold-ping CPU time well under 50ms** (or paid plan documented)
|
||||
- [ ] Auto-pause behavior tested (catchable error confirmed)
|
||||
- [ ] Temporary `/__mongo-ping` route deleted pre-commit
|
||||
- [ ] Baseline cold-ping P95 recorded in `docs/using-mongodb.md` (drives Phase 06 gate)
|
||||
- [ ] `docs/using-mongodb.md` written
|
||||
- [ ] README mentions M0 auto-pause
|
||||
- [ ] `npm run lint` passes (with secret-leak check)
|
||||
|
||||
## Success Criteria
|
||||
- `wrangler dev` connects to Atlas, ping returns OK.
|
||||
- Bundle-size gate passes.
|
||||
- CPU-time gate passes (or operator escalates to paid plan).
|
||||
- Auto-pause yields catchable error within 5s, not a hang.
|
||||
- Cold-start ping P95 wall-clock recorded as `BASELINE_COLD_PING_MS` for Phase 06 abort threshold derivation.
|
||||
- Rollback steps documented + verifiable (revert flag → redeploy → KV/D1 path intact).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Connection string leaks via log | M | H | `check-secret-leaks.js` wired into lint at this phase. |
|
||||
| `0.0.0.0/0` is exploited | L | H | SCRAM-SHA-256 + TLS; rotate password quarterly; non-guessable username. **Permanent risk** without paid CF static-egress add-on. |
|
||||
| `nodejs_compat_v2` breaks existing modules | L | M | `compatibility_date 2025-10-01` (post-flag-stabilization). Step 9 inventory + `wrangler dev` smoke before any prod deploy. Vitest does NOT catch this (debugger #19); rely on smoke. |
|
||||
| M0 auto-pause hits during low-traffic period | L | M | Bot has 6+ daily crons (any one writing Mongo prevents pause). Phase 02 catches `MongoServerSelectionError` → 503. Phase 08 confirms. |
|
||||
| Bundle exceeds Worker size cap | M | H | **Hard gate at step 12**. Abort to phase-07-alt-pivot if exceeded. |
|
||||
| Cold-start CPU exceeds 50ms (Free plan) | M | CATASTROPHIC | **Hard gate at step 13**. Free plan blocked → escalate to paid OR pivot. |
|
||||
| Atlas API session / MFA expires mid-provisioning | L | M | Step 1 note: complete in one sitting; API token TTL ~30d. |
|
||||
|
||||
## Security Considerations
|
||||
- `MONGODB_URI` contains user+password. Treat as secret-tier (same handling as `TELEGRAM_BOT_TOKEN`).
|
||||
- Password ≥32 chars random.
|
||||
- DB user has `readWrite` only — NOT `dbAdmin` or `clusterAdmin`.
|
||||
- Atlas IP allow-list cannot be tightened without CF Workers static-IP add-on (paid) — accept as documented risk.
|
||||
- All traffic TLS; driver bundles root CA, no manual setup.
|
||||
- Secret-leak lint runs on every `npm run lint` from this phase forward.
|
||||
|
||||
## Rollback (this phase only)
|
||||
1. `wrangler secret delete MONGODB_URI`.
|
||||
2. Revert `wrangler.toml` (remove `compatibility_flags`).
|
||||
3. `npm uninstall mongodb`.
|
||||
4. `npm run deploy` — bot continues on KV/D1 unchanged.
|
||||
5. (Optional) Delete Atlas cluster from UI.
|
||||
6. (Optional) Revert `scripts/check-secret-leaks.js` if migration abandoned entirely. Keep otherwise — rule applies to other secrets too.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 02 (MongoKVStore) needs `MONGODB_URI` available + bundle/CPU gates passed.
|
||||
- **Unblocks:** Phase 02, Phase 03.
|
||||
@@ -1,203 +0,0 @@
|
||||
# Phase 02 — MongoKVStore Implementation
|
||||
|
||||
## Context Links
|
||||
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §2 (KV doc shape), §6 (reference impl)
|
||||
- [Driver report](../reports/researcher-260425-1924-mongodb-atlas-fit-and-driver.md) §"Memoization Pattern"
|
||||
- [Code-reviewer findings #6, #7, #16, #17](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Debugger GAP-C, QW-3](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
- `src/db/kv-store-interface.js` — full contract
|
||||
- `src/db/cf-kv-store.js` — behavioral parity target (108 LOC)
|
||||
- `src/db/create-store.js:40-78` — namespace-prefixing wrapper to mirror
|
||||
- `src/bot.js` `getBot()` — memoization pattern to follow
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0
|
||||
- **Status:** pending
|
||||
- **Description:** Implement `MongoKVStore` (KVStore interface) + `mongo-client.js` shared connection helper. No factory wiring yet — that lands in Phase 04.
|
||||
|
||||
## Key Insights
|
||||
- Store value as **string** (matches `putJSON` serialization). No double-parse risk; preserves null/array/nested fidelity.
|
||||
- Per-module collections (12 KV modules → 12 collections), name == module name with `-` → `_` (e.g. `loldle-emoji` → `loldle_emoji`). (Reviewer #3 recommended single shared collection; user opted to keep per-module.)
|
||||
- TTL index `{ expiresAt: 1 }` with `expireAfterSeconds: 0`, `sparse: true`. Sweeper runs every 60s — stale-read window vs CFKVStore documented + filtered at read time (code-reviewer #7).
|
||||
- Cursor pagination via sorted `_id`, NOT `skip()`. Encode last `_id` as base64.
|
||||
- Memoize `MongoClient` at module scope. **Do NOT** await `connect()` lazily inside every method — first caller awaits, others race the same promise. **On reject, null BOTH `client` and `connectPromise`** (code-reviewer #16) so the next request retries cleanly instead of reusing a dead client.
|
||||
- **`list()` prefix-strip behavior** (code-reviewer #6): MongoKVStore returns keys **WITH prefix preserved** (mirrors CFKVStore). The wrapper in `create-store.js:65` strips. Unambiguous.
|
||||
- **`MongoServerSelectionError`** caught in `getDb()` returning a 503-with-Retry-After path (debugger GAP-C / QW-3) — handles paused-M0 wake without a 5s hang propagating to user.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- `MongoKVStore` implements every method in `kv-store-interface.js`: `get`, `put`, `delete`, `list`, `getJSON`, `putJSON`.
|
||||
- Exact behavioral parity with `CFKVStore` (with TTL stale-window divergence noted):
|
||||
- `get` returns `null` on missing key (NOT `undefined`).
|
||||
- `get` and `getJSON` filter on `expiresAt` at read time (per code-reviewer #7):
|
||||
`findOne({_id, $or: [{expiresAt: {$exists: false}}, {expiresAt: {$gt: new Date()}}]})`.
|
||||
Closes the up-to-60s TTL-sweeper stale-read gap.
|
||||
- `put` with `expirationTtl` writes `expiresAt = now + ttl*1000`. Without it, removes any existing `expiresAt`.
|
||||
- `delete` is idempotent (no-op on missing key).
|
||||
- `list` returns `{ keys, cursor, done }` with `done=true` when no more pages. **Keys returned WITH prefix preserved** (parity with CFKVStore — wrapper strips).
|
||||
- `getJSON` returns `null` on missing OR malformed JSON; logs `console.warn`. Never throws.
|
||||
- `putJSON` throws on `undefined` or cyclic value.
|
||||
- TTL index created idempotently on first connect per collection.
|
||||
- `getDb(env)` catches `MongoServerSelectionError` and rethrows a tagged error so callers can map to 503 + Retry-After.
|
||||
|
||||
### Non-functional
|
||||
- File ≤200 LOC. Split into:
|
||||
- `src/db/mongo-client.js` — singleton client + `getDb(env)` (≤80 LOC).
|
||||
- `src/db/mongo-kv-store.js` — class itself (≤200 LOC; if approaching limit, extract `mongo-list-cursor.js`).
|
||||
- JSDoc on every export.
|
||||
- No `process.env`; only `env.MONGODB_URI`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as Module Handler
|
||||
participant W as create-store.js (Phase 04)
|
||||
participant K as MongoKVStore
|
||||
participant C as mongo-client.js
|
||||
participant A as Atlas
|
||||
|
||||
H->>W: createStore("wordle", env)
|
||||
W->>K: new MongoKVStore(env, "wordle")
|
||||
H->>K: getJSON("games:42")
|
||||
K->>C: getDb(env)
|
||||
alt cold isolate
|
||||
C->>A: TLS + SCRAM (≈1500ms)
|
||||
C-->>K: Db
|
||||
else warm isolate
|
||||
C-->>K: Db (memoized)
|
||||
else paused M0
|
||||
C-->>K: throws tagged MongoServerSelectionError → 503
|
||||
end
|
||||
K->>K: ensureIndex once
|
||||
K->>A: findOne({_id: "wordle:games:42", expiresAt-filter})
|
||||
A-->>K: {value: "{...}"}
|
||||
K-->>H: parsed object
|
||||
```
|
||||
|
||||
### Document shape
|
||||
```js
|
||||
// collection: wordle (per-module)
|
||||
{ _id: "wordle:games:42", value: "{\"word\":\"apple\"}", expiresAt: ISODate? }
|
||||
```
|
||||
|
||||
**Prefix:** the namespace prefix (`wordle:`) is preserved inside `_id` AND in the keys returned by `list()`. The wrapper in `create-store.js:65` strips on the way out (parity with CFKVStore). MongoKVStore does **not** strip prefixes — it stores and returns keys verbatim. Regression test: 2-level prefix (`wordle:games:`) round-trips through wrapper → stripped to `games:`.
|
||||
|
||||
### Connection memoization (with reject-handling)
|
||||
```js
|
||||
// mongo-client.js — sketch (NOT for copy-paste; phase-02 step writes the real version)
|
||||
let client = null;
|
||||
let connectPromise = null;
|
||||
|
||||
export async function getDb(env) {
|
||||
if (client) return client.db("miti99bot");
|
||||
if (!connectPromise) {
|
||||
client = new MongoClient(env.MONGODB_URI, {
|
||||
maxPoolSize: 1,
|
||||
minPoolSize: 0,
|
||||
serverSelectionTimeoutMS: 5000,
|
||||
connectTimeoutMS: 10000,
|
||||
});
|
||||
// Reject path: null BOTH so next call retries cleanly (code-reviewer #16)
|
||||
connectPromise = client.connect().catch((err) => {
|
||||
client = null;
|
||||
connectPromise = null;
|
||||
throw err;
|
||||
});
|
||||
}
|
||||
try {
|
||||
await connectPromise;
|
||||
} catch (err) {
|
||||
// M0 may be auto-paused; surface actionable log (debugger QW-3)
|
||||
if (err?.name === "MongoServerSelectionError") {
|
||||
console.warn(JSON.stringify({ event: "mongo_server_selection_failed", note: "M0 may be paused; caller should map to 503" }));
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
return client.db("miti99bot");
|
||||
}
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/mongo-client.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/mongo-kv-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/fakes/fake-mongo.js` — surface re-derived from phase-03 + phase-02 (code-reviewer #17): `findOne`, `updateOne` (upsert + `$set` + `$unset`), `deleteOne`, `find()` returning chainable `.sort().skip().limit().project().toArray()`, `insertOne`, `insertMany`, `distinct`, `deleteMany`, `countDocuments`, `createIndex` (no-op).
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/mongo-kv-store.test.js`
|
||||
|
||||
### MODIFY
|
||||
- (none in this phase — wiring deferred to Phase 04)
|
||||
|
||||
### DELETE
|
||||
- (none)
|
||||
|
||||
## Implementation Steps
|
||||
1. Create `tests/fakes/fake-mongo.js` first — defines the surface area MongoKVStore + MongoTradesStore (phase-03) must use. Methods (re-derived per code-reviewer #17): `collection(name)` returns object with `findOne`, `updateOne` (with upsert + `$set` + `$unset`), `deleteOne`, `find(query)` returning chainable `.sort().skip().limit().project().toArray()`, `insertOne`, `insertMany`, `distinct`, `deleteMany`, `createIndex` (no-op), `countDocuments`. Backed by `Map<collectionName, Map<_id, doc>>`. TTL is NOT simulated (TTL is server-side; tests check `expiresAt` field only — and the read-time `expiresAt` filter is exercised against `Date.now()`).
|
||||
2. Create `src/db/mongo-client.js`:
|
||||
- `getDb(env)` — module-scope memoized client + connect promise. **On `client.connect()` reject, null both** (code-reviewer #16).
|
||||
- Catch + log `MongoServerSelectionError` with actionable message (debugger QW-3).
|
||||
- `closeMongo()` — for tests/teardown only.
|
||||
- JSDoc on both.
|
||||
3. Create `src/db/mongo-kv-store.js`:
|
||||
- Constructor `(env, collectionName)` — defer connect.
|
||||
- `_ensureIndex()` — runs once per collection per isolate (use a `Set<string>` at module scope).
|
||||
- Methods mirror `cf-kv-store.js` line-for-line (same null semantics, same warn-on-corrupt-JSON), **except**: `get` and `getJSON` filter on `expiresAt` at read time.
|
||||
- `list()` uses `escapeRegex` + `sort({_id:1})` + `limit(N+1)` + base64 cursor of last `_id`. Returns keys WITH prefix.
|
||||
4. Write `tests/db/mongo-kv-store.test.js`:
|
||||
- Inject `fake-mongo` via dependency injection (constructor takes optional `dbOverride` for tests).
|
||||
- Cover: get-missing → null, put → get round trip, putJSON → getJSON round trip, getJSON of corrupt → null + warn, put with TTL writes `expiresAt`, put without TTL clears `expiresAt`, delete idempotent, list with prefix returns keys WITH prefix preserved, **2-level prefix regression** (`wordle:games:`), list with cursor, list `done` flag.
|
||||
- **TTL stale-read regression** (code-reviewer #7): put with `expirationTtl: 1` second, advance time 2s (mock `Date.now()` or use a tiny real sleep), assert `get` returns null even before TTL sweeper would run.
|
||||
- **Connection-reject retry regression** (code-reviewer #16): mock first `connect()` to reject; assert second `getDb()` call retries (does not reuse dead client).
|
||||
- Cover edge: `putJSON(undefined)` throws, `putJSON(circular)` throws.
|
||||
5. `npm test -- mongo-kv-store` → passes.
|
||||
6. `npm run lint` → passes.
|
||||
|
||||
## Todo List
|
||||
- [ ] `tests/fakes/fake-mongo.js` created with full surface (re-derived from phase-02 + phase-03)
|
||||
- [ ] `src/db/mongo-client.js` created (≤80 LOC, JSDoc, reject-resets-state, MongoServerSelectionError logged)
|
||||
- [ ] `src/db/mongo-kv-store.js` created (≤200 LOC, JSDoc)
|
||||
- [ ] `tests/db/mongo-kv-store.test.js` created
|
||||
- [ ] All KVStore methods covered with parity tests vs CFKVStore semantics
|
||||
- [ ] `expiresAt` read-time filter tested (TTL stale-read regression)
|
||||
- [ ] Connect-reject retry regression tested
|
||||
- [ ] 2-level prefix list regression tested
|
||||
- [ ] `list()` cursor pagination tested with > 1 page; keys returned WITH prefix
|
||||
- [ ] `getJSON` corrupt-data path returns null without throwing
|
||||
- [ ] `npm test` passes
|
||||
- [ ] `npm run lint` passes
|
||||
- [ ] No file > 200 LOC
|
||||
|
||||
## Success Criteria
|
||||
- All tests in `mongo-kv-store.test.js` pass.
|
||||
- Behavioral diff vs `cf-kv-store.js` is zero for the 6 KVStore methods (verified by symmetric test cases) **except** the documented TTL stale-read divergence (which the read-time filter eliminates).
|
||||
- `mongo-client.js` connect-promise is awaited exactly once per isolate under concurrent first calls; on reject, both `client` and `connectPromise` are nulled.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| `JSON.parse` throws despite warn-and-null contract | L | M | Wrap in try/catch identical to `cf-kv-store.js:85-91`. |
|
||||
| TTL stale-read window divergence vs CFKVStore | M | L | Read-time `expiresAt` filter (code-reviewer #7); tested via 1s-TTL + 2s-sleep regression. **Functional & risk sections both call out this divergence explicitly.** |
|
||||
| Memoized client lingers across hot-reloads in `wrangler dev` | M | L | `closeMongo()` exposed for test teardown; in prod, isolate teardown handles it. |
|
||||
| Concurrent `_ensureIndex` calls race | L | L | Idempotent on Mongo side. Track per-collection in module-scope `Set` to skip extra round-trips. |
|
||||
| Driver throws on absent `expiresAt` field with sparse index | L | M | Verified by report §2 (sparse:true). Test covers no-TTL case. |
|
||||
| Dead-client reuse after `connect()` rejection | M | H | Reject handler nulls both; regression tested (code-reviewer #16). |
|
||||
| Paused M0 cluster causes hang | M | H | `serverSelectionTimeoutMS: 5000` + caught + logged + caller maps to 503 (debugger GAP-C). |
|
||||
| `list()` cursor encodes `_id` containing colon → base64 fine | L | L | Already opaque per interface contract; no consumer parses cursor. |
|
||||
|
||||
## Security Considerations
|
||||
- Connection only via `env.MONGODB_URI` — never accept URI from request input.
|
||||
- `MongoClient` constructor must NOT log URI on failure. Wrap in try/catch with redacted message.
|
||||
- Reads/writes never echo full document into logs (PII risk: trading user_id).
|
||||
- Test fakes never make network calls.
|
||||
|
||||
## Rollback (this phase only)
|
||||
1. Delete created files.
|
||||
2. `npm uninstall mongodb` (if not needed by Phase 03 yet — but it will be).
|
||||
3. No runtime impact: nothing in Phase 02 is wired into the request path yet.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 04 (dual-write wraps this).
|
||||
- **Unblocks:** Phase 03 can proceed in parallel (independent file).
|
||||
@@ -1,203 +0,0 @@
|
||||
# Phase 03 — MongoTradesStore + Trading Refactor
|
||||
|
||||
## Context Links
|
||||
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §3 (trading mapping), §6 (reference impl)
|
||||
- [Brainstormer Findings #2, #6, #7](../reports/brainstormer-260425-2034-atlas-plan-critique.md)
|
||||
- [Code-reviewer Findings #1, #3, #13](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Debugger #17](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
- `src/db/sql-store-interface.js` — full contract (kept; `MongoSqlStore` is a thin shim)
|
||||
- `src/db/cf-sql-store.js` — behavioral parity target (80 LOC)
|
||||
- `src/modules/trading/history.js`, `src/modules/trading/retention.js` — direct refactor targets
|
||||
- `src/modules/trading/migrations/0001_trades.sql`
|
||||
- `tests/db/create-sql-store.test.js:48-52` — pre-existing assertion that `last_row_id` is present (constrains return shape)
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0
|
||||
- **Status:** pending
|
||||
- **Description:** Replace the SQL-pattern dispatcher approach with a direct **`MongoTradesStore`** (~80 LOC, 6 explicit methods). Refactor `trading/history.js` + `trading/retention.js` to call it directly. Keep `MongoSqlStore` as a thin shim (returns `MongoTradesStore` for the trading module) so `create-sql-store.js` factory branching is unchanged and `tests/db/create-sql-store.test.js` continues to pass.
|
||||
|
||||
## Key Insights
|
||||
- Trading uses 6 distinct queries today (history.js + retention.js). The dispatcher approach (regex-match SQL strings) is brittle: statements 4 & 5 cannot be distinguished by first-30-char prefix (code-reviewer #3); a future 7th statement silently breaks.
|
||||
- **Direct refactor is simpler AND safer** (brainstormer #6): explicit methods make the persistence boundary honest. The trading module already isolates SQL strings to two small files (~30 LOC of changes total).
|
||||
- **Preserve `legacy_id`** (code-reviewer #13 / debugger #17): backfill writes `{_id: ObjectId, legacy_id: <orig_int>, user_id, ...}`. Historical trade IDs in logs/dashboards stay joinable.
|
||||
- **`last_row_id` contract** (code-reviewer #1): existing test `tests/db/create-sql-store.test.js:48-52` asserts `result.last_row_id` is present. `MongoSqlStore.run` returns `{changes: 1, last_row_id: 0}` for inserts (number, NOT hex). Trading code does NOT consume `last_row_id` (verified in step 1) — `0` is safe.
|
||||
- **Confirm zero arithmetic on `id`** (debugger #17): `grep -r "\.id\s*[-+*/<>]" src/modules/trading/` must return empty. If any match, escalate before refactor.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- `src/db/mongo-trades-store.js` (~80 LOC). 6 explicit methods:
|
||||
- `insert(trade)` → `db.trading_trades.insertOne({_id: ObjectId(), legacy_id: null, ...trade})` returns `{changes: 1, last_row_id: 0}`.
|
||||
- `byUser(userId, limit)` → `find({user_id: userId}).sort({ts:-1}).limit(limit).toArray()` → adapt each doc.
|
||||
- `distinctUsers()` → `distinct("user_id")`.
|
||||
- `oldRowsForUser(userId, keepN)` → `find({user_id: userId}).sort({ts:-1}).skip(keepN).project({_id:1}).toArray()` → ids.
|
||||
- `oldRows(keepN)` → `find({}).sort({ts:-1}).skip(keepN).project({_id:1}).toArray()` → ids.
|
||||
- `deleteByIds(ids)` → `deleteMany({_id: {$in: ids.map(toObjectIdOrLegacy)}})`.
|
||||
- `src/db/mongo-sql-store.js` (~40 LOC) is a **thin shim** that satisfies the `SqlStore` interface for `create-sql-store.js` factory branching:
|
||||
- For trading module, returns a wrapper exposing `run`/`all`/`first`/`prepare`/`batch` that delegates to `MongoTradesStore`. `prepare`/`batch` throw `"unsupported"`.
|
||||
- `run` returns `{changes: 1, last_row_id: 0}` for inserts. `0` is a number — satisfies `tests/db/create-sql-store.test.js:48-52`.
|
||||
- `tablePrefix` exposed (matches `create-sql-store.js:39-42`).
|
||||
- Refactor `src/modules/trading/history.js`: replace SQL-string `.run()` / `.all()` / `.first()` calls with `MongoTradesStore.insert` / `.byUser`. (~15 LOC delta.)
|
||||
- Refactor `src/modules/trading/retention.js`: replace SQL-string calls with `.distinctUsers` / `.oldRowsForUser` / `.oldRows` / `.deleteByIds`. (~15 LOC delta.)
|
||||
- During dual-write window the SqlStore interface still delegates to D1 for primary OR Mongo for primary (Phase 04 wraps both). `MongoTradesStore` is the Mongo-side concrete; `CFSqlStore` is the D1-side concrete.
|
||||
|
||||
### Non-functional
|
||||
- `src/db/mongo-trades-store.js` ≤100 LOC.
|
||||
- `src/db/mongo-sql-store.js` ≤80 LOC (thin shim only).
|
||||
- JSDoc on every export.
|
||||
- No SQL strings anywhere in `src/db/mongo-*.js`.
|
||||
- Trading module changes: ~30 LOC total across 2 files.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Module
|
||||
H[trading/history.js]
|
||||
R[trading/retention.js]
|
||||
end
|
||||
subgraph DB
|
||||
S[MongoSqlStore shim]
|
||||
T[MongoTradesStore]
|
||||
D[CFSqlStore]
|
||||
end
|
||||
H -->|insert / byUser| S
|
||||
R -->|distinctUsers / oldRows* / deleteByIds| S
|
||||
S -->|delegates if Mongo primary| T
|
||||
T -->|insertOne / find / distinct / deleteMany| M[(Mongo: trading_trades)]
|
||||
S -.->|delegates if D1 primary| D
|
||||
```
|
||||
|
||||
### Document shape (collection: `trading_trades`)
|
||||
```js
|
||||
{
|
||||
_id: ObjectId,
|
||||
legacy_id: number | null, // D1 autoincrement id, preserved during backfill (code-reviewer #13)
|
||||
user_id: number,
|
||||
symbol: string,
|
||||
side: "buy" | "sell",
|
||||
qty: number,
|
||||
price_vnd: number,
|
||||
ts: number // ms timestamp, parity with D1 column
|
||||
}
|
||||
```
|
||||
|
||||
### Indexes
|
||||
```js
|
||||
db.trading_trades.createIndex({ user_id: 1, ts: -1 });
|
||||
db.trading_trades.createIndex({ ts: -1 });
|
||||
db.trading_trades.createIndex({ legacy_id: 1 }, { sparse: true }); // for legacy joins
|
||||
```
|
||||
|
||||
### Why direct refactor over dispatcher
|
||||
| Concern | Dispatcher | Direct |
|
||||
|---------|-----------|--------|
|
||||
| 7th SQL statement | Silent breakage (M likelihood per dispatcher's own risk row) | Cannot happen — explicit method per use-case |
|
||||
| Statements 4 vs 5 disambiguation | Requires inspecting beyond first 30 chars (code-reviewer #3) | N/A — different methods |
|
||||
| `LIMIT -1 OFFSET ?` sqlite-ism | Handler must ignore LIMIT silently | N/A — `oldRowsForUser/oldRows` use `.skip()` only |
|
||||
| Test surface | Pattern-match coverage of every variation | One test per method |
|
||||
| LOC | ~150 dispatcher + ~200 handlers | ~80 store + ~30 module changes |
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/mongo-trades-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/mongo-sql-store.js` (thin shim, ≤80 LOC)
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/mongo-trades-store.test.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/mongo-sql-store.test.js` (covers shim run/all/first contract — `last_row_id: 0` for inserts)
|
||||
|
||||
### MODIFY
|
||||
- `/config/workspace/tiennm99/miti99bot/src/modules/trading/history.js` — replace SQL strings with `MongoTradesStore` method calls.
|
||||
- `/config/workspace/tiennm99/miti99bot/src/modules/trading/retention.js` — same.
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/modules/trading/*.test.js` — update where SQL strings appear; switch to `MongoTradesStore` mocks.
|
||||
|
||||
### READ FOR CONTEXT (do not edit)
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/sql-store-interface.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/cf-sql-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/modules/trading/migrations/0001_trades.sql`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/create-sql-store.test.js:48-52` — `last_row_id` shape constraint
|
||||
|
||||
### DELETE
|
||||
- (none — D1 binding stays through Phase 07)
|
||||
|
||||
## Implementation Steps
|
||||
1. Grep `src/modules/trading/` for every `.run(`, `.all(`, `.first(`, `.prepare(`, `.batch(` call. Confirm only the 6 known queries. If a 7th appears, list before proceeding (refactor must include it).
|
||||
2. **Verify zero arithmetic on `id`** (debugger #17): `grep -r "\.id\s*[-+*/<>]" src/modules/trading/`. Must return empty. Document the grep result inline in the phase plan.
|
||||
3. Confirm `last_row_id` is unused by trading: `grep -r "last_row_id" src/modules/trading/` — must return empty (only `cf-sql-store.js` and tests reference it).
|
||||
4. Create `src/db/mongo-trades-store.js`:
|
||||
- Constructor `(env)` — defers connect.
|
||||
- 6 methods listed above; each ≤15 LOC.
|
||||
- `_ensureIndexes()` lazy init (3 indexes).
|
||||
- JSDoc.
|
||||
5. Create `src/db/mongo-sql-store.js` thin shim:
|
||||
- Constructor `(env, moduleName)`.
|
||||
- `tablePrefix = ${moduleName}_`.
|
||||
- `run/all/first` delegate to a `MongoTradesStore` instance for trading module by introspecting query for hard-coded markers (or — simpler — accept a method-name in module init). **Decision:** the shim wraps `MongoTradesStore` and only the trading module ever instantiates it; the shim's `run/all/first` exist purely for `SqlStore` interface compliance + the existing factory-test contract. The trading module DOES NOT call the shim — it imports `MongoTradesStore` directly. Shim's `run/all/first` translate the 6 known queries OR throw `"MongoSqlStore: unsupported query — call MongoTradesStore directly"` for everything else.
|
||||
- **Insert returns `{changes: 1, last_row_id: 0}` (number, not hex)** per code-reviewer #1.
|
||||
- `prepare`/`batch` throw `"unsupported in MongoSqlStore"`.
|
||||
6. Refactor `src/modules/trading/history.js`:
|
||||
- Replace SQL-string `.run(...)` with `tradesStore.insert({...})`.
|
||||
- Replace SQL-string `.all(...)` for history with `tradesStore.byUser(userId, limit)`.
|
||||
- Module init signature change: `init({ db, sql, tradesStore, env })` — Phase 04 factory passes the right `tradesStore` (Mongo or null when D1-primary). When Mongo unavailable + D1 primary, fall back to old SQL path through `sql`.
|
||||
7. Refactor `src/modules/trading/retention.js`:
|
||||
- Replace SQL-string `.all(...)` for distinct → `tradesStore.distinctUsers()`.
|
||||
- Replace per-user retention SELECT → `tradesStore.oldRowsForUser(userId, keepN)`.
|
||||
- Replace global retention SELECT → `tradesStore.oldRows(keepN)`.
|
||||
- Replace DELETE IN (...) → `tradesStore.deleteByIds(ids)`.
|
||||
8. Write `tests/db/mongo-trades-store.test.js`:
|
||||
- Inject `fake-mongo` via constructor.
|
||||
- One test per method; cover edge cases (empty list, IDs with `legacy_id` mix, ordering).
|
||||
9. Write `tests/db/mongo-sql-store.test.js`:
|
||||
- Verify `run` returns `{changes, last_row_id: 0}` (number) — satisfies `tests/db/create-sql-store.test.js:48-52`.
|
||||
- Verify `prepare`/`batch` throw.
|
||||
10. Update `tests/modules/trading/*.test.js` to use `MongoTradesStore` mock + injected `tradesStore`.
|
||||
11. `npm test` passes (including `tests/db/create-sql-store.test.js`). `npm run lint` passes.
|
||||
|
||||
## Todo List
|
||||
- [ ] Grep confirms exactly 6 SQL statements in trading module (or extends list)
|
||||
- [ ] **Grep confirms zero arithmetic on `id`** (debugger #17)
|
||||
- [ ] `last_row_id` confirmed unused by trading
|
||||
- [ ] `src/db/mongo-trades-store.js` created (≤100 LOC, 6 methods, JSDoc)
|
||||
- [ ] `src/db/mongo-sql-store.js` shim created (≤80 LOC, returns `last_row_id: 0`)
|
||||
- [ ] `trading/history.js` refactored to use `MongoTradesStore` directly
|
||||
- [ ] `trading/retention.js` refactored
|
||||
- [ ] Trading module init signature accepts `tradesStore`
|
||||
- [ ] `legacy_id` field shape documented + index created
|
||||
- [ ] `tests/db/mongo-trades-store.test.js` covers all 6 methods
|
||||
- [ ] `tests/db/mongo-sql-store.test.js` covers shim contract incl. `last_row_id: 0`
|
||||
- [ ] Trading module tests updated
|
||||
- [ ] `tests/db/create-sql-store.test.js:48-52` still passes (no contract regression)
|
||||
- [ ] `npm test` passes
|
||||
- [ ] `npm run lint` passes
|
||||
|
||||
## Success Criteria
|
||||
- `MongoTradesStore` provides exact behavioral parity with the 6 D1 queries (verified per-method).
|
||||
- Trading module no longer contains SQL string literals.
|
||||
- `MongoSqlStore` shim satisfies `SqlStore` interface for factory branching.
|
||||
- `last_row_id: 0` (number) — `tests/db/create-sql-store.test.js:48-52` unchanged and passing.
|
||||
- `legacy_id` preserved on backfilled trades.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| 7th SQL statement appears mid-refactor | L | M | Step 1 grep gates the refactor. Direct method approach makes additions explicit; cannot silently break. |
|
||||
| `id` arithmetic somewhere unseen | L | H | Step 2 grep gate; if found, escalate. |
|
||||
| `last_row_id: 0` consumer somewhere | L | M | Step 3 grep gate; trading confirmed not to consume. |
|
||||
| Trading test regressions on signature change | M | M | Step 10 updates tests in same commit. Run `npm test` between steps. |
|
||||
| `legacy_id` lookups slow without index | L | L | Sparse index on `legacy_id` created (step 4 lazy init). |
|
||||
| Backwards-compat for in-flight D1 reads during dual-write | L | M | Phase 04 factory selects which concrete (Mongo or D1) per primary flag. Module sees uniform interface. |
|
||||
|
||||
## Security Considerations
|
||||
- Bind values pass through unmodified; no string concatenation into queries.
|
||||
- `_id` exposed as hex; not sensitive (server-generated).
|
||||
- No SQL strings anywhere in trading code post-refactor — eliminates an entire class of injection foot-gun.
|
||||
|
||||
## Rollback (this phase only)
|
||||
1. Revert `src/modules/trading/*.js` changes (single commit).
|
||||
2. Delete `src/db/mongo-trades-store.js` + `src/db/mongo-sql-store.js`.
|
||||
3. No runtime impact (not wired in yet).
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 04 (dual-write SQL wrapper needs the shim + concrete).
|
||||
- **Parallel-safe with:** Phase 02 (independent files; both feed into Phase 04).
|
||||
@@ -1,233 +0,0 @@
|
||||
# Phase 04 — Dual-Write Wrappers + Storage Flag + e2e
|
||||
|
||||
## Context Links
|
||||
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §5 (dual-write mechanics)
|
||||
- [Code-reviewer Findings #2, #8, #18, #20, #27](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Brainstormer Finding #10](../reports/brainstormer-260425-2034-atlas-plan-critique.md) (e2e moves earlier)
|
||||
- [Debugger #8](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
- `src/db/create-store.js` — KV factory to extend (≤80 LOC currently)
|
||||
- `src/db/create-sql-store.js` — SQL factory to extend (≤80 LOC currently)
|
||||
- `scripts/stub-kv.js` — needs sibling `stubMongo` (duck-typed, NOT a string)
|
||||
- `scripts/register.js:75` — `buildRegistry({MODULES, KV: stubKv, AI: stubAi})` call site
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0
|
||||
- **Status:** pending
|
||||
- **Description:** Wrap KV and SQL stores so writes hit BOTH backends and reads hit the configured primary. Single env flag `STORAGE_PRIMARY` toggles read source. **Also lands the e2e storage-roundtrip test** (moved here from phase-08 per brainstormer #10) so dual-write code is end-to-end-tested before any prod soak. This is the ONLY phase that touches the request path's data layer pre-cutover.
|
||||
|
||||
## Key Insights
|
||||
- Single source of truth: `env.STORAGE_PRIMARY` ∈ {`kv`, `mongo`}. Default `kv`.
|
||||
- Dual-write is **always on** when MongoDB credentials are present, regardless of `STORAGE_PRIMARY`. Goal: keep secondary warm + ready for read-flip.
|
||||
- Writes to BOTH must be parallel via `Promise.allSettled`. If secondary write fails → log error AND push the failed key onto a small KV retry queue (`__retry:mongo-failed`) for later drain. Throw only on primary failure. (code-reviewer #8 / debugger #8)
|
||||
- A separate **drift-verifier cron** (1/hr or 1/6h, sample N keys) drains the retry queue + spot-checks parity. Replaces the original "one-shot reconciliation" approach.
|
||||
- During cutover (Phase 07), the flag flips to `mongo`. After soak, secondary writes can be turned off via a second flag `DUAL_WRITE` (default `1`, set to `0` post-cutover).
|
||||
- `register.js` runs at deploy time — does NOT need a Mongo connection. Pass **duck-typed `stubMongo`** + `STUB_SENTINEL` so factories short-circuit even if flags drift (code-reviewer #2).
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- `DualKVStore` class implementing `KVStore` interface:
|
||||
- `get`/`getJSON`/`list` → read from primary only.
|
||||
- `put`/`putJSON`/`delete` → write to both via `Promise.allSettled`. Log secondary failures with key + error AND enqueue to `__retry:mongo-failed` (a KV list bound to the same `env.KV` namespace, prefix `__retry:mongo-failed:`). Throw only on primary failure.
|
||||
- Expose `_kind === "dual"` sentinel for test-side identification (replaces the `_implementations` array seam — code-reviewer #27).
|
||||
- `DualSqlStore` class implementing `SqlStore` interface:
|
||||
- `all`/`first` → read primary only.
|
||||
- `run` → write both. Same fault tolerance as KV (retry-queue same namespace, prefix `__retry:mongo-sql-failed:`).
|
||||
- `prepare`/`batch` → primary only (D1 stays authoritative for legacy paths until cutover).
|
||||
- `create-store.js` factory updated:
|
||||
- Reads `env.STORAGE_PRIMARY` (default `"kv"`) and `env.DUAL_WRITE` (default `"1"`).
|
||||
- **If `env.MONGODB_URI === STUB_SENTINEL`** → unconditionally return CFKVStore-only (deploy-time path).
|
||||
- If `MONGODB_URI` absent → return CFKVStore-only (legacy path).
|
||||
- If `DUAL_WRITE=0` AND `STORAGE_PRIMARY=mongo` → return MongoKVStore-only.
|
||||
- Otherwise → return `DualKVStore(primary, secondary)`.
|
||||
- **Pre-design post-cutover shape** (code-reviewer #20): when both flags are absent / unset, factory returns MongoKVStore-only directly (no flag inspection). Phase-07 simplification just deletes the KV branches.
|
||||
- `create-sql-store.js` factory mirrors logic for SQL (uses `MongoSqlStore` shim from phase-03).
|
||||
- `register.js` + `scripts/stub-kv.js`: add **duck-typed** `stubMongo` (NOT a string) + `STUB_SENTINEL` so deploy-time registry build doesn't crash.
|
||||
- **Drift-verifier cron** (replaces one-shot phase-05 reconciliation):
|
||||
- New module `src/cron/drift-verifier.js` (or co-located in misc) registered in `wrangler.toml [triggers] crons` at 1/hr (configurable).
|
||||
- Drains `__retry:mongo-failed` queue (each entry: re-attempt the secondary write).
|
||||
- Spot-checks parity: sample N keys per module, hash-compare values, log mismatches.
|
||||
- **e2e test** lands HERE (not phase-08 — brainstormer #10): `tests/e2e/storage-roundtrip.test.js` boots a fake env with `MongoKVStore` + `MongoSqlStore` against `fake-mongo.js`, dispatches a representative wordle command + a trading insert, asserts state persisted.
|
||||
|
||||
### Non-functional
|
||||
- Each new file ≤200 LOC.
|
||||
- New files: `src/db/dual-kv-store.js`, `src/db/dual-sql-store.js`, `src/cron/drift-verifier.js`, `tests/e2e/storage-roundtrip.test.js`.
|
||||
- All new exports JSDoc'd.
|
||||
- Behavior must be deterministic given env flags (testable via fake env).
|
||||
- Logging via `console.warn`/`console.error` with structured fields (`{ phase, op, key, err }`).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Read/write flow (during dual-write window)
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as Handler
|
||||
participant F as createStore (factory)
|
||||
participant D as DualKVStore
|
||||
participant K as CFKVStore (primary)
|
||||
participant M as MongoKVStore (secondary)
|
||||
participant Q as KV retry queue
|
||||
|
||||
H->>F: createStore("wordle", env)
|
||||
F->>F: read env.STORAGE_PRIMARY (=kv) + env.DUAL_WRITE (=1)
|
||||
F-->>H: DualKVStore(K, M)
|
||||
H->>D: getJSON("games:42")
|
||||
D->>K: getJSON
|
||||
K-->>D: doc
|
||||
D-->>H: doc
|
||||
|
||||
H->>D: putJSON("games:42", v, {ttl: 86400})
|
||||
par parallel
|
||||
D->>K: putJSON
|
||||
and
|
||||
D->>M: putJSON
|
||||
end
|
||||
K-->>D: ok
|
||||
M--xD: error (caught, logged)
|
||||
D->>Q: enqueue(__retry:mongo-failed:games:42)
|
||||
D-->>H: ok (primary succeeded)
|
||||
```
|
||||
|
||||
### Drift-verifier cron flow
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Cron[1/hr cron] --> V[drift-verifier]
|
||||
V -->|drain| Q[(KV: __retry:mongo-failed)]
|
||||
V -->|sample N keys| K[(CF KV)]
|
||||
V -->|sample N keys| M[(Atlas)]
|
||||
V -->|hash compare| Log[CF Observability]
|
||||
V -->|retry secondary| M
|
||||
```
|
||||
|
||||
### Flag matrix
|
||||
|
||||
| `STORAGE_PRIMARY` | `DUAL_WRITE` | `MONGODB_URI` | Result |
|
||||
|--------------------|--------------|----------------|--------|
|
||||
| (unset) or `kv` | `1` (default) | set | DualKV: read KV, write both |
|
||||
| (unset) or `kv` | `0` | any | CFKVStore only (legacy / rollback) |
|
||||
| `mongo` | `1` | set | DualKV: read Mongo, write both (cutover phase) |
|
||||
| `mongo` | `0` | set | MongoKVStore only (post-cutover) |
|
||||
| any | any | unset | CFKVStore only (Phase 02/03 not deployed yet) |
|
||||
| any | any | === STUB_SENTINEL | CFKVStore only (deploy-time register path) |
|
||||
|
||||
### Stub for register (duck-typed; code-reviewer #2)
|
||||
```js
|
||||
// scripts/stub-kv.js — sketch
|
||||
export const STUB_SENTINEL = "__stub_mongo__";
|
||||
export const stubMongo = {
|
||||
// duck-typed MongoClient surface that no-ops without network IO
|
||||
db() { return { collection: () => { throw new Error("stubMongo: no IO"); } }; },
|
||||
connect: async () => undefined,
|
||||
close: async () => undefined,
|
||||
};
|
||||
// register.js passes env.MONGODB_URI = STUB_SENTINEL; factories short-circuit on sentinel
|
||||
```
|
||||
Test: assert zero `MongoClient.connect()` calls when stub is used (`vi.spyOn(MongoClient.prototype, 'connect')`).
|
||||
|
||||
### Rollback semantics (code-reviewer #18)
|
||||
**Rollback to KV-primary AFTER any Mongo-primary period requires reverse-backfill.** During the Mongo-primary window, secondary writes to KV may have failed silently and KV may be missing rows that exist only in Mongo. Cross-link to Phase 07 reverse-backfill scripts (which become Stage-2 prerequisites — see phase-07 step 11 prereq).
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/dual-kv-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/dual-sql-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/cron/drift-verifier.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/dual-kv-store.test.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/dual-sql-store.test.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/e2e/storage-roundtrip.test.js`
|
||||
|
||||
### MODIFY
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/create-store.js` — read flags, branch, honor STUB_SENTINEL.
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/create-sql-store.js` — same.
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/stub-kv.js` — add `stubMongo` (duck-typed) + `STUB_SENTINEL`.
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/register.js:75` — pass `MONGODB_URI: STUB_SENTINEL, STORAGE_PRIMARY: "kv", DUAL_WRITE: "0"` in env stub.
|
||||
- `/config/workspace/tiennm99/miti99bot/wrangler.toml` — add `[vars] STORAGE_PRIMARY = "kv"`, `DUAL_WRITE = "1"`; add 1/hr cron for drift-verifier.
|
||||
|
||||
### DELETE
|
||||
- (none)
|
||||
|
||||
## Implementation Steps
|
||||
1. Grep `env.KV` and `env.DB` usage outside `src/db/` — should be zero (encapsulated). Confirm.
|
||||
2. Create `src/db/dual-kv-store.js`:
|
||||
- Constructor `(primary, secondary, retryQueue, logger=console)`.
|
||||
- 6 methods. Reads → primary. Writes → `Promise.allSettled` with secondary errors logged AND enqueued to `retryQueue`.
|
||||
- Expose `_kind = "dual"` (code-reviewer #27).
|
||||
- JSDoc.
|
||||
3. Create `src/db/dual-sql-store.js` — same shape, SqlStore interface.
|
||||
4. Create `src/cron/drift-verifier.js`:
|
||||
- Cron handler signature `(event, ctx)`. `ctx.db / ctx.sql` provide the dual store.
|
||||
- Drain `__retry:mongo-failed` (and SQL queue) — re-attempt secondary writes; remove on success.
|
||||
- Sample N keys per module via dual store; hash-compare; log mismatches.
|
||||
- Tunable N via `env.DRIFT_SAMPLE_N` (default 50).
|
||||
5. Modify `src/db/create-store.js`:
|
||||
- **Short-circuit on STUB_SENTINEL** (code-reviewer #2).
|
||||
- Branch on `env.STORAGE_PRIMARY`, `env.DUAL_WRITE`, presence of `env.MONGODB_URI`.
|
||||
- When constructing MongoKVStore, pass collection name (sanitized: replace `-` with `_`).
|
||||
- Keep the existing prefixed-wrapper closure intact.
|
||||
- **Comment the post-cutover shape** (code-reviewer #20): `// post-Phase-07: this entire function returns MongoKVStore-only; KV branches removed`.
|
||||
6. Modify `src/db/create-sql-store.js` mirrored. Use `MongoSqlStore` shim from phase-03; trading module's init wires `MongoTradesStore` directly via `tradesStore` param.
|
||||
7. Add `stubMongo` + `STUB_SENTINEL` to `scripts/stub-kv.js`. Update `scripts/register.js:75` to pass `MONGODB_URI: STUB_SENTINEL, STORAGE_PRIMARY: "kv", DUAL_WRITE: "0"`.
|
||||
8. Update `wrangler.toml` `[vars]` block to declare `STORAGE_PRIMARY = "kv"`, `DUAL_WRITE = "1"`, `DRIFT_SAMPLE_N = "50"`. Add cron `"0 * * * *"` (or `"0 */6 * * *"` if log volume is a concern).
|
||||
9. Tests:
|
||||
- `dual-kv-store.test.js`: write succeeds when both succeed; write succeeds when secondary fails (logs + enqueues retry); write fails when primary fails; reads always from primary; `_kind === "dual"`.
|
||||
- `dual-sql-store.test.js`: same pattern.
|
||||
- **stubMongo regression** (code-reviewer #2): construct factory with `STUB_SENTINEL` + every flag combo; `vi.spyOn(MongoClient.prototype, 'connect')` asserts ZERO calls.
|
||||
- Integration test: factory with all flag combos returns correct concrete type (verify via `_kind` sentinel).
|
||||
- **e2e test** `tests/e2e/storage-roundtrip.test.js`: boot fake env (`fake-mongo`), build registry, dispatch a wordle command (KV path) + trading insert (SQL path), assert persistence.
|
||||
- Update register dry-run test to confirm command list still derives.
|
||||
10. Run `npm test`, `npm run lint`, `npm run register:dry`. All pass.
|
||||
|
||||
## Todo List
|
||||
- [ ] `dual-kv-store.js` created (with retry-queue + `_kind` sentinel)
|
||||
- [ ] `dual-sql-store.js` created
|
||||
- [ ] `src/cron/drift-verifier.js` created + cron registered in wrangler.toml
|
||||
- [ ] `create-store.js` reads env flags + honors STUB_SENTINEL + post-cutover shape commented
|
||||
- [ ] `create-sql-store.js` mirrors logic
|
||||
- [ ] Duck-typed `stubMongo` + `STUB_SENTINEL` added to `stub-kv.js`
|
||||
- [ ] `register.js` passes Mongo stub + flags
|
||||
- [ ] `wrangler.toml` declares `STORAGE_PRIMARY` + `DUAL_WRITE` + `DRIFT_SAMPLE_N` + drift cron
|
||||
- [ ] All test files written, passing
|
||||
- [ ] **stubMongo never reaches MongoClient.connect** asserted in test
|
||||
- [ ] **e2e storage-roundtrip test** passes (wordle KV + trading SQL)
|
||||
- [ ] `npm run register:dry` succeeds
|
||||
- [ ] `npm test` passes
|
||||
- [ ] `npm run lint` passes
|
||||
- [ ] Manual smoke: `wrangler dev` with `MONGODB_URI` set + `DUAL_WRITE=1` → write to wordle module → verify both KV and Mongo received it
|
||||
|
||||
## Success Criteria
|
||||
- All 6 flag/sentinel combos behave per matrix.
|
||||
- Secondary failure does NOT fail user-facing request; failure IS logged AND enqueued to retry queue.
|
||||
- `register:dry` continues to work without Mongo creds; zero MongoClient.connect calls.
|
||||
- Drift-verifier cron drains retry queue + flags any divergence ≥ threshold.
|
||||
- e2e roundtrip passes for both KV (wordle) and SQL (trading) paths against `fake-mongo`.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Secondary write silently masks data divergence | M | H | Retry queue + drift-verifier cron (1/hr). Phase 06 monitors error rate. |
|
||||
| Write amplification doubles latency on every put | H | M | `Promise.allSettled` parallel; latency = max(KV, Mongo). Cold p99 = 1500ms — flagged in Phase 06 abort gate. |
|
||||
| `register:dry` connects to Atlas and fails | L | H | `STUB_SENTINEL` short-circuit; regression-tested in step 9. |
|
||||
| Flag drift between wrangler.toml and .env.deploy | M | M | Phase 08 docs include checklist; lint script checks both files declare `STORAGE_PRIMARY`. |
|
||||
| `MONGODB_URI` missing in prod → factory falls back silently | L | H | Boot-time assertion: if `STORAGE_PRIMARY=mongo` but URI absent (and not STUB_SENTINEL), throw on first request. |
|
||||
| Order-of-operations bug: read after write reads stale primary | L | M | Same-isolate writes to primary block reads (single-threaded JS); no interleaving issue. |
|
||||
| Mongo write fails because collection didn't exist + index not created | L | M | MongoKVStore `_ensureIndex` creates lazily; first write triggers creation. Tested in Phase 02. |
|
||||
| Drift-verifier cron itself burns M0 ops | L | L | Default 1/hr × N=50 = ~1200 reads/day, well under M0 envelope. Tunable via `DRIFT_SAMPLE_N`. |
|
||||
| **Rollback to KV-primary after Mongo-primary loses data** | M | H | **Reverse-backfill scripts (phase-07) are Stage-2 prerequisites** (code-reviewer #18). Cross-linked. |
|
||||
|
||||
## Security Considerations
|
||||
- Logged errors must NOT include the document `value` (PII risk for trading).
|
||||
- Retry-queue values may contain document blobs — same redaction rules for any cron-driven log emission.
|
||||
- `STORAGE_PRIMARY` and `DUAL_WRITE` are public-readable env vars (in `wrangler.toml`). Acceptable — they're routing flags, not secrets.
|
||||
- `STUB_SENTINEL` is a public string; safe — not a credential.
|
||||
|
||||
## Rollback (this phase only)
|
||||
1. Set `DUAL_WRITE=0` in wrangler.toml `[vars]`. Redeploy.
|
||||
2. CFKVStore + CFSqlStore become sole path. Identical to pre-Phase-04 behavior.
|
||||
3. Disable drift-verifier cron (remove from wrangler.toml triggers).
|
||||
4. (Optional) Revert phase commits if rollback is permanent.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 05 (backfill must run with dual-write live so concurrent writes don't bypass Mongo).
|
||||
- **Unblocks:** Phase 05.
|
||||
@@ -1,213 +0,0 @@
|
||||
# Phase 05 — Backfill + Verification (Local-Only, No Admin Routes)
|
||||
|
||||
## Context Links
|
||||
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §5 Phase 3+4 (backfill, verify)
|
||||
- [Brainstormer Finding #5](../reports/brainstormer-260425-2034-atlas-plan-critique.md) (no admin routes)
|
||||
- [Code-reviewer #4, #21](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Debugger #16, #10](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
- `docs/architecture.md` § 10 — explicitly rejects admin HTTP surface; this phase complies.
|
||||
- `scripts/migrate.js` — D1 migration runner pattern (Node script using wrangler envs)
|
||||
- `wrangler.toml:14-22` — KV + D1 binding IDs
|
||||
- 12 KV-using modules: util, wordle, loldle, loldle-emoji, loldle-quote, loldle-ability, loldle-splash, misc, lolschedule, semantle, doantu, twentyq
|
||||
- 1 D1-using module: trading
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0
|
||||
- **Status:** pending
|
||||
- **Description:** One-shot **local-node** scripts to copy historical KV → Mongo, D1 → `trading_trades`. Plus a verifier comparing counts + sample-value hashes. Run AFTER Phase 04 dual-write is live so concurrent writes are already going to Mongo. **No admin HTTP routes** — uses CF KV REST API + `wrangler d1 export` per `docs/architecture.md` § 10 (brainstormer #5).
|
||||
|
||||
## Key Insights
|
||||
- **Order matters**: dual-write must be deployed BEFORE backfill. Otherwise: writes during backfill window go to KV only → backfill misses them OR overwrites newer Mongo state.
|
||||
- **No admin HTTP surface** (architectural compliance per `docs/architecture.md` § 10):
|
||||
- KV reads via CF KV REST API: `https://api.cloudflare.com/client/v4/accounts/{id}/storage/kv/namespaces/{nsid}/keys` (paginated) + `/values/{key}` per key. Account token (already standard for `wrangler kv` ops).
|
||||
- D1 reads via `npx wrangler d1 export miti99bot-db --remote --output=trades.sql` (already used in phase-07; mirror here).
|
||||
- Mongo writes via local node + `mongodb` SDK (no Worker constraints).
|
||||
- Backfill scripts run from operator's local machine; no `ADMIN_TOKEN` secret introduced.
|
||||
- KV `list()` REST endpoint paginated 1000/page. Each value via `/values/{key}` GET. Expect minutes for full sweep across 12 modules. **Metadata** (`expirationTtl`) included on list response — propagate to `expiresAt` on Mongo upsert (debugger #10).
|
||||
- D1 `trading_trades` is small (<300KB / few thousand rows). Single export + insertMany ok.
|
||||
- Mongo Atlas M0 throughput ~100 ops/sec. Budget backfill at 50 ops/sec to leave headroom for live traffic. `await sleep(20)` between writes.
|
||||
- **Upsert semantics**: `updateOne({_id}, {$setOnInsert: {...}, $set: {value, expiresAt}}, {upsert: true})` — skip-if-exists for new docs, but ensure `expiresAt` reflects source TTL.
|
||||
- **Verifier sample size** (code-reviewer #21): `N = √(total) capped at 500`; full-scan compare on collections <10K docs (cheap on M0 with small data).
|
||||
- **Cursor checkpoint** (debugger #16): write last-processed key to `.backfill-cursor-{module}.json` per module so a CPU-budget OOM doesn't lose progress.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- `scripts/backfill-kv-to-mongo.js`:
|
||||
- Local node script using CF KV REST API + `mongodb` SDK directly.
|
||||
- Reads `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN`, `KV_NAMESPACE_ID`, `MONGODB_URI` from `.env.deploy` (mirror of existing `wrangler kv` cred usage).
|
||||
- For each module in `MODULES`: paginated KV list → for each key → if Mongo doc absent, copy with **`expiresAt` propagated from KV `metadata.expirationTtl`** (debugger #10).
|
||||
- **Cursor checkpoint** to `.backfill-cursor-{module}.json` after each REST page; resume on restart (debugger #16).
|
||||
- Logs progress per module: `[wordle] 142 keys: 138 copied, 4 skipped (already in Mongo)`.
|
||||
- Idempotent: re-running is safe (skip-if-exists).
|
||||
- `scripts/backfill-d1-to-mongo.js`:
|
||||
- Local node script.
|
||||
- Step 1: `npx wrangler d1 export miti99bot-db --remote --output=./.backfill/trades.sql`.
|
||||
- Step 2: parse the SQL dump (or use `wrangler d1 execute --command "SELECT * FROM trading_trades" --json --remote` for direct JSON).
|
||||
- For each row: `insertOne` with `_id: ObjectId()`, `legacy_id: row.id` (code-reviewer #13), other fields mapped 1:1.
|
||||
- Pre-flight: skip if `trading_trades.countDocuments() > 0` AND `--force` not passed.
|
||||
- `scripts/verify-mongo-parity.js`:
|
||||
- Local node script. Same CF KV REST API + Mongo SDK + `wrangler d1 execute`.
|
||||
- Per-module: count via REST list (paginate to total) vs Mongo `countDocuments`. Allowable diff: ±1% (live writes during run).
|
||||
- Sample `N = min(500, ceil(sqrt(total)))` random keys per module; SHA256(value) cross-compare. **Full-scan compare for collections <10K docs.** (code-reviewer #21)
|
||||
- For trades: full-scan compare on `legacy_id`, ts, user_id, symbol, qty.
|
||||
- Output report: pass/fail per module, mismatch list with redacted keys.
|
||||
- Exit code: 0 on pass, 1 on fail.
|
||||
- All three scripts read `MONGODB_URI` + CF creds from `.env.deploy` via `node --env-file-if-exists` pattern matching `register.js`.
|
||||
|
||||
### Non-functional
|
||||
- Each script ≤200 LOC. If approaching: extract `scripts/lib/migration-helpers.js` (CF REST pagination, Mongo upsert wrapper, hash helper).
|
||||
- All scripts use the SAME `MongoClient` instance (single connection) per run.
|
||||
- Logs printable + grep-friendly (`[module] action: details`).
|
||||
- Dry-run mode (`--dry-run`) for all three.
|
||||
- **No admin HTTP surface** — zero changes to `src/index.js`; zero new secrets.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Local-only flow (no Worker route)
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Local[backfill-kv-to-mongo.js Node script]
|
||||
REST[CF KV REST API]
|
||||
KV[(CF KV)]
|
||||
Mongo[(Atlas M0)]
|
||||
Cursor[(.backfill-cursor-*.json)]
|
||||
|
||||
Local -->|GET /accounts/.../keys| REST
|
||||
REST -->|enumerate| KV
|
||||
KV --> REST
|
||||
REST -->|page of {key, metadata}| Local
|
||||
Local -->|GET /values/{key}| REST
|
||||
REST --> KV
|
||||
KV --> REST
|
||||
REST -->|value| Local
|
||||
Local -->|updateOne $setOnInsert with expiresAt| Mongo
|
||||
Local -->|checkpoint last key| Cursor
|
||||
```
|
||||
|
||||
### D1 flow
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Local[backfill-d1-to-mongo.js]
|
||||
Wrangler[wrangler d1 execute --remote --json]
|
||||
D1[(D1)]
|
||||
Mongo[(Atlas M0)]
|
||||
|
||||
Local -->|SELECT * FROM trading_trades| Wrangler
|
||||
Wrangler --> D1
|
||||
D1 --> Wrangler
|
||||
Wrangler -->|JSON rows| Local
|
||||
Local -->|insertOne with legacy_id| Mongo
|
||||
```
|
||||
|
||||
### Verification flow
|
||||
```
|
||||
1. For each module (KV-using):
|
||||
a. CF REST: enumerate keys with prefix `module:` → N_kv
|
||||
b. Mongo: countDocuments({}) on collection `module` → N_mongo
|
||||
c. assert |N_kv - N_mongo| / max(N_kv, 1) < 0.01
|
||||
|
||||
2. For each module:
|
||||
- if N_kv < 10000: FULL-SCAN compare (code-reviewer #21)
|
||||
- else: sample N = min(500, ceil(sqrt(N_kv))) random keys
|
||||
- SHA256(KV value) === SHA256(Mongo doc.value) ?
|
||||
- Compare expiresAt bucket presence/absence + within ±5min (debugger #10)
|
||||
|
||||
3. trading_trades:
|
||||
a. D1: SELECT COUNT(*) → N_d1
|
||||
b. Mongo: countDocuments → N_mongo
|
||||
c. FULL-SCAN compare on legacy_id, ts, user_id, symbol, qty (collection is small)
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-kv-to-mongo.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-d1-to-mongo.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/verify-mongo-parity.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/lib/migration-helpers.js` (if size demands)
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/scripts/verify-mongo-parity.test.js` — unit-test helper functions (count diff, hash compare)
|
||||
|
||||
### MODIFY
|
||||
- `/config/workspace/tiennm99/miti99bot/.env.deploy.example` — add placeholders: `CLOUDFLARE_ACCOUNT_ID=`, `CLOUDFLARE_API_TOKEN=` (KV-read scope), `KV_NAMESPACE_ID=`.
|
||||
- `/config/workspace/tiennm99/miti99bot/package.json` `scripts`:
|
||||
- `"backfill:kv": "node --env-file-if-exists=.env.deploy scripts/backfill-kv-to-mongo.js"`
|
||||
- `"backfill:d1": "node --env-file-if-exists=.env.deploy scripts/backfill-d1-to-mongo.js"`
|
||||
- `"verify:mongo": "node --env-file-if-exists=.env.deploy scripts/verify-mongo-parity.js"`
|
||||
|
||||
### DELETE
|
||||
- (none)
|
||||
|
||||
### EXPLICITLY NOT CREATED
|
||||
- ~~`src/admin/dump-routes.js`~~ — DROPPED. Architecture compliance.
|
||||
- ~~`ADMIN_TOKEN` secret~~ — DROPPED.
|
||||
- ~~`/__admin/*` routes in `src/index.js`~~ — DROPPED.
|
||||
|
||||
## Implementation Steps
|
||||
1. Add CF account creds to `.env.deploy` (operator-side; mirror in `.env.deploy.example` placeholders only).
|
||||
2. Write `scripts/backfill-kv-to-mongo.js`:
|
||||
- Connect Mongo via `MONGODB_URI`.
|
||||
- Per module: GET `https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/storage/kv/namespaces/{NS_ID}/keys?prefix=module:` (paginate via `cursor`).
|
||||
- For each key: `GET /values/{key}`; build `{_id: prefixedKey, value, expiresAt: metadata.expiration ? new Date(metadata.expiration*1000) : undefined}`.
|
||||
- `updateOne({_id}, {$setOnInsert: {value, expiresAt}, $set: {}}, {upsert: true})` — skip-if-exists.
|
||||
- Throttle: `await sleep(20)` per write.
|
||||
- Checkpoint last processed key → `.backfill-cursor-{module}.json` after each page (debugger #16).
|
||||
3. Write `scripts/backfill-d1-to-mongo.js`:
|
||||
- Run `npx wrangler d1 execute miti99bot-db --remote --command "SELECT * FROM trading_trades" --json` → parse rows.
|
||||
- Pre-flight: skip if `countDocuments > 0` and no `--force`.
|
||||
- `insertMany` in batches of 100. Each row: `{_id: ObjectId(), legacy_id: row.id, user_id, symbol, side, qty, price_vnd, ts}`.
|
||||
4. Write `scripts/verify-mongo-parity.js`:
|
||||
- Counts + hash-compare per spec. **Full-scan when total < 10000** (code-reviewer #21).
|
||||
- Print human report. Exit code 0/1.
|
||||
- Compare `expiresAt` presence + ±5min bucket (debugger #10).
|
||||
5. Test the verifier with synthetic KV/Mongo via `fake-mongo` + a tiny mock CF REST stub.
|
||||
6. **Dry-run pass first**: `--dry-run` connects, lists modules, prints what WOULD copy without writing.
|
||||
7. **Real run**: dual-write deployed (Phase 04 done) → run `npm run backfill:kv` → `npm run backfill:d1` → `npm run verify:mongo`.
|
||||
|
||||
## Todo List
|
||||
- [ ] CF account ID + API token + KV namespace ID added to `.env.deploy`
|
||||
- [ ] `scripts/backfill-kv-to-mongo.js` written (CF REST + Mongo SDK + cursor checkpoint + expiresAt propagation)
|
||||
- [ ] `scripts/backfill-d1-to-mongo.js` written (legacy_id preserved)
|
||||
- [ ] `scripts/verify-mongo-parity.js` written (full-scan when <10K, sqrt-sample otherwise)
|
||||
- [ ] Helper unit tests pass
|
||||
- [ ] Dry-run all three scripts
|
||||
- [ ] Real run completed; verifier reports PASS for all 13 collections
|
||||
- [ ] Mismatch report saved to `plans/260425-1945-mongodb-atlas-migration/backfill-report.md`
|
||||
- [ ] **No `/__admin/*` routes added; no `ADMIN_TOKEN` secret created**
|
||||
|
||||
## Success Criteria
|
||||
- `verify-mongo-parity.js` exits 0.
|
||||
- All 12 KV modules + `trading_trades` show count parity within 1%.
|
||||
- Hash compare: 0 mismatches on full-scan modules; mismatches on sampled modules explainable by live writes (re-run resolves).
|
||||
- `expiresAt` propagated correctly from KV TTL metadata (verified ±5min bucket).
|
||||
- `backfill-report.md` includes timestamps, durations, counts per module.
|
||||
- **Zero new HTTP routes, zero new secrets** (architectural compliance).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Backfill exhausts M0 throughput → user-facing latency spike | M | H | Throttle 50 ops/sec; run during low-traffic window (UTC 18:00 = local 01:00 VN). Phase 06 monitors. |
|
||||
| CF API token leak via `.env.deploy` accidentally committed | L | H | `.env.deploy` is gitignored; `check-secret-leaks.js` (phase-01) covers MONGODB_URI but NOT CF tokens — extend lint to cover `CLOUDFLARE_API_TOKEN`. |
|
||||
| Live write between backfill read+upsert overwrites newer state | L | M | `$setOnInsert` only (skip-if-exists). Newer dual-write data is preserved. |
|
||||
| `KV.list()` REST cursor expires mid-run | L | M | CF cursors are durable; checkpoint last seen key to disk after each page (debugger #16). |
|
||||
| TTL metadata propagation incorrect → expired data persists in Mongo | L | M | Verify script compares `expiresAt` ±5min bucket (debugger #10). |
|
||||
| `trading_trades` SELECT * blows D1 query cap | L | L | Trading is small; verified via dry-run row count. |
|
||||
| Verifier flags 0.9% mismatch as PASS but real corruption hides | L | M | Full-scan when <10K; sqrt-sample (capped 500) otherwise (code-reviewer #21). |
|
||||
| Backfill OOM mid-stream loses progress | L | M | Cursor checkpoint per page (debugger #16). |
|
||||
| CF REST rate limits during backfill | L | M | 1200 req/5min default for KV reads; throttle adequate. |
|
||||
|
||||
## Security Considerations
|
||||
- **No admin HTTP surface added** — eliminates `ADMIN_TOKEN` rotation, route ordering risk, timing-leak concerns, log-leak concerns from prior plan.
|
||||
- CF API token scope: KV READ + D1 READ only. Operator creates a scoped token; document in `docs/using-mongodb.md`.
|
||||
- Backfill scripts run from operator's local machine; `.env.deploy` is gitignored.
|
||||
- Extend `check-secret-leaks.js` (phase-01) to cover `CLOUDFLARE_API_TOKEN`.
|
||||
|
||||
## Rollback (this phase only)
|
||||
1. Inserts to Mongo are reversible via `db.<collection>.deleteMany({})` (script: `scripts/wipe-mongo.js` — write as part of this phase, gated by interactive `read -p` confirm).
|
||||
2. Delete `.backfill-cursor-*.json` files.
|
||||
3. Revert script commits if rollback is permanent.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 06 (soak needs verified data parity).
|
||||
- **Unblocks:** Phase 06.
|
||||
@@ -1,190 +0,0 @@
|
||||
# Phase 06 — Staged Deploy + Soak (Cold-Start Gate)
|
||||
|
||||
## Context Links
|
||||
- [Driver report](../reports/researcher-260425-1924-mongodb-atlas-fit-and-driver.md) §"Cold-Start Cost" (~1500ms baseline)
|
||||
- [Validation matrix](../reports/researcher-260425-1934-free-db-validation-matrix.md) — Atlas rejected on cold-start grounds; this phase validates user's bet
|
||||
- [Brainstormer Findings #4, #8, #10](../reports/brainstormer-260425-2034-atlas-plan-critique.md)
|
||||
- [Code-reviewer #10, #11](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Debugger GAP-A, GAP-B, #2, #13, Q4](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
- Phase 01 — `BASELINE_COLD_PING_MS` recorded; this phase derives the abort gate from it
|
||||
- `src/bot.js` `getBot()` — memoization model (warm path comparison)
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0 (DECISION GATE)
|
||||
- **Status:** pending
|
||||
- **Description:** Deploy with dual-write live, observe cold-start latency for 24h (extend to 72h only on stop-condition), decide whether to cutover or pivot to Upstash. **This phase contains the hard abort criteria.** Telemetry deployed in Phase 04 is exercised here; no new code paths added except a synthetic burst test pre-deploy.
|
||||
|
||||
## Key Insights
|
||||
- Dual-write means latency = `max(KV, Mongo)` per write. KV ~5ms; Mongo cold ~1500ms; Mongo warm ~50ms. Net effect: write p99 jumps to ~1500ms during cold isolates.
|
||||
- CF Workers spawn new isolates on traffic patterns nobody fully predicts. Cannot guarantee warmth.
|
||||
- User experience proxy: time from `/wordle` keystroke to bot reply. Composed of: Telegram→CF (~50–200ms) + Worker boot + Mongo read + handler logic + reply. **Mongo cold-start is the dominant chunk.**
|
||||
- M0 connection cap = 500. Each cold isolate opens a new client. Pre-deploy synthetic burst test (debugger #2) verifies headroom before live traffic.
|
||||
- **Abort gate is derived from Phase-01 baseline** (brainstormer #4): `2.5 × P95(BASELINE_COLD_PING_MS)`. Not asserted at 3000ms.
|
||||
- **Soak duration = 24h default** (brainstormer #8). Extend to 72h ONLY when stop-condition triggers (see Success/Abort).
|
||||
- **Automated alerts** (debugger GAP-B): Atlas free-tier email alert + CF Observability rule. 5-min config, zero code.
|
||||
- **e2e test already lands in Phase 04** (brainstormer #10) — dual-write code is end-to-end validated before this phase deploys.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- Deploy at start of low-traffic window (UTC 18:00 / local 01:00 VN).
|
||||
- Live config: `STORAGE_PRIMARY=kv`, `DUAL_WRITE=1`, `MONGODB_URI` set.
|
||||
- Telemetry collected (instrumentation lands in Phase 04 or earliest deploy of dual-write — NOT in this phase): per-request timing for `/wordle`, `/loldle`, `/trading`. Bucketed cold (first request in isolate) vs warm.
|
||||
- Telemetry source: `console.log({ts, cmd, isolate_age_ms, mongo_op_ms, total_ms, cold})` parsed from CF Logs.
|
||||
- **Soak window:** default 24h (covers daily cron cycle). Extend to 72h only if {cold-start P95 between 2500ms and the derived gate, error rate >0.5%, traffic <50 req/24h}. (brainstormer #8)
|
||||
- Daily verifier run via `npm run verify:mongo` (manual invocation, captured to `soak-day-N.md`).
|
||||
- **Synthetic burst test pre-deploy** (debugger #2): hit Worker with 20 parallel cold requests, observe Atlas connection counter, abort if >60% of 500 cap.
|
||||
- **Automated alerts** (debugger GAP-B):
|
||||
- Atlas free-tier email alert: cluster unavailability + connections > 400 (already configured in Phase 01 step 4 — verify still active).
|
||||
- CF Observability rule: >10 errors in 1 minute → email.
|
||||
|
||||
### Non-functional
|
||||
- No new code paths added in this phase. Telemetry helper (`src/util/timing.js`) lands with Phase 04 dual-write deploy so from-first-request data exists (debugger #3).
|
||||
- Dashboards: CF Observability built-in views are sufficient; no external service.
|
||||
- Telemetry overhead: `console.log` per request — already captured by CF Observability sampling.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Telemetry helper (lands in Phase 04; SHOWN here for reference — verify against snippet bug, code-reviewer #10)
|
||||
```js
|
||||
// src/util/timing.js
|
||||
export function startTiming(env, cmd) {
|
||||
const t0 = Date.now();
|
||||
const marks = []; // <-- declared at top (was missing; copy-paste-into-prod risk)
|
||||
return {
|
||||
mark(label) { marks.push({ label, dt: Date.now() - t0 }); },
|
||||
end(extra = {}) {
|
||||
const total = Date.now() - t0;
|
||||
console.log(JSON.stringify({ event: "cmd_timing", cmd, total, ...extra, marks }));
|
||||
}
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Cold-start tracking (code-reviewer #11)
|
||||
**Use a module-scoped boolean — NOT `isolate_age_ms < 200ms` (which always misses real cold paths because Mongo connect is 1500ms).** `isolate_age_ms` remains useful as a histogram metric.
|
||||
|
||||
```js
|
||||
// src/index.js — sketch
|
||||
let isFirstRequestInIsolate = true;
|
||||
const ISOLATE_BORN = Date.now();
|
||||
// inside dispatcher:
|
||||
const isCold = isFirstRequestInIsolate;
|
||||
isFirstRequestInIsolate = false;
|
||||
const isolate_age_ms = Date.now() - ISOLATE_BORN; // metric, not classifier
|
||||
console.log({ event: "request", cmd, cold: isCold, isolate_age_ms, total_ms });
|
||||
```
|
||||
|
||||
### Soak data flow
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Users --> CF[CF Worker handler]
|
||||
CF -->|console.log| Obs[CF Observability]
|
||||
CF -->|writes| K[(KV)]
|
||||
CF -->|writes| M[(Mongo)]
|
||||
Obs -->|export| CSV[soak-export.csv]
|
||||
CSV --> Analyzer[scripts/analyze-soak.js]
|
||||
Analyzer --> Report[soak-report.md]
|
||||
Atlas[Atlas] -->|email alert| Operator
|
||||
Obs -->|alert rule| Operator
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/analyze-soak.js` — parse CF logs export → p50/p95/p99 per cmd × cold/warm
|
||||
|
||||
### MODIFY
|
||||
- (telemetry helpers + ISOLATE_BORN landed in Phase 04 — no source-code edits in Phase 06)
|
||||
|
||||
### DELETE
|
||||
- (none)
|
||||
|
||||
## Implementation Steps
|
||||
1. Verify Phase 04 telemetry is live in current deploy. Tail logs for 5 min, confirm `cmd_timing` events emitted with `cold` boolean.
|
||||
2. **Synthetic burst test** (debugger #2): from a local node script or a single test session, fire 20 parallel `/wordle` requests with cache-busting tokens against the deployed Worker. Observe Atlas connection counter in Atlas UI for the next 60s. **Abort if peak > 300 (60% of 500 cap).**
|
||||
3. Verify CF Observability alert rule active: ">10 errors in 1 min → email" (debugger GAP-B). Verify Atlas email alert (cluster unavailable + connections > 400) active.
|
||||
4. **Hour 0–1**: synthetic load test. Trigger 50 cold-start cycles (10+ min spacing OR `wrangler tail` confirming fresh isolate). Record p50/p95/p99 cold-start latency.
|
||||
5. **Hour 1–24**: passive soak. Observe real traffic. Daily snapshot: `npm run verify:mongo` → save report.
|
||||
6. **Hour 24**: run `scripts/analyze-soak.js`. Pull 24h CF Logs export. Compute:
|
||||
- cold-start P50, P95, P99 per command
|
||||
- warm P50, P95, P99
|
||||
- dual-write divergence count (search log for `dual-write:secondary:failed`)
|
||||
- Mongo connection error count
|
||||
- **CPU-time errors** (debugger GAP-A) — search for "Worker exceeded CPU time" in CF logs.
|
||||
7. Decision gate (see Success / Abort below).
|
||||
8. **Cron behavior note** (debugger Q4): during dual-write window, cron handlers receive whichever store the env flag selects. Cron writes during `STORAGE_PRIMARY=kv` go to KV; Mongo's `trading_trades` won't have retention-enforced trims until cutover. Document in `soak-decision.md`. Manual cleanup post-cutover: run `trading-retention` once with Mongo as primary to trim Mongo-side accumulated rows.
|
||||
9. If proceeding: continue soak ONLY IF stop-condition triggers extension to 72h, then go to Phase 07.
|
||||
10. If aborting: see Pivot Path (now linked to phase-07-alt-pivot.md).
|
||||
|
||||
## Todo List
|
||||
- [ ] Phase 04 telemetry confirmed live (cmd_timing events with `cold` boolean)
|
||||
- [ ] Phase 04 e2e storage-roundtrip test passing on current build
|
||||
- [ ] **Synthetic burst test** completed; Atlas connection peak ≤ 300/500
|
||||
- [ ] CF Observability alert rule (>10 errors / min → email) active
|
||||
- [ ] Atlas email alert (unavailable + connections > 400) verified
|
||||
- [ ] Hour-1 synthetic cold-start measurement complete
|
||||
- [ ] Hour-24 passive soak analyzed
|
||||
- [ ] `scripts/analyze-soak.js` produced p50/p95/p99 buckets
|
||||
- [ ] No CPU-time errors in CF logs (debugger GAP-A)
|
||||
- [ ] Daily `verify:mongo` runs (≥1 in 24h soak; ≥3 if extended to 72h) all PASS
|
||||
- [ ] Decision recorded in `soak-decision.md`: PROCEED or ABORT (24h or 72h reason cited)
|
||||
- [ ] If ABORT: execute [phase-07-alt-pivot.md](phase-07-alt-pivot.md)
|
||||
|
||||
## Success Criteria (PROCEED to Phase 07)
|
||||
- Cold-start P95 for `/wordle` and `/loldle` ≤ **`2.5 × BASELINE_COLD_PING_MS`** (derived from Phase-01 step 13; e.g., baseline 1500ms → gate 3750ms).
|
||||
- Warm P95 ≤ 500ms.
|
||||
- Dual-write secondary failure rate < 0.1% over 24h.
|
||||
- M0 connection peak ≤ 400 of 500 cap.
|
||||
- Zero CPU-time exceeded errors in CF logs.
|
||||
- 1 (24h) or 3 (72h) consecutive `verify:mongo` runs PASS with 0 mismatches.
|
||||
- No M0 auto-pause incidents.
|
||||
|
||||
### Soak duration rule
|
||||
- **Default 24h** (covers daily cron cycle).
|
||||
- **Extend to 72h ONLY IF** any of: cold-start P95 between 2500ms and the derived gate, error rate >0.5%, traffic <50 req/24h. (brainstormer #8)
|
||||
- Document the chosen duration + reason in `soak-decision.md`.
|
||||
|
||||
## Abort Criteria (PIVOT to Upstash, do NOT retry)
|
||||
Any ONE of:
|
||||
- Cold-start P95 > derived gate over the soak window.
|
||||
- Dual-write divergence rate > 1% sustained for >1h.
|
||||
- Connection saturation event (>400/500 connections seen).
|
||||
- M0 auto-pause occurs unexpectedly during soak.
|
||||
- Atlas outage > 5min during soak.
|
||||
- Verifier reports >0.5% data drift on any module.
|
||||
- **Worker CPU-time exceeded errors** observed (debugger GAP-A).
|
||||
|
||||
### Pivot Path (if aborted)
|
||||
1. Set `STORAGE_PRIMARY=kv`, `DUAL_WRITE=0` → redeploy. Bot returns to KV/D1 only.
|
||||
2. Leave Mongo cluster + collections in place for forensic analysis (24h, then take a `mongoexport` snapshot).
|
||||
3. Execute [phase-07-alt-pivot.md](phase-07-alt-pivot.md) — pre-written Upstash skeleton.
|
||||
4. Decommission Atlas only after Upstash live and stable for 7 days.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Cold-start > derived gate under real traffic | M (research says probable) | H | Pivot path documented above + pre-written. No retries on Atlas; the limit is architectural. |
|
||||
| Dual-write doubles end-to-end p99 even when warm | H | M | `Promise.allSettled` parallelism keeps warm penalty small (~50ms). p99 dominated by Mongo cold; expected. |
|
||||
| CF Observability log volume exceeds 200k events/day cap | L | M | Sampling already at 1.0 in `wrangler.toml`; can drop to 0.1 if hit. |
|
||||
| Soak window misses peak traffic (weekend pattern) | M | M | Stop-condition extends to 72h when traffic <50 req/24h or borderline cold-start. |
|
||||
| Operator misreads verifier "PASS within 1%" as exact match | L | M | analyze-soak.js prints exact diff numbers. |
|
||||
| Mongo cluster wakes mid-soak triggering 30s+ delay | L | H | Detect via timing logs + Atlas alert; treat as auto-pause incident → ABORT. |
|
||||
| Instrumentation itself adds latency | L | L | `console.log` in CF is async + cheap; <1ms. |
|
||||
| `analyze-soak.js` parser breaks on log format changes | L | L | Pin format; tests on synthetic CSV. |
|
||||
| Burst test triggers M0 cap before live deploy | M | M | Step 2 abort condition prevents proceeding if peak > 300/500. |
|
||||
| Cron-during-soak writes go to wrong primary | M | L | Documented in step 8; manual cleanup post-cutover. |
|
||||
|
||||
## Security Considerations
|
||||
- Telemetry logs: never log message content, user_id raw, command arguments. Only `cmd` (e.g. `/wordle`), durations, isolate age, cold flag.
|
||||
- `analyze-soak.js` runs locally; CSV export has no secrets but may contain user IDs — handle as PII.
|
||||
|
||||
## Rollback (this phase only)
|
||||
- Set `DUAL_WRITE=0` → redeploy. Mongo writes stop. Telemetry remains (cheap, useful diagnostic).
|
||||
- Telemetry can be disabled by deleting `startTiming` calls; not required.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 07 cutover.
|
||||
- **Unblocks:** Phase 07 (if PROCEED) OR phase-07-alt-pivot (if ABORT).
|
||||
- **Decision artifact:** `plans/260425-1945-mongodb-atlas-migration/soak-decision.md` (created during this phase).
|
||||
@@ -1,105 +0,0 @@
|
||||
# Phase 07-ALT — Pivot to Upstash Redis (STANDBY)
|
||||
|
||||
## Context Links
|
||||
- [Free DB validation matrix](../reports/researcher-260425-1934-free-db-validation-matrix.md) §"Final Recommendation" + §"Next Steps (If Upstash Recommended)"
|
||||
- [Brainstormer Finding #4](../reports/brainstormer-260425-2034-atlas-plan-critique.md) — phantom abort path
|
||||
- [Debugger GAP-D / QW-6, #23](../reports/debugger-260425-2034-atlas-plan-failure-modes.md) — pre-write Upstash plan
|
||||
- Phase 06 — abort criteria
|
||||
|
||||
## Overview
|
||||
- **Priority:** STANDBY (execute only if Phase 06 ABORT)
|
||||
- **Status:** standby
|
||||
- **Description:** Skeleton plan to pivot from Atlas → Upstash Redis when the cold-start gate trips. Eliminates "draft new plan under outage pressure" by pre-writing the steps. Estimated ~3-4 days net (smaller than Atlas migration because Upstash is HTTP-native and dual-write doesn't have cold-start amplification).
|
||||
|
||||
## Trigger
|
||||
Execute when ANY Phase-06 abort criterion fires:
|
||||
- Cold-start P95 > derived gate over the soak window.
|
||||
- Dual-write divergence > 1% sustained for >1h.
|
||||
- M0 connection saturation events (>400/500).
|
||||
- M0 auto-pause occurs unexpectedly during soak.
|
||||
- Atlas outage > 5 min during soak.
|
||||
- Verifier reports >0.5% data drift.
|
||||
- Worker CPU-time exceeded errors observed.
|
||||
|
||||
## Key Insights
|
||||
- Upstash Redis is **HTTP-native** (REST API + `@upstash/redis` SDK). No TLS+SCRAM cold-start cost — first request is sub-100ms.
|
||||
- No long-lived connection pool to manage; no `serverSelectionTimeoutMS` hangs; no auto-pause behavior.
|
||||
- Free tier: 10K commands/day, 256MB storage. Sufficient for KV scope (~615KB total data).
|
||||
- Trading (D1) STAYS — Upstash is KV-only; Mongo trading work is dropped or reverted.
|
||||
|
||||
## Approach (5 steps)
|
||||
|
||||
### Step 1 — Provision + secrets (30 min)
|
||||
1. Sign up at https://upstash.com (free).
|
||||
2. Create Redis database in nearest region (likely `us-east-1` or `eu-west-1` since Upstash free tier doesn't offer SEA).
|
||||
3. Copy `UPSTASH_REDIS_REST_URL` + `UPSTASH_REDIS_REST_TOKEN`.
|
||||
4. `wrangler secret put UPSTASH_REDIS_REST_URL`
|
||||
5. `wrangler secret put UPSTASH_REDIS_REST_TOKEN`
|
||||
6. Mirror in `.env.deploy`.
|
||||
|
||||
### Step 2 — UpstashKVStore (~60 LOC; 1-2h)
|
||||
- New file `src/db/upstash-kv-store.js`. Wraps `Redis.fromEnv()` to satisfy the `KVStore` interface.
|
||||
- Methods: `get/put/delete/list/getJSON/putJSON`.
|
||||
- TTL via `EX` flag on `SET`. List via `SCAN` with `MATCH prefix:*`.
|
||||
- `npm install @upstash/redis`. Verify bundle size remains under CF Workers cap (~50KB; very small).
|
||||
|
||||
### Step 3 — Fake + tests (1h)
|
||||
- New `tests/fakes/fake-upstash.js` — Map-backed; satisfy the surface used (`set/get/del/scan/expire`).
|
||||
- Mirror Phase 02 test scaffolding for `UpstashKVStore`.
|
||||
- Reuse the e2e storage-roundtrip test pattern (Phase 04) for KV path only (trading stays D1).
|
||||
|
||||
### Step 4 — Factory wiring (30 min)
|
||||
- Edit `src/db/create-store.js`: add `STORAGE_PRIMARY=upstash` branch.
|
||||
- Drop or revert MongoKVStore branches (depending on what shipped).
|
||||
- Trading: `src/db/create-sql-store.js` returns CFSqlStore (D1) only. Revert phase-03 MongoTradesStore work in `src/modules/trading/*.js` IF shipped, OR drop unshipped.
|
||||
|
||||
### Step 5 — Re-run dual-write + cutover (1-2 days)
|
||||
- Dual-write phase: KV (primary) + Upstash (secondary). Same `DualKVStore` from Phase 04 (parameterized — secondary is now UpstashKVStore).
|
||||
- Backfill from KV → Upstash using the same local-node script pattern as Phase 05 (now writes to Upstash via REST).
|
||||
- Soak 24h (no cold-start risk on Upstash; Phase 06's gate is irrelevant here).
|
||||
- Cutover: `STORAGE_PRIMARY=upstash` → `DUAL_WRITE=0` after 24h overlap → delete CF KV namespace via guarded script (Phase 07 step 19 pattern).
|
||||
|
||||
## Drop / revert from Atlas plan
|
||||
|
||||
### If Phase 02–04 SHIPPED but Phase 07 cutover NOT executed (typical abort timing)
|
||||
- Revert: `src/db/mongo-*.js`, `src/db/dual-*.js` (or re-parameterize for Upstash), `src/cron/drift-verifier.js`.
|
||||
- Revert: `src/modules/trading/*` (if Mongo refactor landed) — restore D1-only path.
|
||||
- Revert: `wrangler.toml` Mongo config + `STORAGE_PRIMARY` flag enum (now includes `upstash`).
|
||||
- Keep: `scripts/check-secret-leaks.js` (general secret hygiene).
|
||||
- Mongo cluster: leave running 7 days for forensic analysis, then `mongoexport` snapshot + delete.
|
||||
|
||||
### Mark plan.md updates
|
||||
- Change `STORAGE_PRIMARY` enum to include `"upstash"`.
|
||||
- Mark Atlas-related files for deletion in Phase 07-ALT cutover.
|
||||
- Add cross-link from plan.md "Abort criteria" section.
|
||||
|
||||
## Estimated Effort
|
||||
- Step 1: 30 min
|
||||
- Step 2: 1-2h
|
||||
- Step 3: 1h
|
||||
- Step 4: 30 min
|
||||
- Step 5: 1-2 days (mostly soak)
|
||||
- **Total: ~3-4 days net**
|
||||
|
||||
## Success Criteria
|
||||
- `UpstashKVStore` passes `KVStore` interface contract tests.
|
||||
- Backfill verifier reports PASS for all 12 KV modules.
|
||||
- Soak shows no error spikes; cold-start latency < 200ms (Upstash HTTP fundamental).
|
||||
- Atlas cluster decommissioned after 7-day forensic window.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Upstash region latency higher than Atlas SEA | M | L | Free-tier regions are US/EU — adds ~150ms vs SEA-routed Atlas, BUT HTTP avoids cold TLS+SCRAM, so net cold path is still faster. |
|
||||
| `@upstash/redis` bundle pushes Worker over size cap | L | H | Bundle is ~50KB; far smaller than mongodb (~4-5 MB). Verify via `wrangler deploy --dry-run` (same gate pattern as Phase 01). |
|
||||
| Free-tier 10K commands/day cap hit | L | M | Audit current ops/day before pivot; if borderline, escalate to paid ($0.20/100K). |
|
||||
| Operator panics mid-pivot, mixes Mongo + Upstash code | M | H | This file IS the pre-written runbook (debugger QW-6); follow steps 1–5 in order. |
|
||||
|
||||
## Rollback
|
||||
- Same as Phase 06 pivot rollback: `STORAGE_PRIMARY=kv`, `DUAL_WRITE=0`. KV remains authoritative until Upstash backfill verified.
|
||||
|
||||
## Next Steps (when triggered)
|
||||
- Inform user: cold-start gate failed; executing Upstash pivot.
|
||||
- Begin Step 1.
|
||||
- Update plan.md status: `Atlas migration: ABORTED on YYYY-MM-DD; Upstash pivot in progress.`
|
||||
@@ -1,239 +0,0 @@
|
||||
# Phase 07 — Cutover + Decommission
|
||||
|
||||
## Context Links
|
||||
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §5 Phase 5–6 (cutover + decommission)
|
||||
- [Code-reviewer #4, #9, #12](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Debugger #12, #14](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
- `wrangler.toml:14-22` — KV + D1 bindings to remove
|
||||
- Phase 04 — flag matrix (this phase flips primary)
|
||||
- Phase 06 — soak decision must be PROCEED before this phase starts
|
||||
- Phase 07-ALT — fallback if soak ABORTed
|
||||
|
||||
## Overview
|
||||
- **Priority:** P0 (irreversible after step 18)
|
||||
- **Status:** pending (blocked on Phase 06 PROCEED)
|
||||
- **Description:** Flip read primary to Mongo, soak Mongo-primary, then disable secondary writes, then delete CF KV + D1 bindings + dependent code. **First reversible step** (flag flip) and **first irreversible step** (binding deletion) clearly demarcated.
|
||||
|
||||
## Key Insights
|
||||
- Cutover is a **two-stage** flip:
|
||||
1. **Read flip** (reversible): `STORAGE_PRIMARY=mongo`, `DUAL_WRITE=1`. Reads now from Mongo, writes still to both. **Rollback = flip flag back.**
|
||||
2. **Write decommission**: `DUAL_WRITE=0` AFTER 24h Stage-2 dual-write overlap (code-reviewer #9). Writes to Mongo only. KV/D1 frozen.
|
||||
3. **Binding deletion** (irreversible): `wrangler kv namespace delete`, `wrangler d1 delete`. Data gone.
|
||||
- **Dual-write overlap** (code-reviewer #9): keep `DUAL_WRITE=1` for the FIRST 24h of Stage 2; only flip to `DUAL_WRITE=0` after 24h of Mongo-primary stability. Reduces single-region M0 data-loss exposure.
|
||||
- **Atlas snapshot at start AND end of Stage 2** (code-reviewer #9): `wrangler d1 export miti99bot-db --remote --output=...` once at Stage-2 start (KV/D1 baseline) and `mongoexport` at end. Local-disk checkpoints, not committed.
|
||||
- 7-day cooldown between each stage. Catch slow-burn issues.
|
||||
- `register.js` stub setup from Phase 04 already handles Mongo absence — no changes needed at deploy time.
|
||||
- Trading module D1 tables are gone post-decommission. The D1 binding in `wrangler.toml` must be removed; `createSqlStore` already returns null when `env.DB` absent — safe.
|
||||
- **`package.json` deploy chain** (code-reviewer #12): MUST remove `&& npm run db:migrate` from `deploy` script AND remove the `db:migrate` script entry. Otherwise next deploy ENOENTs on `scripts/migrate.js`.
|
||||
- **Reverse-backfill scripts pre-built** (code-reviewer #4): `scripts/backfill-mongo-to-kv.js` + `scripts/backfill-mongo-to-d1.js` written BEFORE Stage 2, tested against fakes. Otherwise rollback under outage pressure → guaranteed data loss.
|
||||
- **Stage-2 rollback = N-day game state loss** (debugger #14): explicit user-comm note. Operator informs users before executing rollback.
|
||||
- **Irreversible-step guard** (debugger #12): wrap `wrangler kv namespace delete` + `wrangler d1 delete` in a tiny shell function requiring typed `CONFIRM`. Document: never run in CI.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- **Stage 1 (Read flip)**: deploy with `STORAGE_PRIMARY=mongo`, `DUAL_WRITE=1`. Soak 7 days. Daily `verify:mongo` runs. Reverse-backfill scripts pre-built before this stage.
|
||||
- **Stage 2 (Stop dual-write)**:
|
||||
- Stage-2 day 0: take D1 export snapshot + run `mongoexport` snapshot (start baseline).
|
||||
- Stage-2 day 0–1: deploy with `DUAL_WRITE=1` still on (24h overlap; code-reviewer #9).
|
||||
- Stage-2 day 1: set `DUAL_WRITE=0`. Deploy. Soak remaining 6 days.
|
||||
- Stage-2 day 7: `mongoexport` end snapshot.
|
||||
- **Stage 3 (Final cleanup, irreversible)**:
|
||||
- Verify both Stage-2 snapshots committed to local-disk backups (no commit).
|
||||
- Delete dependent code: `cf-kv-store.js`, `cf-sql-store.js`, `dual-kv-store.js`, `dual-sql-store.js`, drift-verifier cron.
|
||||
- Refactor tests off `fake-d1.js`; delete `fake-d1.js`.
|
||||
- Remove KV + D1 bindings from `wrangler.toml`.
|
||||
- **Edit `package.json`** (code-reviewer #12): remove `&& npm run db:migrate` from `deploy` chain; remove `db:migrate` script entry.
|
||||
- Delete `scripts/migrate.js`.
|
||||
- Wrap KV/D1 namespace deletes in confirm-guard (debugger #12).
|
||||
- `wrangler kv namespace delete --namespace-id <id>`.
|
||||
- `wrangler d1 delete miti99bot-db`.
|
||||
- Update `src/db/create-store.js` to remove KV branch. Become Mongo-only (no flag).
|
||||
- Update `src/db/create-sql-store.js` similarly.
|
||||
|
||||
### Non-functional
|
||||
- Each stage has a single-line summary in `cutover-log.md`.
|
||||
- Backup files committed only as references in plan dir (NOT raw data — privacy).
|
||||
- Bindings removed from `wrangler.toml` in same commit that removes the dependent code.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
gantt
|
||||
title Cutover Sequence
|
||||
dateFormat YYYY-MM-DD
|
||||
section Stage 1
|
||||
Read flip (STORAGE_PRIMARY=mongo) :s1, 2026-04-30, 7d
|
||||
section Stage 2
|
||||
Snapshots + DUAL_WRITE overlap (24h) :s2a, after s1, 1d
|
||||
Stop dual-write (DUAL_WRITE=0) :s2b, after s2a, 6d
|
||||
Stage 2 end snapshot :s2c, after s2b, 1d
|
||||
section Stage 3
|
||||
Verify backups + remove code :s3a, after s2c, 1d
|
||||
Delete bindings + KV/D1 namespaces :s3b, after s3a, 1d
|
||||
```
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Phase 06 PROCEED] --> B[Stage 1: read flip]
|
||||
B -->|7-day soak| C{verifier PASS?}
|
||||
C -->|no| R1[Rollback: STORAGE_PRIMARY=kv]
|
||||
C -->|yes| D[Stage 2: 24h overlap]
|
||||
D --> E[Stage 2: stop dual-write]
|
||||
E -->|6-day soak| F{Mongo errors?}
|
||||
F -->|yes| R2[Rollback: re-enable DUAL_WRITE,<br/>flip primary back, RUN reverse-backfill,<br/>USER-COMM about N-day game state loss]
|
||||
F -->|no| G[Stage 3: snapshots + code cleanup]
|
||||
G --> H[Confirm-guarded namespace delete]
|
||||
H --> Z[Done — Mongo-only]
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE (BEFORE Stage 2 — code-reviewer #4)
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-mongo-to-kv.js` — reverse-backfill script for emergency rollback.
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-mongo-to-d1.js` — same for trading.
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/wrangler-delete-guard.sh` — wraps `wrangler kv namespace delete` / `wrangler d1 delete` in `read -p CONFIRM` prompt (debugger #12).
|
||||
- `/config/workspace/tiennm99/miti99bot/plans/260425-1945-mongodb-atlas-migration/cutover-log.md` — running log of stages, dates, observations.
|
||||
|
||||
### MODIFY
|
||||
- `/config/workspace/tiennm99/miti99bot/wrangler.toml` — Stage 1: change `STORAGE_PRIMARY = "mongo"`. Stage 2 day 1: `DUAL_WRITE = "0"`. Stage 3: remove `[[kv_namespaces]]`, `[[d1_databases]]`, drop `STORAGE_PRIMARY`/`DUAL_WRITE`/`DRIFT_SAMPLE_N`/drift-verifier cron.
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/create-store.js` — Stage 3: remove KV branch, simplify to Mongo-only (matches the post-cutover shape pre-designed in Phase 04).
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/create-sql-store.js` — Stage 3: same.
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/register.js` — Stage 3: replace `stubKv`/`stubMongo` shims with single Mongo stub OR skip Mongo entirely.
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/stub-kv.js` — Stage 3: rename to `stub-bindings.js`, drop KV/AI-only stubs as appropriate.
|
||||
- `/config/workspace/tiennm99/miti99bot/package.json` — **Stage 3 step 17: remove `&& npm run db:migrate` from `deploy` chain AND remove the `db:migrate` script entry** (code-reviewer #12).
|
||||
|
||||
### DELETE (Stage 3 only)
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/cf-kv-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/cf-sql-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/dual-kv-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/db/dual-sql-store.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/cron/drift-verifier.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-kv-to-mongo.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-d1-to-mongo.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/verify-mongo-parity.js` (or move to `archive/`)
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/analyze-soak.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/migrate.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/fakes/fake-d1.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/dual-kv-store.test.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/db/dual-sql-store.test.js`
|
||||
- `/config/workspace/tiennm99/miti99bot/src/modules/trading/migrations/` (D1 migrations no longer relevant)
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-mongo-to-kv.js` (reverse-backfill no longer needed post-cutover)
|
||||
- `/config/workspace/tiennm99/miti99bot/scripts/backfill-mongo-to-d1.js`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Pre-Stage 2 Prerequisites (CODE-REVIEWER #4)
|
||||
0a. Pre-build `scripts/backfill-mongo-to-kv.js` (sketch: enumerate Mongo collection → for each `{_id, value, expiresAt}`, write back to CF KV via REST API + `expirationTtl`). Test against `fake-mongo` + a tiny REST-API mock.
|
||||
0b. Pre-build `scripts/backfill-mongo-to-d1.js` (enumerate `trading_trades` → use `legacy_id` if present, else generate sequential int → emit SQL `INSERT INTO trading_trades` → pipe through `wrangler d1 execute --remote --file`). Test against fakes.
|
||||
|
||||
### Stage 1 — Read flip (reversible)
|
||||
1. Verify Phase 06 PROCEED decision recorded.
|
||||
2. Set `STORAGE_PRIMARY = "mongo"` in `wrangler.toml`.
|
||||
3. `npm run deploy`.
|
||||
4. Smoke: send `/wordle` from a test chat. Confirm reply.
|
||||
5. Tail logs: confirm reads now hit Mongo (`mongo_op_ms` field populated).
|
||||
6. Soak 7 days. Daily `verify:mongo`.
|
||||
7. **Rollback**: revert flag, redeploy. KV still has writes (dual-write still on).
|
||||
|
||||
### Stage 2 — Snapshots + 24h overlap → stop dual-write (reversible until KV deleted)
|
||||
8. **Stage-2 day 0**: take baseline snapshots (code-reviewer #9):
|
||||
- `npx wrangler d1 export miti99bot-db --remote --output=./.backups/d1-stage2-start.sql`
|
||||
- `mongoexport --uri "$MONGODB_URI" --collection trading_trades --out=./.backups/mongo-trades-stage2-start.json` (and per-KV-collection for completeness)
|
||||
9. Deploy with `DUAL_WRITE=1` still on. Run 24h overlap window. Verify dual-write health (no new divergence).
|
||||
10. **Stage-2 day 1**: set `DUAL_WRITE = "0"` in `wrangler.toml`. Deploy. Smoke. Soak 6 more days.
|
||||
11. **Stage-2 day 7**: `mongoexport ... --out=./.backups/mongo-trades-stage2-end.json`.
|
||||
12. **Rollback** (irreversible until KV deletion is the next stage; rollback = real work):
|
||||
- Re-enable `DUAL_WRITE=1`, flip `STORAGE_PRIMARY=kv`. Redeploy.
|
||||
- Run `node scripts/backfill-mongo-to-kv.js && node scripts/backfill-mongo-to-d1.js` to recover Mongo-only writes back into KV/D1.
|
||||
- **Inform users that game state for the last N days is reverting** (debugger #14).
|
||||
|
||||
### Stage 3 — Backup + delete bindings (IRREVERSIBLE after step 18)
|
||||
13. Verify Stage-2 backup files exist on operator local disk.
|
||||
14. Refactor any test still using `fake-d1.js` to `fake-mongo.js`. Then delete `fake-d1.js`.
|
||||
15. Simplify `create-store.js` and `create-sql-store.js` to Mongo-only (matches Phase-04 pre-designed shape). Delete dual-store files + drift-verifier cron + reverse-backfill scripts.
|
||||
16. Delete `cf-kv-store.js` + `cf-sql-store.js`.
|
||||
17. **Edit `package.json`** (code-reviewer #12):
|
||||
- Remove `&& npm run db:migrate` from `deploy` script.
|
||||
- Remove the `db:migrate` script entry.
|
||||
- **Verify `npm run deploy` parses + runs to dry-run completion BEFORE step 18.**
|
||||
18. Remove `[[kv_namespaces]]` + `[[d1_databases]]` from `wrangler.toml`. Remove `STORAGE_PRIMARY` + `DUAL_WRITE` + `DRIFT_SAMPLE_N` vars + drift-verifier cron entry.
|
||||
19. **IRREVERSIBLE** (debugger #12; never run in CI):
|
||||
```sh
|
||||
bash scripts/wrangler-delete-guard.sh kv f7f190fcb2fa42eb84a05542911334b0
|
||||
bash scripts/wrangler-delete-guard.sh d1 miti99bot-db
|
||||
```
|
||||
Each script prompts `Type CONFIRM to delete:` before invoking the destructive wrangler command.
|
||||
20. Delete `scripts/migrate.js`, `scripts/backfill-*.js`, `scripts/verify-mongo-parity.js`, `scripts/analyze-soak.js`, `scripts/wrangler-delete-guard.sh`, trading D1 migrations folder.
|
||||
21. `npm run lint`, `npm test`, `npm run deploy`. All green.
|
||||
22. Update `cutover-log.md` with completion timestamp.
|
||||
|
||||
## Todo List
|
||||
- [ ] **Pre-Stage 2: `scripts/backfill-mongo-to-kv.js` + `scripts/backfill-mongo-to-d1.js` written + tested against fakes**
|
||||
- [ ] Stage-1 deployed (`STORAGE_PRIMARY=mongo`)
|
||||
- [ ] Stage-1 soak 7 days, verifier PASS daily
|
||||
- [ ] **Stage-2 day-0 baseline snapshots** (D1 export + mongoexport)
|
||||
- [ ] Stage-2 24h dual-write overlap completed
|
||||
- [ ] Stage-2 `DUAL_WRITE=0` deployed
|
||||
- [ ] Stage-2 6-day soak, no Mongo errors
|
||||
- [ ] **Stage-2 end-of-window mongoexport** snapshot
|
||||
- [ ] Stage-3 backups verified on local disk
|
||||
- [ ] Tests refactored off `fake-d1`
|
||||
- [ ] `create-store.js` simplified
|
||||
- [ ] `create-sql-store.js` simplified
|
||||
- [ ] Dual stores + drift-verifier + reverse-backfill scripts deleted
|
||||
- [ ] CFKVStore + CFSqlStore deleted
|
||||
- [ ] **`package.json` `deploy` chain edited (no `db:migrate`); `db:migrate` script removed; dry-run deploy passes**
|
||||
- [ ] `wrangler.toml` bindings removed
|
||||
- [ ] `scripts/wrangler-delete-guard.sh` written + manually verified to prompt
|
||||
- [ ] KV namespace deleted via guarded script
|
||||
- [ ] D1 database deleted via guarded script
|
||||
- [ ] `migrate.js` deleted
|
||||
- [ ] Final deploy passes; bot operational on Mongo-only
|
||||
|
||||
## Success Criteria
|
||||
- Bot serves all 13 modules from Mongo only. No KV or D1 reads anywhere in code.
|
||||
- `wrangler.toml` has no KV or D1 bindings.
|
||||
- `package.json` `deploy` script no longer references `db:migrate`.
|
||||
- `npm test` passes with no `fake-d1` references.
|
||||
- Mongo collections exhibit expected sizes (compared against pre-cutover backups).
|
||||
- `cutover-log.md` documents each stage with timestamp + verifier result.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Stage 1 flip surfaces hidden read-path bug not caught in dual-write | M | H | Smoke test in 5 commands first; tail logs for 1h before declaring stable. Rollback flag. |
|
||||
| Stage 2 surfaces write-path bug (KV was masking Mongo write failure) | M | H | 24h overlap window catches early. Phase 04 logged secondary failures; review log diff. |
|
||||
| Single-region M0 data loss during Stage 2 soak | L | H | Snapshots at start AND end of Stage 2 (code-reviewer #9). 24h overlap reduces single-source window. |
|
||||
| Operator forgets to back up before binding delete | L | CRITICAL | Step 13 gates step 19; require Stage-2 snapshot files on disk. |
|
||||
| Wrangler CLI namespace-delete accidentally runs in CI | L | CATASTROPHIC | `scripts/wrangler-delete-guard.sh` requires typed CONFIRM (debugger #12). Documented: never in CI. |
|
||||
| Tests still importing `fake-d1` break after delete | L | M | Step 14 grep `fake-d1` and refactor before step 19. |
|
||||
| Mongo collection accidentally dropped during cleanup | L | CRITICAL | No `db.dropCollection` in any script. Operator-only commands. |
|
||||
| Atlas auto-pause triggers post-cutover during low traffic | L | M | Bot has 6+ daily crons that write Mongo post-cutover. Phase 08 verifies. |
|
||||
| `register.js` breaks because stubKv removed | L | M | Step 15 includes register stub refactor; `npm run register:dry` is a release gate. |
|
||||
| `npm run deploy` breaks on `db:migrate` ENOENT | M | H | **Step 17 fixes package.json BEFORE step 19 namespace delete** (code-reviewer #12). |
|
||||
| Stage-2 rollback reverses N days of Mongo-only writes | guaranteed if rollback | H | Reverse-backfill scripts pre-built; user-comm note required (debugger #14). |
|
||||
|
||||
## Security Considerations
|
||||
- Backup files contain user IDs — store on encrypted disk; do NOT commit.
|
||||
- After binding deletion, the only path to historical data is the local backup file.
|
||||
- Connection string + DB user remain; rotate password at Stage 3 step 21 to invalidate any leaked-during-migration creds.
|
||||
|
||||
## Rollback
|
||||
|
||||
| Stage | Reversible? | Action |
|
||||
|-------|-------------|--------|
|
||||
| Stage 1 | YES | Flip `STORAGE_PRIMARY=kv`, redeploy. Latency: <2 min. |
|
||||
| Stage 2 day 0–1 (overlap) | YES | Flip `STORAGE_PRIMARY=kv` while DUAL_WRITE still on. KV is fresh. |
|
||||
| Stage 2 day 1+ (DUAL_WRITE=0) | YES (with stale KV) | Re-enable `DUAL_WRITE=1` + flip primary. **Run reverse-backfill scripts** to recover Mongo-only writes. **User-comm: N-day game state revert.** |
|
||||
| Stage 3 step 13–18 | YES | Revert commits, redeploy. Bindings still exist. |
|
||||
| Stage 3 step 19+ | NO | Restore from local backup files into a fresh KV namespace. Manual operator process; expect 4–8h. |
|
||||
|
||||
## "If aborting" — link to alternative
|
||||
If at any stage the operator decides to abandon the migration (cold-start regression discovered, M0 limits hit, etc.), execute [phase-07-alt-pivot.md](phase-07-alt-pivot.md). Reverse-backfill scripts (built in pre-Stage 2) are reusable for the abort path.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** Phase 08 (final tests + docs).
|
||||
- **Unblocks:** Phase 08.
|
||||
- **Post-completion:** Atlas cluster monitoring becomes routine. Cost guardrail doc due in Phase 08.
|
||||
@@ -1,179 +0,0 @@
|
||||
# Phase 08 — Tests + Docs (Trimmed)
|
||||
|
||||
## Context Links
|
||||
- All phases 01–07
|
||||
- [Brainstormer #9, #10](../reports/brainstormer-260425-2034-atlas-plan-critique.md)
|
||||
- `docs/architecture.md` § 8 "Swapping the backends"
|
||||
- `tests/fakes/` — fakes inventory
|
||||
- `CLAUDE.md` (root) — project conventions to mirror in new docs
|
||||
- `docs/using-d1.md` — pattern for the new `using-mongodb.md` (Phase 01 began it; this phase finalizes)
|
||||
|
||||
## Overview
|
||||
- **Priority:** P1
|
||||
- **Status:** pending
|
||||
- **Description:** Doc-only finalization. Most tests + secret-leak lint + e2e roundtrip already landed in earlier phases (01, 04). This phase verifies CI integration and updates documentation.
|
||||
|
||||
## Key Insights
|
||||
- All unit tests landed alongside their phase (Phases 02, 03, 04, 05).
|
||||
- **e2e test landed in Phase 04** (brainstormer #10), not here.
|
||||
- **Secret-leak lint landed in Phase 01** (brainstormer #10), not here. Phase 08 verifies CI runs it + existing files are clean.
|
||||
- **M0 auto-pause cron heartbeat is NOT an unresolved question** (brainstormer #9): bot has 6+ daily crons; post-cutover any cron that writes data prevents pause. Phase 08 confirms.
|
||||
- The single biggest doc owe: `docs/using-mongodb.md` — runbook covering provisioning, connection, troubleshooting, auto-pause, password rotation, M0→Flex upgrade. Started in Phase 01; finalized here.
|
||||
- Cost guardrail: M0 is free, but plan for the day it isn't. Stub doc explaining when to upgrade.
|
||||
- Test fakes: `fake-mongo.js` may have grown; freeze the surface and document.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
- **Verify** secret-leak lint runs in CI (`npm run lint` chain) and existing files are clean. (Lint introduced in Phase 01.)
|
||||
- **Verify** e2e storage-roundtrip test (Phase 04) passes in current CI build.
|
||||
- **Verify** daily crons write to Mongo post-cutover (brainstormer #9 — replaces unresolved Q). If `misc` daily cron is read-only, add a single `db.runCommand({ping:1})` line to it.
|
||||
- Doc updates:
|
||||
- `docs/using-mongodb.md` — finalized runbook (started Phase 01).
|
||||
- `docs/architecture.md` — § 8 updated to describe Mongo-only state, removal of CF KV/D1.
|
||||
- `docs/code-standards.md` — add note: "All persistence goes through `MongoKVStore` / `MongoTradesStore`. Modules NEVER touch the Mongo client directly."
|
||||
- `docs/cost-tracking.md` (new, stub) — when M0 fills, what triggers upgrade, expected cost ladder (M0 → Flex → M10).
|
||||
- `docs/development-roadmap.md` — **verify migration is NOT listed** (per user feedback memory: roadmap = future-only). Remove if present.
|
||||
- `docs/project-changelog.md` — append migration entry.
|
||||
- `README.md` — update database section.
|
||||
- `CLAUDE.md` (root) — update Architecture section: replace KV/D1 references with Mongo.
|
||||
- Append "Alternatives considered" section to plan.md (already done at plan creation; verify still accurate).
|
||||
|
||||
### Non-functional
|
||||
- New docs ≤200 lines each.
|
||||
- Tests use existing fakes — no new mocks created in this phase.
|
||||
- Run final `npm run lint && npm test && npm run register:dry` before sign-off.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Tests[tests/]
|
||||
Tests --> U1[unit: mongo-kv-store.test.js — Phase 02]
|
||||
Tests --> U2[unit: mongo-trades-store.test.js — Phase 03]
|
||||
Tests --> U3[unit: mongo-sql-store.test.js — Phase 03]
|
||||
Tests --> U4[unit: dual-*.test.js — DELETED in Phase 07]
|
||||
Tests --> E[e2e: storage-roundtrip.test.js — Phase 04, persists to Mongo-only post-Phase-07]
|
||||
Tests --> Mod[module unit tests — unchanged, use fake-mongo]
|
||||
|
||||
Docs[docs/]
|
||||
Docs --> D1[architecture.md — UPDATED]
|
||||
Docs --> D2[using-mongodb.md — finalized]
|
||||
Docs --> D3[cost-tracking.md — NEW stub]
|
||||
Docs --> D4[code-standards.md — UPDATED]
|
||||
Docs --> D5[project-changelog.md — APPENDED]
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
|
||||
### CREATE
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/cost-tracking.md` (stub)
|
||||
|
||||
### MODIFY
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/architecture.md` — § 8 + any KV/D1 reference
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/using-mongodb.md` — finalize from Phase-01 partial
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/code-standards.md`
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/project-changelog.md` (append)
|
||||
- `/config/workspace/tiennm99/miti99bot/docs/development-roadmap.md` (verify migration NOT listed; remove if present)
|
||||
- `/config/workspace/tiennm99/miti99bot/README.md`
|
||||
- `/config/workspace/tiennm99/miti99bot/CLAUDE.md` (root) — Architecture section
|
||||
- `/config/workspace/tiennm99/miti99bot/tests/fakes/fake-mongo.js` — comment block documenting frozen surface
|
||||
- `/config/workspace/tiennm99/miti99bot/src/modules/misc/index.js` — IF the daily cron does not already write data, add a `db.runCommand({ping:1})` line (brainstormer #9 — replaces unresolved Q)
|
||||
|
||||
### DELETE
|
||||
- (none new in this phase; Phase 07 already removed all CF-specific code)
|
||||
|
||||
## Implementation Steps
|
||||
1. Grep all `docs/` and `README.md` for `KV`, `Cloudflare KV`, `D1`, `CFKVStore`, `CFSqlStore`. Build edit list.
|
||||
2. Update `docs/architecture.md` § 8: replace "Swap CFKVStore → other backend" narrative with "Mongo is the backend; swap target is `mongo-kv-store.js`". Update §10 unchanged (no admin HTTP surface — Phase 05 complies).
|
||||
3. Finalize `docs/using-mongodb.md` (started Phase 01):
|
||||
- Provisioning steps (Atlas UI walkthrough)
|
||||
- Connection string format
|
||||
- Auto-pause behavior + how to wake (just send a request)
|
||||
- Password rotation procedure (90 days, owner = repo maintainer)
|
||||
- Failure modes: connection error, SCRAM auth fail, SRV resolution fail, server-selection timeout (paused M0)
|
||||
- Node-API surface inventory (from Phase 01 step 9)
|
||||
- Atlas alert config (cluster unavailable + connections > 400)
|
||||
- 0.0.0.0/0 IP allowlist permanence + paid CF static-egress upgrade path
|
||||
- Baseline cold-ping P95 (Phase 01 step 13)
|
||||
- When to upgrade (link to cost-tracking.md)
|
||||
4. Write `docs/cost-tracking.md` stub:
|
||||
- M0 limits (storage 512MB, connections 500, ~100 ops/sec)
|
||||
- Triggers to upgrade: storage > 400MB sustained, connection saturation, ops/sec degraded
|
||||
- Cost ladder: M0 (free) → Flex ($8–30/mo) → M10 ($57/mo)
|
||||
- Monthly review checklist
|
||||
5. Update `docs/code-standards.md`: add Mongo persistence rule (modules go through `MongoKVStore` / `MongoTradesStore`).
|
||||
6. Append to `docs/project-changelog.md` with date, summary, plan link.
|
||||
7. Verify `docs/development-roadmap.md` does NOT list this migration. Remove if present (per user feedback memory: roadmap = future-only).
|
||||
8. Edit root `CLAUDE.md` Architecture section: replace KV/D1 mentions with Mongo.
|
||||
9. Edit root `README.md` database section.
|
||||
10. **Verify daily cron writes Mongo** (brainstormer #9): grep `src/modules/misc/index.js` cron handler for any `db.put`/`tradesStore.insert`/equivalent. If absent, add a single `await db.put("misc:last_cron_ping", String(Date.now()))` line. Document inline.
|
||||
11. **Verify e2e test passes** (brainstormer #10): `npm test -- tests/e2e/storage-roundtrip.test.js`.
|
||||
12. **Verify secret-leak lint runs** (brainstormer #10): `npm run lint` — confirm `scripts/check-secret-leaks.js` is in the chain and passes on current source.
|
||||
13. Run `npm run lint`, `npm test`, `npm run register:dry`. All pass.
|
||||
14. Final commit message: `feat(db): complete migration to MongoDB Atlas M0`.
|
||||
|
||||
## Todo List
|
||||
- [ ] All `docs/` references to KV/D1 grep-clean
|
||||
- [ ] `docs/architecture.md` § 8 rewritten
|
||||
- [ ] `docs/using-mongodb.md` finalized (with all Phase-01 + Phase-07 additions)
|
||||
- [ ] `docs/cost-tracking.md` stub written
|
||||
- [ ] `docs/code-standards.md` updated
|
||||
- [ ] `docs/project-changelog.md` appended
|
||||
- [ ] `docs/development-roadmap.md` verified migration NOT listed
|
||||
- [ ] `README.md` database section updated
|
||||
- [ ] Root `CLAUDE.md` Architecture section updated
|
||||
- [ ] **Daily cron Mongo-write verified** (brainstormer #9 closes the unresolved Q)
|
||||
- [ ] e2e storage-roundtrip test passing in CI
|
||||
- [ ] Secret-leak lint passing in CI
|
||||
- [ ] `tests/fakes/fake-mongo.js` surface documented in header comment
|
||||
- [ ] Final `npm run lint && npm test && npm run register:dry` all green
|
||||
|
||||
## Success Criteria
|
||||
- `grep -ri "Cloudflare KV" docs/ src/ README.md` returns zero relevant hits.
|
||||
- `grep -ri "CFKVStore\|CFSqlStore" .` returns zero hits (in source; git history is fine).
|
||||
- E2E test passes for both wordle (KV path) and trading (Mongo path) modules.
|
||||
- Lint catches a deliberately-introduced `console.log(env.MONGODB_URI)` (negative test, then revert).
|
||||
- Documentation reads coherently end-to-end: README → architecture → using-mongodb → cost-tracking.
|
||||
- Daily cron confirmed writing Mongo (auto-pause non-issue).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Doc drift between architecture.md and code reality | M | M | Step 1 grep comprehensive; reviewer verifies against git diff. |
|
||||
| E2E test passes against fake but fails against real Atlas | L | M | Phase 06 soak validated against real Atlas; e2e here is regression coverage. |
|
||||
| Lint script false-positives break CI | L | M | Tight regex (Phase 01); add allowlist comments for legitimate references in tests if needed. |
|
||||
| `cost-tracking.md` becomes stale | H | L | Stub by design; review monthly per `documentation-management.md`. |
|
||||
| Future contributor adds raw `MongoClient` use bypassing store | M | M | code-standards.md note + lint rule (if feasible) flags `import { MongoClient }` outside `src/db/`. |
|
||||
|
||||
## Security Considerations
|
||||
- `using-mongodb.md` must not include real connection string (only redacted form).
|
||||
- Cost-tracking doc references billing, not credentials.
|
||||
- Lint check covers `MONGODB_URI`, `TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `CLOUDFLARE_API_TOKEN`.
|
||||
|
||||
## Rollback
|
||||
- All edits are doc + tests; revert commits if needed without runtime impact.
|
||||
|
||||
## Next Steps
|
||||
- **Blocks:** none (final phase).
|
||||
- **Post-completion:** schedule monthly cost-tracking review; confirm Atlas dashboard alerts wired (storage > 400MB, connections > 400 — Phase 01 already configured).
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions (plan-level)
|
||||
|
||||
1. **`tests/fakes/fake-mongo.js` TTL semantics** — Phase 02 deferred TTL simulation. Read-time `expiresAt` filter is testable (covered in Phase 02 `Date.now()` mock). Acceptable: assert `expiresAt` field presence; trust real Atlas TTL in soak.
|
||||
2. **`MongoSqlStore` future query expansion** — Phase 03 replaced dispatcher with direct `MongoTradesStore`. New trading queries = new explicit method, not silent dispatcher break. Closed by direct refactor.
|
||||
3. **`MONGODB_URI` rotation cadence** — 90 days; owner = repo maintainer; documented in `docs/using-mongodb.md` rotation section. Calendar entry in `docs/cost-tracking.md` review cycle.
|
||||
|
||||
## Alternatives considered (reviewer dissent)
|
||||
|
||||
Reviewers (see brainstormer + code-reviewer + debugger reports in `plans/reports/`) recommended:
|
||||
1. **Maintenance-window cutover** (~10 min downtime) instead of dual-write — saves ~6h, removes latency amplification.
|
||||
2. **Defer trading migration** entirely (D1 free tier handles ~100 writes/day indefinitely).
|
||||
3. **Single shared `kv` collection** instead of 12 per-module collections.
|
||||
4. **Pivot to Upstash** before starting (smaller bundle, HTTP-native, no cold-start TLS cost).
|
||||
5. **`MongoTradesStore` direct refactor** instead of SQL-pattern dispatcher — APPLIED (Phase 03).
|
||||
|
||||
User chose to proceed with full Atlas migration to validate cold-start UX firsthand. Reviewer correctness/safety findings have been applied throughout phases 01–08; architectural recommendations 1–4 are documented but not adopted. If the cold-start gate trips, [phase-07-alt-pivot.md](phase-07-alt-pivot.md) executes recommendation 4.
|
||||
@@ -1,76 +0,0 @@
|
||||
---
|
||||
title: "Migrate miti99bot from CF KV+D1 to MongoDB Atlas M0"
|
||||
description: "Dual-write migration to Atlas M0 with explicit cold-start abort threshold and Upstash pivot path."
|
||||
status: superseded
|
||||
priority: P2
|
||||
effort: 22h
|
||||
branch: main
|
||||
tags: [storage, migration, mongodb, atlas, cloudflare-workers]
|
||||
created: 2026-04-25
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
supersededBy: 260508-2222-go-port-cloud-run
|
||||
---
|
||||
|
||||
> **SUPERSEDED 2026-05-08** — superseded by [260508-2222-go-port-cloud-run](../260508-2222-go-port-cloud-run/plan.md). Direction changed: rather than swap CF KV+D1 for MongoDB on the same Cloudflare Worker, the bot is being rewritten in Go and deployed to Google Cloud Run with Firestore Native as the storage backend (all free-tier). All architectural concerns this plan addressed (cold-start, dual-write, abort criteria) are re-addressed in the Go-port plan against the new stack.
|
||||
|
||||
# Plan: KV+D1 → MongoDB Atlas M0
|
||||
|
||||
User-chosen path despite research recommending Upstash. Goal: validate cold-start UX firsthand with safe rollback to KV/D1 (or pivot to Upstash) if M0 cold-start P95 exceeds derived threshold.
|
||||
|
||||
## Reports
|
||||
- [Atlas fit + driver](../reports/researcher-260425-1924-mongodb-atlas-fit-and-driver.md)
|
||||
- [Schema + migration mechanics](../reports/researcher-260425-1924-mongodb-schema-and-migration.md)
|
||||
- [Free DB validation matrix](../reports/researcher-260425-1934-free-db-validation-matrix.md)
|
||||
- [Brainstormer critique](../reports/brainstormer-260425-2034-atlas-plan-critique.md)
|
||||
- [Code-reviewer correctness](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
|
||||
- [Debugger failure-modes](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
|
||||
|
||||
## Constraints (locked)
|
||||
- Backend: MongoDB Atlas M0 free, region `aws-ap-southeast-1`.
|
||||
- Driver: official `mongodb` npm v6.7+; `nodejs_compat_v2`; `compatibility_date >= 2025-03-20` (current `2025-10-01` already qualifies — no change needed).
|
||||
- Strategy: dual-write → backfill → verify → read-flip → soak → decommission.
|
||||
- Scope: BOTH `KV` (12 modules) and `D1` (`trading`) → MongoDB.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Owner files |
|
||||
|---|-------|--------|--------|-------------|
|
||||
| 01 | [Atlas setup + wrangler config](phase-01-atlas-setup.md) | pending | 2h | `wrangler.toml`, `.env.deploy.example`, `scripts/check-secret-leaks.js` |
|
||||
| 02 | [MongoKVStore implementation](phase-02-mongo-kv-store.md) | pending | 3h | `src/db/mongo-*.js` |
|
||||
| 03 | [MongoTradesStore + trading refactor](phase-03-mongo-sql-store.md) | pending | 3h | `src/db/mongo-trades-store.js`, `src/modules/trading/*` |
|
||||
| 04 | [Dual-write wrappers + flag + e2e](phase-04-dual-write-wrappers.md) | pending | 4h | `src/db/dual-*.js`, factories, `tests/e2e/*` |
|
||||
| 05 | [Backfill + verification (local-only)](phase-05-backfill-scripts.md) | pending | 3h | `scripts/backfill-*.js` |
|
||||
| 06 | [Staged deploy + soak (cold-start gate)](phase-06-staged-deploy-and-soak.md) | pending | 4h | runtime telemetry |
|
||||
| 07 | [Cutover + decommission](phase-07-cutover-and-decommission.md) | pending | 3h | wrangler bindings |
|
||||
| 07-ALT | [Pivot to Upstash (STANDBY)](phase-07-alt-pivot.md) | standby | (3-4d if triggered) | `src/db/upstash-*.js` |
|
||||
| 08 | [Tests + docs](phase-08-tests-and-docs.md) | pending | 1h | `tests/`, `docs/` |
|
||||
|
||||
## Critical dependencies
|
||||
- 01 → 02, 03 (Atlas creds + bundle-size gate required)
|
||||
- 02 + 03 → 04 (wrappers need both stores; e2e lands here)
|
||||
- 04 → 05 (dual-write must be live before backfill so concurrent writes hit Mongo too)
|
||||
- 05 → 06 (no soak without verified data)
|
||||
- 06 → 07 (cutover blocked on cold-start gate; if breached, **abort to [phase-07-alt-pivot.md](phase-07-alt-pivot.md)**)
|
||||
- 02–05 land throughout; 08 finalizes after 07
|
||||
|
||||
## Abort criteria (must trigger pivot, not retry)
|
||||
- Phase 06: cold-start P95 for `/wordle` or `/loldle` > `2.5 × P95(cold-ping baseline from Phase 01)` over 24h soak window.
|
||||
- Phase 06: dual-write divergence rate > 1% sustained for >1h.
|
||||
- Phase 06: M0 connection saturation events (>400 of 500 cap) observed during burst.
|
||||
- Phase 06: Worker CPU-time exceeded errors observed on cold start (Free plan 50ms ceiling).
|
||||
- Pivot path: leave dual-write running → revert `STORAGE_PRIMARY=kv` → execute [phase-07-alt-pivot.md](phase-07-alt-pivot.md).
|
||||
|
||||
## Rollback per phase
|
||||
Every phase file lists explicit rollback. Headline: until Phase 07 deletes KV/D1 bindings, the original data path is one env-flag flip away. Phase 07 is the only irreversible cutover step.
|
||||
|
||||
## Alternatives considered (reviewer dissent)
|
||||
|
||||
Reviewers (see brainstormer + code-reviewer + debugger reports) recommended:
|
||||
1. **Skip dual-write** in favor of a 10-minute maintenance window (saves ~6h, removes latency amplification).
|
||||
2. **Defer trading migration** entirely (D1 free tier handles ~100 writes/day indefinitely).
|
||||
3. **Single shared `kv` collection** instead of 12 per-module collections.
|
||||
4. **Pivot to Upstash** before starting (smaller bundle, HTTP-native, no cold-start TLS cost).
|
||||
5. **Direct `MongoTradesStore`** instead of SQL-pattern dispatcher (applied — see Phase 03).
|
||||
|
||||
User chose to proceed with full Atlas migration to validate cold-start UX firsthand. Reviewer correctness/safety findings have been applied throughout phases 01–08; architectural recommendations 1–4 are documented but not adopted. If the cold-start gate trips, [phase-07-alt-pivot.md](phase-07-alt-pivot.md) executes recommendation 4.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Critical blockers"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "30min"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 1: Critical blockers
|
||||
|
||||
## Overview
|
||||
Two cross-phase blockers that prevent the next merge from working: Go-version mismatch breaks `docker build`; three nil-deref sites match a pattern Phase 5a fixed but never propagated. Plus a CI gap that let issue #1 ship silently.
|
||||
|
||||
## Requirements
|
||||
- Functional: `docker build` succeeds locally and in CI; misc/help handlers tolerate `update.Message == nil`.
|
||||
- Non-functional: future Go-version drift surfaces in CI, not at deploy time.
|
||||
|
||||
## Architecture
|
||||
Three independent fixes; no design choices needed beyond version-bump direction.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `Dockerfile` — bump builder image
|
||||
- Modify: `.github/workflows/ci.yml` — bump `go-version`, add `docker build` step
|
||||
- Modify: `go.mod` (alternative path: lower `go` directive)
|
||||
- Modify: `internal/modules/misc/misc.go` — guard at lines 54, 79, 94
|
||||
- Modify: `internal/modules/util/help.go` — guard at line 100
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### 1. Pick Go-version direction
|
||||
Two options, equivalent outcome:
|
||||
- **(a) Bump up** — `Dockerfile:1` → `golang:1.25-alpine`; `ci.yml:19` `go-version: '1.25'`. Match local toolchain.
|
||||
- **(b) Lower go.mod** — `go.mod:3` → `go 1.23.0`. Codebase uses no 1.24/1.25 features; cheapest fix.
|
||||
|
||||
Recommended: **(b)** unless team standard is 1.25.
|
||||
|
||||
### 2. Add `docker build` to CI
|
||||
After `go build` step in `.github/workflows/ci.yml`:
|
||||
```yaml
|
||||
- name: Docker build
|
||||
run: docker build -t miti99bot-go .
|
||||
```
|
||||
~3 lines. Catches future Dockerfile/go.mod drift.
|
||||
|
||||
### 3. Add nil-message guards
|
||||
Match the pattern at `internal/modules/util/info.go:36`:
|
||||
```go
|
||||
if update.Message == nil {
|
||||
return nil
|
||||
}
|
||||
```
|
||||
Apply to:
|
||||
- `misc.go:54` (pingCommand handler)
|
||||
- `misc.go:79` (mstatsCommand handler)
|
||||
- `misc.go:94` (fortytwoCommand handler)
|
||||
- `util/help.go:100` (helpCommand handler)
|
||||
|
||||
## Success Criteria
|
||||
- [x] `go.mod`, `Dockerfile`, `ci.yml` agree on Go version (all 1.25; bumped Dockerfile + CI up to match go.mod)
|
||||
- [x] `docker build -t miti99bot-go .` succeeds locally
|
||||
- [x] CI workflow includes `docker build` step
|
||||
- [x] Four handlers have `update.Message == nil` guard at top (misc ×3, util/help ×1)
|
||||
- [x] `go vet ./...` and `go test -race -count=1 ./...` clean
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk:** Bumping go.mod down hides a feature use we missed → `go build` catches at compile time.
|
||||
- **Risk:** New CI step adds ~30s build time → acceptable; container build is what production uses.
|
||||
- **Mitigation:** Test locally before push; CI matrix runs on every PR.
|
||||
@@ -1,127 +0,0 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "High-priority hardening"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "2-3h"
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 2: High-priority hardening
|
||||
|
||||
## Overview
|
||||
Pre-public-launch security/reliability fixes: env allowlist (H1), panic recovery (M9), visibility enforcement (M2), cron timeout reduction (M4), 413/400 header bug (M3), emoji HTML escape (M7). These four items are the gate before exposing the bot publicly.
|
||||
|
||||
## Requirements
|
||||
- Functional: protected commands gated by admin check; panicking handler does not trigger Telegram retry storm; future API keys do not auto-leak to all modules.
|
||||
- Non-functional: defense-in-depth at trust boundaries.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Env allowlist (H1)
|
||||
Replace `secretEnvKeys` denylist with explicit allowlist, opt-in per module via `MODULE_<NAME>_*` convention. Module declares required keys; `Build` filters env to only declared keys.
|
||||
|
||||
### Visibility enforcement (M2)
|
||||
Two-tier dispatcher gate:
|
||||
- `VisibilityProtected` → require `update.Message.From.ID` ∈ `ADMIN_USER_IDS` env (comma-separated).
|
||||
- `VisibilityPrivate` → bot-owner-only (single ID).
|
||||
- `VisibilityPublic` → unchanged.
|
||||
|
||||
Cheaper than per-chat admin lookup; defer Telegram `getChatMember` call to a future iteration if needed.
|
||||
|
||||
### Panic recovery (M9)
|
||||
Wrap `b.ProcessUpdate` in `defer recover()` inside `webhook.go` handler. Log panic, return 200, prevent retry storm.
|
||||
|
||||
### Cron timeout (M4)
|
||||
Lower `defaultCronTimeout` from 5m to 60s. Document long-running cron pattern (publish to PubSub, exit fast).
|
||||
|
||||
### Header-shadow fix (M3)
|
||||
Detect `MaxBytesError` separately from generic decode errors; do not call `http.Error` after MaxBytesReader has already written 413.
|
||||
|
||||
### Emoji HTML escape (M7)
|
||||
`html.EscapeString(emojis)` at `loldleemoji/render.go:20`.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `cmd/server/main.go` — replace `secretEnvKeys` with allowlist resolver
|
||||
- Modify: `internal/modules/module.go` — add `RequiredEnv []string` field on `Module`
|
||||
- Modify: `internal/modules/registry.go` — filter env per module
|
||||
- Modify: `internal/modules/dispatcher.go` — visibility gate
|
||||
- Modify: `internal/telegram/webhook.go` — panic recovery + MaxBytesError handling
|
||||
- Modify: `internal/server/timeouts.go` — cron timeout 5m → 60s
|
||||
- Modify: `internal/modules/loldleemoji/render.go` — html.EscapeString
|
||||
- Modify: `internal/modules/loldle/loldle.go`, `internal/modules/loldleemoji/loldleemoji.go` — declare protected commands need admin
|
||||
- Test: `webhook_test.go`, `dispatcher_test.go`, `registry_test.go`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### 1. Env allowlist
|
||||
1. Add `RequiredEnv []string` to `Module`.
|
||||
2. In `registry.Build`, build `Deps.Env` from `intersect(os env, mod.RequiredEnv)`.
|
||||
3. Delete `secretEnvKeys` (no longer needed; nothing leaks by default).
|
||||
4. Update misc/util/loldle/loldleemoji modules — none currently need env, declare empty.
|
||||
5. Tests: assert unrelated env var does not appear in `Deps.Env`.
|
||||
|
||||
### 2. Visibility enforcement
|
||||
1. Add `ADMIN_USER_IDS` env parsing in `loadConfig` → `[]int64`.
|
||||
2. Add `BOT_OWNER_ID` env parsing → `int64`.
|
||||
3. In `dispatcher.Install`, before invoking handler, check `cmd.Visibility`:
|
||||
- `Private`: require `update.Message.From.ID == BOT_OWNER_ID`
|
||||
- `Protected`: require `update.Message.From.ID ∈ ADMIN_USER_IDS`
|
||||
- Else: proceed
|
||||
4. Reject denied calls silently (no reply — avoid leak that protected command exists).
|
||||
5. Tests: protected command from non-admin returns no-op; from admin proceeds.
|
||||
|
||||
### 3. Panic recovery in webhook
|
||||
At `internal/telegram/webhook.go:58`:
|
||||
```go
|
||||
func() {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
log.Printf("webhook handler panic: %v", r)
|
||||
}
|
||||
}()
|
||||
b.ProcessUpdate(ctx, &update)
|
||||
}()
|
||||
```
|
||||
Test: register a handler that panics; assert webhook returns 200 and no goroutine leaks.
|
||||
|
||||
### 4. Lower cron timeout
|
||||
`internal/server/timeouts.go:8`: `defaultCronTimeout = 60 * time.Second`. Update doc comment.
|
||||
|
||||
### 5. Header-shadow fix
|
||||
At `internal/telegram/webhook.go:49-54`, check `errors.As(err, &maxBytesErr)`:
|
||||
```go
|
||||
var maxBytesErr *http.MaxBytesError
|
||||
if errors.As(err, &maxBytesErr) {
|
||||
// 413 already written by MaxBytesReader
|
||||
return
|
||||
}
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
```
|
||||
Update `TestWebhookHandler_RejectsOversizedBody` to assert exact 413 status.
|
||||
|
||||
### 6. Emoji escape
|
||||
`render.go:20`: `clue := "🎭 " + html.EscapeString(emojis)`.
|
||||
|
||||
## Success Criteria
|
||||
- [x] Env allowlist: `Deps.Env` is empty by default; denylist + `envForModules` deleted. Phase 07 will add per-module allowlist plumbing.
|
||||
- [x] Future `GEMINI_API_KEY` cannot auto-leak (no env flows by default)
|
||||
- [x] Non-admin caller of Protected/Private commands silently denied via `Auth.Permits` in dispatcher
|
||||
- [x] Handler that panics → 200 to Telegram, stack logged via `runtime/debug.Stack()` (test: `TestWebhookHandler_RecoversPanicAndReturns200`)
|
||||
- [x] Cron handler timeout = 60s (`internal/server/timeouts.go:8`)
|
||||
- [x] Oversized webhook body returns clean 413 (`*http.MaxBytesError` branch, test rewritten with valid-prefixed JSON)
|
||||
- [x] Emoji string `html.EscapeString` in `loldleemoji/render.go:28`
|
||||
- [x] All existing tests pass; `Auth.Permits` table-driven test added; panic-recovery test added
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk:** Visibility gate breaks dev workflow if `ADMIN_USER_IDS` unset → default to "deny all protected/private when env unset" with a startup warning. Bot owner must set env explicitly.
|
||||
- **Risk:** Panic recovery hides bugs → still log full stack trace via `runtime/debug.Stack()` so Cloud Logging captures it.
|
||||
- **Risk:** 60s cron timeout is too aggressive for a future heavy cron → document escape hatch via `Cron.Timeout` override field.
|
||||
|
||||
## Security Considerations
|
||||
- Visibility gate uses constant-time comparison? Not needed — IDs are small ints, equality check is fine.
|
||||
- Panic recovery must NOT echo internal error to user — only log server-side.
|
||||
- Env allowlist prevents future leak class entirely.
|
||||
|
||||
## Next Steps
|
||||
Phase 03 (helper extraction) can run in parallel after Phase 02 lands; they touch different files.
|
||||
@@ -1,77 +0,0 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Shared helper extraction"
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: "1-2h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 3: Shared helper extraction
|
||||
|
||||
## Overview
|
||||
Eliminate helper drift across `wordle`, `loldle`, `loldleemoji`, `misc`. The 6a review flagged this for 6b prep; the architecture review confirmed `subjectFor` variants already differ in shape, and `winRate` truncation drift bit Phase 5b/5c. Extract before the next module port lands and compounds the problem.
|
||||
|
||||
## Requirements
|
||||
- Functional: zero behavior change; helpers must be byte-equivalent at call sites.
|
||||
- Non-functional: single source for chat-helper + champion-name primitives; future modules import rather than copy.
|
||||
|
||||
## Architecture
|
||||
|
||||
Two new packages:
|
||||
|
||||
### `internal/modules/util/chathelper`
|
||||
Generic helpers usable by any module:
|
||||
- `SubjectFor(msg *models.Message) string` — single canonical impl (private/group fallback)
|
||||
- `ArgAfterCommand(text string) string`
|
||||
- `NowMillis() int64`
|
||||
- `Reply(ctx, b, msg, text) error`
|
||||
- `ReplyHTML(ctx, b, msg, text) error`
|
||||
- `WinRate(wins, played int) int` — `math.Round` correctly
|
||||
|
||||
### `internal/champname`
|
||||
Loldle-specific:
|
||||
- `Normalize(s string) string`
|
||||
- `FindChampion[T any](needle string, all []T, name func(T) string) (T, bool)` — generic over champion type
|
||||
|
||||
(Or keep as helpers in `internal/modules/util/chathelper` — see unresolved Q2 from arch report.)
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/modules/util/chathelper/chathelper.go`
|
||||
- Create: `internal/modules/util/chathelper/chathelper_test.go`
|
||||
- Create: `internal/champname/champname.go` (or fold into chathelper)
|
||||
- Create: `internal/champname/champname_test.go`
|
||||
- Modify: `internal/modules/wordle/handlers.go` — delete local helpers, import
|
||||
- Modify: `internal/modules/loldle/handlers.go` — same
|
||||
- Modify: `internal/modules/loldleemoji/handlers.go` — same
|
||||
- Modify: `internal/modules/misc/misc.go` — same (uses `nowMillis`)
|
||||
- Modify: `internal/modules/loldle/lookup.go` — delete local `findChampion`/`normalize`
|
||||
- Modify: `internal/modules/loldleemoji/lookup.go` — same
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Decide canonical `SubjectFor`** — pick the loldle/emoji shape (no `ChatTypePrivate` special-case — `default` branch already handles it). Document in a comment.
|
||||
2. **Write chathelper package** with all 6 helpers + table-driven tests.
|
||||
3. **Migrate wordle** — replace 4 local helpers with imports; run tests; assert no behavior change.
|
||||
4. **Migrate loldle** — same.
|
||||
5. **Migrate loldleemoji** — same.
|
||||
6. **Migrate misc** — only `nowMillis`.
|
||||
7. **Write champname package** with `Normalize` + generic `FindChampion`. Tests cover prefix-match, ambiguous-prefix, exact-match, accent-insensitive.
|
||||
8. **Migrate loldle/loldleemoji** lookup paths.
|
||||
9. **Run full test suite + race detector.**
|
||||
|
||||
## Success Criteria
|
||||
- [x] Single `SubjectFor` impl; zero copies in modules (`internal/modules/util/chathelper`)
|
||||
- [x] Single `Normalize` + `Find` (generic) impl (`internal/champname`)
|
||||
- [x] All wire-format tests still pass (no behavior drift)
|
||||
- [x] `go test -race -count=1 ./...` clean
|
||||
- [x] Net LOC reduction across handler files: ~290 net lines removed (589 deletions vs 299 insertions across all files; loldle/loldleemoji/wordle handlers each ~50–60 lines slimmer)
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk:** Generic `FindChampion[T]` may not compile cleanly with current Go version → fallback to interface + type assertion or per-module thin wrapper.
|
||||
- **Risk:** Subtle `SubjectFor` divergence (private channel with no From) → covered by table-driven tests with all chat types.
|
||||
- **Mitigation:** Migrate one module at a time, run tests between each.
|
||||
|
||||
## Next Steps
|
||||
- Phase 06 file-size splits become mechanical after this lands.
|
||||
- Phase 07+ AI modules import these helpers instead of copying.
|
||||
@@ -1,88 +0,0 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Structured logging"
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: "2-3h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 4: Structured logging
|
||||
|
||||
## Overview
|
||||
Forward-port Phase 11's "Cloud Logging structured JSON" from the port plan. Cloud Run treats `stdout` lines as records but only parses JSON for severity/labels/trace correlation. Every `log.Printf` site added before this lands is a future migration. Also closes log-injection class (J3) by making newlines safe-by-construction.
|
||||
|
||||
## Requirements
|
||||
- Functional: same log content emitted, JSON-encoded.
|
||||
- Non-functional: severity levels, structured fields, trace ID propagation hooks.
|
||||
|
||||
## Architecture
|
||||
|
||||
New package `internal/log` (or `internal/obs`):
|
||||
```go
|
||||
package log
|
||||
|
||||
import "log/slog"
|
||||
|
||||
var defaultLogger = slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
|
||||
Level: slog.LevelInfo,
|
||||
}))
|
||||
|
||||
func Info(msg string, args ...any) { defaultLogger.Info(msg, args...) }
|
||||
func Warn(msg string, args ...any) { defaultLogger.Warn(msg, args...) }
|
||||
func Error(msg string, args ...any) { defaultLogger.Error(msg, args...) }
|
||||
func Fatal(msg string, args ...any) { defaultLogger.Error(msg, args...); os.Exit(1) }
|
||||
|
||||
func With(args ...any) *slog.Logger { return defaultLogger.With(args...) }
|
||||
```
|
||||
|
||||
Slog's JSONHandler is stdlib (Go 1.21+), zero deps. Cloud Logging auto-recognizes `severity`, `time`, `message` keys.
|
||||
|
||||
### 18 call sites to rewire
|
||||
- `cmd/server/main.go` (×9) — startup messages
|
||||
- `internal/server/router.go:77, 86` — cron logging
|
||||
- `internal/modules/dispatcher.go:24` — handler error
|
||||
- `internal/modules/misc/misc.go:51` — KV write failure
|
||||
|
||||
Mechanical translation:
|
||||
```go
|
||||
log.Printf("misc /ping: putJSON failed: %v", err)
|
||||
// becomes
|
||||
log.Error("misc ping putJSON failed", "module", "misc", "command", "ping", "err", err)
|
||||
```
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/log/log.go` (~40 LOC)
|
||||
- Create: `internal/log/log_test.go`
|
||||
- Modify: every file with `log.Printf` (18 sites)
|
||||
- Modify: `internal/telegram/webhook.go` — panic recovery (Phase 02) uses new logger
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Create `internal/log` package** with stdlib slog.JSONHandler.
|
||||
2. **Add log level env** — `LOG_LEVEL=info|debug|warn|error` (default info).
|
||||
3. **Write tests** — capture output, assert JSON shape with `severity`, `time`, custom fields.
|
||||
4. **Migrate `cmd/server/main.go`** — 9 sites. `log.Fatalf` → `log.Fatal`.
|
||||
5. **Migrate `internal/server/router.go`** — 2 sites. Add structured fields (`route=/cron`, `name=$name`).
|
||||
6. **Migrate `internal/modules/dispatcher.go`** — 1 site.
|
||||
7. **Migrate `internal/modules/misc/misc.go`** — 1 site.
|
||||
8. **Migrate `internal/telegram/webhook.go`** — panic recovery from Phase 02.
|
||||
9. **Search-grep `log.Printf` and `log.Fatalf`** — confirm zero remaining.
|
||||
10. **Smoke test locally** — run server, hit endpoint, verify Cloud-Logging-friendly JSON in stdout.
|
||||
|
||||
## Success Criteria
|
||||
- [x] Zero stdlib `"log"` imports outside `internal/log` (verified via grep)
|
||||
- [x] All log lines are valid JSON via `slog.JSONHandler` writing to stdout
|
||||
- [x] Each line has `level`, `time`, `msg`, plus structured fields
|
||||
- [x] Cron + dispatcher error logs no longer have CRLF-injection risk (J3) — newlines are escaped in field values (test: `TestNewlineEscaping_NoLogInjection`)
|
||||
- [x] `LOG_LEVEL=debug|info|warn|error` honoured at startup
|
||||
- [x] `go test -race -count=1 ./...` clean across all 13 packages
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk:** Newline handling — slog escapes newlines in field values, so error wrapping `%v` of a newline-bearing error becomes safe automatically.
|
||||
- **Risk:** Test output noise — tests can use `slog.NewTextHandler(io.Discard, ...)` injected via init or env flag.
|
||||
- **Risk:** Performance regression — slog is ~2× slower than `log.Printf` per call but well under 1µs; negligible for webhook latency.
|
||||
|
||||
## Next Steps
|
||||
- Phase 6b/7+ modules use `log` package from day one — no migration debt.
|
||||
- Once Cloud Logging structured queries work, build error-rate dashboard (Phase 11 telemetry concern).
|
||||
@@ -1,112 +0,0 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Test coverage gaps"
|
||||
status: completed
|
||||
priority: P2
|
||||
effort: "6-8h"
|
||||
dependencies: [3]
|
||||
---
|
||||
|
||||
# Phase 5: Test coverage gaps
|
||||
|
||||
## Overview
|
||||
Coverage at 44.7% with handler-layer at 0% in 5 modules and Firestore ops skipped on CI. Implement handler integration tests + Firestore emulator in CI to reach ≥60% coverage and gain confidence in the dispatch path that currently has no test exercising it end-to-end.
|
||||
|
||||
## Requirements
|
||||
- Functional: every handler reachable via `bot.ProcessUpdate` exercised in tests with realistic `*models.Update` fixtures.
|
||||
- Non-functional: tests run in <30s on CI; emulator setup adds <60s startup; no flakes.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Handler test pattern
|
||||
- New `testutil/update.go` package: builders for `NewPrivateMessage(userID, text)`, `NewGroupMessage(chatID, userID, text)`, `NewChannelMessage(chatID, text)`.
|
||||
- Bot mock: capture sent messages via a `recordingBot` that stores `SendMessageParams` instead of calling Telegram. (`*bot.Bot` has unexported fields — alternative: spin httptest server that replies to `sendMessage` API and use `bot.WithServerURL`.)
|
||||
- Per-module `handlers_test.go` exercises each handler with a real in-memory KV provider + recording bot.
|
||||
|
||||
### Firestore emulator on CI
|
||||
Add GitHub Actions service or Docker step:
|
||||
```yaml
|
||||
- name: Start Firestore emulator
|
||||
run: |
|
||||
gcloud --quiet components install beta cloud-firestore-emulator
|
||||
gcloud beta emulators firestore start --host-port=localhost:8080 &
|
||||
until nc -z localhost 8080; do sleep 1; done
|
||||
- name: Run tests
|
||||
env:
|
||||
FIRESTORE_EMULATOR_HOST: localhost:8080
|
||||
GOOGLE_CLOUD_PROJECT: test-project
|
||||
run: go test -race ./internal/storage/...
|
||||
```
|
||||
|
||||
Or use `firestore-emulator` Docker image with service container.
|
||||
|
||||
## Related Code Files
|
||||
- Create: `internal/testutil/update.go` — Update fixture builders
|
||||
- Create: `internal/testutil/recordbot.go` — recording bot helper (httptest-based)
|
||||
- Create: `internal/modules/wordle/handlers_test.go`
|
||||
- Create: `internal/modules/loldle/handlers_test.go`
|
||||
- Create: `internal/modules/loldleemoji/handlers_test.go`
|
||||
- Create: `internal/modules/util/handlers_test.go` (info/help/stickerid)
|
||||
- Create: `internal/modules/misc/handlers_test.go`
|
||||
- Modify: `.github/workflows/ci.yml` — emulator setup + env
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Build `internal/testutil`**
|
||||
- `NewPrivateMessage`, `NewGroupMessage`, `NewChannelMessage` builders.
|
||||
- `NewRecordingBot()` returns `*bot.Bot` wired to httptest server that records `SendMessage`/`SendSticker` requests; expose `Sent() []SendMessageParams`.
|
||||
- Tests for the test util itself.
|
||||
|
||||
2. **Wordle handler tests** (~25% coverage gain)
|
||||
- `TestHandleWordle_Win` — guess equals target → win path, sticker, stats.
|
||||
- `TestHandleWordle_Loss` — exhaust max guesses → loss path.
|
||||
- `TestHandleWordle_InvalidWord` — non-dictionary word → reject.
|
||||
- `TestHandleNew` — abandon active round, autoGiveup recorded.
|
||||
- `TestHandleGiveup` — reveal, idempotency on finished.
|
||||
- `TestHandleStats` — win rate calc with wins/losses.
|
||||
- `TestHandleWordle_NilMessage` — nil-guard path.
|
||||
|
||||
3. **Loldle handler tests** (~10% gain) — same pattern.
|
||||
|
||||
4. **Loldleemoji handler tests** (~20% gain) — same pattern.
|
||||
|
||||
5. **Util handler tests** (~15% gain)
|
||||
- `TestInfoCommand_*` — chat-id/sender-id echo.
|
||||
- `TestHelpCommand_*` — registry render.
|
||||
- `TestStickerIDCommand_*` — sticker echo, no-sticker case.
|
||||
|
||||
6. **Misc handler tests** (~10% gain)
|
||||
- `TestPingCommand_*` — KV write best-effort, reply.
|
||||
- `TestMstatsCommand_*` — GetJSON missing → fresh state, formatting.
|
||||
- `TestFortytwoCommand_*` — easter egg reply.
|
||||
|
||||
7. **Firestore emulator on CI**
|
||||
- Add gcloud emulator service to GitHub Actions workflow.
|
||||
- Set `FIRESTORE_EMULATOR_HOST` for storage package tests.
|
||||
- Verify all 5 currently-skipped tests run on CI.
|
||||
|
||||
8. **Coverage gate**
|
||||
- Add `-coverprofile=cov.out` to CI test command.
|
||||
- Optional: gate at ≥60% (start with warn, escalate to fail when stable).
|
||||
|
||||
## Success Criteria
|
||||
- [x] Coverage 69.8% (target ≥60% reached, +25% absolute from 44.7% baseline)
|
||||
- [x] Every handler has happy-path + error-path tests (misc 4, util 7, wordle 10, loldle 9, loldleemoji 8)
|
||||
- [x] CI now starts gcloud Firestore emulator on `localhost:8090`; storage tests run with `FIRESTORE_EMULATOR_HOST` set instead of `t.Skip`
|
||||
- [x] `go test -race -count=1 ./...` clean (15 packages)
|
||||
- [x] No flaky tests observed locally
|
||||
- [x] Coverage summary added to CI output
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk:** Recording bot via httptest is brittle if `go-telegram/bot` changes serialization → pin bot library version; add integration smoke test.
|
||||
- **Risk:** Firestore emulator startup adds 30-60s to CI → acceptable; this is industry-standard.
|
||||
- **Risk:** Tests over-mock and miss real bugs → use real in-memory KVStore (already standard); recording bot only stubs the network.
|
||||
- **Risk:** Handler tests duplicate state-layer tests → keep handler tests focused on dispatch + reply text + side effects, not game logic.
|
||||
|
||||
## Security Considerations
|
||||
- Test fixtures use synthetic IDs (no real Telegram user IDs).
|
||||
- Emulator runs in-CI, not exposed externally.
|
||||
|
||||
## Next Steps
|
||||
- Coverage trend tracked in CI; future modules require ≥60% to merge.
|
||||
- Phase 06 cleanup (file splits) easier with comprehensive tests.
|
||||
@@ -1,135 +0,0 @@
|
||||
---
|
||||
phase: 6
|
||||
title: "Cleanup and tooling"
|
||||
status: completed
|
||||
priority: P3
|
||||
effort: "2-3h"
|
||||
dependencies: [3]
|
||||
---
|
||||
|
||||
# Phase 6: Cleanup and tooling
|
||||
|
||||
## Overview
|
||||
Bundle remaining Medium/Low items from review reports: file-size splits, lint/vuln scanners, image-digest pinning, dead-code removal, hygiene fixes. None are individually urgent; bundled to land cleanly in one PR after Phase 03 mechanically simplifies the file-size work.
|
||||
|
||||
## Requirements
|
||||
- Functional: no behavior change.
|
||||
- Non-functional: stricter CI gates, smaller files, less surprise from supply-chain.
|
||||
|
||||
## Architecture
|
||||
Six independent fixes; pick whichever order is convenient.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `.github/workflows/ci.yml` — golangci-lint + govulncheck
|
||||
- Modify: `Dockerfile` — pin base images by digest
|
||||
- Modify: `internal/modules/loldle/handlers.go` — split per handler (post Phase 03)
|
||||
- Modify: `internal/modules/wordle/handlers.go` — same
|
||||
- Modify: `internal/modules/loldleemoji/handlers.go` — same
|
||||
- Modify: `internal/modules/loldle/compare.go` — split year/multi/exact
|
||||
- Modify: `internal/storage/firestore_kv.go` — extract validate/prefixSuccessor
|
||||
- Modify: `cmd/server/main.go` — extract config.go + provider.go
|
||||
- Modify: `internal/modules/registry.go` — Module.Name guard (M4)
|
||||
- Modify: `internal/storage/kv_provider.go` — `MemoryProvider.Base()` to test-tag (M7)
|
||||
- Modify: `internal/storage/firestore_provider.go` — validate moduleName in `For` (N2)
|
||||
- Delete: `internal/modules/wordle/state.go` constants (N3 — `gameTTLSeconds`)
|
||||
- Delete: `internal/modules/wordle/daily.go` if `pickDaily` unused after audit (N6)
|
||||
- Delete: `internal/modules/modules.go` (N7 — vestigial)
|
||||
- Create: `.golangci.yml` — config
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### 1. golangci-lint + govulncheck on CI
|
||||
Add `.golangci.yml`:
|
||||
```yaml
|
||||
linters:
|
||||
enable:
|
||||
- gofmt
|
||||
- errcheck
|
||||
- staticcheck
|
||||
- gosec
|
||||
- govet
|
||||
- ineffassign
|
||||
- unused
|
||||
```
|
||||
CI step:
|
||||
```yaml
|
||||
- uses: golangci/golangci-lint-action@v6
|
||||
with:
|
||||
version: latest
|
||||
- run: go install golang.org/x/vuln/cmd/govulncheck@latest && govulncheck ./...
|
||||
```
|
||||
Fix any findings (likely small — code is already clean).
|
||||
|
||||
### 2. Pin Docker base images by digest
|
||||
```dockerfile
|
||||
FROM golang:1.23-alpine@sha256:<actual-digest> AS builder
|
||||
...
|
||||
FROM gcr.io/distroless/static:nonroot@sha256:<actual-digest>
|
||||
```
|
||||
Use `docker pull` + `docker inspect` or `crane digest` to fetch digests. Document refresh procedure in `Dockerfile` comment.
|
||||
|
||||
### 3. File-size splits (post Phase 03)
|
||||
After Phase 03 helper extraction, expected residual files >200 LOC:
|
||||
- `loldle/handlers.go` → split into `handle_loldle.go`, `handle_giveup.go`, `handle_stats.go`, `handle_setmax.go`
|
||||
- `wordle/handlers.go` → same shape
|
||||
- `loldleemoji/handlers.go` → same
|
||||
- `loldle/compare.go` → split by attr type (`compare_year.go`, `compare_multi.go`)
|
||||
- `firestore_kv.go` → extract `firestore_keys.go` (validate + prefixSuccessor)
|
||||
- `cmd/server/main.go` → extract `config.go` (loadConfig + envForModules + splitCSV) + `provider.go` (buildProvider)
|
||||
|
||||
Verify each split with `wc -l` and tests.
|
||||
|
||||
### 4. Module.Name guard
|
||||
At `registry.go:119`:
|
||||
```go
|
||||
if mod.Name != "" && mod.Name != name {
|
||||
return nil, fmt.Errorf("module factory for %q returned mismatched Name=%q", name, mod.Name)
|
||||
}
|
||||
mod.Name = name
|
||||
```
|
||||
Test: factory that returns wrong Name → Build fails.
|
||||
|
||||
### 5. MemoryProvider.Base test-tag
|
||||
Move `Base()` method to `kv_provider_test.go` with `//go:build testtag`-style guard, or to a `storagetest` helper package. Update test imports.
|
||||
|
||||
### 6. FirestoreProvider validate
|
||||
At `firestore_provider.go:22 For`, call `validateCollection(moduleName)` even though upstream validates — defense in depth. Add test.
|
||||
|
||||
### 7. Dead-code removal
|
||||
- Delete `gameTTLSeconds` constant.
|
||||
- If `pickDaily` is genuinely unused, delete it + its test.
|
||||
- Delete `internal/modules/modules.go` (move package doc into `module.go`).
|
||||
|
||||
### 8. Bonus: PORT validation (L2)
|
||||
At `loadConfig`, validate `PORT` is numeric:
|
||||
```go
|
||||
if _, err := strconv.Atoi(port); err != nil {
|
||||
return nil, fmt.Errorf("invalid PORT %q: %w", port, err)
|
||||
}
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
- [x] `golangci-lint run` passes (0 issues; config tuned for the codebase style)
|
||||
- [x] `govulncheck ./...` runs on CI as informational; `golang.org/x/net` bumped v0.52.0 → v0.54.0 to resolve GO-2026-4918
|
||||
- [ ] Docker base images pinned by digest — **deferred** (Dependabot handles in practice)
|
||||
- [ ] Source-file size splits — **deferred** (largest file 279 LOC; the 200 LOC ceiling is a guideline, not a hard limit)
|
||||
- [x] `Module.Name` mismatch surfaces as error (`TestBuild_RejectsFactoryNameMismatch`)
|
||||
- [x] `FirestoreProvider.For` re-validates module name — invalid names return an `invalidStore` whose ops error with `ErrInvalidModuleName`
|
||||
- [x] `PORT` env validated numerically + 0..65535
|
||||
- [x] Dead code removed: `gameTTLSeconds` const, `pickDaily`/`hashDJB2`/`todayUTC` helpers + tests, `daily.go` renamed to `pick_random.go`
|
||||
- [x] `go test -race -count=1 ./...` clean across all 15 packages
|
||||
- [x] CI lint job + govulncheck job added
|
||||
|
||||
## Risk Assessment
|
||||
- **Risk:** golangci-lint surfaces 50+ findings → fix ones blocking, defer rest with `//nolint:` and a TODO comment.
|
||||
- **Risk:** Digest pinning causes CI failure on next base-image update → document refresh policy (monthly via dependabot or manual).
|
||||
- **Risk:** File splits introduce import cycles → unlikely (handlers are leaves), but verify with `go vet`.
|
||||
|
||||
## Security Considerations
|
||||
- Digest pinning hardens supply chain.
|
||||
- gosec finds common Go security mistakes (e.g., G104 unhandled errors, G304 path traversal).
|
||||
- govulncheck flags dependency CVEs.
|
||||
|
||||
## Next Steps
|
||||
- Plan complete; merge sequence: 01 → 02 → 03 → 04 → 05 → 06.
|
||||
- Update `260508-2222-go-port-cloud-run/plan.md` Phase 11 to mark structured-logging done (Phase 04 here forward-ports it).
|
||||
@@ -1,51 +0,0 @@
|
||||
---
|
||||
title: "Fix all review findings (architecture + security + tests)"
|
||||
description: "Remediation plan covering Critical/High/Major/Medium findings from the 2026-05-09 whole-project review (architecture, security, test-coverage)."
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: 1.5-2d
|
||||
branch: main
|
||||
tags: [fixes, review, hardening, tests, ci]
|
||||
created: 2026-05-09
|
||||
completed: 2026-05-09
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
---
|
||||
|
||||
# Plan: Fix all review findings
|
||||
|
||||
Six phases ordered by risk-gate. Phase 1 must land before next merge (Dockerfile/go.mod mismatch breaks `docker build`). Phases 2–4 are pre-public-launch hardening. Phase 5 closes the handler-layer test gap. Phase 6 is cleanup.
|
||||
|
||||
## Source reports
|
||||
- [Architecture & code quality](../reports/code-reviewer-260509-1248-whole-project-architecture.md)
|
||||
- [Security audit](../reports/code-reviewer-260509-1248-whole-project-security.md)
|
||||
- [Test coverage audit](../reports/tester-260509-1249-whole-project-coverage.md)
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort | Key deliverable |
|
||||
|---|-------|--------|--------|-----------------|
|
||||
| 01 | [Critical blockers](phase-01-critical-blockers.md) | done | 30min | Go-version alignment + 4 nil-deref guards + CI docker-build step |
|
||||
| 02 | [High-priority hardening](phase-02-high-priority-hardening.md) | done | 2-3h | Env allowlist, panic recovery, visibility enforcement, cron timeout |
|
||||
| 03 | [Shared helper extraction](phase-03-shared-helper-extraction.md) | done | 1-2h | `internal/modules/util/chathelper` + `internal/champname` (DRY) |
|
||||
| 04 | [Structured logging](phase-04-structured-logging.md) | done | 2-3h | `internal/log` slog.JSONHandler + 22-site rewire (forward-port from Phase 11) |
|
||||
| 05 | [Test coverage gaps](phase-05-test-coverage-gaps.md) | done | 6-8h | Handler integration tests (wordle/misc/util/loldle/loldleemoji) + Firestore emulator on CI — coverage 44.7% → 69.8% |
|
||||
| 06 | [Cleanup and tooling](phase-06-cleanup-and-tooling.md) | done | 2-3h | golangci-lint + govulncheck on CI, defensive guards (Module.Name, PORT, FirestoreProvider validate), dead-code removal. Docker digest pinning + LOC splits deferred (low value). |
|
||||
|
||||
## Key dependencies
|
||||
- Phase 03 must precede next module port in `260508-2222-go-port-cloud-run` (Phase 6b/7) so future modules don't compound helper drift.
|
||||
- Phase 04 is forward-port of Phase 11 from the active port plan; landing earlier reduces migration cost on each new module.
|
||||
- Phase 06 file splits are mechanically cleaner after Phase 03 helper extraction.
|
||||
|
||||
## Out of scope
|
||||
- Phase 9 OIDC migration for `/cron/*` (tracked in port plan).
|
||||
- Async-with-detached-context webhook dispatch (M2 from architecture report — defer until cold-start telemetry exists).
|
||||
- `pickRandom` cryptographic upgrade (L5 — non-issue today).
|
||||
- Sticker `file_id` rotation procedure (L6 — operations doc, not code).
|
||||
|
||||
## Validation
|
||||
- `go vet ./...` clean
|
||||
- `go test -race -count=1 ./...` clean
|
||||
- `docker build -t miti99bot-go .` succeeds
|
||||
- New CI workflow steps green
|
||||
- Coverage rises from 44.7% → ≥60%
|
||||
-147
@@ -1,147 +0,0 @@
|
||||
# Phase 01 — Wire `telegram-setup` Into `deploy.yml`
|
||||
|
||||
**Status:** implemented (pending live verification on next push to main)
|
||||
**Priority:** P1 (next deploy needs it for full automation)
|
||||
**Estimate:** ~30 min implementation + 1 deploy cycle to verify
|
||||
|
||||
## Context links
|
||||
|
||||
- Parent plan: `../plan.md`
|
||||
- Existing GH workflow: `.github/workflows/deploy.yml`
|
||||
- Reference Makefile targets: `Makefile` lines 101-148 (`telegram-setup`, `telegram-webhook`, `telegram-commands`)
|
||||
- Commands payload: `aws/telegram-commands.json`
|
||||
- Deploy role IAM: `aws/README.md` § 4 (already has `AmazonSSMFullAccess`)
|
||||
- Cutover docs: `docs/deploy-aws.md` line 64, `docs/deploy-aws-free-tier-guide.md` line 270 (current manual steps)
|
||||
|
||||
## Overview
|
||||
|
||||
Add two steps to `deploy.yml` after the existing **Smoke test** step:
|
||||
|
||||
1. **Register Telegram webhook** — read `FunctionUrl` from CFN, read token + secret from SSM, `POST` `setWebhook`.
|
||||
2. **Register Telegram command menu** — read token from SSM, `POST` `setMyCommands` with `aws/telegram-commands.json`.
|
||||
|
||||
Mirror the existing inline-CLI pattern used by the Smoke step (no `make` indirection — Makefile uses `--profile admin` which is wrong for CI).
|
||||
|
||||
## Key insights
|
||||
|
||||
- Deploy role already has `AmazonSSMFullAccess` and `AWSCloudFormationFullAccess` → no IAM change.
|
||||
- `setWebhook` / `setMyCommands` are idempotent → safe to run every push.
|
||||
- CFN `Outputs.FunctionUrl` (template.yaml:208-210) → reused from smoke step.
|
||||
- SSM param paths fixed at `/miti99bot/prod/{telegram-bot-token,telegram-webhook-secret}` (per `aws/README.md` § 3, `samconfig.toml`).
|
||||
- Bot token must be **masked** before any shell echo (it is a credential in URL path).
|
||||
- `aws/telegram-commands.json` is committed → `--data-binary "@aws/telegram-commands.json"` works directly.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
1. After a green SAM deploy + smoke test, the workflow registers the webhook + commands.
|
||||
2. Webhook URL format: `${FunctionUrl%/}/webhook` (trim trailing slash; Function URLs sometimes include it).
|
||||
3. `allowed_updates` must equal `["message","callback_query"]` (matches `Makefile:120`).
|
||||
4. `secret_token` from SSM is sent in the `setWebhook` payload — bot validates this on every incoming update.
|
||||
5. Job fails (non-zero exit) if either Telegram call returns non-2xx **or** `"ok": false`.
|
||||
|
||||
### Non-functional
|
||||
- Token never appears in plaintext logs (`::add-mask::TOKEN` before use).
|
||||
- No new secrets in GitHub repo settings — everything still flows through SSM.
|
||||
- No new third-party action — only `aws` CLI + `curl` + `jq` (all preinstalled on `ubuntu-latest`).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
deploy.yml job: deploy (existing)
|
||||
├─ checkout (existing)
|
||||
├─ setup-go (existing)
|
||||
├─ setup-sam (existing)
|
||||
├─ configure-aws-creds (existing)
|
||||
├─ build-lambda (existing)
|
||||
├─ sam-deploy (existing)
|
||||
├─ smoke-test (existing) — reads FunctionUrl from CFN
|
||||
├─ register-webhook (NEW) — reads FunctionUrl + SSM token/secret, POST setWebhook
|
||||
└─ register-commands (NEW) — reads SSM token, POST setMyCommands w/ aws/telegram-commands.json
|
||||
```
|
||||
|
||||
The Function URL is fetched twice (smoke + register-webhook). Acceptable: CFN describe-stacks is fast and the steps stay independent / debuggable. Optimization (cache URL in a step output) is out of scope.
|
||||
|
||||
## Related code files
|
||||
|
||||
**Modify**
|
||||
- `.github/workflows/deploy.yml` — append two steps after **Smoke test**
|
||||
|
||||
**Read (no change)**
|
||||
- `Makefile` lines 101-148 — reference implementation
|
||||
- `aws/telegram-commands.json` — payload body
|
||||
|
||||
**Possibly update**
|
||||
- `docs/deploy-aws.md` line 64 — current "manual setWebhook" instructions become "automatic on push; manual command kept for emergencies"
|
||||
- `docs/deploy-aws-free-tier-guide.md` line 270 — same note
|
||||
|
||||
## Implementation steps
|
||||
|
||||
1. **Open** `.github/workflows/deploy.yml`. Locate the `- name: Smoke test (Function URL responds)` step (last step today).
|
||||
2. **Append step `Register Telegram webhook`** after smoke-test:
|
||||
- Reuse `STACK_NAME` env (already at job level).
|
||||
- `URL=$(aws cloudformation describe-stacks ... FunctionUrl ...)` — copy pattern from smoke step.
|
||||
- `TOKEN=$(aws ssm get-parameter --name /miti99bot/prod/telegram-bot-token --with-decryption --query Parameter.Value --output text)`
|
||||
- `echo "::add-mask::$TOKEN"` immediately after read.
|
||||
- `SECRET=$(aws ssm get-parameter --name /miti99bot/prod/telegram-webhook-secret --with-decryption --query Parameter.Value --output text)`
|
||||
- `echo "::add-mask::$SECRET"` immediately after read.
|
||||
- `WEBHOOK_URL="${URL%/}/webhook"`
|
||||
- `RESP=$(curl -fsS -X POST "https://api.telegram.org/bot${TOKEN}/setWebhook" -d "url=${WEBHOOK_URL}" -d "secret_token=${SECRET}" -d 'allowed_updates=["message","callback_query"]')`
|
||||
- `echo "$RESP" | jq -e '.ok == true' >/dev/null || { echo "setWebhook failed: $RESP"; exit 1; }`
|
||||
- `echo "$RESP" | jq '{ok, result}'`
|
||||
3. **Append step `Register Telegram command menu`**:
|
||||
- Read `TOKEN` from SSM (same path) and re-mask. (Step env doesn't persist across steps; re-read is fine — single SSM call is cheap.)
|
||||
- `RESP=$(curl -fsS -X POST "https://api.telegram.org/bot${TOKEN}/setMyCommands" -H 'Content-Type: application/json' --data-binary "@aws/telegram-commands.json")`
|
||||
- Same `jq -e '.ok == true'` validation + pretty-print.
|
||||
4. **Lint locally** with `actionlint` if available (or just YAML parse): `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/deploy.yml'))"`.
|
||||
5. **Update docs**:
|
||||
- `docs/deploy-aws.md` line 64: add note "as of <date>, push-to-main auto-runs `setWebhook` + `setMyCommands`; manual command below kept for break-glass".
|
||||
- Same in `docs/deploy-aws-free-tier-guide.md` line 270.
|
||||
6. **Commit** with conventional message: `ci(deploy): auto-register Telegram webhook + commands after SAM deploy`.
|
||||
|
||||
## Todo list
|
||||
|
||||
- [x] Read current `deploy.yml` end (smoke step) to confirm insertion point
|
||||
- [x] Append `Register Telegram webhook` step (with token mask + `jq -e` validation)
|
||||
- [x] Append `Register Telegram command menu` step (with token mask + `jq -e` validation)
|
||||
- [x] Validate YAML parse locally (`yaml.safe_load` → OK)
|
||||
- [x] Update `docs/deploy-aws.md` + `docs/deploy-aws-free-tier-guide.md` notes
|
||||
- [ ] Commit + push, observe first run in GH Actions UI
|
||||
- [ ] Verify `curl https://api.telegram.org/bot$TOKEN/getWebhookInfo` shows the Function URL post-deploy
|
||||
|
||||
## Success criteria
|
||||
|
||||
| Check | How to verify |
|
||||
|-------|---------------|
|
||||
| Step `Register Telegram webhook` shows green | GH Actions run UI |
|
||||
| Step `Register Telegram command menu` shows green | GH Actions run UI |
|
||||
| `setWebhook` response logged as `{ok:true, result:true, description:"Webhook was set"}` | step log |
|
||||
| Token / secret not visible in logs | search step output for first 4 chars of token → must show `***` |
|
||||
| `make telegram-webhook-info` from local shows `url == <FunctionUrl>/webhook` | local `make` after pipeline finishes |
|
||||
| `/help` works in Telegram after deploy | live bot smoke test |
|
||||
|
||||
## Risk assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|-----------|
|
||||
| Telegram API blip → CI fails despite healthy deploy | Low | Low (Lambda still serving) | `-fsS` + manual workflow re-run; document break-glass `make telegram-setup` |
|
||||
| SSM param missing on first-ever deploy | Low | High (deploy red) | Precondition documented in `aws/README.md` § 3 — params must exist before first push (already true today) |
|
||||
| Bot token printed in `set -x`-style verbose log | Medium | High (token leak) | `::add-mask::` immediately after SSM read; never use `set -x` in these steps |
|
||||
| `aws/telegram-commands.json` invalid JSON | Low | Low (single-step fail) | `--data-binary @file` + Telegram validates; `jq -e .ok` catches |
|
||||
|
||||
## Security considerations
|
||||
|
||||
- **Mask tokens / secrets**: `::add-mask::` after every SSM read. GH Actions then redacts that string from all subsequent log lines (including child commands).
|
||||
- **Path-credential leak**: `curl` URL contains `${TOKEN}` — masking covers it, but additionally avoid `set -x`, `-v`, or `echo "$URL"` in any debug temp.
|
||||
- **No new IAM**: deploy role keeps existing scope; no expansion of privileges.
|
||||
- **Webhook secret**: validates inbound requests at `internal/telegram/webhook.go:19-20` (`X-Telegram-Bot-Api-Secret-Token` header). Auto-registration enforces same secret across token rotations.
|
||||
|
||||
## Next steps
|
||||
|
||||
After this phase merges and one deploy cycle confirms green:
|
||||
- (Optional follow-up, separate plan) Replace `AmazonSSMFullAccess` with a scoped policy granting only `ssm:GetParameter` on `/miti99bot/*` — pairs with the broader IAM tightening already deferred in `aws/README.md` § 4.
|
||||
- (Optional) Cache `FunctionUrl` between smoke and register-webhook via `$GITHUB_OUTPUT` — micro-optimization only.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
- None. All decisions locked: always-on registration, inline CLI (no `make` from CI), `jq -e` failure semantics, both docs files updated.
|
||||
@@ -1,60 +0,0 @@
|
||||
# Auto-Register Telegram Webhook + Commands After Deploy
|
||||
|
||||
**Date:** 2026-05-16
|
||||
**Slug:** `260516-1035-auto-register-after-deploy`
|
||||
**Status:** Implemented (pending commit + live verification on next push to main)
|
||||
**Type:** CI/CD enhancement (single phase)
|
||||
**Mode:** fast (no research needed — referenced code paths already exist)
|
||||
|
||||
## Goal
|
||||
|
||||
Every push to `main` already runs `.github/workflows/deploy.yml` → SAM deploy → smoke-test the Function URL. After a successful deploy, register the Telegram webhook + command menu **automatically** (currently done manually via `make telegram-setup`).
|
||||
|
||||
## Why
|
||||
|
||||
- Eliminates manual `make telegram-setup` step after first deploy / handler-path change / webhook secret rotation.
|
||||
- Self-healing: if Telegram's `webhook_url` ever drifts from the Function URL (e.g. secret rotated but `setWebhook` forgotten), the next deploy fixes it.
|
||||
- `setWebhook` and `setMyCommands` are idempotent — running on every deploy is safe and cheap.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Do **not** introduce a new Go binary / Lambda hook / CloudFormation custom resource.
|
||||
- Do **not** rewrite the existing Makefile targets — keep `make telegram-setup` working for local/manual use.
|
||||
- Do **not** touch `aws/telegram-commands.json` content or module behavior.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | File | Status |
|
||||
|---|-------|------|--------|
|
||||
| 01 | Wire telegram-setup into deploy.yml | `phase-01-wire-telegram-setup-into-deploy.md` | implemented |
|
||||
|
||||
## Key files
|
||||
|
||||
- `.github/workflows/deploy.yml` — add post-smoke-test registration step
|
||||
- `Makefile` (lines 101-148) — reference impl (do not modify unless CI parity requires)
|
||||
- `aws/telegram-commands.json` — read by `setMyCommands` step
|
||||
- `aws/README.md` § 4 — deploy role already has `AmazonSSMFullAccess`, no IAM change needed
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Deploy role `github-deploy-miti99bot` already has `AmazonSSMFullAccess` + `AWSCloudFormationFullAccess` (verified in `aws/README.md` § 4).
|
||||
- SSM params `/miti99bot/prod/telegram-bot-token` and `/miti99bot/prod/telegram-webhook-secret` already populated (precondition of first deploy).
|
||||
|
||||
## Risks
|
||||
|
||||
- **Telegram API outage on deploy** → CI fails even though Lambda is healthy. Mitigation: use `curl -fsS` so non-2xx aborts the job; failure surface is loud, recoverable by re-running workflow.
|
||||
- **Token exposed in logs** → use `::add-mask::` for TOKEN before any echo / curl line; do not pass via `-d` URL arg (token is in path, but mask anyway).
|
||||
- **Webhook secret rotation race** → SSM read happens after deploy, so newest secret wins. No race in practice.
|
||||
|
||||
## Success criteria
|
||||
|
||||
After merging this change, the next push to `main`:
|
||||
1. SAM deploy succeeds.
|
||||
2. Smoke-test passes.
|
||||
3. New step: `curl /setWebhook` returns `{"ok":true,...}`.
|
||||
4. New step: `curl /setMyCommands` returns `{"ok":true,...}`.
|
||||
5. `getWebhookInfo` shows `url == <FunctionUrl>/webhook`.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
- None (locked decisions): always-on registration, inline CLI calls (not `make telegram-setup`), fail-loud on API errors.
|
||||
@@ -1,50 +0,0 @@
|
||||
# Fix Group Command Matching + Observable Dispatch Logging
|
||||
|
||||
**Date:** 2026-05-16
|
||||
**Slug:** `260516-1130-group-command-match-fix`
|
||||
**Status:** Implemented (awaiting deploy + group smoke test)
|
||||
**Mode:** fast (single phase, well-scoped)
|
||||
**Linked diagnostic:** `../reports/debugger-260516-1124-group-commands-not-matching.md`
|
||||
|
||||
## Goal
|
||||
|
||||
Bot commands sent in Telegram groups (`/help@miti99bot` form) must match registered handlers. Currently they silently miss because `github.com/go-telegram/bot@v1.20.0/handlers.go:88` does byte-exact equality without stripping the `@<botname>` suffix.
|
||||
|
||||
Add structured pre-dispatch logging so the next "silent drop" symptom is observable in CloudWatch without code archaeology.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | File |
|
||||
|---|-------|------|
|
||||
| 01 | Strip `@suffix` in dispatcher + add update log | `phase-01-strip-botname-suffix-and-log-dispatch.md` |
|
||||
|
||||
## Files
|
||||
|
||||
- `internal/modules/dispatcher.go` — swap `RegisterHandler(..., MatchTypeCommand, ...)` for `RegisterHandlerMatchFunc(...)` with a local matcher that strips `@suffix`
|
||||
- `internal/modules/dispatcher_test.go` — add test cases covering `/help` (DM) and `/help@miti99bot` (group), plus negatives
|
||||
- `internal/telegram/webhook.go` — add one structured log line just before `b.ProcessUpdate` recording `update_id`, `chat.id`, `chat.type`, `text` (truncated)
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Don't fork the upstream library.
|
||||
- Don't add a `TELEGRAM_BOT_USERNAME` env var — Telegram routes `/cmd@otherbot` to the addressed bot, so we never receive a foreign-suffixed command. KISS.
|
||||
- Don't refactor existing Auth / Visibility logic.
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. New unit test passes: `/help@miti99bot` with offset=0 length=15 matches handler registered as `"help"`.
|
||||
2. Existing tests still pass (`make test` clean).
|
||||
3. `golangci-lint run ./...` clean.
|
||||
4. After deploy, repeating "send `/help@miti99bot` in group" produces a log line like `dispatch update_id=... chat_type=group chat_id=-100... text="/help@miti99bot"` AND the help reply is sent.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Custom matcher breaks edge case the library handled | Mirror lib semantics: only match `MessageEntityTypeBotCommand` entities, scope to `update.Message.Text` (HandlerTypeMessageText) |
|
||||
| Log line leaks PII in group chats | Chat IDs and group titles are not secret; user IDs are not logged. Message text truncated to 64 chars. |
|
||||
| Test mutates global state via library bot.New | Tests already use `testutil.NewRecordingBot` — reuse pattern |
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
None — locked decisions: no env var, strip-unconditionally, truncate text to 64 chars in log.
|
||||
@@ -1,296 +0,0 @@
|
||||
# Atlas Migration Plan — Architectural Critique
|
||||
|
||||
**Plan:** `plans/260425-1945-mongodb-atlas-migration/` (8 phases, 24h budgeted)
|
||||
**Scope of critique:** YAGNI/over-engineering; cold-start gate; abort path realism; admin-route reintroduction; SQL-shaped wrapper around a doc store.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. Plan is over-engineered for ~615KB KV + ~300KB/mo trading. Dual-write + verifier + soak machinery costs ~13h to potentially throw away.
|
||||
2. `MongoSqlStore` is a SQL-pattern dispatcher emulating 6 statements — phase-03 is a leaky abstraction; refactoring callers is simpler.
|
||||
3. Phase-05 reintroduces `/__admin/*` HTTP routes that `docs/architecture.md` § 10 explicitly rejected. Direct violation of project posture.
|
||||
4. Abort path (`phase-07-alt-pivot.md`) is a TODO. The "bail-out" is not actually planned — the user has no real escape hatch when the gate trips.
|
||||
5. Trading migration earns nothing on M0 vs D1; phase-03 is ~3h of code with negative business value that just adds risk.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Dual-write is over-engineered for this dataset size
|
||||
|
||||
**Smell.** Phase 04 (3h) + dual-write divergence handling in Phase 06 (parts of 4h) + reverse-backfill stub debt in Phase 07 (Stage 2 rollback) = ~5–6h of complexity, all of which is bypassable.
|
||||
|
||||
**Evidence.**
|
||||
- `phase-04-dual-write-wrappers.md:15-21` — dual-write rationale: "keep secondary warm + ready for read-flip". Worth it for high-traffic continuous-availability systems. **Not a Telegram bot with ~615KB total data.**
|
||||
- `phase-05-backfill-scripts.md:16` — order-of-operations: dual-write must precede backfill or "writes during backfill window go to KV only". This dependency is the entire reason dual-write exists. A maintenance-window cutover removes this constraint.
|
||||
- `phase-04` risk table line `phase-04-dual-write-wrappers.md:159` — "Write amplification doubles latency on every put... Cold p99 = 1500ms". Dual-write WORSENS the very metric Phase 06 is gating on. The plan creates the latency problem then measures it.
|
||||
- KV `list()` is paginated 1000/page (phase-05 line 17). Total dataset spans ~615KB. A one-shot dump is minutes, not hours.
|
||||
|
||||
**Recommendation: CHANGE.** Strongly consider replacing dual-write with a 10-minute maintenance window:
|
||||
1. Set Telegram webhook to a 503 page, OR keep current bot live but disable writes to commands that mutate state.
|
||||
2. Run one-shot KV dump → Mongo upsert (local node script, ~5 min for 615KB).
|
||||
3. Run one-shot D1 dump → Mongo (seconds).
|
||||
4. Flip `STORAGE_PRIMARY=mongo`, redeploy.
|
||||
5. Total downtime: 10 min. Telegram retries webhook deliveries on 503 with backoff so most users never notice.
|
||||
|
||||
This kills phase-04 entirely (-3h), phase-05's `/__admin/*` routes (-2h of phase-05 + risk surface), and most of phase-06's "dual-write divergence" telemetry. Net: ~5–6h saved AND simpler abort path (just don't flip the flag; KV is still untouched on rollback if you keep KV bindings in `wrangler.toml` for one extra deploy cycle).
|
||||
|
||||
**Earns its keep when:** continuous availability is a hard requirement, or dataset is large enough that backfill exceeds an acceptable maintenance window. Neither holds here.
|
||||
|
||||
---
|
||||
|
||||
### 2. Trading migration is dead weight
|
||||
|
||||
**Smell.** Phase-03 (3h) + dual-SQL-store wrapper (part of phase-04) + parity verification (part of phase-05) = ~4–5h to migrate ~100 writes/day.
|
||||
|
||||
**Evidence.**
|
||||
- `phase-03-mongo-sql-store.md:13-15` — "Trading runs exactly 6 distinct SQL statements". Building a SQL-pattern dispatcher to emulate them.
|
||||
- D1 free tier per CF docs: 5M reads/day + 100k writes/day. Trading uses ~100 writes/day. **Trading is using <0.1% of D1's free quota.**
|
||||
- The user's stated goal is "validate cold-start UX firsthand". Trading commands aren't on the cold-start latency hot path — they're rare, transactional, and tolerate a 1.5s spike. Migrating them doesn't validate anything new beyond what KV migration already validates.
|
||||
- `phase-07-cutover-and-decommission.md:81` — wrangler.toml needs D1 binding removed Stage 3. Even if Mongo-side works, you've added MongoSqlStore code that exists for one module.
|
||||
|
||||
**Both sides:**
|
||||
- *For migrating:* operational consistency (one backend), one less binding to manage, eliminates `sql` parameter being conditionally null in module init.
|
||||
- *Against:* MongoSqlStore is a SQL-shaped wrapper around a document store (see Finding 6); D1 is rock-solid for trading's workload; if Atlas pivot to Upstash happens, trading still needs a SQL backend OR a second migration.
|
||||
|
||||
**Recommendation: CHANGE — defer trading migration.** Keep D1 for trading. Migrate KV only. Reasons:
|
||||
- D1 is doing its job well. Don't fix what isn't broken.
|
||||
- The cold-start UX validation is fully covered by KV-using modules (`/wordle`, `/loldle` are listed as the soak gate metrics anyway — phase-06-staged-deploy-and-soak.md:23).
|
||||
- Removes phase-03 (-3h) and SqlStore parts of phase-04 wrapper (-1h) and verifier (-30min).
|
||||
- Upstash pivot becomes simpler: it's a key-value pivot only; D1 stays put.
|
||||
|
||||
**Trade-off:** keeping the dual-backend posture requires `create-sql-store.js` to keep returning `null` when `env.DB` absent. Already true today. Zero cost.
|
||||
|
||||
---
|
||||
|
||||
### 3. Per-module collections vs single shared collection
|
||||
|
||||
**Smell.** 12 collections for 12 modules at ~50KB each. Atlas has no per-collection cost on M0, but ops complexity grows.
|
||||
|
||||
**Evidence.**
|
||||
- `phase-02-mongo-kv-store.md:18-19` — "Per-module collections (12 KV modules → 12 collections), name == module name". Justification: not given beyond convention.
|
||||
- `phase-02-mongo-kv-store.md:78` — keys still carry the prefix in `_id` ("wordle:games:42"). So both the collection name AND the `_id` carry module identity. Redundant.
|
||||
- TTL index needs to be created on each of 12 collections (`phase-02-mongo-kv-store.md:124` — `_ensureIndex` per collection).
|
||||
- Backfill (phase-05) iterates module-by-module — works either way, but verifier counts per-collection.
|
||||
|
||||
**Single-collection alternative.**
|
||||
```js
|
||||
// One collection: kv
|
||||
{ _id: "wordle:games:42", value: "...", expiresAt: ... }
|
||||
```
|
||||
- One TTL index instead of 12.
|
||||
- One `createIndex` call instead of 12 lazy-init paths.
|
||||
- `list()` already filters by `_id` prefix in the per-module wrapper — so cross-module isolation is preserved at the wrapper level, identical semantics.
|
||||
- Index size is meaningfully smaller per-collection only if collections grow non-uniformly. Here, total = 615KB. Irrelevant.
|
||||
|
||||
**Per-module pays off when:** sharding (M10+), per-collection TTL policies differ, or you want per-module backup/restore granularity. None of which apply on M0.
|
||||
|
||||
**Recommendation: CHANGE.** Single `kv` collection. Drops phase-02 complexity slightly (single index init), simplifies verifier, simplifies the dump-routes story (which Finding 5 will recommend deleting anyway). Per-module can be added later if scale demands it — pure YAGNI.
|
||||
|
||||
**Counter-argument the planner could make:** "Per-module collection makes it visually obvious in Atlas UI which module owns what." Valid for ops debugging. But `_id` prefix achieves the same with `db.kv.find({_id:/^wordle:/}).limit(10)`. Cosmetic, not architectural.
|
||||
|
||||
---
|
||||
|
||||
### 4. The phantom abort path
|
||||
|
||||
**Smell.** The whole plan hinges on a "safe rollback to KV/D1 (or pivot to Upstash)" — but the pivot file doesn't exist.
|
||||
|
||||
**Evidence.**
|
||||
- `plan.md:47` — "if P95 > 3s, **abort to phase-07-alt-pivot.md** TODO".
|
||||
- `phase-06-staged-deploy-and-soak.md:130` — "Open `phase-07-alt-pivot.md` (TODO; not pre-written here — wait until needed)."
|
||||
- The 3000ms threshold (`phase-06:109`) is asserted without a citation. Researcher's report on Atlas cold-start gives ~1500ms baseline (`phase-01:50`). 3s is "2× baseline" but **not motivated by a UX research finding** (e.g., "Telegram users abandon at >2.5s").
|
||||
|
||||
**Why this matters.** The user's stated motivation is "validate cold-start UX firsthand with safe bail-out". If the bail-out isn't planned, it isn't safe. When the gate trips at 3 AM and the operator is staring at a 3.2s P95, "draft a new plan" is not an executable rollback procedure — it's a panic invitation.
|
||||
|
||||
The pivot path described in `phase-06:125-130` is high-level (5 bullet points). Real questions unanswered:
|
||||
- How long will the bot run on KV/D1 with `DUAL_WRITE=0` while a new Upstash plan is built? Days? Weeks?
|
||||
- Is `MongoKVStore` reusable for `UpstashKVStore` (similar interface) or starting from scratch?
|
||||
- Does Atlas data get exported before deletion? When?
|
||||
- Does the operator pay for Upstash dev work mid-incident, or is there a 7-day "we're on KV only" window?
|
||||
|
||||
**Threshold motivation.** Research report 3 (free-db-validation-matrix) reportedly recommends Upstash on cold-start grounds. That implies the researchers expect Atlas to fail this gate. Setting the threshold at 3000ms when expected baseline cold is 1500ms gives ~2× headroom — generous, but means the "validate firsthand" exercise will likely PASS, even if UX is degraded for the slow tail. Should be 2× P95 baseline cold-start measured in phase-01 ping (line 88 todo: "ping latency recorded as baseline number") — i.e., DERIVED from measurement, not asserted at 3000ms before measurement.
|
||||
|
||||
**Recommendation: CHANGE.**
|
||||
- (a) Pre-write `phase-07-alt-pivot.md` as a 1-page skeleton: "leave Atlas writes off, KV is authoritative, draft new Upstash plan within 5 days." Make it real, even if minimal. Otherwise rollback is just hope.
|
||||
- (b) Replace the 3000ms threshold with a measurement-derived one: `2.5 × P95(cold-start ping from Phase 01)`. If Phase 01 measures 1400ms cold ping, gate is 3500ms. If 1800ms, gate is 4500ms. Honest measurement, not arbitrary number.
|
||||
- (c) Add a "what does the user feel" research item: 1500ms vs 3000ms vs 5000ms — at what point does a Telegram user retype `/wordle`? Without this, the gate is engineering theater.
|
||||
|
||||
---
|
||||
|
||||
### 5. Re-introducing admin routes violates project architecture
|
||||
|
||||
**Smell.** Phase-05 adds `/__admin/dump-kv` and `/__admin/dump-d1-trades` to the Worker. The architecture doc explicitly rejects this.
|
||||
|
||||
**Evidence.**
|
||||
- `docs/architecture.md:14` (design goals) — "**No admin HTTP surface.** One less attack surface, one less secret. Webhook + menu registration happen out-of-band via a post-deploy node script."
|
||||
- `docs/architecture.md:358-365` (§ 10 "Why the register step is not in the Worker") — explicitly rejected reasons:
|
||||
- "Adds a third secret to manage and rotate."
|
||||
- "Adds an attack surface (even a gated one)."
|
||||
- "Running locally + idempotently means the exact same script works whether invoked by a human, CI, or a git hook."
|
||||
- `phase-05-backfill-scripts.md:53-56` — adds `ADMIN_TOKEN` (a third secret), `/__admin/dump-kv` route, and constant-time-compared auth middleware. **Every objection from § 10 applies verbatim.**
|
||||
- The plan handwaves: routes are "temporary, removed in Phase 07" (`phase-05:99`). But "temporary" admin routes have a way of lingering — and the security review burden lands now, not later.
|
||||
|
||||
**Alternative: local-only dump scripts.**
|
||||
- KV dump: `wrangler kv key list --namespace-id=<id> --remote` paginated, then `wrangler kv key get` per key — slow but works without code changes. OR use the CF KV REST API via `curl` from a local script with an account-token (no Worker change).
|
||||
- D1 dump: `npx wrangler d1 export miti99bot-db --output=trades.sql --remote`. **Already in phase-07 step 12 as a final-stage backup** — same command works for the migration backfill.
|
||||
- Mongo write: from local node, no admin route needed.
|
||||
|
||||
The "wrangler kv key get per-key is too slow" argument (`phase-05:31`) is real but solvable: CF KV REST API supports bulk read via `keys/bulk` endpoint, or chunked workers from local node parallelize the per-key gets. Either is faster than building, securing, deploying, and later removing a Worker route.
|
||||
|
||||
**Recommendation: DELETE.** No `/__admin/*` routes. Use CF KV REST API + `wrangler d1 export` from a local node script. Drops `ADMIN_TOKEN` (a third secret), drops `src/admin/dump-routes.js`, drops the security-considerations debt in phase-05 (lines 165-170), drops the deletion step in phase-07 (lines 13, 128). Net: -1.5h + zero attack surface added.
|
||||
|
||||
**Side note:** the planner KNEW this was sketchy — `phase-05:99` flags routes as "temporary" and `phase-07:13` makes deleting them a pre-cutover step. That's the smell. If something must be deleted before cutover, ask why it had to be added.
|
||||
|
||||
---
|
||||
|
||||
### 6. MongoSqlStore is a SQL-pattern dispatcher emulating SQL on a doc store
|
||||
|
||||
**Smell.** Phase-03 builds a regex-driven dispatcher that pattern-matches 6 SQL statements and translates each to a Mongo call. Worst kind of leaky abstraction: pretends to be SQL, isn't, fails open silently if a 7th statement appears.
|
||||
|
||||
**Evidence.**
|
||||
- `phase-03-mongo-sql-store.md:46` — "Statement matching: trim + collapse whitespace + uppercase first 30 chars; switch on prefix".
|
||||
- `phase-03-mongo-sql-store.md:147` — risk: "7th SQL statement appears (silent breakage)" — likelihood "M", impact "H". The plan's own risk table flags this as MEDIUM-likelihood.
|
||||
- `phase-03-mongo-sql-store.md:153` — "`OFFSET` with `LIMIT -1` (sqlite-ism) is meaningless in Mongo. Handler ignores LIMIT, applies `.skip(N)` only." Behavior is QUIETLY different from D1 — the wrapper isn't actually SQL-compatible, just SQL-shaped.
|
||||
- Trading callers (`history.js`, `retention.js`) are listed as "READ FOR CONTEXT" only (`phase-03:104-107`). They're NOT modified. The wrapper exists to AVOID modifying them. Why? They're 2 small files.
|
||||
|
||||
**Simpler alternative.** Refactor `trading/history.js` + `trading/retention.js` to call a `MongoTradesStore` directly:
|
||||
```js
|
||||
// MongoTradesStore — thin, purpose-built
|
||||
class MongoTradesStore {
|
||||
insert(trade) { /* db.trading_trades.insertOne */ }
|
||||
byUser(userId, limit) { /* find().sort().limit() */ }
|
||||
distinctUsers() { /* distinct */ }
|
||||
oldRowsForUser(userId, keepN) { /* find().skip(keepN) projection */ }
|
||||
oldRows(keepN) { /* find().skip(keepN) projection */ }
|
||||
deleteByIds(ids) { /* deleteMany({_id:{$in}}) */ }
|
||||
}
|
||||
```
|
||||
Six methods, ~80 LOC. Trading module's two files lose their SQL string literals — gain explicit, typed method calls. Reads better. No regex spaghetti. No "7th statement" silent-breakage risk because you can't accidentally route an unknown query.
|
||||
|
||||
**Cost.** Modifying `trading/history.js` and `trading/retention.js`: ~30 LOC of changes total (the SQL strings already isolate the access patterns). That's smaller than the dispatcher + handlers + tests.
|
||||
|
||||
**Counter-argument:** "But then the abstraction leaks into the module — the module knows it's talking to Mongo." Yes. That's HONEST. Today the abstraction leaks the other way — the module knows it's talking to SQL even though it might be Mongo. Either way the module knows. Better to know the truth than the lie.
|
||||
|
||||
**Recommendation: DELETE phase-03.** Replace with: "Add `src/db/mongo-trades-store.js` (~80 LOC) + refactor `trading/history.js` and `trading/retention.js` to use it directly. Delete `SqlStore` interface in phase-07 alongside D1." Saves ~2h vs phase-03 implementation, deletes the dispatcher risk, gives trading module a cleaner persistence boundary.
|
||||
|
||||
**Combined with Finding 2:** if you defer trading migration entirely (recommended), phase-03 deletes outright. ~3h saved.
|
||||
|
||||
---
|
||||
|
||||
### 7. `last_row_id` returned as ObjectId hex is dead-code-by-design
|
||||
|
||||
**Smell.** Plan acknowledges trading doesn't use `last_row_id`, then specifies returning ObjectId hex anyway "for parity".
|
||||
|
||||
**Evidence.**
|
||||
- `phase-03-mongo-sql-store.md:25` — "D1 returns `{ changes, last_row_id }` — last_row_id matters for trading? Grep confirms it's not consumed (insert path discards return). Confirm in step 1."
|
||||
- `phase-03-mongo-sql-store.md:34-35` — "`run` → `{ changes, last_row_id }`. `last_row_id` returns ObjectId hex (since callers don't use it numerically)."
|
||||
- `phase-03-mongo-sql-store.md:153` — "`last_row_id` quietly needed by future caller... if caller does math on it, fails loudly". Defensive hand-wave.
|
||||
|
||||
The whole field exists because the SqlStore *interface* defines it (see `cf-sql-store.js:43`). The wrapper's purpose is interface parity, so it returns the field. But the value is meaningless (a hex string in a contract that historically was an integer). It's dead code dressed up as parity.
|
||||
|
||||
**Recommendation: KEEP if phase-03 stays as-is** (interface parity has weight) — but if the trading-only refactor recommendation (Finding 6) lands, delete `last_row_id` entirely and remove the field from `SqlStore` interface. Two smells solved at once.
|
||||
|
||||
---
|
||||
|
||||
### 8. Soak duration "24-72h" is a vibe, not a criterion
|
||||
|
||||
**Smell.** No exit condition for extending 24h → 72h.
|
||||
|
||||
**Evidence.**
|
||||
- `plan.md:52` — "Phase 06: cold-start P95... > 3000ms over 24h soak window." 24h is the gate.
|
||||
- `phase-06-staged-deploy-and-soak.md:25` — "Soak window: minimum 24h, ideally 72h to span weekly traffic peaks."
|
||||
- `phase-06:109-114` (Success Criteria) only references "24h" and "72h" without defining what data 72h gives that 24h didn't.
|
||||
- Bot has cron-driven traffic (`docs/architecture.md:43` — daily lolschedule, etc.). 24h covers one full daily cycle including all crons. 72h gives 3× sample size but no new patterns unless weekly traffic varies dramatically.
|
||||
|
||||
**Recommendation: CHANGE.** Pick one duration with a stated reason:
|
||||
- 24h with rationale: covers daily cron cycle, sufficient for steady-state.
|
||||
- 72h with rationale: covers a weekend-vs-weekday traffic differential of >1.5×.
|
||||
- Or 7 days if weekly cycle matters.
|
||||
|
||||
Or define a stop condition: "extend to 72h if any of {error rate >0.5%, traffic <50 req/24h, cold-start P95 between 2500–3000ms (borderline)}". Without this, "24-72h" is operator-discretion masquerading as a plan.
|
||||
|
||||
---
|
||||
|
||||
### 9. "Cron heartbeat to prevent M0 auto-pause" is worry-driven
|
||||
|
||||
**Smell.** Listed in plan-level unresolved questions; plan-08-tests-and-docs.md:161.
|
||||
|
||||
**Evidence.**
|
||||
- `phase-01-atlas-setup.md:18` — "M0 auto-pauses after **30 days of zero ops**."
|
||||
- The bot has 6+ daily cron triggers (`wrangler.toml` has `[triggers] crons`).
|
||||
- Trading cron + lolschedule cron + retention crons all hit the DB daily.
|
||||
- 30 days of zero ops would require the bot to be entirely silent for 30 days — implies all crons fail or are removed.
|
||||
|
||||
**Recommendation: DELETE the unresolved Q.** Not a real concern. If ALL crons stop firing for 30 days, you have a bigger problem than auto-pause (the bot is dead). Atlas auto-pause is a non-issue for an active bot. The plan-07 risk table line `phase-07:173` already proposes a "no-op cron heartbeat" — also unnecessary, can be removed.
|
||||
|
||||
If paranoia wins: a single line in the existing `misc` cron handler that does `db.runCommand({ping:1})` covers it. Not worth a separate unresolved question.
|
||||
|
||||
---
|
||||
|
||||
### 10. Tests-and-docs in phase 08 is debt, not closure
|
||||
|
||||
**Smell.** Phase 08 (3h, post-cutover) catches doc updates AND a meaningful new e2e test. Ordering is wrong.
|
||||
|
||||
**Evidence.**
|
||||
- `phase-08-tests-and-docs.md:24-26` — e2e test: "Boot fake env with `MongoKVStore` + `MongoSqlStore` against `fake-mongo.js`. Run a representative request..." This validates the FULL stack. Should land BEFORE the irreversible cutover (phase-07 step 18), not after.
|
||||
- Phases 02–04 each include their own unit tests. Good.
|
||||
- BUT phase-04 dual-store testing (`phase-04-dual-write-wrappers.md:127-131`) only verifies factory behavior with fake env — does not verify a request actually round-trips through both stores.
|
||||
- `phase-08:104` proposes a `scripts/check-secret-leaks.js` lint rule. Should have been added in phase-01 alongside the URI introduction, before any commit could leak. Now it's added after 7 phases worth of commits could already contain leaks.
|
||||
|
||||
**Recommendation: CHANGE.**
|
||||
- (a) Move the e2e test from phase-08 to phase-04 (or a new phase-04.5). Run it before phase-05 backfill. Run it before phase-06 deploy. Run it before phase-07 cutover. It's a regression net.
|
||||
- (b) Move the secret-leak lint to phase-01 step 11 (immediately after `MONGODB_URI` is introduced as a secret). 5-min addition.
|
||||
- (c) Phase 08 stays as docs-only update + final mark-complete. Trims it to ~1h.
|
||||
|
||||
Sequencing rule of thumb: tests for a phase land WITH the phase, not in a stash-everything-into-phase-08 graveyard.
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting observations
|
||||
|
||||
**Plan budget is misallocated.** 24h budget → 8h on dual-write/backfill/admin routes (Findings 1, 5) + 3h on trading SQL emulation (Findings 2, 6) = 11h on machinery the user might not need. Closer to 13h if Phase 06 telemetry overhead and Phase 08 catch-up are included.
|
||||
|
||||
**Plan is over-confident on cold-start.** Phase-06 bakes in the ABORT path (good) but the gate threshold is asserted, not derived; the pivot is a TODO; the user has been told this is "safe to bail out of" but the bail-out is unwritten.
|
||||
|
||||
**Plan respects existing project posture EXCEPT in Finding 5.** The architecture doc is clear and this plan re-introduces the exact pattern the doc rejects. Either:
|
||||
- The plan was drafted without consulting `docs/architecture.md` § 10, OR
|
||||
- The convenience of admin routes felt worth the violation.
|
||||
|
||||
Either way: a planner who reintroduces a rejected pattern owes a one-line justification in the phase file. None present.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized action list (impact order)
|
||||
|
||||
1. **DELETE Phase 05's `/__admin/*` routes.** Use CF KV REST API + `wrangler d1 export` from local. Removes a third secret, removes attack surface, complies with `docs/architecture.md` § 10. (Finding 5)
|
||||
|
||||
2. **CHANGE Phase 04 to maintenance-window cutover.** Drop dual-write entirely. ~5–6h saved + removes the latency-amplification problem that Phase 06 is gating on. (Finding 1)
|
||||
|
||||
3. **CHANGE Phase 03: defer trading migration OR replace with `MongoTradesStore` direct refactor.** Either drops phase-03 entirely (-3h) or simplifies it from SQL-pattern-dispatcher to 6 explicit methods. (Findings 2 + 6 combined)
|
||||
|
||||
4. **PRE-WRITE `phase-07-alt-pivot.md` skeleton + DERIVE the cold-start threshold from Phase-01 baseline measurement.** Make the bail-out real. (Finding 4)
|
||||
|
||||
5. **MOVE e2e test from Phase 08 to before Phase 06.** Catch storage-roundtrip regressions BEFORE deploying the dual-write code (or with the maintenance-window approach, before cutover). Move secret-leak lint to Phase 01. (Finding 10)
|
||||
|
||||
6. **CHANGE Phase 02 to single shared `kv` collection.** YAGNI — per-module collections add ops complexity without M0-scale benefit. (Finding 3)
|
||||
|
||||
7. **DEFINE Phase 06 soak duration with stated criterion.** Pick 24h with reason, OR pick 72h with reason. Stop saying "24-72h". (Finding 8)
|
||||
|
||||
8. **DELETE the "M0 auto-pause cron heartbeat" unresolved question.** Bot has 6+ daily crons. Non-issue. (Finding 9)
|
||||
|
||||
If recommendations 1+2+3 land: budget drops from 24h → ~12h. Plan is half the size, more reversible, fewer secrets, fewer code files to delete in phase-07, simpler architecture-doc story.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved questions for the planner
|
||||
|
||||
- Is "validate cold-start UX firsthand" a UX research goal (with a defined user-experience threshold) or an engineering curiosity? Answer changes whether the 3000ms gate is right.
|
||||
- Is the user willing to accept a 10-minute Telegram bot maintenance window? If yes, dual-write is unnecessary.
|
||||
- Is the user committed to migrating BOTH KV and D1, or is keeping D1 acceptable? Plan assumes both; nobody's challenged it.
|
||||
- What's the SLO for `/wordle` reply latency that the user actually cares about? The plan uses 3000ms; users might tolerate less or more.
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
**Summary:** Plan is structurally sound but over-engineered for dataset size; reintroduces admin-route pattern explicitly rejected by project docs; bail-out path is a TODO. Recommend dropping dual-write, dropping admin routes, deferring trading migration. Cuts plan budget ~50%.
|
||||
@@ -1,328 +0,0 @@
|
||||
# Atlas Migration Plan — Correctness Review
|
||||
|
||||
## TL;DR
|
||||
|
||||
Plan is **mostly implementable but has 5 blockers and ~10 high-severity gaps that will burn the implementer**. Worst gap: `MongoSqlStore` SQL-pattern dispatch is internally contradictory (5 handlers for 6 statements, retention.js queries 3 & 4 differ in shape and bind layout in ways the plan handwaves) AND `last_row_id` is required by the existing test contract (`tests/db/create-sql-store.test.js:48-52`) which directly contradicts plan-03's claim it's unused. Cold-start measurement methodology is hand-waved, instrumentation snippet has a `marks` reference-error bug, and the plan's "stubMongo" approach is effectively a dead-letter URI that risks stalling register if any code path tries to construct a MongoClient with it (which the matrix permits if anyone misconfigures the flags).
|
||||
|
||||
The plan is "rollback-correct" but not "round-trip-correct" — Stage-2 reverse-backfill is explicitly NOT pre-built, so a Stage-2 abort with new Mongo-only writes loses data on rollback to KV.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Severity | File:Section | Issue | Recommendation |
|
||||
|---|----------|--------------|-------|----------------|
|
||||
| 1 | BLOCKER | phase-03 §Key Insights, §Functional | Plan claims `last_row_id` is unused; `tests/db/create-sql-store.test.js:48-52` explicitly asserts `result.last_row_id` is present. Returning ObjectId hex (string) where test expects a number breaks build. | Either return `last_row_id: 0` for INSERT (numeric, matches D1 "no rowid" path) and don't pretend it's the inserted ID; OR refactor the existing test to accept hex. Document choice in phase-03. |
|
||||
| 2 | BLOCKER | phase-04 §Stub for register, §Implementation step 6 | `stubMongo = "mongodb://stub-not-used"` is a string, not a duck-typed `MongoClient`. If any code path constructs `new MongoClient(stubMongo)` (e.g. someone forgets to wire the `DUAL_WRITE=0` short-circuit, or `STORAGE_PRIMARY=mongo` env leaks into register), `register.js` hangs ≥10s on `serverSelectionTimeoutMS`. The matrix correctness depends on factory branching that doesn't exist yet. | Make `stubMongo` a duck-typed object exposing `db()`, `connect()` no-ops (pattern matches `stubKv`). Or: have factories check `env.MONGODB_URI === stubMongo` literal sentinel and short-circuit unconditionally. Add a unit test that constructs the factory with stubMongo + every flag combo and asserts no network call. |
|
||||
| 3 | BLOCKER | phase-03 §Key Insights bullet "Mongo equivalents", §Architecture handlers | Statement 4 (`SELECT id ... WHERE user_id = ? ORDER BY ts DESC LIMIT -1 OFFSET ?`) and statement 5 (`SELECT id ... ORDER BY ts DESC LIMIT -1 OFFSET ?`) are described as one handler "handleRetentionOffset" but plan never specifies how the dispatcher distinguishes per-user vs global. The 3 LIMIT-token resolution rules in `tests/fakes/fake-d1.js:155-170` (negative limit = "all rows") are also missing from plan. Without a spec, implementer cannot write the dispatcher. | Add explicit dispatcher rules: detect `WHERE user_id` substring after normalization; route to per-user vs global. Document that LIMIT -1 is sqlite-ism for "no limit"; Mongo handler ignores it entirely and applies only `.skip(N)`. List the 6 normalized prefixes as constants in the plan. |
|
||||
| 4 | BLOCKER | phase-07 §Rollback Stage 2 | "Reverse-backfill from Mongo to KV. Build script on demand." During Stage-2 7-day soak, Mongo-only writes accumulate. If Stage-2 abort happens on day 7, KV is missing 7 days of writes. Operator must build a backfill script under outage pressure with no test coverage. | Pre-build `scripts/backfill-mongo-to-kv.js` and `backfill-mongo-to-d1.js` BEFORE Stage 2 begins. Test against fake-mongo + fake-kv. List in phase-07 step 8 prerequisites. |
|
||||
| 5 | BLOCKER | phase-01 §Risk + phase-02 §Implementation | `mongodb` v6.7+ npm bundle is **~4-5 MB** per the researcher report. CF Workers Free plan limit is **3 MiB compressed** (10 MiB unbottled paid). Plan's mitigation "measure with `wrangler deploy --dry-run`" is reactive, not preventive. If bundle exceeds limit, Phase 02 is blocked AFTER Phase 01 already committed `mongodb` install + wrangler config changes. | Move bundle-size check to phase-01 step 8 as a HARD gate. Budget ≥1MB headroom. Document fallback: switch to `mongodb-driver-core` (no DNS/SRV resolver) or abandon Atlas before any further phase work. |
|
||||
| 6 | HIGH | phase-02 §Architecture "Decision: keep prefix inside `_id`" + phase-04 §Functional | `create-store.js:65` strips the prefix from results. If MongoKVStore stores `_id="wordle:games:42"` and `list({prefix:"games:"})` is called from inside the wrapper (which prepends `"wordle:"` first → `fullPrefix="wordle:games:"`), MongoKVStore must filter `_id` starting with that AND return KEYS WITH PREFIX so `create-store.js` can strip. Plan inconsistently says "prefix-strip mirrors `create-store.js:65`" (key 22) AND "the store doesn't strip prefixes" (key 76). Ambiguity will yield wrong list results. | Match CFKVStore behavior exactly: MongoKVStore returns `keys` AS-IS (prefixed). The wrapper strips. Make this explicit in phase-02 §Functional and write a regression test using a 2-level prefix (`wordle:games:`) to lock semantics. |
|
||||
| 7 | HIGH | phase-02 §Functional, §Implementation step 3 | TTL stale-read window: Mongo TTL sweeper runs every ~60s. A key with `expirationTtl=10s` is still readable for up to 60s after expiration. Plan acknowledges this in Risk row 2 but `getJSON` does NOT check `expiresAt` field on read. Consumers (e.g. game state with TTL) will see "expired" data. CF KV does NOT have this behavior. | Add explicit `expiresAt` filter on read in MongoKVStore: `findOne({_id, $or: [{expiresAt: {$exists: false}}, {expiresAt: {$gt: new Date()}}]})`. OR document the divergence loudly in the contract. Test must cover: put with 1s TTL, sleep 2s, getJSON returns null. |
|
||||
| 8 | HIGH | phase-04 §Functional, §Architecture sequence | Dual-write order is documented as `Promise.all` parallel. But plan says "Throw only on primary failure" and "secondary failures get reconciled in Phase 05 backfill". The reconciliation path is broken: Phase 05 backfill uses `$setOnInsert` (skip-if-exists). If primary write succeeded but secondary failed mid-dual-write, the doc IS missing in Mongo, so backfill would fill it from KV. **But Phase 05 is documented as a one-shot, not a recurring sweeper.** Divergences accumulating after Phase 05 are silent. | Add a recurring "drift verifier" cron (1/hr) that samples N keys cross-store and logs/alerts mismatches. Or: change secondary-failure handling to push the failed key onto a retry queue (KV list) that Phase 05 verifier drains. |
|
||||
| 9 | HIGH | phase-04 §Architecture flag matrix row 4 | `STORAGE_PRIMARY=mongo` + `DUAL_WRITE=0` post-cutover means writes go ONLY to Mongo. M0 is single-region (aws-ap-southeast-1) with NO BACKUPS (acknowledged in phase-01 key insight 5). For 7-day Stage-2 soak there is no fallback if Mongo loses data. | Add an explicit Stage-2 safeguard: enable mongodump-equivalent (or `npm run backfill:d1` reversed for trading) on day-3 of Stage 2 to checkpoint to local disk. Or: keep `DUAL_WRITE=1` permanently (leave KV writes on for one extra cycle as zero-cost insurance) and only flip after Phase 07 binding deletion is irreversible. |
|
||||
| 10 | HIGH | phase-06 §Architecture "Telemetry helper" | Code snippet refers to `marks` outside its definition: `console.log(JSON.stringify({event:"cmd_timing", cmd, total, ...extra, marks}))` — `marks` is in the outer closure but never declared in the snippet. Implementer copy-pastes → ReferenceError at runtime. | Fix the snippet: declare `const marks = []` at top; have `mark(label)` push to it. Or remove `marks` from output. Snippet quality matters because Phase 06 is the decision gate. |
|
||||
| 11 | HIGH | phase-06 §Functional, §Implementation step 3 | Cold-start detection via `Date.now() - ISOLATE_BORN < 200ms` is unreliable. CF isolate boot can be 50ms but the Mongo connect + first read is 1500ms. By the time the FIRST handler logs, age may already be >200ms. Cold requests get bucketed as warm. | Use a different signal: track a `let isFirstRequest = true; if (isFirstRequest) { isFirstRequest = false; logCold(); }` flag at module scope. Combined with `isolate_age_ms` for the histogram, but use the boolean for cold/warm bucketing. |
|
||||
| 12 | HIGH | phase-08 §Functional doc updates | Plan does not mention removing `npm run db:migrate` from `package.json:13` chain inside `npm run deploy` (line 12). Phase 07 step 19 deletes `scripts/migrate.js` but `npm run deploy` still calls `npm run db:migrate` — deploys break. | Add to phase-07 step 17: edit `package.json` to remove `db:migrate` from the `deploy` chain and remove the `db:migrate` script entry. |
|
||||
| 13 | HIGH | phase-05 §Functional, §Implementation step 4 | Backfill of D1 → Mongo for trading: D1 uses `INTEGER PRIMARY KEY AUTOINCREMENT`; Mongo plan uses `ObjectId`. Plan says "fields mapped 1:1" but the `id` field is the table's primary key and (per phase-03) is exposed as `_id.toHexString()`. Existing trading rows lose their original integer IDs entirely. Any external consumer (logs, exports) referencing trade IDs is broken. Retention's DELETE-by-id pass works (uses string-or-int either way) but historical trade IDs in logs become un-joinable. | Either (a) preserve original integer ID in a `legacy_id` field during backfill so support queries still work; or (b) document that pre-cutover trade IDs are abandoned and add a migration note to changelog. Currently, plan does neither. |
|
||||
| 14 | HIGH | phase-01 §Functional, §Implementation step 7 | `compatibility_flags = ["nodejs_compat_v2"]`: per researcher report this is correct. BUT `nodejs_compat` (v1) and `nodejs_compat_v2` are NOT additive — picking v2 changes process/buffer/streams globals. Existing modules using `crypto.timingSafeEqual` (Phase 05 step 2) or other Node APIs may behave differently. Plan's mitigation "run full test suite after edit" only covers unit tests; suite uses fakes, not workerd. | Add: deploy a `wrangler dev` smoke that exercises every module's first command. Add a section in `docs/using-mongodb.md` listing every `node:` import the codebase uses and which surface they need. Otherwise a silent prod regression is plausible. |
|
||||
| 15 | HIGH | phase-05 §Implementation step 2, §Security | `/__admin/dump-kv` returns raw values including TTL records. Plan correctly uses `crypto.timingSafeEqual` but does NOT specify what happens on token mismatch — bare `return 401` leaks timing via early-return before timing-safe compare runs. Also: the plan does not say where the route mount lives in `src/index.js` request flow vs the existing `/webhook` handler (route-ordering matters; admin routes must NOT be reached without auth). | Specify: (a) compare token first, before any branching by header presence; (b) return 401 with no body; (c) place admin routes BEFORE `/webhook` in the dispatcher with explicit order documented. (d) Set `X-Robots-Tag: noindex` to prevent caching/indexing. (e) Rate-limit (CF native or cheap counter in KV). |
|
||||
| 16 | MEDIUM | phase-02 §Architecture connection memoization | `client = new MongoClient(...)` then `connectPromise = client.connect()` — if `connect()` REJECTS, both `client` and `connectPromise` stay populated. Next call: `if (client) return client.db(...)` — returns a Db on a dead client. All subsequent reads hang or fail with cryptic "no connection" errors. | On reject, null both. Pattern: `connectPromise = client.connect().catch(err => { client = null; connectPromise = null; throw err; })`. |
|
||||
| 17 | MEDIUM | phase-02 §Implementation step 1 fake-mongo surface | Listed methods omit `find().sort().skip()` chaining (used by retention pattern from phase-03). Phase-08 §Unresolved Q1 says TTL semantics deferred. Surface inventory is incomplete. | Re-derive the surface from phase-03 handlers: insertOne, find/sort/skip/limit/project/toArray, distinct, deleteMany, countDocuments, createIndex. Audit phase-03 handlers against fake-mongo capabilities before phase-02 starts (ordering: phase-03 design completes the surface, then fake-mongo). |
|
||||
| 18 | MEDIUM | phase-04 §Architecture flag matrix | Rollback case: "primary=mongo, dual=on" → revert to "primary=kv, dual=on". During the time between cutover (mongo-primary) and rollback decision, Mongo accepted writes that KV may have missed (if dual-write secondary failed silently). When primary flips back to KV, those writes are invisible. The flag matrix doesn't address this; phase-07 only handles the "Stage 2 with KV stale" version. | Document explicitly in phase-04: rollback to KV-primary AFTER any Mongo-primary period requires reverse-backfill of any new Mongo doc. Cross-link to finding #4. |
|
||||
| 19 | MEDIUM | phase-01 §Functional + phase-08 §Lint check | `MONGODB_URI` rotation cadence: phase-08 unresolved Q5 leaves "quarterly proposed". Phase-01 says "rotate quarterly" without owner. Telegram secrets pattern says nothing about cadence either — there is no project-level secret rotation runbook. | Phase-08 Step in `using-mongodb.md`: add a runbook section "Rotation: every 90 days, owner = repo maintainer; calendar entry to be created in `docs/cost-tracking.md` review cycle". Or accept "rotate-on-suspicion-only" and document. |
|
||||
| 20 | MEDIUM | phase-04 §Implementation step 7 | `wrangler.toml` already has `[vars] MODULES = "..."`. Adding `STORAGE_PRIMARY` and `DUAL_WRITE` to same `[vars]` block is correct. But phase-07 step 17 deletes them — and the phase-04 dual-store factory hard-reads them. After phase-07 deletion, factories must default sanely (no flags = mongo-only). Plan does not specify the post-deletion factory behavior beyond "simplify to Mongo-only (no flags)". | Pre-design the post-cutover `create-store.js` shape in phase-04 (so the simplification step is mechanical in phase-07). Specifically: factories default to `MongoKVStore`-only when flags absent; KV/D1 branches removed. |
|
||||
| 21 | MEDIUM | phase-05 §Functional, sample size | Verifier samples 100 keys per module. With ≥1000 keys/module possible (loldle-emoji game state), 100/1000 = 10% sample. At 0.1% real corruption rate, expected mismatches = 0.1 — verifier reports PASS even with 1 in 1000 keys silently corrupted. | Increase sample to N=√(total) capped at 500. Or: full-scan compare on collections <10K docs (cheap on M0 with small data). |
|
||||
| 22 | MEDIUM | phase-06 §Connection saturation criterion | "M0 connection peak ≤ 400 of 500". Plan says observed via Atlas dashboard but does not specify alerting or capture. Operator must check manually each day during 24-72h soak — easy to miss. | Add: enable Atlas free-tier alert email at "current connections > 400". Or: programmatic poll every 5min via Atlas Admin API (free tier supports). Pin in `using-mongodb.md`. |
|
||||
| 23 | MEDIUM | phase-02 §Architecture connection options | `maxPoolSize: 1, minPoolSize: 0`. Single connection per isolate is correct for free tier, but the driver default `maxPoolSize=100` means in dev / locally, ops parallel to 100. `serverSelectionTimeoutMS: 5000` means a paused M0 cluster gives a 5s hang per cold start (acknowledged in Phase 06 abort but not as a normal-ops cost). | Document the latency budget: cold = 1500ms (TLS+SCRAM) + ≤5000ms (server selection) = up to 6.5s p99 absolute worst case. Make sure the abort gate at phase-06 (>3000ms = abort) accounts for this. The 3000ms threshold may be unreachable even in the green case. |
|
||||
| 24 | MEDIUM | phase-03 §Implementation step 1 | "Grep `src/modules/trading/` for every `.run(`, `.all(`, `.first(`, `.prepare(`, `.batch(` call. Confirm only 6 SQL strings exist." Verified in this review: 6 found, 0 prepare/batch. But: the grep is on TODAY's code. If trading evolves before Phase 03 ships (developer adds a 7th query during dual-write deployment window), dispatcher silently fails to match → `Error("MongoSqlStore: pattern not matched")` thrown to user mid-handler. | Add to phase-04 dispatcher: log unmatched queries to telemetry, fall back to D1 read during dual-write (since D1 is still authoritative until Stage 2 starts). After Stage 2, unmatched queries are a hard failure that surfaces in Phase 06 monitor. |
|
||||
| 25 | MEDIUM | phase-08 §Unresolved Q4 | "Cron heartbeat to prevent M0 auto-pause" — `wrangler.toml:43` has crons `["0 17 * * *", "0 1 * * *"]`. Misc module's `last_ping` write is on every ad-hoc command, not on cron. If bot has zero command activity for 30 days, M0 auto-pauses. The user has stated this is unlikely but plan should be deterministic, not probabilistic. | Add a cron schedule that explicitly writes `misc:last_ping` (or any Mongo doc) every 7 days. Verify it executes against Mongo (post-cutover, all writes go to Mongo, so the existing daily cron via `misc` module is sufficient — ONLY if any of those crons writes Mongo, which the misc module crons need to be confirmed to do). |
|
||||
| 26 | LOW | phase-01 §Architecture cold path | "Cold path: ~1500ms (TLS + SCRAM + server selection)." Server-side region is `aws-ap-southeast-1`; CF PoP for VN traffic is also Singapore. Latency floor is ~10ms, so 1500ms is dominated by TLS+SCRAM. Reasonable estimate. No action; calling out for transparency. | None. |
|
||||
| 27 | LOW | phase-04 §Test plan integration test | "Verify via `instanceof` check after exposing `_implementations` array on dual stores for testability." This is a code-smell — exposing internals for tests. | Use a sentinel string like `store._kind === "dual"` (one line, not arrays). Or test behavior end-to-end (write → read from one of two seams). |
|
||||
| 28 | LOW | phase-08 §Functional doc list | `docs/development-roadmap.md` mentioned twice in plan (phase-08 §Functional + Todo). User feedback memory says "roadmap = future only" — completed migration must NOT be added; it should be REMOVED if currently listed. Phase-08 says "remove migration item per future-only convention" — correct. | None — already correct. Just flagging consistency. |
|
||||
| 29 | LOW | phase-02 §Architecture document shape | `value` stored as string. CFKVStore stores strings via `kv.put(key, value)` where value is already serialized. Plan correctly mirrors this. But Mongo allows native typed values — slight space waste. Performance: `JSON.parse(doc.value)` per read. Acceptable; matches contract. | None — matches CFKVStore contract. |
|
||||
| 30 | LOW | phase-07 §Implementation step 18 destructive ops | `npx wrangler kv namespace delete --namespace-id <id>` and `npx wrangler d1 delete miti99bot-db` are interactive on wrangler 4.x (require typed confirmation). Plan flags this in Risk row 4 ("Operator runs the destructive commands manually") — correct. | None — already addressed. |
|
||||
|
||||
---
|
||||
|
||||
## Per-section deep dives — BLOCKER + HIGH only
|
||||
|
||||
### Finding #1: `last_row_id` is consumed by an existing test
|
||||
|
||||
**File:** `tests/db/create-sql-store.test.js:48-52`
|
||||
```js
|
||||
it("returns changes and last_row_id on INSERT", async () => {
|
||||
...
|
||||
expect(result).toHaveProperty("last_row_id");
|
||||
});
|
||||
```
|
||||
|
||||
Plan-03 §Key Insights says "D1 returns `{ changes, last_row_id }` — last_row_id matters for trading? Grep confirms it's not consumed (insert path discards return). Confirm in step 1." This is technically true for `src/modules/trading/`, but the SqlStore CONTRACT (`sql-store-interface.js:19`) declares `last_row_id` as `number`, and the test enforces shape. Returning a hex string violates the existing contract. Decisions:
|
||||
- (a) Keep contract as `number`. New `MongoSqlStore` returns `last_row_id: 0` for inserts (matches "no rowid" semantics). Existing test passes.
|
||||
- (b) Loosen contract to `number | string`. Update test. Risky for any future caller doing arithmetic.
|
||||
|
||||
**Recommendation:** Pick (a). Document in phase-03 that `last_row_id` is non-meaningful in MongoSqlStore; if future caller needs the inserted ID, they must read `_id` from a separate read.
|
||||
|
||||
### Finding #2: `stubMongo` is a string, not a duck-typed binding
|
||||
|
||||
**File:** `phase-04-dual-write-wrappers.md:88-93`
|
||||
|
||||
Plan says: `export const stubMongo = "mongodb://stub-not-used";`
|
||||
|
||||
But `stubKv` is a duck-typed object with `.get`, `.put`, `.list` methods. The plan's matrix relies on every register-time path having `DUAL_WRITE=0` so `MongoClient` is never constructed with `stubMongo`. This is a tight coupling: any future change to factory branching (or accidental flag mis-set) will result in `new MongoClient("mongodb://stub-not-used")` which:
|
||||
- Triggers DNS resolution attempt
|
||||
- Hangs on `serverSelectionTimeoutMS=5000ms`
|
||||
- Fails register, blocking deploy
|
||||
|
||||
Plan gives no test that asserts "stubMongo never reaches MongoClient". Build break risk is real.
|
||||
|
||||
**Recommendation:** Convert to duck-typed object stub:
|
||||
```js
|
||||
export const stubMongo = {
|
||||
db() { return stubMongoDb; },
|
||||
connect: async () => undefined,
|
||||
close: async () => undefined,
|
||||
};
|
||||
```
|
||||
Then `mongo-client.js getDb(env)` checks if `env.MONGODB_URI === STUB_SENTINEL` (a defined constant) and short-circuits. Add unit test in phase-04 that constructs the factory with stubMongo + each flag combination and uses `vi.spyOn(MongoClient.prototype, 'connect')` to assert ZERO connect calls.
|
||||
|
||||
### Finding #3: Retention dispatcher has 2 statements but plan describes 1 handler
|
||||
|
||||
**File:** `phase-03-mongo-sql-store.md:115` ("handles both 4 & 5 — one user-scoped, one global — distinguished by presence of WHERE")
|
||||
|
||||
The plan's normalized-prefix matching scheme breaks here:
|
||||
- Statement 4: `SELECT id FROM trading_trades WHERE user_id = ? ORDER BY ts DESC LIMIT -1 OFFSET ?` (binds: `[userId, offset]`)
|
||||
- Statement 5: `SELECT id FROM trading_trades ORDER BY ts DESC LIMIT -1 OFFSET ?` (binds: `[offset]`)
|
||||
|
||||
After normalization (collapse whitespace, trim, uppercase first 30 chars), prefix is `SELECT ID FROM TRADING_TRADES`. Identical. Dispatcher cannot distinguish on first 30 chars. Plan's "presence of WHERE user_id" is the actual signal — but plan doesn't say to inspect beyond 30 chars.
|
||||
|
||||
Bind layout differs:
|
||||
- Stmt 4: `binds[0]=userId, binds[1]=offset`
|
||||
- Stmt 5: `binds[0]=offset`
|
||||
|
||||
Handler logic must inspect WHERE presence to know how to read binds.
|
||||
|
||||
**Recommendation:** Phase-03 dispatcher must do:
|
||||
1. Normalize full query (not just first 30 chars).
|
||||
2. Match via regex: `WHERE\s+user_id\s*=\s*\?` → user-scoped, binds[0]=userId, binds[1]=offset.
|
||||
3. Else if matches global pattern → binds[0]=offset.
|
||||
4. Document the regex set as constants alongside the 6 statements. List explicitly:
|
||||
|
||||
```js
|
||||
const SQL_PATTERNS = {
|
||||
INSERT_TRADE: /^INSERT INTO TRADING_TRADES \(USER_ID, SYMBOL, SIDE, QTY, PRICE_VND, TS\) VALUES/,
|
||||
HISTORY_QUERY: /^SELECT ID, USER_ID, SYMBOL, SIDE, QTY, PRICE_VND, TS FROM TRADING_TRADES WHERE USER_ID = \? ORDER BY TS DESC LIMIT \?$/,
|
||||
DISTINCT_USERS: /^SELECT DISTINCT USER_ID FROM TRADING_TRADES$/,
|
||||
RETENTION_USER_OFFSET: /^SELECT ID FROM TRADING_TRADES WHERE USER_ID = \? ORDER BY TS DESC LIMIT -1 OFFSET \?$/,
|
||||
RETENTION_GLOBAL_OFFSET: /^SELECT ID FROM TRADING_TRADES ORDER BY TS DESC LIMIT -1 OFFSET \?$/,
|
||||
DELETE_BY_IDS: /^DELETE FROM TRADING_TRADES WHERE ID IN \(/,
|
||||
};
|
||||
```
|
||||
|
||||
Add a test that mismatches one statement (e.g. extra space after `LIMIT`) and asserts the dispatcher errors loudly with "MongoSqlStore: pattern not matched" — before falling back to D1 (during dual-write) or failing (post-cutover).
|
||||
|
||||
### Finding #4: Stage-2 reverse-backfill is undocumented work
|
||||
|
||||
**File:** `phase-07-cutover-and-decommission.md` Stage 2 step 11 + Rollback table
|
||||
|
||||
Stage 2 deploys with `DUAL_WRITE=0`, so for 7 days, KV becomes stale. Rollback path: "Re-enable `DUAL_WRITE=1` + flip `STORAGE_PRIMARY=kv`. KV is stale by N days but recoverable via reverse-backfill (script not pre-written; build only if rollback needed)."
|
||||
|
||||
Building a reverse-backfill UNDER outage pressure is a known anti-pattern: untested code, no time to verify, possible data loss. The script can be written as straightforward inverse of `backfill-mongo.js`:
|
||||
- For each Mongo collection, list all docs with `_id.startsWith("modulePrefix:")`
|
||||
- For each, write to KV via `/__admin/inject-kv` route (auth-gated)
|
||||
- Plus: tracking which keys were updated AFTER the cutover decision so the operator knows the pre-Stage-2 state vs post-Stage-2 deltas
|
||||
|
||||
**Recommendation:** Build `scripts/backfill-mongo-to-kv.js` and `scripts/backfill-mongo-to-d1.js` as part of phase-07 step 11 prerequisites (BEFORE Stage 2 deploys). Test against fakes with full coverage. List in plan as a non-optional artifact.
|
||||
|
||||
### Finding #5: mongodb driver bundle size vs CF Workers limit
|
||||
|
||||
**File:** `phase-01-atlas-setup.md` Risk row 5
|
||||
|
||||
CF Workers Free plan limit: **3 MiB compressed** worker bundle (10 MiB uncompressed for paid). The `mongodb` v6.7+ npm package includes:
|
||||
- Core driver (~600KB minified)
|
||||
- BSON (~300KB)
|
||||
- SRV/SCRAM auth (~150KB)
|
||||
- TLS/socket compatibility shims for nodejs_compat_v2 (~variable)
|
||||
|
||||
Researcher report says "~4-5 MB compressed" — over the free-plan limit.
|
||||
|
||||
If after `wrangler deploy --dry-run` the bundle exceeds 3 MiB, phase-01 has already committed:
|
||||
- `wrangler.toml` change (compatibility flag)
|
||||
- `package.json` change (mongodb dep)
|
||||
- secrets created
|
||||
- atlas cluster running
|
||||
|
||||
…and phase-02 cannot deploy. Operator must rollback all of phase-01.
|
||||
|
||||
**Recommendation:** Phase-01 step 8 (after `npm install mongodb`) MUST check: `npx wrangler deploy --dry-run --outdir=./.tmp-deploy && du -sh ./.tmp-deploy`. Hard gate at 2.7 MiB (10% headroom). If exceeds, abort phase entirely; pivot to Upstash plan (smaller HTTP-only client). Document in phase-01 §Success Criteria.
|
||||
|
||||
### Finding #6: list() prefix-strip ambiguity
|
||||
|
||||
**File:** `phase-02-mongo-kv-store.md` line 22 + line 76 contradict
|
||||
|
||||
Currently `create-store.js:65` does the strip:
|
||||
```js
|
||||
keys: result.keys.map((k) => (k.startsWith(prefix) ? k.slice(prefix.length) : k)),
|
||||
```
|
||||
|
||||
CFKVStore.list() (cf-kv-store.js:71-72) returns keys WITH PREFIX. The wrapper strips. Plan-02 says "list() returns ... module namespace already stripped" (from interface JSDoc) but ALSO "keep prefix inside `_id`... the store doesn't strip prefixes."
|
||||
|
||||
The interface contract (`kv-store-interface.js:27`) literally says "module namespace already stripped" — but that's only true at the wrapper layer (createStore), not at the underlying CFKVStore. CFKVStore returns prefixed keys.
|
||||
|
||||
**Recommendation:** MongoKVStore returns prefixed keys (matches CFKVStore). The wrapper strips. Tests for MongoKVStore directly (without wrapper) assert prefixed keys returned. Tests for `createStore("wordle", env)` assert stripped keys. Make this explicit: phase-02 §Functional updated to "list() returns keys with module namespace **preserved**; the namespace wrapper in `create-store.js:65` strips the prefix."
|
||||
|
||||
### Finding #7: TTL stale-read window
|
||||
|
||||
**File:** `phase-02-mongo-kv-store.md` Risk row 2 + missing read-time check
|
||||
|
||||
The Mongo TTL background task runs every ~60 seconds. Document with `expirationTtl: 10` is still readable for up to 60s after expiration. CFKVStore + CF KV does NOT have this stale-read window — CF KV is consistent on read.
|
||||
|
||||
For game state (`expirationTtl: 7 days`), 60s of slack is invisible.
|
||||
For short-lived caches (e.g. price feeds with `expirationTtl: 60s`), users could see "expired" cache entries.
|
||||
|
||||
**Recommendation:** MongoKVStore.get and getJSON filter at read time:
|
||||
```js
|
||||
async get(key) {
|
||||
const doc = await coll.findOne({
|
||||
_id: key,
|
||||
$or: [
|
||||
{ expiresAt: { $exists: false } },
|
||||
{ expiresAt: { $gt: new Date() } }
|
||||
]
|
||||
});
|
||||
return doc?.value ?? null;
|
||||
}
|
||||
```
|
||||
Cost: minor query complexity; covered by `_id` index. Add explicit test: put with 1s TTL, sleep 2s, get returns null even before TTL sweeper fires.
|
||||
|
||||
### Finding #8: Dual-write divergence detection is one-shot
|
||||
|
||||
**File:** `phase-04-dual-write-wrappers.md` §Functional + phase-05 verifier is a manual run
|
||||
|
||||
Plan: secondary write fails → log + continue. Reconciliation = phase-05 backfill (skip-if-exists).
|
||||
|
||||
Issue: Phase-05 runs ONCE (or daily during soak). Between phase-05 runs, divergences accumulate. Backfill `$setOnInsert` only fills missing docs — it cannot catch a CF KV write that succeeded with newer state where a previous secondary write to Mongo failed (Mongo has stale OR no doc).
|
||||
|
||||
**Recommendation:** Add a recurring drift-verifier cron: every hour, sample N keys per module, compare hashes, log mismatches. OR: change failed-secondary-write handling to push the failed key + value onto a retry queue (small KV list at `__retry:mongo-failed`). A separate worker or cron drains the queue with retry. This decouples user-facing latency from secondary durability.
|
||||
|
||||
### Finding #9: Stage 2 single-region, no backups
|
||||
|
||||
**File:** `phase-07-cutover-and-decommission.md` Stage 2 + phase-01 Key Insight 5
|
||||
|
||||
Stage 2 keeps `DUAL_WRITE=0` for 7 days. M0 has NO backups + single-region. Atlas free-tier has ~99% SLA, not 99.9%. A 7-day window with M0 outage = data loss.
|
||||
|
||||
**Recommendation:** Phase-07 Stage 2 step 9.5 (insert): "Run `wrangler d1 export miti99bot-db` to local file at start of Stage 2 (snapshot 1) and end of Stage 2 (snapshot 2). Document in cutover-log." Plus: leave `DUAL_WRITE=1` for the first 24h of Stage 2 (overlap window), then flip on day 1 → soak 6 more days. Risk row in phase-07 explicitly addresses single-region M0.
|
||||
|
||||
### Finding #10: Telemetry helper has reference error
|
||||
|
||||
**File:** `phase-06-staged-deploy-and-soak.md:38-49`
|
||||
|
||||
```js
|
||||
export function startTiming(env, cmd) {
|
||||
const t0 = Date.now();
|
||||
return {
|
||||
mark(label) { /* push {label, dt} into local array */ },
|
||||
end(extra = {}) {
|
||||
const total = Date.now() - t0;
|
||||
console.log(JSON.stringify({ event: "cmd_timing", cmd, total, ...extra, marks }));
|
||||
}
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
`marks` is referenced but never declared. `mark(label)` is a no-op stub.
|
||||
|
||||
**Recommendation:** Fix snippet — `const marks = []` declared at top, `mark(label) { marks.push({label, dt: Date.now() - t0}); }`. This is a copy-paste-into-prod risk.
|
||||
|
||||
### Finding #11: Cold-start detection threshold
|
||||
|
||||
**File:** `phase-06-staged-deploy-and-soak.md:53` "Cold == age < 200ms"
|
||||
|
||||
Cold isolate boot = ~50ms.
|
||||
First Mongo connect = ~1500ms.
|
||||
By the time the FIRST request handler logs `isolate_age_ms`, age is already >200ms.
|
||||
|
||||
So `isolate_age_ms < 200ms` matches NEVER. All cold requests get bucketed as warm. Phase-06 measurement is broken.
|
||||
|
||||
**Recommendation:**
|
||||
```js
|
||||
let isFirstRequestInIsolate = true;
|
||||
// inside handler:
|
||||
const isCold = isFirstRequestInIsolate;
|
||||
isFirstRequestInIsolate = false;
|
||||
console.log({ event: "request", cmd, cold: isCold, total_ms });
|
||||
```
|
||||
|
||||
`isolate_age_ms` is still useful for histogram ("how warm is warm?") but cold/warm bucketing must use the boolean.
|
||||
|
||||
### Finding #12: package.json deploy chain references migrate.js after deletion
|
||||
|
||||
**File:** `package.json:12` and `phase-07-cutover-and-decommission.md:19`
|
||||
|
||||
```json
|
||||
"deploy": "npm run build && wrangler deploy && npm run db:migrate && npm run register",
|
||||
"db:migrate": "node scripts/migrate.js",
|
||||
```
|
||||
|
||||
Phase-07 step 19 deletes `scripts/migrate.js`. After deletion, `npm run deploy` fails: `node scripts/migrate.js` → ENOENT.
|
||||
|
||||
**Recommendation:** Phase-07 step 17 update: edit `package.json` to remove `&& npm run db:migrate` from `deploy` chain, and remove the `db:migrate` script entry. This is a 2-line edit but easily forgotten.
|
||||
|
||||
### Finding #13: Trading ID type drift on backfill
|
||||
|
||||
**File:** `phase-05-backfill-scripts.md` step 5 + `phase-03-mongo-sql-store.md:24`
|
||||
|
||||
D1 trade rows have `id: 1, 2, 3, ...`. After backfill, Mongo has `_id: ObjectId, id: <hex>`. The original integer 1, 2, 3 are gone. Any historical telemetry, logs, or external dashboards referencing trade IDs are orphaned.
|
||||
|
||||
**Recommendation:** During backfill, set both:
|
||||
```js
|
||||
{ _id: ObjectId(), legacy_id: row.id, ... }
|
||||
```
|
||||
And in `MongoSqlStore.adapt()`, expose `id` as the new ObjectId hex BUT keep `legacy_id` so support queries can still find historical trades. Document in phase-08 changelog: pre-cutover trade IDs preserved as `legacy_id`.
|
||||
|
||||
### Finding #14: nodejs_compat_v2 surface change
|
||||
|
||||
**File:** `phase-01-atlas-setup.md` Risk row 3
|
||||
|
||||
`nodejs_compat` (v1) and `nodejs_compat_v2` are NOT additive — they're alternatives. Switching changes the runtime's process/buffer/streams globals.
|
||||
|
||||
Codebase uses (search needed): `crypto.timingSafeEqual` (Phase 05), maybe `Buffer`, maybe `process.env`. The CLAUDE.md says register script uses `--env-file-if-exists` (Node 20.6+), but that's the deploy-time Node, not the worker. Worker doesn't use `process.env` (good).
|
||||
|
||||
**Recommendation:** Phase-01 step 7 add: `grep -rn "import.*node:\|process\.env\|Buffer\." src/` and document every Node API touched. Test each on `wrangler dev` before phase-02 begins. Add a section in `using-mongodb.md` listing surface dependencies.
|
||||
|
||||
### Finding #15: Admin route timing and ordering
|
||||
|
||||
**File:** `phase-05-backfill-scripts.md` step 2 + step 3
|
||||
|
||||
`crypto.timingSafeEqual` is called only after the route matches. The route matching itself (string compare path, header presence check) leaks timing. Plan does NOT specify:
|
||||
- Where in the request flow `/__admin/*` mounts (before or after `/webhook`?)
|
||||
- What the unauth response body is (could leak existence)
|
||||
- Rate limiting
|
||||
|
||||
**Recommendation:**
|
||||
- Mount admin router BEFORE `/webhook` to ensure no Telegram secret check is short-circuited.
|
||||
- 401 response: empty body, generic `WWW-Authenticate: Bearer` header.
|
||||
- Rate limit via KV counter (5 req/min/IP) — minor, but cheap.
|
||||
- `crypto.timingSafeEqual` requires fixed-length input — pad short tokens or reject early with constant-time check on length.
|
||||
- Add `X-Robots-Tag: noindex, nofollow` to prevent search crawlers.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Bundle-size measurement: has anyone (the user) actually deployed `mongodb` v6.7 to a CF Worker on Free plan and confirmed it fits under 3 MiB compressed? Researcher report says "~4-5 MB compressed". This contradiction must be resolved before phase-01 starts. Recommend: do a smoke deploy of a minimal worker with `import { MongoClient } from "mongodb"` and capture `wrangler deploy --dry-run` size output. If >3 MiB on Free plan, abort the entire plan.
|
||||
|
||||
2. Existing `tests/db/create-sql-store.test.js:48-52` requires `last_row_id` to be present and equality-checkable. Decision: phase-03 returns `0` (number) for inserts, OR refactors the test? Plan does not say.
|
||||
|
||||
3. Misc module's existing daily cron: does it currently write KV? If yes, post-cutover it writes Mongo and prevents auto-pause. If no, M0 auto-pauses after 30 days idle. Phase-08 §Unresolved Q4 leaves this open. Action item: confirm misc cron handler writes any data on each invocation.
|
||||
|
||||
4. `wrangler dev` Mongo support: phase-01 step 9 implies `wrangler dev` works against Atlas via Workers Sockets API. This was added in 2025 and the researcher confirms — but real-world reliability matters. Recommend: a Phase 02 sub-step "test full CRUD against real Atlas from `wrangler dev` for ≥10 min". If it stalls / drops connections, dev experience is broken even if prod works.
|
||||
|
||||
5. M0 pause vs cron heartbeat: phase-08 §Unresolved Q4. Action: edit `misc/index.js` (or add a new `keep-alive` cron if misc doesn't write). Effort: 5 min. Should not be unresolved at this stage.
|
||||
|
||||
6. Stage-1 vs Stage-2 cutover decision criteria: Phase-07 lists soak duration but not abort criteria for Stage 1. What metric triggers Stage-1 rollback? "Daily verifier PASS" but `verify-mongo-parity.js` runs against a 7-day stable Mongo state where new writes happen — is there a definition of "PASS" that distinguishes "drift due to live traffic" from "drift due to bug"?
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
**Summary:** Plan is broadly sound and rollback-safe up to Phase 06, but has 5 blockers (last_row_id contract, stubMongo type, retention dispatcher ambiguity, missing reverse-backfill script, untested bundle size) and 10 high-severity gaps (TTL stale reads, divergence-detection one-shot, telemetry bug, cold-start detection threshold, package.json drift, ID type drift, etc.) that will cause implementation pain or silent prod regressions if not addressed before phase-01 starts. Recommend planner pass before code lands.
|
||||
@@ -1,403 +0,0 @@
|
||||
# Whole-project architecture & code-quality review
|
||||
|
||||
**Date:** 2026-05-09
|
||||
**Scope:** every Go file in `cmd/` + `internal/`, plus `Dockerfile`, `Makefile`, `.github/workflows/ci.yml`, `go.mod`. Phases 02–06a landed; per-phase reports already cover their sub-cooks. This pass focuses on **cross-cutting** issues those sub-cooks could not see.
|
||||
**Build/test status at review time:** `go vet ./...` clean; `go test -race -count=1 ./...` clean (10 pkgs). Local toolchain `go1.26.2`.
|
||||
**Skipped (already in prior reports):** winRate truncation (5c), defaultRNG race (5b), info nil-deref (5a), %q-vs-JS (6a), per-module renderBoard tests (5c-M1), loldle helper extraction (6a-Medium), unbounded keylock map size (keylock package doc), all C/H from phase 02-03 review (every fix landed).
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Two real shipping blockers no phase report could catch because they cross phase boundaries:
|
||||
|
||||
1. **Dockerfile build will fail** — `golang:1.23-alpine` cannot satisfy `go.mod`'s `go 1.25.0` directive without a `toolchain` line. Phase-02's build was logged as green when the local toolchain was 1.23-compatible; the project bumped go.mod since.
|
||||
2. **Three `update.Message` nil-derefs** in shipped modules (`misc.go:54,79,94`, `util/help.go:100`) — same shape as the /info bug Phase 5a fixed, just in commands that 5a's review didn't touch. JS source has the same latency; Go panics on nil deref where JS just throws and the framework swallows.
|
||||
|
||||
Plus a **drift cluster** of three near-identical helpers across four modules (subjectFor, argAfterCommand, nowMillis, normalize, reply, replyHTML) — the 6a report flagged this for 6b prep, but the **subjectFor variants are not byte-equivalent** (wordle vs loldle differ on the channel-with-no-From edge). Either drift will produce a real divergence the next time someone "fixes" only one copy, or 6b extracts now and the drift goes away.
|
||||
|
||||
The rest is hygiene.
|
||||
|
||||
---
|
||||
|
||||
## Critical
|
||||
|
||||
### C1 — Dockerfile builder image is older than go.mod's `go` directive
|
||||
|
||||
**Files:** `Dockerfile:1`, `go.mod:3`
|
||||
|
||||
```
|
||||
Dockerfile: FROM golang:1.23-alpine AS builder
|
||||
go.mod: go 1.25.0
|
||||
```
|
||||
|
||||
Go's `go.mod` `go N` directive is a hard floor: the toolchain refuses to build with `go.mod requires go >= 1.25.0`. Without a `toolchain` line in go.mod, the 1.23 image cannot auto-download 1.25 (auto-toolchain only fires when `toolchain go1.X.Y` is declared). This means **every `docker build`** today fails — and the CI image (`actions/setup-go` with `go-version: '1.23'` in `.github/workflows/ci.yml:19`) will fail too the next time it runs.
|
||||
|
||||
Why no prior review caught it: Phase 02 review pinned the Dockerfile contents at a time when go.mod said `go 1.23`; whoever bumped to 1.25 did not also bump the Dockerfile / CI matrix.
|
||||
|
||||
**Fixes (pick one):**
|
||||
|
||||
a. Bump `Dockerfile` builder to `golang:1.25-alpine` and `.github/workflows/ci.yml` `go-version` to `'1.25'`. Cleanest.
|
||||
b. Lower `go.mod` to `go 1.23` (or whatever version is actually required by the dependencies — `cloud.google.com/go/firestore v1.22.0` only needs 1.22+).
|
||||
c. Add `toolchain go1.25.0` to go.mod and rely on auto-download. Slowest cold builds in CI; not recommended.
|
||||
|
||||
Recommend (a). Verify nothing in the codebase actually needs 1.25 features (no `min`/`max`/`clear`/etc. usage I could spot, so (b) is also safe).
|
||||
|
||||
### C2 — Three `update.Message` nil-derefs in shipped modules
|
||||
|
||||
**Files:**
|
||||
- `internal/modules/misc/misc.go:54` (/ping)
|
||||
- `internal/modules/misc/misc.go:79` (/mstats)
|
||||
- `internal/modules/misc/misc.go:94` (/fortytwo)
|
||||
- `internal/modules/util/help.go:100` (/help)
|
||||
|
||||
Every one writes `update.Message.Chat.ID` without first checking `update.Message != nil`. Phase 5a's review caught the same shape in `info.go:36` and the fix landed there (`if msg == nil { return nil }`). The pattern was not propagated. With `bot.HandlerTypeMessageText` + `bot.MatchTypeCommand` the dispatcher only fires on text-message updates today, so `update.Message` is non-nil in practice — but:
|
||||
|
||||
- The infosec-style "untrusted external input" boundary lives at the webhook decoder. A malformed Telegram payload that decodes into a partially-populated `models.Update` still satisfies `MatchType` matching at the library level but can leave `Message == nil`. Library-level guarantees here are thin.
|
||||
- Future visibility-aware dispatch (callbacks, edited-messages) will route through different handler types, and the same factory-supplied `Command.Handler` may be reused. The /info fix already captured this in a comment; misc and util/help did not get the same hardening.
|
||||
- Defensive cost is one line per handler.
|
||||
|
||||
**Fix:** add `if update.Message == nil { return nil }` (or equivalent guard) at the top of each handler. Cheaper than a test, and matches the pattern Phase 5a established.
|
||||
|
||||
---
|
||||
|
||||
## Major
|
||||
|
||||
### J1 — Helper-function drift across four modules; subjectFor variants are NOT byte-equivalent
|
||||
|
||||
**Files:**
|
||||
- `internal/modules/wordle/handlers.go:30-47` — `subjectFor`
|
||||
- `internal/modules/loldle/handlers.go:33-46` — `subjectFor`
|
||||
- `internal/modules/loldleemoji/handlers.go:35-48` — `subjectFor`
|
||||
|
||||
Phase 6a flagged "extract `normalize`, `subjectFor`, `argAfterCommand`, `findChampion` for 6b prep". The flag was right but undersold the urgency:
|
||||
|
||||
```go
|
||||
// wordle (handlers.go:30-47)
|
||||
switch msg.Chat.Type {
|
||||
case models.ChatTypePrivate:
|
||||
if msg.From != nil { return strconv.FormatInt(msg.From.ID, 10) }
|
||||
case models.ChatTypeGroup, models.ChatTypeSupergroup:
|
||||
return strconv.FormatInt(msg.Chat.ID, 10)
|
||||
default:
|
||||
if msg.From != nil { return strconv.FormatInt(msg.From.ID, 10) }
|
||||
}
|
||||
return ""
|
||||
|
||||
// loldle and loldleemoji (handlers.go:33-46 / 35-48)
|
||||
switch msg.Chat.Type {
|
||||
case models.ChatTypeGroup, models.ChatTypeSupergroup:
|
||||
return strconv.FormatInt(msg.Chat.ID, 10)
|
||||
default:
|
||||
if msg.From != nil { return strconv.FormatInt(msg.From.ID, 10) }
|
||||
}
|
||||
return ""
|
||||
```
|
||||
|
||||
Functionally equivalent **today** because Telegram populates `From` on every private DM. But:
|
||||
|
||||
- For `ChatTypeChannel` with no `From` (anonymous channel post), wordle returns `""` and loldle/emoji also return `""` — same. ✓
|
||||
- For `ChatTypePrivate` with `From == nil` (which Telegram never does, but the type system allows), **wordle** returns `""` and **loldle/emoji** also return `""` via the default branch. Identical.
|
||||
|
||||
The risk isn't current behavior; it's that **someone editing one copy to fix a bug will forget to edit the other two**. Phase 5c found exactly this with `winRate` (truncation bug existed in both wordle and loldle; the 5b review fixed wordle and missed loldle, then 5c had to clean up).
|
||||
|
||||
Other drift in the same files:
|
||||
- `argAfterCommand` is byte-identical across wordle / loldle / loldleemoji (3 copies).
|
||||
- `nowMillis` is byte-identical across wordle / loldle / loldleemoji (3 copies).
|
||||
- `reply(ctx, b, chatID, text)` (loldle/loldleemoji) vs `reply(ctx, b, msg, text)` (wordle) — slightly different signatures; not interchangeable but the implementation body is duplicated.
|
||||
- `replyHTML` is byte-identical across loldle / loldleemoji.
|
||||
- `normalize` is byte-identical across loldle / loldleemoji (and a related `normalizeWord` in wordle that drops digit support — different alphabet, intentional).
|
||||
- `findChampion` shape is identical across loldle / loldleemoji modulo type names.
|
||||
|
||||
**Recommendation:** extract `internal/modules/util/chathelper` (or `internal/champname` for the loldle-specific normalize+findChampion pair). Do it as the **first** commit of phase 6b before any new variant lands; then 6b's quote/ability/splash variants pick up the helpers from a single source and the drift problem disappears. Cost: ~80 LOC of helper + 4 import line changes per module.
|
||||
|
||||
A **shared `WinRate(wins, played int) int`** helper should be part of the same extraction — currently 3 copies (wordle/loldle/loldleemoji) all using `math.Round` correctly today, but one drift opportunity per port.
|
||||
|
||||
### J2 — Logging is `log.Printf` everywhere; Cloud Logging will not parse it
|
||||
|
||||
**Files:** `cmd/server/main.go` (×9), `internal/server/router.go:77,86`, `internal/modules/dispatcher.go:24`, `internal/modules/misc/misc.go:51`. 18 call sites total in non-test code.
|
||||
|
||||
Cloud Run forwards `stdout` to Cloud Logging line-by-line. Cloud Logging treats each line as a record but only **parses structured JSON** for severity, trace correlation, and label-based filtering. `log.Printf("cron %s failed: %v", ...)` becomes a single text payload with severity DEFAULT — every alert filter / dashboard / SLO query will need a regex.
|
||||
|
||||
Phase 11 plans "Cloud Logging structured JSON" so this is on the roadmap. The concern is: **every call site added until Phase 11 is debt** that has to be migrated. With Phase 11 bumping into Phase 6b/7/8 worth of new modules, the migration target is moving.
|
||||
|
||||
**Recommendation:** introduce a tiny `internal/log` (or `internal/obs`) package now with a `WithFields(...)` API that emits JSON lines (Go 1.21+ `slog.JSONHandler` is stdlib, zero deps), and route all current call sites through it. Future modules pick up the structured form for free. ~30 LOC + 18 mechanical edits. Could also be Phase 11's first commit — but the longer it waits, the more sites to mechanically rewrite.
|
||||
|
||||
Phase 5a's `misc.go:51` `log.Printf("misc /ping: putJSON failed: %v", err)` is a good motivating example: `module=misc command=ping op=putJSON err=...` as JSON fields makes the eventual error rate dashboard a one-line query; the current shape needs a regex.
|
||||
|
||||
Bonus: the `log.Printf("cron name=%s", name)` at `router.go:77` is **PII-adjacent** — `name` is operator-controlled (cron names are validated `^[a-z0-9_]{1,32}$`), so no real injection risk, but Phase 11 plan says "Cloud Logging" and the only thing standing between us and a `log_entry_payload_size_too_large` is hand-discipline.
|
||||
|
||||
### J3 — Cron handler chain has no log-injection guard for the **error** branch
|
||||
|
||||
**Files:** `internal/server/router.go:86`
|
||||
|
||||
`log.Printf("cron %s failed: %v", name, err)` — `name` is regex-validated so it's safe. But `err` is whatever the module returned, which is **module-controlled and may include user input**. A loldle module today does `fmt.Errorf("loldle saveGame: %w", err)` then chains downward; if a future module ever does `fmt.Errorf("user input was %q", argAfterCommand(msg.Text))` and that error bubbles up, the log line gets a newline-bearing user string. CWE-117 (log injection) class.
|
||||
|
||||
This is theoretical today (no current handler error-wraps user input). But the bot will only get more user-input-touching modules from here. Phase 11's structured-JSON conversion (J2) makes this naturally safe (JSON encodes newlines as `\n`).
|
||||
|
||||
Stopgap until then: `log.Printf("cron %s failed: %s", name, strings.ReplaceAll(err.Error(), "\n", " "))` — ugly, but bounds the damage. Or **defer to J2** and fix structurally.
|
||||
|
||||
### J4 — `update.Message.Chat.ID` is a soft trust boundary that the codebase doesn't enforce
|
||||
|
||||
Same shape as C2 but a finer point: `models.Update` is decoded directly from the webhook body via `json.NewDecoder(r.Body).Decode(...)`. We trust Telegram's TLS-authenticated webhook (X-Telegram-Bot-Api-Secret-Token validates the *delivery*, not the *payload* — anyone with the secret can send any payload). With the secret leaked, a forged request with `Message.Chat.ID = -<your-target-chat>` would route a guess into another user's stats / send a sticker into someone else's group. Practical risk is low (secret is only-on-Telegram-server today), but defense-in-depth is cheap:
|
||||
|
||||
- Validate `update.UpdateID > 0` (Telegram always positive).
|
||||
- Validate `update.Message.Chat.ID` non-zero before trusting it.
|
||||
- Reject `Message.Date` more than ~24h old (replay window).
|
||||
|
||||
None of these are urgent. Track in Phase 11 alongside structured logging.
|
||||
|
||||
---
|
||||
|
||||
## Medium
|
||||
|
||||
### M1 — `bot.New` may return error for transient network reasons; main.go treats it as fatal
|
||||
|
||||
**File:** `cmd/server/main.go:69-72`
|
||||
|
||||
```go
|
||||
b, err := telegram.NewBot(cfg.TelegramBotToken)
|
||||
if err != nil { log.Fatalf("telegram bot init: %v", err) }
|
||||
```
|
||||
|
||||
`telegram.NewBot` passes `WithSkipGetMe()` so the only thing left to fail is option-application — which is in-process and deterministic. Today `err` is always nil after argument validation. Slight surprise that `bot.New` can return `error` at all in this path. Not actionable; flagging because future contributors might switch back to the GetMe-blocking variant and assume the fail-fast path handles transients gracefully (it doesn't; Cloud Run will hot-loop restart the container).
|
||||
|
||||
Fix: add a comment explaining "with WithSkipGetMe + WithNotAsyncHandlers, bot.New does not perform I/O; this error is unreachable in practice" — or leave alone.
|
||||
|
||||
### M2 — Synchronous webhook dispatch holds Cloud Run instance for full handler duration
|
||||
|
||||
**Files:** `internal/telegram/webhook.go:56-58`, `client.go:19`
|
||||
|
||||
`bot.WithNotAsyncHandlers()` makes handler dispatch synchronous (good — solves H2 from the Phase 02 review). But it means a slow handler (e.g. a `/loldle` first response that does 3 Firestore reads + 1 sticker send) holds the webhook open for hundreds of ms — and Cloud Run min-instance=0 default + 1-concurrent-request-per-instance budget means a queue forms during traffic spikes.
|
||||
|
||||
Numbers from real handlers I reviewed:
|
||||
- `/loldle` first guess: 3 Firestore reads (game, stats, config), 2 Firestore writes (game saveGame, stats put), 1 sticker send, 1 message send. ~250-500ms P95 on warm instance.
|
||||
- `/wordle` similar.
|
||||
|
||||
The `handlerTimeout = 10 * time.Second` cap (`webhook.go:26`) is the right shape but Telegram retries after 60s of no 2xx, so a 10s cap means **on a 10s-stuck handler the user gets a duplicate update (Telegram retry) AND the original eventually completes** — a duplicate-write window during the retry / completion overlap. The keylock per-subject mutex protects same-subject writes, but the **second invocation enters the handler** first and sees state from the abandoned first invocation, possibly skipping the user's intended action.
|
||||
|
||||
**Recommendation:** lower handlerTimeout to 8s (gives 2s margin before Telegram retry), and document the retry / duplicate-update pattern as a known limitation. Or: switch to async-with-detached-context per Phase 02 review's H2 alternative ("`context.WithoutCancel(r.Context())` + own goroutine"). Not v1; phase 11 work.
|
||||
|
||||
### M3 — `Cron` handler error mapping conflates "handler failed" with "cron not found"
|
||||
|
||||
**File:** `internal/server/router.go:81-89`
|
||||
|
||||
```go
|
||||
if err := modules.DispatchScheduled(...); err != nil {
|
||||
if errors.Is(err, modules.ErrCronNotFound) { http.NotFound(...); return }
|
||||
log.Printf("cron %s failed: %v", name, err)
|
||||
http.Error(w, "cron failed", http.StatusInternalServerError)
|
||||
}
|
||||
```
|
||||
|
||||
500 on handler error is correct behavior (Cloud Scheduler retries 5xx, doesn't retry 4xx). But Cloud Scheduler also has a "max retry attempts" cap that, when exceeded, sends to a dead-letter; with bare 500 there's no way for a handler to signal "do not retry, I poisoned this". Today no handler has this need, but a "the user's quota is exhausted" handler would benefit from returning 4xx-class.
|
||||
|
||||
YAGNI today. Worth a `Cron.RetryPolicy` field (or sentinel error like `modules.ErrCronDoNotRetry`) when the first such handler lands. Document in Phase 09's plan.
|
||||
|
||||
### M4 — `Module.Name` can be silently overwritten without warning
|
||||
|
||||
**File:** `internal/modules/registry.go:119`
|
||||
|
||||
```go
|
||||
mod := factory(moduleDeps)
|
||||
mod.Name = name // enforce: module name is its registry key
|
||||
```
|
||||
|
||||
Phase 02-03 review flagged this as L1. Still here. Today: harmless, factories happen to either set `Name: name` themselves or leave it blank. But **if a module factory sets `Name: "different"` for legitimate reasons** (refactor, copy-paste, dynamic name override), that intent is **silently discarded** with no log line. Suggest:
|
||||
|
||||
```go
|
||||
if mod.Name != "" && mod.Name != name {
|
||||
return nil, fmt.Errorf("module factory for %q returned mismatched Name=%q", name, mod.Name)
|
||||
}
|
||||
mod.Name = name
|
||||
```
|
||||
|
||||
Or drop `Module.Name` entirely (registry already keys by module name; the field is redundant). Either is more honest than "silently overwrite".
|
||||
|
||||
### M5 — `firestore_kv.go` → 216 LOC; `loldle/handlers.go` → 334 LOC; `wordle/handlers.go` → 284 LOC; `loldleemoji/handlers.go` → 269 LOC; `loldle/compare.go` → 253 LOC; `cmd/server/main.go` → 211 LOC
|
||||
|
||||
Project rule (CLAUDE.md "Consider Modularization"): files >200 LOC should be considered for splitting. Six files exceed.
|
||||
|
||||
| File | LOC | Suggested split |
|
||||
|------|-----|-----------------|
|
||||
| `loldle/handlers.go` | 334 | Each `handle*` to its own file (handle_loldle.go, handle_giveup.go, handle_stats.go, handle_setmax.go) — most of the 334 is one handler each; helpers like `subjectFor` / `argAfterCommand` move to a sibling file or to the shared package per J1. |
|
||||
| `wordle/handlers.go` | 284 | Same — handle_wordle, handle_new, handle_giveup, handle_stats. |
|
||||
| `loldleemoji/handlers.go` | 269 | Same. |
|
||||
| `loldle/compare.go` | 253 | Year/multi/exact compare functions split into `compare_year.go`, `compare_multi.go` — already cleanly separated by attr type within the file. |
|
||||
| `firestore_kv.go` | 216 | Validate / prefixSuccessor → `firestore_keys.go`. List / Get / Put / Delete stay together. ~60 LOC migration. |
|
||||
| `cmd/server/main.go` | 211 | `loadConfig` + `splitCSV` + `envForModules` → `config.go`. `buildProvider` → already a candidate for `provider.go`. main() shrinks to ~80 LOC of orchestration. |
|
||||
|
||||
This is a guideline, not a hard rule, and J1's helper-extraction will incidentally pull ~50 LOC out of three handlers.go files — making M5 mostly self-resolving. Recommend doing M5 **after** J1, since J1 mechanically dictates which lines move out first.
|
||||
|
||||
### M6 — `loldle/state.go` `getOrInitGame` is identical in shape to `loldleemoji/state.go` `getOrInitGame`; almost identical to `wordle/state.go` `getOrInit`
|
||||
|
||||
Three copies of "load existing or start fresh" with tiny variations in:
|
||||
- whether maxGuesses is dynamic per subject (loldle/emoji yes, wordle no).
|
||||
- whether StartedAt initialises to nil (loldle/emoji) or now-millis (wordle).
|
||||
|
||||
Each module's gameState shape differs enough that a shared interface is awkward, but the **pattern** is so repetitive it screams for extraction. Phase 11 problem; tracking only.
|
||||
|
||||
### M7 — `MemoryProvider.Base()` is production code surface area
|
||||
|
||||
**File:** `internal/storage/kv_provider.go:35`
|
||||
|
||||
Phase 04 review M4 flagged this. Still here. The method is documented "test-only" but is on the public production type. A future module can `provider.(*storage.MemoryProvider).Base()` and bypass module isolation — silently. The phase-04 review suggested moving to a `_test.go` build-tagged file or a `storagetest` helper package. Still recommended; cheap.
|
||||
|
||||
---
|
||||
|
||||
## Minor
|
||||
|
||||
### N1 — `Registry` struct has 5 unexported fields used only internally; `AllCommands` is the lone exported map
|
||||
|
||||
`AllCommands` exposed because `dispatcher.Install` needs to iterate it. `publicCmds`/`protected`/`private` are accessed only via getter methods (`PublicCommands()` etc.) which sort+copy. Inconsistency: the dispatcher could equally use a getter. Cosmetic only; small refactor would close out the "callers can mutate AllCommands post-build" vector that Phase 02 review M7 flagged.
|
||||
|
||||
### N2 — `FirestoreProvider.For` accepts ANY moduleName without validation
|
||||
|
||||
**File:** `internal/storage/firestore_provider.go:22`
|
||||
|
||||
The comment says "Module names are validated by modules.Build before reaching here, so we don't sanitize again." True — but the `KVProvider` interface advertises that anyone may construct one. A test using `provider.For("__reserved__")` (Firestore-banned collection name) gets an error from gRPC, not a clean `validateCollection` rejection. Cheap to add a one-line check in `For`. Not blocking.
|
||||
|
||||
### N3 — `gameTTLSeconds` (wordle) is unused; flag from Phase 5b L3 still present
|
||||
|
||||
**File:** `internal/modules/wordle/state.go:18`
|
||||
|
||||
Constant + comment have stale-doc smell. Phase 11 was supposed to add a TTL cron; until then, delete the constant or move the TTL note to a Markdown doc. Compiler doesn't complain (Go ignores unused package-level constants), so it just sits there.
|
||||
|
||||
### N4 — `loldle/loldle.go:14` `MaxGuesses = 8` is exported but only used internally; same in `loldleemoji/state.go:15` `MaxGuesses = 5`. Same for `MaxGuessesCap`.
|
||||
|
||||
Capitalised constants in domain-private packages with no out-of-package callers. Either lowercase them or move to docs. Style nit.
|
||||
|
||||
### N5 — `loldle/state.go:30` and `loldleemoji/state.go:23-26` define `gameState` (lowercase) — unexported. But `loldle/loldle.go:14` defines `MaxGuesses` exported. Mixed casing within the same package suggests no hard convention. Pick one. Style nit.
|
||||
|
||||
### N6 — `pickDaily` (wordle/daily.go:34) is unused by handlers but kept "for parity"
|
||||
|
||||
Effectively dead code with a passing test. Either wire into a /wordle_daily handler (Phase 06+ work) or delete + delete the test. Same shape as N3.
|
||||
|
||||
### N7 — `internal/modules/modules.go` is a 7-line empty-package comment file
|
||||
|
||||
Vestigial. Could be folded into `module.go`'s package doc. Or kept. Truly cosmetic.
|
||||
|
||||
### N8 — `cmd/server/main.go:184` has misaligned struct-init padding
|
||||
|
||||
```go
|
||||
ModuleEnv: envForModules(envMap),
|
||||
```
|
||||
vs
|
||||
```go
|
||||
TelegramBotToken: envMap["TELEGRAM_BOT_TOKEN"],
|
||||
```
|
||||
|
||||
`gofmt` should rewrite this on save. Probably fine but failing-rule-scope nit.
|
||||
|
||||
---
|
||||
|
||||
## Architectural observations
|
||||
|
||||
### Package boundaries: clean
|
||||
|
||||
- `internal/keylock` is a generic primitive, correctly placed at top-level peer to `storage` / `telegram` / `server`.
|
||||
- `internal/storage` is cleanly factored: `KVStore` interface + two impls + a prefix wrapper. `KVProvider` abstraction is the right shape (modules see `KVStore`, not the provider).
|
||||
- `internal/modules` framework is clean: registry holds maps, validate gates inputs, dispatchers tie to bot/HTTP. Modules are leaves.
|
||||
- `cmd/server` is the composition root and owns the catalog (`factories()`). Comment in `internal/modules/modules.go` correctly explains why the catalog cannot live inside `internal/modules`.
|
||||
|
||||
### Trust boundaries: mostly enforced
|
||||
|
||||
| Boundary | Validation | Gap |
|
||||
|----------|------------|-----|
|
||||
| `MODULES` env → registry | `moduleNameRe` regex, dedupe, factory lookup | None |
|
||||
| Webhook body → handler | `MaxBytesReader(1MiB)`, `json.Decoder` | C2 (nil-deref past decode) |
|
||||
| Webhook auth | constant-time secret compare | None |
|
||||
| Cron route → dispatch | `cronNameRe` regex + constant-time secret | None |
|
||||
| Cron error → log | direct `%v` of internal error | J3 (potential CRLF injection theoretical) |
|
||||
| Module name → KV provider | `moduleNameRe` regex (no `:`) | N2 (Firestore provider doesn't double-check) |
|
||||
| KV key → Firestore | `validateKey` thorough | None (Phase 04 review confirmed) |
|
||||
| User input → reply text | `html.EscapeString` everywhere I checked | None |
|
||||
|
||||
### Concurrency: clean across the board
|
||||
|
||||
- All mutating handlers use `defer s.locks.Acquire(subject)()`.
|
||||
- `math/rand` package-level functions are mutex-protected (used everywhere).
|
||||
- `keylock.Map` uses `sync.Map` correctly.
|
||||
- `Registry` is read-only post-Build by convention (documented).
|
||||
- `srv.Shutdown` waits for in-flight handlers; provider closes after shutdown.
|
||||
- Bot dispatcher is synchronous (`WithNotAsyncHandlers`), so `r.Context()` lives across handler.
|
||||
|
||||
`go test -race -count=1` clean on every package.
|
||||
|
||||
### Error propagation: mostly clean, two patterns
|
||||
|
||||
1. **Module handlers** return `error`; the dispatcher logs and discards. Phase 02-03 review M6 flagged that this is decorative. Still true. Neither metrics nor retry nor user-visible "internal error" reply hooks into the return value. For an early-stage codebase this is fine; mark for Phase 11 observability.
|
||||
|
||||
2. **Storage layer** wraps errors with `fmt.Errorf("firestore put %s/%s: %w", ...)` consistently. `errors.Is` checks against `ErrNotFound` and `ErrCronNotFound` — these are the only sentinels. Good.
|
||||
|
||||
### Configuration: clean
|
||||
|
||||
- All env vars read once in `loadConfig`. Per-module env via `Deps.Env` with explicit deny-list (`secretEnvKeys`). Deny-list is correct shape but requires manual upkeep; Phase 02 review H5's allow-list alternative is still preferable but not blocking.
|
||||
- No env var read after startup. Good.
|
||||
|
||||
### Dead/vestigial code
|
||||
|
||||
| Symbol | File | Justification |
|
||||
|--------|------|---------------|
|
||||
| `gameTTLSeconds` | `wordle/state.go:18` | Documented in Phase 5b review (L3) |
|
||||
| `pickDaily` | `wordle/daily.go:34` | Has a test; unused by handlers. (N6) |
|
||||
| `internal/modules/modules.go` | (entire file) | 7-line empty package doc. (N7) |
|
||||
| `MemoryProvider.Base()` | `kv_provider.go:35` | Test-only on production type. (M7) |
|
||||
| `Module.Name` | `module.go:55` | Overwritten by registry; never read by factory. (M4) |
|
||||
|
||||
None blocking.
|
||||
|
||||
---
|
||||
|
||||
## Tests gap (cross-cutting)
|
||||
|
||||
Per-phase reports already enumerated module-specific gaps. Cross-cutting gaps:
|
||||
|
||||
1. **Webhook handler integration test** — phase-02 review test plan #1 was filed; a `webhook_test.go` file exists. Verified it covers method/secret/decode paths. Confirmed adequate.
|
||||
2. **Cron handler integration test** — `router_test.go` exists. Verified it covers method/secret/cronNameRe/dispatch paths.
|
||||
3. **No CI matrix entry for `make build`** — `.github/workflows/ci.yml` only does `go vet`, `go test`, `go build`. Doesn't `docker build` from CI. Combined with C1, that's why no one noticed the Dockerfile/go.mod mismatch. **Add `- run: docker build -t miti99bot-go .` to CI**. ~3 lines.
|
||||
4. **No emulator-gated tests in CI** — Phase 04 review L1/L3 flagged this. Makefile `test-emulator` exists; CI only runs the no-emulator subset. Acceptable for now (emulator setup adds 30-60s to CI run); track for Phase 10/11.
|
||||
|
||||
---
|
||||
|
||||
## Recommended action order
|
||||
|
||||
| # | Severity | Action | Effort |
|
||||
|---|----------|--------|--------|
|
||||
| 1 | C1 | Bump Dockerfile + CI to Go 1.25 (or lower go.mod to 1.23) | 5min |
|
||||
| 2 | C1 | Add `docker build` step to ci.yml so this never recurs silently | 5min |
|
||||
| 3 | C2 | Add `if update.Message == nil { return nil }` to misc + util/help handlers | 10min |
|
||||
| 4 | J1 | Extract shared helpers to `internal/champname` (or similar) as the FIRST commit of Phase 6b | 1-2h |
|
||||
| 5 | J2 | Introduce `internal/log` with slog.JSONHandler; rewire 18 call sites | 2-3h, or defer to Phase 11 |
|
||||
| 6 | M5 | Split files >200 LOC into per-handler / per-concern files (after J1) | 1-2h |
|
||||
| 7 | M4/M7/N* | Hygiene pass — Module.Name guard, Base() to test-tag, dead-code cleanup | 1h |
|
||||
|
||||
Items 1-3 are blockers for the next deploy / merge. The rest can ride along Phase 6b/11 as natural cleanup.
|
||||
|
||||
---
|
||||
|
||||
## Positive observations
|
||||
|
||||
- **Trust boundaries enumerated and individually fixed** with constant-time compares + regex-validated routes + strict per-key Firestore validation. Each fix has a comment explaining the threat model. Operationally friendly.
|
||||
- **`KVProvider` interface is exactly one method** — easy to mock, hard to misuse. Phase 04's review called this out; six modules later, it's still aging well.
|
||||
- **Per-subject keylock + `WithNotAsyncHandlers` together** turn the goroutine-per-update model into a sequenced-per-subject model. Equivalent guarantee to JS Workers' isolate-per-request. Documented at `keylock/keylock.go:5-12`.
|
||||
- **Wire-format tests** lock JS-parity for every persisted JSON shape (gameState, stats, roundConfig). `*int64` for nullable timestamps. Defends migration goal.
|
||||
- **Embed strategy + panic-on-bad-data** consistently applied across wordle/loldle/loldleemoji. Build-time bug surfaces at startup, not on first user.
|
||||
- **Phase reports themselves**: clear, opinionated, action-ordered. The 6a report's "extract helpers as 6b prep" foresight is exactly the kind of forward-pointing review note that makes the next reviewer's job cheaper.
|
||||
- **CI does `-race -count=1`** from day one. The single most-valuable lint a Go service can have.
|
||||
- **Defensive stripping of secrets from `Deps.Env`** via deny-list. Allow-list would be tighter but the deny-list is honest about its limits and tested.
|
||||
- **`srv.Shutdown(15s) → defer closeProvider()`** ordering is correct and the comment in Phase 04 review M5 is now in code (mostly). Graceful shutdown story is solid.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. **Q1**: Phase 11's "Cloud Logging structured JSON" is the natural home for J2/J3. Bringing it forward to Phase 6b (~2-3h) versus letting it pile up to Phase 11 (~2-4h migration cost) — **which side has the better expected-value tradeoff**? Recommend forward-port: every module added in Phase 6b/7/8 is a J2 caller-site, so the marginal cost of structured-log-from-the-start is lower.
|
||||
2. **Q2**: J1's helper extraction — into `internal/modules/util/` (already exists, but it's the /info /help /stickerid module) or a fresh `internal/modules/util/chathelper/` subpackage? Or a top-level `internal/champname` for the loldle-specific normalize+findChampion pair? Naming bikeshed; but the choice constrains how Phase 7+ AI modules will reuse the same helpers.
|
||||
3. **Q3**: C1 — bump go.mod down to 1.23 (zero feature loss) versus bump Dockerfile + CI to 1.25 (more typical, but adds Go-version churn). The codebase doesn't use any 1.24/1.25 features I found. Cheapest is to lower go.mod.
|
||||
4. **Q4**: M2 — keep `WithNotAsyncHandlers` (synchronous) or switch back to async-with-detached-context per the Phase 02 review's H2 alternative? Synchronous is simpler and matches JS-Worker semantics; async would buy back webhook return latency at the cost of a small goroutine pool. Defer until cold-start / latency telemetry from Phase 11 says one way or the other.
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
**Summary:** Architecture and concurrency are solid; two cross-phase blockers (Dockerfile/go.mod version mismatch + nil-deref pattern not propagated to misc/help) need fixing before the next merge, plus a J1 helper-drift cluster that should be extracted as Phase 6b's first commit before five more modules compound the problem.
|
||||
@@ -1,276 +0,0 @@
|
||||
# code-reviewer · whole-project security audit
|
||||
|
||||
Date: 2026-05-09
|
||||
Scope: full repo (`cmd/`, `internal/`, Dockerfile, CI, go.mod). Focus on production-readiness for Cloud Run + Firestore + Telegram webhook.
|
||||
Verdict: **DONE_WITH_CONCERNS** — no Critical issues. Several Medium items worth resolving before public deploy; one High and a few Lows.
|
||||
|
||||
Prior reports reviewed (issues already fixed not re-flagged):
|
||||
- `code-reviewer-260508-2254-phase02-03-bootstrap.md` — C1/C2/C3, H1-H7 all resolved (verified).
|
||||
- `code-reviewer-260508-2333-phase04-firestore-kv.md` — H1/H2 resolved (verified at firestore_kv.go:75-80 and main.go:135-137).
|
||||
- `code-reviewer-260509-1206-phase6a-loldle-emoji.md` — `%q` divergence still open (cosmetic).
|
||||
|
||||
---
|
||||
|
||||
## Critical
|
||||
|
||||
None.
|
||||
|
||||
---
|
||||
|
||||
## High
|
||||
|
||||
### H1 — `Deps.Env` still leaks `GOOGLE_CLOUD_PROJECT`, OAuth file paths, Cloud Run env to every module
|
||||
File: `cmd/server/main.go:188-197`, `internal/modules/module.go:72-76`.
|
||||
|
||||
`secretEnvKeys` strips only the three named tokens. Modules still receive every other env var, including: `GOOGLE_CLOUD_PROJECT`, `FIRESTORE_EMULATOR_HOST`, `GOOGLE_APPLICATION_CREDENTIALS` (path), Cloud Run-injected `K_SERVICE`/`K_REVISION`/`K_CONFIGURATION`, and any future `*_API_KEY` (Gemini, etc. — phase 7 will add `GEMINI_API_KEY` and unless that exact string lands in `secretEnvKeys` first, it goes to all modules).
|
||||
|
||||
Risk: a future module that reflects/echoes any `Deps.Env` value (debug helper, "system info" command) leaks creds. The previous reviewer's H5 fix was a denylist — denylists don't scale.
|
||||
|
||||
Recommendation: invert to allowlist.
|
||||
- Either pass nothing in `Deps.Env` (modules get truly nothing) and have each module document its own env requirements externally, OR
|
||||
- Adopt a `MODULE_<NAME>_*` convention; only matching keys flow to that module's Deps.
|
||||
|
||||
Phase 07 (Gemini) is the natural moment — once a module needs an external API key, an opt-in allowlist is the only safe pattern. Hardcoding `GEMINI_API_KEY` into `secretEnvKeys` works for one variable but not for the 2nd, 3rd, etc.
|
||||
|
||||
---
|
||||
|
||||
## Medium
|
||||
|
||||
### M1 — `MODULES` env validation rejects bad names *during* module construction; one good factory may have already executed
|
||||
File: `internal/modules/registry.go:98-153`.
|
||||
|
||||
The for-loop validates name → checks dup → looks up factory → **calls factory(moduleDeps)** → validates commands. If a later iteration fails (unknown name, dup name, validation error), modules earlier in the list have already had their factories invoked. For loldle/loldleemoji that's just slice allocation, but a future Factory that opens a file, does a DNS lookup, or holds a long-lived resource (Phase 07 Gemini client) will leak.
|
||||
|
||||
Today: low impact (factories are pure). Document with a comment that factories must be allocation-only and never block / open external resources, OR pre-validate all names + dup detection in a first pass before any factory runs.
|
||||
|
||||
### M2 — `Visibility` field is decorative; `stickerid` (private), `fortytwo` (private), `loldle_setmax` / `loldle_emoji_setmax` (private) are publicly invocable
|
||||
Files: `internal/modules/dispatcher.go:15-29`, `internal/modules/util/stickerid.go:24`, `internal/modules/misc/misc.go:90`, `internal/modules/loldle/loldle.go:36`, `internal/modules/loldleemoji/loldleemoji.go:33`.
|
||||
|
||||
`Install` registers every command with `bot.RegisterHandler` regardless of `Visibility`. The comment at `module.go:14-15` openly notes "the dispatcher does not enforce visibility today." Result: any user in any chat can:
|
||||
- `/stickerid` → echo a sticker file_id back. Information disclosure (minor — sticker IDs aren't secret, but they signal which stickers the bot owner privately uploaded).
|
||||
- `/loldle_setmax 1` → make every group's loldle round trivially solvable. Visible griefing.
|
||||
- `/loldle_emoji_setmax 1` → same.
|
||||
- `/fortytwo` → easter egg, no harm.
|
||||
|
||||
Risk: low confidentiality, real abuse for `setmax` in groups. The `setmax` commands change *group-shared* state (subjectFor() returns chat.ID for groups), so any group member can set max=1 and break the game for everyone.
|
||||
|
||||
Fix options (cheapest first):
|
||||
1. Document `setmax` is intentionally permissive and remove the `VisibilityPrivate` tag (truth-in-advertising). Acceptable if you accept the griefing.
|
||||
2. Hard-code an admin allowlist via env: `ADMIN_USER_IDS=12345,67890` and gate `Visibility >= Protected` commands at the dispatcher.
|
||||
3. For groups, check `getChatMember(chat_id, user_id).status` ∈ {creator, administrator} via the Telegram API before running protected/private commands.
|
||||
|
||||
Option 2 is the minimum production-acceptable answer. Option 3 is the right answer; it costs one extra Telegram API call per protected command invocation.
|
||||
|
||||
### M3 — `MaxBytesReader`-induced 413 returns 200 instead
|
||||
File: `internal/telegram/webhook.go:49-54`.
|
||||
|
||||
```go
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxWebhookBody)
|
||||
var update models.Update
|
||||
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
`MaxBytesReader` writes a 413 status to `w` directly when the cap is hit (this is its documented side effect). After that, `Decode` returns an error, the handler tries to call `http.Error(w, "bad request", 400)` — but headers are already sent (413), so it just appends "bad request\n" to the 413 body. Telegram sees a non-2xx and retries. Not a security hole, but operationally noisy. Test `TestWebhookHandler_RejectsOversizedBody` only checks `!= 200`, which passes for 413, so the bug is silent.
|
||||
|
||||
Also: a 1 MiB cap is generous (Telegram updates with media references are <100 KiB even with thumbnails). Consider 256 KiB.
|
||||
|
||||
Fix: tighten cap and use the documented two-stage check pattern, or accept 413 as the response code (don't shadow it with 400).
|
||||
|
||||
### M4 — Cron handler `defaultCronTimeout = 5m` runs serially against single-instance Cloud Run; no parallelism cap
|
||||
File: `internal/server/timeouts.go:8`, `internal/server/router.go:78-79`.
|
||||
|
||||
Cloud Scheduler can fire several crons within a minute. Each request is 5-min capped. Cloud Run free tier is `min=0,max=1` per the plan — so two crons that arrive 30s apart serialize, the second waits up to 5 minutes. Worse: an attacker who steals or guesses the cron secret can fire many `POST /cron/{any-valid-cron-name}` requests; with body=empty they cost nothing on Telegram side but pin the instance for 5 min × N.
|
||||
|
||||
Mitigations:
|
||||
- Tighten `defaultCronTimeout` to the actual wall-clock budget (e.g. 60s) and document that long crons must publish to PubSub and exit fast. 5 minutes is a footgun.
|
||||
- Cron secret rotation policy: document who rotates and how often. The shared-secret bridge in router.go:23 is "until Phase 09 OIDC" — code comment promises the migration but does not record an SLA.
|
||||
- Phase 09 OIDC + Cloud Run IAM ingress restriction is the proper fix.
|
||||
|
||||
### M5 — Webhook URL secret-token compare leaks length via `MaxBytesReader` ordering
|
||||
File: `internal/telegram/webhook.go:43-49`.
|
||||
|
||||
The constant-time secret compare runs *before* `MaxBytesReader` is installed. An unauthenticated POST with a 100 MiB body still gets streamed up to the auth check… actually no — the auth check only reads the header, body is never read. So no DoS via body upload pre-auth. **But** the body lifetime: an attacker can hold a slow-loris connection sending header bytes; the server's `ReadHeaderTimeout: 10s` (main.go:96) caps that. OK. **Confirmed safe**, just worth a comment that the order matters.
|
||||
|
||||
(Demoting from initial Medium to documentation-only after re-reading.)
|
||||
|
||||
### M6 — `ReadTimeout: 30s` covers webhook AND cron, but cron handlers may take longer than the *body read* allowance
|
||||
File: `cmd/server/main.go:97-101`.
|
||||
|
||||
`ReadTimeout` includes the body. For cron requests with empty bodies this is fine. If a future cron endpoint accepts a JSON payload and Cloud Scheduler ever delivers it slowly, 30s read could fire. Today: harmless. When phase 9 lands and cron payloads grow, revisit.
|
||||
|
||||
### M7 — `loldleemoji` does not HTML-escape the emoji string when rendering
|
||||
File: `internal/modules/loldleemoji/render.go:20`.
|
||||
|
||||
```go
|
||||
clue := "🎭 " + emojis
|
||||
```
|
||||
|
||||
`emojis` is loaded from the embedded `data/emojis.json` (loldleemoji.go:30) and the comment claims emojis "aren't HTML-escaped in the JS source either." True. But the data file is build-time controlled — a malicious / careless edit that puts `<script>` or `<` into an emoji value injects raw HTML into a Telegram message with `ParseMode: HTML`. Telegram's HTML parser is strict (only specific tags allowed) so the practical impact is limited to sending the bot's own users a message that fails to parse (Telegram returns 400; user sees nothing). Not exploitable for XSS — Telegram clients aren't browsers — but should still escape defensively given the rest of the code does.
|
||||
|
||||
Risk: Low (build-time data, Telegram clients sanitize). Fix: `html.EscapeString(emojis)` at render.go:20 or document the data-file invariant.
|
||||
|
||||
### M8 — `go.mod` declares `go 1.25.0` but CI builds with `go 1.23`
|
||||
Files: `go.mod:3`, `.github/workflows/ci.yml:17`.
|
||||
|
||||
```
|
||||
go 1.25.0 # go.mod
|
||||
go: ['1.23'] # CI matrix
|
||||
```
|
||||
|
||||
`go 1.25` in go.mod is the *minimum required toolchain*. Building with 1.23 should fail at `go build` (the go directive is enforced since 1.21+ for the language spec, since 1.22+ for stdlib). Either the CI is breaking and we don't notice, or `go 1.25.0` is wrong (current is 1.23 era; 1.25 is a future release). The Dockerfile uses `golang:1.23-alpine` (Dockerfile:1), so production also conflicts.
|
||||
|
||||
Action: align all three (go.mod / CI / Dockerfile) on the actually-installed toolchain. `1.25.0` looks like a typo for `1.23.0`.
|
||||
|
||||
### M9 — No top-level panic recovery around Telegram dispatch
|
||||
File: `internal/telegram/webhook.go:58`.
|
||||
|
||||
`bot.ProcessUpdate(ctx, &update)` runs synchronously (`WithNotAsyncHandlers`). The library does **not** recover panics in this path (verified against `process_update.go` v1.20.0). A panic inside a handler (say, a Phase 7 module that hits nil deref on a malformed Telegram media struct) propagates up.
|
||||
|
||||
`net/http`'s per-request recovery catches it, prints the stack trace via the server's ErrorLog → Cloud Run captures the stack to stderr. Two consequences:
|
||||
- Logs leak Go file paths and line numbers (low risk; logs are private).
|
||||
- The HTTP response is closed mid-write; Telegram sees non-2xx and retries the same panic-inducing update **forever** (Telegram retries failed webhooks for ~24h).
|
||||
|
||||
Wrap `b.ProcessUpdate` in:
|
||||
```go
|
||||
func() {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
log.Printf("webhook handler panic: %v", r)
|
||||
}
|
||||
}()
|
||||
b.ProcessUpdate(ctx, &update)
|
||||
}()
|
||||
```
|
||||
Then return 200. Telegram won't retry, the bug is logged, the bot stays up for the next update.
|
||||
|
||||
---
|
||||
|
||||
## Low
|
||||
|
||||
### L1 — `info.go` echoes sender ID + chat ID; visible to anyone in the chat
|
||||
File: `internal/modules/util/info.go:38-42`.
|
||||
|
||||
`/info` is `VisibilityPublic` and echoes `chat_id`, `thread_id`, `sender_id`. None are secrets (Telegram exposes them to clients via the API anyway), but in a public group `/info` reveals the numeric Telegram user ID of whoever runs it. Some users assume their UID is private. Document or mark as `VisibilityProtected` once visibility enforcement lands (M2).
|
||||
|
||||
### L2 — `PORT` env var is not validated; non-numeric value crashes net.Listen on `:` addr
|
||||
File: `cmd/server/main.go:172-175`.
|
||||
|
||||
`PORT=abc` → `srv.Addr = ":abc"` → `ListenAndServe` returns `address abc: unknown port`. Fail-fast happens but error is opaque. Cloud Run sets `PORT` correctly, so production-safe; local dev may footgun. One-liner: `if _, err := strconv.Atoi(port); err != nil { log.Fatalf(...) }`.
|
||||
|
||||
### L3 — `keylock.Map` grows unbounded; documented but no eviction
|
||||
File: `internal/keylock/keylock.go:8-12`.
|
||||
|
||||
Comment acknowledges 32 MB at 1M distinct keys, says "Eviction is a Phase 11 concern." Cloud Run instances are ephemeral so this is fine in practice. Just confirm: **per-instance** memory ceiling on free tier is ~512 MiB. 32 MB of locks is 6% of that — comfortable.
|
||||
|
||||
### L4 — `loldle` and `loldleemoji` `findChampion` return ambiguous-prefix → nil; user input "k" silently matches nothing
|
||||
File: `internal/modules/loldle/lookup.go:13-36`.
|
||||
|
||||
Behavior is JS-faithful and intentional. Just noting: a user typing `/loldle k` gets `Champion not found: "k"` even though many champions start with K. The JS source has the same UX, so parity is correct. No action.
|
||||
|
||||
### L5 — `pickRandom` uses `math/rand` global; predictable across instances
|
||||
Files: `internal/modules/loldle/handlers.go:65`, `internal/modules/loldleemoji/handlers.go:62`, `internal/modules/wordle/daily.go:58`.
|
||||
|
||||
Per-instance `math/rand` is mutex-protected (good for concurrency) but seeded with fixed `1` since Go 1.20 (no, actually Go 1.20 changed this — `math/rand` global is now seeded from a global random value at startup). Two instances start with different seeds. If predictability ever matters (it doesn't for game variety), use `math/rand/v2` or `crypto/rand`. Today: fine.
|
||||
|
||||
### L6 — Stickers' `file_id` strings are bot-scoped; rotation across bot tokens is undocumented
|
||||
File: `internal/modules/loldle/stickers.go:9-25`.
|
||||
|
||||
Telegram `file_id`s are scoped to the bot that uploaded them. If the dev/prod bot tokens differ (and they should — see README:31), the `winStickers`/`loseStickers` are dev-bot-scoped. In prod, `b.SendSticker` will fail with `STICKER_ID_INVALID`; `trySendSticker` swallows the error (handlers.go:120-129), so users see no sticker but no error either. Operations gap: prod will silently miss stickers. Document the procedure to re-capture stickers via `/stickerid` against the prod bot, or move sticker IDs to env vars.
|
||||
|
||||
### L7 — Dockerfile has no signed/verified base image digest
|
||||
File: `Dockerfile:1,13`.
|
||||
|
||||
`golang:1.23-alpine` and `gcr.io/distroless/static:nonroot` are pulled by tag, not digest. Supply-chain risk: an attacker compromising the registry (or a typo squatting registry MITM) substitutes a malicious image. Fix: pin both to digests (`golang:1.23-alpine@sha256:...`). CI/CD update: dependabot has Go module support but image-digest pinning needs a separate policy.
|
||||
|
||||
### L8 — `go test -race` is run but no coverage threshold gate, no `golangci-lint`, no `gosec`
|
||||
File: `.github/workflows/ci.yml`.
|
||||
|
||||
Vet + tests + build only. A `gosec` step would catch a future `subtle.ConstantTimeCompare` regression, an `os/exec` slip, etc. Recommended additions:
|
||||
- `golangci-lint run --timeout=5m` (gofmt, errcheck, staticcheck, gosec all in one).
|
||||
- Optional `govulncheck ./...` for known CVE detection in deps.
|
||||
|
||||
### L9 — `Build` returns the partially-populated registry on error from `factory()` panic
|
||||
File: `internal/modules/registry.go:118`.
|
||||
|
||||
If a factory panics (loldle's `loadChampions()` does on bad data), the panic propagates up out of `Build`. `main.go:74` then `log.Fatalf`'s. OK. But there's no `defer recover` in `Build` to convert the panic to an error — the `log.Fatalf` handler uses `%v` on an error, which would lose the panic type. Today: harmless, panics print clean stacks via Go's default. Document or wrap.
|
||||
|
||||
---
|
||||
|
||||
## Inputs / Boundaries Verified
|
||||
|
||||
- `r.URL.Path` for /cron/* validated `^[a-z0-9_]{1,32}$` (router.go:21,72-75) — no log injection possible.
|
||||
- Webhook secret: constant-time compare (webhook.go:44).
|
||||
- Cron secret: constant-time compare (router.go:66).
|
||||
- Webhook body: bounded via `MaxBytesReader` (webhook.go:49) — see M3 caveat.
|
||||
- JSON decode: standard `encoding/json` with no `UseNumber` or custom unmarshalers — JSON-bomb risk is bounded by body cap.
|
||||
- Telegram update parsing: delegated to `models.Update`. The library does not call `ioutil.ReadAll` on inner media; we never read media payloads.
|
||||
- Outbound HTTP: no `http.Get/NewRequest` anywhere. All Telegram traffic is via `b.SendMessage` etc. — URL is the official Telegram API endpoint. No SSRF surface.
|
||||
- No shell exec, no `text/template` or `html/template` used in user-facing paths (only `html.EscapeString` for HTML-mode replies, which is the correct primitive).
|
||||
- KV keys validated against Firestore constraints (firestore_kv.go:52-69). Module names validated `^[a-z0-9_-]{1,32}$` rejecting `:` (registry.go:19).
|
||||
- `keylock.Map` per-subject serialisation prevents Get→mutate→Put races; same-key tests confirm (keylock_test.go).
|
||||
- Container: `nonroot:nonroot` user, distroless base, no shell, `CGO_ENABLED=0` static binary. Good.
|
||||
|
||||
---
|
||||
|
||||
## Cloud Run / Firestore IAM (External — Verify Out-of-Band)
|
||||
|
||||
The bot relies on:
|
||||
1. Cloud Run service account having `roles/datastore.user` (Firestore RW).
|
||||
2. Cloud Run *not* having `allUsers` Invoker role — otherwise `/cron/*` is publicly invocable AND `/webhook` becomes redundantly auth'd by app-layer secret only.
|
||||
3. Cloud Scheduler invoking `/cron/*` via OIDC (Phase 09) or with the shared header today.
|
||||
|
||||
These cannot be verified from code. Recommend an `infra/` directory (Terraform / gcloud commands) committed to repo so IAM intent is reviewable. Today: unverifiable — listed as unresolved Q.
|
||||
|
||||
---
|
||||
|
||||
## Positive observations
|
||||
|
||||
- Constant-time compares on both webhook and cron secrets.
|
||||
- Webhook secret + cron secret fail-fast at startup (main.go:53-58, 82-84).
|
||||
- `bot.WithSkipGetMe()` + `WithNotAsyncHandlers()` correctly chosen for webhook mode.
|
||||
- Per-update 10s timeout (webhook.go:26) keeps Cloud Run instance from being held by a single hung handler.
|
||||
- Per-cron 5-min timeout (timeouts.go:8) — see M4 for parallelism caveat but the time bound is correct.
|
||||
- Distroless + nonroot + static binary + 6 MB image. Well under the 15 MiB target.
|
||||
- KV layer separates trust boundary cleanly: every key path validated, prefix wrapper `:` delimiter is non-bypassable due to module-name regex.
|
||||
- HTML rendering escapes user-controlled fields (`html.EscapeString` on champion names, sticker emoji, set name, etc.) — see M7 for the lone exception.
|
||||
- Sticker errors swallowed only after a comment-justified design choice (handlers.go:118-122).
|
||||
- Tests for negative paths: oversized body, wrong secret, same-prefix secret (timing edge), bad JSON, invalid cron name, nested cron path, log-injection attempt — all present.
|
||||
- Secret env stripping (`secretEnvKeys`) addresses the most obvious leak surface.
|
||||
- `MODULES` env validated with a regex that explicitly rejects `:` (the storage delimiter).
|
||||
|
||||
---
|
||||
|
||||
## Recommended Action Order
|
||||
|
||||
| # | Severity | Action |
|
||||
|---|----------|--------|
|
||||
| 1 | M8 | Align go.mod / CI / Dockerfile Go versions. `go 1.25.0` is a typo or premature. |
|
||||
| 2 | M9 | Wrap `b.ProcessUpdate` in `defer recover` so a buggy module doesn't trigger Telegram retry storms. |
|
||||
| 3 | M2 | Decide visibility-enforcement strategy (admin allowlist env or per-chat-admin check) — the `setmax` commands let any group member break the game. |
|
||||
| 4 | H1 | Replace env denylist with allowlist before Phase 07 (Gemini key) ships. |
|
||||
| 5 | M4 | Tighten `defaultCronTimeout` from 5m → 60s; document long-cron pattern. |
|
||||
| 6 | M3 | Either accept 413 in oversized-body path (drop the `http.Error 400` shadow) or use a pre-cap content-length check. |
|
||||
| 7 | M7 | `html.EscapeString` on `emojis` in loldleemoji/render.go:20 (one-liner). |
|
||||
| 8 | L7 | Pin Docker base images by digest. |
|
||||
| 9 | L8 | Add `golangci-lint` + `govulncheck` to CI. |
|
||||
| 10 | L6 | Document sticker `file_id` rotation when switching bot tokens. |
|
||||
|
||||
H1, M2, M4, M9 are the only items I'd require before opening the bot to a public Telegram username. The rest are hardening.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Cloud Run ingress policy**: is `/cron/*` reachable from the public internet, or is the service ingress-restricted to `internal-and-cloud-load-balancing`? If the latter, the cron shared-secret is defense-in-depth; if the former, M4 is closer to a High.
|
||||
2. **Phase 09 OIDC ETA**: the shared-secret bridge in router.go:23 has no documented retire-by date. If Phase 09 slips past the public launch, what's the fallback (rate-limit per cron name, additional IP allowlist)?
|
||||
3. **Bot token isolation between dev and prod**: README mentions a manually-created dev bot. Is there a documented process for prod bot token issuance + rotation, or does the bot token live in Secret Manager indefinitely? (Secret Manager rotation hooks would be Phase 09+ work.)
|
||||
4. **`VisibilityPrivate` semantics**: should `setmax` be private-as-in-bot-owner-only, private-as-in-chat-admin-only, or just unenforced? The current implementation accepts any caller; the field's existence implies an intent that has not been implemented. Clarify and either implement or remove.
|
||||
5. **Cloud Run min-instances**: free tier is min=0. A cold start currently blocks ~1.5s on `firestore.NewClient`. Telegram's webhook timeout is 10s; first update after idle hits the cold start window. Is that latency budget acceptable, or should min=1 (paid)?
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
**Summary:** No critical defects; production posture is solid given prior reviews already landed the high-risk fixes. The remaining items are: visibility enforcement (M2), env allowlist before Phase 07 (H1), panic recovery around dispatcher (M9), and a Go-version mismatch (M8). Ship-ready for a private bot; resolve M2/H1/M9 before public username.
|
||||
@@ -1,79 +0,0 @@
|
||||
# Code Review — Auto-Register Telegram Webhook + Commands After Deploy
|
||||
|
||||
**Date:** 2026-05-16
|
||||
**Branch:** main (uncommitted)
|
||||
**Scope:** `.github/workflows/deploy.yml` (+46 lines), `docs/deploy-aws.md` (+2), `docs/deploy-aws-free-tier-guide.md` (+2)
|
||||
**Plan:** `plans/260516-1035-auto-register-after-deploy/`
|
||||
|
||||
## Status
|
||||
|
||||
**DONE — no must-fix issues.** Implementation faithfully mirrors `Makefile:101-148` reference, hardens it with `set -euo pipefail` + `jq -e` validation, and masks credentials before they can reach any log line. All 8 acceptance criteria from the plan are met by the diff.
|
||||
|
||||
## Critical findings
|
||||
|
||||
None. No blockers, no security regressions, no contract breaks.
|
||||
|
||||
## Verified safe
|
||||
|
||||
1. **Mask timing — no leak window.**
|
||||
- `.github/workflows/deploy.yml:76-79` — `TOKEN=$(...)` then `echo "::add-mask::$TOKEN"` on the very next line. AWS CLI with `--output text --query Parameter.Value` writes nothing else to stdout/stderr, so the value never escapes between capture and mask.
|
||||
- Same pattern at lines 80-83 (SECRET) and 100-103 (re-read TOKEN in commands step).
|
||||
|
||||
2. **No URL-leak via curl error.** Empirically verified `curl 8.5.0 -fsS` on 4xx prints `curl: (22) The requested URL returned error: <code>` — URL is **not** in the message. Even if it were, `::add-mask::` was issued at line 79 before line 86's `curl`, so GH Actions redacts the token from all subsequent log output (stdout + stderr) in the same job. No `set -x`, no `-v`, no `curl -w` interpolating the URL.
|
||||
|
||||
3. **Failure semantics — fail-loud, no silent swallows.**
|
||||
- `set -euo pipefail` + `RESP=$(curl -fsS ...)` — empirically confirmed: curl exit-22 propagates through command substitution, `set -e` kills the shell before next line. (Note: `errexit` *does* propagate from `$(...)` since bash 4.0.)
|
||||
- `jq -e '.ok == true' >/dev/null || { echo ...; exit 1; }` — catches Telegram returning HTTP 200 with `{"ok":false}` body. The `|| { ... }` keeps `set -e` honest (no pipefail-with-tee anti-patterns).
|
||||
- Both steps lack `continue-on-error: true` — failures bubble up to the job.
|
||||
|
||||
4. **Webhook contract matches code.**
|
||||
- Path: workflow sends to `${URL%/}/webhook`; router accepts at `internal/server/router.go:51` (`mux.Handle("/webhook", ...)`).
|
||||
- Secret: workflow forwards `secret_token=${SECRET}`; Telegram echoes back via `X-Telegram-Bot-Api-Secret-Token` header; `internal/telegram/webhook.go:22,48-52` validates with `subtle.ConstantTimeCompare`. Same SSM param (`/miti99bot/prod/telegram-webhook-secret`) feeds both setWebhook and Lambda env (via `template.yaml` `:1` resolver), so values stay in sync.
|
||||
- `allowed_updates=["message","callback_query"]` — matches `Makefile:120` reference exactly.
|
||||
|
||||
5. **setMyCommands payload format.** `aws/telegram-commands.json` has top-level `{"commands": [...]}` which matches Telegram API spec for `--data-binary @file` with `Content-Type: application/json`. Verified via Telegram Bot API docs.
|
||||
|
||||
6. **IAM permissions present.** `aws/README.md:74` lists `AmazonSSMFullAccess` and `:69` lists `AWSCloudFormationFullAccess` attached to `github-deploy-miti99bot`. No new grants needed. Plan claim verified.
|
||||
|
||||
7. **Concurrency safe.** `.github/workflows/deploy.yml:12-14` — `concurrency.group: deploy-prod`, `cancel-in-progress: false` → serial queueing, no parallel setWebhook race. SSM secret values are versioned and read-after-deploy, so the newest value wins by ordering, not by collision.
|
||||
|
||||
8. **YAML structural validity.** 9 total steps (was 7, +2 new). Indentation consistent with existing steps; `env:`, `run:` blocks well-formed (read at `.github/workflows/deploy.yml:67-93,95-111`). No actionlint available locally to formally validate, but eye-parse is clean.
|
||||
|
||||
9. **Docs labeling is unambiguous.**
|
||||
- `docs/deploy-aws.md:56` — "auto-runs `setWebhook` + `setMyCommands` after every push to `main`. The snippet below is the break-glass equivalent..."
|
||||
- `docs/deploy-aws-free-tier-guide.md:261` — "For first-time setup only. After Step 6 wires the GitHub workflow, every push to `main` auto-runs `setWebhook` + `setMyCommands`; this manual block is the break-glass path."
|
||||
- Both clearly tag the manual blocks as break-glass / first-time-only. Low risk of a reader running them every deploy.
|
||||
|
||||
10. **Idempotency.** `setWebhook` and `setMyCommands` are documented as idempotent by Telegram. Running on every push is safe and self-healing (per plan's stated goal).
|
||||
|
||||
11. **No YAGNI/scope creep.** Diff is exactly the two new steps + the two one-line doc notes. No defensive plumbing, no caching layer, no follow-up IAM tightening (correctly deferred to a separate plan).
|
||||
|
||||
## Recommendations (defer — non-blocking)
|
||||
|
||||
| # | Location | Note |
|
||||
|---|----------|------|
|
||||
| R1 | `.github/workflows/deploy.yml:91-92,109-110` | `echo "setWebhook failed: $RESP"` prints the full Telegram response on failure. Telegram's `description` field is generic ("Bad Request: ..."), so token-in-body leak is implausible, but masking is already in place as belt-and-suspenders. Keep as-is — useful for diagnosing rare API rejections. |
|
||||
| R2 | `docs/deploy-aws.md:67`, `docs/deploy-aws-free-tier-guide.md:273` | Break-glass manual blocks still use `${URL}webhook` (no `%/` trim). This relies on CFN's `FunctionUrl` always ending with `/`, which is the documented Lambda behavior. Not a regression (pre-existing). Tighten next time the file is touched. |
|
||||
| R3 | `.github/workflows/deploy.yml:76-78,100-102` | Two SSM calls in two adjacent steps to read the same token. Plan already acknowledges this as acceptable (single SSM call is cheap, keeps steps independently re-runnable). Single-step consolidation or `$GITHUB_OUTPUT` is the documented follow-up. |
|
||||
| R4 | `.github/workflows/deploy.yml:67-93` | No `if: success()` guard on the new steps. GH Actions defaults to running a step only when prior steps succeed, so this is redundant — but adding it explicitly would make the intent obvious to a reader. Optional. |
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
None. All adversarial vectors closed by verification against codebase + empirical curl/bash tests.
|
||||
|
||||
---
|
||||
|
||||
## Citations
|
||||
|
||||
- Workflow diff: `.github/workflows/deploy.yml:67-111`
|
||||
- Webhook handler validation: `internal/telegram/webhook.go:22,48-52`
|
||||
- Router mount: `internal/server/router.go:51`
|
||||
- IAM policy attachments: `aws/README.md:69,74`
|
||||
- CFN output declaration: `template.yaml:207-210`
|
||||
- Reference Makefile targets: `Makefile:101-148`
|
||||
- Commands JSON: `aws/telegram-commands.json:1-96`
|
||||
- Doc break-glass labels: `docs/deploy-aws.md:56`, `docs/deploy-aws-free-tier-guide.md:261`
|
||||
- Concurrency control: `.github/workflows/deploy.yml:12-14`
|
||||
|
||||
**Status:** DONE
|
||||
**Summary:** Implementation is correct, secure, and matches the plan. Token masking is in place before any line that could log the value; `curl -fsS` + `jq -e` + `set -euo pipefail` produce loud failures with no silent swallows; `/webhook` path and secret-token contract match `internal/server/router.go:51` and `internal/telegram/webhook.go:48-52`. Docs clearly label manual blocks as break-glass. No must-fix issues; safe to commit.
|
||||
@@ -1,102 +0,0 @@
|
||||
---
|
||||
title: "Adversarial Failure-Mode Analysis — Atlas M0 Migration Plan"
|
||||
date: 2026-04-25
|
||||
scope: plans/260425-1945-mongodb-atlas-migration/ (phases 01–08)
|
||||
---
|
||||
|
||||
# Atlas Migration: Failure-Mode Analysis
|
||||
|
||||
## 1. TL;DR — Top 3 by Likelihood × Impact
|
||||
|
||||
1. **Cold-start hang exceeds Worker CPU budget (scenario #1)** — likelihood HIGH × impact CATASTROPHIC. Research already shows ~1500ms TLS+SCRAM; Worker free plan CPU limit is 50ms (paid is 30s wall-clock). If the driver's initial TCP connect blocks the CPU thread, the isolate is killed before any response. Plan documents 1500ms latency but does NOT verify whether this is CPU-time or wall-clock time. One CPU-bound TLS frame could exceed the 50ms limit silently. No mitigation in any phase file.
|
||||
|
||||
2. **M0 auto-pause: 30-60s cold-wake hangs the bot completely (scenario #4)** — likelihood MEDIUM × impact CATASTROPHIC. Plan flags this as "unresolved question 4" in phase-08 and mentions it in phase-01/phase-06 risk tables, but the only mitigation listed is "Phase 02 must wrap connect() with timeout + retry; user-friendly error if pause-resume exceeds 30s." No timeout is actually specified in the `mongo-client.js` snippet (phase-02 shows `serverSelectionTimeoutMS: 5000`, which means after 5s the driver throws — but during that 5s the Worker is hung and cannot respond to Telegram, which will retry and create new requests that also hang). No user-facing error path is coded. The driver does NOT auto-retry a paused M0 wake; it fails with a server-selection timeout after the configured window.
|
||||
|
||||
3. **Backfill stale-data overwrite race (scenario #10)** — likelihood MEDIUM × impact SEVERE. The plan's ordering (`dual-write FIRST, then backfill`) is correct, but `$setOnInsert` (skip-if-exists) is documented as the backfill upsert strategy. However, `$setOnInsert` fires only when the document is being *inserted* (upsert: true, doc absent). If the document already exists from dual-write, backfill does nothing — correct. But if dual-write wrote doc A, backfill ran, then dual-write wrote doc B (update), then backfill re-runs (idempotent re-run), backfill's `$setOnInsert` does nothing (doc exists) — also correct. Race exists only during the original window where backfill reads KV and before dual-write has yet written that key to Mongo. The ordering in phase-05 says dual-write must be live first, which mitigates this. HOWEVER: the backfill reads from KV via the `/__admin/dump-kv` Worker route (phase-05), which reads through CFKVStore *directly* (bypassing the dual wrapper) at the time of the batch. If a key expires between KV read and Mongo upsert, Mongo gets a stale value that then persists beyond the original TTL unless `expiresAt` is correctly propagated — the plan states it IS propagated, but this is not explicitly verified in the verify script.
|
||||
|
||||
---
|
||||
|
||||
## 2. Failure Modes Table
|
||||
|
||||
| # | Scenario | Likelihood | Impact | Detection Time | Plan Coverage | Recommended Addition |
|
||||
|---|----------|-----------|--------|----------------|---------------|----------------------|
|
||||
| 1 | Atlas TLS handshake blocks Worker CPU budget (50ms free / 30s paid) | HIGH | CATASTROPHIC | Immediate on first deploy | NONE — plan conflates CPU-time with wall-clock time throughout | Test CPU-time specifically via `wrangler dev` profiling; document free-plan CPU limit constraint; add abort condition to phase-06: "if cold-start triggers CPU exceeded errors, abort" |
|
||||
| 2 | Cold-start stampede on deploy exhausts M0 500-connection cap | MEDIUM | SEVERE | Minutes post-deploy (CF spawns 10–50 isolates) | PARTIAL — phase-01 mentions cap, phase-06 abort criterion is >400 connections, but no pre-deploy estimate of isolate count vs cap | Add pre-deploy synthetic burst test (phase-06 step 5.5): hit Worker with 20 parallel cold requests; observe Atlas connection counter in Atlas UI before proceeding |
|
||||
| 3 | Atlas PoP-to-Atlas jitter spikes cold-start to 5s+ without crossing 3s P95 gate | MEDIUM | SEVERE | After hours to days (soak) | PARTIAL — phase-06 specifies P95 > 3000ms gate but instrumentation (`src/util/timing.js`) is not deployed until phase-06 step 1; gap between phase-04 deploy and instrumentation deploy means early soak hours produce no P95 data | Deploy timing.js in phase-04 or as part of phase-04 smoke test, not phase-06; ensures from-first-request telemetry |
|
||||
| 4 | M0 auto-pause: 30-60s wake stalls all requests (no queuing, all hang) | MEDIUM | CATASTROPHIC | Next request after 30d inactivity | PARTIAL — phase-01 notes it, phase-06 abort criterion includes it, but driver behavior during wake is undocumented; `serverSelectionTimeoutMS: 5000` causes timeout exception after 5s, not a user message | Add explicit catch in `getDb()` for server-selection timeout → return 503 "Service temporarily unavailable" to Telegram; add a Mongo heartbeat cron (verify phase-07 §"Atlas auto-pause" risk uses `misc` cron — phase-08 unresolved Q4 flags this but doesn't resolve it) |
|
||||
| 5 | M0 500-connection limit: cold isolate burst queues then times out | MEDIUM | SEVERE | Minutes under traffic spike | PARTIAL — phase-06 abort criterion ">400 connections" exists but detection relies on Atlas UI polling, not programmatic; no driver behavior on connection-limit documented | Document that MongoDB driver throws `MongoServerSelectionError` when connection limit hit; add this error class to phase-06 log search pattern; wire into abort decision |
|
||||
| 6 | Atlas API token expired / MFA session expired mid-provisioning (phase-01) | LOW | MODERATE | Phase-01 execution | NONE — plan has no mention of Atlas API token lifecycle or MFA requirement | Add note in phase-01 step 1: "Atlas UI sessions expire; complete provisioning in one sitting; API token required for scripted ops has 30-day TTL" |
|
||||
| 7 | `0.0.0.0/0` IP allowlist — security tradeoff not documented as permanent risk | LOW | MODERATE | Never (silent) | PARTIAL — phase-01 documents it but classifies mitigation as "SCRAM+TLS" without acknowledging that M0's connection count + SCRAM is the only auth barrier; no CF Workers static-IP plan | Phase-01 should note: the only upgrade path to tighter IP allowlist is CF Workers paid static egress IP add-on (~$10/mo); document cost in phase-08 cost-tracking.md |
|
||||
| 8 | Network partition: KV write succeeds, Mongo write fails → silent divergence | MEDIUM | SEVERE | Only via verify:mongo script (manual, not automated) | PARTIAL — phase-04 logs secondary failures; phase-05 verify script is manual; no alerting if divergence accumulates between manual runs | Add automated daily verify run via cron (not manual invocation); OR add a divergence counter that triggers CF Observability alert threshold |
|
||||
| 9 | Concurrent writes from different isolates: KV gets value A, Mongo gets value B (race) | LOW | SEVERE | Silent — may never be detected | NONE — plan does not address multi-isolate concurrent write races; KV is last-write-wins; Mongo is also last-write-wins; if both writes race, final state in KV ≠ final state in Mongo | Acknowledge in phase-04 risk table; for game state (non-transactional KV) this is acceptable; for trading (atomic intent), confirm `run()` is called only from single-user contexts |
|
||||
| 10 | Backfill stale-data propagation: TTL not propagated → expired data persists in Mongo | MEDIUM | MODERATE | Only via hash-compare in verify (catches value mismatch not TTL mismatch) | PARTIAL — plan mentions TTL propagation but verify script checks value SHA256, not expiresAt field | Add to verify script: for each sampled key, also compare `expiresAt` bucket (not exact time, but present/absent and within ±5min) |
|
||||
| 11 | KV lazy TTL vs Mongo 60s sweep: divergence detector flags valid state as mismatch | MEDIUM | LOW (false positives) | During verify runs | PARTIAL — phase-02 notes "tests assert field only, not deletion timing" but phase-05 verify uses count parity within 1%; if KV TTL keys haven't been swept yet, count may exceed Mongo | Acknowledge in verify script: count KV excluding expired (if API supports it) OR document that verify is run-after-traffic-drains (UTC 18:00) to minimize in-flight TTLs |
|
||||
| 12 | Phase-07 "irreversible step" (step 18): no guard against accidental execution | LOW | CATASTROPHIC | After execution (too late) | PARTIAL — phase-07 documents step 18 clearly, requires operator to run manually, notes CLI interactive confirm; but the CLI `wrangler kv:namespace delete` and `wrangler d1 delete` do NOT always require interactive confirm in CI/scripted contexts | Add explicit `read -p "Type CONFIRM to delete: "` check in a wrapper script before calling wrangler delete commands; document: "never run these in CI automation" |
|
||||
| 13 | Atlas outage at 4am during soak; bot dead for hours before user wakes | MEDIUM | SEVERE | Hours (no alerting) | PARTIAL — phase-06 abort criterion "Atlas outage > 5min" exists but relies on operator manually checking logs; no automated alert | Set up Atlas free-tier email alert for cluster unreachable; OR wire CF Observability alert: if >10% of requests return 500 in a 5-min window, send notification |
|
||||
| 14 | Rollback after cutover: KV/D1 data is 7+ days stale | MEDIUM | SEVERE | On rollback attempt | PARTIAL — phase-07 Stage 2 rollback mentions "reverse-backfill (script not pre-written)"; 7-day game state loss is acknowledged as out-of-scope | Explicitly document in phase-07: "rollback after Stage 2 = game state loss for last N days; inform users." Consider if this is acceptable before executing Stage 2 |
|
||||
| 15 | `/__admin/*` routes: ADMIN_TOKEN in URL (not header) or in server logs | LOW | HIGH | Never (silent credential leak) | PARTIAL — phase-05 specifies header-only auth (`X-Admin-Token`) and constant-time compare; BUT phase-05 step 2 says "log requests but NOT response bodies" — does not explicitly say NOT to log headers; access logs in CF Observability may include headers | Explicitly suppress `X-Admin-Token` header from CF Observability logs in the admin route handler; add to phase-05 security checklist |
|
||||
| 16 | Backfill runs out of CPU mid-stream; resumability relies on KV cursor durability | LOW | MODERATE | Mid-run (script exits) | PARTIAL — phase-05 mentions KV cursor expires in 60s, handled by restart from last seen key; but "last seen key" is not checkpointed to disk — only in-memory | Add cursor checkpoint: write last processed key to a local file after each page; resume from checkpoint on restart |
|
||||
| 17 | Integer IDs (D1 autoincrement) vs hex ObjectId: downstream comparison breakage | LOW | MODERATE | After cutover, if new feature uses id numerically | PARTIAL — phase-03 notes `id` is returned as hex string; grepping confirms `IN (...)` list usage is string-compare-safe; BUT: if any code does `id - 1` or numeric sort by id, it silently breaks | Add `grep -r "\.id\s*[-+*/<>]" src/modules/trading/` to phase-03 verification step; confirm zero arithmetic on id field |
|
||||
| 18 | `MONGODB_URI` in `.env.deploy` not updated → register script breaks after password rotation | MEDIUM | MODERATE | Next deploy after rotation | PARTIAL — phase-01 documents two-copy problem (same pattern as TELEGRAM_BOT_TOKEN); phase-08 docs include rotation procedure; BUT no lint/check prevents deploy with stale URI | Add to phase-08 `check-secret-leaks.js`: also verify `.env.deploy` and CF secret URI share same host hash (not password, just host match) OR document a rotation runbook with explicit two-step instructions |
|
||||
| 19 | `nodejs_compat_v2` enables Node globals; existing module checks for Workers env via their absence | LOW | MODERATE | Only in wrangler dev / production deploy (not in vitest tests) | PARTIAL — phase-01 risk table mentions it; "run full test suite after edit" is the mitigation; but vitest runs in Node so cannot detect this regression | Add explicit integration test with `wrangler dev` (not vitest) as part of phase-01 success criteria; specifically test grammY and each module's `init()` |
|
||||
| 20 | `compatibility_date` bump contradiction: wrangler.toml already has `2025-10-01` which is AFTER `2025-03-20`; plan says "bump" but no bump is needed | LOW | LOW (plan confusion only) | Phase-01 execution | NONE — plan says "bump compatibility_date" but the current `2025-10-01` already satisfies the `>= 2025-03-20` requirement | Correct plan text in phase-01 step 7: "no compatibility_date change needed — current value `2025-10-01` already satisfies requirement"; note this to avoid a confusing no-op commit |
|
||||
| 21 | `fake-mongo.js` does not simulate TTL; tests pass but production fails on TTL-sensitive paths | MEDIUM | MODERATE | Production only (tests always green) | PARTIAL — phase-02 explicitly documents this gap; phase-08 unresolved Q1 acknowledges it | Document which TTL scenarios are unverifiable in unit tests and MUST be validated in soak; add to phase-06 checklist: "trigger a game with TTL expiry and verify it behaves correctly" |
|
||||
| 22 | MongoDB driver behaves differently in Workers (cloudflare:sockets) vs Node (net); unit tests pass but Worker fails | MEDIUM | SEVERE | Production only | NONE — architecture.md §12 "Pure-logic unit tests only. No workerd pool." — this is a known project constraint but the plan adds a TCP-socket driver that is specifically sensitive to this distinction | Add to phase-01 success criteria: successful `wrangler dev` smoke test with real Atlas (not just vitest); this is the only way to catch Workers-specific driver behavior |
|
||||
| 23 | Alt-pivot path is a "TODO" — user is Atlas-all-in; if Atlas fails UX test, there is no ready Upstash plan | HIGH | MODERATE | Phase-06 abort | PARTIAL — plan mentions "Open new plan: plans/<date>-upstash-migration/ (TODO)" and references researcher report; but writing a new plan takes days; during that time bot is running degraded (dual-write adds ~1500ms p99) OR rolled back to KV/D1 quota exhaustion | Pre-write a phase-07-alt-pivot.md NOW with the Upstash swap steps from researcher report §"Next Steps"; it's 3-4 hours of work that eliminates a week of replanning under pressure |
|
||||
|
||||
---
|
||||
|
||||
## 3. Critical Gaps (Must Address Before Execution)
|
||||
|
||||
### GAP-A: CPU-time vs wall-clock time for TLS handshake (scenario #1)
|
||||
**Evidence:** Cloudflare Workers free plan has a 50ms CPU time limit (not wall-clock). The mongodb driver's TLS handshake involves CPU-bound operations (TLS key exchange, SCRAM PBKDF2). If these exceed 50ms CPU, the Worker is killed silently — no error to user, Telegram retries, creating a stampede. The plan only discusses latency as wall-clock. The researcher report cites ~1500ms but does not specify what fraction is CPU-bound. Architecture.md §14 states `nodejs_compat` flag not needed currently — but adding `nodejs_compat_v2` (phase-01) moves the bot to a different execution context with different CPU accounting rules.
|
||||
**Action required:** Before any code is written, verify with a minimal `wrangler dev` test that connects to Atlas and measures both wall-clock AND CPU time (`performance.now()` does not measure CPU; use `Date.now()` delta combined with CF dashboard CPU metric). If CPU time during cold connect approaches 50ms, the free plan is incompatible and migration must not proceed.
|
||||
|
||||
### GAP-B: No automated divergence alerting (scenarios #8, #13)
|
||||
**Evidence:** The verify script is invoked manually. Phase-06 requires "daily `verify:mongo` runs via manual invocation". Phase-06 also requires operator to manually check Atlas UI for connection count. The bot is a personal project; the operator sleeps. An Atlas outage at 4am + no alert = silent bot death for 8+ hours. Phase-06 says "monitoring gap" is one thing to address but proposes no monitoring solution beyond "CF Observability built-in views".
|
||||
**Action required:** Add to phase-06: set up at least one automated alert. Options: (a) CF Observability alert rule: >10 errors in 1 minute → email; (b) Atlas free-tier email alert for cluster unavailability. Both take 5 minutes to configure and require no code.
|
||||
|
||||
### GAP-C: M0 auto-pause wake behavior not tested before soak (scenario #4)
|
||||
**Evidence:** Phase-01 mentions auto-pause behavior but defers handling to phase-02. Phase-02's `mongo-client.js` snippet sets `serverSelectionTimeoutMS: 5000` — this means a paused M0 will cause the driver to throw after 5s, not wait 60s for cluster wake. The user gets a 500 error, not a "please wait" message. Phase-08 unresolved Q4 asks "confirm one cron touches Mongo" — but the weekly cron heartbeat doesn't prevent pause (needs daily activity, not weekly). The cron schedule in wrangler.toml shows `"0 17 * * *"` and `"0 1 * * *"` — both daily, which would prevent pause IF they perform a Mongo write. But this is phase-08 unresolved Q4, not confirmed.
|
||||
**Action required:** (a) In phase-01, add explicit test: pause the cluster manually in Atlas UI, then test first-request behavior. Confirm driver throws with a catchable error, not a hang. (b) Confirm daily cron writes to Mongo (not KV only) before declaring heartbeat sufficient. (c) In `mongo-client.js`, add specific catch for `MongoServerSelectionError` and return a 503 with Retry-After header.
|
||||
|
||||
### GAP-D: No pre-written Upstash pivot plan (scenario #23)
|
||||
**Evidence:** plan.md §"Abort criteria" says "escalate to Upstash plan"; phase-06 §"Pivot Path" says "Open new plan: phase-07-alt-pivot.md (TODO)". The researcher report §"Next Steps (If Upstash Recommended)" provides a 5-step implementation guide that is ~3-4 hours of work. If Atlas fails the 24h soak gate, the user has no working game bot (KV quota still exhausted, Atlas degraded) and needs to plan Upstash under pressure.
|
||||
**Action required:** Write `phase-07-alt-pivot.md` before starting execution. It does not need to be fully detailed — a 50-line stub with the Upstash steps from the researcher report is sufficient. Having it pre-written means the pivot can start within hours of an abort decision.
|
||||
|
||||
---
|
||||
|
||||
## 4. Quick Wins (Small Plan Additions, High Impact)
|
||||
|
||||
**QW-1 (30 min):** In phase-01 step 9 (smoke test route), add CPU-time measurement alongside wall-clock. Log both: `{wall_ms, cpu_note: "check CF dashboard CPU column"}`. Adds zero code complexity; creates evidence for decision.
|
||||
|
||||
**QW-2 (10 min):** Add Atlas free-tier email alert for cluster unavailability in phase-06 step 4. Zero code; closes GAP-B partially. Note: Atlas free tier allows alerting on cluster health events.
|
||||
|
||||
**QW-3 (15 min):** In `mongo-client.js` (phase-02), add explicit catch for `MongoServerSelectionError` with a comment: "M0 may be auto-paused; 5s timeout is correct; catch and log with actionable message." This does not fix the pause but gives the operator a clear log entry.
|
||||
|
||||
**QW-4 (20 min):** In phase-01 step 8 (risk table), add explicit note: "Worker free plan: 50ms CPU limit. Paid plan: 30s wall-clock. This migration requires Workers paid plan OR testing confirms TLS handshake CPU time < 40ms." This is a deployment-blocker check, not a code change.
|
||||
|
||||
**QW-5 (15 min):** Fix the `compatibility_date` confusion in phase-01 step 7. Current `2025-10-01` already satisfies `>= 2025-03-20`. Remove "bump compatibility_date" instruction; replace with "verify current date satisfies requirement (it does)." Prevents a confusing no-op commit.
|
||||
|
||||
**QW-6 (1 hour):** Pre-write `phase-07-alt-pivot.md` stub using the researcher report's 5-step Upstash guide. Labels it "STANDBY — execute only if phase-06 ABORT." Closes GAP-D.
|
||||
|
||||
**QW-7 (20 min):** In phase-05 `dump-routes.js` auth middleware, explicitly suppress `X-Admin-Token` from any access log: `const token = request.headers.get("X-Admin-Token"); /* do NOT log token value */`. One comment line; prevents accidental CF Observability header capture.
|
||||
|
||||
**QW-8 (30 min):** In phase-06, add `wrangler tail --format=json | grep "dual-write:secondary:failed"` as a manual monitoring step during soak. This surfaces the divergence signal in real-time without waiting for daily verify runs.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. Is the Worker on the free plan or paid plan? This is critical for scenario #1 — free plan's 50ms CPU limit may be incompatible with TLS handshake CPU cost. The plan never states which plan is in use. (wrangler.toml has no `[limits]` or `account_id` override.)
|
||||
|
||||
2. What is the actual daily active user count + ops/day? Phase-06's connection saturation analysis assumes "typical CF replication factor = ~50–100 isolates". This is a guess, not a measurement. If the bot has 5 users, connection exhaustion is irrelevant. If it has 200, it matters.
|
||||
|
||||
3. Does M0 in `aws-ap-southeast-1` have a different SRV lookup latency from CF's Singapore PoP vs US PoPs? The researcher cites ~1500ms for US; SEA-optimized routing may be different (better or worse). No benchmark cited for this specific PoP combination.
|
||||
|
||||
4. The `trading` module uses `retention.js` which runs a retention DELETE job. This job is presumably triggered by cron. After cutover, does the cron handler receive a `MongoSqlStore` or `CFSqlStore`? Phase-06 §"Soak data flow" shows `dispatchScheduled` creating stores, but `cron-dispatcher.js` creates fresh stores per invocation (confirmed by architecture.md §9). The store type depends on env flags — if `STORAGE_PRIMARY=kv` during soak, cron jobs read from KV and write to KV, not Mongo. Trading retention running against KV while trading history reads from KV (correct), but Mongo trading_trades may have stale records not being cleaned up. Plan does not address cron jobs during dual-write window.
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
**Summary:** 23 failure modes identified; 4 critical gaps require pre-execution action; most severe unaddressed risk is CPU-time vs wall-clock ambiguity for the TLS handshake, which could make the migration impossible on the free Worker plan regardless of latency.
|
||||
@@ -1,144 +0,0 @@
|
||||
# Debug Report — Bot Commands Silent in Groups, Work in DM
|
||||
|
||||
**Date:** 2026-05-16
|
||||
**Branch:** main (HEAD `df89431`)
|
||||
**Status:** DONE — root cause identified with code-level + log-level evidence; proposed fix below; nothing changed yet.
|
||||
|
||||
## Symptom
|
||||
|
||||
User reports: bot responds correctly to commands sent in a private chat (DM). Same commands sent in a Telegram group do nothing — no reply, no error, no logged warning.
|
||||
|
||||
## Root cause
|
||||
|
||||
`github.com/go-telegram/bot@v1.20.0/handlers.go:85-93` does a **byte-exact equality check** on the bot-command entity text without stripping the `@botname` suffix that Telegram clients append in groups.
|
||||
|
||||
```go
|
||||
if h.matchType == MatchTypeCommand {
|
||||
for _, e := range entities {
|
||||
if e.Type == models.MessageEntityTypeBotCommand {
|
||||
if data[e.Offset+1:e.Offset+e.Length] == h.pattern {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For private chats:
|
||||
- User sends `/help`.
|
||||
- Telegram sets entity Offset=0, Length=5.
|
||||
- Slice `data[1:5]` = `"help"` → equals `h.pattern="help"` → ✅ match.
|
||||
|
||||
For groups (this is what Telegram clients send when there are 2+ bots in the room OR when the user picks the command from the autocomplete menu):
|
||||
- User sends `/help@miti99bot`.
|
||||
- Telegram sets entity Offset=0, Length=15 (the `@suffix` is **inside** the entity).
|
||||
- Slice `data[1:15]` = `"help@miti99bot"` → does NOT equal `"help"` → ❌ no match.
|
||||
|
||||
The library never strips the `@botname` portion. No test in `handlers_test.go` covers the group form (only bare `/foo`). All 27 commands registered via `dispatcher.go:52-71` `Install` use `MatchTypeCommand`, so all 27 commands fail the same way in groups when sent with the suffix.
|
||||
|
||||
## Evidence
|
||||
|
||||
### 1. Library source
|
||||
|
||||
`/config/go/pkg/mod/github.com/go-telegram/bot@v1.20.0/handlers.go:85-93` — exact match logic shown above.
|
||||
|
||||
`/config/go/pkg/mod/github.com/go-telegram/bot@v1.20.0/handlers_test.go` — only test cases:
|
||||
| Input | Length | Result |
|
||||
|---|---|---|
|
||||
| `/foo` | 4 | match ✓ |
|
||||
| `a /foo` | 4 (offset 2) | match ✓ |
|
||||
| `a /bar` | 4 | no match ✓ |
|
||||
|
||||
No `/foo@botname` cases tested.
|
||||
|
||||
### 2. Production logs
|
||||
|
||||
CloudWatch `/aws/lambda/miti99bot`, last 60 min:
|
||||
|
||||
| Time (UTC) | Event | Notes |
|
||||
|---|---|---|
|
||||
| 04:22:02 | `POST /webhook 200 1ms` + `[TGBOT] [UPDATE] ID:835070937` | Update arrived. No metric for it. |
|
||||
| 04:22:59 | `POST /webhook 200 1372ms` | Long-running handler — this is the DM that worked. |
|
||||
| 04:23:56 | `metrics commands={trade_stats:1}` | Metric flush — only **1** command counted in the cron interval. |
|
||||
| 04:23:56 | `POST /webhook 200 0ms` + `[TGBOT] [UPDATE] ID:835070939` | Update arrived, returned immediately with no command dispatched. |
|
||||
|
||||
So 3 webhooks, 1 fired a handler (`trade_stats`). The other 2 are the group attempts: payload accepted, secret validated, JSON decoded, dispatched into `b.ProcessUpdate` — and silently dropped because no handler matched. **No errors, no panics, no `unauthorized`, no `request body too large`**. Exact signature of "the match function returned false for every registered handler".
|
||||
|
||||
### 3. Webhook config matches expectation
|
||||
|
||||
`getWebhookInfo` (output from current production state, verified by previous workflow run logs):
|
||||
- `url=https://<lambda>.lambda-url.ap-southeast-1.on.aws/webhook`
|
||||
- `allowed_updates=["message","callback_query"]`
|
||||
- `pending_update_count` ≈ 0 (no delivery backlog)
|
||||
|
||||
Group messages **are** being delivered. Telegram is not the problem.
|
||||
|
||||
### 4. Bot Privacy Mode is NOT the root cause
|
||||
|
||||
Privacy mode would prevent the update from arriving at all. We see group updates arrive (rows 1 and 4 above). Privacy mode is therefore either OFF, or the user typed a command (which Telegram delivers even with privacy ON). Either way, the failure is downstream of delivery — in the dispatcher's match logic.
|
||||
|
||||
## Hypotheses ruled out
|
||||
|
||||
| Hypothesis | Why ruled out |
|
||||
|---|---|
|
||||
| Webhook secret mismatch in groups | Would return 401; logs show 200. |
|
||||
| Body-too-large from group payloads | Would return 413; logs show 200. |
|
||||
| Panic in handler | Would log `webhook handler panic` (`webhook.go:75-81`). Nothing. |
|
||||
| Auth denial (Visibility check) | Help/info commands are `VisibilityPublic` (cf. `dispatcher.go:26`) — auth.Permits returns true unconditionally. Also a denial would silently return AFTER the handler closure runs, so the metric `IncCommand` at `dispatcher.go:63` would NOT fire. We'd see no metrics — same outward symptom — but a private-chat `/trade_stats` did increment, so the path works when match succeeds. Auth not the issue. |
|
||||
| Module disabled in groups | No code path filters by chat type in dispatch. `wordle/handlers.go:219`, `loldle/handlers.go:247` use `Chat.Type == ChatTypePrivate` only as a presentation switch (DM uses richer formatting); they do not gate command matching. |
|
||||
| Telegram delivering group updates to wrong endpoint | Single Function URL; only one webhook registered. |
|
||||
|
||||
## Proposed fix (not applied — awaiting your decision)
|
||||
|
||||
**Option A — Local wrapper using `MatchFunc` (recommended)**
|
||||
|
||||
Replace `b.RegisterHandler(..., bot.MatchTypeCommand, ...)` in `internal/modules/dispatcher.go:52-71` with `b.RegisterHandlerMatchFunc(matchFunc, ...)`, where `matchFunc` strips the trailing `@<bot-username>` from the entity bytes before comparing. Single localized change; no library fork.
|
||||
|
||||
Sketch:
|
||||
```go
|
||||
matchCmd := func(name string) bot.MatchFunc {
|
||||
return func(update *models.Update) bool {
|
||||
msg := update.Message
|
||||
if msg == nil {
|
||||
return false
|
||||
}
|
||||
for _, e := range msg.Entities {
|
||||
if e.Type != models.MessageEntityTypeBotCommand || e.Offset != 0 {
|
||||
continue
|
||||
}
|
||||
tok := msg.Text[e.Offset+1 : e.Offset+e.Length]
|
||||
if i := strings.IndexByte(tok, '@'); i >= 0 {
|
||||
tok = tok[:i]
|
||||
}
|
||||
if tok == name {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Pros: zero library upgrade pressure; precise behavior; easy to add a unit test (which the upstream library lacks).
|
||||
Cons: bypasses `bot.MatchTypeCommand`'s niceties (none we use); adds ~15 LOC + tests.
|
||||
|
||||
**Option B — Force users to type bare `/help`**
|
||||
|
||||
Documented workaround only. Doesn't fix the bot. Reject.
|
||||
|
||||
**Option C — File upstream issue / PR**
|
||||
|
||||
Worth doing in parallel, but blocked on upstream review / release timeline. Don't wait.
|
||||
|
||||
Recommend Option A + a fresh unit test in `internal/modules/dispatcher_test.go` covering both `/help` and `/help@miti99bot` against a registered `"help"` handler.
|
||||
|
||||
## Secondary findings (out of scope for this report)
|
||||
|
||||
1. `[TGBOT] [UPDATE]` log lines (`2026/05/16 04:22:02 [TGBOT] [UPDATE] &{ID:... Message:0x40002dc488 ...}`) print struct pointers, not contents. They are emitted by `bot.WithDebug(...)` in the go-telegram lib. Recommend adding a thin pre-dispatch log line in `internal/telegram/webhook.go:82` (just before `b.ProcessUpdate`) that records `update_id`, `chat.id`, `chat.type`, and `message.text` (truncated, no PII concerns since group IDs are not secret). Would have made this exact bug trivially observable instead of requiring a 3-source triangulation. ~10 LOC.
|
||||
|
||||
2. Library version `v1.20.0` was released some time back; the head of the repo may or may not have this fixed — worth a quick GitHub check before Option C.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
- Confirm with user whether they want Option A applied now (single phase, ~30 min) or queued behind other work.
|
||||
- Do you want the extra dispatch-time log line (secondary finding 1) included in the same fix?
|
||||
@@ -1,143 +0,0 @@
|
||||
# Documentation Audit Report
|
||||
|
||||
## Summary
|
||||
|
||||
Audited 12 documentation files covering architecture, module setup, deployment, code standards, and module READMEs. Found **3 major stale sections** related to module implementation status (wordle/loldle described as stubs when now fully implemented) and **1 test count discrepancy**. No broken links detected. Modular architecture docs are accurate.
|
||||
|
||||
## Doc Files Inventory
|
||||
|
||||
| File | Purpose | Status |
|
||||
|------|---------|--------|
|
||||
| README.md | Top-level overview, setup, deploy, troubleshooting | **MAJOR-DRIFT** |
|
||||
| CLAUDE.md | Dev guidance, commands, module contract, testing | **FRESH** |
|
||||
| docs/architecture.md | Deep dive: cold-start, registry, storage, crons, deploy | **MINOR-DRIFT** |
|
||||
| docs/adding-a-module.md | Step-by-step module authoring guide | **FRESH** |
|
||||
| docs/code-standards.md | Formatting, JSDoc, file org, naming, testing | **FRESH** |
|
||||
| docs/codebase-summary.md | Tech stack, active modules table, data flows | **MAJOR-DRIFT** |
|
||||
| docs/deployment-guide.md | CF setup, KV, D1, secrets, deploy steps, rollback | **FRESH** |
|
||||
| docs/using-d1.md | When to use D1 vs KV, SQL API, migration examples | **FRESH** |
|
||||
| docs/using-cron.md | Cron syntax, handler signature, examples | **FRESH** |
|
||||
| src/modules/wordle/README.md | Commands, architecture, KV schema | **FRESH** |
|
||||
| src/modules/loldle/README.md | Commands, architecture, KV schema | **MAJOR-DRIFT** |
|
||||
| src/modules/misc/README.md | Commands, KV demo, schema | **FRESH** |
|
||||
|
||||
## Detailed Findings
|
||||
|
||||
### 1. README.md — MAJOR-DRIFT
|
||||
|
||||
**Lines 14, 67-68: Test count and module status discrepancies**
|
||||
|
||||
- **Line 14 claim:** "105+ vitest unit tests"
|
||||
- **Actual:** 200 tests (verified: `npm test` output shows "Tests 200 passed")
|
||||
- **Fix:** Update to "200+ vitest unit tests"
|
||||
|
||||
- **Line 67 claim:** "wordle/ # stub — proves plugin system"
|
||||
- **Line 68 claim:** "loldle/ # stub"
|
||||
- **Actual:** Both are now full implementations:
|
||||
- Wordle: 4 commands (guessing game, new round, giveup, stats) with KV state, render, daily word, compare logic
|
||||
- Loldle: 4 commands (guessing game, new round, giveup, stats) with KV state, champions data, compare logic
|
||||
- Commit 8a9a6af: "feat(wordle): port classic 5-letter guessing game"
|
||||
- **Fix:** Change line 67 to "wordle/ # Classic 5-letter word guessing game (full impl)" and line 68 to "loldle/ # League of Legends champion guessing game (full impl)"
|
||||
|
||||
### 2. docs/codebase-summary.md — MAJOR-DRIFT
|
||||
|
||||
**Lines 25-26: Module status table outdated**
|
||||
|
||||
| Row | Claim | Actual |
|
||||
|-----|-------|--------|
|
||||
| `wordle` | "Status: Stub" | Full implementation: `/wordle`, `/wordle_new`, `/wordle_giveup`, `/wordle_stats` |
|
||||
| `wordle` | "Commands: `/wordle`, `/wstats`, `/konami`" | Wrong commands listed; actual: `/wordle`, `/wordle_new`, `/wordle_giveup`, `/wordle_stats` (all public) |
|
||||
| `wordle` | "Storage: —" | Uses KV (see src/modules/wordle/README.md) |
|
||||
| `loldle` | "Status: Stub" | Full implementation: `/loldle`, `/loldle_new`, `/loldle_giveup`, `/loldle_stats` |
|
||||
| `loldle` | "Commands: `/loldle`, `/ggwp`" | Missing 3 commands; actual: `/loldle`, `/loldle_new`, `/loldle_giveup`, `/loldle_stats` |
|
||||
| `loldle` | "Storage: —" | Uses KV (see src/modules/loldle/index.js lines 10, 16) |
|
||||
|
||||
**Fix:** Update the "Active Modules" table rows for wordle and loldle with actual command counts, visibility, and KV storage.
|
||||
|
||||
### 3. docs/architecture.md — MINOR-DRIFT
|
||||
|
||||
**Line 33: Module classification outdated**
|
||||
|
||||
- **Line 33 claim:** "wordle/ loldle/ — stub modules proving the plugin system"
|
||||
- **Actual:** Both are now production implementations with full game logic
|
||||
- **Fix:** Update to "wordle/ loldle/ — classic word/champion guessing games (full implementations)" or remove stub reference
|
||||
|
||||
**Line 362: Test count**
|
||||
|
||||
- **Line 362 claim:** "105 tests run in ~500ms"
|
||||
- **Actual:** 200 tests run in ~2.26s (from `npm test`)
|
||||
- **Fix:** Update to "200 tests run in ~2.26s"
|
||||
|
||||
### 4. src/modules/loldle/README.md — MAJOR-DRIFT
|
||||
|
||||
**Lines 1-3: Module described as stub**
|
||||
|
||||
- **Current claim:** "League of Legends guessing game — currently a stub proving the plugin system."
|
||||
- **Actual:** Full implementation with handlers imported (line 8: `import { handleGiveup, handleLoldle, handleNew, handleStats } from "./handlers.js"`), 4 commands, KV state (lines 10-17)
|
||||
- **Fix:** Rewrite to match wordle/README.md pattern: describe the 4 commands, handlers, architecture (handlers.js, compare.js, lookup.js, daily.js, render.js, state.js, champions-data.js), and KV schema
|
||||
|
||||
**Line 15: "No KV usage currently"**
|
||||
|
||||
- **Actual:** Module has `init` hook and KV state management (lines 14-17)
|
||||
- **Fix:** Document the `loldle:` namespace and game/stats keys (same pattern as wordle)
|
||||
|
||||
**Lines 6-16: Commands are stubs with stub responses**
|
||||
|
||||
- **Actual:** Commands have real handler implementations (handlers.js exists with 400+ LOC)
|
||||
- **Fix:** Remove "stub" references, document actual commands and their behavior
|
||||
|
||||
### 5. Cross-Reference Checks
|
||||
|
||||
**Internal links verified:**
|
||||
|
||||
- `README.md` → `docs/adding-a-module.md` ✓ (exists, correct relative path)
|
||||
- `README.md` → `docs/architecture.md` ✓
|
||||
- `README.md` → `docs/using-d1.md` ✓
|
||||
- `README.md` → `docs/using-cron.md` ✓
|
||||
- `README.md` → `docs/deployment-guide.md` ✓
|
||||
- `docs/architecture.md` → `src/modules/<name>/README.md` ✓ (pattern reference, not broken link)
|
||||
- All plan references in README.md reference existing directories ✓
|
||||
|
||||
**No broken links found.**
|
||||
|
||||
### 6. Missing Documentation
|
||||
|
||||
None. All required docs exist:
|
||||
|
||||
- ✓ Top-level README with setup, deploy, troubleshooting
|
||||
- ✓ Architecture deep-dive
|
||||
- ✓ Adding a module guide
|
||||
- ✓ Code standards
|
||||
- ✓ Codebase summary
|
||||
- ✓ Deployment guide (KV + D1)
|
||||
- ✓ D1 usage guide
|
||||
- ✓ Cron usage guide
|
||||
- ✓ Per-module READMEs (wordle, loldle, misc, trading, util)
|
||||
|
||||
Note: `docs/todo.md` exists but appears to be an internal tracking doc, not user-facing documentation.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Fix List
|
||||
|
||||
### Immediate (blocking user confusion)
|
||||
|
||||
1. **README.md line 14:** Update "105+ vitest" → "200+" (test count)
|
||||
2. **README.md lines 67-68:** Remove "stub" label from wordle/loldle in architecture snapshot
|
||||
3. **docs/codebase-summary.md lines 25-26:** Update module status table (wordle/loldle to "Complete", fix commands, add KV storage)
|
||||
4. **src/modules/loldle/README.md:** Full rewrite to describe actual implementation (4 commands, handlers, KV schema) not stub
|
||||
|
||||
### Secondary (clarity improvement)
|
||||
|
||||
5. **docs/architecture.md line 33:** Remove "stub" reference from module list description
|
||||
6. **docs/architecture.md line 362:** Update test count from "105" → "200" and runtime from "~500ms" → "~2.26s"
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Loldle module is fully implemented (files: handlers.js, compare.js, lookup.js, daily.js, render.js, state.js, champions-data.js, champions.json) but its README still describes it as a stub — this is the most glaring discrepancy.
|
||||
- Wordle module is correctly documented in its own README but incorrectly labeled "stub" in architecture snapshot and codebase summary.
|
||||
- Test count increased from 105 to 200 (likely due to additional trading module tests or recent test additions) — this is a growth metric worth celebrating.
|
||||
- All module-specific README files are accurate except loldle.
|
||||
- No architectural issues, just stale descriptive text that contradicts code reality.
|
||||
@@ -1,80 +0,0 @@
|
||||
# Leaguepedia API — Verification Report
|
||||
|
||||
**Date:** 2026-04-21
|
||||
**Purpose:** Verify the Leaguepedia MediaWiki/Cargo API can provide today / this-week LoL matches for a miti99bot module. No implementation, verification only.
|
||||
**Verdict:** **YES — usable.** Endpoint is public, no auth, returns structured JSON. One caveat on `where=` clauses needs rechecking from Cloudflare Workers egress.
|
||||
|
||||
---
|
||||
|
||||
## Endpoint
|
||||
|
||||
- Base: `https://lol.fandom.com/api.php`
|
||||
- Action: `cargoquery` (MediaWiki Cargo extension)
|
||||
- Auth: none
|
||||
- Format: `format=json` → clean `{cargoquery:[{title:{…}}]}` payload
|
||||
- Related `mw.Api` JS wrapper (doc-wikimedia link) works the same; for a Worker we use plain `fetch`, not `mw.Api`
|
||||
|
||||
## Relevant table: `MatchSchedule`
|
||||
|
||||
Primary table for both upcoming and played matches. Confirmed fields (live API, 2026-04-21):
|
||||
|
||||
| Field | Type | Example |
|
||||
|---|---|---|
|
||||
| `DateTime_UTC` | datetime | `"2026-06-14 09:00:00"` |
|
||||
| `Team1`, `Team2` | string | `"T1"`, `"TBD"` |
|
||||
| `Tournament` | string | Tournament name/slug |
|
||||
| `BestOf` | int | |
|
||||
| `Winner` | string | empty until played |
|
||||
| `OverviewPage` | string | wiki page for tournament |
|
||||
| `_pageName` | string | wiki row page |
|
||||
|
||||
Complementary tables: `Tournaments` (metadata), `ScoreboardGames` (per-game stats), `Teams`.
|
||||
|
||||
## Query syntax (verified working)
|
||||
|
||||
Use **table + field aliases** — the bare form `fields=MatchSchedule.DateTime_UTC` hits an `MWException`. Alias form is the idiomatic Leaguepedia convention:
|
||||
|
||||
```
|
||||
tables=MatchSchedule=MS
|
||||
fields=MS.DateTime_UTC=DateTime, MS.Team1=T1, MS.Team2=T2, MS.Tournament=Tournament
|
||||
order_by=MS.DateTime_UTC ASC
|
||||
limit=20
|
||||
```
|
||||
|
||||
Live sample (no where-filter) returned real rows. Ordering, limit, and aliasing all confirmed working.
|
||||
|
||||
Intended week-window query (to be re-verified from CF Worker egress):
|
||||
|
||||
```
|
||||
where=MS.DateTime_UTC >= "2026-04-21 00:00:00"
|
||||
AND MS.DateTime_UTC < "2026-04-28 00:00:00"
|
||||
```
|
||||
|
||||
## Limitations & operational notes
|
||||
|
||||
- **Strict anonymous rate limit.** From a single shared egress IP the API throttled after 1–2 req/min with `ratelimited`. Mitigations for Workers:
|
||||
- Use Worker's distributed egress (many IPs) — in practice won't hit the same bucket
|
||||
- Cache responses in KV (e.g. 60–300 s for upcoming schedule, 5–15 min for results)
|
||||
- Use `cf: { cacheTtl, cacheEverything: true }` on `fetch`
|
||||
- **User-Agent required.** Fandom's policy expects a contact UA, e.g. `miti99bot/0.1 (https://t.me/miti99bot; minhtienit99@gmail.com)`.
|
||||
- **Help page is Cloudflare-challenged.** `https://lol.fandom.com/wiki/Help:Leaguepedia_API` returns 403 to non-browser UAs — consult it from a browser, not from `fetch` code.
|
||||
- **No official JS SDK.** MediaWiki's `mw.Api` is on-wiki JS only. Community Python wrapper (`mwrogue` / `leaguepedia_parser`) is the reference implementation — we port the query shape, not the lib.
|
||||
- **`where=` clause returned MWException from this verification IP** even on trivial filters (`MS.Team1="T1"`). Likely an upstream filter on the shared egress, not a protocol limitation — the exact form is documented and heavily used. **Needs one confirmation curl from a CF Worker before building on it.**
|
||||
|
||||
## Feasibility verdict
|
||||
|
||||
| Requirement | Feasible? | Notes |
|
||||
|---|---|---|
|
||||
| Fetch today's matches | ✅ | `DateTime_UTC >= today AND < tomorrow` |
|
||||
| Fetch this week's matches | ✅ | 7-day window on `DateTime_UTC` |
|
||||
| Filter by region/league | ✅ | `Tournament LIKE "LCK%"` or `OverviewPage` |
|
||||
| Include results/winners | ✅ | `Winner`, `BestOf` already on row |
|
||||
| Run from Cloudflare Worker | ✅ | plain `fetch` + JSON; add UA + KV cache |
|
||||
| Scheduled daily digest | ✅ | fits existing `cron-dispatcher.js` pattern |
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Does `where=` with comparison operators (`>=`, `<`) work from CF Worker egress, or do we need to alternate filter form (`HOLDS`, `LIKE`, full-table-scan + client-side filter)?
|
||||
2. Timezone UX — show UTC, VN time (UTC+7), or let the `/matches` command take a region arg?
|
||||
3. Caching window — ~60 s for "live today" vs ~5 min for week-view; confirm TTL with a real command spec before implementing.
|
||||
4. Do we want the command to also surface `Winner`/score once a match has finished, or keep it schedule-only?
|
||||
@@ -1,55 +0,0 @@
|
||||
# Leaguepedia / Fandom API — Auth Token Verification
|
||||
|
||||
**Date:** 2026-04-21
|
||||
**Follow-up to:** `researcher-260421-0845-leaguepedia-api-verification.md`
|
||||
**Question:** Can we register and use a token to avoid the rate limit?
|
||||
**Verdict:** **No useful token available. Caching + CF Worker egress is the right answer.**
|
||||
|
||||
---
|
||||
|
||||
## What's NOT available on Fandom
|
||||
|
||||
| Mechanism | Status | Evidence |
|
||||
|---|---|---|
|
||||
| `Special:BotPasswords` | Disabled | 403 CF + disabled in UCP platform (documented in Fandom community) |
|
||||
| OAuth 1.0a / 2.0 (`Special:OAuthConsumerRegistration`) | Not offered | 403; Fandom never enabled the OAuth extension |
|
||||
| `action=clientlogin` (MW native login) | Disabled | `authmanagerinfo` returns only `RememberMeAuthenticationRequest`, no password field |
|
||||
| WMF-style API-key header | N/A | MediaWiki has no such thing; Fandom has none either |
|
||||
|
||||
```bash
|
||||
# Live probe
|
||||
curl '.../api.php?action=query&meta=authmanagerinfo&amirequestsfor=login&format=json'
|
||||
# → only returns RememberMeAuthenticationRequest — native login is off
|
||||
```
|
||||
|
||||
## What Fandom *does* expose
|
||||
|
||||
- **Helios SSO** at `services.fandom.com/mobile-fandom-app/fandom-auth/login`
|
||||
- POST `username` + `password` → `access_token` cookie
|
||||
- Used by Leaguepedia's official `mwcleric` Python lib (`LoginCredentials`)
|
||||
- Cookie is carried on subsequent `api.php` requests from same session
|
||||
|
||||
## Does an authenticated cookie lift the rate limit?
|
||||
|
||||
**No, not meaningfully.**
|
||||
|
||||
- MediaWiki's `noratelimit` right only belongs to specific wiki groups (`sysop`, `bot`). Regular logged-in users have the same API limits as anonymous.
|
||||
- Joining the `bot` group on Leaguepedia requires wiki-admin (Leaguepedia staff) approval — not practical for a side project.
|
||||
- The throttling we hit earlier is **Fandom's Cloudflare-edge IP rate limit**, which is session-agnostic. Auth cookies don't bypass it.
|
||||
- `siprop=ratelimits` is stripped on Fandom (`Unrecognized value`) — we can't even enumerate the limits.
|
||||
|
||||
## Right answer for miti99bot (Cloudflare Worker)
|
||||
|
||||
No token registration needed. Mitigate via:
|
||||
|
||||
1. **Edge caching** — `fetch(url, { cf: { cacheTtl: 60, cacheEverything: true } })`. Many bot users share one cached response.
|
||||
2. **KV result cache** — wrap the query in `create-store.js`, key = `matches:{from}:{to}`, TTL 60 s for "today", 5 min for "week".
|
||||
3. **Cron pre-warm** — add a module cron (existing `cron-dispatcher.js` pattern) that refreshes the week window every 15 min. Telegram `/matches` then reads pre-warmed cache.
|
||||
4. **CF Worker egress diversity** — Worker outbound IPs are many; per-IP buckets rarely hit 429 in practice.
|
||||
5. **Honor `Retry-After`** on 429 and surface "data momentarily unavailable" to the user instead of stalling.
|
||||
6. **Proper UA** — `miti99bot/0.1 (https://t.me/miti99bot; minhtienit99@gmail.com)` (already planned). Missing UA is itself a throttle signal on Fandom.
|
||||
|
||||
## Unresolved questions
|
||||
|
||||
1. Do CF Worker-origin fetches hit the same 429 as this shared egress does? (low risk — worth one real test before shipping)
|
||||
2. Is the module's read pattern bursty or steady? If steady, cron pre-warm + long TTL removes all pressure. If bursty (many users hit `/matches` at game time), KV cache is still the lever.
|
||||
@@ -1,395 +0,0 @@
|
||||
# Semantle API Alternatives Research Report
|
||||
**Date:** 2026-04-22 | **Project:** miti99bot
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Recommendation: Cloudflare Workers AI (BGE-base-en-v1.5) + Vectorize for production. Runner-up: Self-hosted precomputed embeddings (GloVe/R2).**
|
||||
|
||||
ConceptNet's unreliability (502 errors) requires immediate replacement. The consensus winner is **Cloudflare Workers AI embeddings** because:
|
||||
- Native to CF Workers (no fetch latency overhead, binding-based)
|
||||
- Proven at scale with edge inference (<100ms cold start, ~50-200ms per embedding)
|
||||
- Free tier: 10M input tokens/month (sufficient for 10k-20k single-word requests)
|
||||
- Cosine similarity built-in (768-dim BGE vectors match ConceptNet's semantic space)
|
||||
- Solves OOV detection via vocabulary check on ingestion
|
||||
|
||||
For "free tier only" projects (100% cost-conscious), **precomputed GloVe vectors in R2/KV** is feasible if you accept a one-time ~10MB upload and manual vocab checking.
|
||||
|
||||
---
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Provider | Auth | Free Tier | Latency | Similarity API | OOV Support | CF Workers Fit | Verdict |
|
||||
|----------|------|-----------|---------|----------------|-------------|----------------|---------|
|
||||
| **CF Workers AI (BGE)** | Binding | 10M tokens/mo | ~50-200ms | Cosine (768d) | Via vocab list | Native ⭐⭐⭐ | **RECOMMENDED** |
|
||||
| **CF Vectorize** | Binding | 30M dimensions/mo | ~30ms | Cosine query | Via storage | Native ⭐⭐⭐ | Best for scale |
|
||||
| **HuggingFace Inference** | API key | ~100 req/hr free | 500ms-2s cold | Cosine (384d) | Yes | Fetch OK ⭐⭐ | Viable but slow |
|
||||
| **OpenAI text-embedding-3-small** | API key | $0.02/1M tokens | ~200-500ms | Cosine | Yes | Fetch OK ⭐⭐ | Overkill, cost adds up |
|
||||
| **Replicate** | API key | Free w/ credits | 500ms+ | Cosine | Yes | Fetch OK ⭐⭐ | Slower than CF AI |
|
||||
| **Datamuse API** | None (free) | ∞ | ~100-300ms | No (ranking only) | Yes | Fetch OK ⭐⭐⭐ | **No similarity score** ❌ |
|
||||
| **GloVe (self-hosted KV/R2)** | None | ✓ (one-time) | ~10-50ms | Cosine (300d) | Manual | Fastest ⭐⭐⭐ | Great for small vocab |
|
||||
| **Word2Vec REST APIs** | Varies | Mostly down | Variable | Cosine | Yes | Fetch | Dead/unreliable |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Option Analysis
|
||||
|
||||
### 1. ⭐ **Cloudflare Workers AI (BGE-base-en-v1.5) — RECOMMENDED**
|
||||
|
||||
**What it does:** Generates 768-dimensional sentence/word embeddings via BAAI's BGE model, runs on CF edge infrastructure.
|
||||
|
||||
**Call pattern:**
|
||||
```javascript
|
||||
const embedding = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
|
||||
text: "word_to_embed"
|
||||
});
|
||||
// Returns { shape: [768], data: [float...], pooling: "mean" }
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- No fetch overhead — native CF Workers binding, direct to inference layer
|
||||
- Cold start <100ms, typical latency 50-200ms for single words (ideal for game speed)
|
||||
- 768 dimensions match semantic similarity expectations (ConceptNet-like quality)
|
||||
- Free tier: 10M input tokens/month = ~100k single-word embeddings (sufficient for 100-200 games/day if each game = 5 guesses)
|
||||
- Cosine similarity scoring built-in at client (just compute `dot(a, b) / (||a|| * ||b||)`)
|
||||
- Paid tier: $0.067/M input tokens (cheap relative to OpenAI)
|
||||
|
||||
**Cons:**
|
||||
- Requires Workers paid plan for Workers AI access (~$15/month base, then token-metered on top)
|
||||
- OOV detection not native to embeddings — must maintain separate vocab list or batch-verify words
|
||||
- 768 dimensions = ~3KB per cached embedding; not huge but adds up for 10k words
|
||||
|
||||
**OOV handling:** Pre-load Google-10000-english into KV (~300KB), check membership before calling similarity. Or call embeddings on both words; if confidence is suspiciously uniform, flag OOV.
|
||||
|
||||
**Real-world latency:** One production report: "global latency under 80ms p50" for embeddings via CF Workers.
|
||||
|
||||
**Cost at scale:**
|
||||
- 100 games/day × 5 guesses/game × 2 words/guess = 1000 embeddings/day
|
||||
- 1000 × 30 days = 30k tokens/month → free tier sufficient (10M tokens)
|
||||
- At paid tier: negligible ($0.002/month)
|
||||
|
||||
**Recommendation:** Ship with this. It's the path of least resistance and best latency.
|
||||
|
||||
---
|
||||
|
||||
### 2. ⭐⭐ **Cloudflare Vectorize — BEST FOR SCALE**
|
||||
|
||||
**What it does:** Managed vector database; store embeddings by word key, query by cosine similarity.
|
||||
|
||||
**Architecture:**
|
||||
1. Pre-compute all 10k words' embeddings via Workers AI in a setup task
|
||||
2. Store in Vectorize index (768 dimensions, cosine metric)
|
||||
3. At game start, query Vectorize: `index.query(targetWordEmbedding, topK=1)` to verify it's in vocab
|
||||
4. On each guess, fetch both embeddings from Vectorize (cached) + compute similarity client-side OR use Vectorize search with custom scoring
|
||||
|
||||
**Pros:**
|
||||
- Median query latency 30-31ms (faster than Workers AI)
|
||||
- Free tier: 30M queried vectors/month = ~1M queries/month (plenty for hobby traffic)
|
||||
- Pre-computed vectors cached globally (no re-embedding per-game)
|
||||
- Deterministic: same two words always return same score
|
||||
|
||||
**Cons:**
|
||||
- Setup overhead: must pre-compute and upload all 10k embeddings once
|
||||
- Requires Vectorize binding + Workers AI for initial embedding generation
|
||||
- Query cost if not in free tier ($0.01/1M queried dimensions)
|
||||
- Overkill for simple 10k-word game; adds operational complexity
|
||||
|
||||
**Cost at scale:** Negligible if free tier applies. At paid tier (unlikely): $0.000001 per query.
|
||||
|
||||
**Recommendation:** Consider after MVP ships. For initial launch, Workers AI direct is simpler. Migrate to Vectorize if you scale beyond 100k guesses/month or want <50ms guarantees.
|
||||
|
||||
---
|
||||
|
||||
### 3. **HuggingFace Inference API — VIABLE BUT SLOW**
|
||||
|
||||
**What it does:** API to sentence-transformers (all-MiniLM-L6-v2, all-mpnet-base-v2, etc.). Free tier has rate limits.
|
||||
|
||||
**Call pattern:**
|
||||
```javascript
|
||||
const response = await fetch("https://api-inference.huggingface.co/models/sentence-transformers/all-MiniLM-L6-v2", {
|
||||
headers: { Authorization: `Bearer ${HF_TOKEN}` },
|
||||
method: "POST",
|
||||
body: JSON.stringify({ inputs: ["word"] })
|
||||
});
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Free tier available (~few hundred req/hr limit)
|
||||
- Many sentence-transformer models to choose from
|
||||
- OOV is implicit (any word gets an embedding)
|
||||
- No account setup beyond free HF profile
|
||||
|
||||
**Cons:**
|
||||
- Cold start latency: 30-60 seconds on free tier (models unloaded on-demand)
|
||||
- Even after warmup, 500ms-2s per request
|
||||
- Rate limits strict on free tier (~100 requests/hour)
|
||||
- 384 dimensions (smaller than BGE, may affect quality)
|
||||
- Fetch round-trip from CF Workers adds another 50-100ms
|
||||
|
||||
**Adoption risk:** Free tier is unreliable for production games (rate limits + cold starts violate 5s timeout). Requires paid tier ($9/month for 2M credits) to be usable.
|
||||
|
||||
**Recommendation:** Skip. Workers AI is superior and same cost (or free).
|
||||
|
||||
---
|
||||
|
||||
### 4. **OpenAI text-embedding-3-small — OVERKILL**
|
||||
|
||||
**What it does:** Industry-standard embeddings via OpenAI API.
|
||||
|
||||
**Pros:**
|
||||
- Best-in-class embeddings quality
|
||||
- Simple API, well-documented
|
||||
- No cold starts
|
||||
|
||||
**Cons:**
|
||||
- $0.02/1M tokens = $0.0002 per word embedding (adds up fast)
|
||||
- 100 games/day × 5 guesses × 2 words = 1000 tokens/day = $0.02/day = $0.60/month
|
||||
- Overkill for single-word similarity (trained on sentences, wasted capacity)
|
||||
- Slower than CF Workers AI (200-500ms typical)
|
||||
- Requires API key in Worker env (security overhead)
|
||||
- Rate limits apply
|
||||
|
||||
**Recommendation:** Too expensive and slow. Only if you value maximum embedding quality above cost/latency (not applicable here).
|
||||
|
||||
---
|
||||
|
||||
### 5. **Replicate — SLIGHTLY WORSE CF OPTION**
|
||||
|
||||
**What it does:** Cloud inference platform supporting embeddings models (multilingual-e5-large, all-mpnet-base-v2, etc.).
|
||||
|
||||
**Pros:**
|
||||
- Competitive pricing (~$0.11 per run vs OpenAI's $0.51)
|
||||
- Wide model choice
|
||||
- Cloudflare acquired Replicate in Nov 2025 → may improve integration
|
||||
|
||||
**Cons:**
|
||||
- Latency 500ms+ (slower than Workers AI + fetch round-trip)
|
||||
- Requires separate account + API key
|
||||
- Not a binding, so full fetch overhead from CF Workers
|
||||
- Pricing less clear (charged by time, not tokens)
|
||||
|
||||
**Recommendation:** Skip in favor of Workers AI. Close competitor if CF AI binding becomes unavailable.
|
||||
|
||||
---
|
||||
|
||||
### 6. **Datamuse API — INSUFFICIENT**
|
||||
|
||||
**What it does:** Free word relationship API (rhyming, meaning, spelling, sound-alike).
|
||||
|
||||
**Why it fails:**
|
||||
- **No numeric similarity score between two words.** Returns ranked lists of related words, not pairwise scores.
|
||||
- Scores have "no interpretable meaning" (per official API docs); used for ranking results only.
|
||||
- Cannot compute "target vs guess" similarity in the Semantle game format.
|
||||
|
||||
**Recommendation:** Rejected. Core requirement not met.
|
||||
|
||||
---
|
||||
|
||||
### 7. ⭐⭐⭐ **Self-Hosted Precomputed Embeddings (GloVe/R2) — COST-OPTIMAL**
|
||||
|
||||
**What it does:** Pre-download GloVe vectors (300d, 6B tokens, 822MB), extract vectors for google-10k words, store in R2, load into KV for fast lookup.
|
||||
|
||||
**Architecture:**
|
||||
```
|
||||
1. Download glove.6B.300d.txt (free, public domain)
|
||||
2. Extract 10k words + vectors → ~30MB JSON
|
||||
3. Gzip → ~8-10MB, store in R2 (free tier includes 10GB)
|
||||
4. On Worker startup: fetch from R2 (or lazy-load per region), cache in KV
|
||||
5. similarity(a, b) = cosine(glove[a], glove[b])
|
||||
6. OOV: word not in glove dict → return null
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Truly free (no API calls, no Workers AI quota)
|
||||
- Fastest: cosine similarity is ~5-10ms JS computation
|
||||
- Completely deterministic, no API reliability concerns
|
||||
- Can pre-compute all 10k × (10k-1) / 2 similarity pairs into KV (expensive, not needed)
|
||||
- Offline-first: no upstream dependency
|
||||
|
||||
**Cons:**
|
||||
- One-time setup: download, parse, compress, upload to R2 (~30 min work)
|
||||
- GloVe is ~7 years old; quality lag behind BGE (but still good for word similarity)
|
||||
- 300 dimensions vs 768 in BGE (may affect semantic quality, but acceptable for Semantle)
|
||||
- ~3KB per cached embedding × 10k words = 30MB in memory if fully loaded (within CF Worker limits, but tight)
|
||||
- Manual OOV check: need separate google-10000-english list in KV
|
||||
|
||||
**Adoption risk:** Low. GloVe is stable, no API changes. Can cache vectors indefinitely. Only upgrade path is re-download if you want newer embeddings.
|
||||
|
||||
**Real cost:** $0 per month.
|
||||
|
||||
**Recommendation:** Highly viable for hobby projects. Better than any API if you're cost-optimizing and accept slight quality trade-off. Hybrid: use GloVe for now, upgrade to Workers AI if you need better semantics later.
|
||||
|
||||
---
|
||||
|
||||
### 8. **Word2Vec REST APIs — OBSOLETE**
|
||||
|
||||
Search found several GitHub repos (quhfus/DoSeR, 3Top/word2vec-api, bmzhao/word2vec-rest-api) but:
|
||||
- None maintained recently (last commits 2020-2022)
|
||||
- No public instances available
|
||||
- Self-hosting them defeats the purpose (you'd run a server alongside CF Workers)
|
||||
|
||||
**Recommendation:** Skip. GloVe is better maintained if you want self-hosted embeddings.
|
||||
|
||||
---
|
||||
|
||||
## Migration Sketch for Recommended Option
|
||||
|
||||
### Implementation Plan: Cloudflare Workers AI (BGE) + Vectorize Fallback
|
||||
|
||||
**Phase 1: Workers AI (MVP)**
|
||||
|
||||
Modify `api-client.js`:
|
||||
|
||||
```javascript
|
||||
export function createClient(options = {}) {
|
||||
const { env, useVectorize = false } = options;
|
||||
|
||||
// Vocab check: load google-10000-english from KV
|
||||
async function isInVocab(word) {
|
||||
const vocab = await env.KV.get("semantle:vocab-10000");
|
||||
if (!vocab) return true; // pessimistic: assume yes if missing
|
||||
return vocab.includes(word.toLowerCase());
|
||||
}
|
||||
|
||||
return {
|
||||
async randomWord() {
|
||||
// Pick from pool; verify it's "in vocab" by checking if we can embed it
|
||||
for (let i = 0; i < MAX_RANDOM_ATTEMPTS; i++) {
|
||||
const candidate = pickFromPool();
|
||||
try {
|
||||
const inVocab = await isInVocab(candidate);
|
||||
if (inVocab) return { word: candidate, verified: true };
|
||||
} catch {
|
||||
// continue
|
||||
}
|
||||
}
|
||||
return { word: pickFromPool(), verified: false };
|
||||
},
|
||||
|
||||
async similarity(a, b) {
|
||||
const inVocabB = await isInVocab(b);
|
||||
if (!inVocabB) {
|
||||
return {
|
||||
a, b, canonical_a: a, canonical_b: b,
|
||||
in_vocab_a: true, in_vocab_b: false, similarity: null
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
// Call Workers AI to get embeddings
|
||||
const [embA, embB] = await Promise.all([
|
||||
env.AI.run("@cf/baai/bge-base-en-v1.5", { text: a }),
|
||||
env.AI.run("@cf/baai/bge-base-en-v1.5", { text: b })
|
||||
]);
|
||||
|
||||
// Compute cosine similarity
|
||||
const sim = cosineSimilarity(embA.data, embB.data);
|
||||
return {
|
||||
a, b, canonical_a: a, canonical_b: b,
|
||||
in_vocab_a: true, in_vocab_b: true,
|
||||
similarity: sim
|
||||
};
|
||||
} catch (err) {
|
||||
throw new UpstreamError("workers-ai embedding failed", { cause: err });
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function cosineSimilarity(vecA, vecB) {
|
||||
let dotProduct = 0, normA = 0, normB = 0;
|
||||
for (let i = 0; i < vecA.length; i++) {
|
||||
dotProduct += vecA[i] * vecB[i];
|
||||
normA += vecA[i] * vecA[i];
|
||||
normB += vecB[i] * vecB[i];
|
||||
}
|
||||
return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
|
||||
}
|
||||
```
|
||||
|
||||
**Phase 2: Setup Task**
|
||||
|
||||
Add to `wrangler.toml`:
|
||||
```toml
|
||||
[env.production]
|
||||
ai = true
|
||||
kv_namespaces = [{ binding = "KV", id = "..." }]
|
||||
```
|
||||
|
||||
Pre-populate KV with vocab list:
|
||||
```bash
|
||||
# Download google-10000-english, store in KV
|
||||
curl -s https://raw.githubusercontent.com/first20hours/google-10000-english/master/google-10000-english.txt | \
|
||||
jq -Rs 'split("\n") | map(select(length > 0))' | \
|
||||
npx wrangler kv:key put --binding=KV "semantle:vocab-10000" -
|
||||
```
|
||||
|
||||
**Phase 3: Vectorize (Optional, Future)**
|
||||
|
||||
Once MVP is stable:
|
||||
1. Pre-compute all 10k word embeddings
|
||||
2. Store in Vectorize index
|
||||
3. Replace `similarity()` with cached lookups
|
||||
4. ~30ms per query vs ~200ms (7x speedup, negligible for gameplay)
|
||||
|
||||
---
|
||||
|
||||
## Cost Breakdown (Monthly)
|
||||
|
||||
| Option | Setup | Per-Game (avg) | 100 games/mo | 1000 games/mo | 10k games/mo |
|
||||
|--------|-------|----------------|--------------|---------------|--------------|
|
||||
| Workers AI (BGE) | $0 | $0 (free tier) | Free | Free | $0.07 |
|
||||
| Vectorize | 1h | $0 (cached) | Free | Free | $0.001 |
|
||||
| HuggingFace (paid) | $0 | $0.0002 | $0.02 | $0.20 | $2.00 |
|
||||
| OpenAI | $0 | $0.0002 | $0.02 | $0.20 | $2.00 |
|
||||
| Replicate | $0 | $0.0002 | $0.02 | $0.20 | $2.00 |
|
||||
| GloVe (self-host) | 1h | $0 | Free | Free | Free |
|
||||
|
||||
---
|
||||
|
||||
## Recommendation Summary
|
||||
|
||||
| Use Case | Recommendation |
|
||||
|----------|---|
|
||||
| **MVP / Immediate fix** | Cloudflare Workers AI (BGE-base-en-v1.5) + Google-10k vocab in KV |
|
||||
| **Ultra cost-conscious** | GloVe vectors in R2 + KV (one-time setup, zero ongoing cost) |
|
||||
| **Production scale (>1k games/mo)** | Workers AI → migrate to Vectorize for caching |
|
||||
| **Maximum semantic quality** | Workers AI (BGE is excellent, no need for OpenAI overkill) |
|
||||
|
||||
**Ship recommendation:** Go with Workers AI. 50-200ms latency is acceptable for game UX (faster than ConceptNet ever was), free tier covers hobby traffic, and if you grow, Vectorize is a drop-in upgrade.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **GloVe quality for single-word semantics?** GloVe trained on document context; single-word embeddings may be noisier than BGE (which uses contrastive learning for dense retrieval). Needs A/B testing if semantics matter (probably doesn't for game).
|
||||
|
||||
2. **BGE "mean" vs "cls" pooling?** Current approach uses default "mean" pooling. Does "cls" (CLS token) pooling improve single-word similarity? Requires testing on Semantle target words.
|
||||
|
||||
3. **OOV detection robustness?** Relying on google-10000-english for vocab checking; what if player guesses a valid English word outside this list (e.g., "cryptocurrency")? Current approach: fallback to "not in vocabulary" (conservative). Could call embeddings on all words and use confidence/variance heuristics, but adds latency.
|
||||
|
||||
4. **Vectorize v2 latency in practice?** Cited 30-31ms median; but is that from CF Workers client or external? If external, add 50-100ms fetch. Need real-world benchmark from within a Worker.
|
||||
|
||||
5. **Workers AI quota enforcement?** 10M tokens/month free tier — is this enforced? What happens on overage? (Assumed: immediate billing, no auto-overage blocking, similar to other CF quotas.)
|
||||
|
||||
6. **Cloudflare API stability for Workers AI?** ConceptNet is failing; is Workers AI more reliable? (Assumption: yes, it's Cloudflare's own service, not external upstream. Still risk, but lower.)
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [Cloudflare BGE-base-en-v1.5 embeddings docs](https://developers.cloudflare.com/workers-ai/models/bge-base-en-v1.5/)
|
||||
- [Cloudflare Workers AI models](https://developers.cloudflare.com/workers-ai/models/)
|
||||
- [Cloudflare Vectorize pricing](https://developers.cloudflare.com/vectorize/platform/pricing/)
|
||||
- [Cloudflare Vectorize get-started](https://developers.cloudflare.com/vectorize/get-started/embeddings/)
|
||||
- [HuggingFace Inference API](https://huggingface.co/docs/api-inference/en/index)
|
||||
- [HuggingFace pricing](https://huggingface.co/docs/inference-providers/pricing)
|
||||
- [OpenAI text-embedding-3-small pricing](https://developers.openai.com/api/docs/pricing)
|
||||
- [Datamuse API docs](https://www.datamuse.com/api/)
|
||||
- [GloVe word embeddings Stanford NLP](https://nlp.stanford.edu/projects/glove/)
|
||||
- [google-10000-english corpus](https://github.com/first20hours/google-10000-english)
|
||||
- [Replicate embeddings models](https://replicate.com/collections/embedding-models)
|
||||
- [Cloudflare Workers AI latency benchmarks](https://www.kalviumlabs.ai/blog/production-ai-on-cloudflare-workers/)
|
||||
- [Embeddings API comparison 2026](https://supermemory.ai/blog/best-open-source-embedding-models-benchmarked-and-ranked/)
|
||||
- [Cloudflare KV + Vectorize integration](https://dev.to/andyjessop/building-ai-powered-second-brain-in-a-cloudflare-worker-with-cloudflare-vectorize-and-openai-23di)
|
||||
@@ -1,204 +0,0 @@
|
||||
# BGE-M3 Cosine Similarity Calibration for Semantle Clone
|
||||
|
||||
**Report Date:** 2026-04-22
|
||||
**Work Context:** Cloudflare Workers bot, Semantle-style word guessing
|
||||
**Model:** BAAI/bge-m3 (1024-dim, multilingual)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Your complaint (random words scoring 40-70%) is **mathematically valid** for high-dim embeddings. Raw cosine in 1024-dim space concentrates toward 0.3-0.4 for unrelated pairs due to high-dimensional geometry. Recommended fix: **percentile-stretch with sigmoid**, not linear rescale. Maps raw cosine ∈ [0.3, 1.0] → [0, 100] with tunable inflection. No precomputed vocab matrix needed; calibrates against empirical percentile anchors.
|
||||
|
||||
---
|
||||
|
||||
## Q1: Cosine Distribution for Random Pairs (BGE-M3)
|
||||
|
||||
### Findings
|
||||
- **BGE-M3 embedding dimension:** 1024-dim dense vectors (confirmed via Hugging Face model card)
|
||||
- **Random cosine baseline (1024-dim):** Beta(511.5, 511.5) distribution → mean ≈ 0, mode around 0.0–0.1, tail out to ~0.3 max for 99th percentile
|
||||
- **Empirical rule for high-dim (d=1024):** Among 10k random pairs, ~0% exceed cosine 0.3; ~99th percentile ≈ 0.25–0.3
|
||||
|
||||
### Key Insight
|
||||
Your observation is correct: random unrelated words naturally cluster around 0.35–0.5 because of high-dimensional geometry, not model failure. This is **expected mathematical behavior** for 1024-dim spaces per Beta distribution theory.
|
||||
|
||||
### Sources
|
||||
- [Sungwon Kim: Random Cosine Similarity Distribution](https://sungwon-kim.com/blog/2025/random-cosine-similarity/) — beta distribution parameterization
|
||||
- [BAAI/bge-m3 Model Card](https://huggingface.co/BAAI/bge-m3) — confirms 1024-dim dense output
|
||||
- [Vaibhav Garg Medium: Why Cosine Similarities Almost Always Positive](https://vaibhavgarg1982.medium.com/why-are-cosine-similarities-of-text-embeddings-almost-always-positive-6bd31eaee4d5) — high-dim concentration
|
||||
|
||||
---
|
||||
|
||||
## Q2: Original Semantle Score Formula
|
||||
|
||||
### Findings
|
||||
- **Semantle (semantle.com):** Uses GoogleNews-vectors-negative300 (Word2Vec, older model)
|
||||
- **Score formula:** `score = raw_cosine * 100`, range [-100, 100] in theory; [-34, 100] in practice
|
||||
- **No rescaling:** Semantle relies on Word2Vec's flatter cosine distribution (300-dim, older training) which naturally spreads unrelated pairs lower
|
||||
|
||||
### Key Insight
|
||||
Semantle **cannot be directly copied** — it worked because Word2Vec 300-dim spreads unrelated words lower naturally. BGE-M3 1024-dim has higher clustering. You need active calibration, not just multiplication.
|
||||
|
||||
### Sources
|
||||
- [Victoria Ritvo: Semantle Solver Blog](https://victoriaritvo.com/blog/semantle-solver/) — game mechanics
|
||||
- [Semantle FAQ](https://semantle.com/faq/) — confirms Word2Vec GoogleNews model
|
||||
- [Andy Chen: Writing a Semantle Solver](https://andychen.io/posts/2024-10-15-semantle-solver/) — reverse-engineering score logic
|
||||
|
||||
---
|
||||
|
||||
## Q3: Practical Calibration Techniques for Workers
|
||||
|
||||
### Option 1: Linear Rescale with Floor (Simplest)
|
||||
```javascript
|
||||
// Subtract empirical baseline, stretch
|
||||
const floor = 0.30; // 30th percentile for random pairs
|
||||
const ceil = 1.0; // Perfect match
|
||||
const raw_cosine = 0.45; // Example guess
|
||||
|
||||
const calibrated = Math.max(0, (raw_cosine - floor) / (ceil - floor) * 100);
|
||||
// 0.45 → (0.15 / 0.70) * 100 = 21.4 (unrelated, good)
|
||||
// 0.85 → (0.55 / 0.70) * 100 = 78.6 (related, good)
|
||||
```
|
||||
**Pros:** Zero overhead, 1 division.
|
||||
**Cons:** Sharp cliff at floor; doesn't distinguish weak vs strong similarity gracefully.
|
||||
|
||||
### Option 2: Sigmoid Stretch (Recommended)
|
||||
```javascript
|
||||
// Logistic function centered on mean of random distribution
|
||||
const logit = (x, floor = 0.30, center = 0.50, scale = 3.0) => {
|
||||
return 1.0 / (1.0 + Math.exp(-scale * (x - center)));
|
||||
};
|
||||
|
||||
const calibrated = (logit(cosine) - logit(floor)) / (1.0 - logit(floor)) * 100;
|
||||
// Adjustable `scale` controls inflection steepness
|
||||
```
|
||||
**Pros:** Smooth S-curve; tunable inflection; graceful tail-off for low scores.
|
||||
**Cons:** 2 exp() calls per guess (negligible on modern CPUs, fine on Workers).
|
||||
|
||||
### Option 3: Gamma/Power Curve
|
||||
```javascript
|
||||
const gamma = (x, floor = 0.30, exp = 2.0) => {
|
||||
const norm = Math.max(0, (x - floor) / (1.0 - floor));
|
||||
return Math.pow(norm, exp) * 100;
|
||||
};
|
||||
// Quadratic: even more aggressive separation, exp=2
|
||||
// Cubic: exp=3 for steeper curves
|
||||
```
|
||||
**Pros:** Cheap (one Math.pow); tunable exponent.
|
||||
**Cons:** Less smooth than sigmoid; may over-amplify mid-range.
|
||||
|
||||
### Option 4: Percentile Mapping (No Precomputed Matrix)
|
||||
Sample 50 random word pairs from your 10k vocab at round start, compute their cosines, use as local distribution anchor. Then map: `score = percentile_rank(guess_cosine, samples) * 100`.
|
||||
|
||||
**Pros:** Data-driven, adapts to actual vocab.
|
||||
**Cons:** Requires 50 cosine computations upfront; adds latency (~5–10ms if parallelized via Promise.all).
|
||||
|
||||
---
|
||||
|
||||
## Q4: Shipping Precomputed Reference Distribution
|
||||
|
||||
### Feasibility
|
||||
**Not recommended for Workers context:**
|
||||
- 10k vocab × 100 samples = 1M cosines → 4MB as float32, 1MB as int8
|
||||
- Bundle limit is typically 1–5 MB shared; eating 1MB for calibration matrix is wasteful
|
||||
- Worker inference budget better spent on actual embeddings (round-start + per-guess)
|
||||
|
||||
### Better Approach
|
||||
**Use Option 2 (Sigmoid)** with **static empirical constants** derived once from literature:
|
||||
- `floor = 0.30` (99th percentile of random baseline, universal for 1024-dim)
|
||||
- `center = 0.50` (midpoint of meaningful range, tunable per game difficulty)
|
||||
- `scale = 3.0` (controls inflection, tunable for warmth UX)
|
||||
|
||||
No matrix ship needed; constants are 12 bytes.
|
||||
|
||||
---
|
||||
|
||||
## Q5: Recommended Formula & Constants
|
||||
|
||||
### Algorithm: Sigmoid-Stretched Percentile
|
||||
|
||||
```javascript
|
||||
function calibrateScore(rawCosine) {
|
||||
// Empirical constants for BGE-M3 1024-dim
|
||||
const FLOOR = 0.30; // Random baseline (99th pct)
|
||||
const CENTER = 0.50; // Inflection point (tunable: 0.45–0.55)
|
||||
const SCALE = 3.0; // Steepness (tunable: 2.0–4.0)
|
||||
|
||||
// Sigmoid stretch
|
||||
const sigmoid = (x) => 1.0 / (1.0 + Math.exp(-SCALE * (x - CENTER)));
|
||||
|
||||
const raw_sig = sigmoid(rawCosine);
|
||||
const floor_sig = sigmoid(FLOOR);
|
||||
const one_sig = sigmoid(1.0);
|
||||
|
||||
// Normalize sigmoid range to [0, 100]
|
||||
const normalized = (raw_sig - floor_sig) / (one_sig - floor_sig);
|
||||
return Math.min(100, Math.max(0, normalized * 100));
|
||||
}
|
||||
|
||||
// Examples (CENTER=0.50, SCALE=3.0):
|
||||
// rawCosine=0.30 → score ≈ 0
|
||||
// rawCosine=0.40 → score ≈ 5
|
||||
// rawCosine=0.45 → score ≈ 20
|
||||
// rawCosine=0.50 → score ≈ 50 (inflection)
|
||||
// rawCosine=0.65 → score ≈ 85
|
||||
// rawCosine=0.90 → score ≈ 98
|
||||
```
|
||||
|
||||
### Tuning Knobs
|
||||
- **CENTER (0.45–0.55):** Move left for harder game (more low scores), right for easier.
|
||||
- **SCALE (2.0–4.0):** Higher = steeper cliff around inflection; lower = smoother spread.
|
||||
- **FLOOR (0.28–0.32):** Adjust if empirical random baseline differs.
|
||||
|
||||
### Why This Works
|
||||
1. **Respects geometry:** Accounts for 1024-dim clustering toward 0.3–0.5
|
||||
2. **Readable UX:** Unrelated (0.30–0.40) → 0–15; weak (0.45) → 20; strong (0.65+) → 80+
|
||||
3. **Tunable:** Constants easy to adjust without code changes
|
||||
4. **Fast:** One sigmoid + 3 arithmetic ops; sub-1ms on Workers
|
||||
|
||||
---
|
||||
|
||||
## Q6: Gotchas & Caveats
|
||||
|
||||
### 1. **Vietnamese vs English**
|
||||
BGE-M3 is multilingual trained; cosine distributions are **similar across languages** (symmetric training). Use same constants for both. Verify empirically if playing both languages heavily.
|
||||
|
||||
### 2. **Math.exp() Edge Cases**
|
||||
Sigmoid for very small x (< 0.1) → exp returns 0, might cause division issues. Clamp floor to 0.25 to be safe.
|
||||
|
||||
```javascript
|
||||
// Safe sigmoid
|
||||
const safe_sigmoid = (x) => Math.max(0.001, Math.min(0.999, 1.0 / (1.0 + Math.exp(-SCALE * (x - CENTER)))));
|
||||
```
|
||||
|
||||
### 3. **Round-to-Round Variance**
|
||||
Different target words have different average cosine distributions with their vocab (e.g., "cat" is closer to more animals than "fluorine" is). **This is expected.** Calibration is per-target, not global. If needed, add a per-target offset, but keep it small.
|
||||
|
||||
### 4. **Bundle Size**
|
||||
Sigmoid constants are negligible; no precomputed matrix needed. Stay under 10KB total.
|
||||
|
||||
### 5. **Testing**
|
||||
Before shipping:
|
||||
- Generate 100 random word pairs, confirm scores in [5, 25] range
|
||||
- Test 50 synonyms/strong neighbors, confirm scores in [70, 95] range
|
||||
- Test 20 hand-picked "warmth edge cases" (e.g., "run" vs "walk")
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Exact p50/p95 for BGE-M3 specifically:** No published distribution stats for bge-m3 random baselines; derived from beta-distribution math. Recommend empirical validation on your 10k vocab.
|
||||
2. **Optimal CENTER/SCALE for your UX:** Tuning is subjective (game difficulty). Recommend A/B testing with 2–3 different profiles.
|
||||
3. **Multilingual calibration drift:** Untested whether Vietnamese and English have identical random baselines; assume yes per symmetry, verify with ~1k random pairs of each.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [BAAI/bge-m3 Model Card (HF)](https://huggingface.co/BAAI/bge-m3)
|
||||
- [M3-Embedding Paper (arXiv:2402.03216)](https://arxiv.org/abs/2402.03216)
|
||||
- [Sungwon Kim: Random Cosine Distribution](https://sungwon-kim.com/blog/2025/random-cosine-similarity/)
|
||||
- [Sentence-Transformers Normalization (GitHub #1084)](https://github.com/UKPLab/sentence-transformers/issues/1084)
|
||||
- [Victoria Ritvo: Semantle Solver](https://victoriaritvo.com/blog/semantle-solver/)
|
||||
- [Blue Yonder: Text Embedding & Cosine Similarity](https://tech.blueyonder.com/text-embedding-and-cosine-similarity/)
|
||||
- [Cloudflare Vectorize Docs](https://developers.cloudflare.com/vectorize/get-started/embeddings/)
|
||||
@@ -1,357 +0,0 @@
|
||||
# Research: Loldle Ability & Splash Modes for Telegram Bot
|
||||
|
||||
**Date:** 2026-04-24
|
||||
**Context:** Adapting Loldle's image-based game modes into Telegram bot commands on Cloudflare Workers. Existing classic mode scrapes champion data from JS bundle; need feasibility analysis for Ability and Splash modes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Gameplay Mechanics
|
||||
|
||||
### Ability Mode
|
||||
- **What player sees:** Single zoomed-in ability icon (no kit context)
|
||||
- **Reveal mechanic:** No progressive zoom on guesses; user either guesses correctly or incorrectly
|
||||
- **Two-stage guessing:**
|
||||
- First: Identify the champion who owns the ability
|
||||
- Bonus: Identify which ability slot (Passive / Q / W / E / R) after champion is guessed
|
||||
- **Icon source:** One random ability per daily reset from pool of 5+ per champion
|
||||
- **Challenge:** 170+ champions × 5 abilities each = ~850+ unique icons; icon color/shape themes repeat across classes, making recognition difficult
|
||||
- **No explicit hint system:** Unlike Classic mode, ability mode provides no "closeness" feedback—binary win/loss only
|
||||
|
||||
### Splash Mode
|
||||
- **What player sees:** Highly zoomed-in crop of splash art (detail only: fragment of weapon, armor, background)
|
||||
- **Reveal mechanic:** Progressive zoom—each wrong guess zooms OUT further, revealing more of the full image
|
||||
- **Guessing limit:** Implied from "LoLdle Unlimited" variant; daily classic has some limit (exact number unconfirmed, but likely 6-8 based on Wordle convention)
|
||||
- **Art sources:** Base splash art or skin splash art (adds difficulty; same champion may have 5-10+ splash variants)
|
||||
- **Single-champion constraint:** Only single-champion splashes; multi-champion art excluded
|
||||
- **Hint via reveal:** Gradual visual context helps players narrow down champion identity over failed attempts
|
||||
|
||||
**Key difference:** Ability = binary guessing; Splash = progressive reveal with feedback.
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Source Investigation
|
||||
|
||||
### JavaScript Bundle Structure
|
||||
Loldle uses minified Vue.js app bundles with versioned filenames:
|
||||
- Main: `js/index.45d55fd2197ccf548738.1774994503850.js` (3.9MB minified)
|
||||
- Chunk vendors: `js/chunk-vendors.45d55fd2197ccf548738.1774994503850.js`
|
||||
- Hash changes on each site update; version embedded in HTML `<link rel="preload">`
|
||||
|
||||
**Champion data extraction:** Search minified bundle for `championId` property to locate champion data object. Data is UTF-8 encoded (handles multi-language text). Python regex tools exist (e.g., `extract-champlist.py` from joulsen/loldle-information-theory repo) to parse the JS object and convert to JSON.
|
||||
|
||||
### Image Source: Loldle vs. Data Dragon
|
||||
Two options identified:
|
||||
|
||||
#### Option A: Riot Data Dragon CDN (Direct)
|
||||
**Pros:**
|
||||
- Official, guaranteed up-to-date with game patches
|
||||
- High availability, CDN-distributed globally
|
||||
- No scraping needed; public documented API
|
||||
- Standardized URL structure; easy to construct URLs
|
||||
|
||||
**Cons:**
|
||||
- Requires calling DDragon for each patch version to get ability icon filenames
|
||||
- Ability icons keyed by internal `SpellKey` (not champion-friendly; requires champion JSON lookup)
|
||||
- Passive icons separate from spell icons (different endpoint prefix)
|
||||
|
||||
**URL patterns:**
|
||||
```
|
||||
https://ddragon.leagueoflegends.com/cdn/{version}/img/spell/{SpellKey}.png
|
||||
https://ddragon.leagueoflegends.com/cdn/{version}/img/passive/{PassiveKey}.png
|
||||
https://ddragon.leagueoflegends.com/cdn/img/champion/splash/{ChampionName}_0.jpg (base)
|
||||
https://ddragon.leagueoflegends.com/cdn/img/champion/splash/{ChampionName}_{skinId}.jpg (skins)
|
||||
```
|
||||
|
||||
**Example:** For Ahri's Q ability, DDragon provides SpellKey `FoxFireTwo` → fetch from spell endpoint. Champion JSON (en_US) specifies slot, image.full filename, and spell data.
|
||||
|
||||
#### Option B: Loldle.net JS Bundle Scraping
|
||||
**Pros:**
|
||||
- Image URLs likely embedded directly in JS bundle (faster runtime lookup)
|
||||
- Already aligned with Loldle's data schema
|
||||
- Single extraction step (no DDragon API calls)
|
||||
|
||||
**Cons:**
|
||||
- Requires re-extraction on each site update (monitor for bundle hash changes)
|
||||
- Image sources may still point to DDragon or Loldle CDN; need inspection
|
||||
- Minified JS harder to parse without exact regex knowledge
|
||||
- Breaking changes if Loldle refactors data structure
|
||||
|
||||
**Status:** Bundle not yet decompiled in this research; assumption that URLs are embedded awaits verification.
|
||||
|
||||
### Recommended approach: **Use Data Dragon directly**
|
||||
Rationale: Official, stable, no brittle scraping. Trade-off is one extra API call to fetch champion.json and one iteration to map SpellKey → URL, but both are lightweight. Splash URLs follow consistent pattern; ability lookup requires JSON traversal but is deterministic.
|
||||
|
||||
---
|
||||
|
||||
## 3. Scraping Feasibility
|
||||
|
||||
### Ability Mode Data Requirements
|
||||
**For each champion, capture:**
|
||||
- Champion name/key (standard)
|
||||
- 5 ability slot icons (Q/W/E/R/Passive)
|
||||
- Q/W/E/R: spells[i].image.full from champion.json
|
||||
- Passive: passive.image.full
|
||||
- Ability names (for bonus second-guess hint)
|
||||
|
||||
**Per-round selection:** Random pick 1 of the 5 abilities → construct URL from SpellKey.
|
||||
|
||||
**Feasibility:** ✅ Fully feasible. DDragon champion.json includes all spell metadata. Static per-patch; refresh on League patch cycle (~2 weeks).
|
||||
|
||||
### Splash Mode Data Requirements
|
||||
**For each champion, capture:**
|
||||
- Champion name/key
|
||||
- List of splash art URLs (base + all skins)
|
||||
- Base: `{ChampionName}_0.jpg`
|
||||
- Skins: `{ChampionName}_{skinId}.jpg`
|
||||
|
||||
**Challenge:** DDragon doesn't list all skin IDs directly; must scrape from champion.json `skins[]` array, which includes `id`, `name`, `num` fields.
|
||||
|
||||
**Per-round selection:** Random pick 1 splash from pool of available skins → crop image on first guess, zoom out on each wrong attempt.
|
||||
|
||||
**Feasibility:** ✅ Fully feasible. Skin IDs available in champion.json. All URLs follow predictable pattern. No additional API calls needed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Image Source Notes & Verification
|
||||
|
||||
### Data Dragon URL Construction
|
||||
|
||||
**Ability Icons:**
|
||||
```javascript
|
||||
const version = "16.8.1"; // from /api/versions.json
|
||||
const championData = await fetch(`https://ddragon.leagueoflegends.com/cdn/${version}/data/en_US/champion/Ahri.json`).then(r => r.json());
|
||||
// championData.data.Ahri.spells[0].image.full = "FoxFireTwo.png"
|
||||
const abilityUrl = `https://ddragon.leagueoflegends.com/cdn/${version}/img/spell/FoxFireTwo.png`;
|
||||
```
|
||||
|
||||
**Passive Icons:**
|
||||
```javascript
|
||||
// championData.data.Ahri.passive.image.full = "AhriPassive.png"
|
||||
const passiveUrl = `https://ddragon.leagueoflegends.com/cdn/${version}/img/passive/AhriPassive.png`;
|
||||
```
|
||||
|
||||
**Splash Art:**
|
||||
```javascript
|
||||
// championData.data.Ahri.skins = [{id: 0, name: "Classic", num: 0}, {id: 1, name: "Dynasty Ahri", num: 1}, ...]
|
||||
const baseUrl = `https://ddragon.leagueoflegends.com/cdn/img/champion/splash/Ahri_0.jpg`;
|
||||
const skinUrl = `https://ddragon.leagueoflegends.com/cdn/img/champion/splash/Ahri_1.jpg`; // skin 1
|
||||
```
|
||||
|
||||
### Verification Status
|
||||
- ✅ DDragon endpoints exist and are documented (hextechdocs.dev, riot-api-libraries)
|
||||
- ✅ Ability icon URLs follow `spell/{SpellKey}.png` and `passive/{PassiveKey}.png` pattern
|
||||
- ✅ Splash URLs follow `champion/splash/{Name}_{skinId}.jpg` pattern
|
||||
- ✅ Champion JSON includes all required metadata (skins[], spells[], passive)
|
||||
- ⚠️ **Not yet verified in browser:** Actual image availability at these URLs (assumed 100% coverage per Riot CDN reliability, but spot checks recommended)
|
||||
|
||||
---
|
||||
|
||||
## 5. Telegram Adaptation Strategy
|
||||
|
||||
### Challenge: Progressive Reveal Mechanic
|
||||
Loldle's core appeal is the zoom-in/zoom-out reveal. Telegram doesn't natively support:
|
||||
- Sending image crops inline (no built-in image editing API in bot SDK)
|
||||
- Real-time photo replacement in same message (must delete + resend, causing UX jank)
|
||||
|
||||
### Three Options Evaluated
|
||||
|
||||
#### Option A: Cloudflare Image Resizing API (RECOMMENDED)
|
||||
**How it works:**
|
||||
1. Store full ability icon / splash URL
|
||||
2. On guess, construct Cloudflare Image Resizing URL with crop/resize parameters
|
||||
3. Send cropped image to Telegram
|
||||
4. On next wrong guess, send new URL with larger viewport (zoom out)
|
||||
|
||||
**Cloudflare Images API supports:**
|
||||
- **Crop:** `?format=webp&crop=smartcrop` or `crop=left,top,right,bottom` (relative coords 0.0–1.0)
|
||||
- **Resize:** `?width=X&height=Y&fit=cover` with `crop=<side>` or `crop=<x>x<y>`
|
||||
- **Chain:** Ability icon small (64×64 crop) → medium (128×128) → full (256×256)
|
||||
- **Splash:** Crop top-left 20% → top-left 40% → top-left 60% → full image
|
||||
|
||||
**Pros:**
|
||||
- Preserves Loldle's core UX (progressive reveal works)
|
||||
- Runs at edge (Cloudflare Workers); <100ms latency
|
||||
- No server-side image processing needed (Workers have no native PIL/ImageMagick)
|
||||
- Scales to millions of guesses
|
||||
- No cost per image (included in Cloudflare Images plan)
|
||||
|
||||
**Cons:**
|
||||
- Requires Cloudflare Images product (adds ~$20–100/mo to existing Workers bill, depending on transforms)
|
||||
- Must delete old message and send new one on each guess (Telegram API limitation)
|
||||
- Message history grows; requires cleanup after game ends
|
||||
|
||||
**Feasibility:** ✅ **Viable and recommended.**
|
||||
|
||||
#### Option B: Full Ability Icon / Simple Splash (NO CROP)
|
||||
**How it works:**
|
||||
1. Send full 256×256 ability icon without cropping
|
||||
2. Send full splash art without cropping
|
||||
3. Skip zoom mechanic; just reveal full image on player request or after N wrong guesses
|
||||
|
||||
**Pros:**
|
||||
- Zero image processing; just send URL
|
||||
- Works in standard Cloudflare Workers (no external APIs)
|
||||
- Simple implementation
|
||||
- Telegram inline keyboards show full image in context
|
||||
|
||||
**Cons:**
|
||||
- Loses Loldle's signature zoom-in reveal (core gameplay appeal)
|
||||
- Splash art mode becomes trivial if full image visible from start
|
||||
- Reduced challenge/fun factor
|
||||
- Defeats the purpose of porting mode
|
||||
|
||||
**Feasibility:** ✅ Viable but **not recommended**—defeats design intent.
|
||||
|
||||
#### Option C: Image Processing in Workers (NOT VIABLE)
|
||||
**How it works:** Use Sharp.js or WASM image library in Worker to crop/resize on-the-fly.
|
||||
|
||||
**Cons:**
|
||||
- Workers have 128 MB CPU execution limit; image processing = slow
|
||||
- Worker size limit (1 MB script); Sharp.js alone is 500+ KB
|
||||
- No native file I/O; must stream into memory
|
||||
- Latency: 5–10 seconds per image
|
||||
- Cost overruns (Workers compute-heavy)
|
||||
|
||||
**Feasibility:** ❌ **Not recommended.** CPU/size constraints make this impractical.
|
||||
|
||||
### Recommendation: **Option A (Cloudflare Image Resizing)**
|
||||
|
||||
**Telegram adaptation flow:**
|
||||
1. **Guess submission:** User taps inline button with champion name
|
||||
2. **Validation:** Check against daily answer
|
||||
3. **If wrong:**
|
||||
- Construct new Cloudflare Image Resizing URL with expanded crop/zoom
|
||||
- Delete previous message (edit doesn't work well for photos)
|
||||
- Send new photo message with updated keyboard
|
||||
- Update guess counter
|
||||
4. **If correct:**
|
||||
- Edit keyboard to show "✓ Correct! Next round in 24h"
|
||||
- Log stats (guesses taken, time elapsed)
|
||||
|
||||
**Cost estimate:** ~1–3 image transforms per game × daily players. If 1000 games/day × 4 guesses avg = 4000 transforms. Cloudflare Images pricing: $0.03–0.10/1000 transforms = **$0.12–0.40/day** (~$3.50–12/month).
|
||||
|
||||
**Trade-off:** Small cost for best UX. Alternative (Option B) is free but kills the mode's appeal.
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation Roadmap
|
||||
|
||||
### Ability Mode
|
||||
1. Fetch latest DDragon version from `/api/versions.json`
|
||||
2. Cache champion.json (en_US) for current patch
|
||||
3. On game start:
|
||||
- Pick random champion
|
||||
- Pick random ability (Q/W/E/R/Passive)
|
||||
- Construct spell/passive icon URL
|
||||
4. Serve icon via Cloudflare Image Resizing (64×64 crop for first guess)
|
||||
5. On each wrong guess, expand crop (128×128, 192×192, full)
|
||||
6. After champion guessed, show ability slot choices (multiple choice buttons)
|
||||
|
||||
**Storage:** Pre-generate crop params at startup; store in KV cache (champion → ability → crop dimensions)
|
||||
|
||||
### Splash Mode
|
||||
1. Cache champion.json with skins[] data
|
||||
2. On game start:
|
||||
- Pick random champion
|
||||
- Pick random skin
|
||||
- Construct splash URL ({Name}_{skinId}.jpg)
|
||||
3. Serve via Image Resizing (20% viewport crop, top-left)
|
||||
4. On each wrong guess, expand viewport (40%, 60%, 80%, 100%)
|
||||
5. Guess from champion select dropdown
|
||||
|
||||
**Storage:** Pre-generate viewport crop params; store in KV
|
||||
|
||||
### Shared Data Pipeline
|
||||
```
|
||||
cron: every patch (2 weeks) or manual trigger
|
||||
→ fetch https://ddragon.leagueoflegends.com/api/versions.json
|
||||
→ get latest version
|
||||
→ fetch https://ddragon.leagueoflegends.com/cdn/{version}/data/en_US/champion.json
|
||||
→ extract championId, skins[], spells[], passive.image.full
|
||||
→ store to KV: key={championId}, value={JSON struct}
|
||||
→ seed RNG for daily selection (same seed = same champion daily across users)
|
||||
```
|
||||
|
||||
**Cloudflare Workers implementation:** Standard fetch + KV bindings. Add Cloudflare Images binding for image transforms.
|
||||
|
||||
---
|
||||
|
||||
## 7. Risk Assessment & Adoption Hazards
|
||||
|
||||
### Data Dragon Risks
|
||||
- **Patch conflicts:** If game hotfixes champion abilities, DDragon may lag by hours
|
||||
- *Mitigation:* Add patch version selector in-game; cache aggressively
|
||||
- **Skin data completeness:** Not all skins may have splash URLs (rare legacy content)
|
||||
- *Mitigation:* Validate URLs at startup; filter out 404s
|
||||
- **Rate limits:** Unlikely for small-scale bot, but no published limits documented
|
||||
- *Mitigation:* Cache all data locally; refresh weekly, not per-request
|
||||
|
||||
### Cloudflare Images Risks
|
||||
- **Cost unpredictability:** Transforms per guess; volume scaling unknown
|
||||
- *Mitigation:* Monitor transform count weekly; set alerts at $50/mo
|
||||
- **Service availability:** CDN outage = bot can't render images (falls back to text)
|
||||
- *Mitigation:* Graceful fallback: "Sorry, image unavailable; here's a text hint instead"
|
||||
- **Transform latency:** Edge compute may be <100ms, but add Telegram API roundtrip
|
||||
- *Mitigation:* Pre-compute crop params at startup; cache Image URLs
|
||||
|
||||
### Telegram API Risks
|
||||
- **Message deletion jank:** Deleting + resending on each guess = slow UX
|
||||
- *Mitigation:* Edit message caption (text) instead; keep photo static, update hints in text
|
||||
- **Alternative:** Use editMessageMedia to replace photo in-place (cleaner, if Cloudflare URL stable)
|
||||
- **Inline keyboard timeout:** Users may not guess within reasonable time; stale keyboards
|
||||
- *Mitigation:* 24h timeout per game; archive messages after completion
|
||||
|
||||
### Game Design Risks
|
||||
- **Ability mode too hard:** 850+ icons; players may not recognize obscure abilities
|
||||
- *Mitigation:* Add multiple-choice dropdown (narrow from 170 → 10 candidates); or hint system
|
||||
- **Splash mode too easy:** Full image reveal may happen in <2 guesses for popular champs
|
||||
- *Mitigation:* Start with smaller crop (10% instead of 20%); require more guesses for full reveal
|
||||
|
||||
---
|
||||
|
||||
## 8. Unresolved Questions
|
||||
|
||||
1. **Loldle.net JS bundle:** Does it embed image URLs directly, or fetch from DDragon? Need to decompress and search.
|
||||
2. **Exact guess limit:** How many guesses allowed in daily Ability/Splash before forfeit? Search results mentioned Wordle convention but not Loldle's specific rule.
|
||||
3. **Splash art scope:** Does Loldle include ALL skins or a curated subset? DDragon lists 10+ skins per champ; scraping all is safe but may inflate data.
|
||||
4. **Ability hint system:** Does ability mode provide any visual feedback (e.g., "close"/"warmer") or is it binary? Confirmation from X post suggests binary.
|
||||
5. **Image URL stability:** Are Cloudflare Image Resizing URLs cacheable by Telegram clients, or regenerated per request? Affects message edit efficiency.
|
||||
6. **Legacy champion coverage:** Do all 170+ champions have ability icons in DDragon? Or are alpha/removed champs missing?
|
||||
7. **Performance baseline:** Average response time from guess → image delivery in production. Need benchmark on low-power Workers.
|
||||
|
||||
---
|
||||
|
||||
## 9. Recommendation Summary
|
||||
|
||||
| Aspect | Finding |
|
||||
|--------|---------|
|
||||
| **Gameplay Mechanics** | Confirmed: Ability = binary guessing; Splash = progressive zoom reveal. Both feasible to port. |
|
||||
| **Data Source** | DDragon (Option A) > Loldle JS scraping (Option B). Official, stable, no brittle parsing. |
|
||||
| **Scraping Feasibility** | ✅ Yes. DDragon champion.json includes all ability icons + skin IDs. One-time cache per patch. |
|
||||
| **Image Source** | DDragon CDN URLs are standardized, documented, and reliable. Verified URL patterns. |
|
||||
| **Telegram Adaptation** | Cloudflare Image Resizing (Option A) best preserves UX. Option B (no crop) viable but kills appeal. Option C (in-Worker processing) not feasible. |
|
||||
| **Implementation Complexity** | Low-medium. Fetch + cache + URL construction + Telegram inline keyboards. ~300–500 LOC per mode. |
|
||||
| **Cost** | ~$5–15/month (Cloudflare Images transforms) + existing Workers bill. |
|
||||
| **Risk Level** | Low-medium. DDragon stable; Image API documented; Telegram API mature. Main hazards: cost overruns, user adoption (difficulty tuning). |
|
||||
|
||||
**Next Step:** Confirm Loldle's guess limit and verify image URL stability via live game testing on ability/splash modes. Then proceed to implementation plan.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [LoLdle Answers Today (Daily Solutions)](https://www.esports.net/wiki/guides/loldle-answers-today/)
|
||||
- [LOLDLE Answer Today: Classic, Quote, Ability, Emoji & Splash](https://phonenumble.com/loldle-wordle/)
|
||||
- [LoLdle – Splash Mode](https://loldle.net/splash)
|
||||
- [LoLdle – Ability Mode](https://loldle.net/ability)
|
||||
- [LoLdle Bonus Ability Guess (Passive/Q/W/E/R)](https://x.com/loldlegame/status/1583815117355249665)
|
||||
- [GitHub: joulsen/loldle-information-theory](https://github.com/joulsen/loldle-information-theory)
|
||||
- [GitHub: Kerrders/LoLdleData](https://github.com/Kerrders/LoLdleData)
|
||||
- [Riot API Libraries: Data Dragon Documentation](https://riot-api-libraries.readthedocs.io/en/latest/ddragon.html)
|
||||
- [HexTech Docs: Data Dragon](https://hextechdocs.dev/data-dragon/)
|
||||
- [Cloudflare Images: Transform via Workers](https://developers.cloudflare.com/images/transform-images/transform-via-workers/)
|
||||
- [Cloudflare Images: Cropping Features](https://developers.cloudflare.com/images/optimization/features/)
|
||||
- [GitHub: cvzi/telegram-bot-cloudflare](https://github.com/cvzi/telegram-bot-cloudflare)
|
||||
- [Telegram Bot API: Inline Keyboards and Message Editing](https://core.telegram.org/bots/api)
|
||||
- [grammY: Inline and Custom Keyboards](https://grammy.dev/plugins/keyboard)
|
||||
- [Data Dragon API – Tested Daily](https://www.freepublicapis.com/data-dragon-api/)
|
||||
@@ -1,179 +0,0 @@
|
||||
# Loldle Modes & Emoji Mode Research Report
|
||||
|
||||
**Date:** April 24, 2026
|
||||
**Scope:** All Loldle game modes discovery + Emoji mode technical analysis + cross-mode data audit
|
||||
|
||||
---
|
||||
|
||||
## Section 1: Complete Loldle Modes Inventory
|
||||
|
||||
All five game modes confirmed as of April 2026:
|
||||
|
||||
| Mode | URL | Type | Resets | Description |
|
||||
|------|-----|------|--------|-------------|
|
||||
| **classic** | `loldle.net/` or `loldle.net/classic` | Daily | Daily @ 00:00 UTC | Guess champion from attribute hints (gender, role, species, resource, range type, region, release year) |
|
||||
| **quote** | `loldle.net/quote` | Daily | Daily @ 00:00 UTC | Guess champion from in-game voice line (audio + text) |
|
||||
| **ability** | `loldle.net/ability` | Daily | Daily @ 00:00 UTC | Guess champion from ability UI icon + name (Passive, Q, W, E, R) |
|
||||
| **emoji** | `loldle.net/emoji` | Daily | Daily @ 00:00 UTC | Guess champion from progressive emoji sequence; unlocks one emoji per wrong guess |
|
||||
| **splash** | `loldle.net/splash` | Daily | Daily @ 00:00 UTC | Guess champion from cropped splash art image (may be any skin) |
|
||||
|
||||
**Key finding:** NO "title", "catchphrase", or sixth mode exists as of April 2026.
|
||||
|
||||
**Unlimited variant:** loldle.org offers an "unlimited" version allowing repeated plays, but primary loldle.net modes are daily-only.
|
||||
|
||||
---
|
||||
|
||||
## Section 2: Emoji Mode Deep Dive
|
||||
|
||||
### Gameplay Mechanics
|
||||
|
||||
- **Input:** 1-3 emojis shown at start; progressive reveal on each wrong guess
|
||||
- **Guesses:** Unlimited attempts until correct answer
|
||||
- **Hint system:** Each incorrect guess unlocks a new emoji
|
||||
- **Output:** After victory, player sees complete emoji sequence; all players see identical emojis for the daily champion
|
||||
|
||||
### Emoji Mapping Examples
|
||||
|
||||
Emojis reference lore, abilities, skins, or thematic traits:
|
||||
- **🦊✨💫** → Ahri (fox + magic particles + stars = nine-tailed fox theme)
|
||||
- **🔥👊** → Lee Sin or Brand (fire + punch = aggression; fire + kick = abilities)
|
||||
- **⚔️🛡️** → Sword/shield-wielding champions
|
||||
- Weapon emojis (🗡️, 🏹, ⚡) = kit identity
|
||||
- Animal emojis (🦁, 🐺, 🦊) = champion lore
|
||||
- Region symbols (👑, 🏰) = Noxus/Demacia/etc
|
||||
|
||||
### Data Source Structure
|
||||
|
||||
**Location:** Champion → emoji sequence mapping embedded in loldle.net JavaScript bundle
|
||||
|
||||
**Extraction method:**
|
||||
1. Inspect `www.loldle.net` page source (DevTools)
|
||||
2. Locate champion data in bundled JS file
|
||||
3. Extract `{championName: "emojiSequence"}` mappings
|
||||
4. Store as JSON for bot reuse
|
||||
|
||||
**Structure (inferred):**
|
||||
```json
|
||||
{
|
||||
"Ahri": "🦊✨💫🌙",
|
||||
"LeeSin": "🔥👊🌊🥋",
|
||||
"Brand": "🔥💣☠️",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**Scope:** Emoji data covers **all 168+ champions** in League (full champion pool, not limited).
|
||||
|
||||
---
|
||||
|
||||
## Section 3: Cross-Mode Data Audit
|
||||
|
||||
### Single Bundle vs Split Strategy
|
||||
|
||||
**Finding:** All mode data likely resides in **one primary JS bundle** on loldle.net:
|
||||
|
||||
1. **Classic mode:** Champion stats (gender, role, resource, region, release year)
|
||||
2. **Quote mode:** Champion voice lines + audio assets
|
||||
3. **Ability mode:** Champion spell icons + names
|
||||
4. **Emoji mode:** Champion → emoji sequence mapping
|
||||
5. **Splash mode:** Champion skin splash art references
|
||||
|
||||
**Source:** GitHub project `joulsen/loldle-information-theory` confirms data extraction via loldle.net JS bundle inspection. The `resources/loldle-champ-data.json` file is maintained by extracting from live Loldle JS.
|
||||
|
||||
### Data Extraction Strategy (Recommended)
|
||||
|
||||
**Approach:** Single scrape operation with mode-aware parsing
|
||||
|
||||
```
|
||||
GET loldle.net
|
||||
→ Parse JS bundle
|
||||
→ Extract entire champion object
|
||||
→ Split into mode-specific datasets:
|
||||
- classic_stats.json (attributes)
|
||||
- emoji_map.json (emoji sequences)
|
||||
- quotes.json (voice lines)
|
||||
- abilities.json (spell info)
|
||||
- splash_references.json (skin images)
|
||||
```
|
||||
|
||||
**Cost:** One HTTP request + parsing overhead = ~1-2 seconds per update.
|
||||
|
||||
**Frequency:** Daily rotation (mirrors official daily reset @ 00:00 UTC). No need to scrape more than once per day unless implementing unlimited mode.
|
||||
|
||||
---
|
||||
|
||||
## Section 4: Emoji Mode — Telegram Bot Adaptation
|
||||
|
||||
### Simplicity Assessment: ✅ TRIVIAL
|
||||
|
||||
**Emoji rendering in Telegram:** Native support. Zero translation overhead.
|
||||
|
||||
**Bot implementation outline:**
|
||||
1. Load emoji_map.json (champion → emojis)
|
||||
2. On `/emoji` command:
|
||||
- Pick random champion from pool
|
||||
- Show 1-3 emojis
|
||||
- Accept user guess via `/guess ChampionName`
|
||||
- Reveal next emoji on wrong guess
|
||||
- End on correct guess or 10 attempts
|
||||
3. Track guesses per user per day (daily reset @ 00:00 UTC)
|
||||
|
||||
**Complexity:** ~50-100 lines of Node.js (much simpler than classic mode).
|
||||
|
||||
---
|
||||
|
||||
## Section 5: Technical Implementation Notes
|
||||
|
||||
### Existing Codebase Integration
|
||||
|
||||
Your project already:
|
||||
- Scrapes champion data from loldle.net JS bundle ✅
|
||||
- Stores as JSON ✅
|
||||
- Classic mode operational ✅
|
||||
|
||||
**Emoji mode add-on requires:**
|
||||
1. Extract `championName → emojiString` from bundle (likely already present)
|
||||
2. Parse emojis into array for progressive reveal
|
||||
3. Add `/emoji` command handler (~100 LOC)
|
||||
4. Reuse existing daily reset logic
|
||||
|
||||
### Data Freshness
|
||||
|
||||
- Loldle updates champion pool when new champs release (rare, ~1-2/year)
|
||||
- Emoji sequences stable for existing champions
|
||||
- Daily puzzle seed: separate rotation (independent per mode)
|
||||
- **Scrape frequency:** Once per day or on-demand after new champion release
|
||||
|
||||
### Limitations
|
||||
|
||||
1. **Emoji ambiguity:** Some emojis can map to multiple interpretations (🔥 = Brand, Lee Sin, Udyr, etc.). Loldle handles this via progressive reveal.
|
||||
2. **Custom emoji selection:** Loldle's emoji assignments appear handcrafted (not algorithmically derived). You cannot compute emojis on-the-fly; must extract from their data.
|
||||
3. **Audio assets (Quote mode):** Not trivial to replicate; requires hosting audio files or linking to Loldle's CDN (legal gray area). Emoji mode avoids this entirely.
|
||||
|
||||
---
|
||||
|
||||
## Section 6: Unresolved Questions
|
||||
|
||||
1. **Exact emoji data format in bundle:** Is it a simple string ("🦊✨💫"), array ["🦊", "✨", "💫"], or object with reveal order? → Requires bundle inspection
|
||||
2. **Emoji uniqueness:** Are emoji sequences guaranteed 1:1 to champions, or can multiple champions share sequences? → Likely 1:1 but unconfirmed
|
||||
3. **Future mode expansion:** Loldle.net roadmap (if public) — any planned new modes? → Not found in search results
|
||||
4. **Unlimited mode emoji data:** Does loldle.org use identical emoji mappings as loldle.net? → Likely yes (separate frontend, same data)
|
||||
5. **Regional CDN:** Does loldle.net serve different data to different regions? → Probably not (Wordle-style games are region-agnostic)
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [LoLdle Game Modes Overview — Phone Numble](https://phonenumble.com/loldle-wordle/)
|
||||
- [LoLdle Answers Today — GFinityEsports](https://www.gfinityesports.com/article/loldle-answer-today)
|
||||
- [LoLdle Answers for Today — Twinfinite](https://twinfinite.net/guides/loldle-answers-today/)
|
||||
- [LoLdle Official Site](https://loldle.net/)
|
||||
- [LoLdle Emoji Mode](https://loldle.net/emoji)
|
||||
- [LoLdle Information Theory Solver — GitHub](https://github.com/joulsen/loldle-information-theory)
|
||||
- [LoLdle Data Fetch — GitHub](https://github.com/Kerrders/LoLdleData)
|
||||
- [LOL Champions Data — GitHub ngryman](https://github.com/ngryman/lol-champions)
|
||||
- [LoLdle Unlimited Variant — loldle.org](https://loldle.org/unlimited)
|
||||
|
||||
---
|
||||
|
||||
**Report Status:** COMPLETE. All five modes documented. Emoji mode analyzed as "trivial for Telegram adaptation." Cross-mode data audit suggests single-bundle extraction is feasible.
|
||||
@@ -1,278 +0,0 @@
|
||||
# Loldle Quote Mode Research Report
|
||||
|
||||
**Date:** 2026-04-24
|
||||
**Focus:** Loldle Quote mode mechanics, data sources, and Telegram bot adaptation feasibility
|
||||
|
||||
---
|
||||
|
||||
## 1. Gameplay Mechanics
|
||||
|
||||
**Core Loop:**
|
||||
- Player presented with champion **quote text** (single line of in-game dialogue)
|
||||
- Player has up to **6 incorrect guesses** to identify the champion
|
||||
- After 6 failed guesses, **audio clue unlocks** — the voice track of the champion speaking that exact quote
|
||||
- Binary feedback: correct/incorrect (no gradual hints like classic mode)
|
||||
- Daily reset at 00:00 UTC (one quote per day, same for all players)
|
||||
|
||||
**Difficulty Factor:**
|
||||
- Many champions share similar tone, thematic dialogue, generic lines
|
||||
- Short quotes often feel interchangeable across champions
|
||||
- Audio hint helps but champions with similar-sounding voices remain ambiguous
|
||||
- Requires genuine champion knowledge, not just systematic elimination (unlike classic mode)
|
||||
|
||||
**Comparison to Classic Mode:**
|
||||
- Classic mode: feedback based on champion metadata (region, year, role, etc.)
|
||||
- Quote mode: immediate right/wrong, then audio clue only
|
||||
- Quote mode is harder — no attribute-based elimination strategy
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Source & Infrastructure
|
||||
|
||||
### Quote Text Source
|
||||
**WHERE:** Embedded in client-side JavaScript bundle (minified `app.{hash}.js`)
|
||||
|
||||
**HOW TO ACCESS:**
|
||||
1. Visit https://loldle.net/quote
|
||||
2. Extract minified bundle from page source (find `app.xxx.js` in script tags)
|
||||
3. Search bundle using regex: `championId` property locates champion data
|
||||
4. Use extraction script (see: joulsen/loldle-information-theory repo)
|
||||
|
||||
**FOUND REPOSITORY:** [joulsen/loldle-information-theory](https://github.com/joulsen/loldle-information-theory)
|
||||
- Provides `resources/extract-champlist.py` for automated extraction
|
||||
- Provides pre-extracted `resources/loldle-champ-data.json`
|
||||
- Regex pattern: `=(\\\[\\{\_id:"\[^{}\]+championId:".+?\\}\\\])`
|
||||
|
||||
**DATA STRUCTURE:** Quote data likely stored same way as classic-mode champion data:
|
||||
- Each champion has array of properties (name, region, role, hp, etc.)
|
||||
- Quote mode adds `quote` field with the dialogue text
|
||||
- Quote may map to voice line ID or include URL reference
|
||||
|
||||
### Audio Source
|
||||
**WHERE:** Likely League of Legends Wiki or Riot-hosted CDN (cached during quote reveal)
|
||||
|
||||
**MECHANICS:**
|
||||
- Initially: only text shown
|
||||
- After 6 wrong guesses: audio file loads via HTTPS
|
||||
- Probable source: Riot Games CDN (per League Wiki structure)
|
||||
- Format: OGG or MP3 (standard web audio)
|
||||
- No direct URL exposed in initial puzzle request (audio fetched only after hint unlock)
|
||||
|
||||
### Cache Endpoint
|
||||
**https://cache.loldle.net/cache.json**
|
||||
- Response is **Salted base64-encoded** (OpenSSL encryption)
|
||||
- Contains aggregated game state/metadata
|
||||
- Cannot be directly parsed without decryption key
|
||||
- Likely syncs game state across devices, not primary data source
|
||||
|
||||
### Data Freshness
|
||||
- Quote data baked into JS bundle (no live API call for quote text)
|
||||
- Audio file fetched at hint reveal (cached, not live-generated)
|
||||
- Bundle updates when new champions added or quotes change (likely patch-synced)
|
||||
|
||||
---
|
||||
|
||||
## 3. Scraping Feasibility
|
||||
|
||||
### ✅ Can We Extract Quote-Champion Pairs?
|
||||
|
||||
**YES** — with caveats:
|
||||
|
||||
**Option A: Direct Bundle Extraction (Reliable)**
|
||||
```
|
||||
1. Fetch https://loldle.net/quote
|
||||
2. Parse HTML, find <script> with app bundle URL
|
||||
3. Download app.{hash}.js
|
||||
4. Extract using regex: championId property block
|
||||
5. Convert minified JS to JSON (via js-to-json converter)
|
||||
6. Filter for quote-only champions (some may lack quotes)
|
||||
7. Store as JSON: [ { name, quote, championId }, ... ]
|
||||
```
|
||||
|
||||
**Exact Regex:** `=(\\\[\\{\_id:"\[^{}\]+championId:".+?\\}\\\])`
|
||||
|
||||
**Output File:** Pre-made at [joulsen repo](https://github.com/joulsen/loldle-information-theory/blob/master/resources/loldle-champ-data.json)
|
||||
|
||||
**Option B: API Reverse-Engineering (Uncertain)**
|
||||
- No public loldle.net API endpoint discovered
|
||||
- cache.json encrypted (not viable)
|
||||
- Quote-of-the-day: only exposed via frontend; no direct REST endpoint found
|
||||
- Would require Cloudflare Workers interception (harder, rate-limited)
|
||||
|
||||
### ⚠️ Limitations
|
||||
|
||||
- **Only daily quote exposed:** True historical quote list not documented
|
||||
- Bundle hash changes on updates: extraction must re-run per patch
|
||||
- **No official API:** Community consensus is bundle extraction only method
|
||||
- New champions may lack quotes (deprecated `championId` field noted in joulsen repo)
|
||||
|
||||
### 📊 Data Format Expected
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "Ahri",
|
||||
"championId": 103,
|
||||
"quote": "The true face of desire.",
|
||||
"audioUrl": null // only populated after hint unlock
|
||||
},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Telegram Bot Adaptation
|
||||
|
||||
### Architecture Design
|
||||
|
||||
**Bot Module: `/loldle/modes/quote.js`**
|
||||
|
||||
```
|
||||
User sends: /loldle-quote
|
||||
Bot responds:
|
||||
1. Fetch today's quote-champion pair from cached data (or re-extract if stale)
|
||||
2. Send Markdown message:
|
||||
```
|
||||
🎭 **Today's Quote**
|
||||
"The true face of desire."
|
||||
|
||||
Guess the champion (6 attempts remaining)
|
||||
```
|
||||
3. User replies: /guess Ahri
|
||||
4. Bot checks against answer, updates attempt counter
|
||||
5. After 6 fails, reply with: *Audio hint unlocked!* [voice message file_id]
|
||||
```
|
||||
|
||||
### Telegram Media Handling
|
||||
|
||||
**Text Only (RECOMMENDED):**
|
||||
- Send quote as Markdown code block
|
||||
- Users guess via `/guess ChampionName`
|
||||
- Audio unnecessary for bot (text-based is cleaner)
|
||||
- **Pros:** Fast, no storage, text-searchable logs
|
||||
- **Cons:** Loses immersion of original web game
|
||||
|
||||
**Text + Optional Audio (ADVANCED):**
|
||||
- After hint unlock, fetch voice line from LoL Wiki/CDN
|
||||
- Send via `sendVoice()` API (Telegram supports OGG, MP3)
|
||||
- Requires one-time download + local cache or stream from CDN
|
||||
- **Pros:** Full feature parity with web
|
||||
- **Cons:** Storage overhead, CDN bandwidth cost, TOS risk (Riot asset rehost)
|
||||
|
||||
**Recommendation:** Text-only. Simpler, faster, no legal/storage issues. Audio hint can be optional (`/hint` command triggers audio fetch).
|
||||
|
||||
### Bot Command Set
|
||||
```
|
||||
/loldle-quote → Today's quote puzzle
|
||||
/guess <champion> → Submit guess
|
||||
/hint → Unlock audio (after 6 fails)
|
||||
/skip → Give up, reveal answer
|
||||
/quote-stats → Player's quote-mode stats
|
||||
```
|
||||
|
||||
### Data Storage (Cloudflare D1)
|
||||
```sql
|
||||
CREATE TABLE loldle_quote_attempts (
|
||||
id UUID PRIMARY KEY,
|
||||
user_id INT,
|
||||
date TIMESTAMP DEFAULT NOW(),
|
||||
champion TEXT,
|
||||
guesses_used INT (0-7),
|
||||
solved BOOLEAN,
|
||||
quote_text TEXT
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Pool Size & Champion Coverage
|
||||
|
||||
### Total Champions Available
|
||||
- **~165 champions** in current League roster
|
||||
- **ALL have voice lines** (via League Wiki)
|
||||
- **Estimated ~140-155 have "iconic" quotes** in loldle pool (inferred)
|
||||
|
||||
### Why Not 165?
|
||||
- Some newer/reworked champions may lack distinct quotes in loldle's curated set
|
||||
- Loldle creator (Pimeko) likely hand-selected quotes for memorability
|
||||
- Deprecated `championId` field in newer champions suggests staggered adoption
|
||||
|
||||
### Quote Dataset References
|
||||
1. [Allan-Cao/lol-voice-lines](https://github.com/Allan-Cao/lol-voice-lines) — 163 champions, cleaned quotes
|
||||
2. [Kaggle: League Voice Lines 13.10](https://www.kaggle.com/datasets/taupiphi/league-of-legends-voice-lines) — Patch 13.10 snapshot
|
||||
3. [League Wiki Champion Audio](https://wiki.leagueoflegends.com/en-us/Category:LoL_Champion_audio) — 175+ pages, official source
|
||||
|
||||
---
|
||||
|
||||
## 6. Technical Recommendations
|
||||
|
||||
### For Bot Implementation
|
||||
|
||||
**Priority 1: Extract Quote Data**
|
||||
```bash
|
||||
curl https://loldle.net/quote | grep -oP 'src="[^"]*app\.[a-z0-9]+\.js"' | xargs curl > bundle.js
|
||||
python3 ~/.claude/skills/extract-quotes.py bundle.js > quotes.json
|
||||
```
|
||||
|
||||
**Priority 2: Build Quote Module**
|
||||
- Fetch from D1 cache (or re-extract weekly)
|
||||
- Hash-based dedup (same quote, different champions → handle edge case)
|
||||
- Timezone handling (UTC reset, but bot may serve multiple timezones)
|
||||
|
||||
**Priority 3: Integrate with Existing Classic Mode**
|
||||
- Reuse champion list, verification logic
|
||||
- Add `/loldle` menu: classic | quote | ability | emoji | splash (when ready)
|
||||
|
||||
### Risk Assessment
|
||||
|
||||
| Risk | Level | Mitigation |
|
||||
|------|-------|-----------|
|
||||
| Riot TOS (champion data) | LOW | Quote data is public on loldle.net; rehost only curated subset |
|
||||
| Audio CDN bandwidth | MED | Skip audio feature; or fetch on-demand and cache 24h |
|
||||
| Bundle extraction brittleness | MED | Monitor for hash changes; add fallback to joulsen repo cache |
|
||||
| Daily reset race condition | LOW | Use UTC timestamp, cache daily answer at 00:01 UTC |
|
||||
| Quote ambiguity false positives | LOW | Case-insensitive matching, accept "Ahri" or "AHRI" |
|
||||
|
||||
### Estimated Effort
|
||||
- **Data Extraction:** 2-4 hours (prototype extraction script)
|
||||
- **Bot Commands:** 3-6 hours (reuse classic mode structure)
|
||||
- **Audio Integration:** 4-8 hours (if audio feature included; skip for MVP)
|
||||
- **Testing:** 2-3 hours
|
||||
- **Total MVP (text-only):** ~8-12 hours
|
||||
|
||||
---
|
||||
|
||||
## Key Findings Summary
|
||||
|
||||
1. ✅ **Quote data IS extractable:** Embedded in loldle.net JS bundle, regex-accessible
|
||||
2. ✅ **No API barrier:** Bundle extraction beats API reverse-engineering (no auth, no rate limits)
|
||||
3. ✅ **~150 champions supported:** Enough diversity for daily rotation without repeats (400+ days)
|
||||
4. ✅ **Telegram-friendly:** Text quotes work perfectly; audio is optional complexity
|
||||
5. ⚠️ **Audio source ambiguous:** Likely LoL Wiki/CDN but not documented; fetch at hint-reveal only
|
||||
6. ⚠️ **One quote per day:** Only today's quote exposed; historical quotes unavailable (not ideal for infinite mode)
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Where exactly is audio hosted?** Riot CDN vs. LoL Wiki vs. loldle.net's own cache — needs network inspection
|
||||
2. **Do ALL ~165 champions have quotes in loldle's pool?** Or curated subset? Exact count unconfirmed
|
||||
3. **Can we extract entire quote history?** Only today's quote is documented; older puzzles not exposed
|
||||
4. **What's the bundle hash update frequency?** Is it per-patch or more granular? Impacts extraction stability
|
||||
5. **Are voice lines guaranteed to be stable?** Or do champions get re-voiced, causing quote mismatches?
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [Loldle.net - Quote Mode](https://loldle.net/quote)
|
||||
- [Phone Numble - LoLdle Answer Guide](https://phonenumble.com/loldle-wordle/)
|
||||
- [GGrecon - LoLdle Answers](https://www.ggrecon.com/word-games/loldle-answer-today/)
|
||||
- [Esports.net - LoLdle Answers & Guides](https://www.esports.net/wiki/guides/loldle-answers-today/)
|
||||
- [Digi Magazine - LoLdle Answers](https://digimagazine.net/games/loldle-answers/)
|
||||
- [GitHub - joulsen/loldle-information-theory](https://github.com/joulsen/loldle-information-theory)
|
||||
- [GitHub - Allan-Cao/lol-voice-lines](https://github.com/Allan-Cao/lol-voice-lines)
|
||||
- [Kaggle - League of Legends Voice Lines](https://www.kaggle.com/datasets/taupiphi/league-of-legends-voice-lines)
|
||||
- [League of Legends Wiki - Champion Audio](https://wiki.leagueoflegends.com/en-us/Category:LoL_Champion_audio)
|
||||
- [GitHub - Peter-DeVries/Discord-LoLdle-Bot](https://github.com/Peter-DeVries/Discord-LoLdle-Bot)
|
||||
- [GitHub - Derpthemeus/LeagueOfQuotes](https://github.com/Derpthemeus/LeagueOfQuotes)
|
||||
@@ -1,410 +0,0 @@
|
||||
# MongoDB Atlas Fit Analysis for miti99bot — Cloudflare Workers
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Recommendation: DO NOT use MongoDB Atlas M0 for this project.**
|
||||
|
||||
Atlas M0 (Free, 512MB) **cannot solve the stated KV quota pain** — it has **100 ops/sec throughput** but **NO daily cap**, which sounds good until you account for:
|
||||
1. **Cold-start cost** — Every fresh isolate incurs ~1500ms TLS handshake + SCRAM auth. No connection pooling across stateless invocations.
|
||||
2. **No money-for-slots trade-off** — Upstash and Turso are HTTP-based, eliminating TLS overhead and working natively in Workers.
|
||||
3. **Operational fit mismatch** — The bot is read-heavy game state + append-only trading ledger. KV + D1 are already optimized for this; Atlas adds complexity without benefit.
|
||||
|
||||
**Better option: Upstash Redis** — FREE tier: 500K commands/month (was 10K/day), unlimited storage in free tier, HTTP-native, memoizable connection per isolate, proven on Workers. **Backup: Turso** for SQL-shaped data (trading module move away from D1).
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Current storage:
|
||||
- **Cloudflare KV**: 100k reads, 1k writes/day free (resets UTC 00:00)
|
||||
- **Cloudflare D1**: 1 query/sec free tier
|
||||
- **Workload**: 13 active modules, many KV-read-heavy (games, schedules), 1 append-only SQL (trading)
|
||||
- **Pain**: Even modest daily traffic exhausts KV quota mid-game; users hit quota-exhausted errors
|
||||
|
||||
User hypothesis: MongoDB Atlas Free (M0) lifts the cap. Reality: **M0 has a throughput cap (100 ops/sec) not a daily-quota cap**, and the architecture cost is substantial.
|
||||
|
||||
---
|
||||
|
||||
## Option Evaluation
|
||||
|
||||
### 1. MongoDB Atlas M0 (Proposed)
|
||||
|
||||
#### Specification
|
||||
- **Storage**: 512 MB (plenty for bot state)
|
||||
- **Throughput**: 100 ops/sec (no daily operation limit)
|
||||
- **Connections**: 500 max
|
||||
- **Regions**: Multi-region (AWS us-east-1, eu-west-1, etc.)
|
||||
- **Cost**: Free; auto-pauses after 30 days inactivity
|
||||
- **Deprecated features**: Backups, server-side JS, sharding, auditing
|
||||
|
||||
#### Cloudflare Workers Compatibility ✅ (as of March 2025)
|
||||
|
||||
**Yes, but with caveats.** Cloudflare shipped `node:net`, `node:tls` (TLS socket support) in Q1 2025.
|
||||
|
||||
**Requirements:**
|
||||
- `wrangler.toml`: `compatibility_flags = ["nodejs_compat_v2"]` + `compatibility_date = "2025-03-20"`
|
||||
- Driver: `npm install mongodb` v6.7+
|
||||
- Connection string: `mongodb+srv://user:pass@cluster.mongodb.net/db` (works; explicit-host strings also work)
|
||||
- Auth: SCRAM over TLS (native)
|
||||
|
||||
**Evidence:** [Cloudflare Workers and MongoDB (March 2025)](https://alexbevi.com/blog/2025/03/25/cloudflare-workers-and-mongodb/) demonstrates full CRUD: drop, insert, query operations.
|
||||
|
||||
#### Operational Trade-offs
|
||||
|
||||
| Dimension | Status |
|
||||
|-----------|--------|
|
||||
| **Cold-start cost** | **❌ SEVERE** — ~1500ms TLS+SCRAM per new isolate; Cloudflare Workers are stateless per-invocation, so every cold start = new connection = handshake latency visible to user |
|
||||
| **Connection pooling** | **❌ IMPOSSIBLE** — Cloudflare Workers V8 isolates have no shared connection pool; memoizing a MongoClient per isolate helps warm requests but cold starts still stall |
|
||||
| **Driver bundle size** | ~4-5 MB compressed (mongodb npm pkg); moderate but adds to bundle cost |
|
||||
| **TLS root CA** | ✅ Bundled in driver; no manual CA setup needed |
|
||||
| **Durable Objects support** | ⚠️ Possible but adds complexity; DO provides in-memory queue to batch inserts, but defeats simplicity goal |
|
||||
| **Local dev (`wrangler dev`)** | ✅ Works as-is; Workers Sockets API routes TCP to Atlas |
|
||||
| **Transactions** | ❌ Not on M0; single-document ACID only |
|
||||
| **Change streams** | ❌ Not on M0 |
|
||||
| **GridFS** | ❌ Not on M0 |
|
||||
|
||||
#### Cost-Benefit for miti99bot
|
||||
|
||||
**Against:**
|
||||
1. **Throughput bottleneck is fake** — 100 ops/sec is enough for this workload (12 active modules, casual daily traffic), but doesn't justify the architecture cost
|
||||
2. **Cold-start latency** — User sends `/wordle` command → Telegram → Worker cold-start → TLS+auth to Atlas = added 1500ms+ wait. KV is instant (cached). Upstash is HTTP (200-300ms).
|
||||
3. **No improvement over KV quota** — User's real pain is hitting the 1k/day write limit. Atlas removes that cap but adds latency. Upstash removes both.
|
||||
4. **Operational debt** — Miti99bot was built around KV/D1 abstractions. Swapping to MongoDB means:
|
||||
- Rewriting `src/db/create-store.js` → `mongodb-store.js`
|
||||
- Rewriting every module's data model (arrays → collections, expiry → TTL indexes)
|
||||
- No transactions → can't safely increment leaderboards atomically
|
||||
- Unfamiliar failure modes (connection errors, timeouts, replica-set failovers)
|
||||
|
||||
**For:**
|
||||
- "Real" database (schemas, indexes, aggregation pipeline)
|
||||
- Free tier is permanent (unlike some services)
|
||||
- Strong community + documentation
|
||||
|
||||
---
|
||||
|
||||
### 2. Upstash Redis (Recommended Primary)
|
||||
|
||||
#### Specification
|
||||
- **FREE tier (March 2025)**:
|
||||
- 500K commands/month (was 10K/day, increased 50x)
|
||||
- Unlimited storage in free tier
|
||||
- 1GB data limit on free tier (soft; upgrade if exceeded)
|
||||
- 200 GB bandwidth/month free
|
||||
- **Cost**: $1 / million commands on paid tier ($0.0000000001 per command); free tier never bills
|
||||
- **Auth**: Token-based HTTP (no TCP)
|
||||
- **Regions**: global edge locations
|
||||
|
||||
#### Cloudflare Workers Compatibility ✅ (Native)
|
||||
|
||||
**Perfect fit.** Upstash is **designed for Workers.**
|
||||
|
||||
**Setup:**
|
||||
- `npm install @upstash/redis`
|
||||
- Environment: REST endpoint + token (HTTP auth, no TLS handshake cost)
|
||||
- `wrangler.toml`: no special flags needed; standard env vars
|
||||
|
||||
**Evidence:**
|
||||
- [Use Redis in Cloudflare Workers (Upstash docs)](https://upstash.com/docs/redis/tutorials/cloudflare_workers_with_redis)
|
||||
- [Cloudflare Workers database integration with Upstash](https://blog.cloudflare.com/cloudflare-workers-database-integration-with-upstash/)
|
||||
- [New Pricing (March 2025)](https://upstash.com/blog/redis-new-pricing) — 500K cmd/month free
|
||||
|
||||
#### Operational Trade-offs
|
||||
|
||||
| Dimension | Status |
|
||||
|-----------|--------|
|
||||
| **Cold-start cost** | ✅ MINIMAL — HTTP-based, no TLS handshake; typical latency 50-200ms to nearest Upstash PoP |
|
||||
| **Connection pooling** | ✅ NOT NEEDED — Stateless HTTP; no per-connection overhead |
|
||||
| **Memoization** | ✅ Cache client per isolate (module-scope static) for warm requests |
|
||||
| **KV abstraction fit** | ✅ PERFECT — Redis strings/hashes are KV-shaped; `@upstash/redis` API mirrors `getJSON`/`putJSON` |
|
||||
| **Expiry (TTL)** | ✅ Redis `EX` / `EXAT` — supported natively |
|
||||
| **Transactions** | ✅ Redis multi/exec (ACID per-key) |
|
||||
| **Data types** | ✅ Strings, hashes, lists, sets, sorted sets (richer than KV) |
|
||||
| **Persistence** | ✅ Redis is durable; data survives restarts |
|
||||
| **Local dev** | ✅ Works in `wrangler dev` with real Upstash instance (not local redis needed) |
|
||||
| **Operations limit** | ✅ 500K/month = ~16,600 ops/day free; daily quota of games (wordle, loldle×5, trading, etc.) is ~2-5k reads + 500-1k writes = within quota |
|
||||
| **Cost overflow** | ✅ If bot goes viral: $1/million commands on overage; at 100k ops/day = $3/month extra |
|
||||
|
||||
#### Fit for Modules
|
||||
|
||||
| Module | Current | Proposed | Notes |
|
||||
|--------|---------|----------|-------|
|
||||
| `wordle`, `loldle*`, `lolschedule`, `semantle`, `doantu`, `twentyq`, `misc` | KV store | Upstash (string/hash) | Direct swap; prefixing works |
|
||||
| `trading` | D1 + KV | **Turso** (SQL) | Append-only log + leaderboard queries; SQL better fit |
|
||||
| `util` | KV | Upstash | Subscription lists, daily state |
|
||||
|
||||
**Trade-off:** Lose `list()` pagination (KV has it; Upstash Redis does not). Solution: use Redis `KEYS` pattern matching or maintain an explicit set of subscription keys.
|
||||
|
||||
#### Cost Breakdown (Realistic Scenario)
|
||||
|
||||
Assume 5k reads + 1k writes/day (conservative for active bot):
|
||||
- **Monthly ops**: (5k + 1k) × 30 = 180,000 → **under 500k free limit ✅**
|
||||
- **Bandwidth**: Game assets (loldle splash art, etc.) ~10MB/day → 300MB/month → **under 200GB free limit ✅**
|
||||
- **Cost**: $0 (free tier)
|
||||
- **Overflow risk** (10x traffic): 1.8M ops → $1.80/month ✅
|
||||
|
||||
---
|
||||
|
||||
### 3. Turso (libSQL) — SQL Alternative / D1 Replacement
|
||||
|
||||
#### Specification
|
||||
- **FREE tier**:
|
||||
- 1 billion row reads/month
|
||||
- 25 million writes/month
|
||||
- 5 GB storage
|
||||
- 3 databases
|
||||
- 3 locations (replication)
|
||||
- **Cost**: Pay-as-you-go ($0.000002/read, $0.00002/write after free tier)
|
||||
- **Protocol**: HTTP (REST API via `@libsql/client`)
|
||||
- **Advantage**: SQLite-compatible, edge-hosted
|
||||
|
||||
#### Cloudflare Workers Compatibility ✅ (Native HTTP)
|
||||
|
||||
**Setup:**
|
||||
- `npm install @libsql/client`
|
||||
- Standard HTTP; no TCP needed
|
||||
|
||||
**Evidence:** [Turso + Cloudflare Workers](https://developers.cloudflare.com/workers/tutorials/connect-to-turso-using-workers/)
|
||||
|
||||
#### Trade-offs for miti99bot
|
||||
|
||||
| Dimension | vs. KV/D1 | vs. Upstash |
|
||||
|-----------|-----------|------------|
|
||||
| **For game state (KV-shaped)** | ⚠️ Overkill — SQL adds latency (prepare → bind → execute) vs. simple get/put | ⚠️ SQL is slower than Redis for simple lookups |
|
||||
| **For trading ledger (SQL-shaped)** | ✅ Natural fit; D1 equiv + cheaper | ✅ SQL is natural; better than Redis (which lacks JOIN) |
|
||||
| **Replication** | ❌ D1 is zoned; Turso is geo-replicated | ✅ Geo-replication on free tier |
|
||||
| **Cold-start** | ✅ HTTP (same as Upstash) | ✅ HTTP |
|
||||
| **Operations ceiling** | ✅ 1B reads/month >> KV 100k/day | ✅ 1B reads/month >> game load |
|
||||
| **Storage** | ✅ 5GB >> 512MB KV | ✅ 5GB >> game state |
|
||||
| **Transactions** | ❌ D1 transactions not on Turso free (?) | Need to verify |
|
||||
|
||||
**Practical use:** Keep Upstash for KV (games, state); move `trading` table to Turso. Or go all-in on Turso and use SQLite tables as KV (slower but unifies stack).
|
||||
|
||||
---
|
||||
|
||||
### 4. Neon (Postgres) — SQL Alternative
|
||||
|
||||
#### Specification
|
||||
- **FREE tier** (Oct 2025 update):
|
||||
- 100 CU-hours/month (0.25 CU = 400 hours continuous, 1 CU = 100 hours)
|
||||
- 0.5 GB storage per branch
|
||||
- 5 compute units per month (tiny)
|
||||
- **Cost**: $14/month pro plan
|
||||
- **Advantage**: Full PostgreSQL semantics; better for complex analytics
|
||||
|
||||
#### Cloudflare Workers Compatibility ⚠️ (Hyperdrive Required)
|
||||
|
||||
**Setup:** Neon + [Cloudflare Hyperdrive](https://developers.cloudflare.com/workers/observability/) (connection pooling). Hyperdrive is included free on all Workers plans.
|
||||
|
||||
**Trade-offs:**
|
||||
- ✅ Full SQL power
|
||||
- ⚠️ Free tier is weak (0.5 GB storage, 100 CU-h/mo) — trading module alone could exhaust this
|
||||
- ⚠️ Hyperdrive adds latency (one more hop)
|
||||
- **Not recommended for miti99bot** — overkill for the workload
|
||||
|
||||
---
|
||||
|
||||
### 5. PlanetScale (MySQL) — Quick Mention
|
||||
|
||||
- **FREE tier**: One serverless cluster, 1 GB storage, 1M row reads/month
|
||||
- **Workers compatible**: Yes, via Hyperdrive or `@planetscale/database` client
|
||||
- **Trade**: Like Neon, overkill for bot state. Better for e-commerce platforms.
|
||||
|
||||
---
|
||||
|
||||
### 6. Status Quo + Optimization (KV Batching)
|
||||
|
||||
Can the bot survive on current KV + D1 by batching writes and using Durable Objects?
|
||||
|
||||
#### Approach
|
||||
- **Durable Objects** as write coordinator: batch 5-10 game turns into 1 KV write
|
||||
- Trade: adds 50-200ms latency for coordination
|
||||
- Complexity: need to manage DO lifecycle, billing (Durable Objects are $0.15/million requests + $0.15/GB storage)
|
||||
|
||||
#### Reality Check
|
||||
- Solves the **daily quota problem** (batches reduce write count by 80%)
|
||||
- But **adds latency** (DO round-trip)
|
||||
- **Costs more** than Upstash free tier ($0 vs. $0.15/million reqs + storage)
|
||||
- **Operational debt** — Durable Objects are stateful; risk of stale writes if DO crashes
|
||||
|
||||
**Verdict:** Not recommended. Upstash is simpler + cheaper + faster.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation Ranking
|
||||
|
||||
### 🥇 **Primary: Upstash Redis**
|
||||
|
||||
**Rationale:**
|
||||
1. **Solves the pain** — 500K cmd/month free = 16.6k/day = 10x current KV quota
|
||||
2. **Minimal latency** — HTTP native; no TLS handshake cost
|
||||
3. **KV-compatible** — Direct `@upstash/redis` drop-in for current `KVStore` interface
|
||||
4. **Free** — Overflow cost is negligible ($3/month at 10x traffic)
|
||||
5. **Proven** — Officially supported by Cloudflare, battle-tested on Workers
|
||||
|
||||
**Implementation effort:** 3-4 hours
|
||||
- Implement `upstash-store.js` (KVStore wrapper around `@upstash/redis`)
|
||||
- Update `create-store.js` to instantiate Upstash
|
||||
- Update `wrangler.toml` (add env vars)
|
||||
- Swap KV namespaces → Upstash endpoint + token
|
||||
- Test `npm test` (should pass with fakes unchanged)
|
||||
|
||||
**Risk:** Low. Redis is simpler than MongoDB; no transactions needed for games.
|
||||
|
||||
---
|
||||
|
||||
### 🥈 **Secondary (Future): Turso for Trading Module Only**
|
||||
|
||||
**When to do:** After Upstash is live, if trading module grows (leaderboard queries, filtering by date range).
|
||||
|
||||
**Plan:**
|
||||
- Keep Upstash for game state (KV)
|
||||
- Move `trading_trades` table to Turso
|
||||
- Rationale: SQL is better for aggregates (top 10 traders, this month's stats)
|
||||
|
||||
**Implementation effort:** 2-3 hours
|
||||
- Implement `turso-sql-store.js`
|
||||
- Migrate `trading` migrations from D1 to Turso
|
||||
- Test
|
||||
|
||||
---
|
||||
|
||||
### ❌ **Reject: MongoDB Atlas M0**
|
||||
|
||||
**Why not:**
|
||||
1. **Cold-start latency** — TLS handshake adds 1-2s per cold start; game commands feel slow
|
||||
2. **No quota relief** — Throughput is 100 ops/sec (fine), but no daily op cap (irrelevant; current bottleneck is KV quota, not throughput)
|
||||
3. **Architectural mismatch** — Bot is read-heavy game state + append-only log; KV + D1 are already optimized; MongoDB adds schema management overhead
|
||||
4. **Operational complexity** — SCRAM auth, connection pooling issues, single-document ACID only (no leaderboard transactions)
|
||||
5. **Zero upside** — Upstash does everything Atlas does for this workload, faster, cheaper, simpler
|
||||
|
||||
---
|
||||
|
||||
## MongoDB Driver Specifics (If You Ignore This Recommendation)
|
||||
|
||||
If the team insists on MongoDB, here's what works:
|
||||
|
||||
### Requirements
|
||||
- **Node.js driver**: v6.7+ (`npm install mongodb@^6.7.0`)
|
||||
- **wrangler.toml**:
|
||||
```toml
|
||||
compatibility_date = "2025-03-20"
|
||||
compatibility_flags = ["nodejs_compat_v2"]
|
||||
```
|
||||
- **Env var**: `MONGODB_URI = "mongodb+srv://user:pass@cluster-hash.mongodb.net/db"`
|
||||
|
||||
### Cold-Start Cost
|
||||
- First request per isolate: ~1500ms (TLS + SCRAM auth)
|
||||
- Subsequent requests on same warm isolate: ~50-100ms (connection reused)
|
||||
- **Problem**: Cloudflare Workers can spawn new isolates at any time; you can't rely on warmth
|
||||
|
||||
### Memoization Pattern
|
||||
```js
|
||||
let mongoClient = null;
|
||||
|
||||
export async function getMongoClient(env) {
|
||||
if (!mongoClient) {
|
||||
mongoClient = new MongoClient(env.MONGODB_URI, {
|
||||
maxPoolSize: 1, // minimize resource use
|
||||
minPoolSize: 0, // no background connections
|
||||
serverSelectionTimeoutMS: 5000,
|
||||
});
|
||||
await mongoClient.connect();
|
||||
}
|
||||
return mongoClient;
|
||||
}
|
||||
```
|
||||
|
||||
**Caveat:** This doesn't solve cold-start latency; it just reuses the connection on warm requests.
|
||||
|
||||
### Connection String Format
|
||||
- ✅ `mongodb+srv://...` works (Cloudflare resolves SRV records via DNS)
|
||||
- ✅ `mongodb://shard0.host,shard1.host:27017` works (explicit hosts)
|
||||
- ✅ TLS is default; root CA bundled in driver
|
||||
|
||||
### Known Limitations
|
||||
- ❌ No replica-set sessions (sharded transactions)
|
||||
- ❌ No change streams
|
||||
- ❌ No GridFS
|
||||
- ❌ M0 has no backups, auditing, or custom auth mechanisms
|
||||
|
||||
### Local Dev
|
||||
- `wrangler dev` → Workers Sockets API → Atlas (works as-is)
|
||||
- No `--local` flag needed; no manual routing required
|
||||
|
||||
---
|
||||
|
||||
## Comparison Matrix
|
||||
|
||||
| Dimension | KV (Status Quo) | Upstash | Turso | Neon | **Atlas** |
|
||||
|-----------|-----------------|---------|-------|------|-----------|
|
||||
| **Daily quota** | 100k reads, 1k writes | 500k cmd/mo | 1B reads, 25M writes | 100 CU-h/mo | Unlimited (100 ops/sec) |
|
||||
| **Cold-start latency** | Instant (cached) | ~100ms HTTP | ~100ms HTTP | ~200ms Hyperdrive | **~1500ms TLS** |
|
||||
| **KV fit** | ✅ Native | ✅ Perfect (Redis) | ⚠️ SQL overhead | ⚠️ SQL overhead | ⚠️ Document model |
|
||||
| **SQL fit** | ❌ None | ❌ No SQL | ✅ SQL | ✅ PostgreSQL | ⚠️ BSON |
|
||||
| **Memoization** | N/A | ✅ HTTP client | ✅ HTTP client | ✅ Pooled | ⚠️ Connection pooling hard |
|
||||
| **Free cost** | $0 (quota limit) | $0 | $0 | $0 (weak quota) | $0 (auto-pause after 30d) |
|
||||
| **Operational complexity** | Low | Low | Low | Medium | **High** |
|
||||
| **Workers native** | ✅ KV binding | ✅ HTTP | ✅ HTTP | ⚠️ Hyperdrive | ⚠️ Node.js compat |
|
||||
| **Risk level** | Low (proven) | Low (proven) | Low (proven) | Medium (big compute) | **High (cold-start, pooling)** |
|
||||
|
||||
---
|
||||
|
||||
## Adoption Risk Assessment
|
||||
|
||||
### Upstash
|
||||
- **Maturity**: Production-ready (2018+, YC-backed)
|
||||
- **Breaking changes**: Rare; API stable since 2021
|
||||
- **Abandonment risk**: Very low (funded, profitable, enterprise customers)
|
||||
- **Community**: 10k+ GitHub stars; active Discord
|
||||
|
||||
### Turso / libSQL
|
||||
- **Maturity**: 2023+; newer but growing fast
|
||||
- **Breaking changes**: Some API churn early on, stabilizing
|
||||
- **Abandonment risk**: Low (Chiselstrike backing, open-source)
|
||||
- **Community**: 5k+ GitHub stars; smaller but active
|
||||
|
||||
### MongoDB
|
||||
- **Maturity**: Highly stable (20+ years)
|
||||
- **Breaking changes**: Rare
|
||||
- **Abandonment risk**: None (massive company)
|
||||
- **Community**: Massive
|
||||
- **Problem**: Not designed for serverless; Cloudflare compat is **new and fragile** (Q1 2025)
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Turso transaction support on free tier** — Need to confirm if free tier allows `BEGIN`/`COMMIT` or only single-statement atomicity.
|
||||
2. **Upstash key namespace collision across modules** — If two modules both use a key named `state`, does Upstash handle prefixing? (Likely yes, but need to test)
|
||||
3. **Redis KEYS pattern on large free tier storage** — If bot accumulates 1GB in Upstash free tier, will `KEYS wordle:*` scans become slow? (Probably fine; Upstash is optimized for this)
|
||||
4. **MongoDB cold-start in production** — The blog post tests locally; real-world Cloudflare production cold-start latency is unknown. Need real-world telemetry.
|
||||
5. **Durable Objects write coalescing** — Can DO batch the writes below into a single KV transaction? (Answer: yes, automatic; but do we need it for games?)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps (If Upstash Recommended)
|
||||
|
||||
1. **Create `src/db/upstash-store.js`** — Implement `KVStore` interface using `@upstash/redis`
|
||||
2. **Update `src/db/create-store.js`** — Swap `new CFKVStore(env.KV)` → `new UpstashStore(env)`
|
||||
3. **Update `wrangler.toml`** — Remove KV namespace bindings; add `UPSTASH_REDIS_REST_URL` and `UPSTASH_REDIS_REST_TOKEN`
|
||||
4. **Test locally** — `npm test` (fakes unchanged; tests should pass)
|
||||
5. **Deploy** — `npm run deploy`
|
||||
6. **Monitor** — Track Redis command usage via Upstash console; should stay under 500k/mo
|
||||
|
||||
**Estimated effort**: ~4-6 hours (including testing + docs update).
|
||||
|
||||
---
|
||||
|
||||
## Sources Cited
|
||||
|
||||
1. [Cloudflare Workers and MongoDB (March 2025)](https://alexbevi.com/blog/2025/03/25/cloudflare-workers-and-mongodb/)
|
||||
2. [A Year of Improving Node.js Compatibility in Cloudflare Workers](https://blog.cloudflare.com/nodejs-workers-2025/)
|
||||
3. [MongoDB Atlas Free Cluster Limits (Official Docs)](https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/)
|
||||
4. [Cloudflare Workers KV Pricing & Limits](https://developers.cloudflare.com/kv/platform/pricing/)
|
||||
5. [New Pricing and Increased Limits for Upstash Redis (March 2025)](https://upstash.com/blog/redis-new-pricing)
|
||||
6. [Use Redis in Cloudflare Workers (Upstash Docs)](https://upstash.com/docs/redis/tutorials/cloudflare_workers_with_redis)
|
||||
7. [Cloudflare Workers Database Integration with Upstash](https://blog.cloudflare.com/cloudflare-workers-database-integration-with-upstash/)
|
||||
8. [Turso + Cloudflare Workers Integration](https://developers.cloudflare.com/workers/tutorials/connect-to-turso-using-workers/)
|
||||
9. [Neon with Cloudflare Workers (Docs)](https://developers.cloudflare.com/workers/databases/third-party-integrations/neon/)
|
||||
10. [Connect to Databases (Cloudflare Workers Docs)](https://developers.cloudflare.com/workers/databases/connecting-to-databases/)
|
||||
11. [Cloudflare Durable Objects — Write Coalescing](https://developers.cloudflare.com/durable-objects/concepts/what-are-durable-objects/)
|
||||
@@ -1,783 +0,0 @@
|
||||
# MongoDB Atlas Data Layer: Schema Design & Migration Plan
|
||||
|
||||
**Status:** Research-only. No assumptions made on whether Atlas is chosen; schema designs are transferable to other document stores.
|
||||
|
||||
**Date:** 2026-04-25
|
||||
**Scope:** Data layer architecture for `miti99bot` assuming MongoDB Atlas backend; includes M0 cost ceiling, KV/SQL schema mappings, dual-write strategy.
|
||||
|
||||
---
|
||||
|
||||
## 1. M0 Hard Ceiling (Specifications)
|
||||
|
||||
### Storage
|
||||
- **512 MB** hard limit (includes data + indexes)
|
||||
- Index overhead: ~5–10% per index. TTL indexes add minimal overhead (~1–2% per collection).
|
||||
- No tiered storage; no compression options on M0.
|
||||
|
||||
### Connections
|
||||
- **500 concurrent connections** shared across all Workers.
|
||||
- No per-project or per-cluster overrides.
|
||||
|
||||
### Throughput
|
||||
- No explicit ops/sec cap published; shared resources mean degradation under heavy load.
|
||||
- Typical M0: 100–200 ops/sec sustained before queueing (non-SLA).
|
||||
|
||||
### Network
|
||||
- No egress cap; egress charges apply to paid tiers only.
|
||||
- No cross-region replication on M0 (M0 = single region only).
|
||||
|
||||
### Backup / PITR
|
||||
- **No backups** on M0. Single-point-of-failure design.
|
||||
- Snapshots not available.
|
||||
- Implications: Migrations must be append-only or dual-write validated.
|
||||
|
||||
### Regions (M0 Support)
|
||||
M0 available in: **aws-eu-west-1** (Ireland), **aws-ap-southeast-1** (Singapore), **aws-ap-southeast-2** (Sydney), **aws-us-east-1** (N. Virginia), **aws-us-west-2** (Oregon).
|
||||
|
||||
- **For miti99bot:** Recommend **aws-ap-southeast-1** (Singapore) — lowest latency to Cloudflare SEA PoPs; alternative **aws-eu-west-1** if EU users dominant.
|
||||
- Cloudflare Workers run in 275+ edge locations globally; data gravity favors nearest geographic center.
|
||||
|
||||
### Upgrade Path (M0 → Paid Tiers)
|
||||
- **M0 → Flex Tier (recommended):** $8–$30/month, auto-scales, includes backups.
|
||||
- **M0 → M10 (dedicated):** $57/month, fixed capacity, backups, multi-region replication.
|
||||
- M2/M5 (shared tiers) **deprecated as of 2026** — no longer available for new projects.
|
||||
|
||||
### When M0 Hits Ceiling?
|
||||
| Metric | Current miti99bot | M0 Limit | Runway |
|
||||
|--------|---|---|---|
|
||||
| Storage | ~1 MB/month (615 KB KV + 300 KB D1 trades) | 512 MB | **512 months** (43 years) |
|
||||
| Connections | ~10–50/min bursts | 500 | **10x headroom** |
|
||||
| Ops/sec (nominal) | ~5–20 ops/sec sustained | ~100–200 | **5–40x headroom** |
|
||||
|
||||
**Verdict:** M0 sufficient for 2–3 years at current growth. Monitor: (1) active user count, (2) trades/day (trading table grows fastest), (3) API cache churn (lolschedule, trading prices).
|
||||
|
||||
---
|
||||
|
||||
## 2. KV → Document Schema
|
||||
|
||||
### Design Decision: Per-Module Collections vs. Shared KV Collection
|
||||
|
||||
**Choice: Per-module collections** (one collection per module).
|
||||
|
||||
**Rationale:**
|
||||
- **Separation of concerns** — indexes scoped per module; smaller collections reduce scan overhead.
|
||||
- **Query isolation** — each module's `_id` index is independent; no compound key overhead.
|
||||
- **Operational clarity** — `db.wordle.find()` vs. `db.kv.find({module: "wordle"})`.
|
||||
- **Cons** — 13 collections instead of 1 (schema complexity); requires per-module migration.
|
||||
|
||||
**Alternative rejected:** Shared `kv` collection with `{module, key}` compound `_id`.
|
||||
- Pro: single schema.
|
||||
- Con: noisy queries, shared index, harder to reason about data size per module.
|
||||
|
||||
### Document Shape
|
||||
|
||||
```javascript
|
||||
{
|
||||
_id: "<key>", // string, e.g. "games:42"
|
||||
value: <JSON string>, // serialized (matches today's putJSON)
|
||||
expiresAt: ISODate | undefined // for TTL; absent = no expiration
|
||||
}
|
||||
```
|
||||
|
||||
**Why store `value` as string (not native BSON object)?**
|
||||
- Today: KVStore uses `putJSON(key, obj)` → serializes to string internally.
|
||||
- Preserves round-trip fidelity for nested objects, arrays, null values.
|
||||
- Avoids schema drift (BSON doesn't have a true null field vs. missing field distinction).
|
||||
- Trade-off: Slightly higher query cost (string parsing in app layer). Mitigated by per-module collections (smaller indexes).
|
||||
|
||||
**Example documents:**
|
||||
|
||||
```javascript
|
||||
// wordle game state
|
||||
{ _id: "games:42", value: "{\"word\": \"apple\", \"guesses\": [...]}", expiresAt: ISODate("2026-04-26T00:00:00Z") }
|
||||
|
||||
// loldle stats (no expiration)
|
||||
{ _id: "stats:42", value: "{\"wins\": 5, \"streak\": 2}" }
|
||||
|
||||
// trading portfolio
|
||||
{ _id: "user:123", value: "{\"vnd\": 50000, \"holdings\": [{\"symbol\": \"ACB\", \"qty\": 100}]}" }
|
||||
```
|
||||
|
||||
### TTL Index (for expirationTtl support)
|
||||
|
||||
Create on every module collection:
|
||||
```javascript
|
||||
db.<module>.createIndex(
|
||||
{ expiresAt: 1 },
|
||||
{ expireAfterSeconds: 0, sparse: true }
|
||||
);
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `expireAfterSeconds: 0` — respect exact `expiresAt` timestamp (not relative TTL).
|
||||
- `sparse: true` — don't index documents without `expiresAt` (stats, permanent data).
|
||||
|
||||
**Behavior:** MongoDB background thread checks every 60 seconds; documents deleted 0–60 sec after expiration. Acceptable for games (session TTL ~24h).
|
||||
|
||||
### list({prefix, limit, cursor}) Implementation
|
||||
|
||||
**Current KV API:**
|
||||
```javascript
|
||||
const result = await db.list({ prefix: "games:", limit: 10, cursor: "..." });
|
||||
// Returns: { keys: [...], cursor?: "...", done: boolean }
|
||||
```
|
||||
|
||||
**MongoDB equivalent (cursor pagination):**
|
||||
|
||||
```javascript
|
||||
async function list(opts = {}) {
|
||||
const { prefix = "", limit = 10, cursor } = opts;
|
||||
|
||||
// Build regex for prefix matching
|
||||
const query = prefix
|
||||
? { _id: { $regex: `^${escapeRegex(prefix)}` } }
|
||||
: {};
|
||||
|
||||
// Decode cursor (opaque base64 string = last seen _id)
|
||||
const after = cursor ? Buffer.from(cursor, "base64").toString() : null;
|
||||
if (after) {
|
||||
query._id = { ...query._id, $gt: after };
|
||||
}
|
||||
|
||||
// Fetch limit + 1 to detect if more pages exist
|
||||
const docs = await collection
|
||||
.find(query)
|
||||
.sort({ _id: 1 })
|
||||
.limit(limit + 1)
|
||||
.toArray();
|
||||
|
||||
const keys = docs.slice(0, limit).map(d => d._id.replace(prefix, ""));
|
||||
const hasMore = docs.length > limit;
|
||||
const nextCursor = hasMore
|
||||
? Buffer.from(docs[limit - 1]._id).toString("base64")
|
||||
: null;
|
||||
|
||||
return {
|
||||
keys,
|
||||
cursor: nextCursor,
|
||||
done: !hasMore
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Why sorted `_id` cursor instead of `$regex` + `skip()`:**
|
||||
- `skip()` is O(n) on large collections; cursor is O(1) pointer.
|
||||
- Regex `/^prefix/` is index-optimizable (MongoDB recognizes prefix patterns).
|
||||
- Combined: scan stops after `limit` docs; cursor encodes the breakpoint.
|
||||
|
||||
### Indexes per Module Collection
|
||||
|
||||
```javascript
|
||||
// Automatically created on all module collections:
|
||||
db.<module>.createIndex({ expiresAt: 1 }, { expireAfterSeconds: 0, sparse: true });
|
||||
|
||||
// On collections that support prefix queries (most):
|
||||
db.<module>.createIndex({ _id: 1 }); // implicit, already exists as primary key
|
||||
|
||||
// Example: if a module needs to query by a secondary field:
|
||||
// (not used today, but extensible — e.g., lolschedule might index by subscriber ID)
|
||||
db.lolschedule.createIndex({ subscriber_id: 1 }); // for bulk deletes
|
||||
```
|
||||
|
||||
**Index count:** 13 collections × 2 indexes (PK + TTL sparse) = 26 indexes. Well under M0's soft limit (~50 before performance degrades).
|
||||
|
||||
---
|
||||
|
||||
## 3. SqlStore → Document Model (trading)
|
||||
|
||||
### Current D1 Schema
|
||||
```sql
|
||||
CREATE TABLE trading_trades (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
user_id INTEGER NOT NULL,
|
||||
symbol TEXT NOT NULL,
|
||||
side TEXT NOT NULL CHECK (side IN ('buy','sell')),
|
||||
qty INTEGER NOT NULL,
|
||||
price_vnd INTEGER NOT NULL,
|
||||
ts INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_trading_trades_user_ts ON trading_trades(user_id, ts DESC);
|
||||
CREATE INDEX idx_trading_trades_ts ON trading_trades(ts);
|
||||
```
|
||||
|
||||
### Mapped to MongoDB
|
||||
|
||||
**Collection:** `trading_trades` (one document per trade).
|
||||
|
||||
**Document:**
|
||||
```javascript
|
||||
{
|
||||
_id: ObjectId(), // MongoDB auto-generated
|
||||
user_id: 123, // integer, matches D1
|
||||
symbol: "ACB", // string
|
||||
side: "buy" | "sell", // enum-like
|
||||
qty: 100, // integer
|
||||
price_vnd: 25000, // integer (VND in paisa, not float)
|
||||
ts: 1713976800000, // timestamp (milliseconds)
|
||||
createdAt: ISODate(...) // for immutable audit log feel (optional)
|
||||
}
|
||||
```
|
||||
|
||||
**Indexes:**
|
||||
```javascript
|
||||
db.trading_trades.createIndex({ user_id: 1, ts: -1 });
|
||||
db.trading_trades.createIndex({ ts: -1 });
|
||||
```
|
||||
|
||||
**Why no `_id` on `user_id + ts`?**
|
||||
- D1 uses `id (autoincrement)` as PK; MongoDB's ObjectId already serves that role.
|
||||
- Compound `_id` would be overkill; separate indexes are cleaner.
|
||||
|
||||
### Aggregation Pipelines (for queries trading does)
|
||||
|
||||
**Query: Last 10 trades for user_id=123**
|
||||
|
||||
D1:
|
||||
```sql
|
||||
SELECT * FROM trading_trades WHERE user_id = ? ORDER BY ts DESC LIMIT 10;
|
||||
```
|
||||
|
||||
MongoDB:
|
||||
```javascript
|
||||
db.trading_trades.aggregate([
|
||||
{ $match: { user_id: 123 } },
|
||||
{ $sort: { ts: -1 } },
|
||||
{ $limit: 10 }
|
||||
]);
|
||||
```
|
||||
|
||||
**Query: Leaderboard (top 5 users by trade count)**
|
||||
|
||||
D1 (hypothetical):
|
||||
```sql
|
||||
SELECT user_id, COUNT(*) as trade_count
|
||||
FROM trading_trades
|
||||
GROUP BY user_id
|
||||
ORDER BY trade_count DESC
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
MongoDB:
|
||||
```javascript
|
||||
db.trading_trades.aggregate([
|
||||
{ $group: { _id: "$user_id", trade_count: { $sum: 1 } } },
|
||||
{ $sort: { trade_count: -1 } },
|
||||
{ $limit: 5 }
|
||||
]);
|
||||
```
|
||||
|
||||
### No Transactions Required (for now)
|
||||
|
||||
- All trading operations are **append-only writes** (INSERT trades).
|
||||
- Portfolio updates (KV: `user:123`) are idempotent.
|
||||
- No multi-document ACID needed.
|
||||
|
||||
If future features (e.g., reversals, corrections) require atomicity, MongoDB 4.0+ supports multi-document transactions. M0 supports replica sets; transactions work on replica sets.
|
||||
|
||||
---
|
||||
|
||||
## 4. Per-Module Storage Map
|
||||
|
||||
| Module | Today | Prefix | Key Shapes | Est. Doc Count | Est. Size | TTL | Aggregation |
|
||||
|--------|-------|--------|-----------|---|---|---|---|
|
||||
| **wordle** | KV | `wordle:` | `games:<uid>`, `stats:<uid>` | 100 | ~70 KB | game state 24h | none |
|
||||
| **loldle** | KV | `loldle:` | `games:<uid>`, `stats:<uid>` | 100 | ~70 KB | game state 24h | none |
|
||||
| **loldle-emoji** | KV | `loldle_emoji:` | `games:<uid>`, `stats:<uid>` | 100 | ~70 KB | game state 24h | none |
|
||||
| **loldle-quote** | KV | `loldle_quote:` | `games:<uid>`, `stats:<uid>` | 100 | ~70 KB | game state 24h | none |
|
||||
| **loldle-ability** | KV | `loldle_ability:` | `games:<uid>`, `stats:<uid>` | 100 | ~70 KB | game state 24h | none |
|
||||
| **loldle-splash** | KV | `loldle_splash:` | `games:<uid>`, `stats:<uid>` | 100 | ~70 KB | game state 24h | none |
|
||||
| **lolschedule** | KV | `lolschedule:` | `events:<day>`, `subscribers` | 10 | ~60 KB | events 24h | none |
|
||||
| **trading** | KV + D1 | `trading:` | `user:<uid>`, `symbols:<symbol>`, `forex:cache` | 100 + trades/day | ~60 KB (KV) + 300 KB/mo (D1) | cache 1–24h | yes (leaderboard, stats) |
|
||||
| **semantle** | KV | `semantle:` | `games:<uid>`, `stats:<uid>` | 50 | ~35 KB | game state 24h | none |
|
||||
| **doantu** | KV | `doantu:` | `games:<uid>`, `stats:<uid>` | 50 | ~35 KB | game state 24h | none |
|
||||
| **twentyq** | KV | `twentyq:` | `game:<subject_id>`, `stats:<uid>` | 30 | ~21 KB | game 24h | none |
|
||||
| **misc** | KV | `misc:` | `last_ping` | 1 | <1 KB | none | none |
|
||||
| **util** | — | — | — | — | — | — | — |
|
||||
| **TOTAL** | — | — | — | ~700 docs | ~800 KB + 300 KB/mo | — | — |
|
||||
|
||||
**Notes:**
|
||||
- Most collections fit in single page of indexes.
|
||||
- No module does complex aggregations today.
|
||||
- `trading` is append-only (grows fastest).
|
||||
|
||||
---
|
||||
|
||||
## 5. Migration Mechanics (Dual-Write Window)
|
||||
|
||||
### Phase 1: Dual-Write Wrapper (no cutover yet)
|
||||
|
||||
Wrap both CF KV and Mongo KV in a `DualKVStore`:
|
||||
|
||||
```javascript
|
||||
// src/db/dual-kv-store.js
|
||||
export class DualKVStore {
|
||||
constructor(cfKv, mongoKv, logger) {
|
||||
this.cf = cfKv;
|
||||
this.mongo = mongoKv;
|
||||
this.logger = logger; // log divergences
|
||||
}
|
||||
|
||||
async get(key) {
|
||||
const cfVal = await this.cf.get(key);
|
||||
const mongoVal = await this.mongo.get(key);
|
||||
|
||||
if (cfVal !== mongoVal) {
|
||||
this.logger.warn(`divergence:get`, { key, cf: cfVal?.length, mongo: mongoVal?.length });
|
||||
}
|
||||
// Read from CF (primary)
|
||||
return cfVal;
|
||||
}
|
||||
|
||||
async put(key, value, opts) {
|
||||
// Write to both; fail if either fails
|
||||
const [cfRes, mongoRes] = await Promise.all([
|
||||
this.cf.put(key, value, opts),
|
||||
this.mongo.put(key, value, opts).catch(err => {
|
||||
this.logger.error(`dual-write:mongo:failed`, { key, err });
|
||||
throw err;
|
||||
})
|
||||
]);
|
||||
return cfRes;
|
||||
}
|
||||
|
||||
// ... delete, list, getJSON, putJSON with same dual-write + primary-read pattern
|
||||
}
|
||||
```
|
||||
|
||||
**Usage in `create-store.js`:**
|
||||
```javascript
|
||||
export function createStore(moduleName, env) {
|
||||
const cfKv = new CFKVStore(env.KV);
|
||||
const mongoKv = new MongoKVStore(env.MONGO_URL, moduleName);
|
||||
|
||||
// Dual-write enabled only if DUAL_WRITE_MODE=1
|
||||
if (env.DUAL_WRITE_MODE === "1") {
|
||||
return new DualKVStore(cfKv, mongoKv, env.logger);
|
||||
}
|
||||
// Otherwise use one or the other
|
||||
return env.USE_MONGO === "1" ? mongoKv : cfKv;
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Dual-Write for trading (D1 → Mongo)
|
||||
|
||||
Similar wrapper for `SqlStore`:
|
||||
|
||||
```javascript
|
||||
export class DualSqlStore {
|
||||
constructor(cfSql, mongoSql, logger) {
|
||||
this.cf = cfSql;
|
||||
this.mongo = mongoSql;
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
async run(query, ...binds) {
|
||||
const cfRes = await this.cf.run(query, ...binds);
|
||||
|
||||
// Map INSERT/UPDATE/DELETE to Mongo equivalent
|
||||
// For trading: all writes are INSERT, so just forward
|
||||
const mongoRes = await this.mongo.run(query, ...binds).catch(err => {
|
||||
this.logger.error(`dual-write:mongo:run:failed`, { query, err });
|
||||
throw err; // abort entire write on Mongo failure
|
||||
});
|
||||
|
||||
return cfRes; // return CF result (primary)
|
||||
}
|
||||
|
||||
async all(query, ...binds) {
|
||||
// READ: only from CF (primary)
|
||||
return this.cf.all(query, ...binds);
|
||||
}
|
||||
|
||||
// ... first, batch, prepare with same pattern
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Backfill Script (Historical Data)
|
||||
|
||||
Run once before cutover to copy all existing data from CF to Mongo:
|
||||
|
||||
```javascript
|
||||
// scripts/backfill-mongo.js
|
||||
async function backfill(cfKv, mongoKv, mongoDb) {
|
||||
for (const moduleName of MODULES) {
|
||||
console.log(`Backfilling ${moduleName}...`);
|
||||
|
||||
// KV backfill
|
||||
let cursor = null;
|
||||
let total = 0;
|
||||
|
||||
do {
|
||||
const { keys, cursor: nextCursor, done } = await cfKv.list({
|
||||
prefix: `${moduleName}:`,
|
||||
limit: 100,
|
||||
cursor
|
||||
});
|
||||
|
||||
for (const key of keys) {
|
||||
const value = await cfKv.get(`${moduleName}:${key}`);
|
||||
await mongoKv.put(key, value); // put without prefix (mongo prefixes internally)
|
||||
total++;
|
||||
}
|
||||
|
||||
cursor = nextCursor;
|
||||
} while (!done);
|
||||
|
||||
console.log(` KV: ${total} keys backfilled`);
|
||||
|
||||
// D1 backfill (trading only)
|
||||
if (moduleName === "trading") {
|
||||
const trades = await cfSql.all("SELECT * FROM trading_trades");
|
||||
const mongoCollection = mongoDb.collection("trading_trades");
|
||||
|
||||
if (trades.length > 0) {
|
||||
const docs = trades.map(row => ({
|
||||
user_id: row.user_id,
|
||||
symbol: row.symbol,
|
||||
side: row.side,
|
||||
qty: row.qty,
|
||||
price_vnd: row.price_vnd,
|
||||
ts: row.ts
|
||||
}));
|
||||
await mongoCollection.insertMany(docs);
|
||||
console.log(` D1: ${docs.length} trades backfilled`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: Verification (Before Cutover)
|
||||
|
||||
Compare key counts + sample values:
|
||||
|
||||
```javascript
|
||||
async function verify(cfKv, mongoDb) {
|
||||
const mismatches = [];
|
||||
|
||||
for (const moduleName of MODULES) {
|
||||
// Count keys in CF
|
||||
let cfCount = 0;
|
||||
let cursor = null;
|
||||
do {
|
||||
const { keys, cursor: next, done } = await cfKv.list({
|
||||
prefix: `${moduleName}:`,
|
||||
limit: 1000,
|
||||
cursor
|
||||
});
|
||||
cfCount += keys.length;
|
||||
cursor = next;
|
||||
} while (!cursor);
|
||||
|
||||
// Count docs in Mongo
|
||||
const mongoCount = await mongoDb.collection(moduleName).countDocuments();
|
||||
|
||||
if (cfCount !== mongoCount) {
|
||||
mismatches.push({
|
||||
module: moduleName,
|
||||
cf: cfCount,
|
||||
mongo: mongoCount,
|
||||
diff: mongoCount - cfCount
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (mismatches.length > 0) {
|
||||
console.error("VERIFICATION FAILED:", mismatches);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log("Verification passed. All modules match.");
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 5: Cutover (Single Env Flag)
|
||||
|
||||
Deploy with `USE_MONGO=1`, which flips all reads to Mongo:
|
||||
|
||||
```javascript
|
||||
// In create-store.js, simplified:
|
||||
if (env.USE_MONGO === "1") {
|
||||
return new MongoKVStore(env.MONGO_URL, moduleName);
|
||||
} else {
|
||||
return new CFKVStore(env.KV);
|
||||
}
|
||||
```
|
||||
|
||||
**Cutover steps:**
|
||||
1. Dual-write window: 1–7 days. Monitor logs for divergences.
|
||||
2. If divergences found: investigate, re-backfill, extend window.
|
||||
3. If clean: deploy with `USE_MONGO=1`.
|
||||
4. Monitor: latency, error rates, logs for 24 hours.
|
||||
5. If stable: disable `DUAL_WRITE_MODE` in next deploy (read-only Mongo).
|
||||
|
||||
### Phase 6: Decommission CF KV + D1
|
||||
|
||||
After 30 days of Mongo-only operation:
|
||||
1. Export D1 `trading_trades` (in case).
|
||||
2. Delete KV namespace via CLI: `npx wrangler kv:namespace delete --binding=KV`
|
||||
3. Delete D1 database via CLI: `npx wrangler d1 delete miti99bot-db`
|
||||
4. Remove KV/D1 bindings from `wrangler.toml`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation: MongoKVStore & MongoSqlStore
|
||||
|
||||
### MongoKVStore (implements KVStore interface)
|
||||
|
||||
```javascript
|
||||
// src/db/mongo-kv-store.js
|
||||
import { MongoClient } from "mongodb";
|
||||
|
||||
export class MongoKVStore {
|
||||
constructor(mongoUrl, moduleName) {
|
||||
this.mongoUrl = mongoUrl;
|
||||
this.moduleName = moduleName;
|
||||
this.client = null;
|
||||
this.db = null;
|
||||
this.collection = null;
|
||||
}
|
||||
|
||||
async init() {
|
||||
this.client = new MongoClient(this.mongoUrl);
|
||||
await this.client.connect();
|
||||
this.db = this.client.db("miti99bot");
|
||||
this.collection = this.db.collection(this.moduleName);
|
||||
|
||||
// Ensure TTL index
|
||||
await this.collection.createIndex(
|
||||
{ expiresAt: 1 },
|
||||
{ expireAfterSeconds: 0, sparse: true }
|
||||
);
|
||||
}
|
||||
|
||||
async get(key) {
|
||||
const doc = await this.collection.findOne({ _id: key });
|
||||
return doc?.value || null;
|
||||
}
|
||||
|
||||
async put(key, value, opts = {}) {
|
||||
const update = { value };
|
||||
if (opts.expirationTtl) {
|
||||
update.expiresAt = new Date(Date.now() + opts.expirationTtl * 1000);
|
||||
} else {
|
||||
update.$unset = { expiresAt: "" }; // remove expiration
|
||||
}
|
||||
|
||||
await this.collection.updateOne(
|
||||
{ _id: key },
|
||||
{ $set: update },
|
||||
{ upsert: true }
|
||||
);
|
||||
}
|
||||
|
||||
async delete(key) {
|
||||
await this.collection.deleteOne({ _id: key });
|
||||
}
|
||||
|
||||
async list(opts = {}) {
|
||||
const { prefix = "", limit = 10, cursor } = opts;
|
||||
|
||||
const query = {};
|
||||
if (prefix) {
|
||||
query._id = { $regex: `^${this.escapeRegex(prefix)}` };
|
||||
}
|
||||
if (cursor) {
|
||||
const after = Buffer.from(cursor, "base64").toString();
|
||||
query._id = { ...query._id, $gt: after };
|
||||
}
|
||||
|
||||
const docs = await this.collection
|
||||
.find(query)
|
||||
.sort({ _id: 1 })
|
||||
.limit(limit + 1)
|
||||
.toArray();
|
||||
|
||||
const keys = docs.slice(0, limit).map(d => d._id.replace(prefix, ""));
|
||||
const hasMore = docs.length > limit;
|
||||
const nextCursor = hasMore
|
||||
? Buffer.from(docs[limit - 1]._id).toString("base64")
|
||||
: null;
|
||||
|
||||
return { keys, cursor: nextCursor, done: !hasMore };
|
||||
}
|
||||
|
||||
async getJSON(key) {
|
||||
const val = await this.get(key);
|
||||
if (!val) return null;
|
||||
try {
|
||||
return JSON.parse(val);
|
||||
} catch (err) {
|
||||
console.warn(`getJSON: corrupt JSON at key="${key}"`, err);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async putJSON(key, value, opts = {}) {
|
||||
if (value === undefined || this.hasCycle(value)) {
|
||||
throw new Error(`putJSON: cannot serialize value`);
|
||||
}
|
||||
await this.put(key, JSON.stringify(value), opts);
|
||||
}
|
||||
|
||||
escapeRegex(str) {
|
||||
return str.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
}
|
||||
|
||||
hasCycle(obj, seen = new WeakSet()) {
|
||||
if (typeof obj !== "object" || obj === null) return false;
|
||||
if (seen.has(obj)) return true;
|
||||
seen.add(obj);
|
||||
for (const key in obj) {
|
||||
if (this.hasCycle(obj[key], seen)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
async close() {
|
||||
if (this.client) await this.client.close();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### MongoSqlStore (implements SqlStore interface)
|
||||
|
||||
```javascript
|
||||
// src/db/mongo-sql-store.js
|
||||
export class MongoSqlStore {
|
||||
constructor(mongoUrl, moduleName) {
|
||||
this.mongoUrl = mongoUrl;
|
||||
this.moduleName = moduleName;
|
||||
this.tablePrefix = `${moduleName}_`;
|
||||
this.client = null;
|
||||
this.db = null;
|
||||
}
|
||||
|
||||
async init() {
|
||||
this.client = new MongoClient(this.mongoUrl);
|
||||
await this.client.connect();
|
||||
this.db = this.client.db("miti99bot");
|
||||
}
|
||||
|
||||
async run(query, ...binds) {
|
||||
// Parse SQL query; route INSERT/UPDATE/DELETE to appropriate Mongo operations.
|
||||
// For trading: only INSERT today.
|
||||
|
||||
if (query.toUpperCase().startsWith("INSERT")) {
|
||||
return this.handleInsert(query, binds);
|
||||
}
|
||||
if (query.toUpperCase().startsWith("UPDATE")) {
|
||||
return this.handleUpdate(query, binds);
|
||||
}
|
||||
if (query.toUpperCase().startsWith("DELETE")) {
|
||||
return this.handleDelete(query, binds);
|
||||
}
|
||||
|
||||
throw new Error(`Unsupported query: ${query}`);
|
||||
}
|
||||
|
||||
async handleInsert(query, binds) {
|
||||
// Parse: INSERT INTO trading_trades (user_id, symbol, ...) VALUES (?, ?, ...)
|
||||
const match = query.match(
|
||||
/INSERT INTO (\w+)\s*\((.*?)\)\s*VALUES\s*\((.*?)\)/i
|
||||
);
|
||||
if (!match) throw new Error(`Cannot parse INSERT: ${query}`);
|
||||
|
||||
const [, tableName, colStr, placeholderStr] = match;
|
||||
const columns = colStr.split(",").map(c => c.trim());
|
||||
const placeholders = placeholderStr.split(",").map(p => p.trim());
|
||||
|
||||
const doc = {};
|
||||
for (let i = 0; i < columns.length; i++) {
|
||||
doc[columns[i]] = binds[i];
|
||||
}
|
||||
|
||||
const result = await this.db.collection(tableName).insertOne(doc);
|
||||
return { changes: 1, last_row_id: result.insertedId };
|
||||
}
|
||||
|
||||
async all(query, ...binds) {
|
||||
// Parse SELECT; return cursor results as array
|
||||
const match = query.match(/SELECT\s+(.*?)\s+FROM\s+(\w+)(.*)/i);
|
||||
if (!match) throw new Error(`Cannot parse SELECT: ${query}`);
|
||||
|
||||
const [, cols, tableName, rest] = match;
|
||||
const collection = this.db.collection(tableName);
|
||||
|
||||
// Parse WHERE clause if present
|
||||
const mongoQuery = this.parseWhereClause(rest, binds);
|
||||
|
||||
return collection.find(mongoQuery).toArray();
|
||||
}
|
||||
|
||||
async first(query, ...binds) {
|
||||
const results = await this.all(query, ...binds);
|
||||
return results[0] || null;
|
||||
}
|
||||
|
||||
// ... handleUpdate, handleDelete, parseWhereClause (SQL parser — complex, scope this to minimal set)
|
||||
|
||||
async close() {
|
||||
if (this.client) await this.client.close();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** Full SQL parser is out-of-scope here. For trading (the only D1 user), implement minimal support: INSERT, SELECT with user_id + ts WHERE clause. Revisit if more modules use D1.
|
||||
|
||||
---
|
||||
|
||||
## 7. Unresolved Questions
|
||||
|
||||
1. **SQL Parser Scope:** How much of SQL syntax must MongoSqlStore support? Trading uses only `INSERT`, `SELECT WHERE user_id = ? ORDER BY ts DESC LIMIT ?`. If future modules add GROUP BY / aggregate queries, needs expansion or fallback to raw SQL driver.
|
||||
|
||||
2. **Dual-Write Divergence:** What's acceptable divergence rate before automatic rollback? (e.g., >1% mismatch triggers alert?)
|
||||
|
||||
3. **Cold-Start Latency:** MongoDB Atlas M0 first connection: ~50–200 ms (cold start). Workers are ≤50 ms cold start today. Acceptable latency increase?
|
||||
|
||||
4. **Multi-Document Transactions:** Trading is append-only today. If future features require atomicity across KV + D1 (e.g., portfolio update + trade record in single atomic operation), MongoDB replica sets support it, but M0 is standalone. Upgrade path to M10 required?
|
||||
|
||||
5. **Network Egress Costs:** At what monthly traffic does Mongo's egress charges ($0.10/GB) exceed Cloudflare KV's (zero egress)? Current traffic: <1 GB/month. Safe for 2–3 years.
|
||||
|
||||
6. **Point-in-Time Recovery (PITR):** M0 has no backups. Is D1 backup export + manual recovery acceptable, or should we upgrade to Flex tier (includes backups) when M0 is outgrown?
|
||||
|
||||
---
|
||||
|
||||
## 8. Summary & Recommendations
|
||||
|
||||
### Data Model
|
||||
- **KV:** One collection per module. Document shape: `{_id, value (JSON string), expiresAt?}`. TTL index on `expiresAt` with `expireAfterSeconds: 0`.
|
||||
- **SQL:** Append-only `trading_trades` collection. Indexes on `(user_id, ts)` for range queries. No transactions needed today.
|
||||
|
||||
### Migration Path
|
||||
1. **Dual-write wrapper** (Phase 1–2): Write to both CF + Mongo, read from CF.
|
||||
2. **Backfill** (Phase 3): Copy all historical data from CF to Mongo.
|
||||
3. **Verify** (Phase 4): Compare key counts; sample check.
|
||||
4. **Cutover** (Phase 5): Flip single env flag `USE_MONGO=1`; read from Mongo only.
|
||||
5. **Decommission** (Phase 6): Drop CF KV namespace + D1 after 30-day grace period.
|
||||
|
||||
### M0 Ceiling
|
||||
- **Storage:** 512 MB. Current usage ~1 MB/month. Runway: **43 years** at steady state; **2–3 years** if user count grows 10x.
|
||||
- **Connections:** 500 concurrent. Current: 10–50/min. **10x headroom.**
|
||||
- **Ops/sec:** ~100–200 nominal. Current: 5–20 sustained. **5–40x headroom.**
|
||||
- **Regions:** Recommend **aws-ap-southeast-1** (Singapore) for SEA latency. EU users: **aws-eu-west-1** (Ireland).
|
||||
|
||||
### Upgrade Decision
|
||||
If traffic grows to 10x+ and M0 hits limits, upgrade to **Flex Tier** ($8–$30/month, auto-scales, backups). M2/M5 (deprecated) are not an option as of 2026.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [MongoDB Pricing](https://www.mongodb.com/pricing)
|
||||
- [Atlas Free Cluster Limits](https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/)
|
||||
- [Atlas Service Limits](https://www.mongodb.com/docs/atlas/reference/atlas-limits/)
|
||||
- [FAQ: Storage](https://www.mongodb.com/docs/atlas/reference/faq/storage/)
|
||||
- [Expire Data from Collections by Setting TTL](https://www.mongodb.com/docs/manual/tutorial/expire-data/)
|
||||
- [TTL Indexes](https://www.mongodb.com/docs/manual/core/index-ttl/)
|
||||
- [Cloud Providers and Regions](https://www.mongodb.com/docs/atlas/cloud-providers-regions/)
|
||||
- [Amazon Web Services (AWS)](https://www.mongodb.com/docs/atlas/reference/amazon-aws/)
|
||||
- [Transactions](https://www.mongodb.com/docs/manual/core/transactions/)
|
||||
- [$regex (query predicate operator)](https://www.mongodb.com/docs/manual/reference/operator/query/regex/)
|
||||
- [MongoDB Pricing Explained: A 2026 Guide To MongoDB Costs](https://www.cloudzero.com/blog/mongodb-pricing/)
|
||||
|
||||
@@ -1,256 +0,0 @@
|
||||
# Free Database Validation Matrix — miti99bot
|
||||
|
||||
**Date:** 2026-04-25
|
||||
**Scope:** Authoritative comparison of database backends to solve KV quota exhaustion on Cloudflare Workers.
|
||||
**Recommendation:** Upstash Redis (primary) + optional Turso for trading module growth.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
**Pick: Upstash Redis.** Removes 1k/day KV write limit → 500K cmds/mo free (≈16.6k ops/day). Cold-start latency 50–200ms (vs. MongoDB's 1500ms). Direct KV adapter swap, 3–4 hours. **Risk: Low. Migration cost: 4 days.**
|
||||
|
||||
---
|
||||
|
||||
## Decision Matrix
|
||||
|
||||
| **Option** | **Free Tier Ceiling** | **Workers Cold-Start** | **KV Fit** | **SQL Fit** | **Quota Relief** | **Lock-in / Portability** | **Migration (days)** | **Risk Level** |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **Cloudflare KV (Status Quo)** | 100k reads, 1k writes/day | <1ms (instant, cached) | ✅ Native | ❌ None | ❌ Hits limit daily | ✅ CF ecosystem | 0 | Low (proven) |
|
||||
| **Cloudflare D1 (Status Quo)** | 5M row reads, 100k writes/day | 20–100ms | ⚠️ Slow for KV | ✅ Good | ✅ Better for SQL | ✅ CF ecosystem | 0 | Low (proven) |
|
||||
| **Upstash Redis** | 500k cmds/mo, 256MB storage | 50–200ms (HTTP) | ✅ Perfect (Redis strings) | ❌ No SQL | ✅ 10x quota relief | ✅ HTTP, easy exit | 3–4 | **Low** |
|
||||
| **Turso (libSQL)** | 500M reads, 10M writes/mo, 5GB | 100–150ms (HTTP) | ⚠️ SQL overhead for KV | ✅ Excellent | ✅ 100x reads relief | ✅ HTTP, open source | 2–3 (SQL-only) | Low |
|
||||
| **MongoDB Atlas M0** | 512MB storage, 100 ops/sec | **1500ms+ (TLS+SCRAM)** | ⚠️ Document model drift | ⚠️ BSON limits | ✅ Removes ops limit | ⚠️ Medium lock-in | 5–7 | **High** |
|
||||
| **Neon Postgres** | 100 CU-h/mo, 0.5GB storage | 200–400ms (Hyperdrive) | ❌ SQL overhead | ✅ Full PostgreSQL | ⚠️ Weak quota (0.5GB) | ✅ Standard Postgres | 4–5 | Medium |
|
||||
| **Supabase (Postgres)** | ~Not found (Postgres backend) | Similar to Neon | ❌ SQL overhead | ✅ Full PostgreSQL | ⚠️ Weak quota | ✅ Standard Postgres | 4–5 | Medium |
|
||||
| **Fauna** | Fetch timeout (site unreachable) | 300–500ms (proprietary) | ⚠️ Graph model | ✅ Transactional | ✅ Removes ops limit | ❌ Proprietary, high lock-in | 6–8 | **High** |
|
||||
| **PlanetScale (MySQL)** | Free tier removed (2024) | N/A | N/A | N/A | ❌ No longer free | ❌ | N/A | **Deprecated** |
|
||||
| **KV + Durable Objects** | 1k writes + DO requests | 50–200ms (DO overhead) | ✅ Possible (batching) | ❌ None | ✅ 80% write reduction | ✅ CF ecosystem | 5–6 | Medium (complex) |
|
||||
| **KV + Read Cache Layer** | 100k reads, 1k writes | <1ms (read cache) | ✅ KV with warmth | ❌ None | ⚠️ Partial relief (reads OK) | ✅ CF ecosystem | 3–4 | Low |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Scoring (per option)
|
||||
|
||||
### Cloudflare KV (Status Quo)
|
||||
**Fit: ❌ Does not solve the pain.** User hits 1k write limit mid-day during peak game traffic. Reads OK (100k/day >> 5-10k games). **Worst case:** `/wordle` command at 23:59 UTC when quota is exhausted → "quota exceeded" error, bot unavailable for 1+ minute until reset.
|
||||
|
||||
**Why still listed:** Baseline; combined with batching (Durable Objects) or read-cache, partial relief possible but adds operational complexity.
|
||||
|
||||
---
|
||||
|
||||
### Cloudflare D1 (Status Quo)
|
||||
**Fit: ⚠️ Marginal for games; good for SQL.** D1 free tier is 5M reads/100k writes/day — excellent for trading module. But trading module is low-volume (append-only ledger, ~100 writes/day). **For KV games:** Storing game state in relational tables is slow (row → JSON deserialize on every read). **Worst case:** Single `/wordle` command triggers SELECT + prepare + bind + execute on D1 (20–100ms), stalls user experience vs. instant KV.
|
||||
|
||||
**Verdict:** Keep D1 for trading (as-is); don't migrate game state to D1.
|
||||
|
||||
---
|
||||
|
||||
### Upstash Redis ⭐ RECOMMENDED PRIMARY
|
||||
**Fit: ✅ Solves the pain, minimal trade-offs.**
|
||||
|
||||
**Free tier verified (Apr 2026):** 500K commands/month, 256MB storage, 10GB bandwidth/month.
|
||||
|
||||
**Ops/day:** 500k ÷ 30 = 16.6k operations/day. **Current workload estimate:** 5–10k reads + 1k writes/day = under 16.6k/day → stays free. **Headroom:** 2–3x buffer before paid tier.
|
||||
|
||||
**Cold-start:** HTTP-native, no TCP handshake. Typical latency 50–200ms to nearest Upstash PoP. **UX:** `/wordle` command returns in ~500–600ms total (Telegram latency + Worker bootstrap + HTTP fetch + Redis op). Acceptable.
|
||||
|
||||
**KV fit:** Redis strings/hashes directly replace Cloudflare KV. `@upstash/redis` library mirrors `getJSON()`/`putJSON()` semantics. **Per-module prefixing:** Works (use Redis KEYS pattern matching or maintain explicit subscriptions set).
|
||||
|
||||
**Transactions & Expiry:** Redis MULTI/EXEC (ACID per-key). Redis EX / EXAT (TTL). ✅ Sufficient for game turns.
|
||||
|
||||
**Trade-off:** Lose `list()` pagination (KV has it; Upstash SCAN-equivalent less efficient for large datasets). **Mitigation:** Maintain explicit set of active-game keys; scan that set instead.
|
||||
|
||||
**Cost overflow:** 10x traffic → 1.8M ops/month → $1.80/month. Negligible.
|
||||
|
||||
**Implementation:** 3–4 hours (write `upstash-store.js`, swap in `create-store.js`, test).
|
||||
|
||||
---
|
||||
|
||||
### Turso (libSQL) — Secondary Option for SQL
|
||||
**Fit: ✅ Excellent for trading, overkill for games.**
|
||||
|
||||
**Free tier verified (Apr 2026):** 500M reads, 10M writes/month, 5GB storage.
|
||||
|
||||
**Ops/day:** 500M ÷ 30 ≈ 16.7M reads/day; 10M ÷ 30 ≈ 333k writes/day. **Current workload:** ~5–10k reads, ~1k writes/day = negligible quota use → free forever.
|
||||
|
||||
**SQL fit:** Perfect for trading module (append-only ledger, aggregations, leaderboards). **KV fit:** Possible but inefficient — SQLite tables as KV is slower than Redis strings (prepare → bind → execute vs. GET).
|
||||
|
||||
**Cold-start:** HTTP-native (libSQL client), 100–150ms. Acceptable.
|
||||
|
||||
**Recommendations:**
|
||||
1. **Immediate:** Migrate KV to Upstash (games, state).
|
||||
2. **Future (phase 2):** Migrate D1 trading module to Turso if trading grows (complex queries, leaderboards, date-range filters). D1 is fine for now (low volume).
|
||||
|
||||
**Implementation:** 2–3 hours (write `turso-sql-store.js`, migrate trading migrations, test).
|
||||
|
||||
---
|
||||
|
||||
### MongoDB Atlas M0
|
||||
**Fit: ❌ DO NOT CHOOSE. Solves quota pain but introduces worse pain.**
|
||||
|
||||
**Free tier verified (Apr 2026):** 512MB storage, 100 ops/sec throughput, 500 connections, auto-pauses after 30 days.
|
||||
|
||||
**Throughput analysis:** "100 ops/sec" sounds fine (5–20 ops/sec typical for bot), but **throughput ≠ daily quota**. Atlas has **no daily operation limit** (the stated advantage), but Upstash solves the actual pain (daily quota) at lower cost/latency.
|
||||
|
||||
**Cold-start:** CRITICAL FLAW. Workers Sockets API enables native TCP (2025 addition). MongoDB driver handshake:
|
||||
1. TLS negotiation: ~300–500ms
|
||||
2. SCRAM authentication: ~500–800ms
|
||||
3. Server selection: ~200–500ms
|
||||
4. **Total per cold isolate: ~1500ms**
|
||||
|
||||
**UX impact:** `/wordle` command → Worker cold-start → TLS+auth stall → 1.5s+ before bot can respond. Users perceive bot as slow. KV is instant; Upstash is 50–200ms. **Trade-off unjustified.**
|
||||
|
||||
**Connection pooling:** Impossible across stateless Worker invocations. Memoizing a MongoClient per isolate helps warm requests but cold starts still stall.
|
||||
|
||||
**Operational debt:**
|
||||
- Rewrite `src/db/create-store.js` → MongoDB adapter
|
||||
- All modules' data models shift: arrays become collections, TTL becomes indexes
|
||||
- No transactions on M0 (single-document ACID only) — can't atomically increment leaderboards
|
||||
- Unfamiliar failure modes (connection errors, replica-set failovers, BSON limits)
|
||||
|
||||
**Lock-in:** Medium (not proprietary, but requires rewrite to exit; harder than swapping HTTP backends).
|
||||
|
||||
**Verdict:** MongoDB is production-ready and well-documented, but **architecturally misfit for a stateless serverless bot**. Upstash solves all problems without the latency penalty.
|
||||
|
||||
---
|
||||
|
||||
### Neon Postgres
|
||||
**Fit: ⚠️ Overkill. Weak free tier.**
|
||||
|
||||
**Free tier verified (Apr 2026):** 100 CU-hours/month, 0.5GB storage.
|
||||
|
||||
**CU-hours breakdown:** 0.25 CU = 400 hours/month continuous. 100 CU-hours = 400 hours continuous running at 0.25 CU, then scales to zero after 5 min idle. **Real-world:** Trading module generates ~100 writes/day, wordle/loldle generate ~5–10k reads/day. Over 30 days: ~3000 write ops (trivial) + ~150k read ops (trivial). **But storage:** Trading table grows fastest (append-only). If uncapped, could hit 0.5GB in ~1–2 months. **Verdict:** Free tier insufficient; need to upgrade.
|
||||
|
||||
**Cold-start:** Hyperdrive required (connection pooling layer). 200–400ms typical. Better than MongoDB but worse than Upstash.
|
||||
|
||||
**Verdict:** Full PostgreSQL power is unnecessary for this workload. Turso or Upstash are simpler choices.
|
||||
|
||||
---
|
||||
|
||||
### Supabase (Postgres)
|
||||
**Fit: ⚠️ Same as Neon (shared Postgres backend).**
|
||||
|
||||
Free tier not verified (WebFetch failed; assuming parity with Neon or slightly better). PostgreSQL is overkill; Upstash solves the problem faster.
|
||||
|
||||
---
|
||||
|
||||
### Fauna
|
||||
**Fit: ❌ Proprietary document DB; unreachable (site timeout).**
|
||||
|
||||
Fauna is a graph-document database (FQL). No cold-start data available; site unreachable during research. **Skip.**
|
||||
|
||||
---
|
||||
|
||||
### PlanetScale (MySQL)
|
||||
**Status: ❌ DEPRECATED.** PlanetScale removed free tier in 2024. Paid tier starts at ~$80/month. **Not viable.**
|
||||
|
||||
---
|
||||
|
||||
### KV + Durable Objects (Batching)
|
||||
**Fit: ✅ Solves write quota pain, but adds complexity.**
|
||||
|
||||
**Architecture:** Durable Objects act as write coordinator. Each game turn POSTs to DO; DO batches 5–10 turns into a single KV write. **Write reduction:** 1k writes → 100–200 writes/day, stays under limit.
|
||||
|
||||
**Latency impact:** Each game turn incurs 50–200ms DO round-trip (coordination). **UX:** Noticeable slowdown (game turns feel sluggish).
|
||||
|
||||
**Cost:** Durable Objects are paid: $0.15/million requests + $0.15/GB storage. At 5–10k daily requests, ~$2–3/month. **Upstash free tier is cheaper.**
|
||||
|
||||
**Operational risk:** DOs are stateful; risk of stale writes if DO crashes between batch flushes.
|
||||
|
||||
**Verdict:** Solves the problem but **adds latency + cost + operational burden**. Upstash is cleaner.
|
||||
|
||||
---
|
||||
|
||||
### KV + Read-Cache Layer
|
||||
**Fit: ⚠️ Partial relief only.**
|
||||
|
||||
**Architecture:** Use a local memory cache in Worker to cache frequently-read keys (game state, leaderboards). First read → KV (counts quota). Subsequent reads → memory cache (no quota). **Effect:** Reduces *read* pressure, not *write* pressure.
|
||||
|
||||
**Limitation:** Bursty traffic (peak hours) exhausts KV read quota faster than cache helps. Write quota is the real blocker (1k writes/day is the reported pain).
|
||||
|
||||
**Verdict:** Helpful but insufficient. Upstash removes both constraints.
|
||||
|
||||
---
|
||||
|
||||
## Verdict: Is MongoDB Suitable for This Project?
|
||||
|
||||
**Answer: NO.** Three reasons:
|
||||
|
||||
1. **Cold-start latency mismatch** — 1500ms TLS+auth per cold isolate vs. 50–200ms for HTTP backends. Telegram users expect <2s round-trip; MongoDB consumes 75% of that budget before querying. Unacceptable for a game bot.
|
||||
|
||||
2. **Architectural debt** — Rewriting KVStore → MongoDB adapter, refactoring all modules' data models, handling BSON limits, managing unfamiliar failure modes. **5–7 days of effort for no UX gain.**
|
||||
|
||||
3. **No upside** — Upstash solves all stated pain (quota) + solves unstated pain (latency) + costs zero + takes 3–4 hours. MongoDB solves quota but adds latency. **Trade-off is unjustifiable.**
|
||||
|
||||
**MongoDB is a great database** (stable, well-documented, transactional), but it's designed for persistent server processes, not stateless serverless invocations. Cloudflare Workers is the wrong primitive for MongoDB.
|
||||
|
||||
---
|
||||
|
||||
## Hybrid Options
|
||||
|
||||
### Option A: Upstash (KV) + Turso (SQL) — Recommended Future State
|
||||
- **Now:** KV games on Upstash, trading on D1 (low volume, acceptable).
|
||||
- **Phase 2 (if trading grows):** Move trading to Turso (better SQL semantics, cheaper reads).
|
||||
- **Rationale:** Best-of-breed for each workload shape. Upstash for state (fast, Redis-native), Turso for ledger (SQL-native).
|
||||
- **Implementation:** 5–7 hours total (Upstash now, Turso later).
|
||||
|
||||
### Option B: Turso for Everything (SQLite)
|
||||
- **Architecture:** Flatten game state into SQLite tables; use Turso as unified backend.
|
||||
- **Pros:** Single SQL backend; join support; aggregations.
|
||||
- **Cons:** Storing JSON in BLOB columns is slower than Redis strings; game state = read-heavy (SQL overhead hurts).
|
||||
- **Verdict:** Works but sub-optimal UX. Upstash + Turso hybrid is better.
|
||||
|
||||
---
|
||||
|
||||
## Final Recommendation
|
||||
|
||||
### 🥇 PRIMARY: Upstash Redis
|
||||
|
||||
**Deploy: Immediately.** Solves the stated pain (daily quota exhaustion), minimizes latency impact, zero lock-in, low implementation cost.
|
||||
|
||||
**Steps:**
|
||||
1. Write `src/db/upstash-store.js` (wraps `@upstash/redis`, implements `KVStore` interface).
|
||||
2. Update `src/db/create-store.js` to instantiate Upstash.
|
||||
3. Update `wrangler.toml` with `UPSTASH_REDIS_REST_URL` and `UPSTASH_REDIS_REST_TOKEN`.
|
||||
4. Test: `npm test` (fakes unchanged; should pass).
|
||||
5. Deploy: `npm run deploy`.
|
||||
|
||||
**Effort:** 3–4 hours.
|
||||
**Risk:** Low (Redis is simpler than MongoDB; no transactions needed for games).
|
||||
**Cost:** $0 (free tier sufficient for current + 3x traffic).
|
||||
|
||||
### 🥈 SECONDARY (Phase 2): Turso for Trading Module
|
||||
|
||||
**Deploy: After Upstash is live and performing well.** Move `trading` module from D1 to Turso if module grows (leaderboard queries, aggregations, date-range filtering).
|
||||
|
||||
**Why phase 2, not now:** D1 is adequate for current trading volume (~100 writes/day, <1MB storage). Turso's SQL advantage shines with complex queries; simple append-only ledger works fine on D1.
|
||||
|
||||
**Effort:** 2–3 hours (write `turso-sql-store.js`, migrate migrations, test).
|
||||
|
||||
### ❌ REJECT: MongoDB Atlas M0
|
||||
|
||||
Cold-start latency (1500ms) is architectural mismatch for serverless bots. Upstash solves all problems with 1/10th the latency and 1/10th the implementation cost.
|
||||
|
||||
---
|
||||
|
||||
## One Thing That Could Change the Recommendation
|
||||
|
||||
**If cold-start latency were irrelevant (e.g., user is willing to accept 2–3s response times for game commands), MongoDB Atlas M0 becomes defensible.** But that's not the case here — Telegram users expect snappy responses. Upstash is the clear winner.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Upstash KEYS pattern performance at scale** — If bot accumulates 256MB data, will `KEYS wordle:*` scans be slow? (Likely fine; Upstash is optimized for this, but benchmark before phase 2.)
|
||||
|
||||
2. **Module prefixing across Upstash + Turso** — If trading module moves to Turso, does per-module key prefixing conflict with SQL table naming (`trading_trades`)? (No; SQL uses tables, KV uses keys; different namespaces.)
|
||||
|
||||
3. **Real-world Upstash bandwidth under bot load** — Free tier is 10GB/month. Game assets (loldle splash art, trading prices) + JSON payloads — will bandwidth stay under limit? (Current estimate: ~300MB/month; room for 10x growth.)
|
||||
|
||||
4. **Fauna current status (2026)** — Site unreachable during research. If Fauna is operational, worth reconsidering (transactional document DB, serverless-native). Recommend user checks directly if interested.
|
||||
|
||||
---
|
||||
|
||||
**Status:** DONE
|
||||
**Summary:** Recommend Upstash Redis (primary, 3–4 days migration). Solves KV quota exhaustion without cold-start latency penalty. Reject MongoDB (1500ms cold-start, 5–7 days effort, no upside over Upstash).
|
||||
@@ -1,407 +0,0 @@
|
||||
# Test Audit: miti99bot-go Coverage & Quality Report
|
||||
|
||||
**Date:** 2026-05-09
|
||||
**Auditor:** QA Lead
|
||||
**Scope:** Full project test suite (72 Go files, 22 test files, 6360 LOC)
|
||||
**Status:** DONE_WITH_CONCERNS
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
- **Test Execution:** ✅ All tests pass (count=1 to avoid flakes)
|
||||
- **Race Detector:** ✅ No data races detected across concurrent access patterns
|
||||
- **Overall Coverage:** 44.7% (below industry 60-80% target)
|
||||
- **Build Status:** ✅ go vet passes, no linting errors
|
||||
- **Critical Gap:** Handler functions in wordle, loldleemoji, misc, util have 0% coverage — handlers never tested via integration tests
|
||||
|
||||
**High-Risk Packages:** wordle (37.1%), loldleemoji (36.3%), misc (21.1%)
|
||||
|
||||
---
|
||||
|
||||
## Per-Package Coverage Table
|
||||
|
||||
| Package | Coverage | Status | Primary Gap |
|
||||
|---------|----------|--------|-------------|
|
||||
| internal/keylock | 100.0% ✅ | Excellent | None — concurrent access well-tested |
|
||||
| internal/telegram | 100.0% ✅ | Excellent | None — webhook auth tested end-to-end |
|
||||
| internal/modules | 71.6% | Good | Registry/Build; public accessors untested (0%) |
|
||||
| internal/server | 71.4% | Good | Router integration; cron timeout edge cases |
|
||||
| internal/modules/loldle | 53.0% | Below target | Handler functions (handleLoldle, handleGiveup, etc.) untested |
|
||||
| internal/storage | 42.9% | Poor | Firestore ops skip on CI (emulator-only); GetJSON/PutJSON unused |
|
||||
| internal/modules/loldleemoji | 36.3% | Poor | Handler layer untested; state functions incomplete |
|
||||
| internal/modules/wordle | 37.1% | Poor | **All 5 handler functions have 0% coverage** |
|
||||
| internal/modules/util | 39.2% | Poor | Handler layer untested (infoCommand, helpCommand, stickerIDCommand all 0%) |
|
||||
| internal/modules/misc | 21.1% | Poor | Handlers only tested via KV contract, never end-to-end with bot |
|
||||
| cmd/server | 0.0% ❌ | No tests | main() entry point untestable; buildProvider() and config loading untested |
|
||||
|
||||
**Total: 44.7%** (below 60% threshold)
|
||||
|
||||
---
|
||||
|
||||
## Top 5 Highest-Risk Coverage Gaps
|
||||
|
||||
### 1. **Wordle Handler Layer (0% coverage)**
|
||||
**Files:** `internal/modules/wordle/handlers.go`
|
||||
**Functions untested:**
|
||||
- `handleWordle` (121–189): Main /wordle command — guess submission, board display, win/loss logic
|
||||
- `handleNew` (193–221): /wordle_new — round abandonment, auto-giveup stats recording
|
||||
- `handleGiveup` (225–253): /wordle_giveup — reveal answer, idempotency on finished rounds
|
||||
- `handleStats` (256–284): /wordle_stats — win rate calculation (math.Round call), streak display
|
||||
- `subjectFor`, `argAfterCommand`, `rejectMessage`, `reply`: All wrapper helpers untested
|
||||
|
||||
**Why it matters:** Handlers encapsulate game flow logic, context cancellation, KV error propagation, Telegram API replies. A broken `subjectFor` or missing nil-check on `msg.From` would only surface in production.
|
||||
|
||||
**Edge cases not tested:**
|
||||
- `msg == nil` paths (lines 123–124, 194–195, etc.) — guard clauses exist but never executed
|
||||
- Context timeout during KV operations (saveGame, loadGame failures)
|
||||
- Nil map/slice operations (e.g., `msg.Chat.Type` when msg is non-nil but Chat is nil)
|
||||
- Empty chat ID or user ID edge cases
|
||||
- Concurrent access: two simultaneous /wordle guesses on same subject race on keylock (tested in isolation, not in handler context)
|
||||
|
||||
---
|
||||
|
||||
### 2. **Misc Module Handler Handlers (11% coverage in handler functions)**
|
||||
**Files:** `internal/modules/misc/misc.go`
|
||||
**Functions untested as handlers:**
|
||||
- `pingCommand` (lines 42–60): Handler closure — KV write best-effort path, bot.SendMessage error propagation
|
||||
- `mstatsCommand` (lines 62–85): Handler closure — GetJSON missing key, error handling, time formatting
|
||||
- `fortytwoCommand` (lines 87–100): Handler closure — easter egg reply
|
||||
|
||||
**Why it matters:** Misc is the "framework-validating" module; if its handlers fail, the whole bot's command routing is in question.
|
||||
|
||||
**Coverage detail:** Tests verify KV contract (Put/Get round-trip) but **never invoke the actual handler closures** via bot.SendMessage or with real Telegram Update objects.
|
||||
|
||||
---
|
||||
|
||||
### 3. **Util Module Handlers (0% coverage)**
|
||||
**Files:** `internal/modules/util/util.go` + `internal/modules/util/help.go`, `info.go`, `stickerid.go`
|
||||
**Functions untested:**
|
||||
- `infoCommand` (info.go:15): /info handler — never tested
|
||||
- `helpCommand` (help.go:92): /help handler — RenderHelp (100% tested) but handler closure untested
|
||||
- `stickerIDCommand` (stickerid.go:21): /stickerid handler and its `stickerFrom` helper (0% coverage)
|
||||
|
||||
**Why it matters:** /help is critical for user onboarding. A nil registry or missing module would break silently.
|
||||
|
||||
---
|
||||
|
||||
### 4. **Firestore Integration Tests Skipped on CI**
|
||||
**Files:** `internal/storage/firestore_kv_test.go`
|
||||
**Status:** 5 out of 11 Firestore tests skip when `FIRESTORE_EMULATOR_HOST` unset (standard CI environment)
|
||||
|
||||
**Tests skipped:**
|
||||
- `TestFirestoreKV_PutGetRoundTrip`: Basic round-trip (skipped)
|
||||
- `TestFirestoreKV_GetMissingReturnsErrNotFound`: ErrNotFound mapping (skipped)
|
||||
- `TestFirestoreKV_PutGetJSON`: JSON marshal/unmarshal (skipped)
|
||||
- `TestFirestoreKV_DeleteIdempotent`: Delete semantics (skipped)
|
||||
- `TestFirestoreKV_ListByPrefix`: Prefix iteration (skipped)
|
||||
|
||||
**What IS tested on CI:** Only validation (key format, reserved names) — happy path ops untested.
|
||||
|
||||
**Risk:** Any breakage in firestore.Client.Get, Put, Delete, List surfaces only in production. Module-level KV operations (recordResult, loadGame, etc.) invoke these untested paths.
|
||||
|
||||
---
|
||||
|
||||
### 5. **Loldleemoji Handlers (0% coverage on handler layer)**
|
||||
**Files:** `internal/modules/loldleemoji/` (New + handlers for loldleemoji_* commands)
|
||||
**No handlers_test.go exists.** State and render tested in isolation; command dispatch untested.
|
||||
|
||||
**Untestable seams:**
|
||||
- Handler returns `error` but no tests verify error propagation to bot.SendMessage
|
||||
- Concurrent state mutations via keylock not tested in handler context
|
||||
|
||||
---
|
||||
|
||||
## Quality Analysis: Test Patterns & Brittle Areas
|
||||
|
||||
### ✅ Strengths
|
||||
|
||||
1. **Keylock mutex tests are excellent** (keylock_test.go:16–73)
|
||||
- Distinct keys don't block (timing test, 40ms timeout)
|
||||
- Same key serializes correctly (32 goroutines, 100 iterations each, atomic counter)
|
||||
- No flakes observed in race detector runs
|
||||
|
||||
2. **Table-driven tests properly structured** (e.g., validate_test.go, modules/registry_test.go)
|
||||
- Consistent naming: TestXxx_CaseName/CaseName
|
||||
- Subtests enable per-case failure isolation
|
||||
|
||||
3. **Mock-light design:** Most tests use real in-memory KVStore (storage.NewMemoryKVStore())
|
||||
- Avoids mock divergence from prod Firestore
|
||||
- Catches JSON serialization bugs
|
||||
|
||||
4. **Telegram webhook tests** (telegram/webhook_test.go) test auth + parsing
|
||||
- Secret constant-time comparison verified
|
||||
- Oversized body rejected (5 MB limit)
|
||||
- Malformed JSON rejected
|
||||
|
||||
### ⚠️ Concerns
|
||||
|
||||
1. **Handler functions never called in tests**
|
||||
- Handlers take `context.Context`, `*bot.Bot`, `*models.Update` but tests only exercise KV layer
|
||||
- `reply()` helper never invoked; any bot.SendMessage error would be silent in tests
|
||||
- Telegram API reply format never validated in tests (unlike JS version which tests reply text)
|
||||
|
||||
2. **Missing nil checks on traversals**
|
||||
- `msg.Chat.Type` assumes msg.Chat exists but only msg == nil is guarded
|
||||
- `msg.From` checked in subjectFor but could be nil in other paths (argAfterCommand doesn't check)
|
||||
- No test for `msg == nil` in update dispatch
|
||||
|
||||
3. **Storage KV contract untested for operations actually used**
|
||||
- `GetJSON`, `PutJSON`, `Delete` have 0% or near-0% coverage in Firestore impl
|
||||
- MemoryKVStore covers them but divergence possible if JSON marshal logic differs
|
||||
- No concurrent Put+Get race test on same key (only keylock, not KV semantics)
|
||||
|
||||
4. **Context cancellation edge cases**
|
||||
- Handlers have `ctx` but no tests cancel mid-operation
|
||||
- Firestore ops check context but integration tests don't verify timeout handling
|
||||
- /cron/{name} has 6-minute timeout; no tests stress it
|
||||
|
||||
5. **No end-to-end integration test**
|
||||
- No test that spins up full bot + registry + storage + Telegram client
|
||||
- Config loading (splitCSV, envForModules, secretEnvKeys stripping) never tested
|
||||
- main() is untestable as written (flag parsing, signal handling, blocking ListenAndServe)
|
||||
|
||||
6. **Firestore emulator-only on local dev**
|
||||
- CI doesn't run `make test-emulator` (if it exists)
|
||||
- List() operation (storage/firestore_kv.go:170) only tested with prefix validation; actual iteration untested
|
||||
- Delete semantics untested in CI
|
||||
|
||||
---
|
||||
|
||||
## Error Handling & Edge Cases Audit
|
||||
|
||||
### Well-Tested ✅
|
||||
- Module name validation (kebab-case, hyphens, reserved names)
|
||||
- Command name validation (lowercase, 1–32 chars)
|
||||
- Firestore key validation (no `/`, `.`, `..`, `__x__`)
|
||||
- Prefix validation in List
|
||||
- HTTP cron auth (constant-time secret check)
|
||||
- Cron name validation (regex-enforced)
|
||||
|
||||
### Untested or Partial ⚠️
|
||||
- **Empty/nil inputs:** msg == nil tested in guard clauses but not invoked
|
||||
- **JSON errors:** decode failures in GetJSON not tested; Firestore variant untested
|
||||
- **Concurrent mutations:** keylock tested in isolation; concurrent handler invocations on same subject not tested
|
||||
- **Context timeout:** handlers accept ctx but no tests cancel it during KV ops
|
||||
- **KV errors mid-transaction:** startFresh writes game state; if saveGame fails, state is inconsistent—not tested
|
||||
- **Oversized payloads:** JSON encode limit not tested (if target word is huge)
|
||||
- **HTTP chunked encoding:** webhook handler reads body size; chunked/streaming untested
|
||||
- **Signal handling:** graceful shutdown in main.go untestable
|
||||
|
||||
---
|
||||
|
||||
## Race Detector Results
|
||||
|
||||
**Command:** `go test -race ./...` (10 seconds per pkg with race instrumentation)
|
||||
|
||||
**Result:** ✅ **PASS** — No data races detected.
|
||||
|
||||
**Tested concurrent patterns:**
|
||||
- Keylock per-key mutual exclusion (keylock_test.go)
|
||||
- Nil RNG usage in wordle/daily_test.go:TestPickRandom_NilRNGIsRaceFree — safe to use shared rand.Rand without lock
|
||||
|
||||
**NOT stress-tested by race detector:**
|
||||
- Concurrent handler invocations (handlers not tested)
|
||||
- Firestore client concurrent Get/Put (emulator skipped on CI)
|
||||
- In-memory KV concurrent access under module handlers (only in isolation)
|
||||
|
||||
---
|
||||
|
||||
## Performance Observations
|
||||
|
||||
- **Test execution time:** ~0.08s total for all test suites (fast ✅)
|
||||
- **Slowest package:** wordle (0.083s) — mostly from LoadWords embedding validation
|
||||
- **No slow tests:** All tests complete in <0.1s individually
|
||||
|
||||
---
|
||||
|
||||
## Untestable Seams (By Design or Complexity)
|
||||
|
||||
| Seam | Reason | Impact |
|
||||
|------|--------|--------|
|
||||
| `main()` in cmd/server | Entry point; signal handling, HTTP server startup | Cannot test startup sequence, config loading, provider selection |
|
||||
| `telegram.Client` | External Telegram API | Mocked/stubbed; prod connectivity untested |
|
||||
| `firestore.Client` | Requires emulator or GCP creds | Skipped on CI; only validation tested |
|
||||
| `http.Server.ListenAndServe` | Blocking; requires real port | Tested via httptest (router_test.go) instead |
|
||||
| Command/Cron handler closures | Dispatch layer tested but handler bodies not | Handlers never invoked with real Update objects |
|
||||
|
||||
---
|
||||
|
||||
## Test Organization Quality
|
||||
|
||||
**Good:**
|
||||
- Separate *_test.go files per module (loldle_test.go split into compare_test, render_test, state_test)
|
||||
- Helper functions (noopCmd, noopCron, buildRegistry) reduce duplication
|
||||
- Unique collection names in Firestore tests prevent cross-test pollution
|
||||
|
||||
**Could improve:**
|
||||
- No golden files or snapshot tests for render output (render_test.go uses string comparison)
|
||||
- No helpers for common handler test patterns (would reduce untested handler gap)
|
||||
- No test utilities for building Update objects (telegram/webhook_test.go builds them manually)
|
||||
|
||||
---
|
||||
|
||||
## Coverage Gaps: Specific File:Line References
|
||||
|
||||
### `internal/modules/wordle/handlers.go`
|
||||
- **30–47** `subjectFor`: Guard clauses on msg == nil, msg.From == nil never executed
|
||||
- **51–60** `argAfterCommand`: Empty string, no space, space handling — only indirectly tested via state layer
|
||||
- **64–73** `rejectMessage`: Case coverage complete in lookup tests but not in handler context
|
||||
- **77–83** `reply`: Never invoked; if bot.SendMessage returns error, handler would fail silently
|
||||
- **121–189** `handleWordle`: Main flow untested — 0% coverage
|
||||
- **193–221** `handleNew`: New round initiation untested
|
||||
- **225–253** `handleGiveup`: Idempotency, giveup stat recording untested
|
||||
- **256–284** `handleStats`: Win rate calculation (math.Round) never exercised with actual wins/losses
|
||||
|
||||
### `internal/modules/misc/misc.go`
|
||||
- **42–60** `pingCommand` handler closure: Best-effort KV write, bot.SendMessage never tested
|
||||
- **62–85** `mstatsCommand` handler closure: GetJSON error path, time formatting untested
|
||||
- **87–100** `fortytwoCommand` handler closure: Simple but never invoked
|
||||
|
||||
### `internal/modules/util/util.go` & helpers
|
||||
- **12–19** `New`: Factory returns module but handlers never invoked
|
||||
- **help.go:92** `helpCommand` handler: RenderHelp 100% but handler dispatch untested
|
||||
- **info.go:15** `infoCommand` handler: 0% coverage
|
||||
- **stickerid.go:21** `stickerIDCommand` handler: 0% coverage
|
||||
- **stickerid.go:70** `stickerFrom` helper: 0% coverage
|
||||
|
||||
### `internal/modules/loldle/handlers.go` (not listed in reports but inferred)
|
||||
- `handleLoldle`, `handleGiveup`, `handleStats`, `handleSetMax`: Handlers untested
|
||||
- Same pattern as wordle — state/render tested in isolation, handler dispatch missing
|
||||
|
||||
### `internal/modules/loldleemoji/` (similar pattern)
|
||||
- Handlers exist but never tested
|
||||
|
||||
### `internal/storage/firestore_kv.go`
|
||||
- **87–114** `Get`: Only validation tested; actual Get + snap.DataAt untested on CI
|
||||
- **115–126** `GetJSON`: 0% coverage in Firestore; MemoryKVStore covers JSON but divergence possible
|
||||
- **127–150** `Put`: 33% (validation only); actual Put untested
|
||||
- **142–151** `PutJSON`: 0% coverage
|
||||
- **152–169** `Delete`: 0% coverage in Firestore
|
||||
- **170–204** `List`: 11% (validation + prefixSuccessor); actual iteration untested
|
||||
|
||||
### `internal/storage/kv_provider.go`
|
||||
- **24–39** Provider constructors: 0% coverage (factories in main.go select them, untestable)
|
||||
|
||||
### `cmd/server/main.go`
|
||||
- **51–118** `main`: Entry point untestable (signal handling, server startup)
|
||||
- **125–152** `buildProvider`: Config → storage backend selection untested; Firestore vs memory fallback untested
|
||||
- **165–186** `loadConfig`: Environment parsing untested
|
||||
|
||||
### `internal/server/router.go`
|
||||
- **45–51** `New`: Router construction untested directly (tested via handlers but not New itself)
|
||||
|
||||
---
|
||||
|
||||
## Recommendations by ROI
|
||||
|
||||
### Critical (Do First)
|
||||
|
||||
1. **Add wordle handler integration tests** (3–4 hours)
|
||||
- Create wordle/handlers_test.go with bot.Bot mock
|
||||
- Test all 5 handlers: handleWordle, handleNew, handleGiveup, handleStats + subjectFor edge cases
|
||||
- Mock bot.SendMessage to verify reply text (win/loss messages, error cases)
|
||||
- **Impact:** 20–25% coverage gain in wordle; blocks production safety gate
|
||||
|
||||
2. **Add misc handler integration tests** (1–2 hours)
|
||||
- Create misc/handlers_test.go exercising pingCommand, mstatsCommand, fortytwoCommand
|
||||
- Verify KV write side effects + bot reply
|
||||
- **Impact:** 10% coverage gain in misc; validates framework end-to-end
|
||||
|
||||
3. **Add firestore emulator to CI** (2–3 hours)
|
||||
- Docker Compose setup or Cloud Emulator in GitHub Actions
|
||||
- Run full Firestore test suite on every push
|
||||
- **Impact:** 10–15% coverage gain in storage; catches Firestore-specific bugs
|
||||
|
||||
### High (Do Next)
|
||||
|
||||
4. **Add util handler tests** (1–2 hours)
|
||||
- Test infoCommand, helpCommand, stickerIDCommand with mock bot
|
||||
- Verify /help output with various registries
|
||||
- **Impact:** 15% coverage gain in util
|
||||
|
||||
5. **Add loldle/loldleemoji handler tests** (2–3 hours)
|
||||
- Same pattern as wordle handlers
|
||||
- **Impact:** 20% coverage gain in loldleemoji, 10–15% in loldle
|
||||
|
||||
6. **Add nil-safety tests for all handler guards** (1 hour)
|
||||
- Test Update.Message == nil path
|
||||
- Test Message.Chat == nil path
|
||||
- Test Message.From == nil path
|
||||
- **Impact:** Covers edge cases, prevents silent failures
|
||||
|
||||
### Medium (Nice to Have)
|
||||
|
||||
7. **Add context cancellation tests** (2–3 hours)
|
||||
- Handlers accept ctx; test timeout during KV ops
|
||||
- Verify error propagation (no silent drops)
|
||||
- **Impact:** Resilience; currently untested
|
||||
|
||||
8. **Add main() integration test** (2–3 hours)
|
||||
- Separate testable config loading from entry point
|
||||
- Test buildProvider logic, config parsing
|
||||
- **Impact:** Catches startup bugs; currently 0% coverage
|
||||
|
||||
9. **Add performance benchmarks** (1–2 hours)
|
||||
- Benchmark CompareChampions, CompareWords (game-critical paths)
|
||||
- Benchmark keylock contention under high concurrency
|
||||
- **Impact:** Prevent performance regression
|
||||
|
||||
### Refactoring (Enables Testing)
|
||||
|
||||
10. **Extract handler helpers into testable functions** (1 hour)
|
||||
- Handlers are closures over `state`; extract reply/error logic into package functions
|
||||
- Allows testing reply paths without mocking bot
|
||||
- **Impact:** Simplifies handler tests; current pattern requires bot mock
|
||||
|
||||
11. **Create test utilities for Update builders** (1 hour)
|
||||
- Helpers for newPrivateMessage, newGroupMessage, newChannelMessage
|
||||
- Reduces boilerplate in handler tests
|
||||
- **Impact:** Enables test proliferation
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Is cmd/server/main.go intentionally untestable?** Should it be refactored to extract testable config/provider logic, or is it acceptable as-is since deployment validates startup?
|
||||
|
||||
2. **Are Firestore emulator tests run in CI?** The skip message says "CI does not run emulator today" — is there a `make test-emulator` target or separate CI job?
|
||||
|
||||
3. **Should handlers be tested via bot.Bot mock or integration test with fake Telegram?** Current approach tests KV contract; mocking bot is simpler but less realistic.
|
||||
|
||||
4. **Are there performance requirements for handler latency?** No benchmarks present; cloud function cold start may be critical.
|
||||
|
||||
5. **Is context cancellation during KV ops handled gracefully?** Handlers don't check ctx.Done(); is this intentional fire-and-forget, or a gap?
|
||||
|
||||
6. **Should private emoji/loldle handler (/loldle_setmax, easter eggs) be tested?** Currently 0% coverage on private commands.
|
||||
|
||||
---
|
||||
|
||||
## Summary: Coverage by Category
|
||||
|
||||
| Category | Tested | Untested | Gap |
|
||||
|----------|--------|----------|-----|
|
||||
| Unit logic (compare, lookup, state) | ✅ | — | 0% |
|
||||
| KV contract (round-trip, not found) | ✅ (memory) | Firestore ops | 40% |
|
||||
| Validation (names, keys, formats) | ✅ | — | 0% |
|
||||
| **Handler dispatch** | ⚠️ (registry OK) | **Handler bodies** | **100%** |
|
||||
| **Bot replies** | ❌ | **All handler text responses** | **100%** |
|
||||
| HTTP routing | ✅ | — | 0% |
|
||||
| Concurrent access | ✅ (keylock, RNG) | **Handler concurrency** | **50%** |
|
||||
| Error propagation | ✅ (isolation) | **Composite errors** | **50%** |
|
||||
| Firestore integration | ✅ (emulator) | **CI coverage** | **100%** |
|
||||
| Main/bootstrap | ❌ | **Entry point, config** | **100%** |
|
||||
|
||||
---
|
||||
|
||||
## Final Assessment
|
||||
|
||||
**Overall Quality:** Good unit test foundation; weak integration coverage.
|
||||
|
||||
**Biggest Risk:** Handler layer has 0% coverage — a broken game flow (e.g., msg.From nil, savegame failure) would only surface in production. This is the #1 blocker for confidence.
|
||||
|
||||
**Second Risk:** Firestore ops untested on CI; any change to Get/Put logic or connection handling is unvalidated until production.
|
||||
|
||||
**Actionable Path:** Implement wordle, misc, util handler tests (4–5 hours total) → coverage jumps to 55–60%. Add Firestore emulator CI (2–3 hours) → 65–70%. Current test architecture is solid; just needs handler-layer extension.
|
||||
|
||||
**Status:** ✅ DONE_WITH_CONCERNS — All tests pass, no races, but coverage is below acceptable threshold and handler layer is entirely untested.
|
||||
Reference in New Issue
Block a user