Code prerequisites for the Phase 06 cold-start soak gate. The 24-72h soak
itself is operator-run; this commit ships the instrumentation + analysis
tools needed to make the PROCEED-or-PIVOT decision.
Telemetry
- src/util/timing.js: startTiming(cmd) returns {mark, end} that emits a
structured cmd_timing log. takeColdFlag() returns {cold, isolateAgeMs}
using a module-scoped boolean — first request in an isolate is cold,
subsequent are warm. This replaces the originally-planned
isolate_age_ms < 200ms classifier (broken because Mongo cold-connect
itself is ~1500ms; cold requests would always bucket as warm —
code-reviewer #11).
- src/util/request-context.js: setLastCold/getLastCold shared state
bridges fetch-level cold detection into the dispatcher middleware
without a circular import.
- src/index.js: takeColdFlag at the top of fetch() emits a request log
and primes the request context for the dispatcher.
- src/modules/dispatcher.js: bot.use() middleware times every command.
Chosen over per-handler wrapping to preserve the existing identity
assertion in tests (handler === reg.allCommands.get(name).cmd.handler)
— single instrumentation point, no contract change.
Soak tools (operator-run)
- scripts/analyze-soak.js: parses CF Logs export (NDJSON or CSV), filters
cmd_timing events, computes p50/p95/p99 per (cmd, cold/warm). Counts
dual-write secondary failures, mongo connection errors, CPU-time
exceeded events. Writes markdown report.
- scripts/synthetic-burst.js: fires N parallel synthetic Telegram updates
at the deployed Worker URL with cache-busting tokens. Used for the
pre-deploy connection-cap stress test (debugger #2 — 20 parallel cold
requests, abort if Atlas peak > 60% of 500-conn cap).
- package.json: analyze:soak + burst:synthetic scripts wired.
Tests
- tests/util/timing.test.js: 8 tests — timing semantics, cold flag flip.
- tests/scripts/analyze-soak.test.js: 22 tests — percentile math, NDJSON
+ CSV parse, aggregation, markdown formatting.
Tests: 667 → 697 (+30). Lint clean.
Operator runbook for Phase 06 (NOT executed by this commit):
1. Verify telemetry live via wrangler tail.
2. Run synthetic burst test: npm run burst:synthetic -- --url <prod>
3. Configure Atlas + CF Observability email alerts.
4. 24h soak (extend to 72h on stop-conditions per phase plan).
5. Daily npm run verify:mongo.
6. npm run analyze:soak -- --input <cf-logs.json> → soak-decision.md.
7. PROCEED to Phase 07 if cold-start P95 ≤ 2.5 × BASELINE_COLD_PING_MS;
else execute phase-07-alt-pivot.md (Upstash standby).
Implements the KVStore interface against MongoDB Atlas with full behavioral
parity vs CFKVStore (null-on-missing, swallow-corrupt-JSON, idempotent delete,
throw-on-undefined-putJSON). Not wired into the request path yet — Phase 04
adds dual-write wrappers and factory routing.
- src/db/mongo-client.js: memoized MongoClient + getDb(env). On connect()
reject, nulls both client and connectPromise so next call retries cleanly
(regression-tested). Catches MongoServerSelectionError and emits a
structured warning before rethrow so callers can map to 503.
- src/db/mongo-kv-store.js: KVStore impl. get/getJSON filter on expiresAt
at read time to close the up-to-60s TTL-sweeper stale-read window vs
CFKVStore. list() returns keys WITH prefix preserved (parity — wrapper
in create-store.js:65 strips). Cursor pagination via sorted _id +
limit(N+1), NOT skip(). Lazy ensureIndex per (collection, isolate)
tracked in module-scope Set.
- src/db/mongo-list-cursor.js: extracted cursor encode/decode to keep
mongo-kv-store.js under 200 LOC.
- tests/fakes/fake-mongo.js: Map-backed fake covering the surface needed
by both Phase 02 (KVStore) and Phase 03 (MongoTradesStore).
- tests/db/mongo-kv-store.test.js: 26 tests, including TTL stale-read
regression (1s TTL + time advance), 2-level prefix list regression,
cursor pagination, connect-reject retry, MongoServerSelectionError
structured log.
Tests: 503 → 529 (+26). Lint clean.
Previously seeds carried hand-curated {category, target, initialHint}.
Now SEEDS is a flat string[] of keywords — at round-start, the model
generates {category, initialHint} on the fly. Benefits:
- adding a seed is trivial (just append a word)
- every round gets a fresh cryptic opener (varies across plays of the
same word)
- HINT STYLE rules apply to the opening hint too, so the initial clue
isn't a definitional giveaway
Implementation:
- prompts.buildStartRoundPrompt(target) — with good/bad examples
- ai-client.generateRoundStart(env, target) — same JSON-in-content
approach as judge(), with defensive fallbacks + redactSecret
- handlers.startFreshGame now async; surfaces roundstart errors via the
existing UPSTREAM_FAIL path
Tests: 449 pass (5 new for generateRoundStart, 1 for roundstart error path).
Gemma 4 likely rejects the flat "traditional" tools schema we were sending
(the docs use OpenAI-wrapped shape for this model) — causing env.AI.run to
throw and users to see the "AI service hiccup" reply every turn.
Switch to the universal approach:
- system prompt asks the model for a one-line JSON {is_guess, answer, hint}
- ai-client.extractText handles both Workers-AI and OpenAI response shapes
- parseJudgementJson walks brace-depth to extract JSON from stray prose /
accidental code fences
- logs twentyq_ai_throw / twentyq_ai_unparseable with preview on failure
so future issues surface in wrangler tail immediately
Tests: 7 new (parser + extractText); 444 total pass.
- SqlStore interface + CF D1 wrapper + per-module factory (table prefix convention)
- init signature extended to ({ db, sql, env }); sql is null when DB binding absent
- custom migration runner walks src/modules/*/migrations/*.sql, tracks applied in _migrations table
- npm run db:migrate with --dry-run and --local flags; chained into deploy
- fake-d1 test helper with subset of SQL semantics for retention and history tests
grammY-based bot with a module plugin system loaded from the MODULES env
var. Three command visibility levels (public/protected/private) share a
unified command namespace with conflict detection at registry build.
- 4 initial modules (util, wordle, loldle, misc); util fully implemented,
others are stubs proving the plugin system end-to-end
- util: /info (chat/thread/sender ids) + /help (pure renderer over the
registry, HTML parse mode, escapes user-influenced strings)
- KVStore interface with CFKVStore and a per-module prefixing factory;
getJSON/putJSON convenience helpers; other backends drop in via one file
- Webhook at POST /webhook with secret-token validation via grammY's
webhookCallback; no admin HTTP surface
- Post-deploy register script (npm run deploy = wrangler deploy && node
--env-file=.env.deploy scripts/register.js) for setWebhook and
setMyCommands; --dry-run flag for preview
- 56 vitest unit tests across 7 suites covering registry, db wrapper,
dispatcher, help renderer, validators, and HTML escaper
- biome for lint + format; phased implementation plan under plans/
tiennm99