feat: notify admin via Telegram on webhook errors

Send error details to ADMIN_CHAT_ID (optional secret) whenever the
statuspage webhook handler hits an error path. Uses waitUntil() so
the notification never blocks the 200 response.
This commit is contained in:
2026-04-13 22:50:27 +07:00
parent 3857c1e16f
commit 62bf203c22
6 changed files with 69 additions and 10 deletions

View File

@@ -22,6 +22,7 @@ No linter configured.
- `BOT_TOKEN` — Telegram bot token - `BOT_TOKEN` — Telegram bot token
- `WEBHOOK_SECRET` — Secret token in Statuspage webhook URL path - `WEBHOOK_SECRET` — Secret token in Statuspage webhook URL path
- `ADMIN_CHAT_ID` — Telegram chat ID to receive webhook error notifications (optional)
## Architecture ## Architecture
@@ -39,7 +40,7 @@ Cloudflare Workers with two entry points exported from `src/index.js`:
### Data Flow ### Data Flow
1. **Statuspage → Worker**: Webhook POST → verify URL secret (timing-safe via `crypto-utils.js`) → parse incident/component event → filter subscribers by type + component → `sendBatch` to CF Queue 1. **Statuspage → Worker**: Webhook POST → verify URL secret (timing-safe via `crypto-utils.js`) → parse incident/component event → filter subscribers by type + component → `sendBatch` to CF Queue. On any error, admin is notified via Telegram (`admin-notifier.js`, non-blocking via `waitUntil`)
2. **Queue → Telegram**: Consumer processes batches of 30 → `sendMessage` via `telegram-api.js` helper → auto-removes blocked subscribers (403/400), retries on 429 2. **Queue → Telegram**: Consumer processes batches of 30 → `sendMessage` via `telegram-api.js` helper → auto-removes blocked subscribers (403/400), retries on 429
3. **User → Bot**: Telegram webhook → grammY handles `/help`, `/start`, `/stop`, `/status`, `/subscribe`, `/history`, `/uptime` commands → reads/writes KV 3. **User → Bot**: Telegram webhook → grammY handles `/help`, `/start`, `/stop`, `/status`, `/subscribe`, `/history`, `/uptime` commands → reads/writes KV

View File

@@ -55,6 +55,10 @@ npx wrangler secret put BOT_TOKEN
npx wrangler secret put WEBHOOK_SECRET npx wrangler secret put WEBHOOK_SECRET
# Choose a random secret string for the Statuspage webhook URL # Choose a random secret string for the Statuspage webhook URL
npx wrangler secret put ADMIN_CHAT_ID
# (Optional) Your Telegram chat ID — receive webhook error alerts via Telegram
# Use the bot's /info command to find your chat ID
``` ```
### 5. Deploy ### 5. Deploy

View File

@@ -63,6 +63,7 @@ A middleware in `index.js` normalizes double slashes in URL paths (Statuspage oc
| `kv-store.js` | ~140 | KV key management, subscriber CRUD, filtered listing | | `kv-store.js` | ~140 | KV key management, subscriber CRUD, filtered listing |
| `status-fetcher.js` | ~120 | Statuspage API client, HTML formatters, status helpers | | `status-fetcher.js` | ~120 | Statuspage API client, HTML formatters, status helpers |
| `crypto-utils.js` | ~12 | Timing-safe string comparison (SHA-256 hashed) | | `crypto-utils.js` | ~12 | Timing-safe string comparison (SHA-256 hashed) |
| `admin-notifier.js` | ~25 | Sends webhook error alerts to admin via Telegram (non-blocking) |
| `telegram-api.js` | ~9 | Telegram Bot API URL builder | | `telegram-api.js` | ~9 | Telegram Bot API URL builder |
## KV Storage ## KV Storage
@@ -103,6 +104,7 @@ Binding: `claude-status` queue
Enabled via `wrangler.jsonc` `observability` config. Automatic — no code changes required. Enabled via `wrangler.jsonc` `observability` config. Automatic — no code changes required.
- **Logs**: All `console.log`/`console.error` calls, request metadata, exceptions. Persisted with invocation logs enabled. Free tier: 200k logs/day, 3-day retention. - **Logs**: All `console.log`/`console.error` calls, request metadata, exceptions. Persisted with invocation logs enabled. Free tier: 200k logs/day, 3-day retention.
- **Admin alerts**: Webhook errors are forwarded to admin Telegram chat in real-time (requires `ADMIN_CHAT_ID` secret)
- **Traces**: Automatic instrumentation of fetch calls, KV reads, queue operations. Persisted. - **Traces**: Automatic instrumentation of fetch calls, KV reads, queue operations. Persisted.
- **Sampling**: 100% (`head_sampling_rate: 1`) for both logs and traces — reduce for high-volume scenarios - **Sampling**: 100% (`head_sampling_rate: 1`) for both logs and traces — reduce for high-volume scenarios
- **Dashboard**: Cloudflare Dashboard → Workers → Observability - **Dashboard**: Cloudflare Dashboard → Workers → Observability
@@ -110,9 +112,10 @@ Enabled via `wrangler.jsonc` `observability` config. Automatic — no code chang
## Security ## Security
- **Statuspage webhook always-200**: Handler always returns HTTP 200 (even on errors) to prevent Statuspage from removing the webhook subscription. Errors are logged, not surfaced as HTTP status codes. - **Statuspage webhook always-200**: Handler always returns HTTP 200 (even on errors) to prevent Statuspage from removing the webhook subscription. Errors are logged, not surfaced as HTTP status codes.
- **Admin error alerts**: All webhook errors send a Telegram notification to `ADMIN_CHAT_ID` (if set) via `admin-notifier.js`. Uses `waitUntil()` to avoid blocking the response. Silently fails if `ADMIN_CHAT_ID` is not configured.
- **Statuspage webhook auth**: URL path secret validated with timing-safe SHA-256 comparison - **Statuspage webhook auth**: URL path secret validated with timing-safe SHA-256 comparison
- **Telegram webhook**: Registered via `setup-bot.js` — Telegram only sends to the registered URL - **Telegram webhook**: Registered via `setup-bot.js` — Telegram only sends to the registered URL
- **No secrets in code**: `BOT_TOKEN` and `WEBHOOK_SECRET` stored as Cloudflare secrets - **No secrets in code**: `BOT_TOKEN`, `WEBHOOK_SECRET`, and `ADMIN_CHAT_ID` stored as Cloudflare secrets
- **HTML injection**: All user/external strings passed through `escapeHtml()` before Telegram HTML rendering - **HTML injection**: All user/external strings passed through `escapeHtml()` before Telegram HTML rendering
## Dependencies ## Dependencies

32
src/admin-notifier.js Normal file
View File

@@ -0,0 +1,32 @@
import { telegramUrl } from "./telegram-api.js";
/**
* Send an error notification to the admin via Telegram.
* Silently fails — admin alerts must never break the main flow.
* @param {object} env - Worker env bindings
* @param {string} reason - Short error reason
* @param {Record<string, unknown>} [details] - Extra context to include
*/
export async function notifyAdmin(env, reason, details = {}) {
if (!env.ADMIN_CHAT_ID || !env.BOT_TOKEN) return;
const text = [
`<b>[Webhook Error]</b> ${reason}`,
`<pre>${JSON.stringify(details, null, 2)}</pre>`,
].join("\n");
try {
await fetch(telegramUrl(env.BOT_TOKEN, "sendMessage"), {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
chat_id: env.ADMIN_CHAT_ID,
text,
parse_mode: "HTML",
disable_web_page_preview: true,
}),
});
} catch {
// Swallow — never let admin notification break the webhook
}
}

View File

@@ -1,6 +1,7 @@
import { getSubscribersByType } from "./kv-store.js"; import { getSubscribersByType } from "./kv-store.js";
import { humanizeStatus, escapeHtml } from "./status-fetcher.js"; import { humanizeStatus, escapeHtml } from "./status-fetcher.js";
import { timingSafeEqual } from "./crypto-utils.js"; import { timingSafeEqual } from "./crypto-utils.js";
import { notifyAdmin } from "./admin-notifier.js";
/** /**
* Format incident event as Telegram HTML message * Format incident event as Telegram HTML message
@@ -42,13 +43,17 @@ export async function handleStatuspageWebhook(c) {
// Validate URL secret (timing-safe) // Validate URL secret (timing-safe)
// Guard against misconfigured deploy (undefined env var) // Guard against misconfigured deploy (undefined env var)
if (!c.env.WEBHOOK_SECRET) { if (!c.env.WEBHOOK_SECRET) {
console.error(JSON.stringify({ event: "webhook_error", reason: "WEBHOOK_SECRET not configured" })); const reason = "WEBHOOK_SECRET not configured";
console.error(JSON.stringify({ event: "webhook_error", reason }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason));
return c.text("OK", 200); return c.text("OK", 200);
} }
const secret = c.req.param("secret"); const secret = c.req.param("secret");
if (!await timingSafeEqual(secret, c.env.WEBHOOK_SECRET)) { if (!await timingSafeEqual(secret, c.env.WEBHOOK_SECRET)) {
console.error(JSON.stringify({ event: "webhook_error", reason: "invalid_secret" })); const reason = "invalid_secret";
console.error(JSON.stringify({ event: "webhook_error", reason }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason));
return c.text("OK", 200); return c.text("OK", 200);
} }
@@ -57,13 +62,18 @@ export async function handleStatuspageWebhook(c) {
try { try {
body = await c.req.json(); body = await c.req.json();
} catch { } catch {
console.error(JSON.stringify({ event: "webhook_error", reason: "invalid_json" })); const reason = "invalid_json";
console.error(JSON.stringify({ event: "webhook_error", reason }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason));
return c.text("OK", 200); return c.text("OK", 200);
} }
const eventType = body?.meta?.event_type; const eventType = body?.meta?.event_type;
if (!eventType) { if (!eventType) {
console.error(JSON.stringify({ event: "webhook_error", reason: "missing_event_type" })); const reason = "missing_event_type";
const details = { keys: Object.keys(body || {}), meta: body?.meta ?? null };
console.error(JSON.stringify({ level: "error", event: "webhook_error", reason, ...details }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason, details));
return c.text("OK", 200); return c.text("OK", 200);
} }
@@ -73,21 +83,27 @@ export async function handleStatuspageWebhook(c) {
let category, html, componentName; let category, html, componentName;
if (eventType.startsWith("incident.")) { if (eventType.startsWith("incident.")) {
if (!body.incident) { if (!body.incident) {
console.error(JSON.stringify({ event: "webhook_error", reason: "missing_incident_data", eventType })); const reason = "missing_incident_data";
console.error(JSON.stringify({ event: "webhook_error", reason, eventType }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason, { eventType }));
return c.text("OK", 200); return c.text("OK", 200);
} }
category = "incident"; category = "incident";
html = formatIncidentMessage(body.incident); html = formatIncidentMessage(body.incident);
} else if (eventType.startsWith("component.")) { } else if (eventType.startsWith("component.")) {
if (!body.component) { if (!body.component) {
console.error(JSON.stringify({ event: "webhook_error", reason: "missing_component_data", eventType })); const reason = "missing_component_data";
console.error(JSON.stringify({ event: "webhook_error", reason, eventType }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason, { eventType }));
return c.text("OK", 200); return c.text("OK", 200);
} }
category = "component"; category = "component";
componentName = body.component.name || null; componentName = body.component.name || null;
html = formatComponentMessage(body.component, body.component_update); html = formatComponentMessage(body.component, body.component_update);
} else { } else {
console.error(JSON.stringify({ event: "webhook_error", reason: "unknown_event_type", eventType })); const reason = "unknown_event_type";
console.error(JSON.stringify({ event: "webhook_error", reason, eventType }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason, { eventType }));
return c.text("OK", 200); return c.text("OK", 200);
} }
@@ -106,7 +122,9 @@ export async function handleStatuspageWebhook(c) {
return c.text("OK", 200); return c.text("OK", 200);
} catch (err) { } catch (err) {
// Catch-all: log error but still return 200 to prevent Statuspage from removing us // Catch-all: log error but still return 200 to prevent Statuspage from removing us
console.error(JSON.stringify({ event: "webhook_error", reason: "unexpected", error: err.message })); const reason = "unexpected";
console.error(JSON.stringify({ event: "webhook_error", reason, error: err.message }));
c.executionCtx.waitUntil(notifyAdmin(c.env, reason, { error: err.message }));
return c.text("OK", 200); return c.text("OK", 200);
} }
} }

View File

@@ -41,4 +41,5 @@
// Secrets (set via `wrangler secret put`): // Secrets (set via `wrangler secret put`):
// BOT_TOKEN - Telegram bot token // BOT_TOKEN - Telegram bot token
// WEBHOOK_SECRET - Statuspage webhook URL secret // WEBHOOK_SECRET - Statuspage webhook URL secret
// ADMIN_CHAT_ID - Telegram chat ID to receive error notifications
} }