Files
Duy /zuey/andGitHub ddf8e1099f feat(webhooks): HTTP webhooks to trigger agents with HMAC auth + durable callbacks (#2)
* feat(webhooks): HTTP webhooks to trigger agents with HMAC auth and durable callbacks

Add multi-tenant HTTP webhook endpoints for agent triggering:
- /v1/webhooks/message: send messages to channels
- /v1/webhooks/llm: sync/async LLM prompts with HMAC-signed callbacks
- HMAC-256 + bearer token authentication
- Rate limiting and tenant isolation
- Durable callback worker with exponential backoff
- PG 000056 + SQLite schema v25 migrations
- Unit + integration tests, P0 tenant isolation invariants
- Channel media capability helpers for attachment routing
- Comprehensive webhook documentation and i18n strings

* fix(webhooks): address post-review findings (K1-K10)

Comprehensive post-merge fixes addressing 10 blocking code review issues
and 2 adversarial re-audit findings in webhook-agent-triggering feature:

K1: Fix auth middleware tenant context lookup sequencing — move
    tenant context injection before authenticate() call to prevent
    unscoped secret lookups.

K2: Canonicalize JSON payload format for jsonb compatibility across
    PostgreSQL and SQLite — ensure consistent serialization without
    whitespace variance to prevent hash mismatches.

K3: Add fail-closed JSON parsing in body hash extraction with explicit
    error handling for malformed payloads before HMAC verification.

K4: Fix worker queue wedge by properly draining slot reservations
    when delivery succeeds, preventing permanent slot occupancy.

K5: Implement lease-token optimistic concurrency control to prevent
    duplicate webhook delivery under high concurrency or retry storms.

K6: Add AES-256-GCM encrypted secret storage at rest with fail-fast
    skip-mount when GOCLAW_ENCRYPTION_KEY environment variable unset.

K7: Implement IP allowlist enforcement supporting both CIDR ranges
    and exact IP matching with proper X-Forwarded-For parsing.

K8: Add HMAC replay nonce cache (5min expiry, non-blocking async flush)
    to prevent request replay attacks on webhook handler.

K9: Fix invariant test schema selection — replace hardcoded assumption
    with explicit schema name from config to support multi-schema testing.

K10: Consolidate rate limiters into single shared instance to prevent
     per-endpoint limiter starvation and ensure fair rate limiting.

New database migrations:
- 000057: webhook_calls.lease_token for optimistic concurrency
- 000058: webhooks.encrypted_secret_key for AES-256-GCM encryption

New i18n keys: MsgWebhookIPDenied, MsgWebhookEncryptionUnavailable
(with English, Vietnamese, Chinese translations).

New modules:
- internal/http/webhooks_payload.go: JSON canonicalization + body hash
- internal/http/webhooks_nonce.go: Replay nonce cache implementation
- internal/http/webhooks_idempotency_test.go: Integration tests

Documentation updates:
- docs/webhooks.md: §13-14 security sections, encryption flow
- docs/00-architecture-overview.md: webhook subsystem security overview
- docs/codebase-summary.md: webhook security patterns
- docs/project-changelog.md: webhook fixes changelog

Test coverage: 53 webhook tests + 4 P0 invariant tests all passing.
No tenant isolation violations. All security gates enforced.

* docs(journals): webhook feature ship + fix cycle entries

* fix(webhooks): address Claude review findings

- webhooks_llm.go: remove misleading ptr() helper; use &completedAt
  pattern for error-path audit rows (matches success path)
- webhooks_auth.go: wrap TouchLastUsed context in WithoutCancel so
  background DB update isn't cancelled when HTTP response completes
- store GetByIDUnscoped (PG+SQLite): add NOT revoked / revoked = 0
  filter for defense-in-depth parity with GetByHashUnscoped
- webhooks/sign.go: fix package doc — HMAC key is raw plaintext
  secret bytes, not hex-decoded SHA-256
- webhooks_admin.go: check auth before encKey guard to avoid leaking
  config state to unauthenticated callers
- webhooks_ratelimit.go: two-phase Load→LoadOrStore to avoid per-call
  entry allocation on the hot path

* docs(webhooks): fix Sign() function doc to match actual key input

Function-level comment still referenced hex-decoded SecretHash after
the package-level doc was corrected. Align with actual caller usage
([]byte(rawSecret)).

* fix(webhooks): use WithoutCancel for worker execute DB updates

Terminal status writes in execute() ran through the worker main-loop
ctx, which is cancelled on graceful shutdown. If the outbound send
completed but the status update raced with shutdown, the row stayed
in 'running' and got re-delivered via reclaimStale. WithoutCancel
lets the DB write survive worker cancellation while preserving
propagated values (tenant ID, etc.).

* fix(webhooks): move tctx init before panic defer in worker execute

Panic recovery called updateRetry with raw ctx (no tenant ID), making
requireTenantID fail and the reset-to-retry DB write silently drop.
Row stayed 'running' until reclaimStale (~90s delay). Init tctx first
so defer closure captures tenant-scoped non-cancellable context.

* fix(webhooks): pass tenant-scoped tctx to invokeAgent in worker

execute() was passing the raw worker-loop ctx (no tenant ID) to
invokeAgent → router.Get → PGAgentStore.GetByID. GetByID reads
TenantIDFromContext which returned uuid.Nil, making every lookup
return 'agent not found'. Async LLM webhook calls silently failed
all retries. Pass tctx (already tenant-scoped + WithoutCancel) so
the router resolves the agent correctly.

* fix(tests): resolve integration test compile errors

- Remove duplicate contains() in mcp_grant_revoke_test.go (already
  defined in tts_gemini_live_test.go)
- Update webhooks_admin_test.go RotateSecret call to match current
  5-arg signature (newSecretHash, newPrefix, newEncryptedSecret)

* fix(webhooks): default nil scopes/ip_allowlist to empty slice in Create

PG columns are NOT NULL DEFAULT '{}'. Explicit NULL from pqStringArray(nil)
violated the constraint, breaking TestWebhookAdminCRUD/TenantIsolation.
Coerce nil slices to empty []string{} so the default applies at the DB layer.

* chore: trigger CI on digitopvn/goclaw fork

* ci: retrigger workflows

* fix(webhooks): renumber migrations to 000059-000061 for merge train
2026-05-11 13:29:24 +07:00

174 lines
9.2 KiB
Go

package store
import (
"context"
"errors"
"time"
"github.com/google/uuid"
)
// ErrIdempotencyConflict is returned when a webhook_call with the same
// (webhook_id, idempotency_key) already exists (partial unique index violation).
var ErrIdempotencyConflict = errors.New("idempotency key conflict: call already exists")
// ErrLeaseExpired is returned by UpdateStatusCAS when 0 rows were affected,
// meaning the row's lease_token no longer matches — it was reclaimed by reclaimStale
// and possibly re-claimed by another worker iteration. The caller should log and drop.
var ErrLeaseExpired = errors.New("webhook call lease expired: row reclaimed by stale sweeper")
// WebhookData represents a registered webhook.
// SecretHash is never serialized to JSON (auth token, server-side only).
// EncryptedSecret holds crypto.Encrypt(raw_secret, encKey) — decrypted at HMAC sign time.
// Existing webhooks with EncryptedSecret="" require rotation before HMAC auth is accepted.
type WebhookData struct {
ID uuid.UUID `json:"id" db:"id"`
TenantID uuid.UUID `json:"tenant_id" db:"tenant_id"`
AgentID *uuid.UUID `json:"agent_id,omitempty" db:"agent_id"`
Name string `json:"name" db:"name"`
Kind string `json:"kind" db:"kind"` // "llm" | "message"
SecretPrefix string `json:"secret_prefix" db:"secret_prefix"`
SecretHash string `json:"-" db:"secret_hash"` // SHA-256 hex; bearer-token lookup only; never serialized
EncryptedSecret string `json:"-" db:"encrypted_secret"` // AES-256-GCM of raw secret; never serialized
Scopes []string `json:"scopes" db:"scopes"`
ChannelID *uuid.UUID `json:"channel_id,omitempty" db:"channel_id"`
RateLimitPerMin int `json:"rate_limit_per_min" db:"rate_limit_per_min"`
IPAllowlist []string `json:"ip_allowlist" db:"ip_allowlist"`
RequireHMAC bool `json:"require_hmac" db:"require_hmac"`
LocalhostOnly bool `json:"localhost_only" db:"localhost_only"`
Revoked bool `json:"revoked" db:"revoked"`
CreatedBy string `json:"created_by" db:"created_by"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
LastUsedAt *time.Time `json:"last_used_at,omitempty" db:"last_used_at"`
}
// WebhookCallData represents a single webhook invocation (queued, in-flight, or terminal).
// DeliveryID is stable across retries — used as X-Webhook-Delivery-Id header.
// StartedAt is set on ClaimNext to detect stale-running calls.
// Attempts is incremented post-send by the worker (NOT on ClaimNext).
// LeaseToken is a random UUID set atomically by ClaimNext; UpdateStatus CAS guards with AND lease_token = $N.
// If CAS hits 0 rows, the row was reclaimed by reclaimStale — the worker logs and drops the update.
type WebhookCallData struct {
ID uuid.UUID `json:"id" db:"id"`
TenantID uuid.UUID `json:"tenant_id" db:"tenant_id"`
WebhookID uuid.UUID `json:"webhook_id" db:"webhook_id"`
AgentID *uuid.UUID `json:"agent_id,omitempty" db:"agent_id"`
DeliveryID uuid.UUID `json:"delivery_id" db:"delivery_id"` // stable across retries
IdempotencyKey *string `json:"idempotency_key,omitempty" db:"idempotency_key"`
Mode string `json:"mode" db:"mode"` // "sync" | "async"
Status string `json:"status" db:"status"` // "queued"|"running"|"done"|"failed"|"dead"
CallbackURL *string `json:"callback_url,omitempty" db:"callback_url"`
Attempts int `json:"attempts" db:"attempts"`
NextAttemptAt *time.Time `json:"next_attempt_at,omitempty" db:"next_attempt_at"`
StartedAt *time.Time `json:"started_at,omitempty" db:"started_at"` // set on ClaimNext
LeaseToken *string `json:"lease_token,omitempty" db:"lease_token"` // CAS guard; set by ClaimNext, cleared by ReclaimStale
RequestPayload []byte `json:"request_payload,omitempty" db:"request_payload"`
Response []byte `json:"response,omitempty" db:"response"`
LastError *string `json:"last_error,omitempty" db:"last_error"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
CompletedAt *time.Time `json:"completed_at,omitempty" db:"completed_at"`
}
// WebhookListFilter controls filtering for WebhookStore.List.
type WebhookListFilter struct {
AgentID *uuid.UUID // filter by bound agent (nil = all)
Limit int // 0 = default (50)
Offset int
}
// WebhookCallListFilter controls filtering for WebhookCallStore.List.
type WebhookCallListFilter struct {
WebhookID *uuid.UUID // filter by parent webhook (nil = all in tenant)
Status string // "" = all statuses
Limit int // 0 = default (50)
Offset int
}
// WebhookStore manages webhook registry entries.
// All methods are tenant-scoped via context (store.TenantIDFromContext).
type WebhookStore interface {
// Create inserts a new webhook. ID + CreatedAt + UpdatedAt should be
// pre-filled by the caller.
Create(ctx context.Context, w *WebhookData) error
// GetByID returns a webhook by its UUID.
// Returns sql.ErrNoRows if not found or tenant mismatch.
GetByID(ctx context.Context, id uuid.UUID) (*WebhookData, error)
// GetByHash returns an active (non-revoked) webhook by its secret_hash.
// Returns sql.ErrNoRows if not found.
GetByHash(ctx context.Context, secretHash string) (*WebhookData, error)
// GetByHashUnscoped looks up a webhook by secret_hash WITHOUT requiring tenant
// in context. Used exclusively in WebhookAuthMiddleware for pre-auth resolution;
// downstream queries remain tenant-scoped after WithTenantID injection.
// security_hash is globally unique (uq_webhooks_secret) so no tenant filter needed.
GetByHashUnscoped(ctx context.Context, secretHash string) (*WebhookData, error)
// GetByIDUnscoped looks up a webhook by UUID WITHOUT requiring tenant in context.
// Used exclusively in WebhookAuthMiddleware for HMAC pre-auth resolution.
GetByIDUnscoped(ctx context.Context, id uuid.UUID) (*WebhookData, error)
// List returns webhooks for the context tenant, with optional agent filter.
List(ctx context.Context, f WebhookListFilter) ([]WebhookData, error)
// Update applies a partial update via column→value map.
// Caller validates keys; store validates against allowlist.
Update(ctx context.Context, id uuid.UUID, updates map[string]any) error
// RotateSecret replaces the secret_hash, secret_prefix, and encrypted_secret.
// Callers (webhooks_admin.go) generate hash + prefix + encrypted form above the store layer.
RotateSecret(ctx context.Context, id uuid.UUID, newSecretHash, newPrefix, newEncryptedSecret string) error
// Revoke marks a webhook as revoked. Returns sql.ErrNoRows if not found.
Revoke(ctx context.Context, id uuid.UUID) error
// TouchLastUsed updates last_used_at. Best-effort — failures are not fatal.
TouchLastUsed(ctx context.Context, id uuid.UUID) error
}
// WebhookCallStore manages webhook call state (queued → running → terminal).
// All methods are tenant-scoped via context.
type WebhookCallStore interface {
// Create inserts a new call record (status = "queued").
// Returns ErrIdempotencyConflict if (webhook_id, idempotency_key) already exists.
Create(ctx context.Context, call *WebhookCallData) error
// GetByID returns a call by its UUID.
// Returns sql.ErrNoRows if not found or tenant mismatch.
GetByID(ctx context.Context, id uuid.UUID) (*WebhookCallData, error)
// GetByIdempotency returns the existing call for a given (webhookID, key).
// Returns sql.ErrNoRows if no match.
GetByIdempotency(ctx context.Context, webhookID uuid.UUID, key string) (*WebhookCallData, error)
// UpdateStatus updates mutable fields after a send attempt.
// Callers may set status, attempts, next_attempt_at, response, last_error, completed_at.
UpdateStatus(ctx context.Context, id uuid.UUID, updates map[string]any) error
// UpdateStatusCAS is like UpdateStatus but guards with AND lease_token = lease.
// Returns ErrLeaseExpired if 0 rows affected (row was reclaimed by reclaimStale).
// Worker callers must use this instead of UpdateStatus for all post-ClaimNext updates.
UpdateStatusCAS(ctx context.Context, id uuid.UUID, lease string, updates map[string]any) error
// ClaimNext atomically claims the next queued call due for processing.
// Sets status="running", started_at=now, and lease_token=new UUID.
// Does NOT increment attempts — the worker does that on terminal UpdateStatus.
// Returns sql.ErrNoRows if the queue is empty.
ClaimNext(ctx context.Context, tenantID uuid.UUID, now time.Time) (*WebhookCallData, error)
// List returns calls for the context tenant with optional filters.
List(ctx context.Context, f WebhookCallListFilter) ([]WebhookCallData, error)
// DeleteOlderThan deletes terminal calls (done/failed/dead) older than ts.
// If tenantID is uuid.Nil, deletes across all tenants (retention worker).
DeleteOlderThan(ctx context.Context, tenantID uuid.UUID, ts time.Time) (int64, error)
// ReclaimStale resets rows stuck in status='running' with started_at older than
// staleThreshold back to status='queued'. Called on worker startup and periodically
// (every 60s) to recover from crashes between ClaimNext and UpdateStatus.
// Returns the number of rows reclaimed.
ReclaimStale(ctx context.Context, staleThreshold time.Time) (int64, error)
}