mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-08-06 18:22:50 +00:00
* feat(webhooks): HTTP webhooks to trigger agents with HMAC auth and durable callbacks
Add multi-tenant HTTP webhook endpoints for agent triggering:
- /v1/webhooks/message: send messages to channels
- /v1/webhooks/llm: sync/async LLM prompts with HMAC-signed callbacks
- HMAC-256 + bearer token authentication
- Rate limiting and tenant isolation
- Durable callback worker with exponential backoff
- PG 000056 + SQLite schema v25 migrations
- Unit + integration tests, P0 tenant isolation invariants
- Channel media capability helpers for attachment routing
- Comprehensive webhook documentation and i18n strings
* fix(webhooks): address post-review findings (K1-K10)
Comprehensive post-merge fixes addressing 10 blocking code review issues
and 2 adversarial re-audit findings in webhook-agent-triggering feature:
K1: Fix auth middleware tenant context lookup sequencing — move
tenant context injection before authenticate() call to prevent
unscoped secret lookups.
K2: Canonicalize JSON payload format for jsonb compatibility across
PostgreSQL and SQLite — ensure consistent serialization without
whitespace variance to prevent hash mismatches.
K3: Add fail-closed JSON parsing in body hash extraction with explicit
error handling for malformed payloads before HMAC verification.
K4: Fix worker queue wedge by properly draining slot reservations
when delivery succeeds, preventing permanent slot occupancy.
K5: Implement lease-token optimistic concurrency control to prevent
duplicate webhook delivery under high concurrency or retry storms.
K6: Add AES-256-GCM encrypted secret storage at rest with fail-fast
skip-mount when GOCLAW_ENCRYPTION_KEY environment variable unset.
K7: Implement IP allowlist enforcement supporting both CIDR ranges
and exact IP matching with proper X-Forwarded-For parsing.
K8: Add HMAC replay nonce cache (5min expiry, non-blocking async flush)
to prevent request replay attacks on webhook handler.
K9: Fix invariant test schema selection — replace hardcoded assumption
with explicit schema name from config to support multi-schema testing.
K10: Consolidate rate limiters into single shared instance to prevent
per-endpoint limiter starvation and ensure fair rate limiting.
New database migrations:
- 000057: webhook_calls.lease_token for optimistic concurrency
- 000058: webhooks.encrypted_secret_key for AES-256-GCM encryption
New i18n keys: MsgWebhookIPDenied, MsgWebhookEncryptionUnavailable
(with English, Vietnamese, Chinese translations).
New modules:
- internal/http/webhooks_payload.go: JSON canonicalization + body hash
- internal/http/webhooks_nonce.go: Replay nonce cache implementation
- internal/http/webhooks_idempotency_test.go: Integration tests
Documentation updates:
- docs/webhooks.md: §13-14 security sections, encryption flow
- docs/00-architecture-overview.md: webhook subsystem security overview
- docs/codebase-summary.md: webhook security patterns
- docs/project-changelog.md: webhook fixes changelog
Test coverage: 53 webhook tests + 4 P0 invariant tests all passing.
No tenant isolation violations. All security gates enforced.
* docs(journals): webhook feature ship + fix cycle entries
* fix(webhooks): address Claude review findings
- webhooks_llm.go: remove misleading ptr() helper; use &completedAt
pattern for error-path audit rows (matches success path)
- webhooks_auth.go: wrap TouchLastUsed context in WithoutCancel so
background DB update isn't cancelled when HTTP response completes
- store GetByIDUnscoped (PG+SQLite): add NOT revoked / revoked = 0
filter for defense-in-depth parity with GetByHashUnscoped
- webhooks/sign.go: fix package doc — HMAC key is raw plaintext
secret bytes, not hex-decoded SHA-256
- webhooks_admin.go: check auth before encKey guard to avoid leaking
config state to unauthenticated callers
- webhooks_ratelimit.go: two-phase Load→LoadOrStore to avoid per-call
entry allocation on the hot path
* docs(webhooks): fix Sign() function doc to match actual key input
Function-level comment still referenced hex-decoded SecretHash after
the package-level doc was corrected. Align with actual caller usage
([]byte(rawSecret)).
* fix(webhooks): use WithoutCancel for worker execute DB updates
Terminal status writes in execute() ran through the worker main-loop
ctx, which is cancelled on graceful shutdown. If the outbound send
completed but the status update raced with shutdown, the row stayed
in 'running' and got re-delivered via reclaimStale. WithoutCancel
lets the DB write survive worker cancellation while preserving
propagated values (tenant ID, etc.).
* fix(webhooks): move tctx init before panic defer in worker execute
Panic recovery called updateRetry with raw ctx (no tenant ID), making
requireTenantID fail and the reset-to-retry DB write silently drop.
Row stayed 'running' until reclaimStale (~90s delay). Init tctx first
so defer closure captures tenant-scoped non-cancellable context.
* fix(webhooks): pass tenant-scoped tctx to invokeAgent in worker
execute() was passing the raw worker-loop ctx (no tenant ID) to
invokeAgent → router.Get → PGAgentStore.GetByID. GetByID reads
TenantIDFromContext which returned uuid.Nil, making every lookup
return 'agent not found'. Async LLM webhook calls silently failed
all retries. Pass tctx (already tenant-scoped + WithoutCancel) so
the router resolves the agent correctly.
* fix(tests): resolve integration test compile errors
- Remove duplicate contains() in mcp_grant_revoke_test.go (already
defined in tts_gemini_live_test.go)
- Update webhooks_admin_test.go RotateSecret call to match current
5-arg signature (newSecretHash, newPrefix, newEncryptedSecret)
* fix(webhooks): default nil scopes/ip_allowlist to empty slice in Create
PG columns are NOT NULL DEFAULT '{}'. Explicit NULL from pqStringArray(nil)
violated the constraint, breaking TestWebhookAdminCRUD/TenantIsolation.
Coerce nil slices to empty []string{} so the default applies at the DB layer.
* chore: trigger CI on digitopvn/goclaw fork
* ci: retrigger workflows
* fix(webhooks): renumber migrations to 000059-000061 for merge train
174 lines
9.2 KiB
Go
174 lines
9.2 KiB
Go
package store
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
)
|
|
|
|
// ErrIdempotencyConflict is returned when a webhook_call with the same
|
|
// (webhook_id, idempotency_key) already exists (partial unique index violation).
|
|
var ErrIdempotencyConflict = errors.New("idempotency key conflict: call already exists")
|
|
|
|
// ErrLeaseExpired is returned by UpdateStatusCAS when 0 rows were affected,
|
|
// meaning the row's lease_token no longer matches — it was reclaimed by reclaimStale
|
|
// and possibly re-claimed by another worker iteration. The caller should log and drop.
|
|
var ErrLeaseExpired = errors.New("webhook call lease expired: row reclaimed by stale sweeper")
|
|
|
|
// WebhookData represents a registered webhook.
|
|
// SecretHash is never serialized to JSON (auth token, server-side only).
|
|
// EncryptedSecret holds crypto.Encrypt(raw_secret, encKey) — decrypted at HMAC sign time.
|
|
// Existing webhooks with EncryptedSecret="" require rotation before HMAC auth is accepted.
|
|
type WebhookData struct {
|
|
ID uuid.UUID `json:"id" db:"id"`
|
|
TenantID uuid.UUID `json:"tenant_id" db:"tenant_id"`
|
|
AgentID *uuid.UUID `json:"agent_id,omitempty" db:"agent_id"`
|
|
Name string `json:"name" db:"name"`
|
|
Kind string `json:"kind" db:"kind"` // "llm" | "message"
|
|
SecretPrefix string `json:"secret_prefix" db:"secret_prefix"`
|
|
SecretHash string `json:"-" db:"secret_hash"` // SHA-256 hex; bearer-token lookup only; never serialized
|
|
EncryptedSecret string `json:"-" db:"encrypted_secret"` // AES-256-GCM of raw secret; never serialized
|
|
Scopes []string `json:"scopes" db:"scopes"`
|
|
ChannelID *uuid.UUID `json:"channel_id,omitempty" db:"channel_id"`
|
|
RateLimitPerMin int `json:"rate_limit_per_min" db:"rate_limit_per_min"`
|
|
IPAllowlist []string `json:"ip_allowlist" db:"ip_allowlist"`
|
|
RequireHMAC bool `json:"require_hmac" db:"require_hmac"`
|
|
LocalhostOnly bool `json:"localhost_only" db:"localhost_only"`
|
|
Revoked bool `json:"revoked" db:"revoked"`
|
|
CreatedBy string `json:"created_by" db:"created_by"`
|
|
CreatedAt time.Time `json:"created_at" db:"created_at"`
|
|
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
|
|
LastUsedAt *time.Time `json:"last_used_at,omitempty" db:"last_used_at"`
|
|
}
|
|
|
|
// WebhookCallData represents a single webhook invocation (queued, in-flight, or terminal).
|
|
// DeliveryID is stable across retries — used as X-Webhook-Delivery-Id header.
|
|
// StartedAt is set on ClaimNext to detect stale-running calls.
|
|
// Attempts is incremented post-send by the worker (NOT on ClaimNext).
|
|
// LeaseToken is a random UUID set atomically by ClaimNext; UpdateStatus CAS guards with AND lease_token = $N.
|
|
// If CAS hits 0 rows, the row was reclaimed by reclaimStale — the worker logs and drops the update.
|
|
type WebhookCallData struct {
|
|
ID uuid.UUID `json:"id" db:"id"`
|
|
TenantID uuid.UUID `json:"tenant_id" db:"tenant_id"`
|
|
WebhookID uuid.UUID `json:"webhook_id" db:"webhook_id"`
|
|
AgentID *uuid.UUID `json:"agent_id,omitempty" db:"agent_id"`
|
|
DeliveryID uuid.UUID `json:"delivery_id" db:"delivery_id"` // stable across retries
|
|
IdempotencyKey *string `json:"idempotency_key,omitempty" db:"idempotency_key"`
|
|
Mode string `json:"mode" db:"mode"` // "sync" | "async"
|
|
Status string `json:"status" db:"status"` // "queued"|"running"|"done"|"failed"|"dead"
|
|
CallbackURL *string `json:"callback_url,omitempty" db:"callback_url"`
|
|
Attempts int `json:"attempts" db:"attempts"`
|
|
NextAttemptAt *time.Time `json:"next_attempt_at,omitempty" db:"next_attempt_at"`
|
|
StartedAt *time.Time `json:"started_at,omitempty" db:"started_at"` // set on ClaimNext
|
|
LeaseToken *string `json:"lease_token,omitempty" db:"lease_token"` // CAS guard; set by ClaimNext, cleared by ReclaimStale
|
|
RequestPayload []byte `json:"request_payload,omitempty" db:"request_payload"`
|
|
Response []byte `json:"response,omitempty" db:"response"`
|
|
LastError *string `json:"last_error,omitempty" db:"last_error"`
|
|
CreatedAt time.Time `json:"created_at" db:"created_at"`
|
|
CompletedAt *time.Time `json:"completed_at,omitempty" db:"completed_at"`
|
|
}
|
|
|
|
// WebhookListFilter controls filtering for WebhookStore.List.
|
|
type WebhookListFilter struct {
|
|
AgentID *uuid.UUID // filter by bound agent (nil = all)
|
|
Limit int // 0 = default (50)
|
|
Offset int
|
|
}
|
|
|
|
// WebhookCallListFilter controls filtering for WebhookCallStore.List.
|
|
type WebhookCallListFilter struct {
|
|
WebhookID *uuid.UUID // filter by parent webhook (nil = all in tenant)
|
|
Status string // "" = all statuses
|
|
Limit int // 0 = default (50)
|
|
Offset int
|
|
}
|
|
|
|
// WebhookStore manages webhook registry entries.
|
|
// All methods are tenant-scoped via context (store.TenantIDFromContext).
|
|
type WebhookStore interface {
|
|
// Create inserts a new webhook. ID + CreatedAt + UpdatedAt should be
|
|
// pre-filled by the caller.
|
|
Create(ctx context.Context, w *WebhookData) error
|
|
|
|
// GetByID returns a webhook by its UUID.
|
|
// Returns sql.ErrNoRows if not found or tenant mismatch.
|
|
GetByID(ctx context.Context, id uuid.UUID) (*WebhookData, error)
|
|
|
|
// GetByHash returns an active (non-revoked) webhook by its secret_hash.
|
|
// Returns sql.ErrNoRows if not found.
|
|
GetByHash(ctx context.Context, secretHash string) (*WebhookData, error)
|
|
|
|
// GetByHashUnscoped looks up a webhook by secret_hash WITHOUT requiring tenant
|
|
// in context. Used exclusively in WebhookAuthMiddleware for pre-auth resolution;
|
|
// downstream queries remain tenant-scoped after WithTenantID injection.
|
|
// security_hash is globally unique (uq_webhooks_secret) so no tenant filter needed.
|
|
GetByHashUnscoped(ctx context.Context, secretHash string) (*WebhookData, error)
|
|
|
|
// GetByIDUnscoped looks up a webhook by UUID WITHOUT requiring tenant in context.
|
|
// Used exclusively in WebhookAuthMiddleware for HMAC pre-auth resolution.
|
|
GetByIDUnscoped(ctx context.Context, id uuid.UUID) (*WebhookData, error)
|
|
|
|
// List returns webhooks for the context tenant, with optional agent filter.
|
|
List(ctx context.Context, f WebhookListFilter) ([]WebhookData, error)
|
|
|
|
// Update applies a partial update via column→value map.
|
|
// Caller validates keys; store validates against allowlist.
|
|
Update(ctx context.Context, id uuid.UUID, updates map[string]any) error
|
|
|
|
// RotateSecret replaces the secret_hash, secret_prefix, and encrypted_secret.
|
|
// Callers (webhooks_admin.go) generate hash + prefix + encrypted form above the store layer.
|
|
RotateSecret(ctx context.Context, id uuid.UUID, newSecretHash, newPrefix, newEncryptedSecret string) error
|
|
|
|
// Revoke marks a webhook as revoked. Returns sql.ErrNoRows if not found.
|
|
Revoke(ctx context.Context, id uuid.UUID) error
|
|
|
|
// TouchLastUsed updates last_used_at. Best-effort — failures are not fatal.
|
|
TouchLastUsed(ctx context.Context, id uuid.UUID) error
|
|
}
|
|
|
|
// WebhookCallStore manages webhook call state (queued → running → terminal).
|
|
// All methods are tenant-scoped via context.
|
|
type WebhookCallStore interface {
|
|
// Create inserts a new call record (status = "queued").
|
|
// Returns ErrIdempotencyConflict if (webhook_id, idempotency_key) already exists.
|
|
Create(ctx context.Context, call *WebhookCallData) error
|
|
|
|
// GetByID returns a call by its UUID.
|
|
// Returns sql.ErrNoRows if not found or tenant mismatch.
|
|
GetByID(ctx context.Context, id uuid.UUID) (*WebhookCallData, error)
|
|
|
|
// GetByIdempotency returns the existing call for a given (webhookID, key).
|
|
// Returns sql.ErrNoRows if no match.
|
|
GetByIdempotency(ctx context.Context, webhookID uuid.UUID, key string) (*WebhookCallData, error)
|
|
|
|
// UpdateStatus updates mutable fields after a send attempt.
|
|
// Callers may set status, attempts, next_attempt_at, response, last_error, completed_at.
|
|
UpdateStatus(ctx context.Context, id uuid.UUID, updates map[string]any) error
|
|
|
|
// UpdateStatusCAS is like UpdateStatus but guards with AND lease_token = lease.
|
|
// Returns ErrLeaseExpired if 0 rows affected (row was reclaimed by reclaimStale).
|
|
// Worker callers must use this instead of UpdateStatus for all post-ClaimNext updates.
|
|
UpdateStatusCAS(ctx context.Context, id uuid.UUID, lease string, updates map[string]any) error
|
|
|
|
// ClaimNext atomically claims the next queued call due for processing.
|
|
// Sets status="running", started_at=now, and lease_token=new UUID.
|
|
// Does NOT increment attempts — the worker does that on terminal UpdateStatus.
|
|
// Returns sql.ErrNoRows if the queue is empty.
|
|
ClaimNext(ctx context.Context, tenantID uuid.UUID, now time.Time) (*WebhookCallData, error)
|
|
|
|
// List returns calls for the context tenant with optional filters.
|
|
List(ctx context.Context, f WebhookCallListFilter) ([]WebhookCallData, error)
|
|
|
|
// DeleteOlderThan deletes terminal calls (done/failed/dead) older than ts.
|
|
// If tenantID is uuid.Nil, deletes across all tenants (retention worker).
|
|
DeleteOlderThan(ctx context.Context, tenantID uuid.UUID, ts time.Time) (int64, error)
|
|
|
|
// ReclaimStale resets rows stuck in status='running' with started_at older than
|
|
// staleThreshold back to status='queued'. Called on worker startup and periodically
|
|
// (every 60s) to recover from crashes between ClaimNext and UpdateStatus.
|
|
// Returns the number of rows reclaimed.
|
|
ReclaimStale(ctx context.Context, staleThreshold time.Time) (int64, error)
|
|
}
|