Files
goclaw/tests/integration/webhooks_admin_test.go
Duy /zuey/andGitHub ddf8e1099f feat(webhooks): HTTP webhooks to trigger agents with HMAC auth + durable callbacks (#2)
* feat(webhooks): HTTP webhooks to trigger agents with HMAC auth and durable callbacks

Add multi-tenant HTTP webhook endpoints for agent triggering:
- /v1/webhooks/message: send messages to channels
- /v1/webhooks/llm: sync/async LLM prompts with HMAC-signed callbacks
- HMAC-256 + bearer token authentication
- Rate limiting and tenant isolation
- Durable callback worker with exponential backoff
- PG 000056 + SQLite schema v25 migrations
- Unit + integration tests, P0 tenant isolation invariants
- Channel media capability helpers for attachment routing
- Comprehensive webhook documentation and i18n strings

* fix(webhooks): address post-review findings (K1-K10)

Comprehensive post-merge fixes addressing 10 blocking code review issues
and 2 adversarial re-audit findings in webhook-agent-triggering feature:

K1: Fix auth middleware tenant context lookup sequencing — move
    tenant context injection before authenticate() call to prevent
    unscoped secret lookups.

K2: Canonicalize JSON payload format for jsonb compatibility across
    PostgreSQL and SQLite — ensure consistent serialization without
    whitespace variance to prevent hash mismatches.

K3: Add fail-closed JSON parsing in body hash extraction with explicit
    error handling for malformed payloads before HMAC verification.

K4: Fix worker queue wedge by properly draining slot reservations
    when delivery succeeds, preventing permanent slot occupancy.

K5: Implement lease-token optimistic concurrency control to prevent
    duplicate webhook delivery under high concurrency or retry storms.

K6: Add AES-256-GCM encrypted secret storage at rest with fail-fast
    skip-mount when GOCLAW_ENCRYPTION_KEY environment variable unset.

K7: Implement IP allowlist enforcement supporting both CIDR ranges
    and exact IP matching with proper X-Forwarded-For parsing.

K8: Add HMAC replay nonce cache (5min expiry, non-blocking async flush)
    to prevent request replay attacks on webhook handler.

K9: Fix invariant test schema selection — replace hardcoded assumption
    with explicit schema name from config to support multi-schema testing.

K10: Consolidate rate limiters into single shared instance to prevent
     per-endpoint limiter starvation and ensure fair rate limiting.

New database migrations:
- 000057: webhook_calls.lease_token for optimistic concurrency
- 000058: webhooks.encrypted_secret_key for AES-256-GCM encryption

New i18n keys: MsgWebhookIPDenied, MsgWebhookEncryptionUnavailable
(with English, Vietnamese, Chinese translations).

New modules:
- internal/http/webhooks_payload.go: JSON canonicalization + body hash
- internal/http/webhooks_nonce.go: Replay nonce cache implementation
- internal/http/webhooks_idempotency_test.go: Integration tests

Documentation updates:
- docs/webhooks.md: §13-14 security sections, encryption flow
- docs/00-architecture-overview.md: webhook subsystem security overview
- docs/codebase-summary.md: webhook security patterns
- docs/project-changelog.md: webhook fixes changelog

Test coverage: 53 webhook tests + 4 P0 invariant tests all passing.
No tenant isolation violations. All security gates enforced.

* docs(journals): webhook feature ship + fix cycle entries

* fix(webhooks): address Claude review findings

- webhooks_llm.go: remove misleading ptr() helper; use &completedAt
  pattern for error-path audit rows (matches success path)
- webhooks_auth.go: wrap TouchLastUsed context in WithoutCancel so
  background DB update isn't cancelled when HTTP response completes
- store GetByIDUnscoped (PG+SQLite): add NOT revoked / revoked = 0
  filter for defense-in-depth parity with GetByHashUnscoped
- webhooks/sign.go: fix package doc — HMAC key is raw plaintext
  secret bytes, not hex-decoded SHA-256
- webhooks_admin.go: check auth before encKey guard to avoid leaking
  config state to unauthenticated callers
- webhooks_ratelimit.go: two-phase Load→LoadOrStore to avoid per-call
  entry allocation on the hot path

* docs(webhooks): fix Sign() function doc to match actual key input

Function-level comment still referenced hex-decoded SecretHash after
the package-level doc was corrected. Align with actual caller usage
([]byte(rawSecret)).

* fix(webhooks): use WithoutCancel for worker execute DB updates

Terminal status writes in execute() ran through the worker main-loop
ctx, which is cancelled on graceful shutdown. If the outbound send
completed but the status update raced with shutdown, the row stayed
in 'running' and got re-delivered via reclaimStale. WithoutCancel
lets the DB write survive worker cancellation while preserving
propagated values (tenant ID, etc.).

* fix(webhooks): move tctx init before panic defer in worker execute

Panic recovery called updateRetry with raw ctx (no tenant ID), making
requireTenantID fail and the reset-to-retry DB write silently drop.
Row stayed 'running' until reclaimStale (~90s delay). Init tctx first
so defer closure captures tenant-scoped non-cancellable context.

* fix(webhooks): pass tenant-scoped tctx to invokeAgent in worker

execute() was passing the raw worker-loop ctx (no tenant ID) to
invokeAgent → router.Get → PGAgentStore.GetByID. GetByID reads
TenantIDFromContext which returned uuid.Nil, making every lookup
return 'agent not found'. Async LLM webhook calls silently failed
all retries. Pass tctx (already tenant-scoped + WithoutCancel) so
the router resolves the agent correctly.

* fix(tests): resolve integration test compile errors

- Remove duplicate contains() in mcp_grant_revoke_test.go (already
  defined in tts_gemini_live_test.go)
- Update webhooks_admin_test.go RotateSecret call to match current
  5-arg signature (newSecretHash, newPrefix, newEncryptedSecret)

* fix(webhooks): default nil scopes/ip_allowlist to empty slice in Create

PG columns are NOT NULL DEFAULT '{}'. Explicit NULL from pqStringArray(nil)
violated the constraint, breaking TestWebhookAdminCRUD/TenantIsolation.
Coerce nil slices to empty []string{} so the default applies at the DB layer.

* chore: trigger CI on digitopvn/goclaw fork

* ci: retrigger workflows

* fix(webhooks): renumber migrations to 000059-000061 for merge train
2026-05-11 13:29:24 +07:00

188 lines
4.9 KiB
Go

//go:build integration
package integration
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"testing"
"github.com/google/uuid"
"github.com/nextlevelbuilder/goclaw/internal/store"
"github.com/nextlevelbuilder/goclaw/internal/store/pg"
)
// seedWebhook creates a webhook in the database and returns its ID + raw secret.
func seedWebhook(t *testing.T, db *sql.DB, tenantID uuid.UUID, kind string) (webhookID uuid.UUID, rawSecret string) {
t.Helper()
webhookID = uuid.New()
rawSecret = "wh_testsecret_" + webhookID.String()[:8]
// Hash the secret as the store does.
h := sha256.Sum256([]byte(rawSecret))
hashHex := hex.EncodeToString(h[:])
_, err := db.Exec(`
INSERT INTO webhooks (id, tenant_id, kind, secret_prefix, secret_hash, status)
VALUES ($1, $2, $3, $4, $5, 'active')
`, webhookID, tenantID, kind, "wh_test", hashHex)
if err != nil {
t.Fatalf("seed webhook: %v", err)
}
t.Cleanup(func() {
db.Exec("DELETE FROM webhook_calls WHERE webhook_id = $1", webhookID)
db.Exec("DELETE FROM webhooks WHERE id = $1", webhookID)
})
return webhookID, rawSecret
}
// TestWebhookAdminCRUD tests basic admin CRUD: create, list, get, update, rotate, revoke.
func TestWebhookAdminCRUD(t *testing.T) {
db := testDB(t)
tenantID, _ := seedTenantAgent(t, db)
// Initialize store.
s := pg.NewPGWebhookStore(db)
ctx := context.Background()
ctx = store.WithTenantID(ctx, tenantID)
// Create webhook.
wh := &store.WebhookData{
ID: uuid.New(),
TenantID: tenantID,
Kind: "llm",
SecretPrefix: "wh_test",
RateLimitPerMin: 60,
}
rawSecret := "wh_testsecret_initial"
h := sha256.Sum256([]byte(rawSecret))
wh.SecretHash = hex.EncodeToString(h[:])
err := s.Create(ctx, wh)
if err != nil {
t.Fatalf("Create failed: %v", err)
}
// Get webhook.
retrieved, err := s.GetByID(ctx, wh.ID)
if err != nil {
t.Fatalf("GetByID failed: %v", err)
}
if retrieved.ID != wh.ID {
t.Errorf("retrieved webhook ID mismatch: got %v, want %v", retrieved.ID, wh.ID)
}
// List webhooks.
list, err := s.List(ctx, store.WebhookListFilter{})
if err != nil {
t.Fatalf("List failed: %v", err)
}
if len(list) < 1 {
t.Errorf("List returned no webhooks")
}
// Update webhook.
err = s.Update(ctx, wh.ID, map[string]any{
"rate_limit_per_min": 120,
})
if err != nil {
t.Fatalf("Update failed: %v", err)
}
// Verify update.
updated, err := s.GetByID(ctx, wh.ID)
if err != nil {
t.Fatalf("GetByID after update failed: %v", err)
}
if updated.RateLimitPerMin != 120 {
t.Errorf("rate limit not updated: got %d, want 120", updated.RateLimitPerMin)
}
// Rotate secret.
newRawSecret := "wh_newsecret_rotated"
newH := sha256.Sum256([]byte(newRawSecret))
newHashHex := hex.EncodeToString(newH[:])
err = s.RotateSecret(ctx, wh.ID, newHashHex, "wh_newrot", "encrypted_placeholder")
if err != nil {
t.Fatalf("RotateSecret failed: %v", err)
}
// Verify old secret hash is now secret_hash_prev.
rotated, err := s.GetByID(ctx, wh.ID)
if err != nil {
t.Fatalf("GetByID after rotate failed: %v", err)
}
if rotated.SecretHash != newHashHex {
t.Errorf("secret_hash not updated: got %s, want %s", rotated.SecretHash, newHashHex)
}
// Revoke webhook.
err = s.Revoke(ctx, wh.ID)
if err != nil {
t.Fatalf("Revoke failed: %v", err)
}
// Verify revoked.
revoked, err := s.GetByID(ctx, wh.ID)
if err != nil {
t.Fatalf("GetByID after revoke failed: %v", err)
}
if !revoked.Revoked {
t.Errorf("webhook not revoked: %+v", revoked)
}
}
// TestWebhookAdminTenantIsolation tests that webhooks from tenant A cannot be accessed by tenant B.
func TestWebhookAdminTenantIsolation(t *testing.T) {
db := testDB(t)
tenantA, _ := seedTenantAgent(t, db)
tenantB, _ := seedTenantAgent(t, db)
sA := pg.NewPGWebhookStore(db)
sB := pg.NewPGWebhookStore(db)
ctxA := context.Background()
ctxA = store.WithTenantID(ctxA, tenantA)
ctxB := context.Background()
ctxB = store.WithTenantID(ctxB, tenantB)
// Tenant A creates a webhook.
whA := &store.WebhookData{
ID: uuid.New(),
TenantID: tenantA,
Kind: "llm",
}
h := sha256.Sum256([]byte("secret_a"))
whA.SecretHash = hex.EncodeToString(h[:])
err := sA.Create(ctxA, whA)
if err != nil {
t.Fatalf("Tenant A create failed: %v", err)
}
// Tenant B tries to access tenant A's webhook directly from DB.
// GetByID should filter by tenant_id in the WHERE clause.
ctxBToGetA := store.WithTenantID(context.Background(), tenantB)
_, err = sB.GetByID(ctxBToGetA, whA.ID)
if err != sql.ErrNoRows {
t.Errorf("Tenant B should not access Tenant A's webhook; got err=%v", err)
}
// Tenant B lists webhooks — should only see their own.
listB, err := sB.List(ctxB, store.WebhookListFilter{})
if err != nil {
t.Fatalf("Tenant B list failed: %v", err)
}
for _, w := range listB {
if w.TenantID != tenantB {
t.Errorf("Tenant B listed webhook with wrong tenant_id: %v", w.TenantID)
}
}
}