mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-08-06 10:21:11 +00:00
* feat(webhooks): HTTP webhooks to trigger agents with HMAC auth and durable callbacks
Add multi-tenant HTTP webhook endpoints for agent triggering:
- /v1/webhooks/message: send messages to channels
- /v1/webhooks/llm: sync/async LLM prompts with HMAC-signed callbacks
- HMAC-256 + bearer token authentication
- Rate limiting and tenant isolation
- Durable callback worker with exponential backoff
- PG 000056 + SQLite schema v25 migrations
- Unit + integration tests, P0 tenant isolation invariants
- Channel media capability helpers for attachment routing
- Comprehensive webhook documentation and i18n strings
* fix(webhooks): address post-review findings (K1-K10)
Comprehensive post-merge fixes addressing 10 blocking code review issues
and 2 adversarial re-audit findings in webhook-agent-triggering feature:
K1: Fix auth middleware tenant context lookup sequencing — move
tenant context injection before authenticate() call to prevent
unscoped secret lookups.
K2: Canonicalize JSON payload format for jsonb compatibility across
PostgreSQL and SQLite — ensure consistent serialization without
whitespace variance to prevent hash mismatches.
K3: Add fail-closed JSON parsing in body hash extraction with explicit
error handling for malformed payloads before HMAC verification.
K4: Fix worker queue wedge by properly draining slot reservations
when delivery succeeds, preventing permanent slot occupancy.
K5: Implement lease-token optimistic concurrency control to prevent
duplicate webhook delivery under high concurrency or retry storms.
K6: Add AES-256-GCM encrypted secret storage at rest with fail-fast
skip-mount when GOCLAW_ENCRYPTION_KEY environment variable unset.
K7: Implement IP allowlist enforcement supporting both CIDR ranges
and exact IP matching with proper X-Forwarded-For parsing.
K8: Add HMAC replay nonce cache (5min expiry, non-blocking async flush)
to prevent request replay attacks on webhook handler.
K9: Fix invariant test schema selection — replace hardcoded assumption
with explicit schema name from config to support multi-schema testing.
K10: Consolidate rate limiters into single shared instance to prevent
per-endpoint limiter starvation and ensure fair rate limiting.
New database migrations:
- 000057: webhook_calls.lease_token for optimistic concurrency
- 000058: webhooks.encrypted_secret_key for AES-256-GCM encryption
New i18n keys: MsgWebhookIPDenied, MsgWebhookEncryptionUnavailable
(with English, Vietnamese, Chinese translations).
New modules:
- internal/http/webhooks_payload.go: JSON canonicalization + body hash
- internal/http/webhooks_nonce.go: Replay nonce cache implementation
- internal/http/webhooks_idempotency_test.go: Integration tests
Documentation updates:
- docs/webhooks.md: §13-14 security sections, encryption flow
- docs/00-architecture-overview.md: webhook subsystem security overview
- docs/codebase-summary.md: webhook security patterns
- docs/project-changelog.md: webhook fixes changelog
Test coverage: 53 webhook tests + 4 P0 invariant tests all passing.
No tenant isolation violations. All security gates enforced.
* docs(journals): webhook feature ship + fix cycle entries
* fix(webhooks): address Claude review findings
- webhooks_llm.go: remove misleading ptr() helper; use &completedAt
pattern for error-path audit rows (matches success path)
- webhooks_auth.go: wrap TouchLastUsed context in WithoutCancel so
background DB update isn't cancelled when HTTP response completes
- store GetByIDUnscoped (PG+SQLite): add NOT revoked / revoked = 0
filter for defense-in-depth parity with GetByHashUnscoped
- webhooks/sign.go: fix package doc — HMAC key is raw plaintext
secret bytes, not hex-decoded SHA-256
- webhooks_admin.go: check auth before encKey guard to avoid leaking
config state to unauthenticated callers
- webhooks_ratelimit.go: two-phase Load→LoadOrStore to avoid per-call
entry allocation on the hot path
* docs(webhooks): fix Sign() function doc to match actual key input
Function-level comment still referenced hex-decoded SecretHash after
the package-level doc was corrected. Align with actual caller usage
([]byte(rawSecret)).
* fix(webhooks): use WithoutCancel for worker execute DB updates
Terminal status writes in execute() ran through the worker main-loop
ctx, which is cancelled on graceful shutdown. If the outbound send
completed but the status update raced with shutdown, the row stayed
in 'running' and got re-delivered via reclaimStale. WithoutCancel
lets the DB write survive worker cancellation while preserving
propagated values (tenant ID, etc.).
* fix(webhooks): move tctx init before panic defer in worker execute
Panic recovery called updateRetry with raw ctx (no tenant ID), making
requireTenantID fail and the reset-to-retry DB write silently drop.
Row stayed 'running' until reclaimStale (~90s delay). Init tctx first
so defer closure captures tenant-scoped non-cancellable context.
* fix(webhooks): pass tenant-scoped tctx to invokeAgent in worker
execute() was passing the raw worker-loop ctx (no tenant ID) to
invokeAgent → router.Get → PGAgentStore.GetByID. GetByID reads
TenantIDFromContext which returned uuid.Nil, making every lookup
return 'agent not found'. Async LLM webhook calls silently failed
all retries. Pass tctx (already tenant-scoped + WithoutCancel) so
the router resolves the agent correctly.
* fix(tests): resolve integration test compile errors
- Remove duplicate contains() in mcp_grant_revoke_test.go (already
defined in tts_gemini_live_test.go)
- Update webhooks_admin_test.go RotateSecret call to match current
5-arg signature (newSecretHash, newPrefix, newEncryptedSecret)
* fix(webhooks): default nil scopes/ip_allowlist to empty slice in Create
PG columns are NOT NULL DEFAULT '{}'. Explicit NULL from pqStringArray(nil)
violated the constraint, breaking TestWebhookAdminCRUD/TenantIsolation.
Coerce nil slices to empty []string{} so the default applies at the DB layer.
* chore: trigger CI on digitopvn/goclaw fork
* ci: retrigger workflows
* fix(webhooks): renumber migrations to 000059-000061 for merge train
250 lines
7.6 KiB
Go
250 lines
7.6 KiB
Go
package channels
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"log/slog"
|
|
"net/http"
|
|
"os"
|
|
"regexp"
|
|
"strings"
|
|
|
|
"github.com/google/uuid"
|
|
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
|
"github.com/nextlevelbuilder/goclaw/internal/store"
|
|
)
|
|
|
|
// WebhookRoute holds a path and handler pair for mounting on the main gateway mux.
|
|
type WebhookRoute struct {
|
|
Path string
|
|
Handler http.Handler
|
|
}
|
|
|
|
// dispatchOutbound consumes outbound messages from the bus and routes them
|
|
// to the appropriate channel. Internal channels are silently skipped.
|
|
func (m *Manager) dispatchOutbound(ctx context.Context) {
|
|
slog.Info("outbound dispatcher started")
|
|
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
slog.Info("outbound dispatcher stopped")
|
|
return
|
|
default:
|
|
msg, ok := m.bus.SubscribeOutbound(ctx)
|
|
if !ok {
|
|
continue
|
|
}
|
|
|
|
// Skip internal channels
|
|
if IsInternalChannel(msg.Channel) {
|
|
continue
|
|
}
|
|
|
|
m.mu.RLock()
|
|
channel, exists := m.channels[msg.Channel]
|
|
m.mu.RUnlock()
|
|
|
|
if !exists {
|
|
slog.Warn("unknown channel for outbound message", "channel", msg.Channel)
|
|
continue
|
|
}
|
|
|
|
// Filter out temp media files that no longer exist (already sent by another dispatch).
|
|
if len(msg.Media) > 0 {
|
|
tmpDir := os.TempDir()
|
|
filtered := msg.Media[:0]
|
|
for _, media := range msg.Media {
|
|
if media.URL != "" && strings.HasPrefix(media.URL, tmpDir) {
|
|
if _, err := os.Stat(media.URL); err != nil {
|
|
slog.Debug("skipping already-delivered temp media", "path", media.URL)
|
|
continue
|
|
}
|
|
}
|
|
filtered = append(filtered, media)
|
|
}
|
|
msg.Media = filtered
|
|
// If only media was in this message and all files are gone, skip entirely.
|
|
if len(msg.Media) == 0 && msg.Content == "" {
|
|
continue
|
|
}
|
|
}
|
|
|
|
// Add tenant context for per-tenant TTS auto-apply
|
|
sendCtx := ctx
|
|
if msg.TenantID != uuid.Nil {
|
|
sendCtx = store.WithTenantID(ctx, msg.TenantID)
|
|
}
|
|
|
|
// Add agent audio context for per-agent TTS voice override
|
|
if msg.AgentID != uuid.Nil && len(msg.AgentOtherConfig) > 0 {
|
|
sendCtx = store.WithAgentAudio(sendCtx, store.AgentAudioSnapshot{
|
|
AgentID: msg.AgentID,
|
|
OtherConfig: msg.AgentOtherConfig,
|
|
})
|
|
}
|
|
|
|
if err := channel.Send(sendCtx, msg); err != nil {
|
|
slog.Error("error sending message to channel",
|
|
"channel", msg.Channel,
|
|
"chat_id", msg.ChatID,
|
|
"content_len", len(msg.Content),
|
|
"content_preview", Truncate(msg.Content, 160),
|
|
"error", err,
|
|
)
|
|
// Try to send a text-only error notification back to the chat.
|
|
// Only for media failures — text-only failures likely mean the chat
|
|
// is inaccessible (kicked, blocked, etc.) so retrying won't help.
|
|
if len(msg.Media) > 0 {
|
|
notifyMsg := bus.OutboundMessage{
|
|
Channel: msg.Channel,
|
|
ChatID: msg.ChatID,
|
|
Content: formatChannelSendError(err),
|
|
Metadata: sendErrorMeta(msg.Metadata),
|
|
TenantID: msg.TenantID,
|
|
}
|
|
if err2 := channel.Send(sendCtx, notifyMsg); err2 != nil {
|
|
slog.Warn("failed to send error notification",
|
|
"channel", msg.Channel, "error", err2)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Clean up temp media files only. Workspace-generated files are preserved
|
|
// so they remain accessible via workspace/web UI after delivery.
|
|
tmpDir := os.TempDir()
|
|
for _, media := range msg.Media {
|
|
if media.URL != "" && strings.HasPrefix(media.URL, tmpDir) {
|
|
if err := os.Remove(media.URL); err != nil {
|
|
slog.Debug("failed to clean up media file", "path", media.URL, "error", err)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// WebhookHandlers returns all webhook handlers from channels that implement WebhookChannel.
|
|
// Used to mount webhook routes on the main gateway mux.
|
|
func (m *Manager) WebhookHandlers() []WebhookRoute {
|
|
m.mu.RLock()
|
|
defer m.mu.RUnlock()
|
|
|
|
var routes []WebhookRoute
|
|
for _, ch := range m.channels {
|
|
if wh, ok := ch.(WebhookChannel); ok {
|
|
if path, handler := wh.WebhookHandler(); path != "" && handler != nil {
|
|
routes = append(routes, WebhookRoute{Path: path, Handler: handler})
|
|
}
|
|
}
|
|
}
|
|
return routes
|
|
}
|
|
|
|
// SendToChannel delivers a message to a specific channel by name.
|
|
func (m *Manager) SendToChannel(ctx context.Context, channelName, chatID, content string) error {
|
|
m.mu.RLock()
|
|
channel, exists := m.channels[channelName]
|
|
m.mu.RUnlock()
|
|
|
|
if !exists {
|
|
return fmt.Errorf("channel %s not found", channelName)
|
|
}
|
|
|
|
msg := bus.OutboundMessage{
|
|
Channel: channelName,
|
|
ChatID: chatID,
|
|
Content: content,
|
|
}
|
|
|
|
return channel.Send(ctx, msg)
|
|
}
|
|
|
|
// SendMediaToChannel delivers a message with media attachments to a specific channel by name.
|
|
// media must be non-empty; use SendToChannel for text-only messages.
|
|
// Returns ErrMediaUnsupported if the channel type does not support media.
|
|
func (m *Manager) SendMediaToChannel(ctx context.Context, channelName, chatID, content string, media []bus.MediaAttachment) error {
|
|
if len(media) == 0 {
|
|
return fmt.Errorf("SendMediaToChannel: media slice must not be empty; use SendToChannel for text-only messages")
|
|
}
|
|
|
|
m.mu.RLock()
|
|
channel, exists := m.channels[channelName]
|
|
m.mu.RUnlock()
|
|
|
|
if !exists {
|
|
return fmt.Errorf("channel %s not found", channelName)
|
|
}
|
|
|
|
if !IsMediaCapable(channel.Type()) {
|
|
return fmt.Errorf("%w: %s (%s)", ErrMediaUnsupported, channelName, channel.Type())
|
|
}
|
|
|
|
msg := bus.OutboundMessage{
|
|
Channel: channelName,
|
|
ChatID: chatID,
|
|
Content: content,
|
|
Media: media,
|
|
}
|
|
|
|
return channel.Send(ctx, msg)
|
|
}
|
|
|
|
// --- Send error notification helpers ---
|
|
|
|
// telegramAPIDescRe extracts the human-readable description from Telegram Bot API errors.
|
|
// Example: `telego: sendPhoto: api: 400 "Bad Request: not enough rights to send photos to the chat"`
|
|
//
|
|
// → "not enough rights to send photos to the chat"
|
|
var telegramAPIDescRe = regexp.MustCompile(`"Bad Request:\s*(.+?)"`)
|
|
|
|
// formatChannelSendError converts a channel.Send error into a user-friendly message.
|
|
// Never exposes raw library/HTTP details.
|
|
func formatChannelSendError(err error) string {
|
|
raw := err.Error()
|
|
lower := strings.ToLower(raw)
|
|
|
|
// Telegram "Bad Request: <description>" — extract description
|
|
if m := telegramAPIDescRe.FindStringSubmatch(raw); len(m) == 2 {
|
|
return fmt.Sprintf("⚠️ Send failed: %s", m[1])
|
|
}
|
|
|
|
// Common Telegram API errors (non-Bad Request)
|
|
switch {
|
|
case strings.Contains(lower, "not enough rights"):
|
|
return "⚠️ Send failed: bot doesn't have permission to send this type of message."
|
|
case strings.Contains(lower, "chat not found"):
|
|
return "⚠️ Send failed: chat not found."
|
|
case strings.Contains(lower, "bot was blocked"):
|
|
return "⚠️ Send failed: bot was blocked by the user."
|
|
case strings.Contains(lower, "user is deactivated"):
|
|
return "⚠️ Send failed: user account is deactivated."
|
|
case strings.Contains(lower, "too many requests") || strings.Contains(lower, "flood"):
|
|
return "⚠️ Send failed: rate limited by Telegram. Please try again later."
|
|
case strings.Contains(lower, "file is too big") || strings.Contains(lower, "wrong file"):
|
|
return "⚠️ Send failed: file is too large or invalid for Telegram."
|
|
}
|
|
|
|
// Generic fallback — don't expose internals
|
|
return "⚠️ Failed to deliver message. Check bot logs for details."
|
|
}
|
|
|
|
// sendErrorMeta copies only the routing fields from outbound metadata.
|
|
// Strips reply_to_message_id, placeholder_key, audio_as_voice, etc.
|
|
// that could cause unintended side effects on the error notification.
|
|
func sendErrorMeta(orig map[string]string) map[string]string {
|
|
if orig == nil {
|
|
return nil
|
|
}
|
|
meta := make(map[string]string)
|
|
for _, k := range []string{"local_key", "message_thread_id"} {
|
|
if v := orig[k]; v != "" {
|
|
meta[k] = v
|
|
}
|
|
}
|
|
if len(meta) == 0 {
|
|
return nil
|
|
}
|
|
return meta
|
|
}
|