tiennm99/goclaw
1
0
Fork 0
mirror of https://github.com/tiennm99/goclaw.git synced 2026-09-17 10:20:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
3084896dc32e94db7e96309b01775c898fa221bb
goclaw/docs/codebase-summary.md
T
Duy /zuey/andGitHub ddf8e1099f feat(webhooks): HTTP webhooks to trigger agents with HMAC auth + durable callbacks (#2) 
* feat(webhooks): HTTP webhooks to trigger agents with HMAC auth and durable callbacks

Add multi-tenant HTTP webhook endpoints for agent triggering:
- /v1/webhooks/message: send messages to channels
- /v1/webhooks/llm: sync/async LLM prompts with HMAC-signed callbacks
- HMAC-256 + bearer token authentication
- Rate limiting and tenant isolation
- Durable callback worker with exponential backoff
- PG 000056 + SQLite schema v25 migrations
- Unit + integration tests, P0 tenant isolation invariants
- Channel media capability helpers for attachment routing
- Comprehensive webhook documentation and i18n strings

* fix(webhooks): address post-review findings (K1-K10)

Comprehensive post-merge fixes addressing 10 blocking code review issues
and 2 adversarial re-audit findings in webhook-agent-triggering feature:

K1: Fix auth middleware tenant context lookup sequencing — move
    tenant context injection before authenticate() call to prevent
    unscoped secret lookups.

K2: Canonicalize JSON payload format for jsonb compatibility across
    PostgreSQL and SQLite — ensure consistent serialization without
    whitespace variance to prevent hash mismatches.

K3: Add fail-closed JSON parsing in body hash extraction with explicit
    error handling for malformed payloads before HMAC verification.

K4: Fix worker queue wedge by properly draining slot reservations
    when delivery succeeds, preventing permanent slot occupancy.

K5: Implement lease-token optimistic concurrency control to prevent
    duplicate webhook delivery under high concurrency or retry storms.

K6: Add AES-256-GCM encrypted secret storage at rest with fail-fast
    skip-mount when GOCLAW_ENCRYPTION_KEY environment variable unset.

K7: Implement IP allowlist enforcement supporting both CIDR ranges
    and exact IP matching with proper X-Forwarded-For parsing.

K8: Add HMAC replay nonce cache (5min expiry, non-blocking async flush)
    to prevent request replay attacks on webhook handler.

K9: Fix invariant test schema selection — replace hardcoded assumption
    with explicit schema name from config to support multi-schema testing.

K10: Consolidate rate limiters into single shared instance to prevent
     per-endpoint limiter starvation and ensure fair rate limiting.

New database migrations:
- 000057: webhook_calls.lease_token for optimistic concurrency
- 000058: webhooks.encrypted_secret_key for AES-256-GCM encryption

New i18n keys: MsgWebhookIPDenied, MsgWebhookEncryptionUnavailable
(with English, Vietnamese, Chinese translations).

New modules:
- internal/http/webhooks_payload.go: JSON canonicalization + body hash
- internal/http/webhooks_nonce.go: Replay nonce cache implementation
- internal/http/webhooks_idempotency_test.go: Integration tests

Documentation updates:
- docs/webhooks.md: §13-14 security sections, encryption flow
- docs/00-architecture-overview.md: webhook subsystem security overview
- docs/codebase-summary.md: webhook security patterns
- docs/project-changelog.md: webhook fixes changelog

Test coverage: 53 webhook tests + 4 P0 invariant tests all passing.
No tenant isolation violations. All security gates enforced.

* docs(journals): webhook feature ship + fix cycle entries

* fix(webhooks): address Claude review findings

- webhooks_llm.go: remove misleading ptr() helper; use &completedAt
  pattern for error-path audit rows (matches success path)
- webhooks_auth.go: wrap TouchLastUsed context in WithoutCancel so
  background DB update isn't cancelled when HTTP response completes
- store GetByIDUnscoped (PG+SQLite): add NOT revoked / revoked = 0
  filter for defense-in-depth parity with GetByHashUnscoped
- webhooks/sign.go: fix package doc — HMAC key is raw plaintext
  secret bytes, not hex-decoded SHA-256
- webhooks_admin.go: check auth before encKey guard to avoid leaking
  config state to unauthenticated callers
- webhooks_ratelimit.go: two-phase Load→LoadOrStore to avoid per-call
  entry allocation on the hot path

* docs(webhooks): fix Sign() function doc to match actual key input

Function-level comment still referenced hex-decoded SecretHash after
the package-level doc was corrected. Align with actual caller usage
([]byte(rawSecret)).

* fix(webhooks): use WithoutCancel for worker execute DB updates

Terminal status writes in execute() ran through the worker main-loop
ctx, which is cancelled on graceful shutdown. If the outbound send
completed but the status update raced with shutdown, the row stayed
in 'running' and got re-delivered via reclaimStale. WithoutCancel
lets the DB write survive worker cancellation while preserving
propagated values (tenant ID, etc.).

* fix(webhooks): move tctx init before panic defer in worker execute

Panic recovery called updateRetry with raw ctx (no tenant ID), making
requireTenantID fail and the reset-to-retry DB write silently drop.
Row stayed 'running' until reclaimStale (~90s delay). Init tctx first
so defer closure captures tenant-scoped non-cancellable context.

* fix(webhooks): pass tenant-scoped tctx to invokeAgent in worker

execute() was passing the raw worker-loop ctx (no tenant ID) to
invokeAgent → router.Get → PGAgentStore.GetByID. GetByID reads
TenantIDFromContext which returned uuid.Nil, making every lookup
return 'agent not found'. Async LLM webhook calls silently failed
all retries. Pass tctx (already tenant-scoped + WithoutCancel) so
the router resolves the agent correctly.

* fix(tests): resolve integration test compile errors

- Remove duplicate contains() in mcp_grant_revoke_test.go (already
  defined in tts_gemini_live_test.go)
- Update webhooks_admin_test.go RotateSecret call to match current
  5-arg signature (newSecretHash, newPrefix, newEncryptedSecret)

* fix(webhooks): default nil scopes/ip_allowlist to empty slice in Create

PG columns are NOT NULL DEFAULT '{}'. Explicit NULL from pqStringArray(nil)
violated the constraint, breaking TestWebhookAdminCRUD/TenantIsolation.
Coerce nil slices to empty []string{} so the default applies at the DB layer.

* chore: trigger CI on digitopvn/goclaw fork

* ci: retrigger workflows

* fix(webhooks): renumber migrations to 000059-000061 for merge train
2026-05-11 13:29:24 +07:00

204 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Codebase Summary
High-level map of GoClaw modules and key cross-cutting concerns.
For system design see `docs/00-architecture-overview.md`; for API contract see `docs/18-http-api.md`.
---
## Module Map
| Path | Purpose |
|------|---------|
| `cmd/` | CLI (cobra): `serve`, `onboard`, `migrate` commands |
| `internal/agent/` | Agent loop (think→act→observe), router, input guard |
| `internal/audio/` | TTS provider layer (see §TTS below) |
| `internal/bootstrap/` | SOUL/IDENTITY system prompts + per-user seed |
| `internal/channels/` | Telegram, Feishu, Zalo, Discord, WhatsApp connectors |
| `internal/config/` | JSON5 config loading + env overlay |
| `internal/crypto/` | AES-256-GCM encryption for API keys |
| `internal/gateway/` | WS + HTTP server, client, method router |
| `internal/http/` | HTTP API handlers (`/v1/*`) |
| `internal/i18n/` | Backend message catalog (EN/VI/ZH) + `T(locale, key, args)` |
| `internal/memory/` | pgvector 3-tier memory system |
| `internal/mcp/` | Model Context Protocol bridge |
| `internal/permissions/` | RBAC: admin / operator / viewer |
| `internal/pipeline/` | 8-stage agent pipeline |
| `internal/providers/` | LLM providers (Anthropic, OpenAI-compat, Qwen, Claude CLI) |
| `internal/store/` | Store interfaces + PG + SQLite implementations |
| `internal/tools/` | Tool registry (filesystem, exec, web, MCP, delegate) |
| `internal/tts/` | Back-compat alias package for old import paths |
| `internal/vault/` | Knowledge Vault: wikilinks, hybrid search, FS sync |
| `migrations/` | PostgreSQL migration files |
| `ui/web/` | React SPA (Vite, Tailwind, Radix UI, Zustand) |
| `ui/desktop/` | Wails v2 desktop app (SQLite, embedded gateway) |
---
## TTS Subsystem
### Overview
TTS lives in `internal/audio/` (canonical) with a back-compat alias at `internal/tts/`.
Five providers ship out-of-the-box: OpenAI, ElevenLabs, Edge TTS, MiniMax, Gemini.
### Provider Capabilities Schema
Each provider implements `audio.TTSProvider` and optionally `audio.CapabilitiesProvider`:
```go
type CapabilitiesProvider interface {
Capabilities() ProviderCapabilities
}
type ProviderCapabilities struct {
Params []ParamSchema
CustomFeatures CustomFeatures
}
```
`ParamSchema` carries `Key`, `Type` (string/number/bool/select/textarea), `Label`, `Help`, `Default`,
`Min`/`Max`, `Options` (for select), `DependsOn` (all-of conditions), and `Hidden` flag.
The UI reads `GET /v1/tts/capabilities` and renders per-provider param editors dynamically.
See `docs/tts-provider-capabilities.md` for full schema documentation.
### Providers
| Provider | Package | Voice source | Key param |
|----------|---------|--------------|-----------|
| OpenAI | `internal/audio/openai/` | Static list | `speed`, `response_format`, `instructions` |
| ElevenLabs | `internal/audio/elevenlabs/` | Dynamic (API) | `voiceSettings.*`, `seed`, `language_code` |
| Edge TTS | `internal/audio/edge/` | Static list | `rate`, `pitch`, `volume` |
| MiniMax | `internal/audio/minimax/` | Dynamic (API) | `speed`, `vol`, `pitch`, `emotion`, `audio.*` |
| Gemini | `internal/audio/gemini/` | Static list (30) | multi-speaker, audio tags, 70+ languages |
### Storage: Dual-Read
Tenant TTS config is stored in `system_config` with two complementary strategies:
- **Legacy flat keys** (`tts.provider`, `tts.voice_id`, `tts.api_key`, …) — written by all versions.
- **Params blob** (`tts.<provider>.params` JSON) — written by v3.x+ for per-provider param overrides.
On read, both paths are merged; blob wins on key conflict. This allows gradual migration:
old clients see flat keys; new clients see the full params blob.
### Manager
`audio.Manager` is the central registry:
- `RegisterProvider(p TTSProvider)` — replaces any existing provider by the same name.
- `Primary()` — returns current primary provider name from config.
- `Synthesize(ctx, text, opts)` — delegates to active provider.
- `ListVoices(ctx, opts)` — delegates to `VoiceListProvider` if implemented.
### HTTP Endpoints
| Endpoint | Role | Notes |
|----------|------|-------|
| `POST /v1/tts/synthesize` | operator | Streams WAV response |
| `POST /v1/tts/test-connection` | operator | Ephemeral provider, returns latency |
| `GET /v1/tts/capabilities` | operator | ProviderCapabilities JSON per provider |
| `GET /v1/tts/config` | admin | Tenant TTS config |
| `POST /v1/tts/config` | admin | Save tenant TTS config |
### Gemini Specifics
- Models: `gemini-3.1-flash-tts-preview` (default), `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts` (preview).
- Multi-speaker: up to 2 simultaneous speakers, each with distinct voice + name annotation.
- Audio tags: inline `<say-as>` / style directives via bracketed prompts.
- Sentinel errors: `ErrInvalidVoice`, `ErrInvalidModel`, `ErrSpeakerLimit` → HTTP 422 with i18n message.
- SSRF guard: `api_base` override validated by `validateProviderURL()` — blocks 127.0.0.1/localhost.
### i18n
Backend validation errors use `i18n.T(locale, key, args...)` pattern.
Locale is extracted from `Accept-Language` HTTP header by `enrichContext` middleware.
UI param labels/help text live in:
- `ui/web/src/i18n/locales/{en,vi,zh}/tts.json`
- `ui/desktop/frontend/src/i18n/locales/{en,vi,zh}/tts.json`
Parity enforced by `ui/web/src/__tests__/i18n-tts-key-parity.test.ts` (vitest).
---
<<<<<<< HEAD
## Image Generation
Native `image_generation` support in the Codex provider (`POST /codex/responses`) + passthrough in the OpenAI-compat path.
**Provider flag:** `ProviderCapabilities.ImageGeneration bool` (`internal/providers/capabilities.go`). Codex sets `true`; other providers default `false`.
**Gate (agent loop):** `ToolDefinition{Type:"image_generation"}` appended iff (provider capability) AND (`AgentConfig.AllowImageGeneration`, default true) AND (request lacks `x-goclaw-no-image-gen` header). Gate logic in `internal/agent/loop_tool_filter.go`.
**Codex native events** (`internal/providers/codex.go`):
- `response.image_generation_call.partial_image` → `ChatResponse.Images` entry with `Partial:true`.
- `response.output_item.done` with `item.type == "image_generation_call"` → final `ChatResponse.Images` entry; partial frames for same `item_id` replaced.
- `response.completed` walks `response.output[]` for image items (non-stream).
**OpenAI-compat parsing:** `choices[0].message.images[]` + `choices[0].delta.images[]` with `data:image/...;base64,...` URLs decoded in `internal/providers/openai_http.go` and `internal/providers/openai_chat.go`. Helper: `parseDataURL()` in `internal/providers/openai_image_url.go`.
**Persistence:** `internal/agent/media.go persistAssistantImages()` writes final images to `{workspace}/media/{sha256}.{ext}`, returns `MediaRef` entries, clears inline `Images[]`. Idempotent on hash. Invoked from `pipeline.FinalizeStage` via `Deps.PersistAssistantImages` callback.
**Web UI:** Download filename resolver (`imageGenDownloadName`) in `ui/web/src/components/chat/media-gallery.tsx`. Image generation works automatically when the agent has the `create_image` tool — no user-facing toggle.
=======
## Webhook Subsystem
External systems invoke agents or send channel messages via webhooks without gateway tokens.
### Components
| Path | Purpose |
|------|---------|
| `internal/http/webhooks_admin.go` | CRUD handlers (create, list, get, patch, rotate, revoke) |
| `internal/http/webhooks_auth.go` | Bearer + HMAC signature verification, IPAllowlist, tenant scope |
| `internal/http/webhooks_nonce.go` | Per-process HMAC replay cache (320s TTL) |
| `internal/http/webhooks_llm.go` | `POST /v1/webhooks/llm` endpoint (sync 30s / async) |
| `internal/http/webhooks_message.go` | `POST /v1/webhooks/message` endpoint (channel delivery) |
| `internal/http/webhooks_ratelimit.go` | Per-webhook + per-tenant rate limiting |
| `internal/http/webhooks_idempotency.go` | `Idempotency-Key` header dedup cache (24h TTL) |
| `internal/http/webhooks_media_fetch.go` | SSRF-guarded media URL fetch + MIME validation |
| `internal/webhooks/worker.go` | Async callback poller + delivery goroutines |
| `internal/webhooks/backoff.go` | Exponential retry schedule `[30s, 2m, 10m, 1h, 6h]` |
| `internal/webhooks/sign.go` | HMAC-SHA256 signing for outbound callbacks |
| `internal/webhooks/limiter.go` | Shared rate limiter for callback delivery |
| `internal/store/webhook_store.go` | `WebhookStore` interface + `WebhookCallStore` |
| `internal/store/pg/webhook_store.go` | PostgreSQL implementation (tenant-scoped) |
| `internal/store/sqlitestore/webhook_store.go` | SQLite implementation (Lite edition) |
| `migrations/` | PG migrations 000056–000058 (webhooks + lease token + encrypted secret) |
### Auth Flow
1. **Bearer auth**: Hash the token, lookup `secret_hash` globally (via `GetByHashUnscoped`) → return webhook + tenantID.
2. **HMAC auth**: Parse `X-Webhook-Id` header, lookup webhook globally → verify signature timestamp + nonce.
3. **Tenant inject**: Re-scope context with webhook's tenantID for all downstream calls.
4. **IP allowlist**: If non-empty, check request source IP (CIDR or exact) against list. Empty = allow all.
5. **Rate limit**: Check per-webhook + per-tenant buckets. Either rejects = 429.
### Idempotency & Lease Tokens
- **Inbound**: `Idempotency-Key` header dedup (24h cache). Same key + same body = cached response; same key + different body = 409 Conflict.
- **Outbound**: Each `webhook_calls` row has `lease_token` (UUID). Worker claims row with CAS. On update, token proves ownership — prevents stale receivers from overwriting.
### Secret Encryption
Raw webhook secret encrypted at rest via AES-256-GCM using `GOCLAW_ENCRYPTION_KEY` (same as LLM provider keys).
- Database: stores `encrypted_secret` column + `secret_hash` (for bearer lookups).
- DB compromise does not leak HMAC material.
- Clients receive plaintext secret once (create/rotate response) — must store securely.
### Audit Payload
All webhook calls logged with canonical `{"body_hash":"<sha256-hex>","meta":{...}}` shape in `webhook_calls.request_payload` (JSON).
Used by idempotency checker to detect body mismatches on replay.
>>>>>>> a83f4090 (fix(webhooks): address post-review findings (K1-K10))
---
## Key Conventions
- **Store layer:** Interface-based; PG (`store/pg/`) + SQLite (`store/sqlitestore/`). Raw SQL, `$1/$2` params.
- **Session token display:** v3 compaction now uses dynamic max_tokens (`in/25` clamped `[1024,8192]`); session token display reads from `sessions.metadata.last_prompt_tokens` and `last_message_count`. Tool schemas counted via `TokenCounter.CountToolSchemas()` and included in ContextStage overhead.
- **Context propagation:** `store.WithLocale`, `store.WithUserID`, `store.WithTenantID`, etc.
- **Security logs:** `slog.Warn("security.*")` for all security events.
- **SSRF prevention:** `validateProviderURL()` in `internal/http/tts_validate.go`.
- **i18n keys:** Add to `internal/i18n/keys.go` + 3 catalog Go files; UI strings in 3 locale JSON dirs.
- **Migrations:** PG (`migrations/`) + SQLite (`store/sqlitestore/schema.go`) — always update both.
Reference in New Issue View Git Blame Copy Permalink