mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-09-04 16:17:08 +00:00
cf16cf53dbbf7aaa8592eb5dfd8a178e059185f3
282
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 viettranx
|
a7c8170c4a |
feat(agent): audio config context propagation in tool callbacks
Add audio config context helpers in pipeline callbacks. Propagate voice/model selections through agent loop resolver for tool invocation. |
||
|
viettranx
|
28a70fd4f0 |
feat(tools): TTS voice/model resolver with agent-level precedence
Add context helpers for audio config routing. Implement resolveVoiceAndModel with agent→user→global precedence. Add 5 precedence tests covering inheritance and override scenarios. |
||
|
therichardngai-codeandGitHub
|
e79a8bbd39 |
fix(mcp): wire per-user MCP tool discovery into agent pipeline
MCP servers with require_user_credentials (e.g. Notion) were defined
but never loaded into the agent's tool registry. Three gaps:
1. getUserMCPTools was defined but never called — add call in
makeBuildFilteredTools before FilterTools runs each iteration.
2. hasMCPTools stayed false when only user-credential servers existed,
so agentToolPolicyWithMCP never injected "group:mcp" into alsoAllow.
Now set true when mcpUserCredSrvs is non-empty.
3. Per-user BridgeTools were registered in the registry but never added
to the "mcp" tool group, so expandSpec("group:mcp") returned empty.
Add MergeToolGroup helper for additive group updates.
Also add debug log when getUserMCPTools skips due to empty userID.
|
||
|
viettranx
|
1ac08155b0 |
feat(trace): reliable stop/abort with ctx-aware streams and 2-phase router
Makes the Stop button on the traces page actually stop running traces. Seven-phase implementation across provider HTTP, agent router, trace persistence, WS events, tool exec, i18n, and integration tests. - Provider HTTP+SSE ctx-aware: close socket on cancel via CtxBody wrapper - Router 2-phase abort: CAS state machine, 3s grace, force-mark fallback - Trace retry: 3 inline retries + 10-max retry queue, stale recovery 10min - trace.status WS event: real-time UI updates (invalidates query on receive) - Tool exec: process-group kill (SIGTERM→3s→SIGKILL), Rod page ctx watch - i18n: 6 abort toast variants in en/vi/zh - Integration: 9 scenarios, -race clean Fixes tenant-ctx loss in forceMarkTraceAborted and retry worker broadcast (caught by code-reviewer: C1/C2). Stale threshold intentionally 10min because start_time-based; last_span_at migration is a follow-up. |
||
|
viettranx
|
221cd78fcf |
fix(agent): emit tool.call event in parallel tool execution path
v3 pipeline's parallel path (makeExecuteToolRaw) skipped the tool.call WebSocket event, so web UI and desktop UI silently dropped tool cards during real-time streaming. Only page refresh (which reloads history) revealed the tool calls. Both UIs rely on tool.call to create entries that later tool.result events can update. Fix: mirror the sequential path's emission in makeExecuteToolRaw. Bus.Broadcast is RWMutex-guarded, safe to call from parallel goroutines. Add tests at two layers to prevent regression: - Pipeline layer (stages_test.go): guards the dispatch contract — multiple tool calls route through ExecuteToolRaw + ProcessToolResult rather than ExecuteToolCall. Previously the parallel path had zero test coverage, which is why this bug escaped. - Agent layer (loop_pipeline_tool_callbacks_test.go): guards the emission contract — both sequential and parallel wrappers emit tool.call with correct payload and routing context. Mutation-verified. |
||
|
viettranx
|
21cc208813 |
fix(agent): use tiktoken for context pruning and protect media tool results
- Replace char-based heuristic (chars/4) with tiktoken BPE for accurate token counting, especially for non-ASCII content (Vietnamese/Chinese) - Add pruningEstimator wrapper with tiktoken/fallback dual-path - Raise default soft trim budget from 3K to 6K chars (3K head + 3K tail) - Media tools (read_image, read_document, read_audio, read_video) get higher soft trim budget (8K: 4K head + 4K tail) and skip hard clear entirely — their vision/audio descriptions are irreplaceable - Add per-result context guard (Pass 0): force-trim any single tool result exceeding 30% of context window |
||
|
viettranx
|
5dc696e3c6 |
fix(agent): preserve up to 30 media refs during history compaction
Both v3 mid-loop compaction and v2 background summarization were dropping MediaRefs when summarizing old messages, making previously shared images/documents permanently inaccessible to the agent. Now collect up to 30 most recent MediaRefs from compacted messages and attach them to the summary/first-kept message so they survive the compaction cycle. |
||
|
viettranx
|
70697988dd |
fix(memory): auto-inject honors share_memory setting for episodic search
When share_memory=true, memory_search tool used MemoryUserID(ctx)="" (cross-user) but auto-inject still passed raw userID, limiting episodic L0 injection to the current user only. Now uses store.MemoryUserID(ctx) so shared-memory agents get cross-user episodic summaries in auto-inject. |
||
|
viettranx
|
a4df5a08e3 |
fix(security): prevent cross-group session data leak in cron jobs
Group-scoped agents could read sessions from other groups via session tools (sessions_list, sessions_history, session_status, sessions_send) because they only checked agent_key, not group context. This caused cron jobs to leak data from unrelated groups into reports. Add isSessionInScope() guard to all 4 session tools with colon-bounded chatID matching. New share_sessions setting (default false) controls cross-group visibility, following the same pattern as share_memory and share_knowledge_graph. Web UI toggle and i18n strings included. 63 test cases covering guild/DM/group users, realistic Zalo IDs, boundary exactness, multi-colon chatIDs, and the exact bug scenario. |
||
|
viettranx
|
4b658a2304 |
feat(tools): tenant-scoped allowed_paths configuration
- Add tenant-level filesystem path restrictions via system_configs table - Merge tenant paths with global skills directories in allowedWithTeamWorkspace() - Propagate tenant paths to subagents via RunContext - Seed allowed_paths from config.json to system_configs on startup - Fix TestStoreTask_RaceToClaimSameTask: use composite PK for team members |
||
|
viettranx
|
02fe3e143e |
fix(agent): remove hardcoded default timezone, ask on-demand instead
Instead of assuming Asia/Saigon or UTC for all users, the model now asks for timezone when the user mentions times/schedules/reminders. After first ask, timezone is stored in USER.md for future sessions. Closes #833 discussion. |
||
|
|
ad893908a5 |
fix(telegram): propagate local_key for forum topic routing in team notifications (#800)
* fix(telegram): propagate local_key for forum topic routing in team notifications
Team task status messages (dispatched, completed, progress) were always
delivered to the General topic in Telegram forum groups because the
notification pipeline had no access to the originating topic's local_key.
Root cause: wireTeamProgressNotifySubscriber in gateway_events.go published
OutboundMessage with no Metadata, so the Telegram adapter had no
message_thread_id to route to the correct forum topic.
Fix has two parts:
1. Team notify path (root cause):
- Add LocalKey field to TeamTaskEventPayload (protocol)
- Extract local_key from tool context in WithContextInfo()
- Add LocalKey to NotifyRoutingMeta
- Pass LocalKey through to OutboundMessage Metadata in both
leader mode (InboundMessage) and direct mode (OutboundMessage)
2. MCP bridge context (supporting):
- Propagate local_key and session_key through bridge HTTP headers
- Add X-Local-Key and X-Session-Key to BridgeContext
- Extract and inject into tool context in gateway middleware
- Include in HMAC signature for integrity
Closes #798
* fix(telegram): pass LocalKey from task metadata in all dispatch/fail broadcast sites
The initial fix added LocalKey to the event payload and WithContextInfo(),
but 4 broadcast call sites use individual With* options instead of
WithContextInfo — so LocalKey was never populated for:
- fallback_dispatch (team_tasks_create.go)
- dispatch_unblocked (team_tool_dispatch.go)
- post_turn dispatch (team_tool_validation.go)
- blocker/fail (team_tasks_blocker.go)
Add WithLocalKey() option function and extract TaskMetaLocalKey from task
metadata at each site, matching the existing TaskMetaPeerKind pattern.
* test(mcp): add HMAC verification tests for extra params (localKey, sessionKey)
- Add tests for SignBridgeContext/VerifyBridgeContext with extra params
- Test backward compat fallback for pre-localKey sessions
- Test that param order matters in signature
- Add clarifying comment for routing context injection security model
---------
Co-authored-by: Jens Henke <jens@henke.dk>
Co-authored-by: viettranx <viettranx@gmail.com>
|
||
|
viettranx
|
56eb686934 |
feat(agent): tenant tool settings overlay via loop ctx injection
Plumb per-tenant tool settings into the agent Loop without touching any tool's Execute signature. Adds WithTenantToolSettings ctx helper and rewrites BuiltinToolSettingsFromCtx with fast-path merge semantics — tenant layer wins over global defaults at tool-name level (no field-level deep merge). Resolver preloads ListAllSettings for the agent's tenant at Loop construction; store errors log + fall back to global. Zero allocs on single-tier reads. Tier 1 (future per-agent override) is reserved and documented in context_keys.go. 8 unit tests cover empty / single-tier / both-merged / RunContext fallback / fast-path semantics. |
||
|
viettranx
|
d77a3664db |
fix(cache): tenant-aware invalidation for builtin tools and skills
Tenant config changes for builtin tools and skills silently failed to invalidate cached agent Loops, leaving tenants stuck on stale tool/skill sets until the 10-minute TTL expired or an unrelated event wiped the cache. Master-level skill CRUD had the same gap in the opposite direction. - Add CacheInvalidatePayload.TenantID so events can scope to one tenant - Add Router.InvalidateTenant(tenantID) with prefix match on "tenantID:agentKey" cache keys; uuid.Nil is a no-op - Rework emitCacheInvalidate helpers in builtin_tools + skills handlers to carry tenant scope; add defense-in-depth uuid.Nil guards in four tenant-config handlers - Update TopicCacheBuiltinTools + TopicCacheSkills subscribers to branch on payload.TenantID (tenant event wipes that tenant, global event keeps the existing InvalidateAll path) - Wire emitCacheInvalidate into master skill CRUD paths (update, delete, toggle, upload, install-deps, rescan-deps, import) that previously only called BumpVersion - Document the system-owner bypass in requireTenantAdmin so handlers keep guarding uuid.Nil themselves |
||
|
viettranx
|
e39e97ee1b |
test(tasks,agent): cover TaskTicker and agent helpers
Add lifecycle and utility function tests: - tasks/task_ticker_test.go: TaskTicker lifecycle, recoverAll, followup - agent/pruning_test.go: resolvePruningSettings, findAssistantCutoff, takeHead/Tail - agent/extractive_memory_test.go: ExtractiveMemoryFallback, dedup - agent/intent_classify_test.go: quickClassify, containsWholeWord, ClassifyIntent - agent/loop_utils_test.go: uniquifyToolCallIDs, shouldShareKG, InvalidateUserWorkspace - agent/inject_and_misc_test.go: truncateForLog, processInjectedMessage, drainInjectChannel |
||
|
viettranx
|
e62c027b7d |
feat(vault): task + delegation auto-linking in enrich worker
- Add enrich_auto_linking.go: deterministic auto-link logic for task/delegation contexts - Add team_task_siblings.go (PG + SQLite): find all tasks in same delegation for batch link - Update enrich_worker.go: call enrich_auto_linking Phase 2 hook + skip re-embed for binary - Update team_store.go: add TaskSiblings interface method + TeamStore binding - Add context_keys.go + test: define ContextDelegationIDKey + ContextTaskIDKey for propagation - Update vault_interceptor.go: extract + inject delegation ID from request - Update run_context.go: propagate DelegationID from request - Update loop_context.go: inject DelegationID from run context - Update teams_tasks.go: DeleteTask(s) → cleanup auto-links via vault_source_cleanup - Update teams_tasks_activity.go: DetachFileFromTask → cleanup related auto-links - Update vault_documents_enrichment.go: populate summary from media_summary - Update cmd/gateway.go: wire TeamStore to handler bootstrap |
||
|
viettranx
|
4b27a337d1 |
test(router): pin stale raw-UUID entry eviction on Get
Pin that a pre-hardening fragmented entry written under the raw UUID cache key (tenantID:<uuidStr>) is still evicted by the TTL branch in Router.Get when a UUID-form caller arrives after TTL expiry. The test synthesizes the fragmented entry directly via a test-only map write, then asserts the subsequent Get evicts the raw-UUID entry, re-invokes the resolver once, and writes the canonical tenantID:agentKey entry — leaving no fragmented entries behind. |
||
|
viettranx
|
7799a9c88a |
fix(router): evict stale canonical entry on double-check TTL miss
Router.Get's canonical double-check branch trusted any existing entry under the canonical key without re-checking TTL. If an earlier agent_key caller wrote the entry and the TTL expired, a later UUID-form caller would resolve fresh, hit the canonical branch, find the stale entry, and return it — indefinitely — because the raw-UUID key was never the map key and the initial-miss eviction branch did nothing. Re-check TTL inside the double-check branch and evict+rewrite when stale. Regression test primes a canonical entry with cachedAt set 2×TTL in the past, then asserts a UUID-form Get re-invokes the resolver and the returned agent reflects the fresh resolver output. |
||
|
viettranx
|
6e86ba9a39 |
fix(agent): use agent_key in v3 workspace path resolver
The v3 workspace resolver block in injectContext was passing l.agentUUID.String() as ResolveParams.AgentID, but the resolver uses this value as a filesystem path segment. The v2 path in loop_pipeline_callbacks.go and the session_key anchor both use agent_key, so a UUID-based v3 path would produce a parallel filesystem for the same logical agent the moment any consumer reads the v3 workspace context. No active bug today because workspace.FromContext has zero consumers, but this closes the landmine before the v3 wiring lands. |
||
|
viettranx
|
c8ebe9a789 |
refactor(comments): remove plan/phase refs from agent identity hardening code
Rewrite inline comments added during the agent identity hardening so they explain the code as it stands today, rather than tying to internal plan terminology (phase numbers, FR/NFR/H/M/C codes, PR references, trap zone labels). Commit history already carries the plan archaeology. Comments now keep the non-obvious invariants (cache boundaries, bypass gaps, silent-nil traps, dual-tenant semantics) and drop the scaffolding. Comment-only — no runtime behavior change. |
||
|
viettranx
|
ca7f7b4f24 |
docs(code): add inline comments at agent identity trap zones
Clarify the agent_key vs UUID contract inline at the four highest-ROI trap zones: the Loop struct identity fields, DomainEvent identity fields, the parseUUID/parseUUIDOrNil store helpers, and the WS method resolver helpers. Each comment explains the invariant and links to docs/agent-identity-conventions.md so future readers can find the full rules without archaeology. Comments only — no runtime behavior change. |
||
|
viettranx
|
6c17aad8d7 |
feat(gateway/methods): add cache-aware resolveAgentUUIDCached helper
Three-part addition so WS handlers can accept agent_key input without a DB roundtrip on every request: 1. Router.GetCached — lock-free TTL cache read, no resolver fallback. 2. Loop.UUID() — canonical UUID accessor (agent_key stays on ID()). 3. resolveAgentUUIDCached — fast-path checks router cache via an agentUUIDProvider interface, falls back to resolveAgentUUID on miss or when input is a UUID string (un-cached post-canonicalization). Helper delegates to pure DB lookup when router is nil, so handlers with no router dependency still work. Tests cover nil-router, cache-miss, and UUID-input paths — cache-hit coverage is exercised via downstream handler integration tests. Phase 3 foundation (TD-2). |
||
|
viettranx
|
c599387248 |
fix(router): canonicalize cache key, exact-segment match, tenant-aware IsRunning
Three related fixes to agent router identity handling: 1. Canonicalize on resolve — Router.Get now stores entries under the canonical tenantID:agent_key after a successful resolver call, regardless of whether the caller passed a UUID or agent_key. Prevents fragmentation where the same logical agent would occupy two cache slots. Callers that pass the UUID form are now un-cached and resolve on every call; all production callers pass agent_key today. 2. Exact-segment match — matchAgentCacheKey helper replaces HasSuffix in Router.Remove and Router.InvalidateAgent. Prevents substring collisions like "tenantX:sub-foo" being wiped when invalidating "foo". Rejects empty agentKey to guard against wildcard wipes. 3. Tenant-scoped IsRunning (C6) — Router.IsRunning now accepts ctx and uses agentCacheKey for lookup. Pre-fix, the bare r.agents[agentID] lookup always returned false in tenant-scoped deployments, causing agents.list to report every live agent as idle. Updates the sole caller at gateway/methods/agents.go:134. Phase 2 of agent identity hardening (TD-3). |
||
|
viettranx
|
89862981e6 |
fix(agent): set AgentUUID in memoryflush SystemPromptConfig
memoryflush.go constructed SystemPromptConfig without AgentUUID, while loop_history.go set both AgentID (agent_key) and AgentUUID. buildRuntimeSection renders below the cache boundary so the omission did not bust the prompt cache — but the runtime identity line dropped the UUID during compaction flush turns. Extract a pure helper buildMemoryFlushPromptConfig so tests can assert the config shape without a full Loop fixture (M7 mitigation from red-team review). Phase 1 Fix A of agent identity hardening. |
||
|
|
9dee3b3d26 |
fix(agent): use agentUUID instead of agent key in DomainEvent for epi… (#826)
* fix(agent): use agentUUID instead of agent key in DomainEvent for episodic worker DomainEvent.AgentID was set to l.id (agent_key string like "goctech-leader") instead of l.agentUUID.String() (UUID). This caused episodic worker to fail with "invalid input syntax for type uuid" on every session.completed event, breaking the entire 3-tier memory consolidation pipeline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(consolidation): parse tenant/agent IDs safely in episodic worker Replace uuid.MustParse with uuid.Parse so a malformed tenant_id or agent_id in a DomainEvent no longer panics the worker goroutine. Parse up front and return a descriptive error before touching the store, so bad IDs surface with context instead of leaking into a raw PG error. Add regression tests asserting Handle rejects non-UUID agent_id and tenant_id with clear errors and skips creation — guards against the l.id vs l.agentUUID mix-up fixed in the prior commit. --------- Co-authored-by: tuannt23065 <tuannt23065@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: viettranx <viettranx@gmail.com> |
||
|
Kai (Tam Nhu) TranandGitHub
|
86ef70906a |
fix(mcp): full reconnect for SSE/HTTP after server-side restart (#812)
* fix(mcp): full reconnect for SSE/HTTP after server-side restart When an MCP server using SSE or streamable-http transport restarts (container redeploy, crash, OOM), the old client holds a stale session ID. tryReconnect only called Ping() on the dead client, which keeps POSTing to /message?sessionId=<dead> — the server returns 404, and after maxReconnectAttempts the connection is permanently dead. For pool connections, poolHealthLoop only set connected=false without attempting any reconnect at all. Fix: - Store connection params (url, headers, command, args, env) in serverState so tryReconnect can create fresh connections. - tryReconnect now has two phases: fast path (ping existing client for transient blips) and slow path (close old client, create fresh one via createClient + Start + Initialize, swap ss.client). - Add updateBridgeToolClients to propagate the new client pointer to all registered BridgeTools after a full reconnect. - Add poolTryReconnect with the same two-phase pattern for pool-managed connections. - Add BridgeTool.swapClient for safe client pointer replacement. Closes #810 * fix(mcp): address review findings — atomic client pointer, close-after-swap, cooldown Red-team review found critical issues in the initial reconnect fix: 1. Data race: BridgeTool.client written by healthLoop, read by Execute concurrently without synchronization. Fix: BridgeTool now holds *atomic.Pointer[mcpclient.Client] shared with serverState.clientPtr. Execute uses atomic Load(), reconnect uses atomic Store(). 2. Close-before-swap: old client was closed before new one was ready. If createClient failed, ss.client pointed to closed client permanently. Fix: create and validate new client first, swap atomically, then close old. 3. Pool BridgeTool orphaning: poolTryReconnect swapped ss.client but existing BridgeTools held old pointer with no update mechanism. Fix: atomic.Pointer propagates automatically — no explicit update needed. 4. Permanent death: after maxReconnectAttempts, server was permanently dead with no recovery. Fix: 5-minute cooldown then reset attempts. 5. healthFailures not reset in fast path: minor inconsistency fixed. Also fixes external caller in loop_mcp_user.go and adds ClientPtr() accessor to poolEntry for atomic pointer sharing. * refactor(mcp): extract fullReconnect helper, add nil guard, clarify dual-pointer design - Extract shared fullReconnect() used by both tryReconnect and poolTryReconnect — eliminates 30-line duplication (-17 lines net) - Add nil guard on clientPtr.Load() in BridgeTool.Execute to prevent nil deref in edge cases (test fixtures, initialization race) - Add doc comment on serverState explaining dual-pointer design (client for healthLoop, clientPtr for BridgeTools) |
||
|
viettranx
|
9f77bfe711 |
feat(vault): tenant-wide rescan with nullable agent_id + media preview
Vault rescan redesigned from per-agent to tenant-wide:
- POST /v1/vault/rescan replaces POST /v1/agents/{id}/vault/rescan
- agent_id nullable in vault_documents (PG migration 046 + SQLite v14)
- Path-based scope inference: agents/{key}/ → personal, teams/{uuid}/ → team, root → shared
- Interceptor sets agent_id=NULL for team-scoped file writes
- Enrichment worker batch key handles empty agent_id
- web-fetch/ directory excluded from vault scan at any depth
- Media preview: images render via authenticated blob URL, binary files show metadata
- Scan button no longer requires agent selection
|
||
|
viettranx
|
4cf66eb379 |
feat(ts-port): reasoning strip, dreaming config + weighted scoring
Phase 6 — Reasoning token stripping: - ReasoningDecision.StripThinking auto-flags Kimi + DeepSeek-Reasoner - Guard clauses in Anthropic/OpenAI/Codex stream handlers - Usage.ThinkingTokens + RawAssistantContent preserved (billing + tool passback safe) Phase 8 — Per-agent dreaming config: - MemoryConfig.Dreaming JSONB (no migration), resolver callback pattern - Enabled/DebounceMs/Threshold/VerboseLog fields with partial-override merge - ConsolidationDeps gains optional AgentStore Phase 10 — Dreaming weighted scoring: - Migration 000045 adds recall_count/recall_score/last_recalled_at on episodic_summaries - ComputeRecallScore 4-component formula (freq/rel/recency/freshness, 14d half-life) - memory_search fire-and-forget RecordRecall; ListUnpromotedScored in DreamingWorker - Bootstrap-friendly filter: unrecalled entries bypass thresholds - Debounce stamped on filter-empty skip to prevent starvation loop Phase 5 follow-up — last_compaction_at in sessions.metadata JSONB: - v3 PruneStage.CompactMessages and v2 maybeSummarize both stamp timestamp - Zero migration; exported const SessionMetaKeyLastCompactionAt RequiredSchemaVersion: 44 → 45 (PG), SchemaVersion: 12 → 13 (SQLite). 27 new tests; builds pass under PG and sqliteonly tags. |
||
|
viettranx
|
dabb1eaa11 |
feat(vault): workspace rescan endpoint with symlink-safe walker
Add POST /v1/agents/{agentID}/vault/rescan to backfill vault_documents
from filesystem. Walks agent workspace, registers missing/changed files,
publishes enrichment events for async summarize + embed + link classify.
Backend:
- SafeWalkWorkspace: symlink-safe walker with exclusion patterns,
resource limits (5K files, 500MB, 50MB/file), context deadline
- RescanWorkspace: hash-based dedup, path-based scope detection
(teams/{id}/ → team scope), idempotent via UpsertDocument ON CONFLICT
- Per-agent mutex (409 Conflict for concurrent rescans)
- Tenant-scoped workspace resolution (config.TenantWorkspace)
- Enrichment worker: semaphore-based parallel summarize (max 3 concurrent)
- Media files without summary skip LLM summarize, embed title+path only
- DRY: export InferTitle/InferDocType from vault pkg, remove from interceptor
- Auto-register text uploads in vault via onTextUploaded callback
Frontend:
- Rescan button in vault page header (FolderSync icon)
- Toast with result counts, 409 warning, error handling
- i18n strings for en/vi/zh
Security: skip all symlinks, boundary checks, per-file size limit,
tenant isolation via server-side workspace resolution.
|
||
|
viettranx
|
8d37dc45ea |
feat(pipeline): per-provider context window, cache/tenant/pipeline hardening
Ship Group A (bug fixes) + Group B (pipeline enhancements) from plans/260410-1009-openclaw-ts-feature-port — five changes that share cmd/ wiring and pipeline plumbing so they commit as a unit. cache: InMemoryCache now supports periodic sweep + max-size cap via variadic options. PermissionCache wires 60s sweep + 10k entry cap + Close() hook in gateway shutdown so long-running gateways don't leak per-user permission entries. Backward compatible for zero-arg callers. store: ContactCollector.seen cache key now includes tenant_id + channel_ instance so the same sender in different tenants (or different bot instances in the same tenant) no longer silently skip upserts against each other. Zero-tenant (Desktop) behaviour preserved. pipeline: EffectiveContextWindow is resolved once per run in ContextStage via a ResolveContextWindow callback (backed by providers.ModelRegistry) so PruneStage bills history against the actual model window instead of a stale static config. Nil resolver / unknown model fall back to Config.ContextWindow for backward compatibility. Locked to the model observed at context build time to prevent mid-run budget drift. pipeline: PipelineConfig.ReserveTokens carves out an optional safety buffer subtracted from the history budget so compaction fires slightly before the hard limit — protects against provider over-delivery and token counter drift on streaming responses. Zero (default) preserves legacy budget math. agent: ModelRegistry flows gateway → ResolverDeps → LoopConfig → Loop → pipeline adapter so resolver can look up per-model capabilities at run time without re-touching gateway internals. 20 regression tests across cache, contact collector, and pipeline cover the critical paths: cross-tenant isolation, cache sweep + eviction + Close idempotency, per-model window override + fallback, and reserve token buffer behaviour. All passing with -race on both go build ./... and go build -tags sqliteonly ./.... |
||
|
viettranx
|
6d43de4169 |
fix(evolution): fix 7 bugs in evolution flow — guardrails, cron, dedup, dead types
- Remove broken delta comparison in CheckGuardrails (compared usage rate vs max delta) - Replace hardcoded +0.05 threshold bump with configurable MaxDeltaPerCycle, cap at 0.95 - Remove dead MetricFeedback and SuggestMemoryPrune constants + UI references - Make SuggestToolOrder approval actionable — disables tool via BuiltinToolTenantConfigStore - Replace time.Ticker(24h) with wall-clock scheduling (3 AM daily, Sundays weekly eval) - Add PG advisory lock on pinned connection for multi-instance safety - Add EvolutionSuggest flag check in cron (metrics-only agents skip suggestion analysis) - Change suggestion dedup from type-only to (type, metric_key) composite key |
||
|
viettranx
|
bb365a66c4 |
refactor(vault): remove vault_link/vault_backlinks tools, auto-sync wikilinks in enrichment pipeline
- Remove vault_link and vault_backlinks builtin tools (replaced by auto-linking)
- Add DeleteDocLinksByType to VaultStore interface (PG + SQLite) to selectively
delete links by type without destroying semantic links
- Integrate SyncDocLinks into enrichment worker: [[wikilinks]] are now
auto-extracted from document content and synced as vault links on every write
- Fix SyncDocLinks to use DeleteDocLinksByType("wikilink") instead of
DeleteDocLinks which was deleting all link types including semantic
- Add missing coreToolSummaries for delegate, memory_expand, vault_search
(previously showed as "(custom tool)" in system prompt)
- Increase enrichSimilarityLimit from 5 to 10 for richer auto-linking
|
||
|
Plateau NguyenandGitHub
|
fb8afd41bf |
feat(channels): add Facebook Messenger and Pancake channel integrations (#731)
Add two new channel implementations for Facebook Fanpage (comment + Messenger auto-reply, first inbox DM) and Pancake/pages.fm (multi-platform inbox via Facebook, Zalo, Instagram, TikTok, WhatsApp, LINE). Key features: - Facebook: comment auto-reply, Messenger auto-reply, first inbox DM, HMAC-SHA256 webhook verification, multi-page webhook routing - Pancake: multi-platform inbox, outbound echo dedup with HTML normalization, race-condition-safe echo fingerprinting, platform-aware formatting - Bootstrap skip: pre-fill USER.md from channel metadata (Pancake) - SanitizeDisplayName across all channels (defense in depth) Code audit fixes: - Fix truncateForTikTok byte→rune slicing (UTF-8 corruption) - Fix empty message.ID shared dedup slot (silent message loss) - Fix DisplayName markdown injection in buildPrefilledUser - Consolidate duplicate ChannelMeta type (agent→bootstrap) - Compile-time interface assertions, alphabetical type constants - Per-message logs demoted to slog.Debug, errors.As for wrapped errors - UI: alphabetical channel ordering, complete config schemas - Remove deprecated WhatsApp bridge_url, fix nested error parsing |
||
|
|
8f56ddaa64 |
feat(v3): core architecture redesign — pipeline, memory, vault, evolution, providers, orchestration (#790)
* feat(v3): add core interface contracts and migration for v3 redesign
Foundation interfaces: TokenCounter, WorkspaceContext, DomainEventBus,
ProviderAdapter/Capabilities. Pipeline: Stage, RunState, MessageBuffer,
substates, Pipeline orchestrator. Memory: EpisodicStore, AutoInjector,
KG temporal extensions, consolidation workers. System integration:
PromptConfig, ToolCapability, Retriever. Orchestration: OrchestrationMode,
EvolutionMetrics/SuggestionStore. Migration 000037: episodic_summaries,
evolution tables, KG temporal columns. Schema version 36→37.
* refactor(plans): mark all v3 design phases complete with file references
* fix(v3): address code review findings on design contracts
- C1: add missing l0_abstract column to episodic_summaries migration
- C2: align EpisodicSummary ID/TenantID/AgentID to uuid.UUID
- H1: document tenant_id scoping requirement on EpisodicStore
- H2: add UNIQUE constraint on (agent_id, user_id, source_id) for dedup
- H4: clarify ProviderAdapter vs Provider relationship in doc
- M3: set state.ExitCode on BreakLoop/AbortRun in pipeline
- M6: store full PipelineConfig in Pipeline struct
- Edge: add WHERE embedding IS NOT NULL on HNSW index
* fix(v3): second-pass review fixes
- H1: use context.WithoutCancel for finalize + set ExitCode on ctx cancel
- H2: use utf8.RuneCountInString consistently in FallbackCounter
- H3: longest-prefix-match in ModelContextWindow (prevents wrong tokenizer)
- H4: return unsubscribe cleanup func from consolidation.Register
* feat(v3): implement DomainEventBus with worker pool, dedup, and retry
Worker pool processes events from buffered channel. SourceID-based dedup
prevents duplicate processing. Exponential backoff retry on handler error.
Panic recovery per handler. Graceful shutdown via Drain(). 8/8 tests pass
with race detector.
* feat(v3): implement ProviderAdapter for Anthropic, OpenAI, DashScope, Codex
Add CapabilitiesAware to all 6 providers. Create ProviderAdapter
implementations that delegate to existing buildRequestBody/parseResponse
for DRY. ClaudeCLI and ACP get capabilities only (subprocess transport).
DashScope wraps OpenAI adapter with StreamWithTools=false override.
* feat(v3): implement WorkspaceContext Resolver for 6 scenarios
Stateless resolver produces immutable WorkspaceContext at run start.
Handles personal/group/predefined/team-shared/team-isolated/delegation.
Wired into loop_context.go behind v3PipelineEnabled flag (additive,
v2 path unchanged). Includes delegation path boundary check,
master tenant bypass, and tenant slug path composition.
* feat(v3): implement tiktoken TokenCounter with BPE encoding + cache
Adds tiktoken-go for accurate cl100k_base/o200k_base token counting.
Per-message FNV-1a hash cache avoids re-encoding unchanged history.
Falls back to rune/3 heuristic for unknown models. NewTokenCounter
factory selects implementation at build time.
* feat(v3): promote 12 other_config JSONB fields to dedicated agent columns
Extract emoji, agent_description, thinking_level, max_tokens,
self_evolve, skill_evolve, skill_nudge_interval, reasoning_config,
workspace_sharing, chatgpt_oauth_routing, shell_deny_groups, and
kg_dedup_config from the catch-all other_config JSONB into proper
columns with DB-level types and defaults.
- Migration: PG (000037) + SQLite (schema v6→7) with backfill
- Go: AgentData struct + simplified Parse* methods
- Store: SELECT/INSERT/scan updated for both PG and SQLite
- Gateway: create/update handlers accept promoted fields
- HTTP: export/import with legacy backward compat
- Web UI: all 15 frontend files read/write from top level
* feat(v3): implement Knowledge Vault with unified search, wikilinks, and FS sync
Migration 000038 adds vault_documents (FTS+pgvector), vault_links, vault_versions
tables. VaultStore interface with PG implementation for document CRUD, hybrid
FTS+vector search, and bidirectional link management. All queries enforce
tenant_id isolation including JOIN-based scoping on link operations.
FS sync layer: SHA-256 content hashing, VaultInterceptor hooks into write_file/
read_file for auto-registration and lazy sync, fsnotify watcher with 500ms
debounce. Wikilink engine parses [[target]] syntax, resolves targets via
3-step strategy, and maintains vault_links on write.
VaultSearchService fans out queries across vault, episodic, and KG stores in
parallel with per-source score normalization and weighted merge. AutoInjector
and Retriever implementations for pipeline integration.
Three agent tools: vault_search (unified discovery), vault_link (explicit
linking), vault_backlinks (dependency tracing). Feature-flagged via
v3_vault_enabled agent setting.
* feat(v3): wire vault into gateway startup + add unit tests
Wire VaultStore embedding provider, VaultSearchService, VaultInterceptor
on read/write tools, and register vault_search/vault_link/vault_backlinks
tools in gateway_vault_wiring.go. All wiring gated by stores.Vault != nil.
Add 28 unit tests for ContentHash, ContentHashFile, and ExtractWikilinks
covering edge cases, unicode, display text, context windows, and offsets.
* feat(v3): implement stage-based pipeline loop with 8 pluggable stages
Decompose monolithic agent loop into internal/pipeline/ package:
- 6 stages: Context, Think, Prune+MemoryFlush, Tool, Observe+Checkpoint, Finalize
- Foundation types: Stage interface, RunState with 7 typed substates, MessageBuffer
- Pipeline orchestrator with setup/iteration/finalize 3-phase execution
- Callback-based PipelineDeps avoids circular import with agent package
- Feature-flagged via v3PipelineEnabled in Loop.Run()
- All 7 exit conditions preserved (no tools, max iter, truncation, loop kill,
read-only streak, tool budget, ctx cancel)
* feat(v3): wire pipeline callbacks to Loop methods + add 71 unit tests
Wire 15 of 17 PipelineDeps callbacks from Loop methods via closures:
- Context: LoadContextFiles, BuildMessages, EnrichMedia, InjectReminders
- Think: BuildFilteredTools, CallLLM (stream/sync)
- Prune: PruneMessages, CompactMessages
- Memory: RunMemoryFlush
- Finalize: SanitizeContent, FlushMessages, UpdateMetadata, BootstrapCleanup, MaybeSummarize
- Remaining: ExecuteToolCall, CheckReadOnly (deep loop.go integration)
Add comprehensive test suite (71 tests, all passing with -race):
- MessageBuffer: 10 tests (append, flush, replace, counts)
- Pipeline.Run: 14 tests (3-phase flow, exit conditions, ctx cancel)
- Stage tests: 47 tests (ThinkStage nudges/truncation, PruneStage budget,
ToolStage parallel/exit, ObserveStage content, CheckpointStage interval,
FinalizeStage cleanup)
* feat(v3): wire remaining 2 callbacks (ExecuteToolCall, CheckReadOnly)
Complete callback wiring — 17/17 PipelineDeps callbacks now active:
- ExecuteToolCall: resolves tool name, executes via registry, processes
result via existing processToolResult with loop detection bridge
- CheckReadOnly: delegates to checkReadOnlyStreak via bridge runState
- Bridge runState shares loop detection state between pipeline and agent
* fix(v3): eliminate data race in tool execution + capture injected messages
- Remove parallel tool execution path — serialize all tool calls to avoid
data races on shared bridgeRS (loop detector, media results, deliverables)
- Loop kill checked after each tool (mid-batch early exit)
- BuildFilteredTools: capture and append injected tool-awareness messages
- Rename test to reflect sequential execution
* feat(v3): wire ResolveWorkspace, safe parallel tools, ContextStage tests
- Wire ResolveWorkspace callback via workspace.NewResolver() with
ResolveParams from Loop fields (no longer a nil stub)
- Re-add safe parallel tool execution: split into ExecuteToolRaw
(parallel I/O) + ProcessToolResult (sequential state mutation)
with opaque rawData pass-through (no double execution)
- Add 12 unit tests for ContextStage (8) + MemoryFlushStage (3)
- Split tool callbacks to loop_pipeline_tool_callbacks.go (under 200 lines)
- Capture buildFilteredTools injected messages
* feat(v3): add episodic memory store + temporal KG columns
Phase 1 — Episodic Store:
- Migration 000039: episodic_summaries table with pgvector, FTS, L0 abstracts
- EpisodicStore PG impl: CRUD, hybrid FTS+vector search, ExistsBySourceID,
PruneExpired. Idempotent via source_id UNIQUE constraint.
Phase 2 — Temporal KG:
- Migration 000040: valid_from/valid_until on kg_entities + kg_relations,
partial indexes for current-facts queries, epoch→timestamptz backfill
- ListEntitiesTemporal: current-only, point-in-time, or include-expired modes
- SupersedeEntity: atomic expire-old + insert-new in single transaction
Schema version bumped to 40.
* fix(v3): review fixes for episodic store + temporal KG
- C1: Fix column name mismatch turn_count vs message_count in Go SQL
- C2: Remove redundant migration 000040 (000037 already adds temporal KG columns)
- H1: Use time.Time not int64 for TIMESTAMPTZ columns in SupersedeEntity
- H2: Add tenant_id scoping to Get/Delete for tenant isolation
- M2: Fix scanEntityTemporal to convert TIMESTAMPTZ→UnixMilli correctly
- L1: Remove unused uuid import from episodic_search.go
- Schema version corrected to 39 (only 000039 is new)
* feat(v3): implement consolidation pipeline with 3 event-driven workers
Event chain: session.completed → EpisodicWorker → episodic.created →
SemanticWorker → entity.upserted → DedupWorker
- EpisodicWorker: reuses compaction summary or calls LLM, generates L0
abstract (extractive), idempotent via source_id check
- SemanticWorker: extracts KG facts from episodic summary via existing
Extractor, sets temporal valid_from, publishes entity.upserted
- DedupWorker: runs DedupAfterExtraction on new entity IDs (terminal)
- L0 abstract: sentence-based extraction (~50 tokens), no LLM needed
- All workers registered via DomainEventBus.Subscribe()
* feat(v3): implement progressive loading with L0 auto-inject + unified search
- AutoInjector: searches episodic store, builds L0 prompt section (~200 tokens),
skips trivial messages via stopword filter
- L1Cache: in-memory LRU (500 entries, 1h TTL) for structured overviews
- UnifiedSearch: cross-tier search merging episodic + document results by score
- ContextStage integration: AutoInject callback appends memory section to system prompt
- MemorySection field added to ContextState for observability
* feat(v3): add memory_expand tool for L2 episodic retrieval
New tool: memory_expand(id) returns full episodic summary with metadata.
Complements memory_search L0/L1 results with deep L2 access.
Nil-safe: returns error message when episodic store not available.
Gateway wiring + memory_search depth param + kg_search temporal param
deferred to runtime integration phase.
* feat(v3): complete Phase 5 — tool extensions + gateway wiring
- memory_search: add depth param + episodic tier search merged with docs
- kg_search: add as_of temporal param, use ListEntitiesTemporal
- memory_expand: registered in gateway startup
- Gateway: Episodic field in Stores, PGEpisodicStore in factory,
embedding provider wired, tools connected to episodic store
* fix(v3): Phase 3 review fixes — tenant isolation + AutoInject args
- C1: Add tenant_id filter to ftsSearch, vectorSearch, List queries
(prevents cross-tenant episodic memory leaks)
- C2: Fix AutoInject callback signature — agent/tenant captured by
closure, only userMessage + userID passed explicitly
- H1: Add tenant_id to List query
* feat(v3): wire per-agent v3 flags from DB into dual-mode gate
Parse v3_pipeline_enabled, v3_memory_enabled, v3_retrieval_enabled from
agent other_config JSONB via ParseV3Flags(). Resolver now sets all flags
on LoopConfig so the existing gate in loop_run.go reads from DB.
- V3Flags struct + ParseV3Flags() + ValidateV3Flags() in store layer
- v3MemoryEnabled/v3RetrievalEnabled added to Loop, LoopConfig, PipelineConfig
- Auto-inject gated on V3RetrievalEnabled (was unconditional)
- Structured perf logging for v3 pipeline runs
- v3 flag validation on both WS agent.update and HTTP PUT endpoints
* feat(v3): wire AutoInjector into pipeline for L0 memory auto-inject
Create AutoInjector at gateway startup from episodic store, pass through
ResolverDeps → LoopConfig → Loop. Pipeline adapter builds AutoInject
callback capturing agent/tenant context via closure.
ContextStage already gates on V3RetrievalEnabled + AutoInject != nil.
* feat(v3): add tool metadata map + capability-based deny rules
Registry gains per-tool ToolMetadata map with RegisterWithMetadata()
and GetMetadata() (infers defaults from tool name when not explicit).
PolicyEngine gains DenyCapability() for RBAC integration — tools with
denied capabilities filtered at step 8 after existing 7-step pipeline.
* fix(v3): add RWMutex to PolicyEngine capability deny fields
DenyCapability() and SetRegistry() now guarded by sync.RWMutex.
FilterTools reads snapshot under RLock. Prevents data race when
capability rules are modified concurrently with tool filtering.
* feat(v3): implement delegate tool for inter-agent task delegation
New `delegate` tool wraps existing agent_links infrastructure
(CanDelegate, DelegateTargets). Supports async (fire-and-forget)
and sync (block with timeout) modes. Permission checked via
AgentLinkStore. Events emitted: delegate.sent/completed/failed.
DelegateRunFunc injected by gateway to avoid circular dependency.
* feat(v3): complete 3 deferred implementations
1. OrchestrationMode resolution: ResolveOrchestrationMode() checks
team membership → delegate links → spawn (priority order).
2. PG EvolutionMetricsStore: RecordMetric, QueryMetrics, aggregate
tool/retrieval metrics, TTL cleanup. All queries tenant-scoped.
3. BridgePromptBuilder: implements PromptBuilder interface by
delegating to existing BuildSystemPrompt(). Appends v3 memory
L0 section when enabled. Ready for template engine swap later.
* fix(v3): address code review findings on commits 5-6
- C1: CanDelegate now tenant-scoped (fail-closed on missing tenant)
- H1: Sync delegate timeout capped at 600s
- H2: Async goroutine gets 10min deadline (prevents leaks)
- H3: JSONB casts use COALESCE/NULLIF guards (handles missing fields)
- M1/M2: Remove dead code (formatVaultSection, memoryL0ToStrings)
* fix(teams): stop auto-creating agent_links for team members
Teams use agent_team_members table directly — agent_links caused
context confusion between team dispatch and delegation systems.
- Remove autoCreateTeamLinks() calls from team create + member add
- Remove link cleanup from member remove
- Remove dead autoCreateTeamLinks() function
- Append DELETE to migration 000039: clear team-created agent_links
* fix(v3): tenant isolation for all agent_links queries + PromptBuilder Instructions
- DelegateTargets, GetLinkBetween, SearchDelegateTargets,
SearchDelegateTargetsByEmbedding, DeleteTeamLinksForAgent all now
scoped by tenant_id (fail-closed on missing tenant)
- BridgePromptBuilder now maps Instructions/InstructionContent to
AGENTS.md context file (was silently dropped)
* feat(v3): wire orchestration mode + evolution metrics into agent loop
- Orchestration mode: resolver resolves mode from team/links, tool filter
hides delegate/team_tasks based on mode, prompt builder injects delegation
targets section
- Evolution metrics: non-blocking goroutine records tool execution metrics
(name, success, duration) via EvolutionMetricsStore in both v2 loop and
v3 pipeline paths (sequential + parallel)
- Fix review findings: tenant ID propagated via store.WithTenantID in
background goroutine, 5s timeout prevents goroutine leak
* feat(v3): implement suggestion engine with pluggable analysis rules
- PG EvolutionSuggestionStore: CRUD for agent_evolution_suggestions table
- SuggestionEngine: aggregates 7-day metrics, runs rules, deduplicates
pending suggestions per type before creating new ones
- 3 initial rules: LowRetrievalUsage (usage_rate<0.2), ToolFailure
(success_rate<0.1), RepeatedTool (>100 calls/week → suggest skill)
- EventSuggestionCreated event type added to eventbus
- Cron wiring deferred to gateway startup integration pass
* feat(v3): implement auto-adapt guardrails with apply/rollback
- AdaptationGuardrails: max delta per cycle, min data points, locked
params, rollback-on-drop percentage
- ApplySuggestion: applies threshold suggestions to agent other_config
JSONB, stores baseline for rollback
- RollbackSuggestion: restores baseline values from suggestion params
- EvaluateApplied: compares post-apply metrics to baseline, auto-rolls
back when quality drops beyond threshold
- Scope limited to retrieval params only (never security settings)
* feat(v3): wire evolution stores + daily/weekly cron for suggestions
- Add EvolutionMetrics + EvolutionSuggestions to Stores struct + PG factory
- Wire EvolutionMetricsStore into ResolverDeps (cmd/gateway_managed.go)
- Add gateway_evolution_cron.go: daily suggestion analysis + weekly
evaluation/rollback for applied suggestions
- Cron runs as background goroutine with 5-min timeout per cycle
* fix(v3): address code review findings on evolution engine
- C1: persist baseline parameters before marking suggestion as applied
(was building map but never saving — rollback would always fail)
- H1: add tenant_id isolation to UpdateSuggestionStatus, GetSuggestion,
and new UpdateSuggestionParameters method
* test(v3): add unit tests for orchestration, suggestions, guardrails, prompt
- orchestration_mode_test: orchModeDenyTools (4 modes) + ResolveOrchestrationMode
(4 scenarios with mock stores)
- suggestion_rules_test: LowRetrievalUsage, ToolFailure, RepeatedTool with
threshold boundary tests (at/below/above min data points)
- evolution_guardrails_test: DefaultGuardrails values + CheckGuardrails
(insufficient data, locked params, zero-min fallback)
- prompt_builder_orchestration_test: BridgePromptBuilder orchestration section
presence/absence across 4 scenarios + target content verification
* test(v3): add integration tests for evolution metrics + suggestions
- Test helper: shared PG connection with sync.Once migration, per-test
tenant+agent seed with cleanup
- Evolution metrics: RecordMetric, AggregateToolMetrics (success rate),
Cleanup (TTL deletion)
- Evolution suggestions: full CRUD, UpdateSuggestionParameters (baseline
persist), tenant isolation (cross-tenant read blocked)
- Pipeline E2E: seed 25 failed tools + 55 low-usage retrievals, verify
SuggestionEngine creates suggestions, verify dedup on second run
- Fix: migration 039 de-duped (episodic_summaries already in 037)
- Fix: NULL reviewed_by scan via sql.NullString
* feat(v3): add HTTP API handlers for evolution, vault, episodic, orchestration, v3-flags
5 new handler files exposing v3 backend stores as REST endpoints:
- evolution_handlers.go: metrics query/aggregate + suggestions CRUD
- vault_handlers.go: cross-agent document listing + search + links
- episodic_handlers.go: episodic summaries list + hybrid search
- orchestration_handlers.go: computed mode + delegate targets (read-only)
- v3_flags_handlers.go: per-agent v3 feature flag get/toggle
Store fixes from code review:
- episodic FTS: use inline to_tsvector (no stored tsv column)
- episodic: conditional user_id filter in List + Search (admin view)
- episodic: add tenant_id to ExistsBySourceID + PruneExpired
- evolution: require tenant_id in context (no struct fallback)
- evolution: check RowsAffected on suggestion updates
- vault: optional agent_id filter in ListDocuments (cross-agent)
* feat(v3): add web UI for evolution tab, v3 settings, vault page, episodic memory
Agent Detail enhancements:
- V3 Settings section: pipeline/memory/retrieval flag toggles
- Orchestration section: mode badge + delegate targets display
- Evolution section: added metrics + suggestions v3 flag toggles
- Evolution tab: Recharts metrics charts + suggestion review table
with approve/reject/rollback actions + guardrails card
New pages:
- /vault: Knowledge Vault document registry with cross-agent listing,
hybrid search dialog, document detail with wikilinks
- Memory page: added Episodic Memory tab with summary cards,
expandable details, key topic badges, and hybrid search
Infrastructure:
- HttpClient: added patch() method
- Query keys: v3Flags, orchestration, evolution namespaces
- 4 new hooks: use-v3-flags, use-orchestration, use-evolution-metrics,
use-evolution-suggestions, use-vault, use-episodic
- i18n: vault namespace (en/vi/zh), agents + memory keys updated
- Reused formatRelativeTime from lib/format.ts (eliminated 3 duplicates)
* refactor(http): add bindJSON helper and migrate all decode call sites
Replace 36 json.NewDecoder(r.Body).Decode + error blocks with bindJSON
across 20 HTTP handler files. Standardizes decode error responses to
structured writeError format. Fixes unchecked decode in handleIndexAll.
* refactor(store): adopt sqlx for PG scan operations (Phase 1+2)
Add jmoiron/sqlx v1.4.0 with camelToSnake json tag mapper.
Migrate scan-heavy PG store methods to sqlx Get/Select:
- tracing.go: GetTrace, ListTraces, ListChildTraces, GetTraceSpans, GetCostSummary
- heartbeat.go: Get, ListDue, ListLogs
- providers.go: GetProvider, GetProviderByName, ListProviders, ListAllProviders
- mcp_servers.go: GetServer, GetServerByName, ListServers
- pairing.go: ListPending, ListPaired
- agents_export_queries.go: 5 export functions
- agents_export_team_queries.go: exportTeamMembers, ExportAgentLinks
All writes (INSERT/UPDATE/DELETE), execMapUpdate, and dynamic WHERE
builders remain raw SQL. Zero behavior change.
* refactor(store): adopt sqlx for SQLite scan operations (Phase 3)
Migrate SQLite store scan methods to sqlx Get/Select:
- providers.go: GetProvider, GetProviderByName, ListProviders, ListAllProviders
- tenants.go: GetTenant, GetTenantBySlug, ListTenants, GetTenantUser, ListUsers, ListUserTenants
- mcp_servers.go: GetServer, GetServerByName, ListServers
Create sqlx_scan_structs.go with sqliteTime-aware scan structs
(providerRow, tenantRow, tenantUserRow, mcpServerRow) to handle
SQLite TEXT timestamp parsing via StructScan.
* refactor(store): migrate PG bulk scan operations to sqlx (Phase 4)
Migrate scan-heavy methods across 6 PG store files:
- tenant_store.go: GetTenant, GetTenantBySlug, ListTenants, GetTenantUser,
ListUsers, ListUserTenants — removed 3 scan helpers
- teams.go: ListTeams, GetTeam, ListMembers, ListMembersByTenant
- teams_tasks_activity.go: ListComments, ListEvents, ListFollowUps
- pending_message_store.go: ListPending, ListByHistoryKey
- skills_grants.go: ListAgentGrants
- config_permissions.go: CheckPermission
~20 scan ops converted. Files with encryption post-processing,
pq.Array, pgvector, or dynamic SQL kept raw.
* refactor(store): extract shared CamelToSnake mapper, add UUIDArray usage note
- Move camelToSnake to internal/store/column_mapper.go (DRY)
- Both pg and sqlitestore packages now import shared CamelToSnake
- Add planned-use comment on UUIDArray type
* refactor(cli): migrate commands from config.json to HTTP API, add providers/setup/TUI
- Add unified HTTP client (gateway_http_client.go) with auth, error parsing, typed generics
- Rewrite agent list/add/delete to use gateway HTTP API instead of config.json
- Rewrite channels list to HTTP API, add channels add/delete subcommands
- Replace models command with full providers CRUD (list/add/update/delete/verify)
- Add setup wizard command (provider → agent → channel post-onboard flow)
- Add Bubble Tea TUI behind build tag (tui/!tui with noop fallback)
- Update onboard next-steps to mention goclaw setup
- Add build-tui Makefile target
- Fix URL path injection (url.PathEscape on all user-supplied path segments)
- Fix UTF-8 truncation in skills description display
* refactor(store): add explicit db struct tags, fix sqlx mapper for heartbeat scan error
Switch sqlx mapper from NewMapperFunc (which only applies CamelToSnake to
field names, not tag values) to NewMapperFunc("db", CamelToSnake) with
explicit db:"column_name" tags on all store structs.
Root cause: NewMapperFunc("json", fn) sets mapFunc but not tagMapFunc,
so camelCase json tags like "agentId" were used as-is instead of being
converted to "agent_id", causing "missing destination name" scan errors.
Fix: use db struct tags as the source of truth for column mapping.
Every DB entity field gets db:"column_name", nested JSON configs and
runtime-only structs get db:"-".
* test(store): add integration tests for 13 store interfaces (70 tests)
Cover Tier 1 (critical) + Tier 2 (security) stores with integration tests
running against pgvector pg18. Coverage from 2.4% to ~54%.
Stores tested: Session, Agent, Team/Task, Memory, KnowledgeGraph, Vault,
MCP Server, API Key, ConfigPermission, Contact.
Infrastructure: fixture builders (seedTeam, seedMCPServer, etc.),
mock EmbeddingProvider, multi-tenant helpers, expanded cleanup.
* fix(store): resolve NULL scan bugs in MCP server and task metadata
- mcp_servers: COALESCE nullable TEXT columns (display_name, command,
url, api_key, tool_prefix) to prevent sqlx scan failures
- mcp_servers_access: COALESCE nullable JSONB columns in ListAgentGrants
(tool_allow, tool_deny, config_overrides) to prevent silent row drops
- teams_tasks: default task metadata to '{}' instead of nil to satisfy
NOT NULL constraint on CreateTask
- sqlx_helpers: export InitSqlx for integration test setup
* feat(pipeline): fix v3 pipeline context injection, tracing, KG temporal filters
- Pipeline context: add InjectContext + LoadSessionHistory callbacks to
ContextStage, propagate enriched ctx via state.Ctx for iteration stages
- Pipeline tracing: wrap makeCallLLM with emitLLMSpanStart/End, wrap
makeExecuteToolCall/Raw with emitToolSpanStart/End
- Token counter: switch pipeline from FallbackCounter to TiktokenCounter
- KG temporal: add valid_until IS NULL filter to all entity/relation
queries (list, search, vector, FTS, traversal CTE, stats)
- Skills: add SkillEmbedder interface for future hybrid BM25+vector search
- Cache: remove unused tenantResolve dead code from PermissionCache
- Store: fix NULL scan bugs in tracing metadata and agent skill_nudge
- Test: add TestStoreKG_TemporalFilter integration test
- UI: add v3 version badge, evolution section, memory/traces improvements
* refactor(store): migrate KG store from raw sql.Rows to sqlx StructScan
Migrate 6 knowledge graph store files from manual rows.Scan() to
pkgSqlxDB.GetContext/SelectContext with intermediate scan row structs.
- Add entityRow, relationRow, traversalRow, dedupCandidateRow structs
with json.RawMessage for jsonb and time.Time for timestamptz columns
- Add toEntity()/toRelation() converters (UnixMilli + json.Unmarshal)
- Add sqlxTx() helper for wrapping *sql.Tx with sqlx mapper
- Fix ScanDuplicates passing time.Now().Unix() to TIMESTAMPTZ column
- Fix ListEntitiesTemporal missing tenant scope (scopeClause)
- Fix SupersedeEntity missing tenant scope and tenant_id on INSERT
- Fix DedupCandidate.CreatedAt using Unix() instead of UnixMilli()
- Update agents_export_queries.go to reuse new scan row structs
- Net -160 lines of manual scan boilerplate removed
* refactor(store): migrate memory, skills, agents, sessions, mcp, cron, vault stores to sqlx
Batch migration of 19 store files from raw rows.Scan() to
pkgSqlxDB.GetContext/SelectContext with intermediate scan row structs.
Groups migrated:
- Memory: memory_docs, memory_admin, memory_search, memory_embedding_cache
- Episodic: episodic_search, episodic_summaries
- Skills: skills, skills_admin, skills_embedding, skills_export_queries
- Agents: agents (backfill+shares), agents_context, agents_export_team_standalone
- Sessions: sessions_list (List, ListPaged, ListPagedRich)
- MCP: mcp_servers_access, mcp_export_queries
- Cron: cron_exec (GetRunLog)
- Vault: vault_documents (ListDocuments, ftsSearch, vectorSearch)
- Tenant: tenant_configs (ListDisabled, ListAll)
7 new scan row files created. Net -510 lines of manual scan boilerplate.
INSERT/UPDATE/DELETE and scalar COUNT queries kept as raw SQL.
* fix(store): fix 3 sqlx scan struct db tag issues found by audit
- Fix vault FTS alias mismatch: `AS rank` → `AS score` (critical: runtime scan error)
- Fix episodic key_topics type: json.RawMessage → pq.StringArray (TEXT[] column)
- Fix agentShareRow.CreatedAt: string → time.Time, wire to output struct
* feat(providers): implement Wave 2 provider resilience and intelligence
9-phase implementation covering:
- Request middleware chain with composable body transformers
- OpenAI prompt caching, service tier, and fast mode middlewares
- Error classification (9 categories) with two-tier failover
- Model registry with forward-compat resolvers (Anthropic + OpenAI)
- Embedding providers (OpenAI + Voyage) with 1536-dim validation
- Cooldown/probe system with per-provider:model state tracking
- Markdown-aware chunking shared across 5 channels
- Session recall via FTS + pgvector on episodic summaries
- Dreaming/promotion pipeline for long-term memory consolidation
Migrations: 000040 (episodic search index), 000041 (promoted_at column)
Schema version: 39 → 41
* feat(providers): wire model registry into gateway provider construction
Create InMemoryRegistry with Anthropic + OpenAI forward-compat resolvers
at gateway startup. Pass to all Anthropic and OpenAI providers created
from both config and DB sources.
* feat(consolidation): wire DomainEventBus and consolidation pipeline
Create DomainEventBus at gateway startup, thread through resolver →
LoopConfig → Loop → PipelineDeps. Emit session.completed event after
each run finalization. Register consolidation pipeline (episodic →
semantic → KG dedup → dreaming) with event bus subscriptions.
* fix(store): fix episodic key_topics pq.Array, ON CONFLICT, and migration 040 immutability
- episodic_summaries.go Create: json.Marshal(KeyTopics) → pq.Array (text[] column)
- episodic_search.go scanEpisodic/scanEpisodicRow: json.RawMessage → pq.StringArray
- episodic_summaries.go Create: ON CONFLICT add WHERE source_id IS NOT NULL for partial index
- migration 040: add immutable_array_to_string wrapper (array_to_string is STABLE in PG)
* test(store): add 17 integration tests for skills, cron, episodic, tenant configs
- Skills store: 6 tests (CRUD, grants, tenant isolation)
- Cron store: 4 tests (job CRUD, run log sqlx scan, pagination, tenant isolation)
- Episodic store: 4 tests (summary CRUD, list, FTS search, tenant isolation)
- Tenant configs: 3 tests (tool/skill disable, list, tenant isolation)
- Test helper: add cleanup for skills, cron, episodic tables
* fix(permissions): use cron-specific permission check for cron tool (#725)
* fix(security): harden exec path exemption matching (#721)
- Add absolute path exemption for dataDir/skills-store/ (fixes skill
scripts using absolute paths like /app/data/skills-store/ being denied)
- Strip surrounding quotes before prefix matching (LLMs often quote paths)
- Reject path traversal ("..") in exempt fields to prevent escape
- Switch from "any field exempt → skip" to per-field matching: only exempt
if ALL fields that match the deny pattern are individually exempt
- Closes pipe/comment bypass vectors where an exempt path in one argument
would exempt the entire command including non-exempt paths
Includes 27 test cases covering: legitimate access, quoted paths,
path traversal, unicode bypass, pipe/comment bypass, mixed args.
* fix(permissions): use cron-specific permission check for cron tool
Cron tool was hardcoded to check `file_writer` configType via
CheckFileWriterPermission(), ignoring the `cron` configType that
the UI actually saves when granting cron permissions. This caused
agents in group chats to be denied cron access even with correct
permission configured.
Add ConfigTypeCron constant and CheckCronPermission() that checks
`cron` configType first, falling back to `file_writer`.
---------
Co-authored-by: Viet Tran <viettranx@gmail.com>
* fix(chat): load message history on first conversation click (#730)
* fix(chat): load message history when selecting existing conversation from clean state
The skipNextHistoryRef was unconditionally set when sessionKey transitioned
from empty to non-empty. This prevented loadHistory() from running when
clicking an existing conversation from the initial /chat page. The skip
was only intended for the new-chat send flow where the optimistic message
is already displayed.
Guard the skip with expectingRunRef so it only activates when a message
send is in flight.
Closes #729
* docs: add UI diff evidence for PR #730
Before/after screenshots and HTML comparison report showing
first conversation click behavior fix.
* feat(whatsapp): port native WhatsApp channel with whatsmeow from dev
Cherry-pick
|
||
|
|
7ef6d1d17a |
fix(agent): correct soft-trim head/tail budget allocation when tail is important (#723)
Co-authored-by: quxy5 <quxy5@outlook.com> |
||
|
viettranx
|
63f0c0fe45 |
feat(agent): centralized tenant user identity resolution for credentials
Add CredentialUserID context key that resolves channel contacts to merged tenant users for credential lookups (SecureCLI, MCP). Keeps UserID unchanged for session/workspace scoping. Resolves group senders, group contacts, and unresolved DMs via ContactStore with 60s TTL cache. |
||
|
viettranx
|
0a27e1247a | Merge remote-tracking branch 'origin/main' into dev | ||
|
Viet TranandGitHub
|
e88686b13a |
fix: deterministic prompt ordering for LLM cache hit (#719)
Sort all non-deterministic map iterations that affect system prompt and tool definitions sent to LLM APIs. Go map iteration order is random, causing prompt prefix to change every turn — breaking Anthropic/OpenAI prompt caching (cache by exact prefix match). Fixed 5 sources of non-deterministic ordering: - Registry.List(): sort canonical tool names - Registry.ProviderDefs(): sort tools + aliases before building defs - PolicyEngine.FilterTools(): sort alias iteration (single Aliases() call) - buildMCPToolsInlineSection(): sort MCP tool names in system prompt - GetAgentContextFiles/GetUserContextFiles: ORDER BY file_name (PG+SQLite) Based on PR #718 by @therichardngai-code with additional fixes: - Context files from DB now deterministic (ORDER BY file_name) - FilterTools() calls registry.Aliases() once instead of 3 times |
||
|
viettranx
|
6ddc112940 |
fix(prompt): skip credentialed CLI context when exec tool is denied
Agents with exec in their deny list cannot run CLI commands, so injecting wrangler/gh credential context is misleading — the LLM sees instructions for tools it cannot use. Gate the section on exec being present in the filtered tool list. |
||
|
viettranx
|
41e6c8f5cc |
feat(infra): tracing recovery, browser cleanup, CLI fixes, UI workspace split (#709)
- Tracing: recover stale running traces/spans on startup (PG + SQLite) - Browser: Chrome orphan cleanup via launcher PID, timeouts, Leakless - Claude CLI: WaitDelay 5s + context-cancel early exit - Agent loop: safety-net defer to finalize orphan root traces - UI: split workspace sharing into separate Memory and KG toggles - Minor: for-range idiom, min() builtin |
||
|
viettranx
|
7a266aee36 |
fix(openai): Together-compatible requests with reasoning/stream/vision gating
Port PR #685 fixes for HTTP 400 on Together AI and strict OpenAI-compat hosts, with additional improvements: - Gate reasoning_content on assistant history to allowlisted models only (OpenAI o-series/GPT-5, DeepSeek, Kimi) — prevents HTTP 400 on Together/Qwen - Gate reasoning_effort to models that support it (OpenAI reasoning family) - Skip stream_options for Together endpoints (causes HTTP 400) - Scope DashScope enable_thinking/thinking_budget to DashScope providers only - Reorder multimodal parts: text before images (Together/Qwen preferred order) - Add redacted_thinking tag to sanitization patterns with early-exit guard fix - Upgrade Together detection from URL-only to URL + providerType + name fallback (mirrors dashScopePassthroughKeys pattern for reverse-proxy compatibility) - Add comprehensive tests for all new behavior |
||
|
viettranx
|
156b2dd96c |
feat(secure-cli): per-agent grants with setting overrides
Replace agent_id column on secure_cli_binaries with is_global flag
and new secure_cli_agent_grants table for per-agent access control
with optional deny_args, deny_verbose, timeout_seconds, tips overrides.
- Migration 000036: create grants table, migrate agent-specific rows,
dedup binaries, drop agent_id, add is_global
- Store layer: SecureCLIAgentGrantStore interface + PG implementation,
LookupByBinary with LEFT JOIN grant merge, ListForAgent
- HTTP API: CRUD endpoints at /v1/cli-credentials/{id}/agent-grants
- Agent loop: buildCredentialCLIContext uses ListForAgent for scoped
system prompt (agents only see authorized CLIs)
- Web UI: grants dialog with card list + inline form, is_global toggle
replaces agent dropdown, i18n for en/vi/zh
|
||
|
Viet TranandGitHub
|
cfc69f6f70 |
fix(tracing): use per-request model/provider overrides in trace spans (#667) (#668)
Trace spans and cost calculations were recording the agent's default model/provider instead of the effective overridden values from heartbeat requests. Introduced functional options pattern (spanOption) so all span emitters resolve the correct model/provider. |
||
|
viettranx
|
ade6c7ba33 |
fix(agent): preserve MediaRefs when merging consecutive same-role messages
sanitizeHistory now appends MediaRefs from dropped messages so compaction summary retains knowledge of shared media files. |
||
|
Michael Nguyenandviettranx
|
d79d738148 |
fix(agent): clear RawAssistantContent on dedup + merge consecutive same-role messages
sanitizeHistory dedup rewrites ToolCalls IDs but not RawAssistantContent. Anthropic provider prefers RawAssistantContent → sends stale IDs → HTTP 400. Fix: clear RawAssistantContent when dedup fires. Also merge consecutive same-role messages (user↔user, assistant↔assistant) to fix role alternation errors from session corruption. |
||
|
viettranx
|
7418cb62aa |
fix(agent): reduce system prompt size and fix team context injection (#613)
Prompt compression (Issue #613): - Truncate skill descriptions to 200 runes in inline XML (matches mcpToolDescMaxLen) - Lower skill inline token threshold from 5000 to 3000 - Compact tool descriptions (risk-audited: preserve behavioral hints) - Compact boilerplate sections: media hint, safety, tool call style, spawn, self-evolve, skill creation, team workspace - Fix token estimator to account for description truncation - Restore safety anti-manipulation clauses dropped during compaction Team context injection fix (found during prod audit): - Add IsTeamLead field to LoopConfig/Loop, plumbed from resolver - Gate team context (TEAM.md, workspace, members) on session type: leader inbound → team context; member-only inbound → spawn section; team dispatch → team context - Filter TEAM.md from context files for member-only inbound chat - Skip team member DB query when team context not needed - Rename HasTeam → IsTeamContext for semantic clarity - Add 8 table-driven tests for team context injection scenarios |
||
|
viettranx
|
4e4a835795 |
fix(agent): defer warning messages after parallel tool results
When parallel tool calls trigger loop detection warnings, the warning messages (role="user") were inserted between tool result messages (role="tool"). This breaks the Anthropic API when routed through OpenAI-compatible proxies (e.g. LiteLLM): the proxy groups consecutive tool messages into a single user message with tool_result blocks, but an intervening user warning splits the group, causing orphaned tool_results and HTTP 400 "tool_use ids without tool_result blocks". Fix: accumulate warning messages during parallel result processing and append them after all tool results, preserving the consecutive grouping. Closes #642 |
||
|
viettranx
|
d819e08071 |
fix(security): fix media upload permission denied + symlink protection
- Fix workspace dir ownership in Docker entrypoint: chown dirs not owned by goclaw on startup (handles dirs created by root in previous lifecycle) - Add symlink check on .uploads/ via os.Lstat before file creation to prevent symlink-based attacks replacing .uploads with link to sensitive dir |
||
|
viettranx
|
1b190fa0bb |
fix(prompt): reduce mechanical chat behavior + optimize system prompt
- Add Tool Call Style section with narration minimalism + non-disclosure
rule (from TS reference): agents must never expose tool names to users
- Consolidate 3 redundant memory recall reminders into 1 dedicated section
- Remove "tell the user you checked but found nothing" instruction that
caused agents to describe internal tool mechanics in responses
- Remove 11 tool aliases from system prompt listing (~300 tokens saved);
aliases still work via provider definitions
- Filter alias tool names out of system prompt ToolNames in loop_history
- Update AGENTS.md: remove tool name references from Memory section,
add group chat framing from V1 ("participant, not their proxy")
|
||
|
viettranx
|
c388364d2c |
fix(ui): fix chat streaming race condition + require agent selection + improve chat UX
- Fix race condition where session-change effect cleared runIdRef after run.started already captured it, causing chunk events to be filtered out (user saw "thinking" but no streamed tokens on new chats) - Add SessionRunID to router + return runId in session status response as backup restoration for event filtering - Require explicit agent selection before chat input is shown - Redesign ChatInput: attach icon inside input container, aligned send - Port desktop UX: wobble animation for tool calls, auto-expand thinking block on stream start, amber icon for streaming, iteration step count |