Commit Graph

90 Commits

Author SHA1 Message Date
Duc Nguyen 23d0b5eb0b fix(providers): auto-clamp max_tokens on model rejection (#267)
* fix(providers): auto-clamp max_tokens on model rejection + fix verify for reasoning models

When OpenAI-compat models reject max_tokens as too large (e.g. gpt-3.5-turbo
supports 4096 but we send 8192), parse the model's stated limit from the 400
error, clamp the value, and retry once. This fixes agent creation for models
with lower output token limits without hardcoding model names.

Also increase the provider verify endpoint's max_tokens from 1 to 50 so
reasoning models (gpt-5, o-series) have enough headroom for internal
reasoning during the check call.

Closes #248, closes #245

* refactor(providers): extract chat retry closure + fix clamp log key

- Extract duplicate retry closure into chatRequestFn() to follow DRY
- Fix slog logging wrong key: body["max_tokens"] was nil for reasoning
  models that use max_completion_tokens — now uses clampedLimit() helper
- Remove unnecessary _ = resp in provider verify endpoint

---------

Co-authored-by: viettranx <viettranx@gmail.com>
2026-03-19 08:41:20 +07:00
Duc Nguyen dc51018563 fix: subagent provider routing + api_base fallback (#262)
* fix(subagent): inherit parent agent's provider instead of alphabetical fallback

Subagents previously used a fixed provider (alphabetically first from the
registry, often "anthropic") regardless of which provider the parent agent
used. This caused invalid combos like anthropic/glm-5 when a zai-coding
agent spawned subagents.

- Pass provider registry to SubagentManager for runtime resolution
- Inject parent provider name into context (WithParentProvider)
- Resolve activeProvider from parent context before LLM call
- Fix trace spans to show actual resolved provider, not default

* fix(providers): api_base fallback from config/env for DB providers

DB providers with empty api_base now inherit from config/env vars
(e.g., GOCLAW_ANTHROPIC_BASE_URL). Prevents proxy API keys from being
sent to the real provider API endpoint.

- Add APIBaseForType() method on ProvidersConfig
- registerProvidersFromDB falls back to config when api_base is empty
- ProvidersHandler uses resolveAPIBase() for model listing
- Add api_base, display_name, settings to provider validation whitelist

* fix(tracing): pass resolved provider name to subagent span emitters

- emitSubagentSpanStart now accepts providerName param instead of
  reading sm.provider.Name() — ensures root subagent span reflects
  the inherited parent provider, not the fallback default
- registerInMemory now uses resolveAPIBase() so DB providers with
  empty api_base inherit the config/env fallback (same as startup path)

---------

Co-authored-by: viettranx <viettranx@gmail.com>
2026-03-18 22:40:49 +07:00
Viet Tran ce333c70f3 fix(security): followup hardening — ILIKE ESCAPE, allowlist logging, shell deny, tests (#251)
- Add explicit ESCAPE '\' clause to all ILIKE queries (knowledge_graph,
  custom_tools, channel_instances, channel_contacts) for correct wildcard
  escaping regardless of PostgreSQL standard_conforming_strings setting
- Log warning when filterAllowedKeys drops unknown fields for debuggability
- Widen base64 decode shell deny pattern to catch -di, -dw0 variants
- Add unit tests for filterAllowedKeys, pqStringArray, scanStringArray,
  pqStringArray↔scanStringArray roundtrip, limitedBuffer, base64 deny
2026-03-18 07:48:48 +07:00
Luan Vu a7f5acc1e3 fix(security): harden SQL injection, MCP prompt injection, sandbox fallback, and file serving (#246)
- execMapUpdate: validate column names with strict regex to prevent SQL injection
- HTTP update handlers: add field allowlists (agents, providers, custom_tools, mcp, channel_instances)
- pqStringArray: properly escape array elements to prevent PostgreSQL array literal injection
- scanStringArray: handle quoted elements in PostgreSQL array format
- MCP bridge: wrap tool results as external/untrusted content to prevent prompt injection
- File serving: block access to sensitive system directories (/etc, /proc, /sys, etc.)
- Sandbox: fail closed when Docker unavailable instead of silent fallback to host
- Shell deny: fix base64 --decode bypass, add host exec 1MB output limit
- ILIKE queries: escape % and _ wildcards in knowledge_graph, custom_tools, channel_instances

Co-authored-by: Luvu182 <208665161+Luvu182@users.noreply.github.com>
2026-03-18 07:42:38 +07:00
viettranx 843b550651 feat: runtime packages UI, pkg-helper, configurable shell deny groups (#244)
Runtime package management with security hardening:

- pkg-helper: root-privileged daemon for apk install/uninstall via Unix socket
- HTTP API: /v1/packages (list/install/uninstall/runtimes), admin role required for writes
- Shell deny groups: 15 configurable groups (per-agent overrides via context)
- Packages UI: Web page for managing system/pip/npm packages with confirmation dialogs
- Docker: privilege separation (root entrypoint → su-exec drop), init for zombie reaping
- Security: umask socket creation, persist file validation, deny pattern hardening
  (Node.js fetch/http, Python from/import, curl localhost, sensitive env vars)
- Auth: empty gateway token → admin role (dev/single-user mode)
2026-03-17 19:50:26 +07:00
viettranx b2a74ba487 feat(skills): skill_manage tool + skill_evolve learning loop (#218)
Adds skill_manage — a first-class agent tool for creating, updating, and
deleting skills from within a conversation — paired with per-agent
skill_evolve that nudges predefined agents to capture reusable workflows.

Tool (skill_manage):
- create: write skill from SKILL.md content string (auto-grant, dep scan)
- patch: find/replace producing new immutable version (advisory-locked)
- delete: soft-delete (archive in DB, move to .trash/)
- Security guard: 25 regex patterns block shell injection, credential exfil,
  path traversal, SQL injection, privilege escalation
- Ownership enforced: only skill owner can patch/delete (admin bypass)
- Content size limit: 100KB; companion file copy: 20MB, symlink-safe (WalkDir)
- Enabled: false by default — admin opt-in per agent

Learning loop (skill_evolve, predefined agents only):
- System prompt: SHOULD/SHOULD NOT guidance for skill creation
- Budget nudges: [System] prefix at 70%/90% iteration budget (ephemeral, i18n)
- Postscript: once-per-run suggestion with explicit user consent
- Config: other_config.skill_evolve + skill_nudge_interval (default 15)

Security hardening (pre-existing + new):
- CreateSkillManaged: RETURNING id + pg_advisory_xact_lock (atomic upsert)
- GetNextVersionLocked: advisory lock for race-safe patch versioning
- Ownership checks on HTTP update/delete, gateway update, 4 grant/revoke handlers
- copyOtherFiles: filepath.WalkDir for real symlink detection

UI: Skill Learning toggle + nudge interval in Agent General Tab
i18n: backend (en/vi/zh catalogs) + frontend (en/vi/zh locale files)
2026-03-17 11:38:35 +07:00
badgerbees 365f41f81c fix: pass custom name to DashScopeProvider for correct registry lookup (#228) 2026-03-16 22:54:10 +07:00
viettranx b0bd4d6198 fix(pairing): fix browser approval stuck + security hardening
Squash-merge PR #225 with security fixes:

- Fix browser pairing stuck on "Waiting for approval" (stale closure:
  useState → useRef for senderID in pairing-form)
- Fix auto-kick after pairing (RequireAuth now accepts senderID,
  onAuthFailure skips logout for paired browser sessions)
- Allow browser-paired users to access HTTP APIs via X-GoClaw-Sender-Id
  header with fail-closed IsPaired check
- Remove ad-hoc IsInternalOrBrowser(), use channels.IsInternalChannel()
- Log failed HTTP pairing auth attempts for security monitoring
- Pass senderID to HttpClient for authenticated HTTP requests
2026-03-16 20:09:44 +07:00
hoangvinh14 a44dbf2ba4 feat(providers/dashscope): add Qwen 3.5 series support with per-model thinking capability (#215)
* feat(providers/dashscope): guard enable_thinking injection by per-model capability check
Introduces ModelThinkingCapable interface and ModelSupportsThinking field
on ChatRequest so DashScope can skip thinking-param injection for models
that do not support it (e.g. qwen3-plus, qwen3-turbo), preventing
\"model not supported\" API errors.
- types.go: add ModelThinkingCapable interface + ModelSupportsThinking *bool on ChatRequest
- dashscope.go: add dashscopeThinkingModels whitelist + ModelSupportsThinking(); honour pre-computed hint
- loop.go: detect ModelThinkingCapable and set hint on ChatRequest before LLM call
- provider_models.go: add qwen3.5-plus / qwen3.5-turbo to DashScope model list
- dashscope_test.go: full test suite for whitelist, injection, hint override, budget mapping

* Fix review code.

---------

Co-authored-by: Nguyen Gia Hoang Vinh <vinhngh@runsystem.net>
2026-03-16 07:43:08 +07:00
viettranx d5ede64595 fix(teams): workspace file view/download, scope filter, storage depth
- Workspace binary files: use HTTP /v1/files/ endpoint for blob fetch
  and download instead of WS text content (fixes corruption). Skip WS
  readFile for non-text files, set metadata only for ImageViewer.
- Workspace scope filter: prefix chatID as top-level folder in unscoped
  listing so scope switching shows correct tree structure.
- Workspace WS ops: strip chatID prefix from file_name before WS calls
  (backend already scopes by chat_id param, was causing doubled path).
- Storage depth: record dirs beyond maxDepth with hasChildren flag before
  SkipDir so they appear in tree with lazy-load indicator.
- Remove unused team_workspace_files UNION from ListTaskScopes query.
2026-03-16 07:16:46 +07:00
viettranx 84b1b07634 refactor(config): centralize hardcoded ~/.goclaw paths via config resolution
Replace all hardcoded ~/.goclaw path constructions with configurable
sources (cfg.ResolvedDataDir() for service dirs, cfg.Agents.Defaults.Workspace
for agent workspaces). This fixes data persistence issues in Docker
deployments where paths differ from local dev.

- Add DataDir field to Config with ResolvedDataDir() resolver
- Add ResolvedDataDirFromEnv() package-level helper for packages without Config
- Populate StoreConfig.SkillsStorageDir (was never set, caused hardcoded fallback)
- Agent workspaces now use subdirectory format (workspace/{key}) for volume compatibility
- Remove dead GOCLAW_SESSIONS_STORAGE env/config (sessions moved to PostgreSQL)
- Fix deploy-stg.sh trailing space after backslash + remove deprecated GOCLAW_MODE
- Add GOCLAW_SKILLS_DIR override in docker-compose for volume persistence
2026-03-15 21:20:46 +07:00
Goon 75c570e951 feat(security): credentialed exec + HTTP RBAC + API key cache (#197)
- Secure CLI credential injection via AES-256-GCM encrypted env vars
- API key management with fine-grained RBAC scopes
- resolveAuth/requireAuth middleware across all 25+ HTTP handlers
- In-memory API key cache with TTL, negative caching, pubsub invalidation
- Sandbox-first execution (fails if unavailable, no silent fallback)
- Credential scrubbing, constant-time token comparison, Admin-only CLI creds
- SQL migration 000020: secure_cli_binaries + api_keys tables
- 14 unit tests for cache and RBAC with race detector

Closes #197
2026-03-15 20:13:18 +07:00
viettranx 3cfe31523c feat(teams): redesign teams list with card/list toggle, agent emoji, and member enrichment
- Add card/list view toggle on teams page (matching agents page pattern)
- Card view: member chips with emoji/Bot icon, frontmatter, crown for lead
- List view: comma-separated member names with frontmatter tooltip on hover
- Show version badge from team settings in both views
- Enrich ListTeams API to bulk-fetch members with emoji from other_config
- Add agent emoji field (other_config.emoji) to create/update forms
- Show emoji in team members dialog instead of Bot icon when available
- Force restrict_to_workspace=true system-wide, remove UI toggle
- Add i18n keys for all 3 locales (en/vi/zh)
2026-03-15 18:04:02 +07:00
viettranx 8053acfe8c fix(storage): fix subtree lazy-loading and preserve workspace media files
- Fix storage file listing to skip subtree root when ?path= is used,
  preventing duplicate folder nesting in the file browser tree
- Comment out os.Remove calls in persistMedia to keep original workspace
  files after persisting to media store
2026-03-15 17:26:09 +07:00
Viet Tran 9a9744077e refactor(teams): v2 system cleanup — remove legacy tools, fix followup, add events API (#210)
Major refactoring of the team system with multiple improvements:

## Removed legacy delegation tools
- Delete `delegate.go`, `delegate_async.go`, `delegate_sync.go`, `delegate_events.go`,
  `delegate_policy.go`, `delegate_prep.go`, `delegate_state.go`, `delegate_search_tool.go`
- Delete `evaluate_loop_tool.go`, `handoff_tool.go`
- Remove all references and registrations from tool manager and policy
- Clean up TEAM_PLAYBOOK_IDEAS.md and TEAM_SYSTEM.md (moved to docs)

## Rename await_reply → ask_user
- Rename action `await_reply` → `ask_user`, `clear_followup` → `clear_ask_user`
- Rename functions `executeAwaitReply` → `executeAskUser`, `executeClearFollowup` → `executeClearAskUser`
- Update system prompt with stronger wording to prevent model misuse
- Model was confusing "await_reply" with general waiting; "ask_user" is unambiguous

## Fix auto-followup false positives
- Add `HasActiveMemberTasks(ctx, teamID, excludeAgentID)` store method
- Guard `autoSetFollowup()` in consumer: skip when lead has active member tasks
- Prevents auto-followup when lead is orchestrating teammates (not waiting for user)

## Task identifier zero-padding
- Change format from `T-1-xxxx` → `T-001-xxxx` (3-digit minimum)

## Refactor workspace WS handlers to filesystem-only
- Rewrite `teams.workspace.list/read/delete` to use pure filesystem (os.ReadDir/ReadFile/Remove)
- Remove DB dependency from workspace WS handlers
- Consistent with storage handler and workspace tools
- Simplify TeamWorkspaceFile type and frontend hook

## Add team events listing API
- New WS method `teams.events.list` with team_id, limit, offset params
- New HTTP endpoint `GET /v1/teams/{id}/events` with bearer auth
- New `ListTeamEvents(ctx, teamID, limit, offset)` store method
- JOIN with team_tasks for team-wide event filtering

## Extract team access policy
- New `team_access_policy.go` — centralized team tool access control

## Migration 000019: team_id columns
- Add team_id foreign key columns to relevant tables

## Other improvements
- Add team_id propagation through agent loop, tracing, sessions
- Update i18n locale files (en/vi/zh) for new tool labels
- Update frontend builtin-tools page and require-setup component
- Bump RequiredSchemaVersion for migration 000019
2026-03-15 14:53:19 +07:00
viettranx 28fab9507a feat(storage): add lazy folder loading, SSE size endpoint, and enhanced file viewer
- Backend: depth-limited WalkDir (max 3 levels default) with on-demand subtree loading
- Backend: new GET /v1/storage/size SSE endpoint with 60min in-memory cache
- Backend: raw binary file serving (?raw=true) with MIME detection and download support
- Frontend: lazy tree expansion with loading spinners for deep folders
- Frontend: streaming size display with cache info tooltip
- Frontend: image viewer (blob URL), unsupported file UI, download button, colored size badges
- Frontend: file-type icons for 13 categories (md, json, yaml, images, video, etc.)
- Fix sidebar connection status text overflow on collapse
- Apply go fix modernization (interface{} → any) across http handlers
2026-03-14 18:13:52 +07:00
Goon 5e2fa395c7 feat(providers): add ACP provider for external coding agents (#190)
* feat(providers): add ACP provider for orchestrating external coding agents (#189)

Implement native Go ACP (Agent Client Protocol) client as a new Provider.
Enables GoClaw to orchestrate any ACP-compatible agent (Claude Code, Codex
CLI, Gemini CLI) as a subprocess via JSON-RPC 2.0 over stdio.

- Add bidirectional JSON-RPC 2.0 transport over stdio pipes
- Add subprocess process pool with idle TTL reaping and crash recovery
- Add ACP session lifecycle (initialize, session/new, session/prompt)
- Add tool bridge for agent-initiated fs/terminal/permission requests
- Add workspace sandboxing, shell deny patterns, and env var filtering
- Wire config-based and DB-based provider registration paths
- Export DefaultDenyPatterns from tools package for reuse

* feat(providers): add changelog entry for ACP provider integration

* fix(tools): prevent workspace traversal bypass via /tmp/ fallback in resolveMediaPath

Reject paths containing ".." in the isInTempDir fallback to prevent
workspace escape where traversal path still resolves inside /tmp/.

* fix(tools): block workspace-sibling paths in resolveMediaPath /tmp/ fallback

When workspace is inside /tmp/, traversal paths like workspace/../X
resolve to /tmp/ siblings that pass isInTempDir. Reject paths inside
the workspace parent directory to prevent this escape.

* feat(providers): add ACP provider web UI and live reload via pubsub

Web UI for creating/editing ACP providers with dedicated form fields
(binary, args, idle TTL, permission mode, work directory). ACP providers
now update immediately without gateway restart via cache invalidation
pubsub pattern.

Frontend:
- New ACPSection form component with i18n (en/vi/zh)
- Provider form dialog integration with ACP state management
- ACP type badge on providers list page
- Settings field added to provider TypeScript types

Backend:
- ACP models handler (claude/codex/gemini) without API key requirement
- Binary path validation + LookPath verification in verify handler
- Provider CRUD emits cache.invalidate events via msgBus
- Subscriber in gateway_managed.go re-registers ACP providers from DB
- ACP core improvements from code review (helpers, jsonrpc, process,
  terminal, tool_bridge)

---------

Co-authored-by: viettranx <viettranx@gmail.com>
2026-03-14 16:16:08 +07:00
viettranx 9115169c03 feat: expand audit logging via pub/sub event pattern
Replace direct ActivityStore injection with event-driven audit system.
Handlers emit audit events via msgBus.Broadcast(), a single subscriber
with buffered channel persists to activity_logs table.

Coverage expanded from 3 agent CRUD actions to ~65 audit points across
all HTTP handlers and WebSocket RPC methods including agents, providers,
skills, MCP servers, cron, sessions, teams, pairing, and more.
2026-03-12 18:34:56 +07:00
viettranx 225bd3d3b6 feat(ui): add agent and channel dropdown filters to traces page 2026-03-12 10:38:20 +07:00
Viet Tran ace07509b7 feat(skills): system skills integration — toggle, dep checking, per-item install (#161)
* feat(infra): add runtime package support for skills

Install nodejs, npm, pandoc, github-cli + pre-install Python packages
(openpyxl, pandas, python-pptx, markitdown) and Node packages
(docx, pptxgenjs). Configure runtime dirs for agent pip/npm installs
with PIP_TARGET, NPM_CONFIG_PREFIX, NODE_PATH to enable dynamic
package installation in read-only container environment.

* feat(infra): add bundled skills with runtime package support

- Add 5 bundled skills: docx, pdf, pptx, xlsx, skill-creator from container skills-store
- Wire GOCLAW_BUILTIN_SKILLS_DIR env var in gateway and CLI
- Support optional runtime packages alongside dynamic skill loading
- Update Dockerfile to COPY bundled-skills at /app/bundled-skills/
- Add PIP_CACHE_DIR in docker-entrypoint.sh for clean pip installs
- Document bundled skills in 14-skills-runtime.md section 6

* feat(infra): remove ai-multimodal skill directory from bundled skills

Remove the ai-multimodal skill package as part of consolidating runtime
package support for bundled skills. This directory is no longer needed
in the bundled skills structure.

* feat(ci): add semantic release and Docker Hub publishing

Add go-semantic-release workflow to auto-create semver tags on merge to
main. Extend docker-publish to push all variants to both GHCR and
Docker Hub (digitop/goclaw).

* feat(skills): add system skills infrastructure with is_system column, dep scanning, and seeder

- Migration 000017: add is_system boolean column with partial index
- Store layer: UpsertSystemSkill, delete protection, IsSystemSkill
- ListAccessible auto-includes system skills (no grants needed)
- ListWithGrantStatus returns is_system field
- Dependency scanner: auto-detect deps from scripts/ or skill-manifest.json
- Dependency checker: verify system binaries, Python/Node packages
- Seeder: seed bundled skills into DB on startup (idempotent via hash)
- Gateway wiring: GOCLAW_BUNDLED_SKILLS_DIR env for bundled skills
- HTTP: delete guard (403), slug conflict check (409), rescan-deps endpoint
- UI: System badge, hide delete for system skills, rescan deps button
- Agent skills tab: "Always available" for system skills
- i18n: en/vi/zh keys for system skills, deps scanning

* feat(skills): conditional system prompt, skill manifests, and Zip Slip fix

- System prompt: only show package list when python3/node are available
- Add skill-manifest.json for pdf, docx, xlsx, pptx bundled skills
- Fix Zip Slip vulnerability in office/unpack.py (all 3 copies)

* refactor(skills): extract shared office code to _shared/ and deduplicate

Move office scripts (pack, unpack, validate, schemas, validators) from
duplicated copies in docx/xlsx/pptx to skills/_shared/office/ with
symlinks. Remove soffice.py (non-functional in containers) and update
SKILL.md references to use soffice binary directly. Update seeder
copyDir to follow symlinks.

Removes ~45K lines of duplicate code across 3 skills.

* fix(skills): address code review findings for system skills integration

- H1: Remove dead symlink branch in copyDir (filepath.Walk follows symlinks)
- H3: Fix rescan-deps to query ALL skills (including archived) and re-activate
  when deps become available; add ListAllSkills() + Status field to SkillInfo
- H4: Add Status field to SkillCreateParams, stop overloading Visibility
- M1: Batch Python/Node dep checks into single subprocess per runtime
- M4: Add rows.Err() check in ListSkills to prevent caching partial results

* feat(skills): async dep checking with realtime WS events

Split Seed() into sync DB upsert + async CheckDepsAsync() goroutine.
Gateway startup no longer blocks on Python/Node subprocess dep checks.

- Seed() returns seeded skills list, all initially status="active"
- CheckDepsAsync() runs in background, emits skill.deps.checked per-skill
- skill.deps.complete event emitted when all checks finish
- Each failed dep check: archives skill + BumpVersion() for immediate
  cache invalidation so next agent turn picks up the change
- UI: use-query-invalidation listens to skill.deps.* events → auto-refresh
  skills list in realtime

* feat(skills): system skills integration with toggle, dep checking, and per-item install

- Add is_system, deps, enabled columns to skills table (migration 017)
- Seed bundled core skills (pdf, docx, pptx, xlsx, skill-creator) on startup
- PYTHONPATH-based dep detection — eliminates false positives from local modules
- Per-item dep install UI with individual status (installing/success/error)
- Enable/disable toggle for core and custom skills (independent of dep status)
- Re-run dep check when skill is toggled back on
- Inline skill thresholds: 40 skills / 5000 tokens before switching to search mode
- Fix UpsertSystemSkill: backfill null file_hash without bumping DB version
- Remove redundant skill-manifest.json files (replaced by deps JSONB column)
- Show author from frontmatter in custom skills tab
- Runtime checker for python3/pip3/node/npm availability
- WS events for dep checking/installing progress
- docs: add 15-core-skills-system.md, 16-skill-publishing.md

---------

Co-authored-by: Goon <duy@wearetopgroup.com>
2026-03-12 09:20:41 +07:00
viettranx 6185948cb3 feat: add trace export as gzip with recursive sub-trace collection
Add GET /v1/traces/{traceID}/export endpoint that recursively collects
trace, spans, and child traces (delegations) up to depth 10, returning
gzip-compressed JSON. Add download button with icon+text in trace detail
dialog header.
2026-03-11 19:55:42 +07:00
Goon c25e770d43 feat(ui): multi-skill upload with client-side validation (#149)
* feat(ui): multi-skill upload with client-side validation

Allow uploading multiple skill ZIP files at once with pre-upload
validation. JSZip parses each ZIP client-side to verify SKILL.md
presence, frontmatter format, and slug validity before upload.

- Add JSZip dependency (lazy-loaded, code-split ~30KB gzip)
- Create validate-skill-zip.ts mirroring server-side checks
- Rewrite skill-upload-dialog for multi-file with status badges
- Add concurrent validation, sequential upload with per-file progress
- Add empty SKILL.md check to backend upload handler
- Add i18n keys for all new UI strings (en/vi/zh)

* fix(ui): duplicate entries and validation hang in multi-skill upload

- Move pending list construction to assignment inside updater return
  to prevent StrictMode double-invoke from pushing duplicates
- Wrap per-file validateSkillZip in try/catch so one failure doesn't
  block Promise.all and leave entries stuck in "validating" state

* fix(ui): use static import for JSZip instead of dynamic import

Dynamic import("jszip") fails in browser - bare module specifiers
don't resolve at runtime. Use static import which Vite handles
via its module graph and code-splits automatically.

* feat(ui): add inline visibility toggle on skills table

Click the visibility badge on managed skills to cycle through
private → internal → public. File-based skills stay read-only.

* fix(ui): move dedup logic outside state updater in upload dialog

Avoids reading stale entries inside functional updater. Builds
pending list from current entries state before calling setEntries.

* fix(ui): auto-select first active agent when current agent unavailable

When agents load from API, if the current selected agent is not in the active agents list, automatically select the first available active agent instead of remaining unset. Prevents chat page from being unable to send messages when default agent selection is invalid.

* feat(ui): make agent display name editable in setup wizard

Allow users to customize the agent display name during onboarding instead of keeping it hardcoded to "GoClaw". Removed read-only state from the display name input and added a placeholder for guidance.

* feat: add document path enrichment and media filename support

Backend changes:
- enrichDocumentPaths() in agent/media.go: injects persisted file paths into <media:document> tags
- Document paths allow skills (e.g. pdf skill via exec) to access files directly
- chat.go: support new media format {path, filename} alongside legacy string paths
- Updated read_document tool description to guide agent on using path attribute
- Docker: add pypdf to Python dependencies for PDF processing
- Softened MUST language in read_* tool descriptions (changed to Call this)

Frontend changes:
- chat-input.tsx: attach filename with each uploaded file in media payload
- use-chat-send.ts: send media as {path, filename} objects instead of just paths
- i18n: add "uploaded_files" text in en, vi, zh locales
- chat-page.tsx: minor adjustment for media handling

Enables skills to process uploaded documents directly without intermediate copying.
2026-03-11 16:59:03 +07:00
pdtkts 44e5d663af fix: respect API base when listing Anthropic models (#151) 2026-03-11 16:55:11 +07:00
Viet Tran 73389d2715 fix(ui): align usage data contracts, add timezone setting, and fix empty usage page (#146)
- Fix 6 data contract mismatches between Go backend JSON tags and React
  frontend TypeScript interfaces (field renames, response envelope changes)
- Add timezone selector to topbar with 12 common timezone options
- Replace date-fns formatting with native Intl.DateTimeFormat for
  timezone-aware chart labels (reduces bundle ~20KB)
- Add missing SnapshotTimeSeries fields (memory_docs, memory_chunks,
  kg_entities, kg_relations) that caused empty usage page
- Add error banner to usage page for API error visibility
- Sanitize backend error messages in usage HTTP handlers
- Add batch chunking (max 3000 rows) for snapshot upserts
- Remove userId display from topbar
- Add usage analytics i18n strings for en/vi/zh
2026-03-11 14:22:03 +07:00
Viet Tran 0926d053b0 feat: add token usage tracking, cost analytics, budget enforcement, wake API, and activity audit trail (#142)
- A1+C2: Include token usage in run.completed event payload for WS clients
- A2: Cost tracking with model pricing config, cost calculation, and cost summary API
- A3: Budget enforcement per agent with monthly budget limits (migration 000015)
- C1: External wake/trigger API (POST /v1/agents/{id}/wake) for orchestrators
- C3: Activity audit trail with structured logging and queryable API
- UI: Activity page, cost stat card on overview, budget section in agent detail
- i18n: Complete en/vi/zh translations for all new features
2026-03-11 12:52:12 +07:00
viettranx b7f4082145 feat(mcp): add agent count column and mask sensitive values in form
- Add agent_count to MCP server list API via CountAgentGrantsByServer query
- Show Agents column in MCP servers table
- Mask sensitive header values (Authorization, API keys) and env vars
  (keys containing secret/token/password) in create/edit form
2026-03-10 21:28:03 +07:00
viettranx bec670ead0 fix(pending-messages): run compaction in background, return 202 Accepted
LLM summarization (30-120s) was blocking HTTP response causing browser
timeouts. Now runs in goroutine and returns immediately. UI polls every
5s until completion.
2026-03-10 20:33:16 +07:00
viettranx 4fce73198d feat(agents): contact search in instances tab with auto profile creation
Add searchable contact dropdown in instances sidebar to find channel
contacts and add them as agent instances. Backend EnsureUserProfile
creates user_agent_profiles row on demand when admin adds contacts.
2026-03-10 18:46:44 +07:00
viettranx 9181eebcea fix(channels): defer media download, fix compaction status & context cancel
- Defer Telegram media download until after mention gate — pending
  history now uses lightweight tags (no download) saving bandwidth
  and avoiding errors on large files
- Fix compaction status query: has_summary now false when new messages
  arrive after last compaction, re-enabling the compact button
- Fix HTTP compact endpoint: detach context from request so LLM
  summarization isn't cancelled when browser closes connection
2026-03-10 17:12:13 +07:00
viettranx f1953203c4 feat(contacts): batch resolver API, shared hooks & contact integration across agent UI
- Add GetContactsBySenderIDs batch lookup to ContactStore (DISTINCT ON sender_id)
- Add GET /v1/contacts/resolve?ids= endpoint (max 100 IDs)
- Extract useContactPicker hook from managers tab (DRY refactor)
- Create useContactResolver hook for batch ID→name resolution via React Query
- Agent Shares tab: replace Input with Combobox contact picker + resolve names in list
- Agent Instances tab: resolve user_id to contact name as fallback
- Update i18n placeholders for contact search (en/vi/zh)
2026-03-10 16:56:17 +07:00
viettranx c8dc9917fe feat(contacts): channel contacts table, auto-collector, contacts page & managers tab redesign
- Add channel_contacts migration (000014) with UNIQUE(channel_type, sender_id)
- Add ContactStore interface with UPSERT, list, count, merge operations
- Add ContactCollector with 30-min TTL cache to skip redundant DB writes
- Wire auto-collection into gateway consumer on every inbound message
- Add GET /v1/contacts API with pagination, search, channel_type & peer_kind filters
- Rename Writers tab → Managers tab (UI-only; backend routes unchanged)
- Extract InlineAddForm with scoped state, debounce cleanup, aria-expanded
- Add Combobox contact picker with debounced search + auto-fill
- Add Contacts page with server-side pagination, filters, i18n (en/vi/zh)
- Add shared ChannelContact type, sidebar nav entry, route & query keys
- Fix ILIKE wildcard escape, log CountContacts errors, extract shared type
2026-03-10 14:29:01 +07:00
viettranx 1ddd2e547a feat(compaction): configurable provider, model & max_tokens for pending message compaction
Add Provider, Model, MaxTokens to PendingCompactionConfig so users can
override the LLM used for pending message summarization via the config
UI. Falls back to agent's provider/model when not set. Increase default
max_tokens from 512 to 4096. Add allowEmpty prop to ProviderModelSelect
to prevent auto-selecting first provider when empty means "use default".
2026-03-10 14:27:15 +07:00
viettranx 3b6bf645f3 feat(channels): pending message compaction — fix provider, wire auto-compact, add global config & UI
- Fix compact endpoint using random provider instead of agent's configured provider+model
- Wire auto-compaction for all 5 channel types (telegram, discord, slack, feishu, zalo_personal)
  via PendingCompactable interface and InstanceLoader
- Add global PendingCompactionConfig (threshold, keep_recent) to ChannelsConfig
- Wire global config through InstanceLoader and PendingMessagesHandler
- Increase compaction timeout from 45s to 180s for slow providers
- Add pending compaction config card to Behavior tab in config page
- Add HowItWorksCard (expanded by default) and toast notifications to pending messages page
- Add i18n support for all new strings (en/vi/zh)
2026-03-10 12:49:12 +07:00
viettranx 4df60649e5 feat(channels): real LLM compaction for web UI compact endpoint
Extract CompactGroup() from runCompaction() for shared use by both
auto-compact and HTTP compact endpoint. Wire provider registry into
PendingMessagesHandler so /v1/pending-messages/compact performs actual
LLM summarization (keep 15 recent + summary) instead of hard delete.

Add rows-affected guard in Compact() to prevent duplicate summaries
when concurrent compactions race on the same key.
2026-03-10 00:29:48 +07:00
viettranx 344e2ac7d1 feat(i18n): add full i18n support for backend and web UI
- Add i18next + react-i18next with namespace-split locale files (27 namespaces x 3 languages)
- Add language switcher in topbar (EN/VI/ZH) with localStorage persistence
- Replace hardcoded strings in 160+ React components with t() translations
- Add Go message catalog (internal/i18n) with T(locale, key, args...) function
- Replace 81 hardcoded error strings in gateway methods and HTTP handlers
- Add locale context propagation: WS connect param + HTTP Accept-Language header
- Keep technical terms in English: Agent, Session, Channel, Provider, Skill, Team, MCP, Cron
- Update CLAUDE.md and review-pr skill with i18n compliance checks
2026-03-09 22:22:42 +07:00
viettranx 63eff188ad feat(kg): add knowledge graph with LLM extraction, traversal, and graph visualization
- KnowledgeGraphStore interface + PostgreSQL implementation (recursive CTE traversal, 5s timeout)
- LLM entity extraction pipeline triggered on memory writes (background goroutine)
- knowledge_graph_search agent tool with search + traversal modes
- HTTP API: CRUD entities, traverse, extract, stats, graph endpoints
- Web UI: KG tab on memory page with table/graph toggle, entity detail, manual extraction
- Force-directed graph visualization using @xyflow/react + d3-force
- Builtin tool seed with configurable provider/model/confidence settings
2026-03-09 17:11:20 +07:00
Nam Nguyen Ngoc 11bed0cc01 fix(mcp-bridge): per-session security context + media forwarding (#91)
* fix(mcp-bridge): add per-session agent context and HMAC verification

- Add per-session MCP config with X-Agent-ID/X-User-ID headers instead
  of shared global config file
- Sign bridge context headers with HMAC-SHA256 to prevent forgery
- Add bridgeContextMiddleware to verify signatures on MCP bridge requests
- Store MCP configs in ~/.goclaw/mcp-configs/ outside agent workDir
- Use atomic writes (tmp + rename) for MCP config files
- Fix provider rename leaving ghost registry entries
- Remove provider_type from mutable fields on update
- Tighten temp dir permissions from 0755 to 0700

* feat(mcp-bridge): propagate channel routing context through MCP bridge

- Pass channel, chat_id, and peer_kind from agent loop to CLI provider options
- Inject X-Channel, X-Chat-ID, X-Peer-Kind headers in bridge context middleware
- Add BridgeContext struct to bundle per-call context for MCP config generation
- Include channel routing headers in per-session MCP config files
- Expose "message" tool via MCP bridge for cross-channel messaging
- Add extract helpers for new option keys in claude_cli_session.go

* feat(mcp-bridge): forward media attachments to outbound message bus

- Wire MessageBus into gateway server and MCP bridge handler
- Publish tool result media files to outbound bus for channel delivery
- Extract channel/chatID/peerKind from tool context for proper routing
- Add mimeFromExt helper for content-type detection on attachments

* feat(mcp-bridge): inject per-agent DB-backed MCP servers into Claude CLI config

- Add MCPServerLookup type to resolve agent-specific MCP servers from DB
- Wire MCPServerStore through provider registration and HTTP handler
- Extract mcpServerEntryToConfig helper to deduplicate transport config logic
- Add JSON-to-Go helpers (jsonToStringSlice, jsonToStringMap) for DB fields
- Merge per-agent MCP servers at config write time without overriding static entries

* fix(mcp-bridge): use Media struct fields and prefer explicit MimeType

- Map Media.Path to attachment URL instead of treating Media as string
- Use Media.MimeType when available, fall back to extension-based detection

* refactor(providers): deduplicate option extractors and extract bridge media forwarding

- Replace per-field extractors (extractSessionKey, extractAgentID, etc.) with generic extractStringOpt/extractBoolOpt
- Add bridgeContextFromOpts helper to build BridgeContext in one call
- Extract forwardMediaToOutbound from inline block in makeToolHandler
- Change NewBridgeServer msgBus param from variadic to explicit pointer

* fix(providers): validate provider_type on update instead of silently dropping it

- Add explicit validation against ValidProviderTypes with 400 response
- Remove silent delete(updates, "provider_type") that hid invalid values
- Caller now receives clear error when submitting unsupported provider_type

* fix(providers): add header injection validation to MCP bridge headers

- Extend CRLF/null-byte checks to agentID, channel, chatID, and peerKind
- Previously only userID had header injection prevention
- Prevents HTTP header injection via crafted values in MCP config

* fix(mcp-bridge): sign all context fields in HMAC and remove legacy code

- Sign all 5 bridge context fields (agentID|userID|channel|chatID|peerKind)
  in HMAC instead of only agentID|userID to prevent channel routing forgery
- Propagate context.Context into MCPServerLookup to respect request
  cancellation instead of using context.Background()
- Remove legacy BuildCLIMCPConfig, WithClaudeCLIMCPConfig, mcpConfigPath,
  and mcpCleanup (dead code since system is PG-only)
- Use mime.TypeByExtension before custom fallback in mimeFromExt
- Add debug log when media forwarding is skipped due to missing context
- Add thread-safety comment to SetMCPServerLookup

---------

Co-authored-by: Nam Nguyen Ngoc <namnn.0911@gmail.com>
Co-authored-by: viettranx <viettranx@gmail.com>
2026-03-09 15:23:56 +07:00
viettranx b284f963f5 feat(memory): add memory management page with CRUD, search, and indexing
Add full-stack memory document management:
- Backend: extend MemoryStore with admin queries (ListAllDocumentsGlobal,
  GetDocumentDetail, ListChunks), HTTP handler with auth middleware
- Frontend: memory page with agent/scope filters, document table with
  pagination, view/edit dialog with content and chunks tabs, create dialog
  with scope selection, semantic search dialog
- UI fixes: reduce input/textarea focus ring width, prevent ring clipping
  in dialog scroll containers, widen memory dialogs on desktop
2026-03-09 14:54:41 +07:00
viettranx 5f7ca84876 feat(channels): persist pending messages to PostgreSQL with Web UI
- Add channel_pending_messages table with UUID v7 PK, sender_id tracking
- Implement PendingMessageStore interface with batched flush (3s/20 msgs)
- Add LLM-based auto-compaction when entries exceed threshold (50)
- Wire persistent history into all channel factories (Telegram, Discord, Slack, Feishu, Zalo)
- Extract channel type constants (TypeTelegram, TypeDiscord, etc.) to eliminate magic strings
- Add HTTP API endpoints for pending messages management (list, view, compact, clear)
- Add Pending Messages dashboard page with group titles resolved from session metadata
- Track sender_id across entire pipeline (migration → store → history → handlers)
2026-03-09 12:39:43 +07:00
viettranx e85624ce96 refactor: split large Go files into smaller modules for maintainability
Split 11 Go files exceeding 500 lines into smaller, focused files:
- channels/manager.go → manager.go + dispatch.go + runs.go + events.go
- channels/slack/channel.go → channel.go + send.go + utils.go
- channels/slack/handlers.go → handlers.go + handlers_mention.go + handlers_files.go
- channels/telegram/handlers.go → handlers.go + handlers_utils.go
- channels/zalo/personal/channel.go → channel.go + send.go + listen.go + handlers.go
- gateway/methods/agents.go → agents.go + agents_create.go + agents_update.go + agents_delete.go
- http/summoner.go → summoner.go + summoner_regenerate.go + summoner_prompts.go + summoner_utils.go
- http/skills.go → skills.go + skills_upload.go + skills_versions.go + skills_grants.go
- http/mcp.go → mcp.go + mcp_tools.go + mcp_grants.go + mcp_requests.go
- store/pg/cron.go → cron.go + cron_crud.go + cron_update.go + cron_exec.go

No logic changes — pure file reorganization to keep files under 200 lines.
2026-03-09 10:41:54 +07:00
Duc Nguyen 137a986d4f feat(channels): add Slack channel (#83)
* feat(channels): add Slack channel via Socket Mode (#37)

Implement Slack integration using Socket Mode (xapp-/xoxb- tokens):
- Event-driven messaging via app_mention + message events
- Policy checks: open, pairing, allowlist, disabled (DM + group)
- Thread participation with configurable TTL
- Markdown-to-mrkdwn formatting pipeline
- Streaming support (edit-in-place + native ChatStreamer)
- SSRF-protected file downloads
- Debounce, dedup, reactions, group history context
- 170 unit tests (format, helpers, stream, SSRF)

Fix BaseChannel.HandleMessage allowlist to also check chatID,
enabling group allowlist with channel IDs across all channels.

Closes #37

* feat(slack): add file/media support and edit-to-mention handling

- Wire inbound file download into handleMessage (images, audio, documents)
- Add media.go with resolveMedia, classifyMime, buildMediaTags
- Extract shared ExtractDocumentContent to channels/media_utils.go (DRY with Telegram)
- Support file_share and message_changed subtypes
- Handle edit-to-mention: respond when user edits old message to add @bot
- Add MediaMaxBytes config field (default 20MB)
- Fix debounce media accumulation (was silently dropping files)
- Add 60s HTTP client timeout on file downloads
- Refactor downloadFile signature for slack.File compatibility
2026-03-09 07:02:37 +07:00
viettranx 01d75ac7fe refactor(tools): migrate read_* tools to provider chain and add media models
Refactor read_image, read_document, read_video, read_audio to use
ResolveMediaProviderChain + ExecuteWithChain for consistent fallback behavior.
Add hardcoded model lists for MiniMax, DashScope, and Suno providers.
2026-03-08 20:10:10 +07:00
viettranx 62c28e1b3e feat(channels): unified media pipeline for Discord, Feishu, and WebSocket
Extract shared media utilities (MediaInfo, BuildMediaTags, TranscribeAudio,
DetectMIMEType) into internal/channels/media/ and refactor Telegram to use
them. Add full inbound/outbound media support to Discord and Feishu channels
(STT transcription, document extraction, media tags, voice agent routing).
Add WebSocket media upload/serve endpoints and MIME-aware media tags in
chat.send. Split large channel files for maintainability.
2026-03-08 16:39:46 +07:00
viettranx 47cc11bfc0 feat(metadata): add JSONB metadata to sessions, profiles, and pairing
Persist friendly names (display_name, username, chat_title) from channel
handlers into sessions, user profiles, and pairing records. Web UI renders
metadata with graceful fallback to raw IDs.

- Add migration 000011: metadata JSONB columns on sessions,
  user_agent_profiles, pairing_requests, paired_devices
- Extend SessionStore/AgentStore/PairingStore interfaces with metadata ops
- Extract and persist channel metadata in gateway consumer
- Extend sessions.patch and add PATCH instances metadata HTTP endpoint
- Update frontend sessions page, detail page, and instances tab
- Delete legacy file-based internal/pairing/service.go
- Update docs references to reflect DB-backed pairing
2026-03-08 15:42:44 +07:00
viettranx e1a6801a7a fix(tools): correct Veo API, media ref ordering, video tag, and model verify
- Fix create_video: use predictLongRunning API instead of generateContent
  (async polling flow: POST → poll every 10s → download video from URI)
- Fix durationSeconds as int (not string) per actual Gemini API requirement
- Fix MediaRef collection order: historical first, current last, so
  refs[len-1] always returns the most recent file (fixes read_audio
  picking up old file instead of current voice message)
- Remove misleading "video not yet supported" text from Telegram handler
  that prevented LLM from calling read_video tool
- Add isNonChatModel() to skip chat-based verify for generation models
  (veo-*, dall-e-*, imagen-*, gemini-*-image)
2026-03-08 15:21:08 +07:00
viettranx ea185b3f6c feat(agents): add self-evolution config and instances management for predefined agents
Self-Evolution: predefined agents can now optionally evolve their SOUL.md
(communication style/tone only) when self_evolve is enabled in other_config.
Identity, name, and operating instructions remain locked. Context propagation
flows through LoopConfig → Loop → context.WithValue → interceptor carve-out.
System prompt guides the agent on what it can/cannot evolve.

Instances Tab: new HTTP endpoints and UI tab for viewing/editing per-user
USER.md files on predefined agents. Includes owner-only access checks,
fileName validation (USER.md only), and cache invalidation.

UI: self-evolve toggle in General tab, create dialog, and setup wizard.
Agent type and evolve/static badges with tooltip explanations on cards
and detail header. TooltipProvider added to agents list and detail pages.
2026-03-08 14:27:40 +07:00
viettranx d9cdbb8e6e feat(storage): add workspace file browser with delete and size display
Add Storage page under Monitoring menu to browse, view, and manage
files inside ~/.goclaw/. Skills directories are visible but
deletion-protected. Extract shared file browser components from
skills page for reuse.
2026-03-08 13:18:18 +07:00
viettranx 2a4db0da94 feat(skills): add file browser, version selector, and security hardening
- Add skill file browser with tree panel and content viewer (syntax highlight per file type)
- Add version selector to browse all skill versions on disk
- Add 3 new API endpoints: GET /v1/skills/{id}/versions, files, files/{path}
- Make skill name clickable to open detail dialog (remove redundant View button)
- Filter system artifacts (__MACOSX, .DS_Store, Thumbs.db) from ZIP extraction and file listing
- Add symlink protection in extraction, file listing, and file reading
- Strengthen path traversal prevention with prefix validation
- Strip frontmatter from .md files in file viewer
- Fix double scrollbar in file content panel
2026-03-08 10:19:09 +07:00
viettranx a53f3e092f fix(media): use absolute paths and relative URLs for WS media delivery
- FilesHandler now serves files by absolute path (auth-token protected)
  instead of requiring a workspace root, supporting multiple agent workspaces
- mediaToMarkdown generates relative URLs (/v1/files/...) instead of
  absolute http://host:port URLs so images work from any client origin
- Deduplicate media collection in agent loop: prefer result.Media over
  MEDIA: prefix parsing to prevent duplicate images
2026-03-08 00:41:38 +07:00
viettranx c1962e4659 feat(skills): add edit modal, drag-and-drop upload, fix frontmatter stripping
- Add skill edit dialog (name, description, visibility, tags)
- Add drag-and-drop support to upload modal with visual feedback
- Fix frontmatter not stripped in skill detail view (CRLF normalization)
- Include visibility, tags, version in skill list/detail API responses
- Change default skill upload visibility from private to internal
- Add visibility column and version display to skills table
2026-03-08 00:16:33 +07:00