Commit Graph

25 Commits

Author SHA1 Message Date
viettranx 08a2d95c0c feat: agent heartbeat system — periodic proactive check-ins (#245)
Phase 1 (Core):
- Migration 000022: agent_heartbeats, heartbeat_run_logs, agent_config_permissions tables
- HeartbeatStore + ConfigPermissionStore interfaces with PG implementations
- HeartbeatTicker: background poll → active hours filter → queue-aware skip → run → smart suppression → deliver/log
- Heartbeat tool: status/get/set/toggle/set_checklist/get_checklist/test/logs actions
- Permission check with wildcard scope matching + TTL cache (60s)
- RPC methods: heartbeat.get/set/toggle/test/logs/checklist.get/checklist.set
- HEARTBEAT.md routed via context file interceptor (read/write for both open + predefined agents)
- Session keys: agent:{id}:heartbeat or agent:{id}💓{ts} (isolated)
- PromptMinimal for heartbeat sessions (like cron/subagent)
- Event broadcasting + cache invalidation via bus (heartbeat + config_perms)
- Gateway wiring: ticker init, event wiring, graceful shutdown

Phase 2 (Integration):
- wakeMode: CronPayload.WakeHeartbeat triggers heartbeat after cron job completes
- Queue-aware: Scheduler.HasActiveSessionsForAgent() skips busy agents
- Stagger: deterministic FNV offset spreads heartbeats across interval
- lightContext: RunRequest.LightContext skips context files, only injects checklist
- System prompt distinguishes cron (user-scheduled tasks) vs heartbeat (autonomous monitoring)
2026-03-18 13:11:44 +07:00
viettranx 0b5124a8f1 fix(security): harden pairing auth — fail-closed, rate-limit, TTL
- Change WS pairing check from fail-open to fail-closed on DB error
  (router.go: previously granted RoleOperator on any IsPaired() error)
- Add "browser" to InternalChannels so it's properly excluded from
  outbound dispatch without ad-hoc helpers
- Rate-limit browser.pairing.status endpoint to prevent sender_id
  enumeration (reuses server RateLimiter via PairingMethods injection)
- Add expires_at column to paired_devices with 30-day TTL for
  defense-in-depth; IsPaired() now checks expiry, ListPaired() prunes
- Add confidence_score column to team_tasks, team_messages,
  team_task_comments
- Bump RequiredSchemaVersion to 21
2026-03-16 19:55:08 +07:00
Goon 75c570e951 feat(security): credentialed exec + HTTP RBAC + API key cache (#197)
- Secure CLI credential injection via AES-256-GCM encrypted env vars
- API key management with fine-grained RBAC scopes
- resolveAuth/requireAuth middleware across all 25+ HTTP handlers
- In-memory API key cache with TTL, negative caching, pubsub invalidation
- Sandbox-first execution (fails if unavailable, no silent fallback)
- Credential scrubbing, constant-time token comparison, Admin-only CLI creds
- SQL migration 000020: secure_cli_binaries + api_keys tables
- 14 unit tests for cache and RBAC with race detector

Closes #197
2026-03-15 20:13:18 +07:00
Viet Tran 9a9744077e refactor(teams): v2 system cleanup — remove legacy tools, fix followup, add events API (#210)
Major refactoring of the team system with multiple improvements:

## Removed legacy delegation tools
- Delete `delegate.go`, `delegate_async.go`, `delegate_sync.go`, `delegate_events.go`,
  `delegate_policy.go`, `delegate_prep.go`, `delegate_state.go`, `delegate_search_tool.go`
- Delete `evaluate_loop_tool.go`, `handoff_tool.go`
- Remove all references and registrations from tool manager and policy
- Clean up TEAM_PLAYBOOK_IDEAS.md and TEAM_SYSTEM.md (moved to docs)

## Rename await_reply → ask_user
- Rename action `await_reply` → `ask_user`, `clear_followup` → `clear_ask_user`
- Rename functions `executeAwaitReply` → `executeAskUser`, `executeClearFollowup` → `executeClearAskUser`
- Update system prompt with stronger wording to prevent model misuse
- Model was confusing "await_reply" with general waiting; "ask_user" is unambiguous

## Fix auto-followup false positives
- Add `HasActiveMemberTasks(ctx, teamID, excludeAgentID)` store method
- Guard `autoSetFollowup()` in consumer: skip when lead has active member tasks
- Prevents auto-followup when lead is orchestrating teammates (not waiting for user)

## Task identifier zero-padding
- Change format from `T-1-xxxx` → `T-001-xxxx` (3-digit minimum)

## Refactor workspace WS handlers to filesystem-only
- Rewrite `teams.workspace.list/read/delete` to use pure filesystem (os.ReadDir/ReadFile/Remove)
- Remove DB dependency from workspace WS handlers
- Consistent with storage handler and workspace tools
- Simplify TeamWorkspaceFile type and frontend hook

## Add team events listing API
- New WS method `teams.events.list` with team_id, limit, offset params
- New HTTP endpoint `GET /v1/teams/{id}/events` with bearer auth
- New `ListTeamEvents(ctx, teamID, limit, offset)` store method
- JOIN with team_tasks for team-wide event filtering

## Extract team access policy
- New `team_access_policy.go` — centralized team tool access control

## Migration 000019: team_id columns
- Add team_id foreign key columns to relevant tables

## Other improvements
- Add team_id propagation through agent loop, tracing, sessions
- Update i18n locale files (en/vi/zh) for new tool labels
- Update frontend builtin-tools page and require-setup component
- Bump RequiredSchemaVersion for migration 000019
2026-03-15 14:53:19 +07:00
Viet Tran 1a42dc93a6 feat(teams): team system v2 with bug fixes, workspace scope, versioning, and prompt optimization (#183)
* feat(workspace): add team shared workspace for file collaboration

- Add workspace_write and workspace_read tools for agents to share files across team members
- Create team_workspaces DB table with migration 000017 (file metadata, pinning, tags)
- Implement PostgreSQL store layer for workspace CRUD operations
- Add RPC handlers for workspace list/read/delete from web UI
- Build React workspace tab with file listing, content preview, and delete
- Propagate workspace channel/chatID scope through delegation chain
- Auto-allow workspace tools in agent tool policy when agent belongs to a team
- Inject team workspace guidance into system prompt for team agents
- Add /reset command handler for clearing session history
- Harden MCP bridge context middleware to reject headers when no gateway token
- Add i18n strings for workspace UI in en/vi/zh locales

* feat(teams): add comprehensive task management with followup reminders and recovery

- Add task followup/reminder system with auto-set on lead agent reply and auto-clear when user responds on channel
- Add task recovery ticker to re-dispatch stale/pending tasks periodically
- Add task scopes, filtering by status/channel/chatID, and task events
- Add WS RPC handlers for task CRUD, assignments, comments, events, and bulk operations (teams_tasks.go)
- Add task detail dialog, settings UI for followup config, and scope filtering in web dashboard
- Add migrations 000018 (team_tasks_v2) and 000019 (task_followup)
- Extend team_tasks_tool with await_reply, clear_followup actions
- Auto-complete/fail team tasks when delegate agent finishes
- Add workspace file listing and team tool manager enhancements

* docs(teams): add team system architecture and playbook ideas documentation

- Add TEAM_SYSTEM.md with full architecture design covering task management, shared workspace, and delegation engine subsystems
- Add TEAM_PLAYBOOK_IDEAS.md outlining future team coordination layers (playbook, member capabilities, auto-learned patterns)
- Document data models, status flows, tool actions, followup reminder system, task ticker, execution locking, and workspace scope model

* fix(teams): resolve 6 critical bugs in team task system

- Fix unblock SQL: check array_length after array_remove (not before)
- Enforce single-team leadership in team creation
- Add requireLead() for approve/reject tool actions
- Validate cross-team dependency references in blocked_by
- Add team_id to handoff route for multi-team isolation
- Set blocked_by DEFAULT '{}' to prevent NULL array issues

* refactor(workspace): use stable userID as scope key instead of connection UUID

Workspace scope changed from (team_id, channel, chat_id) to (team_id, userID).
Fixes workspace fragmentation across WS tab refreshes and reconnections.

* feat(teams): add V1/V2 versioning with feature gating and optimized prompts

- IsTeamV2() helper gates advanced features (locking, followup, review, audit)
- V2 tool actions rejected for V1 teams with clear error message
- Ticker, gateway consumer, delegation hooks respect version flag
- TEAM.md renders v1/v2 sections conditionally
- Tool descriptions and params optimized (~38% token reduction)
- UI: version toggle in settings, V2 Beta badge, conditional rendering
- i18n: version modal keys for en/vi/zh

* fix(migration): use VARCHAR(255) for user ID columns and add metadata JSONB

- assignee_user_id, user_id, actor_id: TEXT → VARCHAR(255)
- Add metadata JSONB to team_task_comments and team_task_attachments

---------

Co-authored-by: Nam Nguyen Ngoc <namnn.0911@gmail.com>
2026-03-13 22:41:32 +07:00
Viet Tran ace07509b7 feat(skills): system skills integration — toggle, dep checking, per-item install (#161)
* feat(infra): add runtime package support for skills

Install nodejs, npm, pandoc, github-cli + pre-install Python packages
(openpyxl, pandas, python-pptx, markitdown) and Node packages
(docx, pptxgenjs). Configure runtime dirs for agent pip/npm installs
with PIP_TARGET, NPM_CONFIG_PREFIX, NODE_PATH to enable dynamic
package installation in read-only container environment.

* feat(infra): add bundled skills with runtime package support

- Add 5 bundled skills: docx, pdf, pptx, xlsx, skill-creator from container skills-store
- Wire GOCLAW_BUILTIN_SKILLS_DIR env var in gateway and CLI
- Support optional runtime packages alongside dynamic skill loading
- Update Dockerfile to COPY bundled-skills at /app/bundled-skills/
- Add PIP_CACHE_DIR in docker-entrypoint.sh for clean pip installs
- Document bundled skills in 14-skills-runtime.md section 6

* feat(infra): remove ai-multimodal skill directory from bundled skills

Remove the ai-multimodal skill package as part of consolidating runtime
package support for bundled skills. This directory is no longer needed
in the bundled skills structure.

* feat(ci): add semantic release and Docker Hub publishing

Add go-semantic-release workflow to auto-create semver tags on merge to
main. Extend docker-publish to push all variants to both GHCR and
Docker Hub (digitop/goclaw).

* feat(skills): add system skills infrastructure with is_system column, dep scanning, and seeder

- Migration 000017: add is_system boolean column with partial index
- Store layer: UpsertSystemSkill, delete protection, IsSystemSkill
- ListAccessible auto-includes system skills (no grants needed)
- ListWithGrantStatus returns is_system field
- Dependency scanner: auto-detect deps from scripts/ or skill-manifest.json
- Dependency checker: verify system binaries, Python/Node packages
- Seeder: seed bundled skills into DB on startup (idempotent via hash)
- Gateway wiring: GOCLAW_BUNDLED_SKILLS_DIR env for bundled skills
- HTTP: delete guard (403), slug conflict check (409), rescan-deps endpoint
- UI: System badge, hide delete for system skills, rescan deps button
- Agent skills tab: "Always available" for system skills
- i18n: en/vi/zh keys for system skills, deps scanning

* feat(skills): conditional system prompt, skill manifests, and Zip Slip fix

- System prompt: only show package list when python3/node are available
- Add skill-manifest.json for pdf, docx, xlsx, pptx bundled skills
- Fix Zip Slip vulnerability in office/unpack.py (all 3 copies)

* refactor(skills): extract shared office code to _shared/ and deduplicate

Move office scripts (pack, unpack, validate, schemas, validators) from
duplicated copies in docx/xlsx/pptx to skills/_shared/office/ with
symlinks. Remove soffice.py (non-functional in containers) and update
SKILL.md references to use soffice binary directly. Update seeder
copyDir to follow symlinks.

Removes ~45K lines of duplicate code across 3 skills.

* fix(skills): address code review findings for system skills integration

- H1: Remove dead symlink branch in copyDir (filepath.Walk follows symlinks)
- H3: Fix rescan-deps to query ALL skills (including archived) and re-activate
  when deps become available; add ListAllSkills() + Status field to SkillInfo
- H4: Add Status field to SkillCreateParams, stop overloading Visibility
- M1: Batch Python/Node dep checks into single subprocess per runtime
- M4: Add rows.Err() check in ListSkills to prevent caching partial results

* feat(skills): async dep checking with realtime WS events

Split Seed() into sync DB upsert + async CheckDepsAsync() goroutine.
Gateway startup no longer blocks on Python/Node subprocess dep checks.

- Seed() returns seeded skills list, all initially status="active"
- CheckDepsAsync() runs in background, emits skill.deps.checked per-skill
- skill.deps.complete event emitted when all checks finish
- Each failed dep check: archives skill + BumpVersion() for immediate
  cache invalidation so next agent turn picks up the change
- UI: use-query-invalidation listens to skill.deps.* events → auto-refresh
  skills list in realtime

* feat(skills): system skills integration with toggle, dep checking, and per-item install

- Add is_system, deps, enabled columns to skills table (migration 017)
- Seed bundled core skills (pdf, docx, pptx, xlsx, skill-creator) on startup
- PYTHONPATH-based dep detection — eliminates false positives from local modules
- Per-item dep install UI with individual status (installing/success/error)
- Enable/disable toggle for core and custom skills (independent of dep status)
- Re-run dep check when skill is toggled back on
- Inline skill thresholds: 40 skills / 5000 tokens before switching to search mode
- Fix UpsertSystemSkill: backfill null file_hash without bumping DB version
- Remove redundant skill-manifest.json files (replaced by deps JSONB column)
- Show author from frontmatter in custom skills tab
- Runtime checker for python3/pip3/node/npm availability
- WS events for dep checking/installing progress
- docs: add 15-core-skills-system.md, 16-skill-publishing.md

---------

Co-authored-by: Goon <duy@wearetopgroup.com>
2026-03-12 09:20:41 +07:00
Viet Tran 73389d2715 fix(ui): align usage data contracts, add timezone setting, and fix empty usage page (#146)
- Fix 6 data contract mismatches between Go backend JSON tags and React
  frontend TypeScript interfaces (field renames, response envelope changes)
- Add timezone selector to topbar with 12 common timezone options
- Replace date-fns formatting with native Intl.DateTimeFormat for
  timezone-aware chart labels (reduces bundle ~20KB)
- Add missing SnapshotTimeSeries fields (memory_docs, memory_chunks,
  kg_entities, kg_relations) that caused empty usage page
- Add error banner to usage page for API error visibility
- Sanitize backend error messages in usage HTTP handlers
- Add batch chunking (max 3000 rows) for snapshot upserts
- Remove userId display from topbar
- Add usage analytics i18n strings for en/vi/zh
2026-03-11 14:22:03 +07:00
Viet Tran 0926d053b0 feat: add token usage tracking, cost analytics, budget enforcement, wake API, and activity audit trail (#142)
- A1+C2: Include token usage in run.completed event payload for WS clients
- A2: Cost tracking with model pricing config, cost calculation, and cost summary API
- A3: Budget enforcement per agent with monthly budget limits (migration 000015)
- C1: External wake/trigger API (POST /v1/agents/{id}/wake) for orchestrators
- C3: Activity audit trail with structured logging and queryable API
- UI: Activity page, cost stat card on overview, budget section in agent detail
- i18n: Complete en/vi/zh translations for all new features
2026-03-11 12:52:12 +07:00
viettranx c8dc9917fe feat(contacts): channel contacts table, auto-collector, contacts page & managers tab redesign
- Add channel_contacts migration (000014) with UNIQUE(channel_type, sender_id)
- Add ContactStore interface with UPSERT, list, count, merge operations
- Add ContactCollector with 30-min TTL cache to skip redundant DB writes
- Wire auto-collection into gateway consumer on every inbound message
- Add GET /v1/contacts API with pagination, search, channel_type & peer_kind filters
- Rename Writers tab → Managers tab (UI-only; backend routes unchanged)
- Extract InlineAddForm with scoped state, debounce cleanup, aria-expanded
- Add Combobox contact picker with debounced search + auto-fill
- Add Contacts page with server-side pagination, filters, i18n (en/vi/zh)
- Add shared ChannelContact type, sidebar nav entry, route & query keys
- Fix ILIKE wildcard escape, log CountContacts errors, extract shared type
2026-03-10 14:29:01 +07:00
viettranx 63eff188ad feat(kg): add knowledge graph with LLM extraction, traversal, and graph visualization
- KnowledgeGraphStore interface + PostgreSQL implementation (recursive CTE traversal, 5s timeout)
- LLM entity extraction pipeline triggered on memory writes (background goroutine)
- knowledge_graph_search agent tool with search + traversal modes
- HTTP API: CRUD entities, traverse, extract, stats, graph endpoints
- Web UI: KG tab on memory page with table/graph toggle, entity detail, manual extraction
- Force-directed graph visualization using @xyflow/react + d3-force
- Builtin tool seed with configurable provider/model/confidence settings
2026-03-09 17:11:20 +07:00
viettranx 5f7ca84876 feat(channels): persist pending messages to PostgreSQL with Web UI
- Add channel_pending_messages table with UUID v7 PK, sender_id tracking
- Implement PendingMessageStore interface with batched flush (3s/20 msgs)
- Add LLM-based auto-compaction when entries exceed threshold (50)
- Wire persistent history into all channel factories (Telegram, Discord, Slack, Feishu, Zalo)
- Extract channel type constants (TypeTelegram, TypeDiscord, etc.) to eliminate magic strings
- Add HTTP API endpoints for pending messages management (list, view, compact, clear)
- Add Pending Messages dashboard page with group titles resolved from session metadata
- Track sender_id across entire pipeline (migration → store → history → handlers)
2026-03-09 12:39:43 +07:00
viettranx 47cc11bfc0 feat(metadata): add JSONB metadata to sessions, profiles, and pairing
Persist friendly names (display_name, username, chat_title) from channel
handlers into sessions, user profiles, and pairing records. Web UI renders
metadata with graceful fallback to raw IDs.

- Add migration 000011: metadata JSONB columns on sessions,
  user_agent_profiles, pairing_requests, paired_devices
- Extend SessionStore/AgentStore/PairingStore interfaces with metadata ops
- Extract and persist channel metadata in gateway consumer
- Extend sessions.patch and add PATCH instances metadata HTTP endpoint
- Update frontend sessions page, detail page, and instances tab
- Delete legacy file-based internal/pairing/service.go
- Update docs references to reflect DB-backed pairing
2026-03-08 15:42:44 +07:00
viettranx 5a4d40f458 enhance: rewrite AGENTS.md for managed mode, add Style section to SOUL.md
Remove ~60% redundant content from AGENTS.md (safety, bootstrap, tools, workspace
sections already covered by system prompt). Add Conversational Style rules to reduce
bot-like behavior. Add customizable Style section to SOUL.md template with summoner
metaprompt support. Migration 000010 force-updates all existing AGENTS.md in DB.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 18:33:23 +07:00
viettranx ed32e68c68 feat(quota): channel quota limiter + per-run tool budget + config UI
Part A — Channel quota limiter (managed mode):
- DB-backed per-user/group request quotas with in-memory 60s TTL cache
- Config merge priority: Groups > Channels > Providers > Default
- Per-group quota override via channels.telegram.groups[chatID].quota
- Migration 000009: index on channel_requests for quota queries
- Hot-reload quota config via pub/sub (TopicConfigChanged)

Part B — Per-run tool call budget:
- Soft stop at configurable limit (default 25, per-agent override)
- MaxToolCalls field on AgentDefaults + AgentSpec + LoopConfig
- LLM gets one final call to summarize when budget exceeded

Part C — Web UI + config page refactor:
- QuotaSection with provider/channel dropdowns (useProviders, useChannelInstances)
- Config page refactored to vertical sidebar tabs layout
- Categories: General, Quota, Agents, Tools, Connections, Advanced, Raw Editor
- Fixed config.patch RPC to serialize raw JSON + baseHash correctly
- Config change pub/sub broadcast from handleApply/handlePatch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 10:11:17 +07:00
viettranx e2debfe49a feat: mid-loop context compaction + team task user scoping
Add mid-loop compaction to prevent context overflow during long-running
delegated agent runs (e.g. 225K+ tokens causing DashScope timeouts).
Uses same threshold as maybeSummarize (contextWindow * historyShare)
with actual PromptTokens from LLM response. Only compacts the in-memory
messages slice; pendingMsgs preserves full history for session flush.

Add user_id/channel columns to team_tasks so end users only see their
own tasks. Delegate/system channels bypass the filter to see all tasks.
Group chats use the group-scoped UserID (group:channel:chatID) so all
members share visibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:03:28 +07:00
viettranx 0e75c21e55 feat: overhaul team delegation, Telegram resilience, and artifact forwarding
Team delegation:
- Unify spawn/subagent/delegate into single spawn tool
- Sibling-aware announce suppression with artifact accumulation
- Fix auto-complete race (isLastDelegation guard)
- Add team tasks list limit (20) with search guidance
- Multi-round orchestration patterns in TEAM.md
- Communication guidance for initial vs follow-up delegations

Telegram resilience:
- Add retrySend wrapper (3 attempts, escalating delay) for network errors
- Fix HTML fallback: strip tags + unescape entities instead of showing raw HTML
- Pre-process HTML tags in LLM output to markdown before conversion pipeline
- Skip caption truncation entirely when > 1024 bytes, send text separately
- Auto-send large images (>5MB) as documents to avoid compression

Artifact forwarding:
- Fix missing ContentType on forwarded media (mimeFromExt for result.Media/ForwardMedia)
- Add deliver parameter to write_file for file attachment delivery
- Extend mimeFromExt with document MIME types

UI: fix regenerate dialog overflow, improve task list layout, delegation detail view

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 14:34:36 +07:00
viettranx 86d58e1021 feat: Introduce a new upgrade command and enhance built-in tool settings with provider and model configuration. 2026-02-27 11:38:04 +07:00
viettranx d65d792646 feat: Implement built-in tool management with persistence, API, and UI. 2026-02-27 10:19:19 +07:00
viettranx 6066adc15a feat: Implement agent delegation, quality gates, and a new hooks evaluation system. 2026-02-26 10:15:07 +07:00
viettranx dfd91556f8 feat: Introduce agent teams, agent linking, and advanced agent orchestration features. 2026-02-25 23:24:52 +07:00
viettranx 206c8256fd refactor: merge sql file to repare to public repo 2026-02-25 08:52:47 +07:00
viettranx 16022d77be feat: Implement agent resummoning with UI retry, add provider verification, and introduce created_at timestamps to various tables. 2026-02-24 15:18:25 +07:00
Viet Tran 7b7e9a4248 Add user_id and agent_id filtering to cron jobs with token usage tracking and UUID-based agent resolution
Extend CronJob model with UserID field for multi-tenant isolation. Add agentID and userID filter parameters to ListJobs across all store implementations (file, pg). Support agent lookup by UUID in resolver for cron jobs that store agent_id as UUID. Change cron job handler signature to return CronJobResult struct with content, token usage (input/output tokens), and duration. Track execution metrics
2026-02-22 21:19:29 +07:00
Viet Tran 86b1724050 Add group file writer management for Telegram with permission-based file editing
Implement group file writer allowlist system with Telegram commands (/addwriter, /removewriter, /writers) for managing who can edit protected files in group chats. Wire AgentStore through Telegram factory, inject SenderID context for permission checks, and auto-bootstrap first group member as writer. Only existing writers can manage the list, preventing removal of the last writer.
2026-02-22 18:57:24 +07:00
Viet Tran f3f4c67b36 Initial commit: GoClaw AI agent gateway
Multi-agent AI gateway with WebSocket RPC, HTTP API, and messaging channel integrations.
Go port of OpenClaw with multi-tenant PostgreSQL, per-user isolation, security hardening,
and production observability.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 14:58:07 +07:00