Commit Graph
1209 Commits
Author SHA1 Message Date
viettranx e021934e68 fix(vault): wire provider hot-reload to config.patch event
The previous commit only listened to TopicSystemConfigChanged (periodic
DB refresh), but config.patch from UI fires TopicConfigChanged. Add
subscriber in gateway_lifecycle so vault enrichment picks up provider/
model changes immediately when user saves config.
2026-04-12 15:13:53 +07:00
viettranx ba66a7b67d fix(vault): increase max_tokens + hot-reload provider for enrichment
- Increase classify max_tokens 1024→2048, summarize 1536→4096 to
  prevent truncated JSON from models like gemini-2.5-pro
- Add debug logging: raw LLM output on parse failures, finish_reason
  truncation warnings with model name
- Hot-swap vault enrichment provider/model on config change without
  restart (wired into TopicSystemConfigChanged handler)
- Use RWMutex-guarded llm() accessor for thread-safe provider reads
2026-04-12 15:08:59 +07:00
viettranx fbfae1e618 fix(agents): prevent NOT NULL violation on promoted columns during update
Frontend sent null for empty emoji (emoji.trim() || null), which violated
the NOT NULL constraint on the promoted emoji column (migration 000037).
This caused all agent saves to fail with 500 when emoji was unset.

- Frontend: send empty string instead of null for emoji field
- Backend (PG + SQLite): add null-coercion for all promoted NOT NULL
  columns — TEXT (emoji, agent_description, thinking_level) coerce to "",
  INT (skill_nudge_interval, max_tokens) coerce to 0
2026-04-12 14:31:51 +07:00
viettranx 28d29ba046 fix(vault): enrichment pipeline reliability + cross-agent classify
9 fixes for vault enrichment pipeline:

1. Queue key = tenant-only (was per-agent, caused multiple batches
   blocking EventBus workers and progress bar flashing)
2. Classify chunks 5 candidates per LLM call (prevents response
   truncation that caused parse_still_failed errors)
3. Classify prompt improved: explicit "EXACTLY one entry per
   candidate", 5-entry example, ctx capped at 30 words
4. max_tokens kept at 1024 (sufficient for 5 candidates)
5. Progress AddDone removes !running guard (safe before Start)
6. Rescan defers event publishing via PendingEvents — Start()
   called before workers receive events, eliminating race
7. Upload handler same deferred publish pattern
8. Frontend enrichment timer cancels stale "complete" timeout
   when new enrichment starts (prevents bar disappearing)
9. Sidebar tree reloads after rescan completes

Classify now searches across entire tenant (empty agentID) to
build cross-agent links for future vault sharing. Access control
enforced at query time — agents only see their own docs.
2026-04-12 14:31:36 +07:00
viettranx bfdaeb2382 fix(vault): increase classify max_tokens to prevent truncated JSON
With 10 candidates, classify response can exceed 1024 tokens when
ctx descriptions are verbose, causing json unmarshal failures.
Bump to 2048 to accommodate worst-case 10-candidate responses.
2026-04-12 13:39:52 +07:00
viettranx 5b52ce27a8 fix(vault): aggregate enrichment progress across per-agent batches
After rescan agent_key fix, docs split into per-agent batch queues.
Each batch was calling Start()/Finish() independently, causing the
progress bar to flash 0/1 per agent instead of showing global progress.

Now: handler Start(total) sets global count, worker batches call
TrackBatch()/MarkBatchDone() for lifecycle, AddDone(n) auto-completes
when done >= total. Progress bar shows smooth 0→N across all agents.
2026-04-12 13:38:44 +07:00
viettranx d58e0366fd fix(vault): only reset enrichment progress on new batch, not mid-drain
Start() is called each drain loop in processBatch to update total.
Only reset done=0 when transitioning from idle→running (new batch),
not when already running (same batch, total growing).
2026-04-12 13:35:06 +07:00
viettranx c3d07bf6e9 fix(vault): reset enrichment progress counter on rescan start
Start() now resets done=0 before broadcasting, preventing stale
counter from previous run causing progress bar to jump/reset.
2026-04-12 13:31:40 +07:00
viettranx 7810b8b78a fix(vault): agent filter support + rescan agent_key path matching
- Add AgentID to VaultTreeOptions, pass agent_id filter through
  backend tree endpoint and frontend hook
- Fix rescan inferOwnerFromPath: match root-level {agent_key}/...
  folders against agentMap (workspace uses agent_key directly,
  not agents/ prefix). Keep legacy agents/ prefix for compat
- Preserve full relPath for DB storage in all patterns
- Show filename with extension in tree (use path basename, not title)
- Start tree folders collapsed (prevent stuck loading state)
- Remove orphaned useVaultGraphData call from vault-page
- Tenant isolation verified: agentMap built per-tenant via scopeClause
2026-04-12 13:25:15 +07:00
viettranx e043241dea fix(tools): add missing tools to goclaw group and relax agent update ownership
- Add 10 missing tools to the goclaw group in policy.go: skill_manage,
  publish_skill, use_skill, delegate, memory_expand, knowledge_graph_search,
  vault_search, create_audio, datetime, heartbeat. Fixes skill creation
  permission denied when agents use group:goclaw allow list.

- Relax agent update ownership check: tenant admins can now update any
  agent in their tenant (adminMiddleware + tenant-scoped GetByID already
  ensures proper authorization). Previously only agent owner or system
  owner could update.

- Improve agent update error logging: include user_id and tenant_id in
  structured log, return actual error message instead of generic
  "internal error" for better debugging.
2026-04-12 12:37:53 +07:00
viettranx 35ce8cc2c5 feat(vault): tree sidebar with lazy-load folder hierarchy
Replace flat paginated document list with collapsible tree view.
Backend: two-query-per-level approach (files + DISTINCT deeper paths
for virtual folder derivation), path parsing in Go for PG+SQLite
compatibility, text_pattern_ops index for prefix range scans.
Frontend: VaultTree component with doc_type colored icons, compact
single-line nodes, truncate-middle filenames, scope dots, hover
tooltip for date. Sidebar expanded to 384px on large screens.

Also fixes pre-existing SQLite scanVaultDocRow missing path_basename.
2026-04-12 12:37:10 +07:00
viettranx 31b2662ecf feat(tools): API key management in web search chain form
API keys for Exa, Tavily, and Brave can now be set directly in the
web search provider chain settings dialog. Keys are saved to
config_secrets (AES-256-GCM encrypted, tenant-scoped) and stripped
from the persisted settings blob. Raw key values are never returned
in API responses — only boolean set/unset status.

Backend:
- Add ConfigSecretsStore to BuiltinToolsHandler
- extractAndSaveSecrets: parse api_key from settings, save to
  config_secrets, strip before persisting to builtin_tool_tenant_configs
- getSecretsStatus: returns boolean map per tool (never raw values)
- Enriched handleList response with secrets_set per tool

Frontend:
- Per-provider API key input in web search chain cards
- "Key set" status badge when key exists, "Change" button to replace
- Staged keys sent with settings on save, backend extracts
2026-04-12 12:02:40 +07:00
viettranx 81cd6ef24b fix(ui): add API key hint to web search chain form
Users had no indication where to configure Exa/Tavily/Brave API keys.
Added hint text pointing to Config → Secrets management.
2026-04-12 11:51:23 +07:00
viettranx f994335cb1 fix(ui): always show Settings button for tools with dedicated forms
hasEditableSettings() only checked if tool.settings was non-empty,
hiding the Settings button for web_search, tts, etc. when no settings
existed yet. Now tools with dedicated form components always show the
button so admins can create initial settings.
2026-04-12 11:45:06 +07:00
viettranx c07aa5cf4b feat(ui): web_search chain editor + TTS settings form
Add dedicated UI forms for web_search and TTS builtin tool settings,
replacing the raw JSON editor for these tools.

- web-search-chain-form: drag-drop provider order (Exa/Tavily/Brave),
  DuckDuckGo locked as always-on fallback, color-coded rails,
  per-provider max_results config
- tts-provider-form: primary provider selector + auto mode dropdown
  with inline descriptions (off/inbound/tagged/always)
- Wire both into builtin-tool-settings-dialog dispatch
- i18n strings for en/vi/zh
2026-04-12 11:41:16 +07:00
viettranx 3a8470b8da docs(tools): tenant tool config refactor - phases 7-8 completion
- 03-tools-system.md: Add § 14 Per-Tenant Tool Configuration (4-tier overlay)
  - Comprehensive overlay explanation (per-agent > tenant > global > hardcoded)
  - Opt-in pattern for tool authors with code example
  - Schema contracts for web_search (provider_order), web_fetch (policy), tts (primary)
  - Secret vs non-secret split guidance
  - Tenant admin workflow (Settings → Builtin Tools UI)
  - Feature flag documentation (TenantScopedSingletons)

- 17-changelog.md: Expand Per-Tenant Tool Configuration entry with phase details
  - Phase 5: Builtin tools settings editor on web UI
  - Phase 7 rest (30a40bbe): Exa + Tavily web search providers
    - Credit @kaitranntt for PR 825 original work
    - 11 new unit tests, provider_order config
  - Phase 8 (def1712f, 43ee918b): Tenant-aware singleton pools
    - web_fetch domain policy resolver (6 new tests)
    - tts primary provider resolver (5 new tests)
    - Feature flag gating (LRU pool, 64 tenant limit, 30 min idle timeout)

All phases 1-8 now shipped on dev branch. Docs integrated with existing
23-multi-tenant-architecture.md § 14 reference. Ready for Phase 9 completion.
2026-04-12 11:25:12 +07:00
viettranx 43ee918b9e feat(tools): tenant-aware TTS primary provider resolution
TTS tool now reads per-tenant primary provider override from
BuiltinToolSettingsFromCtx before falling back to the manager's
default. Same per-call resolution pattern as web_search/web_fetch.

Simpler path A approach (no pool) — tenant can select which
registered provider their agents use. Per-tenant API keys deferred
to a future pool implementation if needed.

- Add resolvePrimary(ctx, mgr) with ttsOverride struct
- Refactor Execute to use resolved primary with fallback chain
- 5 new unit tests for tenant provider override scenarios
2026-04-12 11:22:14 +07:00
viettranx def1712f3b feat(tools): tenant-aware web_fetch domain policy resolution
web_fetch now reads per-tenant policy overrides from
BuiltinToolSettingsFromCtx before falling back to the tool's
default policy. Same per-call resolution pattern as web_search.

- Add resolvePolicy(ctx) with webFetchPolicyOverride struct
- Refactor Execute + doFetch + fetchRawContent to use webFetchPolicy
- InProcessExtractor also resolves policy from ctx
- Remove isDomainAllowed/isDomainBlocked (replaced by matchDomainList)
- 6 new unit tests for tenant policy override scenarios
2026-04-12 11:19:00 +07:00
viettranx 30a40bbe59 feat(tools): add Exa + Tavily web search providers with ranked ordering
Port Exa and Tavily provider clients from PR #825 into the tenant-aware
overlay architecture (builtin_tool_tenant_configs.settings).

- Add ExaConfig + TavilyConfig to WebToolsConfig with provider_order
- Add Exa HTTP client (api.exa.ai/search, x-api-key auth)
- Add Tavily HTTP client (api.tavily.com/search, Bearer auth)
- Extend Brave + DDG constructors with per-provider maxResults
- Add config_secrets plumbing for exa/tavily API keys (apply/collect/mask/strip)
- Refactor gateway_setup.go to use WebSearchConfigFromConfig
- Add NormalizeWebSearchProviderOrder (DDG always last, dedup, unknown skip)
- Extract web_search_config.go (builder, normalizer, shared helpers)
- 11 new unit tests for provider order, builder, clamp, normalize

Credit: @kaitranntt (PR #825) for the original Exa + Tavily implementation.
2026-04-12 11:09:32 +07:00
viettranx 7ae9721c2b docs(contrib): tenant-scope guard rules for admin writes
Role checks are not tenant checks — a non-master tenant admin holds
RoleAdmin in their own tenant and passes role-only middleware by
design. CLAUDE.md gains a one-line directive pointing at CONTRIBUTING
for the full decision table + anti-patterns. CONTRIBUTING gains:

- Target-table decision table (global vs tenant-scoped) with the
  matching guard for each (requireMasterScope vs requireTenantAdmin).
- Shared predicate reference: store.IsMasterScope(ctx).
- Anti-pattern list for reviewers: writes to no-tenant_id tables
  without master-scope check, SQL tenant_id IS NULL arms on write
  paths, role-only admin gates, revoke/delete handlers that skip
  pre-fetch ownership verification.
2026-04-12 10:48:31 +07:00
viettranx 714600f2a9 docs: tenant tool config 4-tier overlay + Phase 0b security hotfix
- docs/03-tools-system.md § 14: 4-tier overlay architecture, opt-in
  pattern for tool authors, secret vs non-secret split, cache
  invalidation, master-scope guard pattern.
- docs/17-changelog.md: concise Security + Added entries for the
  2026-04-12 hotfix and refactor.
- docs/23-multi-tenant-architecture.md: extend Per-Tenant Overrides
  table with settings column + cross-reference to § 14; add
  master-scope guard row to the Security table.

Phase 9 of tenant tool config refactor. Documents what shipped in
commits b419f352, 933c2e10, 56eb6869, 96e38c59, ed32f6e6, fbbba5e8,
6d7473b5, 1e5e84d5.
2026-04-12 10:44:05 +07:00
viettranx 1e5e84d55c feat(ui): tenant-scope aware builtin tool settings editor
Phase 5 of tenant tool config refactor. Wires the existing builtin tools
page + settings dialog to the tenant-config HTTP endpoint shipped in
Phase 4, so tenant admins can edit per-tenant tool overrides from the
same "Settings" button master admins already use.

- BuiltinToolData DTO gains tenant_settings (already returned by the
  GET /v1/tools/builtin enrich path when request is tenant-scoped).
- useBuiltinTools hook gains setTenantSettings + clearTenantSettings —
  both hit PUT /v1/tools/builtin/{name}/tenant-config, with null clearing
  the settings column while preserving tenant_enabled (column-list
  upsert on the store side, shipped in Phase 2).
- BuiltinToolSettingsDialog accepts a tenantScope flag. In tenant mode:
    - initial values come from tenant_settings ?? settings (pre-fill
      with global defaults when creating a fresh override)
    - a small badge + hint line communicates the mode
    - optional "Reset to global default" appears when an override exists
- BuiltinToolsPage branches the save callback: master scope → updateTool
  (global settings), non-master → setTenantSettings. The backend enforces
  the same rule defensively per Phase 0b, so this branch is a UX guard
  (avoid 403) not a security boundary.
- New en/vi/zh keys: tenantOverrideBadge, tenantOverrideHint,
  tenantOverrideNewHint, resetToGlobalDefault.

Verified: pnpm build clean (tsc -b + vite).
2026-04-12 10:27:44 +07:00
viettranx 6d7473b56f fix(http): master-scope guards on builtin_tools, packages, api-keys
Phase 0b of tenant tool config refactor. Closes 3 privilege-escalation
vulnerabilities in the same bug class as commit b419f352 (Phase 1
config.* hotfix):

- CRITICAL: PUT /v1/tools/builtin/{name} — non-master tenant admin
  could overwrite global builtin_tools.settings, corrupting tool
  defaults for every tenant.
- CRITICAL: POST /v1/packages/install|uninstall — non-master tenant
  admin could run pip/npm/apk server-wide. Supply-chain vector.
- HIGH: POST /v1/api-keys/{id}/revoke (HTTP + WS) — tenant admin
  could revoke NULL-tenant system keys because store SQL matches
  (tenant_id = \$N OR tenant_id IS NULL).

Implementation:
- Export store.IsMasterScope as the single predicate; rewire Phase 1
  config.* middleware to delegate (no behaviour change).
- Add http.requireMasterScope helper symmetric to requireTenantAdmin.
- Guard handleUpdate (builtin_tools) and handleInstall/handleUninstall
  (packages) with master-scope check before any mutation or shell exec.
- Fix api_keys.Revoke at the handler layer: fetch key via new
  APIKeyStore.Get, verify key.TenantID matches caller tenant for
  non-owner callers. Applies to both HTTP and WS paths.
- Harden WS router to inject role into ctx so store.IsOwnerRole works
  from WS handlers (closes a latent drift between the HTTP and WS
  layers that broke the initial WS api_keys.revoke fix).
- Drop unused APIKeyStore.Delete (YAGNI + removes dormant vuln with
  the same tenant_id IS NULL arm).
- Emit security.tenant_scope_violation and security.api_key_revoke_
  forbidden slog events on every rejection for future SIEM alerting.
- New MsgMasterScopeRequired i18n key + en/vi/zh catalogs.

Tests cover the guard predicate, all 3 HTTP endpoints, and the full
WS api_keys.revoke matrix (cross-tenant deny, system-key deny for
tenant admins, system owner bypass, own-tenant happy path). 14 other
admin-gated write endpoints verified safe by static audit.
2026-04-12 10:24:07 +07:00
viettranx fbbba5e8bb feat(tools): tenant-aware web_search provider chain resolution
Refactor web_search Execute to resolve its provider chain per-request
via BuiltinToolSettingsFromCtx instead of iterating the singleton's
hardcoded provider list. Tenant admins can now reorder or disable
providers via builtin_tool_tenant_configs.settings using the shape
{"provider_order":[...], "brave":{"enabled":false}, ...}. Unknown
provider names in provider_order are logged + skipped — no injection
path via this settings blob. Malformed JSON falls back to defaults
(fail-open so a bad admin paste never crashes the tool). Secrets stay
in config_secrets — tenant cannot inject API keys through this path.
11 unit tests cover reorder, filter, disable, unknown-skip, malformed,
global-layer, and tenant-vs-global precedence.

Note: This is the MVP tenant adoption half of the plan's Phase 7. The
Exa + Tavily feature port from PR 825 and coordination with its author
is deferred to a separate follow-up (the port is net-new feature work
orthogonal to the overlay adoption).
2026-04-12 09:01:01 +07:00
viettranx ed32f6e6ed test(tools): verify tenant tool settings flow through media provider chain
Phase 6 of the tenant tool config refactor adopts zero production code
changes for media tools — media_provider_chain.go:77 already reads via
BuiltinToolSettingsFromCtx, and the Phase 3 merge automatically feeds
tenant overrides into that call. This adds 5 verification tests proving
the end-to-end flow: tenant layer beats global, per-agent arg still wins
over tenant, fallback to global when no tenant row, tenant-only layer
works, and cross-tool isolation (create_image override does not leak
into create_audio). Covers the 4-tier overlay on a real tool without
touching any media tool's Execute signature.
2026-04-12 09:00:58 +07:00
viettranx 96e38c5912 feat(http): tenant-config settings DTO + GET endpoint
Extend PUT /v1/tools/builtin/{name}/tenant-config to accept optional
enabled + settings fields (at least one required). Add GET endpoint for
the combined tenant override view. Enrich list handler with tenant_settings
alongside existing tenant_enabled. Pointer *bool + json.RawMessage DTO
distinguishes "not set" from "explicit false/null". 16KB body cap via
MaxBytesReader prevents trivial DoS. isValidSettingsJSON rejects non-object
non-null payloads so tool-specific schemas stay predictable. Backward
compat: old clients sending {"enabled": bool} still decode cleanly.
17 tests: validator subcases + DTO decode + stub-backed httptest handler.
2026-04-12 08:55:25 +07:00
viettranx 56eb686934 feat(agent): tenant tool settings overlay via loop ctx injection
Plumb per-tenant tool settings into the agent Loop without touching any
tool's Execute signature. Adds WithTenantToolSettings ctx helper and
rewrites BuiltinToolSettingsFromCtx with fast-path merge semantics —
tenant layer wins over global defaults at tool-name level (no field-level
deep merge). Resolver preloads ListAllSettings for the agent's tenant at
Loop construction; store errors log + fall back to global. Zero allocs
on single-tier reads. Tier 1 (future per-agent override) is reserved and
documented in context_keys.go. 8 unit tests cover empty / single-tier /
both-merged / RunContext fallback / fast-path semantics.
2026-04-12 08:55:20 +07:00
viettranx 933c2e10d9 feat(store): tenant tool settings with column-preservation upsert
Wake up the dead builtin_tool_tenant_configs.settings column (exists in
migrations 000027/SQLite 1180 since v3 tenant foundation, never read/written).
Add GetSettings/SetSettings/ListAllSettings interface methods with a
json.RawMessage Settings field on BuiltinToolTenantConfig. Both DBs use
explicit column-list DO UPDATE SET so Set(enabled) and SetSettings(raw)
never clobber each other. Add ErrInvalidTenant sentinel so nil-tenant
callers fail fast (no silent master fallback). ListAll now filters
enabled IS NOT NULL — rows created via SetSettings stay in their own lane.
9 SQLite unit tests + 4 new PG integration tests (round-trip,
column coexist, cross-tenant isolation, nil-tenant guard).
2026-04-12 08:55:15 +07:00
viettranx b419f35248 fix(gateway): guard config.* methods against non-master tenant scope
Non-master tenant admin calling config.patch/apply/get/schema would
corrupt master *config.Config + config.json on disk via m.cfg.ReplaceFrom
and m.Save, leaking master config across tenants. Add requireMasterScope
middleware that rejects non-master tenant ctx (system-owner role still
bypasses for consistency with requireTenantAdmin). Chain before
requireOwner on all 4 config.* methods. Adds MsgConfigMasterScopeOnly
i18n key (en/vi/zh) and 10 unit tests covering helper + middleware +
chained fail-closed behavior.
2026-04-12 08:55:09 +07:00
viettranx dc482ff169 ci: run integration suite against pgvector service container
Integration tests were running locally only — CI never exercised
the 29 v3_*_test.go files under tests/integration/. This left the
session tenant-isolation + vault scan-arity bugs undetected until
a manual run this session.

Changes:
- tests/integration/v3_test_helper.go: testDB() now calls
  pg.InitSqlx(db) inside sharedDBOnce. Previously the test suite
  relied on whichever test ran first happening to call InitSqlx,
  so running with `-run <filter>` could segfault with a nil
  pkgSqlxDB. Removes the ordering-dependency land mine.
- .github/workflows/ci.yaml: add services.pg block running
  pgvector/pgvector:pg18 with pg_isready healthcheck, set
  TEST_DATABASE_URL at job level, and add a new "Integration
  tests" step after unit tests. Uses -timeout=180s (vs 90s unit)
  because the first test runs migrations from scratch.

Local integration suite: 2.9s. CI cold start expected ~30-60s
with image pull + migrations.
2026-04-12 07:18:01 +07:00
viettranx 8893cdbf8b fix(store/pg): session tenant isolation + vault scan arity
Two unrelated integration-test failures, both real source bugs:

1. Session ListPaged/ListPagedRich leaked cross-tenant data.
   buildSessionFilter only honored opts.TenantID and never read
   tenant from ctx — so callers relying on ctx scoping (the common
   case) got no tenant filter at all. List() already had the
   correct pattern using store.TenantIDFromContext(ctx) gated by
   !store.IsCrossTenant(ctx). Replicate inside buildSessionFilter;
   thread ctx into the helper; opts.TenantID still wins as admin
   override. Fixes TestStoreSession_ListPaged_Pagination/tenant_isolation
   and TestStoreSession_ListPagedRich_TenantIsolation.

2. Vault GetDocument / GetDocumentByBasename panicked with
   "sql: expected 15 destination arguments in Scan, not 14".
   Migration 000047 added `path_basename` as a generated column;
   the SELECT lists were updated but the Scan arg lists were not.
   GetDocumentByID had the correct 15-target pattern — copy it.
   vaultDocRow.PathBasename field already exists. Audit sweep of
   vault_links.go + vault_documents_enrichment.go confirms no
   other drift. Fixes TestStoreVault_UpsertAndGetDocument.
2026-04-12 07:17:54 +07:00
viettranx adf3eab01b test(acp): extract processCloseTimeout var for fast-test override
TestProcessPool_Close_TimesOutSlowProcess was burning 5s of wall
time per run to exercise the "fake process never exits" path,
because Close() used a hard-coded `time.After(5 * time.Second)`.

Extract the timeout as a package var `processCloseTimeout` (default
5s — production unchanged) so the test can override to 20ms via
t.Cleanup-scoped swap. Same pattern as the recent cron/vault/
facebook speedups in commit 0c44149f.

Result: acp race suite 17s → 1.6s. No production behavior change.
2026-04-12 07:17:48 +07:00
viettranx 0c44149fad test: speed up retry/cron/facebook tests, drop coverage ratchet gate
Slow tests were dominating CI feedback time and AI dev loop because they
waited through real exponential backoffs and 1s ticker intervals.
Test-only override pattern keeps production behavior 100% identical.

Speed wins (no-race wall-clock per package):
- internal/vault            16.3s -> 0.6s   (-15.7s)
- internal/cron             11.7s -> 1.5s   (-10.2s)
- internal/channels/facebook 6.3s -> 3.0s   (-3.3s)
- Full -race ./... suite     90s+ -> 51s

Changes:
- vault: new fastBackoffsForTest(t) helper overrides enrichRetryBackoffs
  + enrichRetryTimeouts to 1ms in 3 retry tests; drop 2 duplicate tests
  (FirstAttemptSuccess, MaxRetriesConstant)
- cron: extract runLoopTickInterval as package var (default 1s); test-only
  setFastTick(t) helper shortens to 20ms so 6 scheduler tests no longer
  sleep 1.5s each waiting for a tick
- facebook: extract graphBackoffBase as package var (default 1s); newFakeGraph
  helper shortens to 1ms so HTTP retry tests don't burn 6s of real waits

Coverage ratchet removed:
- Delete scripts/check_coverage.go + scripts/coverage_thresholds.json
- Remove "Coverage ratchet gate" CI step
- Keep coverage profile + go tool cover summary as informational only
- Philosophy: signal over coverage %. Forced tests to bump % were the
  root cause of the slowness this commit unwinds.

Production behavior unchanged. Coverage profile shows isolated package
coverage matches prior thresholds (vault 27.4%, cron 73.7%, facebook 81.9%).
2026-04-11 23:53:07 +07:00
viettranx d77a3664db fix(cache): tenant-aware invalidation for builtin tools and skills
Tenant config changes for builtin tools and skills silently failed to
invalidate cached agent Loops, leaving tenants stuck on stale tool/skill
sets until the 10-minute TTL expired or an unrelated event wiped the
cache. Master-level skill CRUD had the same gap in the opposite
direction.

- Add CacheInvalidatePayload.TenantID so events can scope to one tenant
- Add Router.InvalidateTenant(tenantID) with prefix match on
  "tenantID:agentKey" cache keys; uuid.Nil is a no-op
- Rework emitCacheInvalidate helpers in builtin_tools + skills handlers
  to carry tenant scope; add defense-in-depth uuid.Nil guards in four
  tenant-config handlers
- Update TopicCacheBuiltinTools + TopicCacheSkills subscribers to branch
  on payload.TenantID (tenant event wipes that tenant, global event
  keeps the existing InvalidateAll path)
- Wire emitCacheInvalidate into master skill CRUD paths (update, delete,
  toggle, upload, install-deps, rescan-deps, import) that previously
  only called BumpVersion
- Document the system-owner bypass in requireTenantAdmin so handlers
  keep guarding uuid.Nil themselves
2026-04-11 23:22:17 +07:00
viettranx ee75d498e6 ci: cap go test -timeout to 90s to bound CI hang impact
Deadlocked tests previously blocked CI for the full 10-minute Go
default before failing. Wave C had a live example: a test timeout
branch using `<-t.Context().Done()` (which never fires until test
return) combined with a dispatch-suppressing patch caused a 10-min
GH Actions stall.

Cap every test binary at 90s. Wave C race-heavy suites run in <20s
locally, so 90s leaves ample breathing room while failing fast on
any future deadlock. Applied to both CI workflow and Makefile so
local `make test` enforces the same bound.

Unit-level defenses (proper `time.After` timeouts in tests) are still
the right fix — this is a safety net, not a substitute.
2026-04-11 23:17:33 +07:00
viettranx a8b48024fc test(feishu): bound webhook dispatch wait to 2s, not test timeout
larkevents_test used `<-t.Context().Done()` as the timeout branch in
the dispatch wait select. t.Context() is only canceled when the test
function returns, so a missing dispatch call would deadlock the test
for the full default go test timeout (10 min) before failing — a real
CI hazard, discovered when an unrelated HMAC-verify experiment
prevented dispatch and the CI run sat idle for 10 minutes.

Replace with `time.After(2*time.Second)` via a named constant, plus a
comment warning future readers not to revert the pattern.
2026-04-11 23:15:02 +07:00
viettranx e3993ee50e fix(ci): check_coverage preserves thresholds in --update mode
The preserve-loop that backfills threshold entries for packages not
present in the current coverage profile was gated on `!*update`, which
meant `go run scripts/check_coverage.go --update` with a narrow
coverprofile silently wiped floors for every package not measured.
Discovered while ratcheting wave C with a package-scoped profile.

Move the preserve-loop out of the gate so it runs in both check and
update modes. Narrow profiles now touch only observed packages and
leave the rest untouched.
2026-04-11 23:14:53 +07:00
viettranx 26f2279b1c build(ci): ratchet bump wave C coverage floors + changelog
- scripts/coverage_thresholds.json: feishu 0 → 63.89, acp 0 → 80.05.
  Minor drift auto-normalized: backup 18.80 → 19.88,
  facebook 81.80 → 81.85, providers 62.15 → 62.53,
  store/pg 3.45 → 3.51, tools 26.61 → 26.59 (precision)
- docs/17-changelog.md: "Deferred Coverage Waves A-C — Resolved" entry
2026-04-11 22:45:13 +07:00
viettranx 32c6bc1451 test(acp,feishu): wave C greenfield coverage push
- internal/providers/acp 0.0% → 80.0% (7 test files, ~2560 LOC)
  - helpers, types round-trip, jsonrpc framing + adversarial fuzz,
    session, terminal/sandbox, tool_bridge (3 permission modes),
    ProcessPool lifecycle
- internal/channels/feishu 20.6% → 63.9% (15 test files)
  - bot parse/policy, factory, larkws proto + WS lifecycle,
    larkclient HTTP error paths, larkevents AES-CBC decrypt +
    tamper detection, media send/receive, lifecycle

Zero source modifications. Race clean (-race -count=3).
Dual build verified (standard + sqliteonly).
2026-04-11 22:45:08 +07:00
viettranx f2c43c3849 test(facebook,store/pg): wave B coverage push
Phase 03 - facebook 23.1% -> 81.9% (+58.8%, target ≥60%)
  graph_api_test.go: NewGraphClient, VerifyToken, SubscribeApp, GetPost,
  GetComment, GetCommentThread, ReplyToComment, SendMessage, SendTypingOn,
  doRequest retry/backoff matrix (5xx retry, 401 non-retry, 429+Retry-After,
  context cancel during backoff, transport error + retry), 551/subcode 24h
  window non-retry, logRateLimit parse branches.

  handlers_test.go: New required-field validation (4 branches), Factory
  valid+malformed, WebhookHandler single-shot route, handleAPIError health
  state mapping, handleCommentEvent (feature gate, edit/remove drop, page
  routing, self-reply skip, dedup, enriched content path), handleMessagingEvent
  (feature gate, page routing, self skip, receipt drop, text+postback dispatch,
  dedup), Send Messenger+comment paths + missing-metadata error, sendFirstInbox
  one-shot dedup, runDedupCleaner stale eviction, Stop closes deps, webhookRouter
  register/unregister/route-once/ServeHTTP-no-instances, PostFetcher (defaults,
  empty-id, cache hit/miss/expired, error propagation, GetCommentThread
  empty+delegate).

  Minimal refactor: converted package-level graphAPIBase const to var so tests
  can point at httptest server. KISS, zero runtime behavior change.

Phase 04 - store/pg 1.3% -> 3.5% (+2.2%, unit-test-only)
  scan_rows_test.go: pure-logic transformation tests for agentShareRow,
  userInstanceRow, documentInfoRow, documentDetailRow, chunkInfoRow,
  scoredChunkRow, episodicSummaryRow, episodicScoredRow, sessionListRow,
  sessionPagedRow, sessionRichRow, entityRow, entityTemporalRow, relationRow,
  relationExportRow, traversalRow, dedupCandidateRow, mcpAccessRequestRow,
  cronRunLogRow, skillInfoRow+skillInfoRowWithFrontmatter, customSkillExportRow,
  parseDepsColumn, parseFrontmatterAuthor, marshalFrontmatter, buildSkillInfo,
  mergeEpisodicScores, hybridMerge.

  pure_logic_test.go: matchWildcard pattern matching, evalPermRows priority
  resolution (individual deny > individual allow > group deny > group allow),
  contactResolveCache get/set/expired/negative cache.

  Finding: 30% target not reachable with unit tests alone. Store/pg is
  fundamentally DB-bound (~6000 LOC of SQL code vs ~200 LOC pure logic).
  Reaching 30% requires running integration tests with TEST_DATABASE_URL and
  -coverpkg=./internal/store/pg/ — which the current CI ratchet does NOT
  do. Confirmed: integration tests yield 32% coverage on store/pg, but that
  requires a wired-up test database in CI. Deferred as infrastructure change
  outside Phase 04 scope.

Ratchet bumps (scripts/coverage_thresholds.json):
- internal/channels/facebook: 23.08 -> 81.80
- internal/store/pg: 1.28 -> 3.45

Race clean, dual build (default + -tags sqliteonly) clean.
2026-04-11 21:47:22 +07:00
viettranx 20de0e332e feat(feishu): add /addwriter /removewriter /writers commands
Adds parity with Telegram and Discord for file-writer management
commands, closing the UX gap where users saw an error mentioning
/addwriter but the Feishu channel had no handler.

- New maybeHandleWriterCommand routes /addwriter, /removewriter,
  /writers from the Feishu inbound flow. Runs at step 5a — after
  checkGroupPolicy — so commands never bypass allowlist or pairing
  enforcement. Step 2a rejects slash commands in DM chats early so
  users get a clear hint without waking the agent pipeline.
- Target user is identified via reply-to (fetches parent message
  sender) or first non-bot @mention. A bare /addwriter with no
  target shows the usage hint instead of silently self-granting,
  preventing accidental privilege capture in empty-writer groups.
- Refuses to run while botOpenID is unresolved so a @mention of
  the bot itself cannot be mistaken for a human target.
- 10s context timeout on each handler bounds worst-case Lark API
  latency (parent message lookup, permission store access).
- feishu.New() gains variadic Option parameter with WithAgentStore
  and WithConfigPermStore mirroring Telegram's pattern. The gateway
  now wires pgStores.Agents and pgStores.ConfigPermissions into
  Feishu channel on startup.
- 12 new unit tests with fakeConfigPermStore and httptest Lark
  server cover DM rejection, nil-store graceful degradation, bootstrap
  via self-mention, bare-command usage hint, bot-probe race refusal,
  non-writer rejection, grant via mention, remove last-writer guard,
  empty and populated list output, reply-to target resolution via
  Lark im/v1/messages lookup, and non-command passthrough. 40 total
  tests in the feishu package, all green with -race.

Closes #818.
2026-04-11 21:45:37 +07:00
viettranx 934ab7e485 fix(media): route claude-cli PDFs as document blocks, scope disable_tools
read_image and read_document failed when only claude-cli was configured
because the fallback chains only listed 'anthropic'. Adding claude-cli
as fallback (PR #802) surfaced a deeper bug: buildStreamJSONInput
hardcoded 'type: image' for every content block, so PDFs routed
through claude-cli were rejected by the Anthropic API with a MIME
mismatch.

- providers/claude_cli_session.go: buildStreamJSONInput now picks the
  Anthropic block type from MIME — application/pdf -> document,
  image/* -> image.
- tools/read_image.go, read_document.go: add claude-cli to the
  fallback priority and model defaults (empty string lets the provider
  pick its own default model).
- tools/read_image.go, read_document_resolve.go: scope disable_tools to
  claude-cli only instead of leaking a CLI-specific flag into the
  shared Options map every provider in the chain receives.
- providers/claude_cli_session_test.go: table test covering png/pdf/
  mixed/unknown MIME routing plus the empty-text edge case.

Closes #801.
2026-04-11 21:39:47 +07:00
viettranx abc6d62c95 test(providers,zalo): wave A coverage push
Push two deferred packages above their coverage floors:
- internal/providers 57.0% -> 62.2% (+5.2%, target ≥60%)
  Targeted pure-logic adapter transformation code: registry,
  openai/dashscope/codex ToRequest/FromResponse/FromStreamChunk,
  Azure header branch, token source propagation, function_call +
  incomplete + reasoning parse branches.

- internal/channels/zalo 7.2% -> 65.3% (+58.1%, target ≥20%)
  Factory table-driven creds/config cases, New() defaults, callAPI
  success/error/malformed, getMe/getUpdates, sendMessage/sendPhoto,
  Send routing + photo extraction, Stop signal, processUpdate
  dispatch, handleTextMessage empty-sender drop, downloadMedia four
  branches (png/jpg fallback/404/empty).

Minimal refactor: converted package-level apiBase const to var in
internal/channels/zalo/zalo.go so tests can point at httptest server.
Zero runtime behavior change; tests restore the original value via
t.Cleanup.

Ratchet bumps (scripts/coverage_thresholds.json):
- internal/providers: 57.00 -> 62.15
- internal/channels/zalo: 7.20 -> 65.25

Race clean, dual build (default + -tags sqliteonly) clean,
personal/ sub-packages untouched.
2026-04-11 21:29:32 +07:00
viettranx 0ef3b5cd03 fix(backup): surface Detail alongside Hint in preflight warnings
Missing-status checks previously only surfaced Hint to users, leaving
them with a fix (Install postgresqlNN-client) but no explanation of the
cause (pg_dump NN cannot dump PostgreSQL MM server). Now both warning
and missing statuses append Detail before Hint.

Follow-up to #830.
2026-04-11 21:28:22 +07:00
viettranx 1bdd291fac feat(feishu): auto-fetch Lark docx URLs into agent context
When a user pastes a Lark or Feishu docx URL in chat, the channel
now detects the URL and fetches the document's raw text via the
Lark Docs API, injecting the content into the agent input inline
so the model can reason over the linked doc without a tool call.

- New Channel.resolveLarkDocs pipeline step runs before reply
  context fetch in handleMessageEvent (step 7a)
- LarkClient.GetDocRawContent calls /open-apis/docx/v1/documents
  /{id}/raw_content with the existing tenant access token;
  permission and not-found errors map to ErrDocAccessDenied
- Per-channel LRU cache (128 entries, 5 min TTL) dedupes repeat
  URL references within the window; soft failures are NOT cached
  so permission grants become visible immediately
- Rune-safe content truncation at 8000 runes handles CJK docs
  without splitting mid-rune
- Bounded concurrency (max 3 parallel fetches) and a per-message
  cap of 10 doc URLs act as spam guards
- Tight URL regex anchors the hostname class so a lazy match
  cannot bypass via query-string embedding
- 18 new unit tests cover URL extraction edge cases, cache LRU
  and TTL semantics, Lark API error code mapping, resolver
  end-to-end with nil cache, access denied soft failure,
  per-message cap, and UTF-8 truncation

Required Lark app permission: docx:document:readonly, plus
per-document access grant from the doc owner.

Partial fix for #818 (Phase 2 of 3 — thread reply + writer
commands are tracked separately).
2026-04-11 21:26:12 +07:00
viettranx bf272d8dbf fix(feishu): route thread replies via Lark reply endpoint
Bot responses to messages inside Lark topic threads were dropped
outside the thread because outbound Send always used the new-message
endpoint. This change:

- Adds LarkClient.ReplyMessage() that POSTs to
  /open-apis/im/v1/messages/{id}/reply with reply_in_thread=true
- Parses thread_id from im.message.receive_v1 events (distinct from
  root_id which fires on any quote reply) and stamps
  feishu_reply_target_id into the message metadata
- Propagates the key through cmd/gateway_consumer_normal.go and the
  new package-level routingMetaKeys var in internal/channels/events.go
  so block replies and retry notifications also land in thread
- Routes sendText, sendMarkdownCard, sendImage, sendFile, and
  sendMediaAttachment via a new deliverMessage helper that falls back
  to SendMessage with a warning log on reply endpoint errors (e.g.
  thread root deleted)
- Adds 10 unit tests covering routing, fallback, content
  double-encoding, and the thread_id gate that prevents plain quote
  replies from being silently promoted to threads

Closes #818
2026-04-11 21:22:23 +07:00
viettranx e8e00d1884 fix(telegram): preserve italic content in htmlTagToMarkdown
Go regexp's $1_ is parsed as a reference to a named group "1_" (underscores are
valid identifier chars), not $1 followed by literal underscore. This dropped the
captured italic content, turning <i>foo</i> into "__" instead of "_foo_".

Use ${1}_ with explicit braces to delimit the group reference, and update the
format_extended_test suite to assert the full round-trip for <i> and <em>.
2026-04-11 21:22:23 +07:00
viettranx eaddc68796 docs: changelog entry for coverage improvement waves
Document test coverage improvement initiative:
- 43 new test files across 3 waves
- Per-package coverage floors in coverage_thresholds.json
- CI ratchet gate prevents regression
- ~9000 lines of new test code, 61 packages covered
2026-04-11 21:22:23 +07:00
viettranx 83a6b3c68f test(cache,sessions,knowledgegraph): slim wave 3 coverage push
Add Wave 3 package coverage:
- cache/permission_cache_test.go: PermissionCache 9 methods + invalidation
- sessions/key_extra_test.go: session key builders
- sessions/manager_extra_test.go: SetHistory, Save, loadAll
- knowledgegraph/extractor_helpers_test.go: Extract with mock provider, splitChunks, mergeResults
2026-04-11 21:22:23 +07:00
viettranx bdbed9a35e test(channels): extend slack/discord/telegram/whatsapp coverage
Add channel integration handler tests:
- slack/mention_test.go: isBotMentioned, stripBotMention, user cache, sweepMaps
- slack/media_test.go: classifyMime, buildMediaTags
- discord/handler_test.go: resolveDisplayName, tryHandleCommand, classifyMediaType
- telegram/format_extended_test.go: markdown→HTML, table rendering, chunkHTML
- telegram/handlers_utils_test.go: detectMention, hasOtherMention, isServiceMessage
- telegram/channel_parse_test.go: parseChatID, isEnabled, effectiveMentionMode
- whatsapp/inbound_test.go: extractTextContent, extractQuotedText
- whatsapp/chunk_text_test.go: chunkText, markdownToWhatsApp
- whatsapp/media_utils_test.go: mimeToExt, classifyDownloadError
2026-04-11 21:22:23 +07:00