- Update docs/00-architecture-overview.md with TokenCounter and sessions.metadata details
- Update docs/codebase-summary.md with overhead accounting and compaction logic
- Update docs/project-changelog.md with v3.11.0 context-tokens accuracy fixes
* refactor(providers): migrate ToolDefinition.Function to pointer + add image response fields
ToolDefinition.Function becomes *ToolFunctionSchema with omitempty so native tool types (image_generation, web_search, etc.) can be declared without a function body. All 9 internal construction sites updated. CleanToolSchemas refactored — function-shape cleaning extracted into cleanFunctionSchema helper, outer pass-through handles native tools.
Added image response fields needed by the next commits: ChatResponse.Images, StreamChunk.Images, ImageContent.Partial (distinguishes partial frames from final images).
* feat(providers): native image_generation for Codex + OpenAI-compat tracks
Codex native (POST /codex/responses): emit image_generation tool object in request tools[] (type, action, model, output_format, partial_images). Handle SSE events response.image_generation_call.partial_image + response.output_item.done (type image_generation_call) + response.completed output[] walk for non-stream. Dedup per item_id. Extend codexSSEEvent/codexItem with output_format, result, partial_image_b64, partial_image_index.
OpenAI-compat (/v1/chat/completions): serialize ToolDefinition{Type:'image_generation'} as {type:'image_generation'} pass-through. Parse choices[0].message.images[] + delta.images[] (data URLs) via new parseDataURL helper; append to ChatResponse.Images.
ProviderCapabilities.ImageGeneration flag; Codex provider + adapter set true. Other providers default false.
* feat(agent,http,store): persist assistant images + tri-level image_generation gate
Agent loop tri-level gate: (provider capability) AND (AgentConfig.AllowImageGeneration, default true, stored in other_config.allow_image_generation) AND (request lacks x-goclaw-no-image-gen header). Gate in loop_tool_filter.go appends ToolDefinition{Type:'image_generation'} only when all three pass. Per-request opt-out parsed in chat_completions.go and propagated via RunRequest.NoImageGen.
Media persistence: persistAssistantImages writes final images (Partial:false) to {workspace}/media/{sha256}.{ext}, returns MediaRef entries, clears inline Images[] from the message. Idempotent on hash, traversal-safe, symlink-guarded. Invoked from pipeline.FinalizeStage via new Deps.PersistAssistantImages callback — covers both stream-final and non-stream paths.
Agent store reads AllowImageGeneration from other_config JSONB with absent/nil = true default (matches V3Flags pattern). No DB migration — code-only default.
* feat(ui/web): image_generation toggle + streaming placeholder + download filename
Composer chip 'Images' visible only when active agent's provider has ImageGeneration capability. Per-agent localStorage persistence via useImageGenToggle hook. When off, sends noImageGen:true to WS chat method (maps to x-goclaw-no-image-gen on upstream HTTP call path).
ActiveRunZone renders a skeleton placeholder while streaming partial_image frames arrive. MediaGallery assigns generated-YYYYMMDD-HHmmss.png as the download filename for hex/UUID PNGs.
i18n keys added to en/vi/zh chat.json: imageGenToggle, imageGenGenerating, imageGenDownloadName. 8 vitest tests for the toggle hook.
* docs: add Image Generation section to codebase-summary + changelog entry
Documents the new native image_generation pipeline across providers layer (Codex + OpenAI-compat), agent gate, media persistence, and web UI surface.
* fix(ui/web): match Codex-routed providers for image_generation toggle
Image-gen toggle visibility was hard-coded to provider id 'chatgpt_oauth' but real Codex-routed agents in production use provider ids like 'cliproxy-codex'. The toggle never rendered.
Replace the Set-has check with a small helper that accepts the literal ids plus any provider string containing 'codex' (case-insensitive). Same logic applied in both chat-input.tsx (composer chip) and chat-page.tsx (streaming placeholder gate).
Verified against a live Codex-routed agent: toggle now renders, noImageGen:true propagates on toggle-off.
* docs(pr-1002): targeted-mode UX evidence report
Captures the UI integration surface for native image_generation against a live Codex-routed agent on the remote dev backend.
Includes: composer toggle chip (rendered), streaming skeleton placeholder, and honest failure-path capture showing the legacy create_image builtin fallback. Self-contained HTML report in .github/pr-assets/1002/index.html.
* fix(permissions): classify sessions.compact as write method
CI RBAC-drift test (TestMethodRole_DriftCoverage_AllProtocolMethodsClassified) was failing because the new sessions.compact method added upstream was not classified in any of isPublicMethod / isAdminMethod / isWriteMethod / isReadMethod.
Sessions compaction mutates session history (compacts messages into summaries), so it belongs with the other sessions.* write methods.
* fix(tests): remove duplicate contains() in integration package
Both tts_gemini_live_test.go and mcp_grant_revoke_test.go declared a file-local func contains(s, substr string) bool with identical bodies, causing 'contains redeclared in this block' at compile time in the integration job.
Replace all call sites with strings.Contains (same semantics, stdlib) and drop the duplicates. No behavior change.
* feat(providers): NativeImageProvider interface + Codex implementation
Defines a provider-level contract (NativeImageProvider.GenerateImage) that OAuth-backed providers can implement to serve image generation without exposing static API credentials. Re-uses the PR's Track A native wire format (POST /codex/responses with image_generation tool, item.result decoding, SSE fallback).
CodexProvider + CodexAdapter implement it. Also adds MediaRef.Prompt field so downstream layers can propagate the generating prompt alongside the asset.
* feat(tools): route create_image through NativeImageProvider for OAuth providers
create_image.callProvider now checks for a NativeImageProvider implementation before the credentialProvider interface. When the provider chain points at a Codex-family provider (no static API key), the tool delegates to the provider's GenerateImage which executes the native ChatGPT Responses API image_generation flow.
Resolves 'provider X does not expose API credentials required for image generation' errors for openai-codex / cliproxy-codex chains. On success the tool embeds the user's prompt as a PNG tEXt 'Description' chunk (file-local pngEmbedPrompt helper to avoid tools→agent import cycle), writes the image to /tmp, and threads the prompt through result.MediaPrompts for downstream MediaRef propagation.
* feat(agent,pipeline): propagate image prompt through MediaRef + PNG tEXt embed helper
Adds EmbedPNGPrompt public helper in internal/agent/png_metadata.go that inserts a tEXt 'Description' chunk (plus 'Software: goclaw') into PNG byte streams before the IEND chunk. Non-PNG inputs are passed through unchanged — the helper never errors on unknown formats.
FinalizeStage wires MediaResult.Prompt (from create_image tool output) onto MediaRef.Prompt so the UI can render the generating prompt alongside the image. Per-image prompt list threaded via pipeline RunState.
* feat(ui/web): show generating prompt as caption under each image in MediaGallery
When a MediaRef carries a prompt, MediaGallery renders it as a muted italic caption (line-clamp-2) beneath the image with the full text in the title tooltip. Caption is hidden when the prompt is absent so non-assistant images (user uploads, legacy data) look unchanged.
MediaItem + session media_refs types extended with an optional prompt field; the chat-message adapter threads ref.prompt through when converting WS payloads to UI state.
* fix(providers/codex): stream:true + instructions for native image_generation
The ChatGPT Responses API on /codex/responses rejects two things hard:
- stream:false → HTTP 400 "Stream must be set to true"
- missing instructions → HTTP 400 "Instructions are required"
buildNativeImageRequestBody now sets stream:true and a purpose-specific instructions string ("Generate an image matching the user's description using the image_generation tool. Return only the image; do not describe it in text."). The existing parseNativeImageSSE path was already in place for stream parsing; routing changed from the non-stream branch to the SSE branch.
Regression assertions added to TestCodexGenerateImage_BuildsNativeRequest so these two fields can't silently regress.
* feat(providers,tools,ui): image_model whitelist (gpt-image-2 default, gpt-image-1.5 legacy)
Replaces the hardcoded "gpt-image-2" literal in buildNativeImageRequestBody with a user-configurable field threaded through NativeImageRequest.ImageModel. The whitelist is enforced by ValidateImageModel which rejects anything outside {gpt-image-2, gpt-image-1.5} with a clear error — prevents silent upstream 400s from model names the Responses API would reject.
create_image.callProvider reads params.image_model from the chain entry and threads it through. Empty / absent falls back to DefaultImageModel (gpt-image-2).
UI: added an 'Image model' select inside the existing openai-codex Settings panel on the Create Image Provider Chain dialog. Options: Default · gpt-image-2 (recommended) and Legacy · gpt-image-1.5. i18n keys in en/vi/zh tools.json under builtin.mediaChain.
Tests: TestCodexGenerateImage covers default/legacy/rejected model cases; TestCreateImageTool_ThreadsImageModel covers params→request threading with empty/legacy/explicit sub-cases.
* fix(tools): raise media chain default timeout to 600s/1 retry for image gen
gpt-image-2 on complex prompts (dense Vietnamese text, infographic layouts) legitimately takes 4–8 minutes to complete. The previous default of 120s × 2 retries routinely died mid-generation with 'context deadline exceeded' — the upstream run was still producing bytes when our ctx cancelled.
Default is now Timeout: 600 / MaxRetries: 1. Retries dropped to 1 because image generation is stateful per upstream run: a mid-flight timeout leaves orphan server work, and retrying a fresh generation doubles cost for no gain. Surface the failure fast so operators can widen the timeout instead.
Operators can still set a tighter value explicitly via the Chain dialog.
* refactor: remove user-facing Images toggle, keep admin-level AllowImageGeneration
The per-request opt-out toggle (composer chip + streaming placeholder + noImageGen header plumbing) was a support footgun — users toggle OFF, forget, then can't generate images and think it's broken. Removed in full.
Kept: AgentConfig.AllowImageGeneration (admin kill-switch, stored in other_config.allow_image_generation, default true). Tri-level gate simplifies to two tiers: provider capability AND agent config allows.
Removed: useImageGenToggle hook, IMAGE_GEN_PROVIDER_IDS set in chat-input, supportsImageGenProvider helper, agentProvider/agentKey props on ChatInput, showImageGenPlaceholder prop on MessageBubble/ActiveRunZone/ChatThread, noImageGen param on use-chat-send, parseNoImageGen in chat_completions.go, NoImageGen on RunRequest, no_image_gen_header_test.go, imageGenToggle/imageGenGenerating i18n keys. Kept imageGenDownloadName — used by MediaGallery for generated-\*.png filename resolution.
* docs(pr-1002): refreshed UX trace + updated codebase notes
Replaces the earlier stealth-state evidence with a clean three-capture trace from a real successful run: inline image + prompt caption, MediaGallery lightbox expansion, Chain dialog with the new Image model dropdown open. Skill-routing rows scrubbed from the capture — they reflect per-agent skill setup, not anything this PR introduces.
codebase-summary + changelog: reflect final state (toggle removed, image_model selector, 600s default chain timeout, gpt-image-2 as quality baseline).
* fix(pipeline): preserve mid-loop image_generation output across iterations
FinalizeStage previously read state.Think.LastResponse.Images, which holds
only the last iteration's response. If the LLM emitted image_generation_call
in iteration N alongside a function_call, then responded text-only in N+1,
the image from N was silently dropped on finalize.
Accumulate final (non-partial) images into state.Observe.AssistantImages
across every iteration via ObserveStage, and source FinalizeStage from the
accumulator instead of LastResponse. Partial streaming frames are filtered
defensively; response.Images is cleared on drain to prevent double-counting
on re-exec.
* test(pipeline): regression coverage for mid-loop image accumulation
Six ObserveStage cases covering image accumulation semantics:
- single-iter image-only, image+tool_call same iter, mid-loop image
surviving text-only final iter, multiple images across iters, partial
frame filtering, nil-response safety.
Two FinalizeStage cases verifying accumulator is the source of truth:
- PersistsFromObserveAccumulator: image in Observe + empty LastResponse
must still be persisted via PersistAssistantImages.
- NoPersistWhenAccumulatorEmpty: no call when no images were emitted.
---------
Co-authored-by: viettranx <viettranx@gmail.com>
- Fix prose body citation in 15-core-skills-system (removed :410 line ref)
- Normalize File Reference schema in 08-scheduling-cron and 10-tracing-observability to Module/Path/Purpose format
- Add missing hint line to 03-tools-system File Reference section
- Clean Go method-call symbols from 01-agent-loop Mermaid diagrams (Router + Resolver)
- 01-agent-loop: replace Go function names in sanitize step details and
mermaid labels with behavioral descriptions; clean resolver resolved
properties, router cache/run-tracking, and team workspace context
variables of symbol references
- 11-agent-teams: compress mailbox 3-action table + Use Cases into
3 narrative paragraphs; replace WorkspaceDir Go code block with
prose description
Replace trailing File Reference sections in 7 docs with 3-4 row
module-level tables. Column schema: Module | Path | Purpose.
Adds 1-line grep hint at end of each section. Also fixes one
body .go:line citation in 05-channels-messaging.md.
Rewrote docs/03-tools-system.md from 987 lines to 519 lines.
Removed Go type/const definitions, function citations, 47-file
path listing, and static regex patterns. Replaced with concept
tables, complete tool inventory cross-checked against live
internal/tools/ registrations, and 3-row module table.
Root CHANGELOG.md is the user-facing single source of truth. docs/17-changelog.md
was an internal verbose log duplicating git history.
Also removed docs/multi-tenant-architecture.bk.md (gitignored, pre-v3 backup
superseded by 23-multi-tenant-architecture.md).
Phase 01 of plans/260419-1344-docs-audit-and-condense.
Add manifest-origin dependency declarations in SKILL.md frontmatter.
New fields: deps (pip/npm/system) and exclude_deps (override search results).
Validates dependencies against per-category regex allowlists to prevent injection.
Manifest overrides can extend or reduce auto-discovered deps.
Backward compatible: skills without deps:/exclude_deps: behave identically.
Includes regression guard (bundled_smoke_test.go) for 5 bundled skills.
Closes credential-scope bypass where an agent without a `secure_cli_agent_grants` row could still invoke a registered binary via shell fallthrough and pick up inherited env (`$GH_TOKEN`) or on-disk OAuth state. Registration is now an authorization boundary, not only a credential-injection hint.
- Add `SecureCLIStore.IsRegisteredBinary` on PG + SQLite backends — case-insensitive, tenant-scoped.
- New gate branch in `internal/tools/shell.go` (after normalization, before exec approval) with shell-wrapper unwrap depth 3 (sh/bash/zsh/dash -c, env K=V, nohup, stdbuf, timeout), 2s DB-lookup timeout, fail-CLOSED on error. New log events: `security.credentialed_binary_denied`, `security.credentialed_binary_gate_error`, `security.credentialed_binary_wrapper_too_deep`.
- Env scrubbing on host fall-through (`internal/tools/env_scrub.go`) strips static credential keys (GH_TOKEN, AWS_*, OPENAI_API_KEY, …) plus dynamic keys from tenant's registered binaries; preserves HOME/PATH/TERM/LANG/USER/TZ.
- Subagent ExecTool registration (`cmd/gateway_agents.go` → `buildSubagentToolsRegistry`) now receives the same `SecureCLIStore` — parent can't delegate to a child to bypass the gate.
- Binary names lowercased on Create/Update for symmetry with case-insensitive lookup.
- Thread `pgStores.SecureCLI` into `setupSubagents` in `cmd/gateway.go`.
- 25+ unit tests (gate, env-scrub, SQLite IsRegisteredBinary) + 5 PG integration tests (deny-ungranted, allow-granted, unregistered-unchanged, is_global-not-denied regression, shell-wrapper-bypass denied). All green.
* feat(packages): add GitHub Releases binary installer
New runtime source `github:owner/repo[@tag]` for installing Linux CLI
binaries from GitHub Releases. Admin-only, SHA256-verified, ELF-validated.
Backend:
- GitHub API client with 10-min cache + rate-limit mapping
- SSRF-guarded streaming downloader (HTTPS + host allowlist, re-validated
on every redirect hop, literal-IP rejection)
- Checksums.txt / SHA256SUMS lookup with constant-time verify
- Archive extract (tar.gz / zip / raw) with path-traversal + zip-bomb
guards, symlink skip
- ELF magic + 64-bit class + runtime-arch validation
- Atomic manifest persistence (temp + rename)
HTTP:
- POST /v1/packages/install accepts github: spec
- GET /v1/packages/github-releases for picker UI (viewer+, arch-filtered)
- Extended InstalledPackages response with github field
- github-bin runtime probe
Infra:
- Dockerfile creates /app/data/.runtime/bin (goclaw:goclaw 0755)
- docker-entrypoint.sh prepends bin dir to PATH
- Env-only config (never config.json): token, max size, org allowlist,
bin dir, manifest path
UI:
- GitHub Binaries section + release picker modal
- Dismissable musl/glibc compatibility warning (localStorage)
- i18n keys across en/vi/zh
Docs: docs/packages-github.md user guide + 14-skills-runtime.md cross-ref.
Closes#741
* refactor(packages): revert validPkgName broadening + drop unused sentinel
Code review cleanup:
- validPkgName regex had `:` added defensively, but github: specs are
validated separately via skills.ParseGitHubSpec before reaching this
check — the broadening was dead attack surface.
- Drop unused ErrUnknownArchive sentinel + the `_ = ErrUnknownArchive`
stub in extractRaw.
* feat(packages): per-user rate limit on /v1/packages/github-releases
Cap picker endpoint at 30 req/min/user (burst 10) to protect the shared
GitHub API quota. Key is userID (header X-GoClaw-User-Id) or remote IP
for anonymous callers. Returns 429 + Retry-After: 60 when tripped.
Standalone token-bucket limiter (stale-entry cleanup every 5 min) lives
in internal/http rather than importing internal/gateway, which would
create a package cycle.
* fix(ui): guard split()[0] for noUncheckedIndexedAccess strict TS
CI pnpm build failed on TS2345: `.split('@')[0]` returns
`string | undefined` under strict index access. Default to empty
string to satisfy the type checker; runtime behaviour unchanged
because the downstream regex rejects empty strings.
* fix(packages): address Claude review — medium + low findings
Medium
- rate limiter: atomic.Int64 lastSeen + amortized sweep replaces
goroutine-based cleanup → fixes data race on lastSeen and the
goroutine leak when tests swap the package-level limiter.
- checksum pipeline: slog.Warn on ReadFile and ParseChecksums failures
(previously silent). "asset not listed" stays warn+proceed but is now
documented as the publisher's choice — ELF validation remains the
final gate.
- downloader: drop http.Client.Timeout (30s capped the whole request
including body read, aborting large downloads on slow links). Context
deadline from install timeout (5 min) is the correct bound.
Low / style / UI
- extractRaw honors maxUncompressed (ErrFileTooLarge on overflow) so
the helper is safe outside the hot path.
- cmd/gateway_github_installer.go: drop the explicit cfg.Defaults()
call — NewGitHubInstaller already invokes it.
- GitHubPackageEntry: remove unpopulated InstalledBy field + document
why.
- owner regex tightened to 39-char GitHub limit (was 40).
- mu lock comment corrected: serializes only the disk-write phase.
- UI: shared stripPrefixAndTag helper + owner regex mirrors the backend
39-char cap; destructure-with-default kills the split()[0] ?? ""
awkwardness while still satisfying noUncheckedIndexedAccess.
Verified: go build (pg + sqliteonly) · go vet · go test -race
./internal/skills ./internal/http · pnpm build.
* fix(packages): address Claude review round 2
Medium
- validRepoPath now rejects trailing hyphens in the owner segment and
caps at 39 chars, matching gitHubSpecRE exactly. Previously a subtle
drift between the two validators could let `foo-/repo` slip to the
GitHub API and surface as a 502 instead of a clean 400.
- handleGitHubReleases no longer forwards raw err.Error() from the
upstream call. Maps sentinel errors:
ErrGitHubRateLimited → 429 + Retry-After
ErrGitHubNotFound → 404
ErrGitHubUnauthorized → 502 "github authentication failed"
default → 502 "failed to fetch releases"
Avoids leaking rate-limit reset timestamps / server internals to
viewer-tier callers.
Low / UX
- Install response now returns the manifest entry for github: specs
(new lookupGitHubEntry helper; nil-safe fallback to {ok:true}). Lets
the UI display "installed: lazygit v0.42.0" without a list refresh.
- gitHubSpecRE tag segment capped at 1..255 chars (git ref-name bound).
UI isValidFullSpec mirrors the same cap.
* fix(packages): address Claude review round 3
Medium
- github_api: URL-encode owner, repo, and tag via url.PathEscape when
building API paths. Previously a tag containing '#' would be stripped
as a URL fragment and '?' would inject a query parameter, silently
hitting the wrong release.
Low / polish
- Uninstall via full "github:owner/repo[@tag]" spec now falls back to
manifest lookup by owner/repo, handling packages whose binary name
differs from the repo name (cli/cli → gh).
- GitHubClient.cache sweeps expired entries opportunistically when the
map grows past 256 entries (prevents theoretical unbounded growth
over long uptime).
- handleInstall for github: specs now calls GitHubInstaller.Install
directly and returns the freshly-created manifest entry, eliminating
the double manifest read via List() from the lookupGitHubEntry helper.
- pickBinaries comment corrected — actual behavior excludes paths
matched by nonBinaryPathRE rather than enforcing a single-depth limit.
* fix(packages): address Claude review round 4 (style + ordering)
All 4 findings are Low severity:
- github_api.go: replace interface{} with any across the cache type,
cacheGet return, cacheSet param, and doJSON out param.
- doJSON: rename local `url` to `apiURL` to avoid shadowing the
"net/url" package import used by GetRelease/ListReleases.
- Uninstall: save the updated manifest BEFORE removing binaries on
disk. If saveManifest fails we now bail out without leaving a
manifest entry that still claims binaries which have been deleted
(a retried Uninstall would otherwise hit ErrPackageNotInstalled
after the first attempt wiped the files). Disk removal stays
best-effort and warn-on-error, which matches the idempotent intent.
- pickBinaries: inline comment corrected to reflect actual behavior —
depth is not enforced; nonBinaryPathRE filter + downstream ELF
validation are the real gates.
* fix(packages): address Claude review round 5
All 3 findings are Low severity:
- handleInstall github fast-path now wraps the context with
skills.InstallTimeout (5 min) before calling gh.Install and emits the
same "skills: installing dep" / "dep installed" / "github install
failed" log lines as the generic InstallSingleDep path, so
operator-observability is identical between github: and pip:/npm:
install flows.
- installTimeout promoted to exported InstallTimeout so the http layer
shares the single source of truth rather than duplicating the
5-minute constant.
- cacheMaxEntries comment clarifies it is a soft sweep trigger, not a
hard cap — when every entry is still within TTL the map can briefly
exceed the threshold by one insert.
* fix(packages): address Claude review round 6 (final Lows)
Both findings are Low severity (reviewer marked the PR "ready to merge"
already):
- github_installer: "no checksum asset available" downgraded from
slog.Warn to slog.Info. Many popular upstream releases (jq, fzf,
older ripgrep, etc.) ship no checksum file at all — that is publisher
policy, not a problem with the install. The suspicious cases
(checksum file unreadable, unparseable, or missing this asset) stay
at Warn so they stand out.
- handleGitHubReleases response now uses a narrow assetPreview DTO
(name + size_bytes) instead of embedding the full GitHubAsset type
which also carried browser_download_url. The picker UI never rendered
the URL; trimming the response keeps the viewer-tier surface minimal.
UI AssetPreview interface realigned to match.
* fix(packages): address Claude review round 7
Narrow the GET /v1/packages GitHub entry to a viewer-safe projection
(repo/tag/binaries/name/installed_at), mirroring the assetPreview fix
from round 6. Strips asset_url, sha256, and asset_name from the list
response — viewer-level callers no longer see CDN download URLs or
checksum metadata for installed packages. UI types realigned; the
removed fields were never rendered.
Finding #2 (install writes binary before manifest save) left as noted —
reviewer confirmed informational only, self-heals on retry, no security
impact since binaries pass ELF validation before being written.
* fix(packages): address Claude review round 8
Map HTTP 429 (GitHub secondary rate limits — abuse detection,
unauthenticated bursts, search) to ErrGitHubRateLimited in the API
client so the picker endpoint renders 429 "rate limit reached" with
Retry-After: 60 instead of falling through to 502 "failed to fetch
releases". Primary rate limits (403 + X-RateLimit-Remaining: 0) were
already handled; this covers the secondary class documented at
https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api#about-secondary-rate-limits
* fix(packages): address Claude review round 9
Two defensive hardenings flagged as Very Low:
- ParseChecksums: strip leading `./` from checksum filenames.
`sha256sum ./file` emits `./file` in the name column; the caller
looks up by bare asset basename so `./`-prefixed entries would
silently miss. Real release checksums almost never use this form,
but the guard is essentially free.
- doJSON: cap response body at 8 MiB via io.LimitReader before JSON
decode. Current GitHub list/release payloads are well under this
(~1 MiB at per_page=100). Guards against future call sites or a
misbehaving upstream returning an oversized document.
* fix(cron): eliminate cross-test race on runLoopTickInterval
`Service.Stop()` closes stopChan but does not wait for the runLoop
goroutine to exit. In the test suite, test A's `defer cs.Stop()` can
return before the spawned runLoop has reached
`ticker := time.NewTicker(runLoopTickInterval)`. If test B then calls
`setFastTick()` to mutate the package-level var, the race detector
correctly flags it:
Read at runLoopTickInterval by goroutine A (runLoop ticker init)
Previous write by goroutine B (setFastTick in test B)
Fix: snapshot `runLoopTickInterval` inside `Start()` under the mutex
before spawning the goroutine, and pass the value as a parameter to
`runLoop`. The spawned goroutine no longer reads the package-level
var, so the cross-test window is closed. Production behavior
unchanged.
Verified: `go test -race -count=3 ./internal/cron/...` passes three
times in a row; the CI failure on PR #898 reproduced before the fix
and is gone after.
* fix(packages): address review P0/P1/P2 + new DoS vector
P0.1 — UI uninstall 400: parseAndValidatePackage now accepts
github:<bare-name> (manifest Name form, no owner/repo) in addition
to the full spec. UI sends github:${pkg.name} from the manifest;
dispatcher already tolerated bare names — the HTTP validator was
the only gate rejecting them. Install path re-validates strictly
via ParseGitHubSpec and bare-name install returns 400 now (was 500).
P1.1 — ExtractArchive raw-ELF fallback name: add ExtractArchiveAs(
path, fallbackName, max). Installer passes parsed.Repo so raw
(non-archive) ELF assets no longer end up recorded as
/tmp/goclaw-gh-asset-XXXX.bin — that basename would leak into the
manifest Binaries entry and break PATH lookup.
P1.3 — Archive entry count cap: maxArchiveEntries = 10_000 +
ErrTooManyEntries sentinel. Tar: count ALL headers seen (incl.
symlinks/dirs we skip) to block the gzip-bomb-of-headers DoS —
header bytes don't count against maxUncompressed for zero-size
entries. Zip: pre-check via peekZipEntryCount reads the EOCD
record manually and rejects oversized archives BEFORE
zip.OpenReader allocates []*zip.File of declared capacity (this
was a fresh red-team finding; stdlib would otherwise alloc ~1GB
for a crafted 200MB zip claiming 4M entries).
P1.4 — Rate-limit install/uninstall: packagesWriteLimiter
(10/min/user, burst 3). Admin-only mitigates but a compromised
token could otherwise flood upstream (GitHub/pip/npm) or spam
manifest mutations.
P1.6 — Non-Linux early reject: ErrUnsupportedOS guard at the top
of Install(). Windows/macOS hosts no longer waste bandwidth
fetching a Linux asset just to fail at the ELF machine check.
P1.7 — Manifest fsync: OpenFile → Write → Sync → Close → Rename →
dir Sync, with tmp cleanup on every error path. POSIX doesn't
guarantee durability via rename alone; XFS / ext4 with async
journal can reorder.
P2.1 — Belt-and-suspenders zip runtime break when cumulative bytes
reach the cap (pre-declared check already covers it but the
streaming loop now bails immediately).
P2.6 — Binary-name collision warn: slog.Warn when a different repo
already owns the basename we're about to overwrite. Last-writer-
wins unchanged; operator now gets a signal instead of silence.
Hardening — rate-limit key: rateLimitKeyFromRequest prefers
store.UserIDFromContext over the raw X-GoClaw-User-Id header so
an admin can't rotate the header mid-session to dodge the bucket.
Header/IP fallback retained for pre-auth / test callers.
Tests: 9 new cases on parseAndValidatePackage (github full/bare/
empty/traversal/injection/space/leading-hyphen);
TestExtractArchiveAs_RawELFUsesFallbackName;
TestExtractTarGz_EntryCountCap + TestExtractZip_EntryCountCap;
TestPeekZipEntryCount (DoS pre-check path).
Verified: go build ./... && go build -tags sqliteonly ./... &&
go vet ./... && go test -race ./internal/skills/... ./internal/http/...
---------
Co-authored-by: viettranx <viettranx@gmail.com>
- agent-identity-conventions.md: new "ActorID vs UserID in Group Chats"
section with helper table, group behavior table, propagation chain
diagram, group permission policy, legacy-data tolerance notes.
- 17-changelog.md: entry covering the security fix, propagation
additions, ACTOR migration list, scope-intentional sites, tests,
and the no-DB-migration decision with its legacy-fallback rationale.
- webhook_handler.go: demote per-event "page_id resolution" log from Info to
Debug. Firing on every webhook was the exact anti-pattern called out in the
routing-metadata refactor review.
- pancake.go: move reply_to_comment_id guard above rememberOutboundEcho so a
missing-metadata error does not stamp phantom echoes that would pollute
inbound echo dedup.
- docs/journals: drop three Pancake journals (not needed in repo).
* feat(channels): make Pancake platform a required select with 11 options
- Add mandatory platform select field (11 options) to Pancake channel schema
- Add config required validation on channel instance form (create-only)
- Add i18n fieldOptions/fieldConfig for platform in en/vi/zh locales
- Add TDD tests for Pancake config schema (channel-schemas.test.ts)
- Add backend test TestFactoryExplicitPlatformPreserved
- Add slog.Debug for auto-detect path in pancake.go
- Update Platform field comment in types.go for clarity
- Update plan status to completed; add changelog entry
* feat(channels): hide comment_reply for non-social Pancake platforms
Extend showWhen to accept string | string[] values. Apply it to
features.comment_reply so the field only appears for platforms that
support public posts/comments (facebook, instagram, threads, tiktok,
youtube). E-commerce platforms (shopee, lazada, tokopedia) and
messaging-only platforms (line, google, chat_plugin) no longer show
the irrelevant Comment Reply toggle.
Also guard depValue before String() coercion to avoid the "undefined"
literal matching hazard.
* feat(channels): Pancake comment reply fix + platform select + UI improvements
- Fix comment reply: pass message_id (reply_to_comment_id) to Pancake API
ReplyComment now requires messageID param; guard added for empty ID
- Add SendMessageRequest.MessageID field (omitempty) for reply_comment action
- Platform select: required field with 11 options, showWhen gate for comment_reply
- Webhook Page ID moved to Advanced collapsible section in channel form
(auto-expands when existing value is configured)
- routing_metadata.go: centralize routing metadata key constants
- config-flatten.ts: flatten/unflatten nested config for form state
Closes: Pancake comment reply returns 'Missing required field: message_id'
Port goclaw context pruning to match upstream TS design in
openclaw/src/agents/pi-hooks/context-pruning/:
- Opt-in default: prune only when mode="cache-ttl" (was opt-out)
- Remove Pass 0 per-result 30% guard (duplicated Pass 1 with different
suffix, caused wobble)
- Dedupe double prune call per iteration: PruneStage owns the single
entry point; loop_history only runs limitHistoryTurns + sanitizeHistory
- Add cache-TTL gate for Anthropic prompt cache: skip prune while cache
is live, scoped per-session via sync.Map
- Add context.pruned event emission for observability
- Configurable TTL as Go duration string ("5m", "30s")
BREAKING CHANGE: context pruning now opt-in. Add
contextPruning.mode: "cache-ttl" to config.agents.defaults to restore.
Migration 51 / SQLite v19 backfills mode="cache-ttl" for agents with
existing custom context_pruning config missing the mode field, so
previously-configured agents keep pruning after the opt-in flip.
NULL configs stay NULL (new opt-in default applies).
Web UI adds Cache TTL input + toggle wiring mode to cache-ttl/off.
Update architecture overview with streaming TTS provider layer. Expand tools
system docs with voice/model resolution. Update HTTP and WebSocket RPC
documentation for voice endpoints. Record Phase 02 completion in changelog.
Update CLAUDE.md with voice picker and streaming TTS patterns.
Update module map to list internal/audio/ (unified manager, TTS active,
STT/Music/SFX stubbed/partial) and clarify internal/tts/ as a
24-symbol backward-compat alias layer. Add changelog entry for Phase 1.
Stale recovery sweeps traces by `start_time < NOW() - threshold`, which
measures trace age rather than inactivity. Any threshold low enough to
be useful (2-10 min) kills legitimate long-running agent runs: research
chains, large code generation, extended shell commands routinely exceed
10 minutes.
Disabled in Start() — function kept in place for easy re-enable once a
`last_span_at` column is added so recovery can gate on "no activity for
N minutes" instead of "started > N min ago".
Trade-off: zombie traces from gateway crashes may remain `running` in
DB. Accepted: primary abort path (router 2-phase + trace.status WS
event) handles the common case; safety-net gap preferred over false
kills of healthy runs.
Integration test RecoverStaleNow() still works (manual trigger, not
loop-dependent) so coverage of the recovery function itself is
preserved for when it's re-enabled.
- Sigma.js graph: restore doc_type coloring (revert Louvain community detection)
- Fix animation flash after FA2 layout finishes by removing post-processing
camera reset and redundant noverlap/compactOrphans in stopLayout
- Fix vault tree "Load more" state bug when filtering by doc_type:
add treeVersion counter to force re-mount and reset auto-expand state
- Fix meta map loss: loadRoot now merges instead of replacing, preserving
subtree entries from previous loadSubtree calls
- Add compact graph DTO endpoints and hooks for KG and vault graphs
- Add semantic zoom tiers, adaptive FA2 settings, and node sizing
Slow tests were dominating CI feedback time and AI dev loop because they
waited through real exponential backoffs and 1s ticker intervals.
Test-only override pattern keeps production behavior 100% identical.
Speed wins (no-race wall-clock per package):
- internal/vault 16.3s -> 0.6s (-15.7s)
- internal/cron 11.7s -> 1.5s (-10.2s)
- internal/channels/facebook 6.3s -> 3.0s (-3.3s)
- Full -race ./... suite 90s+ -> 51s
Changes:
- vault: new fastBackoffsForTest(t) helper overrides enrichRetryBackoffs
+ enrichRetryTimeouts to 1ms in 3 retry tests; drop 2 duplicate tests
(FirstAttemptSuccess, MaxRetriesConstant)
- cron: extract runLoopTickInterval as package var (default 1s); test-only
setFastTick(t) helper shortens to 20ms so 6 scheduler tests no longer
sleep 1.5s each waiting for a tick
- facebook: extract graphBackoffBase as package var (default 1s); newFakeGraph
helper shortens to 1ms so HTTP retry tests don't burn 6s of real waits
Coverage ratchet removed:
- Delete scripts/check_coverage.go + scripts/coverage_thresholds.json
- Remove "Coverage ratchet gate" CI step
- Keep coverage profile + go tool cover summary as informational only
- Philosophy: signal over coverage %. Forced tests to bump % were the
root cause of the slowness this commit unwinds.
Production behavior unchanged. Coverage profile shows isolated package
coverage matches prior thresholds (vault 27.4%, cron 73.7%, facebook 81.9%).
Adds parity with Telegram and Discord for file-writer management
commands, closing the UX gap where users saw an error mentioning
/addwriter but the Feishu channel had no handler.
- New maybeHandleWriterCommand routes /addwriter, /removewriter,
/writers from the Feishu inbound flow. Runs at step 5a — after
checkGroupPolicy — so commands never bypass allowlist or pairing
enforcement. Step 2a rejects slash commands in DM chats early so
users get a clear hint without waking the agent pipeline.
- Target user is identified via reply-to (fetches parent message
sender) or first non-bot @mention. A bare /addwriter with no
target shows the usage hint instead of silently self-granting,
preventing accidental privilege capture in empty-writer groups.
- Refuses to run while botOpenID is unresolved so a @mention of
the bot itself cannot be mistaken for a human target.
- 10s context timeout on each handler bounds worst-case Lark API
latency (parent message lookup, permission store access).
- feishu.New() gains variadic Option parameter with WithAgentStore
and WithConfigPermStore mirroring Telegram's pattern. The gateway
now wires pgStores.Agents and pgStores.ConfigPermissions into
Feishu channel on startup.
- 12 new unit tests with fakeConfigPermStore and httptest Lark
server cover DM rejection, nil-store graceful degradation, bootstrap
via self-mention, bare-command usage hint, bot-probe race refusal,
non-writer rejection, grant via mention, remove last-writer guard,
empty and populated list output, reply-to target resolution via
Lark im/v1/messages lookup, and non-command passthrough. 40 total
tests in the feishu package, all green with -race.
Closes#818.
When a user pastes a Lark or Feishu docx URL in chat, the channel
now detects the URL and fetches the document's raw text via the
Lark Docs API, injecting the content into the agent input inline
so the model can reason over the linked doc without a tool call.
- New Channel.resolveLarkDocs pipeline step runs before reply
context fetch in handleMessageEvent (step 7a)
- LarkClient.GetDocRawContent calls /open-apis/docx/v1/documents
/{id}/raw_content with the existing tenant access token;
permission and not-found errors map to ErrDocAccessDenied
- Per-channel LRU cache (128 entries, 5 min TTL) dedupes repeat
URL references within the window; soft failures are NOT cached
so permission grants become visible immediately
- Rune-safe content truncation at 8000 runes handles CJK docs
without splitting mid-rune
- Bounded concurrency (max 3 parallel fetches) and a per-message
cap of 10 doc URLs act as spam guards
- Tight URL regex anchors the hostname class so a lazy match
cannot bypass via query-string embedding
- 18 new unit tests cover URL extraction edge cases, cache LRU
and TTL semantics, Lark API error code mapping, resolver
end-to-end with nil cache, access denied soft failure,
per-message cap, and UTF-8 truncation
Required Lark app permission: docx:document:readonly, plus
per-document access grant from the doc owner.
Partial fix for #818 (Phase 2 of 3 — thread reply + writer
commands are tracked separately).
Bot responses to messages inside Lark topic threads were dropped
outside the thread because outbound Send always used the new-message
endpoint. This change:
- Adds LarkClient.ReplyMessage() that POSTs to
/open-apis/im/v1/messages/{id}/reply with reply_in_thread=true
- Parses thread_id from im.message.receive_v1 events (distinct from
root_id which fires on any quote reply) and stamps
feishu_reply_target_id into the message metadata
- Propagates the key through cmd/gateway_consumer_normal.go and the
new package-level routingMetaKeys var in internal/channels/events.go
so block replies and retry notifications also land in thread
- Routes sendText, sendMarkdownCard, sendImage, sendFile, and
sendMediaAttachment via a new deliverMessage helper that falls back
to SendMessage with a warning log on reply endpoint errors (e.g.
thread root deleted)
- Adds 10 unit tests covering routing, fallback, content
double-encoding, and the thread_id gate that prevents plain quote
replies from being silently promoted to threads
Closes#818
Document test coverage improvement initiative:
- 43 new test files across 3 waves
- Per-package coverage floors in coverage_thresholds.json
- CI ratchet gate prevents regression
- ~9000 lines of new test code, 61 packages covered
Distills the dual-identity rules from the agent identity hardening work
into a single permanent reference. Covers all 5 trap zones with post-
hardening status, batch fail-fast contract change, router cache
canonicalization, FK safety net, telemetry type-layer defense, and
dual-tenant agent_key semantics. Adds a CLAUDE.md pointer under Key
Patterns so contributors find it via the standard onboarding path.
viettranx