mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-04 12:13:15 +00:00
* refactor(bitrix24): rename "Path B" framing to maintainer-specified naming [B24:2794] Per maintainer hard rule #10 (no generic "Path A/B" framing) from PR #1061 review. The Bitrix24 MCP auto-onboard flow is Bitrix-specific glue ("Bitrix24 OAuth -> existing mcp_user_credentials bridge"), NOT a generic MCP architecture pattern. Naming convention applied consistently: - First mention per file: full "Bitrix24 OAuth -> existing mcp_user_credentials bridge" (matches maintainer comment verbatim). - Subsequent mentions in same file: shortened "mcp_user_credentials bridge". - Test/log context referencing literal endpoint /api/auto-onboard: keep "auto-onboard" reference (it's the actual API endpoint name). Changes are documentation-only: - Rename in code comments + test descriptions + plan docs. - Clarify framing in mcp_client.go + provisioner.go doc comments to emphasize Bitrix-specific glue (not generic MCP infra). - Reuse existing mcp_user_credentials table + MCPServerStore methods (no schema / store / abstraction change). Files: - cmd/gateway.go (factory registration doc) - internal/channels/bitrix24/{channel,factory,mcp_client,provisioner}.go - internal/channels/bitrix24/{mcp_client,provisioner}_test.go - plan/goclaw-mcp-integration.md (21 occurrences) Verified: go build + MCP-related tests pass (TestProvision*, TestInitMCPProvisioner*, TestMCPClient*). Phase 1 of Path C execution per plans/reports/decision-log-260519-1555-bitrix24-pr-fork-decision.md. * fix: confine outbound media paths to agent workspace [B24:2794] Tool MEDIA:<path> output reached channel file-upload sinks (Bitrix imbot.v2.File.upload, Telegram sendDocument, etc.) verbatim via parseMediaResult, with no workspace-boundary check. A malicious or buggy tool emitting MEDIA:/etc/passwd could exfiltrate arbitrary files to chat. Extract the EvalSymlinks+Rel containment from extractMediaFromContent into a shared confineToWorkspace helper and apply it at the parseMediaResult sink in processToolResult. Fixing at the source/egress boundary protects every channel at once rather than per-channel. Paths that escape the workspace are dropped and logged (security.media_path_rejected). Add TestConfineToWorkspace (boundary unit) and TestParseMediaResultConfinedToWorkspace (sink regression for H2). * feat(bitrix24): support inbound + outbound media via imbot.v2 File API [B24:2794] Bitrix24 channel was text-only; attachments were parsed but dropped. - Inbound: download chat files via imbot.v2.File.download (one-time URL), forward to the agent with MIME preserved (internal/channels/bitrix24/download.go). - Outbound: upload agent media to the chat via imbot.v2.File.upload (internal/channels/bitrix24/send_media.go). - Add BaseChannel.HandleMessageMedia to preserve MIME/filename through the bus. - Per-channel media_max_mb cap (default 20) applies to both directions. Tests: 92 pass (internal/channels/bitrix24 + internal/channels), go vet clean (PG + sqliteonly). * refactor(bitrix24): migrate messaging/bot-list/unregister to imbot v2 API [B24:2794] Move outbound REST calls to the imbot v2 family (keeps register on v1): - imbot.message.add -> imbot.v2.Chat.Message.send (fields.message shape, live-verified) - imbot.bot.list (+ legacy imbot.list fallback) -> imbot.v2.Bot.list; add botListRows to normalize the v2 {bots:[...]} envelope, legacy array, and id-keyed map forms - imbot.unregister -> imbot.v2.Bot.unregister Bot registration stays on v1 imbot.register: v2 imbot.v2.Bot.register changes the event-delivery model (per-event handler URLs -> eventMode), which would require rewriting the inbound event parser. No user-facing behavior change. Tests: bitrix24 package green; go vet ./... clean. * feat(bitrix24): route whisper via v1 SKIP_CONNECTOR + add v2 replyId [B24:2794] Bot was leaking HiddenMessage (whisper) replies to the external Zalo connector because every outbound call went through imbot.v2.Chat.Message.send, which has no equivalent of the v1 SKIP_CONNECTOR flag. Branch the outbound path on inbound visibility: whisper → imbot.message.add + SKIP_CONNECTOR=Y (v1, send_v1.go) public → imbot.v2.Chat.Message.send + fields.replyId (v2, send_v2.go) Pipeline: events.go parse data[PARAMS][PARAMS][COMPONENT_ID]=HiddenMessage into EventParams.IsHiddenMessage (form + JSON variants) handle.go set bitrix_visibility on InboundMessage.Metadata consumer forward visibility + message_id into OutboundMessage send.go resolveSendOptions + sendChunk dispatcher + shared callWithRateLimitRetry helper metadata_keys.go single source of truth for the keys + values Defaults preserve pre-refactor behaviour: callers that don't populate bitrix_visibility still go through v2 public, and replyId is omitted unless a numeric bitrix_message_id arrives in metadata. Tests: TestParseEvent_FormURLEncoded_IsHiddenMessage (3 cases) TestParseEvent_JSON_IsHiddenMessage (3 cases) TestResolveSendOptions (8 cases) TestSend_BranchesOnVisibility (4 cases) * feat(bitrix24): openline sender-tag echo on replies [B24:2794] Openline sender-tag echo (this change): - Capture the connector sender tag ("[name #id]:" or "[name] #id:") from inbound openline group messages, strip it from the body the agent sees, and re-prepend the canonical "[name] #id:" form to the reply so the Open Channel connector routes the answer back to the right external user. - New sender_prefix.go helper (+ test) accepts both inbound layouts and emits one canonical form; scoped to messages carrying the tag, so plain chats are unaffected. - metadata_keys.go: MetaKeySenderPrefix; handle.go capture/strip/stash; gateway_consumer_normal.go forwards the key; send.go prepends it on the first chunk before chunking. Bundled bitrix24 channel-core work already on this branch: - handle.go: @mention is the sole trigger for both staff and connector customers; unmentioned traffic is dropped (was: drop all connector msgs). - isGroupMessageType: treat SONET_GROUP "B" as a group. - handle_test.go, mcp_client_test.go: cover the above. * feat(bitrix24): accept colon-less openline sender tag, echo [name] #id [B24:2794] The Open Channel connector dropped the trailing colon from its sender tag: inbound now arrives as "[Name] #id <msg>" (was "[Name] #id: <msg>"). The id-bearing patterns required the colon, so the tag fell through to the name-only branch and the reply echoed "[Name]" — dropping the #id the connector needs to route the answer back. - sender_prefix.go: make the trailing ":" optional on both id layouts ([name #id] / [name] #id, with or without colon) and echo the canonical "[name] #id" (no colon) to match the connector's current format. Bare "[name]" (no id) still echoes "[name]" for Open Channel only. - handle.go: gate the bare name-only layout to Open Channel (isOpenChannel) so ordinary group chats starting with "[x] ..." are left untouched. - sender_prefix_test.go: cover colon/no-colon x id-inside/id-outside, the name-only openline case, and the non-openline no-op. * fix: security and robustness fixes from the bitrix24 channel review [B24:2794] - download.go: block redirect-based SSRF on inbound media. CheckRedirect re-validates each hop (http(s) only, reject private/loopback/link-local hosts, cap hops); the initial portal-domain pin is no longer bypassable via a 3xx to an internal service. Public-host redirects still allowed. - handle.go: extract/echo the openline sender tag only for Open Channel sessions (was: any group chat), removing bogus prefixes in CRM group chats and narrowing the forged-tag misroute surface. - loop_tools.go + loop_media.go: confine result.Media to the agent / team / tenant-allowed roots (new confineToAnyRoot) before a channel uploads it, so a prompt-injected out-of-workspace path (e.g. /etc/passwd) cannot exfiltrate, while legitimate cross-workspace media (team files, delegatee output) still flows. - send_media.go: bounded outbound read via io.LimitReader replaces the os.Stat + os.ReadFile pair, closing the TOCTOU size-cap bypass; cap a single message's outbound attachments at 10 (mirrors inbound). - register.go: paginate imbot.v2.Bot.list (limit/offset + hasNextPage, capped at 40 pages) so verify/lookup see bots past the first 50. - mcp_client.go: redact access_token / refresh_token / client_secret from an echoed MCP error body before it is logged or returned (+ test). * fix(security): validate resolved dial IP on Bitrix media redirects [B24:2794] The inbound media download redirect guard only string-checked the redirect hostname (isPrivateOrLoopback on req.URL.Hostname()), so a redirect to a public hostname that resolves to 127.0.0.1 / 169.254.169.254 / an RFC1918 address — or a DNS-rebinding swap between check and dial — still passed the guard and the client would connect. Reported in PR review. Add security.NewRedirectFollowingSafeClient: it follows redirects but validates the RESOLVED destination IP of every hop at dial time via net.Dialer.Control, reusing the existing blocked-CIDR list. The IP it checks is the IP actually dialed, so both redirect-to-internal and DNS rebinding are refused, while legitimate public CDN redirects still succeed. download.go now uses it instead of the hostname-string guard. Tests: deterministic dial-control table (loopback / link-local / private / multicast / unspecified / public, v4 + v6), malformed/non-IP addr, test bypass, loopback-dial-blocked client wiring, and redirect cap + scheme checks. * feat(bitrix24): per-participant Zalo openline identity from 3-token sender tag [B24:2794] Parse the connector's "[Name] #uid #msgId" sender tag so each external customer in a shared Open Channel group gets its own contact + USER.md instead of collapsing onto the connector proxy id. Identity minting is gated on IS_CONNECTOR=Y to reject operator forged tags. Echo back the msgId only ("#msgId") on replies; keep the legacy single-number and name-only layouts unchanged. Zero DB migration. - sender_prefix.go: parseOpenlineSenderTag() classifies 3-token / legacy / name-only - handle.go: synthetic senderID "openlines:{instance}:{chat}:{uid}" + participant_user_id metadata, gated on FromIsConnector - gateway_consumer_normal.go: deriveGroupUserID() routes participant -> per-person scope, group fallback otherwise - send.go: buildAddressMention numeric-id guard so synthetic ids don't emit invalid [USER=...] BBCode - MetaKeyMessageID kept as Bitrix MESSAGE_ID (drives v2 fields.replyId); connector msgId surfaced only via echo prefix --------- Co-authored-by: DangTinh311 <dangtinh31193@gmail.com> Co-authored-by: Chinh Dang <chinhdang@192.168.68.104>
242 lines
7.8 KiB
Go
242 lines
7.8 KiB
Go
package agent
|
|
|
|
import (
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
)
|
|
|
|
// parseMediaResult extracts a MediaResult from a tool result string containing "MEDIA:" prefix.
|
|
// Handles formats: "MEDIA:/path/to/file" and "[[audio_as_voice]]\nMEDIA:/path/to/file".
|
|
// Returns nil if no MEDIA: prefix is found.
|
|
//
|
|
// IMPORTANT: Only matches "MEDIA:" at the start of the (trimmed) string to avoid false
|
|
// positives when tool output contains "MEDIA:" in arbitrary text (e.g. a web page
|
|
// mentioning a commit message like "return MEDIA: path from screenshot").
|
|
func parseMediaResult(toolOutput string) *MediaResult {
|
|
s := toolOutput
|
|
asVoice := false
|
|
|
|
// Check for [[audio_as_voice]] tag (TTS voice messages)
|
|
if strings.Contains(s, "[[audio_as_voice]]") {
|
|
asVoice = true
|
|
s = strings.ReplaceAll(s, "[[audio_as_voice]]", "")
|
|
}
|
|
|
|
s = strings.TrimSpace(s)
|
|
|
|
// Only match MEDIA: at the beginning of the string.
|
|
if !strings.HasPrefix(s, "MEDIA:") {
|
|
return nil
|
|
}
|
|
path := strings.TrimSpace(s[6:])
|
|
if path == "" {
|
|
return nil
|
|
}
|
|
// Take only the first line (in case there's trailing text)
|
|
if nl := strings.IndexByte(path, '\n'); nl >= 0 {
|
|
path = strings.TrimSpace(path[:nl])
|
|
}
|
|
|
|
return &MediaResult{
|
|
Path: path,
|
|
ContentType: mimeFromExt(filepath.Ext(path)),
|
|
AsVoice: asVoice,
|
|
}
|
|
}
|
|
|
|
// confineToWorkspace validates that mediaPath resolves to a regular file located
|
|
// inside workspace, then returns the cleaned path. It is the single source of
|
|
// truth for the media path-containment boundary, shared by the two feeders of
|
|
// MediaResult.Path: extractMediaFromContent (LLM-echoed tokens) and the
|
|
// parseMediaResult sink in processToolResult (tool MEDIA: output). Constraining
|
|
// at this boundary protects every outbound channel at once — a path that escapes
|
|
// the workspace never reaches a channel's file-upload egress.
|
|
//
|
|
// Containment applies, in order:
|
|
// - relative paths are resolved against the workspace root;
|
|
// - Lstat (not Stat) rejects a symlink at the leaf outright;
|
|
// - EvalSymlinks resolves ancestor symlinks before the Rel check, so a
|
|
// "<ws>/<symlink-dir>/secret" escape via a dir symlink pointing outside the
|
|
// workspace is caught (a purely lexical Rel check would miss it).
|
|
//
|
|
// Returns the cleaned (symlink-preserving) path and true when the file is safe
|
|
// to ship, or "", false when it must be dropped. An empty workspace yields
|
|
// false: without a boundary there is nothing to validate against, and an
|
|
// unvalidatable path must never reach an external egress.
|
|
//
|
|
// NOTE: the returned path is `cleaned`, NOT the symlink-resolved path. resolved
|
|
// is used ONLY for the containment check — downstream readers (channel senders,
|
|
// history, dedup) must see the same path semantics the tool emitted, otherwise
|
|
// workspaces backed by bind-mounts / dir symlinks suffer dedup misses (observed
|
|
// in production).
|
|
func confineToWorkspace(mediaPath, workspace string) (string, bool) {
|
|
if mediaPath == "" || workspace == "" {
|
|
return "", false
|
|
}
|
|
// Resolve workspace to its real path (follows symlinks). Required because
|
|
// macOS uses symlinks for /tmp → /private/tmp; if we only Clean the
|
|
// workspace but EvalSymlinks the candidate path, the Rel check below would
|
|
// spuriously fail even for legitimate files.
|
|
wsRoot := ""
|
|
if abs, err := filepath.Abs(workspace); err == nil {
|
|
if resolved, err := filepath.EvalSymlinks(abs); err == nil {
|
|
wsRoot = filepath.Clean(resolved)
|
|
} else {
|
|
wsRoot = filepath.Clean(abs)
|
|
}
|
|
}
|
|
if wsRoot == "" {
|
|
return "", false
|
|
}
|
|
path := mediaPath
|
|
if !filepath.IsAbs(path) {
|
|
path = filepath.Join(wsRoot, path)
|
|
}
|
|
cleaned := filepath.Clean(path)
|
|
info, err := os.Lstat(cleaned)
|
|
if err != nil || !info.Mode().IsRegular() {
|
|
return "", false
|
|
}
|
|
resolved, err := filepath.EvalSymlinks(cleaned)
|
|
if err != nil {
|
|
return "", false
|
|
}
|
|
rel, err := filepath.Rel(wsRoot, resolved)
|
|
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
return "", false
|
|
}
|
|
return cleaned, true
|
|
}
|
|
|
|
// confineToAnyRoot accepts mediaPath if it is contained by ANY of the allowed
|
|
// roots (each checked with the hardened confineToWorkspace). Used for the
|
|
// result.Media egress: a tool's media legitimately lives in the agent
|
|
// workspace, the team workspace, OR a tenant-allowed path — the same scopes the
|
|
// producing tools (create_*, send_file, delegate) validate against. A path
|
|
// outside every root (e.g. /etc/passwd from a prompt-injected path) is rejected,
|
|
// so the egress guard holds without dropping legitimate cross-workspace media.
|
|
func confineToAnyRoot(mediaPath string, roots []string) (string, bool) {
|
|
for _, root := range roots {
|
|
if root == "" {
|
|
continue
|
|
}
|
|
if cleaned, ok := confineToWorkspace(mediaPath, root); ok {
|
|
return cleaned, true
|
|
}
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// extractMediaFromContent scans text for MEDIA:<path> tokens the LLM may echo
|
|
// in its final response (e.g. when a tool returned the MEDIA: prefix as plain
|
|
// text instead of setting Result.Media). Relative paths are resolved against
|
|
// workspace. Called before sanitize strips the tokens so the attachments are
|
|
// still delivered.
|
|
//
|
|
// Security: only paths accepted by confineToWorkspace are emitted. An LLM cannot
|
|
// inject attachments pointing at /etc/passwd, a sibling tenant's workspace, or a
|
|
// hallucinated path — the extractor silently drops them.
|
|
func extractMediaFromContent(content, workspace string) []MediaResult {
|
|
if !strings.Contains(content, "MEDIA:") || workspace == "" {
|
|
return nil
|
|
}
|
|
matches := mediaPathPattern.FindAllString(content, -1)
|
|
if len(matches) == 0 {
|
|
return nil
|
|
}
|
|
results := make([]MediaResult, 0, len(matches))
|
|
seen := make(map[string]struct{}, len(matches))
|
|
for _, m := range matches {
|
|
path := strings.TrimSpace(strings.TrimPrefix(m, "MEDIA:"))
|
|
if path == "" {
|
|
continue
|
|
}
|
|
// Drop markdown/JSON trailing punctuation that would otherwise stick:
|
|
// ")", "]", "\"", "'", ",", ";", ".".
|
|
path = strings.TrimRight(path, `)]"',;.`)
|
|
if path == "" {
|
|
continue
|
|
}
|
|
cleaned, ok := confineToWorkspace(path, workspace)
|
|
if !ok {
|
|
continue
|
|
}
|
|
if _, dup := seen[cleaned]; dup {
|
|
continue
|
|
}
|
|
seen[cleaned] = struct{}{}
|
|
results = append(results, MediaResult{
|
|
Path: cleaned,
|
|
ContentType: mimeFromExt(filepath.Ext(cleaned)),
|
|
})
|
|
}
|
|
return results
|
|
}
|
|
|
|
// deduplicateMedia removes duplicate media results by path, keeping the first
|
|
// occurrence. Exact-string match is the ONLY safe comparison: filepath.Abs
|
|
// normalization depends on the process CWD, which varies across deployment
|
|
// environments and was observed to drop legitimate entries in production.
|
|
// The tiny cost of an occasional aliased-path duplicate (e.g. "./x" vs "/abs/x")
|
|
// is preferable to silently eating a real attachment.
|
|
func deduplicateMedia(media []MediaResult) []MediaResult {
|
|
if len(media) <= 1 {
|
|
return media
|
|
}
|
|
seen := make(map[string]bool, len(media))
|
|
result := make([]MediaResult, 0, len(media))
|
|
for _, m := range media {
|
|
if seen[m.Path] {
|
|
continue
|
|
}
|
|
seen[m.Path] = true
|
|
result = append(result, m)
|
|
}
|
|
return result
|
|
}
|
|
|
|
// mimeFromExt returns a MIME type for common media file extensions.
|
|
func mimeFromExt(ext string) string {
|
|
switch strings.ToLower(ext) {
|
|
case ".png":
|
|
return "image/png"
|
|
case ".jpg", ".jpeg":
|
|
return "image/jpeg"
|
|
case ".gif":
|
|
return "image/gif"
|
|
case ".webp":
|
|
return "image/webp"
|
|
case ".mp4":
|
|
return "video/mp4"
|
|
case ".ogg", ".opus":
|
|
return "audio/ogg"
|
|
case ".mp3":
|
|
return "audio/mpeg"
|
|
case ".wav":
|
|
return "audio/wav"
|
|
case ".txt":
|
|
return "text/plain"
|
|
case ".pdf":
|
|
return "application/pdf"
|
|
case ".csv":
|
|
return "text/csv"
|
|
case ".json":
|
|
return "application/json"
|
|
case ".html", ".htm":
|
|
return "text/html"
|
|
case ".xml":
|
|
return "application/xml"
|
|
case ".zip":
|
|
return "application/zip"
|
|
case ".doc", ".docx":
|
|
return "application/msword"
|
|
case ".xls", ".xlsx":
|
|
return "application/vnd.ms-excel"
|
|
case ".md":
|
|
return "text/markdown"
|
|
default:
|
|
return "application/octet-stream"
|
|
}
|
|
}
|