Files
goclaw/internal/providers/codex_native_image_test.go
Kai (Tam Nhu) Tran 4b02c27307 feat: native image_generation for Codex + OpenAI-compat providers (#1002)
* refactor(providers): migrate ToolDefinition.Function to pointer + add image response fields

ToolDefinition.Function becomes *ToolFunctionSchema with omitempty so native tool types (image_generation, web_search, etc.) can be declared without a function body. All 9 internal construction sites updated. CleanToolSchemas refactored — function-shape cleaning extracted into cleanFunctionSchema helper, outer pass-through handles native tools.

Added image response fields needed by the next commits: ChatResponse.Images, StreamChunk.Images, ImageContent.Partial (distinguishes partial frames from final images).

* feat(providers): native image_generation for Codex + OpenAI-compat tracks

Codex native (POST /codex/responses): emit image_generation tool object in request tools[] (type, action, model, output_format, partial_images). Handle SSE events response.image_generation_call.partial_image + response.output_item.done (type image_generation_call) + response.completed output[] walk for non-stream. Dedup per item_id. Extend codexSSEEvent/codexItem with output_format, result, partial_image_b64, partial_image_index.

OpenAI-compat (/v1/chat/completions): serialize ToolDefinition{Type:'image_generation'} as {type:'image_generation'} pass-through. Parse choices[0].message.images[] + delta.images[] (data URLs) via new parseDataURL helper; append to ChatResponse.Images.

ProviderCapabilities.ImageGeneration flag; Codex provider + adapter set true. Other providers default false.

* feat(agent,http,store): persist assistant images + tri-level image_generation gate

Agent loop tri-level gate: (provider capability) AND (AgentConfig.AllowImageGeneration, default true, stored in other_config.allow_image_generation) AND (request lacks x-goclaw-no-image-gen header). Gate in loop_tool_filter.go appends ToolDefinition{Type:'image_generation'} only when all three pass. Per-request opt-out parsed in chat_completions.go and propagated via RunRequest.NoImageGen.

Media persistence: persistAssistantImages writes final images (Partial:false) to {workspace}/media/{sha256}.{ext}, returns MediaRef entries, clears inline Images[] from the message. Idempotent on hash, traversal-safe, symlink-guarded. Invoked from pipeline.FinalizeStage via new Deps.PersistAssistantImages callback — covers both stream-final and non-stream paths.

Agent store reads AllowImageGeneration from other_config JSONB with absent/nil = true default (matches V3Flags pattern). No DB migration — code-only default.

* feat(ui/web): image_generation toggle + streaming placeholder + download filename

Composer chip 'Images' visible only when active agent's provider has ImageGeneration capability. Per-agent localStorage persistence via useImageGenToggle hook. When off, sends noImageGen:true to WS chat method (maps to x-goclaw-no-image-gen on upstream HTTP call path).

ActiveRunZone renders a skeleton placeholder while streaming partial_image frames arrive. MediaGallery assigns generated-YYYYMMDD-HHmmss.png as the download filename for hex/UUID PNGs.

i18n keys added to en/vi/zh chat.json: imageGenToggle, imageGenGenerating, imageGenDownloadName. 8 vitest tests for the toggle hook.

* docs: add Image Generation section to codebase-summary + changelog entry

Documents the new native image_generation pipeline across providers layer (Codex + OpenAI-compat), agent gate, media persistence, and web UI surface.

* fix(ui/web): match Codex-routed providers for image_generation toggle

Image-gen toggle visibility was hard-coded to provider id 'chatgpt_oauth' but real Codex-routed agents in production use provider ids like 'cliproxy-codex'. The toggle never rendered.

Replace the Set-has check with a small helper that accepts the literal ids plus any provider string containing 'codex' (case-insensitive). Same logic applied in both chat-input.tsx (composer chip) and chat-page.tsx (streaming placeholder gate).

Verified against a live Codex-routed agent: toggle now renders, noImageGen:true propagates on toggle-off.

* docs(pr-1002): targeted-mode UX evidence report

Captures the UI integration surface for native image_generation against a live Codex-routed agent on the remote dev backend.

Includes: composer toggle chip (rendered), streaming skeleton placeholder, and honest failure-path capture showing the legacy create_image builtin fallback. Self-contained HTML report in .github/pr-assets/1002/index.html.

* fix(permissions): classify sessions.compact as write method

CI RBAC-drift test (TestMethodRole_DriftCoverage_AllProtocolMethodsClassified) was failing because the new sessions.compact method added upstream was not classified in any of isPublicMethod / isAdminMethod / isWriteMethod / isReadMethod.

Sessions compaction mutates session history (compacts messages into summaries), so it belongs with the other sessions.* write methods.

* fix(tests): remove duplicate contains() in integration package

Both tts_gemini_live_test.go and mcp_grant_revoke_test.go declared a file-local func contains(s, substr string) bool with identical bodies, causing 'contains redeclared in this block' at compile time in the integration job.

Replace all call sites with strings.Contains (same semantics, stdlib) and drop the duplicates. No behavior change.

* feat(providers): NativeImageProvider interface + Codex implementation

Defines a provider-level contract (NativeImageProvider.GenerateImage) that OAuth-backed providers can implement to serve image generation without exposing static API credentials. Re-uses the PR's Track A native wire format (POST /codex/responses with image_generation tool, item.result decoding, SSE fallback).

CodexProvider + CodexAdapter implement it. Also adds MediaRef.Prompt field so downstream layers can propagate the generating prompt alongside the asset.

* feat(tools): route create_image through NativeImageProvider for OAuth providers

create_image.callProvider now checks for a NativeImageProvider implementation before the credentialProvider interface. When the provider chain points at a Codex-family provider (no static API key), the tool delegates to the provider's GenerateImage which executes the native ChatGPT Responses API image_generation flow.

Resolves 'provider X does not expose API credentials required for image generation' errors for openai-codex / cliproxy-codex chains. On success the tool embeds the user's prompt as a PNG tEXt 'Description' chunk (file-local pngEmbedPrompt helper to avoid tools→agent import cycle), writes the image to /tmp, and threads the prompt through result.MediaPrompts for downstream MediaRef propagation.

* feat(agent,pipeline): propagate image prompt through MediaRef + PNG tEXt embed helper

Adds EmbedPNGPrompt public helper in internal/agent/png_metadata.go that inserts a tEXt 'Description' chunk (plus 'Software: goclaw') into PNG byte streams before the IEND chunk. Non-PNG inputs are passed through unchanged — the helper never errors on unknown formats.

FinalizeStage wires MediaResult.Prompt (from create_image tool output) onto MediaRef.Prompt so the UI can render the generating prompt alongside the image. Per-image prompt list threaded via pipeline RunState.

* feat(ui/web): show generating prompt as caption under each image in MediaGallery

When a MediaRef carries a prompt, MediaGallery renders it as a muted italic caption (line-clamp-2) beneath the image with the full text in the title tooltip. Caption is hidden when the prompt is absent so non-assistant images (user uploads, legacy data) look unchanged.

MediaItem + session media_refs types extended with an optional prompt field; the chat-message adapter threads ref.prompt through when converting WS payloads to UI state.

* fix(providers/codex): stream:true + instructions for native image_generation

The ChatGPT Responses API on /codex/responses rejects two things hard:

- stream:false → HTTP 400 "Stream must be set to true"

- missing instructions → HTTP 400 "Instructions are required"

buildNativeImageRequestBody now sets stream:true and a purpose-specific instructions string ("Generate an image matching the user's description using the image_generation tool. Return only the image; do not describe it in text."). The existing parseNativeImageSSE path was already in place for stream parsing; routing changed from the non-stream branch to the SSE branch.

Regression assertions added to TestCodexGenerateImage_BuildsNativeRequest so these two fields can't silently regress.

* feat(providers,tools,ui): image_model whitelist (gpt-image-2 default, gpt-image-1.5 legacy)

Replaces the hardcoded "gpt-image-2" literal in buildNativeImageRequestBody with a user-configurable field threaded through NativeImageRequest.ImageModel. The whitelist is enforced by ValidateImageModel which rejects anything outside {gpt-image-2, gpt-image-1.5} with a clear error — prevents silent upstream 400s from model names the Responses API would reject.

create_image.callProvider reads params.image_model from the chain entry and threads it through. Empty / absent falls back to DefaultImageModel (gpt-image-2).

UI: added an 'Image model' select inside the existing openai-codex Settings panel on the Create Image Provider Chain dialog. Options: Default · gpt-image-2 (recommended) and Legacy · gpt-image-1.5. i18n keys in en/vi/zh tools.json under builtin.mediaChain.

Tests: TestCodexGenerateImage covers default/legacy/rejected model cases; TestCreateImageTool_ThreadsImageModel covers params→request threading with empty/legacy/explicit sub-cases.

* fix(tools): raise media chain default timeout to 600s/1 retry for image gen

gpt-image-2 on complex prompts (dense Vietnamese text, infographic layouts) legitimately takes 4–8 minutes to complete. The previous default of 120s × 2 retries routinely died mid-generation with 'context deadline exceeded' — the upstream run was still producing bytes when our ctx cancelled.

Default is now Timeout: 600 / MaxRetries: 1. Retries dropped to 1 because image generation is stateful per upstream run: a mid-flight timeout leaves orphan server work, and retrying a fresh generation doubles cost for no gain. Surface the failure fast so operators can widen the timeout instead.

Operators can still set a tighter value explicitly via the Chain dialog.

* refactor: remove user-facing Images toggle, keep admin-level AllowImageGeneration

The per-request opt-out toggle (composer chip + streaming placeholder + noImageGen header plumbing) was a support footgun — users toggle OFF, forget, then can't generate images and think it's broken. Removed in full.

Kept: AgentConfig.AllowImageGeneration (admin kill-switch, stored in other_config.allow_image_generation, default true). Tri-level gate simplifies to two tiers: provider capability AND agent config allows.

Removed: useImageGenToggle hook, IMAGE_GEN_PROVIDER_IDS set in chat-input, supportsImageGenProvider helper, agentProvider/agentKey props on ChatInput, showImageGenPlaceholder prop on MessageBubble/ActiveRunZone/ChatThread, noImageGen param on use-chat-send, parseNoImageGen in chat_completions.go, NoImageGen on RunRequest, no_image_gen_header_test.go, imageGenToggle/imageGenGenerating i18n keys. Kept imageGenDownloadName — used by MediaGallery for generated-\*.png filename resolution.

* docs(pr-1002): refreshed UX trace + updated codebase notes

Replaces the earlier stealth-state evidence with a clean three-capture trace from a real successful run: inline image + prompt caption, MediaGallery lightbox expansion, Chain dialog with the new Image model dropdown open. Skill-routing rows scrubbed from the capture — they reflect per-agent skill setup, not anything this PR introduces.

codebase-summary + changelog: reflect final state (toggle removed, image_model selector, 600s default chain timeout, gpt-image-2 as quality baseline).

* fix(pipeline): preserve mid-loop image_generation output across iterations

FinalizeStage previously read state.Think.LastResponse.Images, which holds
only the last iteration's response. If the LLM emitted image_generation_call
in iteration N alongside a function_call, then responded text-only in N+1,
the image from N was silently dropped on finalize.

Accumulate final (non-partial) images into state.Observe.AssistantImages
across every iteration via ObserveStage, and source FinalizeStage from the
accumulator instead of LastResponse. Partial streaming frames are filtered
defensively; response.Images is cleared on drain to prevent double-counting
on re-exec.

* test(pipeline): regression coverage for mid-loop image accumulation

Six ObserveStage cases covering image accumulation semantics:
- single-iter image-only, image+tool_call same iter, mid-loop image
  surviving text-only final iter, multiple images across iters, partial
  frame filtering, nil-response safety.

Two FinalizeStage cases verifying accumulator is the source of truth:
- PersistsFromObserveAccumulator: image in Observe + empty LastResponse
  must still be persisted via PersistAssistantImages.
- NoPersistWhenAccumulatorEmpty: no call when no images were emitted.

---------

Co-authored-by: viettranx <viettranx@gmail.com>
2026-04-23 08:22:39 +07:00

347 lines
12 KiB
Go

package providers
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// minimalPNGForProviders is a 1x1 transparent PNG in base64 used by native image tests.
const minimalPNGForProviders = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
// mockImageServer returns a test server that captures request bodies and returns a
// minimal successful image generation response. The captured pointer is written on
// each request.
func mockImageServer(t *testing.T, captured *[]byte) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
t.Errorf("read body: %v", err)
http.Error(w, "read error", http.StatusInternalServerError)
return
}
*captured = body
resp := map[string]any{
"id": "resp_test",
"status": "completed",
"output": []map[string]any{
{
"type": "image_generation_call",
"result": minimalPNGForProviders,
"output_format": "png",
},
},
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(resp); err != nil {
t.Errorf("encode response: %v", err)
}
}))
}
// TestCodexGenerateImage_BuildsNativeRequest verifies that GenerateImage sends the
// correct JSON body to the Responses API: model, stream:false, input, tools, and
// tool_choice. The test captures the raw request body from a mock server and
// asserts each required field is present and well-formed.
//
// Sub-cases:
// - Default (empty ImageModel) → tools[0].model == "gpt-image-2"
// - Legacy (ImageModel: "gpt-image-1.5") → tools[0].model == "gpt-image-1.5"
// - Rejected (ImageModel: "dall-e-3") → GenerateImage returns error containing "unsupported image model"
func TestCodexGenerateImage_BuildsNativeRequest(t *testing.T) {
var captured []byte
server := mockImageServer(t, &captured)
defer server.Close()
p := NewCodexProvider("codex-test", &staticTokenSource{token: "tok"}, server.URL, "gpt-image-2")
p.retryConfig.Attempts = 1
req := NativeImageRequest{
Model: "gpt-image-2",
Prompt: "A red circle on a white background",
AspectRatio: "16:9",
OutputFormat: "png",
}
result, err := p.GenerateImage(context.Background(), req)
if err != nil {
t.Fatalf("GenerateImage returned error: %v", err)
}
if len(result.Data) == 0 {
t.Fatal("GenerateImage returned empty Data")
}
// Verify outbound request body shape.
var body map[string]any
if err := json.Unmarshal(captured, &body); err != nil {
t.Fatalf("unmarshal captured body: %v", err)
}
// model field (outer Responses API model, not image model)
if model, _ := body["model"].(string); model != "gpt-image-2" {
t.Errorf("body[model] = %q, want %q", model, "gpt-image-2")
}
// Responses API requires stream:true — non-streaming requests are rejected with
// HTTP 400 "Stream must be set to true". Final image is assembled from SSE events.
if stream, _ := body["stream"].(bool); !stream {
t.Error("body[stream] must be true (Responses API rejects stream:false)")
}
// instructions is required by Responses API — must be non-empty.
if instr, _ := body["instructions"].(string); instr == "" {
t.Error("body[instructions] must be non-empty (Responses API rejects requests without instructions)")
}
// input must be an array with one user message
inputs, ok := body["input"].([]any)
if !ok || len(inputs) != 1 {
t.Fatalf("body[input]: expected []any length 1, got %T len %d", body["input"], len(inputs))
}
userMsg, ok := inputs[0].(map[string]any)
if !ok {
t.Fatalf("input[0] is not a map: %T", inputs[0])
}
if role, _ := userMsg["role"].(string); role != "user" {
t.Errorf("input[0].role = %q, want %q", role, "user")
}
contents, ok := userMsg["content"].([]any)
if !ok || len(contents) != 1 {
t.Fatalf("input[0].content: expected []any length 1, got %T len %d", userMsg["content"], len(contents))
}
contentPart, ok := contents[0].(map[string]any)
if !ok {
t.Fatalf("content[0] is not a map: %T", contents[0])
}
if typ, _ := contentPart["type"].(string); typ != "input_text" {
t.Errorf("content[0].type = %q, want %q", typ, "input_text")
}
if text, _ := contentPart["text"].(string); text != req.Prompt {
t.Errorf("content[0].text = %q, want %q", text, req.Prompt)
}
// tools must contain one image_generation entry
tools, ok := body["tools"].([]any)
if !ok || len(tools) != 1 {
t.Fatalf("body[tools]: expected []any length 1, got %T len %d", body["tools"], len(tools))
}
tool, ok := tools[0].(map[string]any)
if !ok {
t.Fatalf("tools[0] is not a map: %T", tools[0])
}
if typ, _ := tool["type"].(string); typ != "image_generation" {
t.Errorf("tools[0].type = %q, want %q", typ, "image_generation")
}
// size should map to 1792x1024 for 16:9
wantSize := SizeFromAspect("16:9")
if size, _ := tool["size"].(string); size != wantSize {
t.Errorf("tools[0].size = %q, want %q", size, wantSize)
}
if fmt.Sprint(tool["output_format"]) != "png" {
t.Errorf("tools[0].output_format = %v, want png", tool["output_format"])
}
// tools[0].model must be gpt-image-2 (default when ImageModel is empty)
if imgModel, _ := tool["model"].(string); imgModel != DefaultImageModel {
t.Errorf("tools[0].model = %q, want %q (default)", imgModel, DefaultImageModel)
}
// tool_choice must force image_generation
toolChoice, ok := body["tool_choice"].(map[string]any)
if !ok {
t.Fatalf("body[tool_choice] is not a map: %T", body["tool_choice"])
}
if typ, _ := toolChoice["type"].(string); typ != "image_generation" {
t.Errorf("tool_choice.type = %q, want %q", typ, "image_generation")
}
}
// TestCodexGenerateImage_ImageModelDefault verifies that an empty ImageModel results
// in the default gpt-image-2 model in the outbound tools[0].model field.
func TestCodexGenerateImage_ImageModelDefault(t *testing.T) {
var captured []byte
server := mockImageServer(t, &captured)
defer server.Close()
p := NewCodexProvider("codex-test", &staticTokenSource{token: "tok"}, server.URL, "gpt-image-2")
p.retryConfig.Attempts = 1
_, err := p.GenerateImage(context.Background(), NativeImageRequest{
Prompt: "test",
ImageModel: "", // explicitly empty — should default to gpt-image-2
AspectRatio: "1:1",
})
if err != nil {
t.Fatalf("GenerateImage returned error: %v", err)
}
var body map[string]any
if err := json.Unmarshal(captured, &body); err != nil {
t.Fatalf("unmarshal captured body: %v", err)
}
tools, _ := body["tools"].([]any)
if len(tools) == 0 {
t.Fatal("tools array is empty")
}
tool, _ := tools[0].(map[string]any)
if imgModel, _ := tool["model"].(string); imgModel != "gpt-image-2" {
t.Errorf("tools[0].model = %q, want gpt-image-2 (default)", imgModel)
}
}
// TestCodexGenerateImage_ImageModelLegacy verifies that ImageModel "gpt-image-1.5"
// is forwarded to the outbound tools[0].model field.
func TestCodexGenerateImage_ImageModelLegacy(t *testing.T) {
var captured []byte
server := mockImageServer(t, &captured)
defer server.Close()
p := NewCodexProvider("codex-test", &staticTokenSource{token: "tok"}, server.URL, "gpt-image-2")
p.retryConfig.Attempts = 1
_, err := p.GenerateImage(context.Background(), NativeImageRequest{
Prompt: "test",
ImageModel: "gpt-image-1.5",
AspectRatio: "1:1",
})
if err != nil {
t.Fatalf("GenerateImage returned error: %v", err)
}
var body map[string]any
if err := json.Unmarshal(captured, &body); err != nil {
t.Fatalf("unmarshal captured body: %v", err)
}
tools, _ := body["tools"].([]any)
if len(tools) == 0 {
t.Fatal("tools array is empty")
}
tool, _ := tools[0].(map[string]any)
if imgModel, _ := tool["model"].(string); imgModel != "gpt-image-1.5" {
t.Errorf("tools[0].model = %q, want gpt-image-1.5 (legacy)", imgModel)
}
}
// TestCodexGenerateImage_ImageModelRejected verifies that an unsupported image model
// causes GenerateImage to return an error containing "unsupported image model" before
// making any HTTP request.
func TestCodexGenerateImage_ImageModelRejected(t *testing.T) {
requestMade := false
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requestMade = true
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
p := NewCodexProvider("codex-test", &staticTokenSource{token: "tok"}, server.URL, "gpt-image-2")
p.retryConfig.Attempts = 1
_, err := p.GenerateImage(context.Background(), NativeImageRequest{
Prompt: "test",
ImageModel: "dall-e-3",
})
if err == nil {
t.Fatal("expected error for unsupported image model, got nil")
}
if !strings.Contains(err.Error(), "unsupported image model") {
t.Errorf("error %q does not contain 'unsupported image model'", err.Error())
}
if requestMade {
t.Error("HTTP request was made despite invalid image model (should have been rejected before the request)")
}
}
// TestCodexGenerateImage_SSEFallback verifies that GenerateImage correctly parses
// an SSE-format response when the server returns streamed lines instead of a JSON blob.
func TestCodexGenerateImage_SSEFallback(t *testing.T) {
imgB64 := minimalPNGForProviders
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/event-stream")
// Emit a response.completed SSE event with an image_generation_call.
ev := codexSSEEvent{
Type: "response.completed",
Response: &codexAPIResponse{
ID: "resp_sse",
Status: "completed",
Output: []codexItem{
{
ID: "ig_1",
Type: "image_generation_call",
OutputFormat: "png",
Result: imgB64,
},
},
Usage: &codexUsage{InputTokens: 5, OutputTokens: 5, TotalTokens: 10},
},
}
b, _ := json.Marshal(ev)
fmt.Fprintf(w, "data: %s\n\n", b)
fmt.Fprint(w, "data: [DONE]\n\n")
}))
defer server.Close()
p := NewCodexProvider("codex-test", &staticTokenSource{token: "tok"}, server.URL, "gpt-image-2")
p.retryConfig.Attempts = 1
result, err := p.GenerateImage(context.Background(), NativeImageRequest{
Prompt: "A blue square",
OutputFormat: "png",
})
if err != nil {
t.Fatalf("GenerateImage SSE fallback: %v", err)
}
if result.MimeType != "image/png" {
t.Errorf("MimeType = %q, want image/png", result.MimeType)
}
want, _ := base64.StdEncoding.DecodeString(imgB64)
if len(result.Data) != len(want) {
t.Errorf("Data length = %d, want %d", len(result.Data), len(want))
}
if result.Usage == nil {
t.Error("Usage is nil")
} else if result.Usage.TotalTokens != 10 {
t.Errorf("Usage.TotalTokens = %d, want 10", result.Usage.TotalTokens)
}
}
// TestCodexGenerateImage_NoPrompt verifies that an empty prompt returns an error
// before making any HTTP request.
func TestCodexGenerateImage_NoPrompt(t *testing.T) {
p := NewCodexProvider("codex-test", &staticTokenSource{token: "tok"}, "http://localhost", "gpt-image-2")
p.retryConfig.Attempts = 1
_, err := p.GenerateImage(context.Background(), NativeImageRequest{Prompt: ""})
if err == nil {
t.Fatal("expected error for empty prompt, got nil")
}
}
// TestSizeFromAspect verifies the aspect ratio → pixel dimension mapping.
func TestSizeFromAspect(t *testing.T) {
cases := []struct {
ratio string
want string
}{
{"1:1", "1024x1024"},
{"16:9", "1792x1024"},
{"9:16", "1024x1792"},
{"4:3", "1365x1024"},
{"3:4", "1024x1365"},
{"", "1024x1024"},
{"custom", "1024x1024"},
}
for _, tc := range cases {
got := SizeFromAspect(tc.ratio)
if got != tc.want {
t.Errorf("SizeFromAspect(%q) = %q, want %q", tc.ratio, got, tc.want)
}
}
}