goclaw/docs/tts-provider-capabilities.md

TTS Provider Capabilities Schema

Reference for the ProviderCapabilities schema, per-provider parameter tables, storage format, and the checklist for adding a new TTS provider.

---

1. ProviderCapabilities Schema

Go Types

// ProviderCapabilities describes everything the frontend needs to render
// dynamic per-provider settings and validate user input.
type ProviderCapabilities struct {
    Provider       string            // provider ID: "openai", "gemini", etc.
    DisplayName    string            // human-readable name
    RequiresAPIKey bool              // show API key field
    Models         []string          // static model list; nil = user freetext
    Voices         []VoiceOption     // static voices; nil = dynamic or none
    Params         []ParamSchema     // ordered list of configurable params
    CustomFeatures map[string]any    // provider-specific UI flags
}

// ParamSchema describes one configurable parameter.
type ParamSchema struct {
    Key         string       // dot-delimited storage key, e.g. "voice_settings.stability"
    Type        ParamType    // "range" | "integer" | "boolean" | "enum" | "string" | "text"
    Label       string       // English label (UI falls back to this if i18n key missing)
    Description string       // English help text (optional)
    Default     any          // default value (nil = omit from request)
    Min         *float64     // range/integer lower bound
    Max         *float64     // range/integer upper bound
    Step        *float64     // slider increment
    Enum        []EnumOption // allowed values for "enum" type
    DependsOn   []Dependency // show/hide rules
}

// Dependency expresses a show/hide condition on another param's value.
type Dependency struct {
    Field string // key of the controlling param
    Op    string // "eq" (only operator currently supported)
    Value string // required value (string-compared after JSON marshal)
}

// EnumOption is one choice inside an enum param.
type EnumOption struct {
    Value string
    Label string
}

ParamType values

Constant	Wire value	UI widget
ParamTypeRange	"range"	Slider + numeric input
ParamTypeInteger	"integer"	Integer slider / stepper
ParamTypeBoolean	"boolean"	Checkbox / toggle
ParamTypeEnum	"enum"	Select dropdown
ParamTypeString	"string"	Single-line text input
ParamTypeText	"text"	Multi-line textarea

DependsOn AND semantics

Multiple DependsOn entries are evaluated with AND logic: the param is shown only when all conditions are satisfied simultaneously.

// Shown only when format=="mp3":
DependsOn: []audio.Dependency{
    {Field: "audio.format", Op: "eq", Value: "mp3"},
}

// Shown only when provider=="gemini" AND mode=="multi":
DependsOn: []audio.Dependency{
    {Field: "provider", Op: "eq", Value: "gemini"},
    {Field: "mode",     Op: "eq", Value: "multi"},
}

CustomFeatures flags

Key	Type	Meaning
multi_speaker	bool	Provider supports multi-speaker mode (Gemini)
audio_tags	bool	Provider supports inline audio expression tags (Gemini)
voices_dynamic	bool	Voices must be fetched at runtime via VoiceListProvider (MiniMax)

Read-only contract

Capabilities() must be a pure function — no side effects, no I/O, no locking. The manager calls it during startup to populate the catalog and again on each GET /v1/tts/capabilities request.

---

2. Per-Provider Parameter Tables

OpenAI (internal/audio/openai/)

Key	Type	Default	Range / Options	DependsOn
speed	range	1.0	0.25–4.0, step 0.05	—
response_format	enum	"mp3"	mp3, opus, aac, flac, wav, pcm	—
instructions	text	""	—	model == "gpt-4o-mini-tts"

ElevenLabs (internal/audio/elevenlabs/)

Key	Type	Default	Range / Options	DependsOn
voice_settings.stability	range	0.5	0–1, step 0.01	—
voice_settings.similarity_boost	range	0.75	0–1, step 0.01	—
voice_settings.style	range	0.0	0–1, step 0.01	—
voice_settings.use_speaker_boost	boolean	true	—	—
voice_settings.speed	range	1.0	0.7–1.2, step 0.01	—
apply_text_normalization	enum	"" (auto)	auto, on, off	—
seed	integer	0	0–2³²−1	—
optimize_streaming_latency	range	0	0–4, step 1	—
language_code	string	""	ISO 639-1	—

Edge TTS (internal/audio/edge/)

No API key required. Full voice list available via edge-tts --list-voices.

Key	Type	Default	Range / Options	DependsOn
rate	integer	0	-50–+100 (%)	—
pitch	integer	0	-50–+50 (Hz)	—
volume	integer	0	-50–+100 (%)	—

MiniMax (internal/audio/minimax/)

Voices are dynamic — fetched at runtime via VoiceListProvider. CustomFeatures["voices_dynamic"] = true signals the frontend to call GET /v1/voices?provider=minimax.

Key	Type	Default	Range / Options	DependsOn
speed	range	1.0	0.5–2.0, step 0.1	—
vol	range	1.0	0.01–10.0, step 0.01	—
pitch	integer	0	-12–+12 (semitones)	—
emotion	enum	""	none/happy/sad/angry/fearful/disgusted/surprised/neutral/excited/anxious	—
text_normalization	boolean	nil (omit)	—	—
audio.format	enum	"mp3"	mp3, pcm, flac, wav	—
audio.sample_rate	enum	"" (default)	8000/16000/22050/24000/32000/44100	—
audio.bitrate	enum	"" (default)	32000/64000/128000/256000 bps	audio.format == "mp3"
audio.channel	enum	"" (default)	1=mono, 2=stereo	—

Gemini (internal/audio/gemini/)

No configurable Params — advanced features exposed via CustomFeatures.

Feature	CustomFeatures key	Notes
Multi-speaker	multi_speaker: true	Up to 2 speakers; each has name + voice
Audio tags	audio_tags: true	Inline [laughs], [sighs], etc.
Preview badge	—	Models are preview-stage; UI shows badge

Sentinel validation errors (422 Unprocessable Entity):

Error	i18n key
ErrInvalidVoice	MsgTtsGeminiInvalidVoice
ErrSpeakerLimit	MsgTtsGeminiSpeakerLimit
ErrInvalidModel	MsgTtsGeminiInvalidModel

All 422 error messages are run through i18n.T(locale, key, args...) using the locale extracted from Accept-Language by requireAuth → enrichContext.

---

3. Storage Format — Dual-Read / Dual-Write

TTS configuration is persisted in system_configs (tenant-scoped key-value store). Two layouts coexist for backward compatibility:

Legacy flat keys

tts.provider            → "openai"
tts.openai.api_base     → "https://api.openai.com/v1"
tts.openai.voice        → "nova"
tts.openai.model        → "tts-1"
tts.gemini.speakers     → [{"name":"Alice","voice":"Kore"}]  (JSON array)

Params blob (Phase C+)

tts.<provider>.params   → {"speed":1.2,"response_format":"mp3"}  (JSON object)

Dual-read logic (resolveTenantProvider)

1. Read legacy flat keys to populate testConnectionRequest fields.
2. Read tts.<provider>.params blob via loadParamsBlob().
3. Blob wins on key conflict — blob values override legacy flat keys for the same logical param.
4. Legacy flat keys fill gaps not covered by the blob.

Rollback

Reverting Phase C+ is non-destructive: legacy flat keys remain valid; the blob is simply ignored when loadParamsBlob returns nil.

---

4. Adding a New Provider

Follow these touchpoints in order. Each step is required; skipping any causes a compile or runtime failure.

Step 1 — Provider package

Create internal/audio/<name>/tts.go implementing audio.TTSProvider:

type Provider struct { /* config fields */ }
func (p *Provider) Name() string { return "<name>" }
func (p *Provider) Synthesize(ctx context.Context, text string, opts audio.TTSOptions) (*audio.SynthResult, error)
func (p *Provider) Capabilities() audio.ProviderCapabilities

Follow the SSRF guard pattern for external HTTP calls (validate apiBase via validateProviderURL before creating the HTTP client).

Step 2 — Capabilities()

Return a fully populated audio.ProviderCapabilities.

• Set RequiresAPIKey: true/false.
• Populate Params with []audio.ParamSchema; use &float64 literals for Min/Max/Step (package-level var to avoid address-of-literal).
• Defaults must match the values your Synthesize applies when the param is absent from opts.Params.
• If voices are dynamic, set CustomFeatures["voices_dynamic"] = true and implement VoiceListProvider (see Step 6).

Step 3 — createEphemeralTTSProvider (internal/http/tts_test_connection.go)

Add a case "<name>": branch:

case "<name>":
    return newpkg.NewProvider(newpkg.Config{
        APIKey:    req.APIKey,
        APIBase:   req.APIBase,
        VoiceID:   req.VoiceID,
        ModelID:   req.ModelID,
        TimeoutMs: req.TimeoutMs,
    }), nil

Also add "<name>": true to supportedTestProviders and, if an API key is needed, to providersRequiringAPIKey.

Step 4 — resolveTenantProvider (internal/http/tts.go)

Add a case "<name>": branch reading the provider's config keys from h.systemConfigs / h.configSecrets and calling loadParamsBlob.

Step 5 — Wire sentinel errors (if any)

If the provider defines sentinel errors (like Gemini's ErrInvalidVoice), add errors.Is checks in both handleSynthesize and handleTestConnection, using i18n.T(locale, MsgXxx, ...) for the response body.

Step 6 — VoiceListProvider (dynamic voices only)

Implement the interface:

type VoiceListProvider interface {
    TTSProvider
    ListVoices(ctx context.Context) ([]VoiceOption, error)
}

The voices handler (GET /v1/voices?provider=<name>) uses a type-assertion to detect this interface at runtime — no registration needed.

Step 7 — tts_config.go schema validation

If the provider has required fields (e.g. group_id for MiniMax), add validation in internal/http/tts_config.go so PUT /v1/tts/config returns a clear 400 when mandatory fields are missing.

Step 8 — i18n keys

For each ParamSchema, add:

1. tts.<name>.<param_key>.label and tts.<name>.<param_key>.help to all 6 locale JSON files (web × 3 + desktop × 3).
2. Any backend validation error keys to internal/i18n/keys.go and all three catalog files (catalog_en.go, catalog_vi.go, catalog_zh.go).

Step 9 — Tests

• Unit test Capabilities() returns non-empty Provider + DisplayName.
• Test createEphemeralTTSProvider builds the provider without error.
• Test Synthesize against a mock HTTP server.
• If sentinel errors exist, test 422 body uses i18n.T for all locales (see TestSynthesize_GeminiInvalidVoice_I18n pattern in tts_test.go).

Consumers to check

File	What to verify
internal/http/tts.go	resolveTenantProvider case
internal/http/tts_test_connection.go	createEphemeralTTSProvider + allow-lists
internal/http/tts_config.go	Schema validation for required fields
internal/http/voices.go	VoiceListProvider type-assertion path
internal/audio/manager.go	Auto-discovers via RegisterProvider; no change needed
ui/web/src/i18n/locales/*/tts.json	Param label/help keys (3 locales)
ui/desktop/frontend/src/i18n/locales/*/tts.json	Desktop locale parity

---

5. Gemini Specifics

Models

Gemini TTS uses preview models only (as of 2026-04):

• gemini-3.1-flash-tts-preview (default — higher Elo, more stable)
• gemini-2.5-flash-preview-tts
• gemini-2.5-pro-preview-tts

The frontend displays a "Preview" badge (i18n key tts.gemini.previewBadge).

Multi-speaker mode

• Up to 2 speakers per request.
• Each speaker: {name: string, voice: string} — voice must be one of the 30 prebuilt Gemini voices.
• Serialized as tts.gemini.speakers JSON blob in system_configs.
• At synthesis time, loadSavedSpeakers restores the blob into opts.Speakers.

Audio tags

Inline expressive markers injected by the user directly into text:

Hello [laughs] world [sighs] how are you?

Categories: Emotion, Pacing, Effect, Voice quality. Full tag list is hardcoded in the frontend (gemini/audio-tags.ts).

Language support

Gemini TTS supports 70+ languages automatically based on input text — no explicit language parameter needed.

SSRF guard

api_base overrides for Gemini (and all providers) are validated by validateProviderURL in internal/http/tts_url_validator.go. Private IP ranges (127.x, 10.x, 192.168.x, etc.) and localhost are rejected with a 400 response to prevent Server-Side Request Forgery.