diff --git a/README.md b/README.md index 300e2190..ec27693d 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,10 @@ ccs proxy start hf eval "$(ccs proxy activate)" ``` +The proxy also supports request-time `profile:model` selectors, scenario-based +model routing through `proxy.routing`, and explicit activation helpers such as +`ccs proxy activate --fish`. + Guide: [OpenAI-Compatible Provider Routing](./docs/openai-compatible-providers.md) ### Related Project: claude-code-router diff --git a/docs/openai-compatible-providers.md b/docs/openai-compatible-providers.md index ddae7656..1cf86650 100644 --- a/docs/openai-compatible-providers.md +++ b/docs/openai-compatible-providers.md @@ -68,8 +68,22 @@ ccs proxy status ccs proxy stop ``` -`ccs proxy activate` prints the `ANTHROPIC_*` exports needed for a local -Anthropic-compatible session against the running proxy. +Useful variants: + +```bash +ccs proxy start hf --host 127.0.0.1 +ccs proxy activate --fish +``` + +`ccs proxy activate` now prints the full local runtime contract: + +- `ANTHROPIC_BASE_URL` +- `ANTHROPIC_AUTH_TOKEN` +- `ANTHROPIC_MODEL` plus tier defaults when present +- `DISABLE_TELEMETRY` +- `DISABLE_COST_WARNINGS` +- `API_TIMEOUT_MS` +- `NO_PROXY` ## One Active Proxy Profile @@ -82,6 +96,56 @@ The current runtime is a single local proxy daemon. This is intentional to avoid breaking an in-flight Claude session by swapping its upstream provider out from under it. +## Request-Time Routing + +The proxy is no longer limited to the startup profile's default model. + +Supported request-time selectors: + +- `profile:model` + Example: `deepseek:deepseek-reasoner` +- `profile` + Example: `openrouter` +- plain model ids + Example: `deepseek-chat` + +Routing behavior: + +1. `profile:model` wins immediately. +2. Scenario routing may override the active profile when configured. +3. Plain model ids are matched against the configured OpenAI-compatible + profiles before falling back to the active profile. + +This means a Claude session launched through one compatible profile can still +request another compatible profile/model when the proxy can resolve it safely. + +## Scenario Routing + +Scenario routing is now supported through `proxy.routing` in your CCS config. + +Example `~/.ccs/config.yaml`: + +```yaml +proxy: + routing: + default: "deepseek:deepseek-chat" + background: "ollama:qwen2.5-coder:0.5b" + think: "deepseek:deepseek-reasoner" + longContext: "openrouter:google/gemini-2.5-pro" + longContextThreshold: 60000 + webSearch: "openrouter:perplexity/sonar-pro" +``` + +Current scenario detection: + +- `background`: requested model contains `haiku` +- `think`: Anthropic `thinking` is enabled +- `longContext`: estimated request tokens exceed `longContextThreshold` +- `webSearch`: tool list includes `web_search` +- `default`: fallback selector when the above do not apply + +Routing decisions are logged through CCS structured logs. + ## How Profile Detection Works CCS keeps these profiles in the normal API/settings-profile flow. @@ -102,6 +166,79 @@ OpenAI-compatible endpoints such as: are routed through the local proxy for Claude-target launches. +## Provider Setup + +### DeepSeek + +Use a settings profile whose env looks like: + +```json +{ + "env": { + "ANTHROPIC_BASE_URL": "https://api.deepseek.com/v1", + "ANTHROPIC_AUTH_TOKEN": "sk-...", + "ANTHROPIC_MODEL": "deepseek-chat", + "CCS_DROID_PROVIDER": "generic-chat-completion-api" + } +} +``` + +Typical override target: + +- `deepseek:deepseek-reasoner` + +### OpenRouter + +```json +{ + "env": { + "ANTHROPIC_BASE_URL": "https://openrouter.ai/api/v1", + "ANTHROPIC_AUTH_TOKEN": "sk-or-...", + "ANTHROPIC_MODEL": "openai/gpt-4.1-mini", + "CCS_DROID_PROVIDER": "generic-chat-completion-api" + } +} +``` + +Useful when you want: + +- model fan-out behind one provider profile +- long-context or web-search scenario targets + +### Ollama / Local Gateways + +```json +{ + "env": { + "ANTHROPIC_BASE_URL": "http://127.0.0.1:11434", + "ANTHROPIC_AUTH_TOKEN": "ollama", + "ANTHROPIC_MODEL": "qwen3-coder", + "CCS_DROID_PROVIDER": "generic-chat-completion-api" + } +} +``` + +For self-signed HTTPS gateways, add `CCS_OPENAI_PROXY_INSECURE=1`. + +### DashScope / Qwen Compatible Mode + +DashScope's compatible endpoint works even when older settings files still +carry a stale Anthropic-style provider hint: + +```json +{ + "env": { + "ANTHROPIC_BASE_URL": "https://dashscope-us.aliyuncs.com/compatible-mode/v1", + "ANTHROPIC_AUTH_TOKEN": "sk-...", + "ANTHROPIC_MODEL": "qwen3.6-plus", + "CCS_DROID_PROVIDER": "anthropic" + } +} +``` + +CCS now infers the OpenAI-compatible route from the base URL and does not let +that stale provider hint block proxy routing. + ## Self-Signed TLS If your upstream gateway uses a self-signed or privately issued certificate, @@ -124,29 +261,50 @@ That flag is respected by both: - `ccs ` with Claude target: auto-starts the local proxy when needed - `ccs proxy start `: starts the proxy explicitly +- `GET /`: proxy info and bound profile details - `GET /health`: proxy liveness check - `GET /v1/models`: local view of the configured model mapping - `POST /v1/messages`: Anthropic-compatible request entrypoint +## Troubleshooting + +### Missing or invalid local proxy token + +- Re-run `eval "$(ccs proxy activate)"` +- Check `ccs proxy status` and confirm the expected profile is running + +### Self-signed or private CA upstream + +- Add `CCS_OPENAI_PROXY_INSECURE=1` to the profile settings +- Restart the proxy after changing the setting + +### Port conflict on `3456` + +- Start with a fixed port: `ccs proxy start hf --port 3457` +- Re-run `ccs proxy activate` after changing the port + +### Provider returns `429` or empty upstream output + +- CCS now preserves upstream rate-limit errors and retry headers +- Empty or malformed provider JSON is returned as Anthropic-style `api_error` + +### Requests route to the wrong model/profile + +- Use an explicit selector such as `profile:model` +- Review `proxy.routing` if scenario routing is enabled +- Check CCS structured logs in `~/.ccs/logs/current.jsonl` for routing decisions + ## Validation The shipped coverage includes: - unit tests for OpenAI-compatible profile detection - unit tests for Anthropic -> OpenAI request translation +- unit tests for request-time profile/model routing and scenario routing - unit tests for multi-line SSE parsing - integration tests for `/v1/messages` request/response translation +- integration tests for rate limits, empty upstream responses, timeout handling, + thinking/tool-call chunk streaming, and request-time routing - integration tests for daemon lifecycle and `/health` / `/v1/models` - e2e tests for `ccs proxy` lifecycle - e2e tests for `ccs ` auto-routing through a mock upstream - -## Current Scope - -The current implementation focuses on the core routing path: - -- local proxy lifecycle -- Anthropic/OpenAI request-response translation -- Claude-target settings profile auto-routing - -Scenario-based routing and token-count-driven model switching remain follow-up -work if they are needed beyond the base provider-routing flow. diff --git a/docs/project-roadmap.md b/docs/project-roadmap.md index 3b82689d..c6271e38 100644 --- a/docs/project-roadmap.md +++ b/docs/project-roadmap.md @@ -41,7 +41,7 @@ All major modularization work is complete. The codebase evolved from monolithic ### Recent Fixes -- **2026-04-14**: **#991** CCS now auto-routes Claude-target settings profiles that use OpenAI-compatible endpoints through a local Anthropic-compatible proxy instead of sending raw Anthropic `/v1/messages` traffic directly to chat-completions backends. The new `ccs proxy` command supports `start`, `status`, `activate`, and `stop`, the runtime exposes `/health` and `/v1/models`, the shared SSE parser now handles multi-line `data:` payloads, and the feature ships with dedicated unit, integration, and e2e coverage. +- **2026-04-14**: **#991** CCS now auto-routes Claude-target settings profiles that use OpenAI-compatible endpoints through a local Anthropic-compatible proxy instead of sending raw Anthropic `/v1/messages` traffic directly to chat-completions backends. The `ccs proxy` command now supports `start`, `status`, `activate`, and `stop` with explicit host binding, shell-aware activation helpers, and a fuller local runtime env contract. The proxy surface now exposes `GET /`, `/health`, `/v1/models`, and `/v1/messages`, logs routing decisions into CCS structured logs, supports Anthropic image blocks plus request-time `profile:model` overrides, and adds config-driven scenario routing (`background`, `think`, `longContext`, `webSearch`) on top of the compatible-profile path. Coverage now includes request routing, rate-limit/timeout/empty-upstream failures, chunked tool-call streaming, and disconnect cleanup alongside the existing unit, integration, and e2e suites. - **2026-04-10**: **#765** `/providers` now includes a first-class Hugging Face preset for API Profiles. CCS exposes Hugging Face Inference Providers through the existing OpenAI-compatible profile flow with the official router endpoint `https://router.huggingface.co/v1`, a short `hf` default profile name, and `hf` preset alias support for both the dashboard chooser and `ccs api create --preset hf`. - **2026-04-10**: **#944** Image Analysis auth readiness no longer collapses to native Read when merged runtime-status dependency overrides include a missing initializer value. CCS now preserves default dependency functions when override entries are `undefined`, still reads token-backed auth status directly in the local readiness path, and includes regression coverage for the missing-initializer case that previously surfaced as `deps.initializeAccounts is not a function`. - **2026-04-10**: **#945** CCS now normalizes Gemini CLI and Antigravity tier signals around an explicit `free / pro / ultra / unknown` model, preserves raw tier ids such as `g1-pro-tier`, enriches Gemini quota responses with provider entitlement evidence, classifies `MODEL_CAPACITY_EXHAUSTED` separately from auth/entitlement failures, fixes the Antigravity CLI quota table so live quota-derived tiers no longer collapse back to stale `unknown`, adds Gemini tier ids to CLI quota output, extends Gemini Flash Lite grouping to cover `gemini-3.1-flash-lite-preview`, and allows Gemini account surfaces to render the same tier badge semantics as Antigravity. diff --git a/src/commands/proxy-command.ts b/src/commands/proxy-command.ts index 9110f3f1..51c406f9 100644 --- a/src/commands/proxy-command.ts +++ b/src/commands/proxy-command.ts @@ -36,12 +36,15 @@ function showHelp(): number { console.log(''); console.log('Options:'); console.log(' --port Override the local proxy port (default: 3456)'); + console.log(' --host Bind the proxy server to a specific host (default: 127.0.0.1)'); console.log(' --shell activate only: auto|bash|zsh|fish|powershell'); + console.log(' --fish activate only: shorthand for --shell fish'); console.log(' --insecure Disable upstream TLS verification'); console.log(''); console.log('Examples:'); console.log(' ccs proxy start hf'); console.log(' eval "$(ccs proxy activate)"'); + console.log(' ccs proxy activate --fish'); console.log(' ccs proxy status'); console.log(' ccs proxy stop'); console.log(''); @@ -61,11 +64,14 @@ function resolveProfile(profileName: string) { async function handleStart(args: string[]): Promise { const profileName = args.find((arg) => !arg.startsWith('-')); if (!profileName) { - console.error(fail('Usage: ccs proxy start [--port ] [--insecure]')); + console.error( + fail('Usage: ccs proxy start [--port ] [--host ] [--insecure]') + ); return 1; } const portValue = parseOptionValue(args, '--port'); + const host = parseOptionValue(args, '--host'); const port = portValue ? Number.parseInt(portValue, 10) || 3456 : undefined; let profile; try { @@ -77,6 +83,7 @@ async function handleStart(args: string[]): Promise { const result = await startOpenAICompatProxy(profile, { ...(port ? { port } : {}), + ...(host ? { host } : {}), insecure: args.includes('--insecure'), }); @@ -101,6 +108,10 @@ async function handleStatus(): Promise { } console.log(ok(`Proxy running on port ${status.port}`)); + if (status.host) { + console.log(` Host: ${status.host}`); + console.log(` Local URL: http://${status.host}:${status.port}`); + } console.log(` Profile: ${status.profileName}`); console.log(` Base URL: ${status.baseUrl}`); if (status.model) { @@ -119,7 +130,7 @@ async function handleActivate(args: string[]): Promise { return 1; } - const shell = detectShell(parseOptionValue(args, '--shell')); + const shell = detectShell(args.includes('--fish') ? 'fish' : parseOptionValue(args, '--shell')); let profile; try { profile = resolveProfile(status.profileName); @@ -128,7 +139,13 @@ async function handleActivate(args: string[]): Promise { return 1; } - const env = buildOpenAICompatProxyEnv(profile, status.port, status.authToken); + const env = buildOpenAICompatProxyEnv( + profile, + status.port, + status.authToken, + undefined, + status.host || '127.0.0.1' + ); Object.entries(env).forEach(([key, value]) => { console.log(formatExportLine(shell, key, value)); }); diff --git a/src/config/unified-config-loader.ts b/src/config/unified-config-loader.ts index 5865ba5d..5cd1254e 100644 --- a/src/config/unified-config-loader.ts +++ b/src/config/unified-config-loader.ts @@ -384,6 +384,18 @@ function mergeWithDefaults(partial: Partial): UnifiedConfig { : defaults.cliproxy.routing?.strategy, }, }, + proxy: { + routing: { + default: partial.proxy?.routing?.default ?? defaults.proxy?.routing?.default, + background: partial.proxy?.routing?.background ?? defaults.proxy?.routing?.background, + think: partial.proxy?.routing?.think ?? defaults.proxy?.routing?.think, + longContext: partial.proxy?.routing?.longContext ?? defaults.proxy?.routing?.longContext, + webSearch: partial.proxy?.routing?.webSearch ?? defaults.proxy?.routing?.webSearch, + longContextThreshold: + partial.proxy?.routing?.longContextThreshold ?? + defaults.proxy?.routing?.longContextThreshold, + }, + }, logging: { enabled: partial.logging?.enabled ?? DEFAULT_LOGGING_CONFIG.enabled, level: partial.logging?.level ?? DEFAULT_LOGGING_CONFIG.level, @@ -674,6 +686,17 @@ function generateYamlWithComments(config: UnifiedConfig): string { ); lines.push(''); + if (config.proxy?.routing) { + lines.push('# ----------------------------------------------------------------------------'); + lines.push('# Proxy Routing: OpenAI-compatible local proxy model selection rules'); + lines.push('# Use profile:model selectors to force a target profile and upstream model.'); + lines.push('# ----------------------------------------------------------------------------'); + lines.push( + yaml.dump({ proxy: config.proxy }, { indent: 2, lineWidth: -1, quotingType: '"' }).trim() + ); + lines.push(''); + } + if (config.logging) { lines.push('# ----------------------------------------------------------------------------'); lines.push('# Logging: CCS-owned structured runtime logs'); diff --git a/src/config/unified-config-types.ts b/src/config/unified-config-types.ts index 3ab0f545..afb7006c 100644 --- a/src/config/unified-config-types.ts +++ b/src/config/unified-config-types.ts @@ -498,6 +498,19 @@ export interface ProxyLocalConfig { auto_start: boolean; } +export interface OpenAICompatProxyRoutingConfig { + default?: string; + background?: string; + think?: string; + longContext?: string; + webSearch?: string; + longContextThreshold?: number; +} + +export interface OpenAICompatProxyConfig { + routing?: OpenAICompatProxyRoutingConfig; +} + /** * CLIProxy server configuration section. * Controls whether CCS uses local or remote CLIProxyAPI instance. @@ -856,6 +869,8 @@ export interface UnifiedConfig { profiles: Record; /** CLIProxy configuration */ cliproxy: CLIProxyConfig; + /** OpenAI-compatible local proxy configuration */ + proxy?: OpenAICompatProxyConfig; /** CCS-owned structured logging configuration */ logging?: LoggingConfig; /** User preferences */ @@ -934,6 +949,12 @@ export const DEFAULT_CLIPROXY_SERVER_CONFIG: CliproxyServerConfig = { }, }; +export const DEFAULT_OPENAI_COMPAT_PROXY_CONFIG: OpenAICompatProxyConfig = { + routing: { + longContextThreshold: 60_000, + }, +}; + /** * Create an empty unified config with defaults. */ @@ -958,6 +979,11 @@ export function createEmptyUnifiedConfig(): UnifiedConfig { strategy: 'round-robin', }, }, + proxy: { + routing: { + ...DEFAULT_OPENAI_COMPAT_PROXY_CONFIG.routing, + }, + }, logging: { ...DEFAULT_LOGGING_CONFIG }, preferences: { theme: 'system', diff --git a/src/proxy/proxy-daemon-entry.ts b/src/proxy/proxy-daemon-entry.ts index 226433bb..0013ef7e 100644 --- a/src/proxy/proxy-daemon-entry.ts +++ b/src/proxy/proxy-daemon-entry.ts @@ -4,6 +4,7 @@ import { startOpenAICompatProxyServer } from './server/proxy-server'; interface RuntimeOptions { port: number; + host: string; profileName: string; settingsPath: string; authToken: string; @@ -12,6 +13,7 @@ interface RuntimeOptions { function parseArgs(argv: string[]): RuntimeOptions { let port = 3456; + let host = '127.0.0.1'; let profileName = ''; let settingsPath = ''; let authToken = ''; @@ -23,6 +25,10 @@ function parseArgs(argv: string[]): RuntimeOptions { port = Number.parseInt(argv[++i] || '', 10) || port; continue; } + if (arg === '--host' && argv[i + 1]) { + host = argv[++i] || host; + continue; + } if (arg === '--profile' && argv[i + 1]) { profileName = argv[++i] || ''; continue; @@ -40,7 +46,7 @@ function parseArgs(argv: string[]): RuntimeOptions { } } - return { port, profileName, settingsPath, authToken, insecure }; + return { port, host, profileName, settingsPath, authToken, insecure }; } function startRuntime(options: RuntimeOptions): void { @@ -62,6 +68,7 @@ function startRuntime(options: RuntimeOptions): void { const server = startOpenAICompatProxyServer({ profile, + host: options.host, port: options.port, authToken: options.authToken, insecure: options.insecure, diff --git a/src/proxy/proxy-daemon-state.ts b/src/proxy/proxy-daemon-state.ts index c1921948..525c5a7e 100644 --- a/src/proxy/proxy-daemon-state.ts +++ b/src/proxy/proxy-daemon-state.ts @@ -9,6 +9,7 @@ import { export interface OpenAICompatProxySession { profileName: string; settingsPath: string; + host: string; port: number; baseUrl: string; authToken: string; diff --git a/src/proxy/proxy-daemon.ts b/src/proxy/proxy-daemon.ts index 603c3097..8a28382e 100644 --- a/src/proxy/proxy-daemon.ts +++ b/src/proxy/proxy-daemon.ts @@ -225,10 +225,11 @@ export async function stopOpenAICompatProxy(): Promise<{ success: boolean; error export async function startOpenAICompatProxy( profile: OpenAICompatProfileConfig, - options: { port?: number; insecure?: boolean } = {} + options: { port?: number; host?: string; insecure?: boolean } = {} ): Promise { return withOpenAICompatProxyLock(async () => { const status = await getOpenAICompatProxyStatus(); + const host = options.host?.trim() || status.host || '127.0.0.1'; const port = typeof options.port === 'number' ? options.port @@ -245,7 +246,12 @@ export async function startOpenAICompatProxy( if (!Number.isInteger(port) || port < 1 || port > 65535) { return { success: false, port, error: `Invalid port: ${port}` }; } - if (status.running && status.profileName === profile.profileName && status.port === port) { + if ( + status.running && + status.profileName === profile.profileName && + status.port === port && + (status.host || '127.0.0.1') === host + ) { return { success: true, alreadyRunning: true, @@ -304,6 +310,8 @@ export async function startOpenAICompatProxy( daemonEntry, '--port', String(port), + '--host', + host, '--profile', profile.profileName, '--settings-path', @@ -323,6 +331,7 @@ export async function startOpenAICompatProxy( writeOpenAICompatProxySession({ profileName: profile.profileName, settingsPath: profile.settingsPath, + host, port, baseUrl: profile.baseUrl, authToken, diff --git a/src/proxy/proxy-env.ts b/src/proxy/proxy-env.ts index ca11813a..ed99a87c 100644 --- a/src/proxy/proxy-env.ts +++ b/src/proxy/proxy-env.ts @@ -4,11 +4,17 @@ export function buildOpenAICompatProxyEnv( profile: OpenAICompatProfileConfig, port: number, authToken: string, - claudeConfigDir?: string + claudeConfigDir?: string, + host = '127.0.0.1' ): Record { + const localBaseUrl = `http://${host}:${port}`; return { - ANTHROPIC_BASE_URL: `http://127.0.0.1:${port}`, + ANTHROPIC_BASE_URL: localBaseUrl, ANTHROPIC_AUTH_TOKEN: authToken, + DISABLE_TELEMETRY: '1', + DISABLE_COST_WARNINGS: '1', + API_TIMEOUT_MS: '600000', + NO_PROXY: host === '127.0.0.1' ? '127.0.0.1,localhost' : `${host},127.0.0.1,localhost`, ...(profile.model ? { ANTHROPIC_MODEL: profile.model } : {}), ...(profile.opusModel ? { ANTHROPIC_DEFAULT_OPUS_MODEL: profile.opusModel } : {}), ...(profile.sonnetModel ? { ANTHROPIC_DEFAULT_SONNET_MODEL: profile.sonnetModel } : {}), diff --git a/src/proxy/request-router.ts b/src/proxy/request-router.ts new file mode 100644 index 00000000..aa758280 --- /dev/null +++ b/src/proxy/request-router.ts @@ -0,0 +1,222 @@ +import { loadConfigSafe, loadSettings } from '../utils/config-manager'; +import { expandPath } from '../utils/helpers'; +import type { ProxyOpenAIRequest } from './transformers/request-transformer'; +import { + loadOpenAICompatProxyRoutingConfig, + type OpenAICompatProxyRoutingConfig, +} from './routing-config'; +import { resolveOpenAICompatProfileConfig, type OpenAICompatProfileConfig } from './profile-router'; + +export type ProxyRoutingScenario = 'default' | 'background' | 'think' | 'longContext' | 'webSearch'; + +export interface ProxyRequestRoute { + profile: OpenAICompatProfileConfig; + model?: string; + scenario?: ProxyRoutingScenario; + estimatedTokens: number; + source: + | 'explicit-profile' + | 'scenario' + | 'profile-model-match' + | 'profile-name' + | 'request-model' + | 'active-default'; +} + +function loadOpenAICompatProfiles( + activeProfile: OpenAICompatProfileConfig +): OpenAICompatProfileConfig[] { + const config = loadConfigSafe(); + const profiles = [activeProfile]; + + for (const [profileName, settingsPath] of Object.entries(config.profiles)) { + if (profileName === activeProfile.profileName) { + continue; + } + + try { + const expandedPath = expandPath(settingsPath); + const settings = loadSettings(expandedPath); + const profile = resolveOpenAICompatProfileConfig( + profileName, + expandedPath, + settings.env || {} + ); + if (profile) { + profiles.push(profile); + } + } catch { + // Ignore invalid profiles while routing a live request. + } + } + + return profiles; +} + +function resolveSelectorTarget( + selector: string, + activeProfile: OpenAICompatProfileConfig, + profiles: OpenAICompatProfileConfig[] +): { profile: OpenAICompatProfileConfig; model?: string; explicitProfile: boolean } | null { + const trimmed = selector.trim(); + if (!trimmed) { + return null; + } + + const colonIndex = trimmed.indexOf(':'); + if (colonIndex > 0) { + const profileName = trimmed.slice(0, colonIndex).trim(); + const profile = profiles.find((candidate) => candidate.profileName === profileName); + if (profile) { + const model = trimmed.slice(colonIndex + 1).trim() || profile.model; + return { profile, model, explicitProfile: true }; + } + } + + const namedProfile = profiles.find((candidate) => candidate.profileName === trimmed); + if (namedProfile) { + return { profile: namedProfile, model: namedProfile.model, explicitProfile: false }; + } + + return { + profile: activeProfile, + model: trimmed, + explicitProfile: false, + }; +} + +function profileSupportsModel(profile: OpenAICompatProfileConfig, model: string): boolean { + return [profile.model, profile.opusModel, profile.sonnetModel, profile.haikuModel].some( + (candidate) => typeof candidate === 'string' && candidate === model + ); +} + +function estimateTokens(request: ProxyOpenAIRequest): number { + let characters = 0; + for (const message of request.messages) { + if (typeof message.content === 'string') { + characters += message.content.length; + continue; + } + if (Array.isArray(message.content)) { + for (const part of message.content) { + characters += part.type === 'text' ? part.text.length : part.image_url.url.length; + } + } + if (Array.isArray(message.tool_calls)) { + for (const toolCall of message.tool_calls) { + characters += toolCall.function.name.length + toolCall.function.arguments.length; + } + } + } + + if (Array.isArray(request.tools)) { + characters += JSON.stringify(request.tools).length; + } + + return Math.max(1, Math.ceil(characters / 4)); +} + +function detectScenario( + request: ProxyOpenAIRequest, + requestedModel: string | undefined, + routing: OpenAICompatProxyRoutingConfig +): { scenario?: ProxyRoutingScenario; selector?: string; estimatedTokens: number } { + const estimatedTokens = estimateTokens(request); + const hasWebSearchTool = + request.tools?.some((tool) => tool.function.name === 'web_search') === true; + const thinkingEnabled = + request.reasoning?.enabled === true || typeof request.reasoning_effort === 'string'; + const modelId = requestedModel || ''; + const longContextThreshold = routing.longContextThreshold ?? 60_000; + + if (hasWebSearchTool && routing.webSearch) { + return { scenario: 'webSearch', selector: routing.webSearch, estimatedTokens }; + } + if (thinkingEnabled && routing.think) { + return { scenario: 'think', selector: routing.think, estimatedTokens }; + } + if (estimatedTokens > longContextThreshold && routing.longContext) { + return { scenario: 'longContext', selector: routing.longContext, estimatedTokens }; + } + if (modelId.toLowerCase().includes('haiku') && routing.background) { + return { scenario: 'background', selector: routing.background, estimatedTokens }; + } + if (!requestedModel && routing.default) { + return { scenario: 'default', selector: routing.default, estimatedTokens }; + } + + return { estimatedTokens }; +} + +export function resolveProxyRequestRoute( + activeProfile: OpenAICompatProfileConfig, + request: ProxyOpenAIRequest +): ProxyRequestRoute { + const profiles = loadOpenAICompatProfiles(activeProfile); + const requestedModel = request.model?.trim() || undefined; + const explicitTarget = requestedModel + ? resolveSelectorTarget(requestedModel, activeProfile, profiles) + : null; + const routing = loadOpenAICompatProxyRoutingConfig(); + + if (explicitTarget?.explicitProfile) { + return { + profile: explicitTarget.profile, + model: explicitTarget.model, + estimatedTokens: estimateTokens(request), + source: 'explicit-profile', + }; + } + + const scenario = detectScenario(request, requestedModel, routing); + if (scenario.selector) { + const scenarioTarget = resolveSelectorTarget(scenario.selector, activeProfile, profiles); + if (scenarioTarget) { + return { + profile: scenarioTarget.profile, + model: scenarioTarget.model, + scenario: scenario.scenario, + estimatedTokens: scenario.estimatedTokens, + source: 'scenario', + }; + } + } + + if (explicitTarget && explicitTarget.profile.profileName !== activeProfile.profileName) { + return { + profile: explicitTarget.profile, + model: explicitTarget.model, + estimatedTokens: scenario.estimatedTokens, + source: 'profile-name', + }; + } + + if (requestedModel) { + const matchedProfile = profiles.find((profile) => + profileSupportsModel(profile, requestedModel) + ); + if (matchedProfile) { + return { + profile: matchedProfile, + model: requestedModel, + estimatedTokens: scenario.estimatedTokens, + source: 'profile-model-match', + }; + } + + return { + profile: activeProfile, + model: requestedModel, + estimatedTokens: scenario.estimatedTokens, + source: 'request-model', + }; + } + + return { + profile: activeProfile, + model: activeProfile.model, + estimatedTokens: scenario.estimatedTokens, + source: 'active-default', + }; +} diff --git a/src/proxy/routing-config.ts b/src/proxy/routing-config.ts new file mode 100644 index 00000000..f069edd6 --- /dev/null +++ b/src/proxy/routing-config.ts @@ -0,0 +1,74 @@ +import * as fs from 'fs'; +import * as yaml from 'js-yaml'; +import { getActiveConfigPath, getConfigPath } from '../utils/config-manager'; + +export interface OpenAICompatProxyRoutingConfig { + default?: string; + background?: string; + think?: string; + longContext?: string; + webSearch?: string; + longContextThreshold?: number; +} + +function readRawConfigObject(configPath: string): Record | null { + if (!fs.existsSync(configPath)) { + return null; + } + + const raw = fs.readFileSync(configPath, 'utf8'); + const parsed = + configPath.endsWith('.yaml') || configPath.endsWith('.yml') ? yaml.load(raw) : JSON.parse(raw); + return typeof parsed === 'object' && parsed !== null ? (parsed as Record) : null; +} + +function normalizeRoutingConfig(value: unknown): OpenAICompatProxyRoutingConfig { + if (typeof value !== 'object' || value === null) { + return {}; + } + + const config = value as Record; + return { + default: + typeof config.default === 'string' && config.default.trim() + ? config.default.trim() + : undefined, + background: + typeof config.background === 'string' && config.background.trim() + ? config.background.trim() + : undefined, + think: + typeof config.think === 'string' && config.think.trim() ? config.think.trim() : undefined, + longContext: + typeof config.longContext === 'string' && config.longContext.trim() + ? config.longContext.trim() + : undefined, + webSearch: + typeof config.webSearch === 'string' && config.webSearch.trim() + ? config.webSearch.trim() + : undefined, + longContextThreshold: + typeof config.longContextThreshold === 'number' && + Number.isFinite(config.longContextThreshold) + ? config.longContextThreshold + : undefined, + }; +} + +export function loadOpenAICompatProxyRoutingConfig(): OpenAICompatProxyRoutingConfig { + const activePath = getActiveConfigPath(); + const rawConfig = readRawConfigObject(activePath); + if (rawConfig?.proxy && typeof rawConfig.proxy === 'object') { + return normalizeRoutingConfig((rawConfig.proxy as Record).routing); + } + + const legacyPath = getConfigPath(); + if (legacyPath !== activePath) { + const legacyConfig = readRawConfigObject(legacyPath); + if (legacyConfig?.proxy && typeof legacyConfig.proxy === 'object') { + return normalizeRoutingConfig((legacyConfig.proxy as Record).routing); + } + } + + return {}; +} diff --git a/src/proxy/server/messages-route.ts b/src/proxy/server/messages-route.ts index 49a66585..ae0d33d9 100644 --- a/src/proxy/server/messages-route.ts +++ b/src/proxy/server/messages-route.ts @@ -1,12 +1,15 @@ import * as http from 'http'; import type { Dispatcher } from 'undici'; import type { OpenAICompatProfileConfig } from '../profile-router'; +import { resolveProxyRequestRoute } from '../request-router'; import { ProxyRequestTransformer } from '../transformers/request-transformer'; import { ProxySseStreamTransformer } from '../transformers/sse-stream-transformer'; import { resolveOpenAIChatCompletionsUrl } from '../upstream-url'; +import { createLogger } from '../../services/logging'; import { pipeWebResponseToNode, readJsonBody, writeJson } from './http-helpers'; -const REQUEST_TIMEOUT_MS = 30_000; +const REQUEST_TIMEOUT_MS = 600_000; +const logger = createLogger('proxy:openai-compat:messages'); class ProxyInputError extends Error { constructor(message: string) { @@ -23,7 +26,10 @@ function buildUpstreamHeaders(profile: OpenAICompatProfileConfig): Record } { let transformed; try { const transformer = new ProxyRequestTransformer(); @@ -32,12 +38,13 @@ function buildUpstreamBody(profile: OpenAICompatProfileConfig, rawBody: unknown) const message = error instanceof Error ? error.message : 'Invalid Anthropic request'; throw new ProxyInputError(message); } + const route = resolveProxyRequestRoute(profile, transformed); const body = { ...transformed, - model: transformed.model || profile.model, + model: route.model || route.profile.model, stream: transformed.stream === true, }; - return JSON.stringify(body); + return { body: JSON.stringify(body), route }; } export function extractIncomingProxyToken(headers: http.IncomingHttpHeaders): string | null { @@ -88,6 +95,20 @@ function buildFetchInit( return init; } +function getRequestTimeoutMs(): number { + const rawValue = process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS; + if (!rawValue) { + return REQUEST_TIMEOUT_MS; + } + + const parsed = Number.parseInt(rawValue, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : REQUEST_TIMEOUT_MS; +} + +function formatTimeoutDuration(timeoutMs: number): string { + return timeoutMs % 1000 === 0 ? `${timeoutMs / 1000} seconds` : `${timeoutMs}ms`; +} + export async function handleProxyMessagesRequest( req: http.IncomingMessage, res: http.ServerResponse, @@ -98,6 +119,9 @@ export async function handleProxyMessagesRequest( const transformer = new ProxySseStreamTransformer(); if (!validateIncomingProxyAuth(req.headers, expectedAuthToken)) { + logger.warn('auth.invalid', 'Rejected proxy message request with invalid auth token', { + remoteAddress: req.socket.remoteAddress || null, + }); await pipeWebResponseToNode( transformer.error(401, 'authentication_error', 'Missing or invalid local proxy token'), res @@ -105,34 +129,77 @@ export async function handleProxyMessagesRequest( return; } + let timeoutMs = REQUEST_TIMEOUT_MS; try { const rawBody = await readJsonBody(req); - const upstreamBody = buildUpstreamBody(profile, rawBody); + const upstream = buildUpstreamRequest(profile, rawBody); + logger.info('request.forward', 'Forwarding Anthropic request to OpenAI-compatible upstream', { + profileName: upstream.route.profile.profileName, + provider: upstream.route.profile.provider, + baseUrl: upstream.route.profile.baseUrl, + model: upstream.route.model || upstream.route.profile.model || null, + routeSource: upstream.route.source, + scenario: upstream.route.scenario || null, + estimatedTokens: upstream.route.estimatedTokens, + }); const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS); - const abortOnDisconnect = () => { + timeoutMs = getRequestTimeoutMs(); + const timeout = setTimeout(() => controller.abort(), timeoutMs); + const abortOnDisconnect = (source: string) => { if (!controller.signal.aborted && !res.writableEnded) { + logger.info( + 'request.disconnect', + 'Aborting upstream request after local client disconnect', + { + profileName: profile.profileName, + source, + } + ); controller.abort(); } }; - req.on('aborted', abortOnDisconnect); - req.on('close', abortOnDisconnect); - res.on('close', abortOnDisconnect); + req.on('aborted', () => abortOnDisconnect('req.aborted')); + req.on('close', () => abortOnDisconnect('req.close')); + req.socket?.on('close', () => abortOnDisconnect('req.socket.close')); + res.on('close', () => abortOnDisconnect('res.close')); + res.socket?.on('close', () => abortOnDisconnect('res.socket.close')); + const disconnectPoll = setInterval(() => { + if ( + req.destroyed || + res.destroyed || + req.socket?.destroyed === true || + res.socket?.destroyed === true + ) { + abortOnDisconnect('poll.destroyed'); + } + }, 50); try { const upstreamResponse = await fetch( - resolveOpenAIChatCompletionsUrl(profile.baseUrl), - buildFetchInit(profile, upstreamBody, controller.signal, insecureDispatcher) + resolveOpenAIChatCompletionsUrl(upstream.route.profile.baseUrl), + buildFetchInit(upstream.route.profile, upstream.body, controller.signal, insecureDispatcher) ); clearTimeout(timeout); + clearInterval(disconnectPoll); + logger.info('response.received', 'Received upstream response', { + profileName: profile.profileName, + routedProfileName: upstream.route.profile.profileName, + status: upstreamResponse.status, + }); const response = await transformer.transform(upstreamResponse); await pipeWebResponseToNode(response, res); } finally { clearTimeout(timeout); + clearInterval(disconnectPoll); } } catch (error) { const message = error instanceof Error ? error.message : 'Unknown proxy error'; + logger.error('request.failed', 'Proxy message request failed', { + profileName: profile.profileName, + error: message, + abort: error instanceof Error && error.name === 'AbortError', + }); const status = error instanceof Error && error.name === 'AbortError' ? 502 @@ -149,7 +216,7 @@ export async function handleProxyMessagesRequest( status, type, error instanceof Error && error.name === 'AbortError' - ? 'The upstream provider did not respond within 30 seconds' + ? `The upstream provider did not respond within ${formatTimeoutDuration(timeoutMs)}` : message ), res diff --git a/src/proxy/server/proxy-server.ts b/src/proxy/server/proxy-server.ts index b808d9ee..4b68f2a1 100644 --- a/src/proxy/server/proxy-server.ts +++ b/src/proxy/server/proxy-server.ts @@ -2,6 +2,7 @@ import * as http from 'http'; import { Agent } from 'undici'; import type { OpenAICompatProfileConfig } from '../profile-router'; import { OPENAI_COMPAT_PROXY_SERVICE_NAME } from '../proxy-daemon-paths'; +import { createLogger } from '../../services/logging'; import { handleProxyMessagesRequest, handleProxyModelsRequest, @@ -11,12 +12,19 @@ import { writeJson } from './http-helpers'; export interface OpenAICompatProxyServerOptions { profile: OpenAICompatProfileConfig; + host?: string; port: number; authToken: string; insecure?: boolean; } export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOptions): http.Server { + const host = options.host?.trim() || '127.0.0.1'; + const logger = createLogger('proxy:openai-compat', { + profileName: options.profile.profileName, + host, + port: options.port, + }); const insecureDispatcher = options.insecure ? new Agent({ connect: { rejectUnauthorized: false } }) : undefined; @@ -31,12 +39,31 @@ export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOpt writeJson(res, 200, { ok: true, service: OPENAI_COMPAT_PROXY_SERVICE_NAME, + host, profile: options.profile.profileName, port: options.port, }); return; } + if (method === 'GET' && pathname === '/') { + writeJson(res, 200, { + ok: true, + service: OPENAI_COMPAT_PROXY_SERVICE_NAME, + bind: { + host, + port: options.port, + }, + profile: { + name: options.profile.profileName, + provider: options.profile.provider, + model: options.profile.model || null, + }, + endpoints: ['/health', '/v1/messages', '/v1/models'], + }); + return; + } + if (method === 'GET' && pathname === '/v1/models') { if (!validateIncomingProxyAuth(req.headers, options.authToken)) { writeJson(res, 401, { @@ -63,13 +90,21 @@ export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOpt return; } + logger.warn('http.not_found', 'Rejected unknown proxy route', { + method, + pathname, + }); writeJson(res, 404, { error: 'Not found' }); }); + logger.info('server.start', 'OpenAI-compatible proxy server listening', { + baseUrl: `http://${host}:${options.port}`, + }); server.on('close', () => { + logger.info('server.stop', 'OpenAI-compatible proxy server stopped'); void insecureDispatcher?.close(); }); - server.listen(options.port, '127.0.0.1'); + server.listen(options.port, host); return server; } diff --git a/src/proxy/transformers/request-transformer.ts b/src/proxy/transformers/request-transformer.ts index 1804d426..6b294dac 100644 --- a/src/proxy/transformers/request-transformer.ts +++ b/src/proxy/transformers/request-transformer.ts @@ -1,20 +1,97 @@ -import { translateAnthropicRequest } from '../../cursor/cursor-anthropic-translator'; +interface AnthropicThinking { + type?: 'enabled' | 'disabled' | string; + budget_tokens?: number; +} + +interface AnthropicTextBlock { + type: 'text'; + text?: string; +} + +interface AnthropicImageBlock { + type: 'image'; + source?: { + type?: string; + media_type?: string; + data?: string; + }; +} + +interface AnthropicToolUseBlock { + type: 'tool_use'; + id?: string; + name?: string; + input?: Record; +} + +interface AnthropicToolResultBlock { + type: 'tool_result'; + tool_use_id?: string; + content?: unknown; +} + +type AnthropicContentBlock = + | AnthropicTextBlock + | AnthropicImageBlock + | AnthropicToolUseBlock + | AnthropicToolResultBlock + | { type: string; [key: string]: unknown }; + +interface AnthropicMessage { + role?: 'user' | 'assistant' | string; + content?: string | AnthropicContentBlock[]; +} interface AnthropicProxyRequestShape { + model?: unknown; + system?: unknown; + messages?: unknown; max_tokens?: unknown; temperature?: unknown; top_p?: unknown; stop_sequences?: unknown; metadata?: unknown; tools?: unknown; + stream?: unknown; + thinking?: AnthropicThinking; } -export interface ProxyOpenAIRequest extends ReturnType { - max_tokens?: number; - temperature?: number; - top_p?: number; - stop?: string[]; - metadata?: Record; +interface OpenAITextPart { + type: 'text'; + text: string; +} + +interface OpenAIImagePart { + type: 'image_url'; + image_url: { + url: string; + }; +} + +type OpenAIContentPart = OpenAITextPart | OpenAIImagePart; + +interface OpenAIMessage { + role: 'system' | 'user' | 'assistant' | 'tool'; + content: string | OpenAIContentPart[] | null; + tool_call_id?: string; + tool_calls?: Array<{ + id: string; + type: 'function'; + function: { + name: string; + arguments: string; + }; + }>; +} + +export interface ProxyOpenAIRequest { + model?: string; + stream: boolean; + reasoning_effort?: string; + reasoning?: { + enabled: boolean; + effort: string; + }; tools?: Array<{ type: 'function'; function: { @@ -23,6 +100,23 @@ export interface ProxyOpenAIRequest extends ReturnType; }; }>; + messages: OpenAIMessage[]; + max_tokens?: number; + temperature?: number; + top_p?: number; + stop?: string[]; + metadata?: Record; +} + +const TOOL_RESULT_SERIALIZATION_FALLBACK = '[unserializable content]'; +const TOOL_USE_ARGUMENTS_FALLBACK = '{}'; + +function assertObject(value: unknown, label: string): Record { + if (typeof value !== 'object' || value === null) { + throw new Error(`${label} must be an object`); + } + + return value as Record; } function asNumber(value: unknown): number | undefined { @@ -46,6 +140,90 @@ function asMetadata(value: unknown): Record | undefined { : undefined; } +function safeJsonStringify(value: unknown, fallback: string): string { + try { + const serialized = JSON.stringify(value); + return typeof serialized === 'string' ? serialized : fallback; + } catch { + return fallback; + } +} + +function flattenTextContent(content: unknown, label: string): string { + if (typeof content === 'string') { + return content; + } + if (!Array.isArray(content)) { + throw new Error(`${label} must be a string or content block array`); + } + + return content + .map((block, index) => { + const parsed = assertObject(block, `${label}[${index}]`); + if (parsed.type !== 'text') { + throw new Error(`${label}[${index}].type "${String(parsed.type)}" is not supported`); + } + return typeof parsed.text === 'string' ? parsed.text : ''; + }) + .join('\n'); +} + +function toToolResultContent(content: unknown, label: string): string { + if (content === undefined) { + return ''; + } + if (typeof content === 'string') { + return content; + } + if (Array.isArray(content)) { + return flattenTextContent(content, label); + } + return safeJsonStringify(content, TOOL_RESULT_SERIALIZATION_FALLBACK); +} + +function createFallbackToolId(messageIndex: number, blockIndex: number): string { + return `toolu_proxy_fallback_${messageIndex}_${blockIndex}`; +} + +function toImagePart(block: AnthropicImageBlock, label: string): OpenAIImagePart { + const source = block.source; + if (!source || source.type !== 'base64' || !source.media_type || !source.data) { + throw new Error(`${label}.source must be a base64 image payload`); + } + + return { + type: 'image_url', + image_url: { + url: `data:${source.media_type};base64,${source.data}`, + }, + }; +} + +function isImageBlock(block: AnthropicContentBlock): block is AnthropicImageBlock { + return block.type === 'image'; +} + +function isToolUseBlock(block: AnthropicContentBlock): block is AnthropicToolUseBlock { + return block.type === 'tool_use'; +} + +function isToolResultBlock(block: AnthropicContentBlock): block is AnthropicToolResultBlock { + return block.type === 'tool_result'; +} + +function flushUserContent(messages: OpenAIMessage[], parts: OpenAIContentPart[]): void { + if (parts.length === 0) { + return; + } + + const onlyText = parts.every((part) => part.type === 'text'); + messages.push({ + role: 'user', + content: onlyText ? parts.map((part) => (part as OpenAITextPart).text).join('\n') : [...parts], + }); + parts.length = 0; +} + function transformTools(value: unknown): ProxyOpenAIRequest['tools'] { if (!Array.isArray(value)) { return undefined; @@ -71,19 +249,179 @@ function transformTools(value: unknown): ProxyOpenAIRequest['tools'] { return tools.length > 0 ? tools : undefined; } +function mapThinkingToReasoning( + thinking: AnthropicThinking | undefined +): Pick { + if (!thinking || thinking.type === 'disabled') { + return {}; + } + + if (thinking.type !== 'enabled') { + throw new Error('thinking.type must be "enabled" or "disabled"'); + } + + const effort = + typeof thinking.budget_tokens === 'number' && thinking.budget_tokens >= 8192 + ? 'high' + : 'medium'; + + return { + reasoning_effort: effort, + reasoning: { + enabled: true, + effort, + }, + }; +} + +function transformMessages(messagesValue: unknown): OpenAIMessage[] { + if (!Array.isArray(messagesValue)) { + throw new Error('messages must be an array'); + } + + const translatedMessages: OpenAIMessage[] = []; + + messagesValue.forEach((message, messageIndex) => { + const parsedMessage = assertObject(message, `messages[${messageIndex}]`) as AnthropicMessage; + const role = parsedMessage.role; + if (role !== 'user' && role !== 'assistant') { + throw new Error(`messages[${messageIndex}].role must be "user" or "assistant"`); + } + + const content = parsedMessage.content; + if (typeof content === 'string') { + translatedMessages.push({ role, content }); + return; + } + + if (!Array.isArray(content)) { + throw new Error(`messages[${messageIndex}].content must be a string or array`); + } + + const userParts: OpenAIContentPart[] = []; + const assistantTextParts: string[] = []; + const toolCalls: NonNullable = []; + let sawToolResult = false; + + content.forEach((block, blockIndex) => { + const parsed = assertObject( + block, + `messages[${messageIndex}].content[${blockIndex}]` + ) as AnthropicContentBlock; + + if (parsed.type === 'text') { + const text = typeof parsed.text === 'string' ? parsed.text : ''; + if (role === 'user') { + userParts.push({ type: 'text', text }); + } else { + assistantTextParts.push(text); + } + return; + } + + if (isImageBlock(parsed)) { + if (role !== 'user') { + throw new Error( + `messages[${messageIndex}].content[${blockIndex}] image requires user role` + ); + } + userParts.push(toImagePart(parsed, `messages[${messageIndex}].content[${blockIndex}]`)); + return; + } + + if (isToolUseBlock(parsed)) { + if (role !== 'assistant') { + throw new Error( + `messages[${messageIndex}].content[${blockIndex}] tool_use requires assistant role` + ); + } + toolCalls.push({ + id: + typeof parsed.id === 'string' && parsed.id.length > 0 + ? parsed.id + : createFallbackToolId(messageIndex, blockIndex), + type: 'function', + function: { + name: typeof parsed.name === 'string' ? parsed.name : 'tool', + arguments: safeJsonStringify(parsed.input ?? {}, TOOL_USE_ARGUMENTS_FALLBACK), + }, + }); + return; + } + + if (isToolResultBlock(parsed)) { + if (role !== 'user') { + throw new Error( + `messages[${messageIndex}].content[${blockIndex}] tool_result requires user role` + ); + } + if (typeof parsed.tool_use_id !== 'string' || parsed.tool_use_id.trim().length === 0) { + throw new Error( + `messages[${messageIndex}].content[${blockIndex}].tool_use_id must be a non-empty string` + ); + } + sawToolResult = true; + flushUserContent(translatedMessages, userParts); + translatedMessages.push({ + role: 'tool', + tool_call_id: parsed.tool_use_id, + content: toToolResultContent( + parsed.content, + `messages[${messageIndex}].content[${blockIndex}].content` + ), + }); + return; + } + + throw new Error( + `messages[${messageIndex}].content[${blockIndex}].type "${String(parsed.type)}" is not supported` + ); + }); + + if (role === 'assistant') { + translatedMessages.push({ + role: 'assistant', + content: assistantTextParts.join('\n'), + tool_calls: toolCalls.length > 0 ? toolCalls : undefined, + }); + return; + } + + if (userParts.length > 0 || !sawToolResult) { + flushUserContent(translatedMessages, userParts); + } + }); + + return translatedMessages; +} + export class ProxyRequestTransformer { transform(raw: unknown): ProxyOpenAIRequest { - const translated = translateAnthropicRequest(raw); - const source = (raw || {}) as AnthropicProxyRequestShape; + const source = assertObject(raw || {}, 'request') as AnthropicProxyRequestShape; + const messages = transformMessages(source.messages); + const system = source.system; + const allMessages = + system !== undefined + ? [ + { role: 'system', content: flattenTextContent(system, 'system') } as OpenAIMessage, + ...messages, + ] + : messages; return { - ...translated, + model: + typeof source.model === 'string' && source.model.trim().length > 0 + ? source.model.trim() + : undefined, + stream: source.stream === true, + messages: allMessages, max_tokens: asNumber(source.max_tokens), temperature: asNumber(source.temperature), top_p: asNumber(source.top_p), stop: asStringArray(source.stop_sequences), metadata: asMetadata(source.metadata), tools: transformTools(source.tools), + ...mapThinkingToReasoning(source.thinking), }; } } diff --git a/src/types/config.ts b/src/types/config.ts index b84658e1..47de4f0d 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -40,6 +40,19 @@ export interface CLIProxyVariantsConfig { [profileName: string]: CLIProxyVariantConfig; } +export interface OpenAICompatProxyRoutingConfig { + default?: string; + background?: string; + think?: string; + longContext?: string; + webSearch?: string; + longContextThreshold?: number; +} + +export interface OpenAICompatProxyConfig { + routing?: OpenAICompatProxyRoutingConfig; +} + /** * Main CCS configuration * Located at: ~/.ccs/config.json @@ -51,6 +64,8 @@ export interface Config { profile_targets?: Record; /** User-defined CLIProxy profile variants (optional) */ cliproxy?: CLIProxyVariantsConfig; + /** OpenAI-compatible local proxy configuration (optional) */ + proxy?: OpenAICompatProxyConfig; /** Legacy continuity inheritance mapping (profile -> source account) */ continuity_inherit_from_account?: Record; } diff --git a/src/types/index.ts b/src/types/index.ts index f88f447d..934f18ae 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -14,6 +14,8 @@ export type { ProfilesRegistry, CLIProxyVariantConfig, CLIProxyVariantsConfig, + OpenAICompatProxyConfig, + OpenAICompatProxyRoutingConfig, } from './config'; export { isConfig, isSettings } from './config'; diff --git a/src/web-server/routes/config-routes.ts b/src/web-server/routes/config-routes.ts index 60b0cc44..3cfa87e6 100644 --- a/src/web-server/routes/config-routes.ts +++ b/src/web-server/routes/config-routes.ts @@ -673,6 +673,10 @@ router.put('/', (req: Request, res: Response): void => { currentConfig.cliproxy = mergeCliproxyConfig(currentConfig, config); } + if (config.proxy !== undefined) { + currentConfig.proxy = config.proxy; + } + if (config.preferences !== undefined) { currentConfig.preferences = config.preferences; } diff --git a/tests/e2e/proxy-command.e2e.test.ts b/tests/e2e/proxy-command.e2e.test.ts index 550ee8a2..021a0919 100644 --- a/tests/e2e/proxy-command.e2e.test.ts +++ b/tests/e2e/proxy-command.e2e.test.ts @@ -68,20 +68,42 @@ describe('proxy command e2e', () => { 'utf8' ); - const started = runCli(['proxy', 'start', 'hf', '--port', String(port)]); + const started = runCli(['proxy', 'start', 'hf', '--port', String(port), '--host', '127.0.0.1']); expect(started.status).toBe(0); const status = runCli(['proxy', 'status']); expect(status.stdout).toContain(`Proxy running on port ${port}`); + expect(status.stdout).toContain('Host: 127.0.0.1'); expect(status.stdout).toContain('Profile: hf'); const activate = runCli(['proxy', 'activate', '--shell', 'bash']); expect(activate.stdout).toContain(`export ANTHROPIC_BASE_URL='http://127.0.0.1:${port}'`); expect(activate.stdout).toMatch(/export ANTHROPIC_AUTH_TOKEN='[a-f0-9]{48}'/); + expect(activate.stdout).toContain("export DISABLE_TELEMETRY='1'"); + expect(activate.stdout).toContain("export DISABLE_COST_WARNINGS='1'"); + expect(activate.stdout).toContain("export API_TIMEOUT_MS='600000'"); + expect(activate.stdout).toContain("export NO_PROXY='127.0.0.1,localhost'"); + + const activateFish = runCli(['proxy', 'activate', '--fish']); + expect(activateFish.stdout).toContain(`set -gx ANTHROPIC_BASE_URL 'http://127.0.0.1:${port}'`); const health = await fetch(`http://127.0.0.1:${port}/health`); expect(health.status).toBe(200); + const info = await fetch(`http://127.0.0.1:${port}/`); + expect(info.status).toBe(200); + await expect(info.json()).resolves.toMatchObject({ + ok: true, + service: 'ccs-openai-compat-proxy', + bind: { + host: '127.0.0.1', + port, + }, + profile: { + name: 'hf', + }, + }); + const stopped = runCli(['proxy', 'stop']); expect(stopped.status).toBe(0); }, 35000); diff --git a/tests/integration/proxy/messages-edge-cases.test.ts b/tests/integration/proxy/messages-edge-cases.test.ts new file mode 100644 index 00000000..9b9898a8 --- /dev/null +++ b/tests/integration/proxy/messages-edge-cases.test.ts @@ -0,0 +1,259 @@ +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import getPort from 'get-port'; +import * as fs from 'fs'; +import * as http from 'http'; +import * as os from 'os'; +import * as path from 'path'; +import { startOpenAICompatProxyServer } from '../../../src/proxy/server/proxy-server'; +import type { OpenAICompatProfileConfig } from '../../../src/proxy/profile-router'; + +let proxyServer: http.Server; +let upstreamServer: http.Server; +let upstreamSockets = new Set(); +let upstreamPort: number; +let proxyPort: number; +let tempDir: string; +let originalTimeoutEnv: string | undefined; +let originalCcsHome: string | undefined; + +async function startUpstream( + handler: (req: http.IncomingMessage, res: http.ServerResponse) => Promise | void +): Promise { + upstreamServer = http.createServer((req, res) => { + void Promise.resolve(handler(req, res)); + }); + upstreamServer.on('connection', (socket) => { + upstreamSockets.add(socket); + socket.on('close', () => { + upstreamSockets.delete(socket); + }); + }); + await new Promise((resolve) => + upstreamServer.listen(upstreamPort, '127.0.0.1', () => resolve()) + ); +} + +async function requestProxy(payload: unknown, signal?: AbortSignal): Promise { + return fetch(`http://127.0.0.1:${proxyPort}/v1/messages`, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'x-api-key': 'test-proxy-token', + }, + body: JSON.stringify(payload), + signal, + }); +} + +beforeEach(async () => { + upstreamPort = await getPort(); + proxyPort = await getPort(); + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-edge-')); + originalTimeoutEnv = process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS; + originalCcsHome = process.env.CCS_HOME; + process.env.CCS_HOME = tempDir; +}); + +afterEach(async () => { + if (originalTimeoutEnv !== undefined) { + process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS = originalTimeoutEnv; + } else { + delete process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS; + } + if (originalCcsHome !== undefined) { + process.env.CCS_HOME = originalCcsHome; + } else { + delete process.env.CCS_HOME; + } + + if (proxyServer) { + await new Promise((resolve) => proxyServer.close(() => resolve())); + } + if (upstreamServer) { + for (const socket of upstreamSockets) { + socket.destroy(); + } + upstreamSockets = new Set(); + await new Promise((resolve) => upstreamServer.close(() => resolve())); + } + fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +describe('openai proxy message edge cases', () => { + async function startProxyWithHandler( + handler: (req: http.IncomingMessage, res: http.ServerResponse) => Promise | void + ) { + await startUpstream(handler); + const profile: OpenAICompatProfileConfig = { + profileName: 'hf', + settingsPath: '/tmp/hf.settings.json', + baseUrl: `http://127.0.0.1:${upstreamPort}`, + apiKey: 'hf_token', + provider: 'generic-chat-completion-api', + model: 'hf-model', + }; + proxyServer = startOpenAICompatProxyServer({ + profile, + port: proxyPort, + authToken: 'test-proxy-token', + }); + } + + it('preserves rate-limit errors from the upstream provider', async () => { + await startProxyWithHandler((_req, res) => { + res.writeHead(429, { + 'Content-Type': 'application/json', + 'Retry-After': '9', + }); + res.end(JSON.stringify({ error: { message: 'rate limited' } })); + }); + + const response = await requestProxy({ + model: 'hf-model', + messages: [{ role: 'user', content: 'hello' }], + }); + + expect(response.status).toBe(429); + expect(response.headers.get('retry-after')).toBe('9'); + await expect(response.json()).resolves.toMatchObject({ + type: 'error', + error: { + type: 'rate_limit_error', + message: 'rate limited', + }, + }); + }); + + it('returns api_error when the upstream JSON response has no usable choices', async () => { + await startProxyWithHandler((_req, res) => { + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify({ id: 'chatcmpl_empty', choices: [] })); + }); + + const response = await requestProxy({ + model: 'hf-model', + messages: [{ role: 'user', content: 'hello' }], + }); + + expect(response.status).toBe(502); + await expect(response.json()).resolves.toMatchObject({ + type: 'error', + error: { + type: 'api_error', + message: 'Failed to translate Cursor JSON response', + }, + }); + }); + + it('streams thinking deltas and chunked tool-call arguments back as Anthropic SSE', async () => { + await startProxyWithHandler((_req, res) => { + res.writeHead(200, { 'Content-Type': 'text/event-stream' }); + res.write( + 'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"role":"assistant"}}]}\n\n' + ); + res.write( + 'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"reasoning_content":"Need to search first."}}]}\n\n' + ); + res.write( + 'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"search","arguments":"{\\"q\\":\\""}}]}}]}\n\n' + ); + res.write( + 'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"docs\\"}"}}]}}]}\n\n' + ); + res.write( + 'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":8,"completion_tokens":6}}\n\n' + ); + res.end('data: [DONE]\n\n'); + }); + + const response = await requestProxy({ + model: 'hf-model', + stream: true, + messages: [{ role: 'user', content: 'search docs' }], + }); + + const body = await response.text(); + expect(response.status).toBe(200); + expect(body).toContain('"type":"thinking_delta"'); + expect(body).toContain('"type":"tool_use"'); + expect(body).toContain('"partial_json":"{\\"q\\":\\""'); + expect(body).toContain('"partial_json":"docs\\"}"'); + expect(body).toContain('event: message_stop'); + }); + + it('returns a timeout error when the upstream does not respond in time', async () => { + process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS = '50'; + await startProxyWithHandler(async (req, _res) => { + await new Promise((resolve) => req.on('close', () => resolve())); + }); + + const response = await requestProxy({ + model: 'hf-model', + messages: [{ role: 'user', content: 'hello' }], + }); + + expect(response.status).toBe(502); + await expect(response.json()).resolves.toMatchObject({ + type: 'error', + error: { + type: 'api_error', + message: 'The upstream provider did not respond within 50ms', + }, + }); + }); + + it('aborts the upstream request when the client disconnects mid-flight', async () => { + await startProxyWithHandler(() => {}); + + await new Promise((resolve) => { + const request = http.request( + { + hostname: '127.0.0.1', + port: proxyPort, + path: '/v1/messages', + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'x-api-key': 'test-proxy-token', + }, + }, + () => resolve() + ); + + request.write( + JSON.stringify({ + model: 'hf-model', + messages: [{ role: 'user', content: 'hello' }], + }) + ); + request.end(); + + setTimeout(() => { + request.destroy(new Error('client aborted')); + resolve(); + }, 50); + }); + + const logPath = path.join(tempDir, '.ccs', 'logs', 'current.jsonl'); + await Promise.race([ + new Promise((resolve, reject) => { + const startedAt = Date.now(); + const timer = setInterval(() => { + if (fs.existsSync(logPath)) { + const content = fs.readFileSync(logPath, 'utf8'); + if (content.includes('"event":"request.disconnect"')) { + clearInterval(timer); + resolve(); + return; + } + } + + if (Date.now() - startedAt > 1500) { + clearInterval(timer); + reject(new Error('proxy did not log disconnect cleanup')); + } + }, 50); + }), + ]); + }); +}); diff --git a/tests/integration/proxy/messages-endpoint.test.ts b/tests/integration/proxy/messages-endpoint.test.ts index ee96ce4c..d6ce7469 100644 --- a/tests/integration/proxy/messages-endpoint.test.ts +++ b/tests/integration/proxy/messages-endpoint.test.ts @@ -173,16 +173,44 @@ describe('openai proxy messages endpoint', () => { expect(body.error?.type).toBe('authentication_error'); }); - it('returns invalid_request_error for unsupported Anthropic content blocks', async () => { + it('translates supported Anthropic image blocks into OpenAI image_url content', async () => { const response = await requestProxy({ model: 'hf-model', - messages: [{ role: 'user', content: [{ type: 'image' }] }], + messages: [ + { + role: 'user', + content: [ + { type: 'text', text: 'Describe this' }, + { + type: 'image', + source: { + type: 'base64', + media_type: 'image/png', + data: 'ZmFrZQ==', + }, + }, + ], + }, + ], }); - const body = (await response.json()) as { error?: { type?: string; message?: string } }; - expect(response.status).toBe(400); - expect(body.error?.type).toBe('invalid_request_error'); - expect(body.error?.message).toContain('is not supported'); + expect(response.status).toBe(200); + const parsedUpstream = upstreamBody as { + messages?: Array<{ + role: string; + content: Array<{ type: string; text?: string; image_url?: { url?: string } }>; + }>; + }; + expect(parsedUpstream.messages?.[0]).toEqual({ + role: 'user', + content: [ + { type: 'text', text: 'Describe this' }, + { + type: 'image_url', + image_url: { url: 'data:image/png;base64,ZmFrZQ==' }, + }, + ], + }); }); it('accepts query strings and trailing slashes on the messages route', async () => { diff --git a/tests/integration/proxy/request-routing.test.ts b/tests/integration/proxy/request-routing.test.ts new file mode 100644 index 00000000..d5a8ce0d --- /dev/null +++ b/tests/integration/proxy/request-routing.test.ts @@ -0,0 +1,217 @@ +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import getPort from 'get-port'; +import * as fs from 'fs'; +import * as http from 'http'; +import * as os from 'os'; +import * as path from 'path'; +import { startOpenAICompatProxyServer } from '../../../src/proxy/server/proxy-server'; +import type { OpenAICompatProfileConfig } from '../../../src/proxy/profile-router'; + +let originalCcsHome: string | undefined; +let tempDir: string; +let proxyServer: http.Server; +let upstreamServers: http.Server[] = []; +let proxyPort: number; + +function startMockUpstream( + port: number, + hitLabel: string, + hits: string[], + bodies: Array<{ label: string; body: unknown }> +): Promise { + return new Promise((resolve) => { + const server = http.createServer(async (req, res) => { + let body = ''; + for await (const chunk of req) { + body += chunk.toString(); + } + hits.push(hitLabel); + bodies.push({ label: hitLabel, body: JSON.parse(body) }); + + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end( + JSON.stringify({ + id: `chatcmpl_${hitLabel}`, + model: hitLabel, + choices: [ + { + index: 0, + message: { role: 'assistant', content: `Reply from ${hitLabel}` }, + finish_reason: 'stop', + }, + ], + usage: { prompt_tokens: 2, completion_tokens: 3 }, + }) + ); + }); + upstreamServers.push(server); + server.listen(port, '127.0.0.1', () => resolve()); + }); +} + +function writeSettings(profileName: string, env: Record): string { + const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`); + fs.writeFileSync(settingsPath, JSON.stringify({ env }, null, 2), 'utf8'); + return settingsPath; +} + +async function requestProxy(payload: unknown): Promise { + return fetch(`http://127.0.0.1:${proxyPort}/v1/messages`, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'x-api-key': 'test-proxy-token', + }, + body: JSON.stringify(payload), + }); +} + +beforeEach(async () => { + originalCcsHome = process.env.CCS_HOME; + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-routing-')); + fs.mkdirSync(path.join(tempDir, '.ccs'), { recursive: true }); + process.env.CCS_HOME = tempDir; + proxyPort = await getPort(); +}); + +afterEach(async () => { + await Promise.all( + upstreamServers.map( + (server) => + new Promise((resolve) => { + server.close(() => resolve()); + }) + ) + ); + upstreamServers = []; + if (proxyServer) { + await new Promise((resolve) => proxyServer.close(() => resolve())); + } + if (originalCcsHome !== undefined) { + process.env.CCS_HOME = originalCcsHome; + } else { + delete process.env.CCS_HOME; + } + fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +describe('openai proxy request routing', () => { + it('routes explicit profile:model selectors to the matching upstream profile', async () => { + const primaryPort = await getPort(); + const secondaryPort = await getPort(); + const hits: string[] = []; + const bodies: Array<{ label: string; body: unknown }> = []; + await startMockUpstream(primaryPort, 'primary', hits, bodies); + await startMockUpstream(secondaryPort, 'secondary', hits, bodies); + + const primarySettings = writeSettings('hf', { + ANTHROPIC_BASE_URL: `http://127.0.0.1:${primaryPort}`, + ANTHROPIC_AUTH_TOKEN: 'hf_token', + ANTHROPIC_MODEL: 'hf-default', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }); + const secondarySettings = writeSettings('deepseek', { + ANTHROPIC_BASE_URL: `http://127.0.0.1:${secondaryPort}`, + ANTHROPIC_AUTH_TOKEN: 'deepseek_token', + ANTHROPIC_MODEL: 'deepseek-chat', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }); + + fs.writeFileSync( + path.join(tempDir, '.ccs', 'config.json'), + JSON.stringify({ profiles: { hf: primarySettings, deepseek: secondarySettings } }, null, 2), + 'utf8' + ); + + const profile: OpenAICompatProfileConfig = { + profileName: 'hf', + settingsPath: primarySettings, + baseUrl: `http://127.0.0.1:${primaryPort}`, + apiKey: 'hf_token', + provider: 'generic-chat-completion-api', + model: 'hf-default', + }; + proxyServer = startOpenAICompatProxyServer({ + profile, + port: proxyPort, + authToken: 'test-proxy-token', + }); + + const response = await requestProxy({ + model: 'deepseek:deepseek-reasoner', + messages: [{ role: 'user', content: 'hello' }], + }); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ + content: [{ type: 'text', text: 'Reply from secondary' }], + }); + expect(hits).toEqual(['secondary']); + expect(bodies[0]?.body).toMatchObject({ model: 'deepseek-reasoner' }); + }); + + it('routes thinking requests through the configured think scenario', async () => { + const primaryPort = await getPort(); + const thinkPort = await getPort(); + const hits: string[] = []; + const bodies: Array<{ label: string; body: unknown }> = []; + await startMockUpstream(primaryPort, 'primary', hits, bodies); + await startMockUpstream(thinkPort, 'thinker', hits, bodies); + + const primarySettings = writeSettings('hf', { + ANTHROPIC_BASE_URL: `http://127.0.0.1:${primaryPort}`, + ANTHROPIC_AUTH_TOKEN: 'hf_token', + ANTHROPIC_MODEL: 'hf-default', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }); + const thinkSettings = writeSettings('thinker', { + ANTHROPIC_BASE_URL: `http://127.0.0.1:${thinkPort}`, + ANTHROPIC_AUTH_TOKEN: 'think_token', + ANTHROPIC_MODEL: 'deepseek-reasoner', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }); + + fs.writeFileSync( + path.join(tempDir, '.ccs', 'config.json'), + JSON.stringify( + { + profiles: { hf: primarySettings, thinker: thinkSettings }, + proxy: { + routing: { + think: 'thinker:deepseek-reasoner', + }, + }, + }, + null, + 2 + ), + 'utf8' + ); + + const profile: OpenAICompatProfileConfig = { + profileName: 'hf', + settingsPath: primarySettings, + baseUrl: `http://127.0.0.1:${primaryPort}`, + apiKey: 'hf_token', + provider: 'generic-chat-completion-api', + model: 'hf-default', + }; + proxyServer = startOpenAICompatProxyServer({ + profile, + port: proxyPort, + authToken: 'test-proxy-token', + }); + + const response = await requestProxy({ + model: 'hf-default', + thinking: { type: 'enabled', budget_tokens: 9000 }, + messages: [{ role: 'user', content: 'think hard' }], + }); + + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ + content: [{ type: 'text', text: 'Reply from thinker' }], + }); + expect(hits).toEqual(['thinker']); + expect(bodies[0]?.body).toMatchObject({ model: 'deepseek-reasoner' }); + }); +}); diff --git a/tests/unit/proxy/request-router.test.ts b/tests/unit/proxy/request-router.test.ts new file mode 100644 index 00000000..eb780493 --- /dev/null +++ b/tests/unit/proxy/request-router.test.ts @@ -0,0 +1,194 @@ +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { resolveProxyRequestRoute } from '../../../src/proxy/request-router'; +import { resolveOpenAICompatProfileConfig } from '../../../src/proxy/profile-router'; +import { loadSettings } from '../../../src/utils/config-manager'; + +let originalCcsHome: string | undefined; +let tempDir: string; + +function writeSettings(profileName: string, env: Record): string { + const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`); + fs.writeFileSync(settingsPath, JSON.stringify({ env }, null, 2), 'utf8'); + return settingsPath; +} + +function buildProfile(profileName: string) { + const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`); + const profile = resolveOpenAICompatProfileConfig( + profileName, + settingsPath, + loadSettings(settingsPath).env || {} + ); + expect(profile).toBeTruthy(); + return profile!; +} + +beforeEach(() => { + originalCcsHome = process.env.CCS_HOME; + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-router-')); + fs.mkdirSync(path.join(tempDir, '.ccs'), { recursive: true }); +}); + +afterEach(() => { + if (originalCcsHome !== undefined) { + process.env.CCS_HOME = originalCcsHome; + } else { + delete process.env.CCS_HOME; + } + fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +describe('resolveProxyRequestRoute', () => { + beforeEach(() => { + process.env.CCS_HOME = tempDir; + + const profiles = { + hf: writeSettings('hf', { + ANTHROPIC_BASE_URL: 'https://router.huggingface.co/v1', + ANTHROPIC_AUTH_TOKEN: 'hf_token', + ANTHROPIC_MODEL: 'hf-default', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }), + deepseek: writeSettings('deepseek', { + ANTHROPIC_BASE_URL: 'https://api.deepseek.com/v1', + ANTHROPIC_AUTH_TOKEN: 'deepseek_token', + ANTHROPIC_MODEL: 'deepseek-chat', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }), + thinker: writeSettings('thinker', { + ANTHROPIC_BASE_URL: 'https://thinking.example.com/v1', + ANTHROPIC_AUTH_TOKEN: 'think_token', + ANTHROPIC_MODEL: 'deepseek-reasoner', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }), + search: writeSettings('search', { + ANTHROPIC_BASE_URL: 'https://search.example.com/v1', + ANTHROPIC_AUTH_TOKEN: 'search_token', + ANTHROPIC_MODEL: 'sonar-pro', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }), + background: writeSettings('background', { + ANTHROPIC_BASE_URL: 'https://background.example.com/v1', + ANTHROPIC_AUTH_TOKEN: 'background_token', + ANTHROPIC_MODEL: 'qwen-small', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }), + long: writeSettings('long', { + ANTHROPIC_BASE_URL: 'https://long.example.com/v1', + ANTHROPIC_AUTH_TOKEN: 'long_token', + ANTHROPIC_MODEL: 'gemini-2.5-pro', + CCS_DROID_PROVIDER: 'generic-chat-completion-api', + }), + }; + + fs.writeFileSync( + path.join(tempDir, '.ccs', 'config.json'), + JSON.stringify( + { + profiles, + proxy: { + routing: { + default: 'hf:hf-default', + background: 'background:qwen-small', + think: 'thinker:deepseek-reasoner', + longContext: 'long:gemini-2.5-pro', + longContextThreshold: 10, + webSearch: 'search:sonar-pro', + }, + }, + }, + null, + 2 + ), + 'utf8' + ); + }); + + it('uses explicit profile:model selectors', () => { + const route = resolveProxyRequestRoute(buildProfile('hf'), { + model: 'deepseek:deepseek-reasoner', + stream: true, + messages: [{ role: 'user', content: 'plan this' }], + }); + + expect(route.profile.profileName).toBe('deepseek'); + expect(route.model).toBe('deepseek-reasoner'); + expect(route.source).toBe('explicit-profile'); + }); + + it('matches plain model ids to the profile that owns them', () => { + const route = resolveProxyRequestRoute(buildProfile('hf'), { + model: 'deepseek-chat', + stream: true, + messages: [{ role: 'user', content: 'hello' }], + }); + + expect(route.profile.profileName).toBe('deepseek'); + expect(route.model).toBe('deepseek-chat'); + expect(route.source).toBe('profile-model-match'); + }); + + it('routes thinking requests through the configured think scenario', () => { + const route = resolveProxyRequestRoute(buildProfile('hf'), { + model: 'hf-default', + stream: true, + reasoning_effort: 'high', + reasoning: { enabled: true, effort: 'high' }, + messages: [{ role: 'user', content: 'work this out carefully' }], + }); + + expect(route.profile.profileName).toBe('thinker'); + expect(route.model).toBe('deepseek-reasoner'); + expect(route.scenario).toBe('think'); + expect(route.source).toBe('scenario'); + }); + + it('routes long-context requests through the configured longContext scenario', () => { + const route = resolveProxyRequestRoute(buildProfile('hf'), { + model: 'hf-default', + stream: true, + messages: [{ role: 'user', content: 'x'.repeat(120) }], + }); + + expect(route.profile.profileName).toBe('long'); + expect(route.model).toBe('gemini-2.5-pro'); + expect(route.scenario).toBe('longContext'); + expect(route.estimatedTokens).toBeGreaterThan(10); + }); + + it('routes web_search tool requests through the configured webSearch scenario', () => { + const route = resolveProxyRequestRoute(buildProfile('hf'), { + model: 'hf-default', + stream: true, + messages: [{ role: 'user', content: 'search the docs' }], + tools: [ + { + type: 'function', + function: { + name: 'web_search', + parameters: { type: 'object' }, + }, + }, + ], + }); + + expect(route.profile.profileName).toBe('search'); + expect(route.model).toBe('sonar-pro'); + expect(route.scenario).toBe('webSearch'); + }); + + it('routes haiku requests through the configured background scenario', () => { + const route = resolveProxyRequestRoute(buildProfile('hf'), { + model: 'claude-3-haiku', + stream: true, + messages: [{ role: 'user', content: 'run this in the background' }], + }); + + expect(route.profile.profileName).toBe('background'); + expect(route.model).toBe('qwen-small'); + expect(route.scenario).toBe('background'); + }); +}); diff --git a/tests/unit/proxy/transformers/request-transformer.test.ts b/tests/unit/proxy/transformers/request-transformer.test.ts index e3d6afad..25687c20 100644 --- a/tests/unit/proxy/transformers/request-transformer.test.ts +++ b/tests/unit/proxy/transformers/request-transformer.test.ts @@ -28,6 +28,7 @@ describe('ProxyRequestTransformer', () => { expect(result.stream).toBe(true); expect(result.reasoning_effort).toBe('high'); + expect(result.reasoning).toEqual({ enabled: true, effort: 'high' }); expect(result.max_tokens).toBe(1024); expect(result.temperature).toBe(0.2); expect(result.top_p).toBe(0.9); @@ -42,6 +43,43 @@ describe('ProxyRequestTransformer', () => { }); }); + it('translates base64 image blocks into OpenAI image_url parts', () => { + const transformer = new ProxyRequestTransformer(); + const result = transformer.transform({ + messages: [ + { + role: 'user', + content: [ + { type: 'text', text: 'Describe this image' }, + { + type: 'image', + source: { + type: 'base64', + media_type: 'image/png', + data: 'ZmFrZS1pbWFnZS1ieXRlcw==', + }, + }, + ], + }, + ], + }); + + expect(result.messages).toEqual([ + { + role: 'user', + content: [ + { type: 'text', text: 'Describe this image' }, + { + type: 'image_url', + image_url: { + url: 'data:image/png;base64,ZmFrZS1pbWFnZS1ieXRlcw==', + }, + }, + ], + }, + ]); + }); + it('drops malformed optional fields but preserves the translated core request', () => { const transformer = new ProxyRequestTransformer(); const result = transformer.transform({