mirror of
https://github.com/tiennm99/ccs.git
synced 2026-07-16 04:18:05 +00:00
feat(proxy): complete openai routing scope
This commit is contained in:
Tam Nhu Tran
@@ -67,6 +67,10 @@ ccs proxy start hf
|
||||
eval "$(ccs proxy activate)"
|
||||
```
|
||||
|
||||
The proxy also supports request-time `profile:model` selectors, scenario-based
|
||||
model routing through `proxy.routing`, and explicit activation helpers such as
|
||||
`ccs proxy activate --fish`.
|
||||
|
||||
Guide: [OpenAI-Compatible Provider Routing](./docs/openai-compatible-providers.md)
|
||||
|
||||
### Related Project: claude-code-router
|
||||
|
||||
@@ -68,8 +68,22 @@ ccs proxy status
|
||||
ccs proxy stop
|
||||
```
|
||||
|
||||
`ccs proxy activate` prints the `ANTHROPIC_*` exports needed for a local
|
||||
Anthropic-compatible session against the running proxy.
|
||||
Useful variants:
|
||||
|
||||
```bash
|
||||
ccs proxy start hf --host 127.0.0.1
|
||||
ccs proxy activate --fish
|
||||
```
|
||||
|
||||
`ccs proxy activate` now prints the full local runtime contract:
|
||||
|
||||
- `ANTHROPIC_BASE_URL`
|
||||
- `ANTHROPIC_AUTH_TOKEN`
|
||||
- `ANTHROPIC_MODEL` plus tier defaults when present
|
||||
- `DISABLE_TELEMETRY`
|
||||
- `DISABLE_COST_WARNINGS`
|
||||
- `API_TIMEOUT_MS`
|
||||
- `NO_PROXY`
|
||||
|
||||
## One Active Proxy Profile
|
||||
|
||||
@@ -82,6 +96,56 @@ The current runtime is a single local proxy daemon.
|
||||
This is intentional to avoid breaking an in-flight Claude session by swapping
|
||||
its upstream provider out from under it.
|
||||
|
||||
## Request-Time Routing
|
||||
|
||||
The proxy is no longer limited to the startup profile's default model.
|
||||
|
||||
Supported request-time selectors:
|
||||
|
||||
- `profile:model`
|
||||
Example: `deepseek:deepseek-reasoner`
|
||||
- `profile`
|
||||
Example: `openrouter`
|
||||
- plain model ids
|
||||
Example: `deepseek-chat`
|
||||
|
||||
Routing behavior:
|
||||
|
||||
1. `profile:model` wins immediately.
|
||||
2. Scenario routing may override the active profile when configured.
|
||||
3. Plain model ids are matched against the configured OpenAI-compatible
|
||||
profiles before falling back to the active profile.
|
||||
|
||||
This means a Claude session launched through one compatible profile can still
|
||||
request another compatible profile/model when the proxy can resolve it safely.
|
||||
|
||||
## Scenario Routing
|
||||
|
||||
Scenario routing is now supported through `proxy.routing` in your CCS config.
|
||||
|
||||
Example `~/.ccs/config.yaml`:
|
||||
|
||||
```yaml
|
||||
proxy:
|
||||
routing:
|
||||
default: "deepseek:deepseek-chat"
|
||||
background: "ollama:qwen2.5-coder:0.5b"
|
||||
think: "deepseek:deepseek-reasoner"
|
||||
longContext: "openrouter:google/gemini-2.5-pro"
|
||||
longContextThreshold: 60000
|
||||
webSearch: "openrouter:perplexity/sonar-pro"
|
||||
```
|
||||
|
||||
Current scenario detection:
|
||||
|
||||
- `background`: requested model contains `haiku`
|
||||
- `think`: Anthropic `thinking` is enabled
|
||||
- `longContext`: estimated request tokens exceed `longContextThreshold`
|
||||
- `webSearch`: tool list includes `web_search`
|
||||
- `default`: fallback selector when the above do not apply
|
||||
|
||||
Routing decisions are logged through CCS structured logs.
|
||||
|
||||
## How Profile Detection Works
|
||||
|
||||
CCS keeps these profiles in the normal API/settings-profile flow.
|
||||
@@ -102,6 +166,79 @@ OpenAI-compatible endpoints such as:
|
||||
|
||||
are routed through the local proxy for Claude-target launches.
|
||||
|
||||
## Provider Setup
|
||||
|
||||
### DeepSeek
|
||||
|
||||
Use a settings profile whose env looks like:
|
||||
|
||||
```json
|
||||
{
|
||||
"env": {
|
||||
"ANTHROPIC_BASE_URL": "https://api.deepseek.com/v1",
|
||||
"ANTHROPIC_AUTH_TOKEN": "sk-...",
|
||||
"ANTHROPIC_MODEL": "deepseek-chat",
|
||||
"CCS_DROID_PROVIDER": "generic-chat-completion-api"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Typical override target:
|
||||
|
||||
- `deepseek:deepseek-reasoner`
|
||||
|
||||
### OpenRouter
|
||||
|
||||
```json
|
||||
{
|
||||
"env": {
|
||||
"ANTHROPIC_BASE_URL": "https://openrouter.ai/api/v1",
|
||||
"ANTHROPIC_AUTH_TOKEN": "sk-or-...",
|
||||
"ANTHROPIC_MODEL": "openai/gpt-4.1-mini",
|
||||
"CCS_DROID_PROVIDER": "generic-chat-completion-api"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Useful when you want:
|
||||
|
||||
- model fan-out behind one provider profile
|
||||
- long-context or web-search scenario targets
|
||||
|
||||
### Ollama / Local Gateways
|
||||
|
||||
```json
|
||||
{
|
||||
"env": {
|
||||
"ANTHROPIC_BASE_URL": "http://127.0.0.1:11434",
|
||||
"ANTHROPIC_AUTH_TOKEN": "ollama",
|
||||
"ANTHROPIC_MODEL": "qwen3-coder",
|
||||
"CCS_DROID_PROVIDER": "generic-chat-completion-api"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For self-signed HTTPS gateways, add `CCS_OPENAI_PROXY_INSECURE=1`.
|
||||
|
||||
### DashScope / Qwen Compatible Mode
|
||||
|
||||
DashScope's compatible endpoint works even when older settings files still
|
||||
carry a stale Anthropic-style provider hint:
|
||||
|
||||
```json
|
||||
{
|
||||
"env": {
|
||||
"ANTHROPIC_BASE_URL": "https://dashscope-us.aliyuncs.com/compatible-mode/v1",
|
||||
"ANTHROPIC_AUTH_TOKEN": "sk-...",
|
||||
"ANTHROPIC_MODEL": "qwen3.6-plus",
|
||||
"CCS_DROID_PROVIDER": "anthropic"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
CCS now infers the OpenAI-compatible route from the base URL and does not let
|
||||
that stale provider hint block proxy routing.
|
||||
|
||||
## Self-Signed TLS
|
||||
|
||||
If your upstream gateway uses a self-signed or privately issued certificate,
|
||||
@@ -124,29 +261,50 @@ That flag is respected by both:
|
||||
|
||||
- `ccs <profile>` with Claude target: auto-starts the local proxy when needed
|
||||
- `ccs proxy start <profile>`: starts the proxy explicitly
|
||||
- `GET /`: proxy info and bound profile details
|
||||
- `GET /health`: proxy liveness check
|
||||
- `GET /v1/models`: local view of the configured model mapping
|
||||
- `POST /v1/messages`: Anthropic-compatible request entrypoint
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing or invalid local proxy token
|
||||
|
||||
- Re-run `eval "$(ccs proxy activate)"`
|
||||
- Check `ccs proxy status` and confirm the expected profile is running
|
||||
|
||||
### Self-signed or private CA upstream
|
||||
|
||||
- Add `CCS_OPENAI_PROXY_INSECURE=1` to the profile settings
|
||||
- Restart the proxy after changing the setting
|
||||
|
||||
### Port conflict on `3456`
|
||||
|
||||
- Start with a fixed port: `ccs proxy start hf --port 3457`
|
||||
- Re-run `ccs proxy activate` after changing the port
|
||||
|
||||
### Provider returns `429` or empty upstream output
|
||||
|
||||
- CCS now preserves upstream rate-limit errors and retry headers
|
||||
- Empty or malformed provider JSON is returned as Anthropic-style `api_error`
|
||||
|
||||
### Requests route to the wrong model/profile
|
||||
|
||||
- Use an explicit selector such as `profile:model`
|
||||
- Review `proxy.routing` if scenario routing is enabled
|
||||
- Check CCS structured logs in `~/.ccs/logs/current.jsonl` for routing decisions
|
||||
|
||||
## Validation
|
||||
|
||||
The shipped coverage includes:
|
||||
|
||||
- unit tests for OpenAI-compatible profile detection
|
||||
- unit tests for Anthropic -> OpenAI request translation
|
||||
- unit tests for request-time profile/model routing and scenario routing
|
||||
- unit tests for multi-line SSE parsing
|
||||
- integration tests for `/v1/messages` request/response translation
|
||||
- integration tests for rate limits, empty upstream responses, timeout handling,
|
||||
thinking/tool-call chunk streaming, and request-time routing
|
||||
- integration tests for daemon lifecycle and `/health` / `/v1/models`
|
||||
- e2e tests for `ccs proxy` lifecycle
|
||||
- e2e tests for `ccs <profile>` auto-routing through a mock upstream
|
||||
|
||||
## Current Scope
|
||||
|
||||
The current implementation focuses on the core routing path:
|
||||
|
||||
- local proxy lifecycle
|
||||
- Anthropic/OpenAI request-response translation
|
||||
- Claude-target settings profile auto-routing
|
||||
|
||||
Scenario-based routing and token-count-driven model switching remain follow-up
|
||||
work if they are needed beyond the base provider-routing flow.
|
||||
|
||||
@@ -41,7 +41,7 @@ All major modularization work is complete. The codebase evolved from monolithic
|
||||
|
||||
### Recent Fixes
|
||||
|
||||
- **2026-04-14**: **#991** CCS now auto-routes Claude-target settings profiles that use OpenAI-compatible endpoints through a local Anthropic-compatible proxy instead of sending raw Anthropic `/v1/messages` traffic directly to chat-completions backends. The new `ccs proxy` command supports `start`, `status`, `activate`, and `stop`, the runtime exposes `/health` and `/v1/models`, the shared SSE parser now handles multi-line `data:` payloads, and the feature ships with dedicated unit, integration, and e2e coverage.
|
||||
- **2026-04-14**: **#991** CCS now auto-routes Claude-target settings profiles that use OpenAI-compatible endpoints through a local Anthropic-compatible proxy instead of sending raw Anthropic `/v1/messages` traffic directly to chat-completions backends. The `ccs proxy` command now supports `start`, `status`, `activate`, and `stop` with explicit host binding, shell-aware activation helpers, and a fuller local runtime env contract. The proxy surface now exposes `GET /`, `/health`, `/v1/models`, and `/v1/messages`, logs routing decisions into CCS structured logs, supports Anthropic image blocks plus request-time `profile:model` overrides, and adds config-driven scenario routing (`background`, `think`, `longContext`, `webSearch`) on top of the compatible-profile path. Coverage now includes request routing, rate-limit/timeout/empty-upstream failures, chunked tool-call streaming, and disconnect cleanup alongside the existing unit, integration, and e2e suites.
|
||||
- **2026-04-10**: **#765** `/providers` now includes a first-class Hugging Face preset for API Profiles. CCS exposes Hugging Face Inference Providers through the existing OpenAI-compatible profile flow with the official router endpoint `https://router.huggingface.co/v1`, a short `hf` default profile name, and `hf` preset alias support for both the dashboard chooser and `ccs api create --preset hf`.
|
||||
- **2026-04-10**: **#944** Image Analysis auth readiness no longer collapses to native Read when merged runtime-status dependency overrides include a missing initializer value. CCS now preserves default dependency functions when override entries are `undefined`, still reads token-backed auth status directly in the local readiness path, and includes regression coverage for the missing-initializer case that previously surfaced as `deps.initializeAccounts is not a function`.
|
||||
- **2026-04-10**: **#945** CCS now normalizes Gemini CLI and Antigravity tier signals around an explicit `free / pro / ultra / unknown` model, preserves raw tier ids such as `g1-pro-tier`, enriches Gemini quota responses with provider entitlement evidence, classifies `MODEL_CAPACITY_EXHAUSTED` separately from auth/entitlement failures, fixes the Antigravity CLI quota table so live quota-derived tiers no longer collapse back to stale `unknown`, adds Gemini tier ids to CLI quota output, extends Gemini Flash Lite grouping to cover `gemini-3.1-flash-lite-preview`, and allows Gemini account surfaces to render the same tier badge semantics as Antigravity.
|
||||
|
||||
@@ -36,12 +36,15 @@ function showHelp(): number {
|
||||
console.log('');
|
||||
console.log('Options:');
|
||||
console.log(' --port <n> Override the local proxy port (default: 3456)');
|
||||
console.log(' --host <addr> Bind the proxy server to a specific host (default: 127.0.0.1)');
|
||||
console.log(' --shell <name> activate only: auto|bash|zsh|fish|powershell');
|
||||
console.log(' --fish activate only: shorthand for --shell fish');
|
||||
console.log(' --insecure Disable upstream TLS verification');
|
||||
console.log('');
|
||||
console.log('Examples:');
|
||||
console.log(' ccs proxy start hf');
|
||||
console.log(' eval "$(ccs proxy activate)"');
|
||||
console.log(' ccs proxy activate --fish');
|
||||
console.log(' ccs proxy status');
|
||||
console.log(' ccs proxy stop');
|
||||
console.log('');
|
||||
@@ -61,11 +64,14 @@ function resolveProfile(profileName: string) {
|
||||
async function handleStart(args: string[]): Promise<number> {
|
||||
const profileName = args.find((arg) => !arg.startsWith('-'));
|
||||
if (!profileName) {
|
||||
console.error(fail('Usage: ccs proxy start <profile> [--port <n>] [--insecure]'));
|
||||
console.error(
|
||||
fail('Usage: ccs proxy start <profile> [--port <n>] [--host <addr>] [--insecure]')
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const portValue = parseOptionValue(args, '--port');
|
||||
const host = parseOptionValue(args, '--host');
|
||||
const port = portValue ? Number.parseInt(portValue, 10) || 3456 : undefined;
|
||||
let profile;
|
||||
try {
|
||||
@@ -77,6 +83,7 @@ async function handleStart(args: string[]): Promise<number> {
|
||||
|
||||
const result = await startOpenAICompatProxy(profile, {
|
||||
...(port ? { port } : {}),
|
||||
...(host ? { host } : {}),
|
||||
insecure: args.includes('--insecure'),
|
||||
});
|
||||
|
||||
@@ -101,6 +108,10 @@ async function handleStatus(): Promise<number> {
|
||||
}
|
||||
|
||||
console.log(ok(`Proxy running on port ${status.port}`));
|
||||
if (status.host) {
|
||||
console.log(` Host: ${status.host}`);
|
||||
console.log(` Local URL: http://${status.host}:${status.port}`);
|
||||
}
|
||||
console.log(` Profile: ${status.profileName}`);
|
||||
console.log(` Base URL: ${status.baseUrl}`);
|
||||
if (status.model) {
|
||||
@@ -119,7 +130,7 @@ async function handleActivate(args: string[]): Promise<number> {
|
||||
return 1;
|
||||
}
|
||||
|
||||
const shell = detectShell(parseOptionValue(args, '--shell'));
|
||||
const shell = detectShell(args.includes('--fish') ? 'fish' : parseOptionValue(args, '--shell'));
|
||||
let profile;
|
||||
try {
|
||||
profile = resolveProfile(status.profileName);
|
||||
@@ -128,7 +139,13 @@ async function handleActivate(args: string[]): Promise<number> {
|
||||
return 1;
|
||||
}
|
||||
|
||||
const env = buildOpenAICompatProxyEnv(profile, status.port, status.authToken);
|
||||
const env = buildOpenAICompatProxyEnv(
|
||||
profile,
|
||||
status.port,
|
||||
status.authToken,
|
||||
undefined,
|
||||
status.host || '127.0.0.1'
|
||||
);
|
||||
Object.entries(env).forEach(([key, value]) => {
|
||||
console.log(formatExportLine(shell, key, value));
|
||||
});
|
||||
|
||||
@@ -384,6 +384,18 @@ function mergeWithDefaults(partial: Partial<UnifiedConfig>): UnifiedConfig {
|
||||
: defaults.cliproxy.routing?.strategy,
|
||||
},
|
||||
},
|
||||
proxy: {
|
||||
routing: {
|
||||
default: partial.proxy?.routing?.default ?? defaults.proxy?.routing?.default,
|
||||
background: partial.proxy?.routing?.background ?? defaults.proxy?.routing?.background,
|
||||
think: partial.proxy?.routing?.think ?? defaults.proxy?.routing?.think,
|
||||
longContext: partial.proxy?.routing?.longContext ?? defaults.proxy?.routing?.longContext,
|
||||
webSearch: partial.proxy?.routing?.webSearch ?? defaults.proxy?.routing?.webSearch,
|
||||
longContextThreshold:
|
||||
partial.proxy?.routing?.longContextThreshold ??
|
||||
defaults.proxy?.routing?.longContextThreshold,
|
||||
},
|
||||
},
|
||||
logging: {
|
||||
enabled: partial.logging?.enabled ?? DEFAULT_LOGGING_CONFIG.enabled,
|
||||
level: partial.logging?.level ?? DEFAULT_LOGGING_CONFIG.level,
|
||||
@@ -674,6 +686,17 @@ function generateYamlWithComments(config: UnifiedConfig): string {
|
||||
);
|
||||
lines.push('');
|
||||
|
||||
if (config.proxy?.routing) {
|
||||
lines.push('# ----------------------------------------------------------------------------');
|
||||
lines.push('# Proxy Routing: OpenAI-compatible local proxy model selection rules');
|
||||
lines.push('# Use profile:model selectors to force a target profile and upstream model.');
|
||||
lines.push('# ----------------------------------------------------------------------------');
|
||||
lines.push(
|
||||
yaml.dump({ proxy: config.proxy }, { indent: 2, lineWidth: -1, quotingType: '"' }).trim()
|
||||
);
|
||||
lines.push('');
|
||||
}
|
||||
|
||||
if (config.logging) {
|
||||
lines.push('# ----------------------------------------------------------------------------');
|
||||
lines.push('# Logging: CCS-owned structured runtime logs');
|
||||
|
||||
@@ -498,6 +498,19 @@ export interface ProxyLocalConfig {
|
||||
auto_start: boolean;
|
||||
}
|
||||
|
||||
export interface OpenAICompatProxyRoutingConfig {
|
||||
default?: string;
|
||||
background?: string;
|
||||
think?: string;
|
||||
longContext?: string;
|
||||
webSearch?: string;
|
||||
longContextThreshold?: number;
|
||||
}
|
||||
|
||||
export interface OpenAICompatProxyConfig {
|
||||
routing?: OpenAICompatProxyRoutingConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* CLIProxy server configuration section.
|
||||
* Controls whether CCS uses local or remote CLIProxyAPI instance.
|
||||
@@ -856,6 +869,8 @@ export interface UnifiedConfig {
|
||||
profiles: Record<string, ProfileConfig>;
|
||||
/** CLIProxy configuration */
|
||||
cliproxy: CLIProxyConfig;
|
||||
/** OpenAI-compatible local proxy configuration */
|
||||
proxy?: OpenAICompatProxyConfig;
|
||||
/** CCS-owned structured logging configuration */
|
||||
logging?: LoggingConfig;
|
||||
/** User preferences */
|
||||
@@ -934,6 +949,12 @@ export const DEFAULT_CLIPROXY_SERVER_CONFIG: CliproxyServerConfig = {
|
||||
},
|
||||
};
|
||||
|
||||
export const DEFAULT_OPENAI_COMPAT_PROXY_CONFIG: OpenAICompatProxyConfig = {
|
||||
routing: {
|
||||
longContextThreshold: 60_000,
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Create an empty unified config with defaults.
|
||||
*/
|
||||
@@ -958,6 +979,11 @@ export function createEmptyUnifiedConfig(): UnifiedConfig {
|
||||
strategy: 'round-robin',
|
||||
},
|
||||
},
|
||||
proxy: {
|
||||
routing: {
|
||||
...DEFAULT_OPENAI_COMPAT_PROXY_CONFIG.routing,
|
||||
},
|
||||
},
|
||||
logging: { ...DEFAULT_LOGGING_CONFIG },
|
||||
preferences: {
|
||||
theme: 'system',
|
||||
|
||||
@@ -4,6 +4,7 @@ import { startOpenAICompatProxyServer } from './server/proxy-server';
|
||||
|
||||
interface RuntimeOptions {
|
||||
port: number;
|
||||
host: string;
|
||||
profileName: string;
|
||||
settingsPath: string;
|
||||
authToken: string;
|
||||
@@ -12,6 +13,7 @@ interface RuntimeOptions {
|
||||
|
||||
function parseArgs(argv: string[]): RuntimeOptions {
|
||||
let port = 3456;
|
||||
let host = '127.0.0.1';
|
||||
let profileName = '';
|
||||
let settingsPath = '';
|
||||
let authToken = '';
|
||||
@@ -23,6 +25,10 @@ function parseArgs(argv: string[]): RuntimeOptions {
|
||||
port = Number.parseInt(argv[++i] || '', 10) || port;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--host' && argv[i + 1]) {
|
||||
host = argv[++i] || host;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--profile' && argv[i + 1]) {
|
||||
profileName = argv[++i] || '';
|
||||
continue;
|
||||
@@ -40,7 +46,7 @@ function parseArgs(argv: string[]): RuntimeOptions {
|
||||
}
|
||||
}
|
||||
|
||||
return { port, profileName, settingsPath, authToken, insecure };
|
||||
return { port, host, profileName, settingsPath, authToken, insecure };
|
||||
}
|
||||
|
||||
function startRuntime(options: RuntimeOptions): void {
|
||||
@@ -62,6 +68,7 @@ function startRuntime(options: RuntimeOptions): void {
|
||||
|
||||
const server = startOpenAICompatProxyServer({
|
||||
profile,
|
||||
host: options.host,
|
||||
port: options.port,
|
||||
authToken: options.authToken,
|
||||
insecure: options.insecure,
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
export interface OpenAICompatProxySession {
|
||||
profileName: string;
|
||||
settingsPath: string;
|
||||
host: string;
|
||||
port: number;
|
||||
baseUrl: string;
|
||||
authToken: string;
|
||||
|
||||
@@ -225,10 +225,11 @@ export async function stopOpenAICompatProxy(): Promise<{ success: boolean; error
|
||||
|
||||
export async function startOpenAICompatProxy(
|
||||
profile: OpenAICompatProfileConfig,
|
||||
options: { port?: number; insecure?: boolean } = {}
|
||||
options: { port?: number; host?: string; insecure?: boolean } = {}
|
||||
): Promise<StartOpenAICompatProxyResult> {
|
||||
return withOpenAICompatProxyLock(async () => {
|
||||
const status = await getOpenAICompatProxyStatus();
|
||||
const host = options.host?.trim() || status.host || '127.0.0.1';
|
||||
const port =
|
||||
typeof options.port === 'number'
|
||||
? options.port
|
||||
@@ -245,7 +246,12 @@ export async function startOpenAICompatProxy(
|
||||
if (!Number.isInteger(port) || port < 1 || port > 65535) {
|
||||
return { success: false, port, error: `Invalid port: ${port}` };
|
||||
}
|
||||
if (status.running && status.profileName === profile.profileName && status.port === port) {
|
||||
if (
|
||||
status.running &&
|
||||
status.profileName === profile.profileName &&
|
||||
status.port === port &&
|
||||
(status.host || '127.0.0.1') === host
|
||||
) {
|
||||
return {
|
||||
success: true,
|
||||
alreadyRunning: true,
|
||||
@@ -304,6 +310,8 @@ export async function startOpenAICompatProxy(
|
||||
daemonEntry,
|
||||
'--port',
|
||||
String(port),
|
||||
'--host',
|
||||
host,
|
||||
'--profile',
|
||||
profile.profileName,
|
||||
'--settings-path',
|
||||
@@ -323,6 +331,7 @@ export async function startOpenAICompatProxy(
|
||||
writeOpenAICompatProxySession({
|
||||
profileName: profile.profileName,
|
||||
settingsPath: profile.settingsPath,
|
||||
host,
|
||||
port,
|
||||
baseUrl: profile.baseUrl,
|
||||
authToken,
|
||||
|
||||
@@ -4,11 +4,17 @@ export function buildOpenAICompatProxyEnv(
|
||||
profile: OpenAICompatProfileConfig,
|
||||
port: number,
|
||||
authToken: string,
|
||||
claudeConfigDir?: string
|
||||
claudeConfigDir?: string,
|
||||
host = '127.0.0.1'
|
||||
): Record<string, string> {
|
||||
const localBaseUrl = `http://${host}:${port}`;
|
||||
return {
|
||||
ANTHROPIC_BASE_URL: `http://127.0.0.1:${port}`,
|
||||
ANTHROPIC_BASE_URL: localBaseUrl,
|
||||
ANTHROPIC_AUTH_TOKEN: authToken,
|
||||
DISABLE_TELEMETRY: '1',
|
||||
DISABLE_COST_WARNINGS: '1',
|
||||
API_TIMEOUT_MS: '600000',
|
||||
NO_PROXY: host === '127.0.0.1' ? '127.0.0.1,localhost' : `${host},127.0.0.1,localhost`,
|
||||
...(profile.model ? { ANTHROPIC_MODEL: profile.model } : {}),
|
||||
...(profile.opusModel ? { ANTHROPIC_DEFAULT_OPUS_MODEL: profile.opusModel } : {}),
|
||||
...(profile.sonnetModel ? { ANTHROPIC_DEFAULT_SONNET_MODEL: profile.sonnetModel } : {}),
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
import { loadConfigSafe, loadSettings } from '../utils/config-manager';
|
||||
import { expandPath } from '../utils/helpers';
|
||||
import type { ProxyOpenAIRequest } from './transformers/request-transformer';
|
||||
import {
|
||||
loadOpenAICompatProxyRoutingConfig,
|
||||
type OpenAICompatProxyRoutingConfig,
|
||||
} from './routing-config';
|
||||
import { resolveOpenAICompatProfileConfig, type OpenAICompatProfileConfig } from './profile-router';
|
||||
|
||||
export type ProxyRoutingScenario = 'default' | 'background' | 'think' | 'longContext' | 'webSearch';
|
||||
|
||||
export interface ProxyRequestRoute {
|
||||
profile: OpenAICompatProfileConfig;
|
||||
model?: string;
|
||||
scenario?: ProxyRoutingScenario;
|
||||
estimatedTokens: number;
|
||||
source:
|
||||
| 'explicit-profile'
|
||||
| 'scenario'
|
||||
| 'profile-model-match'
|
||||
| 'profile-name'
|
||||
| 'request-model'
|
||||
| 'active-default';
|
||||
}
|
||||
|
||||
function loadOpenAICompatProfiles(
|
||||
activeProfile: OpenAICompatProfileConfig
|
||||
): OpenAICompatProfileConfig[] {
|
||||
const config = loadConfigSafe();
|
||||
const profiles = [activeProfile];
|
||||
|
||||
for (const [profileName, settingsPath] of Object.entries(config.profiles)) {
|
||||
if (profileName === activeProfile.profileName) {
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
const expandedPath = expandPath(settingsPath);
|
||||
const settings = loadSettings(expandedPath);
|
||||
const profile = resolveOpenAICompatProfileConfig(
|
||||
profileName,
|
||||
expandedPath,
|
||||
settings.env || {}
|
||||
);
|
||||
if (profile) {
|
||||
profiles.push(profile);
|
||||
}
|
||||
} catch {
|
||||
// Ignore invalid profiles while routing a live request.
|
||||
}
|
||||
}
|
||||
|
||||
return profiles;
|
||||
}
|
||||
|
||||
function resolveSelectorTarget(
|
||||
selector: string,
|
||||
activeProfile: OpenAICompatProfileConfig,
|
||||
profiles: OpenAICompatProfileConfig[]
|
||||
): { profile: OpenAICompatProfileConfig; model?: string; explicitProfile: boolean } | null {
|
||||
const trimmed = selector.trim();
|
||||
if (!trimmed) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const colonIndex = trimmed.indexOf(':');
|
||||
if (colonIndex > 0) {
|
||||
const profileName = trimmed.slice(0, colonIndex).trim();
|
||||
const profile = profiles.find((candidate) => candidate.profileName === profileName);
|
||||
if (profile) {
|
||||
const model = trimmed.slice(colonIndex + 1).trim() || profile.model;
|
||||
return { profile, model, explicitProfile: true };
|
||||
}
|
||||
}
|
||||
|
||||
const namedProfile = profiles.find((candidate) => candidate.profileName === trimmed);
|
||||
if (namedProfile) {
|
||||
return { profile: namedProfile, model: namedProfile.model, explicitProfile: false };
|
||||
}
|
||||
|
||||
return {
|
||||
profile: activeProfile,
|
||||
model: trimmed,
|
||||
explicitProfile: false,
|
||||
};
|
||||
}
|
||||
|
||||
function profileSupportsModel(profile: OpenAICompatProfileConfig, model: string): boolean {
|
||||
return [profile.model, profile.opusModel, profile.sonnetModel, profile.haikuModel].some(
|
||||
(candidate) => typeof candidate === 'string' && candidate === model
|
||||
);
|
||||
}
|
||||
|
||||
function estimateTokens(request: ProxyOpenAIRequest): number {
|
||||
let characters = 0;
|
||||
for (const message of request.messages) {
|
||||
if (typeof message.content === 'string') {
|
||||
characters += message.content.length;
|
||||
continue;
|
||||
}
|
||||
if (Array.isArray(message.content)) {
|
||||
for (const part of message.content) {
|
||||
characters += part.type === 'text' ? part.text.length : part.image_url.url.length;
|
||||
}
|
||||
}
|
||||
if (Array.isArray(message.tool_calls)) {
|
||||
for (const toolCall of message.tool_calls) {
|
||||
characters += toolCall.function.name.length + toolCall.function.arguments.length;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (Array.isArray(request.tools)) {
|
||||
characters += JSON.stringify(request.tools).length;
|
||||
}
|
||||
|
||||
return Math.max(1, Math.ceil(characters / 4));
|
||||
}
|
||||
|
||||
function detectScenario(
|
||||
request: ProxyOpenAIRequest,
|
||||
requestedModel: string | undefined,
|
||||
routing: OpenAICompatProxyRoutingConfig
|
||||
): { scenario?: ProxyRoutingScenario; selector?: string; estimatedTokens: number } {
|
||||
const estimatedTokens = estimateTokens(request);
|
||||
const hasWebSearchTool =
|
||||
request.tools?.some((tool) => tool.function.name === 'web_search') === true;
|
||||
const thinkingEnabled =
|
||||
request.reasoning?.enabled === true || typeof request.reasoning_effort === 'string';
|
||||
const modelId = requestedModel || '';
|
||||
const longContextThreshold = routing.longContextThreshold ?? 60_000;
|
||||
|
||||
if (hasWebSearchTool && routing.webSearch) {
|
||||
return { scenario: 'webSearch', selector: routing.webSearch, estimatedTokens };
|
||||
}
|
||||
if (thinkingEnabled && routing.think) {
|
||||
return { scenario: 'think', selector: routing.think, estimatedTokens };
|
||||
}
|
||||
if (estimatedTokens > longContextThreshold && routing.longContext) {
|
||||
return { scenario: 'longContext', selector: routing.longContext, estimatedTokens };
|
||||
}
|
||||
if (modelId.toLowerCase().includes('haiku') && routing.background) {
|
||||
return { scenario: 'background', selector: routing.background, estimatedTokens };
|
||||
}
|
||||
if (!requestedModel && routing.default) {
|
||||
return { scenario: 'default', selector: routing.default, estimatedTokens };
|
||||
}
|
||||
|
||||
return { estimatedTokens };
|
||||
}
|
||||
|
||||
export function resolveProxyRequestRoute(
|
||||
activeProfile: OpenAICompatProfileConfig,
|
||||
request: ProxyOpenAIRequest
|
||||
): ProxyRequestRoute {
|
||||
const profiles = loadOpenAICompatProfiles(activeProfile);
|
||||
const requestedModel = request.model?.trim() || undefined;
|
||||
const explicitTarget = requestedModel
|
||||
? resolveSelectorTarget(requestedModel, activeProfile, profiles)
|
||||
: null;
|
||||
const routing = loadOpenAICompatProxyRoutingConfig();
|
||||
|
||||
if (explicitTarget?.explicitProfile) {
|
||||
return {
|
||||
profile: explicitTarget.profile,
|
||||
model: explicitTarget.model,
|
||||
estimatedTokens: estimateTokens(request),
|
||||
source: 'explicit-profile',
|
||||
};
|
||||
}
|
||||
|
||||
const scenario = detectScenario(request, requestedModel, routing);
|
||||
if (scenario.selector) {
|
||||
const scenarioTarget = resolveSelectorTarget(scenario.selector, activeProfile, profiles);
|
||||
if (scenarioTarget) {
|
||||
return {
|
||||
profile: scenarioTarget.profile,
|
||||
model: scenarioTarget.model,
|
||||
scenario: scenario.scenario,
|
||||
estimatedTokens: scenario.estimatedTokens,
|
||||
source: 'scenario',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (explicitTarget && explicitTarget.profile.profileName !== activeProfile.profileName) {
|
||||
return {
|
||||
profile: explicitTarget.profile,
|
||||
model: explicitTarget.model,
|
||||
estimatedTokens: scenario.estimatedTokens,
|
||||
source: 'profile-name',
|
||||
};
|
||||
}
|
||||
|
||||
if (requestedModel) {
|
||||
const matchedProfile = profiles.find((profile) =>
|
||||
profileSupportsModel(profile, requestedModel)
|
||||
);
|
||||
if (matchedProfile) {
|
||||
return {
|
||||
profile: matchedProfile,
|
||||
model: requestedModel,
|
||||
estimatedTokens: scenario.estimatedTokens,
|
||||
source: 'profile-model-match',
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
profile: activeProfile,
|
||||
model: requestedModel,
|
||||
estimatedTokens: scenario.estimatedTokens,
|
||||
source: 'request-model',
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
profile: activeProfile,
|
||||
model: activeProfile.model,
|
||||
estimatedTokens: scenario.estimatedTokens,
|
||||
source: 'active-default',
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
import * as fs from 'fs';
|
||||
import * as yaml from 'js-yaml';
|
||||
import { getActiveConfigPath, getConfigPath } from '../utils/config-manager';
|
||||
|
||||
export interface OpenAICompatProxyRoutingConfig {
|
||||
default?: string;
|
||||
background?: string;
|
||||
think?: string;
|
||||
longContext?: string;
|
||||
webSearch?: string;
|
||||
longContextThreshold?: number;
|
||||
}
|
||||
|
||||
function readRawConfigObject(configPath: string): Record<string, unknown> | null {
|
||||
if (!fs.existsSync(configPath)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const raw = fs.readFileSync(configPath, 'utf8');
|
||||
const parsed =
|
||||
configPath.endsWith('.yaml') || configPath.endsWith('.yml') ? yaml.load(raw) : JSON.parse(raw);
|
||||
return typeof parsed === 'object' && parsed !== null ? (parsed as Record<string, unknown>) : null;
|
||||
}
|
||||
|
||||
function normalizeRoutingConfig(value: unknown): OpenAICompatProxyRoutingConfig {
|
||||
if (typeof value !== 'object' || value === null) {
|
||||
return {};
|
||||
}
|
||||
|
||||
const config = value as Record<string, unknown>;
|
||||
return {
|
||||
default:
|
||||
typeof config.default === 'string' && config.default.trim()
|
||||
? config.default.trim()
|
||||
: undefined,
|
||||
background:
|
||||
typeof config.background === 'string' && config.background.trim()
|
||||
? config.background.trim()
|
||||
: undefined,
|
||||
think:
|
||||
typeof config.think === 'string' && config.think.trim() ? config.think.trim() : undefined,
|
||||
longContext:
|
||||
typeof config.longContext === 'string' && config.longContext.trim()
|
||||
? config.longContext.trim()
|
||||
: undefined,
|
||||
webSearch:
|
||||
typeof config.webSearch === 'string' && config.webSearch.trim()
|
||||
? config.webSearch.trim()
|
||||
: undefined,
|
||||
longContextThreshold:
|
||||
typeof config.longContextThreshold === 'number' &&
|
||||
Number.isFinite(config.longContextThreshold)
|
||||
? config.longContextThreshold
|
||||
: undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export function loadOpenAICompatProxyRoutingConfig(): OpenAICompatProxyRoutingConfig {
|
||||
const activePath = getActiveConfigPath();
|
||||
const rawConfig = readRawConfigObject(activePath);
|
||||
if (rawConfig?.proxy && typeof rawConfig.proxy === 'object') {
|
||||
return normalizeRoutingConfig((rawConfig.proxy as Record<string, unknown>).routing);
|
||||
}
|
||||
|
||||
const legacyPath = getConfigPath();
|
||||
if (legacyPath !== activePath) {
|
||||
const legacyConfig = readRawConfigObject(legacyPath);
|
||||
if (legacyConfig?.proxy && typeof legacyConfig.proxy === 'object') {
|
||||
return normalizeRoutingConfig((legacyConfig.proxy as Record<string, unknown>).routing);
|
||||
}
|
||||
}
|
||||
|
||||
return {};
|
||||
}
|
||||
@@ -1,12 +1,15 @@
|
||||
import * as http from 'http';
|
||||
import type { Dispatcher } from 'undici';
|
||||
import type { OpenAICompatProfileConfig } from '../profile-router';
|
||||
import { resolveProxyRequestRoute } from '../request-router';
|
||||
import { ProxyRequestTransformer } from '../transformers/request-transformer';
|
||||
import { ProxySseStreamTransformer } from '../transformers/sse-stream-transformer';
|
||||
import { resolveOpenAIChatCompletionsUrl } from '../upstream-url';
|
||||
import { createLogger } from '../../services/logging';
|
||||
import { pipeWebResponseToNode, readJsonBody, writeJson } from './http-helpers';
|
||||
|
||||
const REQUEST_TIMEOUT_MS = 30_000;
|
||||
const REQUEST_TIMEOUT_MS = 600_000;
|
||||
const logger = createLogger('proxy:openai-compat:messages');
|
||||
|
||||
class ProxyInputError extends Error {
|
||||
constructor(message: string) {
|
||||
@@ -23,7 +26,10 @@ function buildUpstreamHeaders(profile: OpenAICompatProfileConfig): Record<string
|
||||
};
|
||||
}
|
||||
|
||||
function buildUpstreamBody(profile: OpenAICompatProfileConfig, rawBody: unknown): string {
|
||||
function buildUpstreamRequest(
|
||||
profile: OpenAICompatProfileConfig,
|
||||
rawBody: unknown
|
||||
): { body: string; route: ReturnType<typeof resolveProxyRequestRoute> } {
|
||||
let transformed;
|
||||
try {
|
||||
const transformer = new ProxyRequestTransformer();
|
||||
@@ -32,12 +38,13 @@ function buildUpstreamBody(profile: OpenAICompatProfileConfig, rawBody: unknown)
|
||||
const message = error instanceof Error ? error.message : 'Invalid Anthropic request';
|
||||
throw new ProxyInputError(message);
|
||||
}
|
||||
const route = resolveProxyRequestRoute(profile, transformed);
|
||||
const body = {
|
||||
...transformed,
|
||||
model: transformed.model || profile.model,
|
||||
model: route.model || route.profile.model,
|
||||
stream: transformed.stream === true,
|
||||
};
|
||||
return JSON.stringify(body);
|
||||
return { body: JSON.stringify(body), route };
|
||||
}
|
||||
|
||||
export function extractIncomingProxyToken(headers: http.IncomingHttpHeaders): string | null {
|
||||
@@ -88,6 +95,20 @@ function buildFetchInit(
|
||||
return init;
|
||||
}
|
||||
|
||||
function getRequestTimeoutMs(): number {
|
||||
const rawValue = process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS;
|
||||
if (!rawValue) {
|
||||
return REQUEST_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
const parsed = Number.parseInt(rawValue, 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : REQUEST_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
function formatTimeoutDuration(timeoutMs: number): string {
|
||||
return timeoutMs % 1000 === 0 ? `${timeoutMs / 1000} seconds` : `${timeoutMs}ms`;
|
||||
}
|
||||
|
||||
export async function handleProxyMessagesRequest(
|
||||
req: http.IncomingMessage,
|
||||
res: http.ServerResponse,
|
||||
@@ -98,6 +119,9 @@ export async function handleProxyMessagesRequest(
|
||||
const transformer = new ProxySseStreamTransformer();
|
||||
|
||||
if (!validateIncomingProxyAuth(req.headers, expectedAuthToken)) {
|
||||
logger.warn('auth.invalid', 'Rejected proxy message request with invalid auth token', {
|
||||
remoteAddress: req.socket.remoteAddress || null,
|
||||
});
|
||||
await pipeWebResponseToNode(
|
||||
transformer.error(401, 'authentication_error', 'Missing or invalid local proxy token'),
|
||||
res
|
||||
@@ -105,34 +129,77 @@ export async function handleProxyMessagesRequest(
|
||||
return;
|
||||
}
|
||||
|
||||
let timeoutMs = REQUEST_TIMEOUT_MS;
|
||||
try {
|
||||
const rawBody = await readJsonBody(req);
|
||||
const upstreamBody = buildUpstreamBody(profile, rawBody);
|
||||
const upstream = buildUpstreamRequest(profile, rawBody);
|
||||
logger.info('request.forward', 'Forwarding Anthropic request to OpenAI-compatible upstream', {
|
||||
profileName: upstream.route.profile.profileName,
|
||||
provider: upstream.route.profile.provider,
|
||||
baseUrl: upstream.route.profile.baseUrl,
|
||||
model: upstream.route.model || upstream.route.profile.model || null,
|
||||
routeSource: upstream.route.source,
|
||||
scenario: upstream.route.scenario || null,
|
||||
estimatedTokens: upstream.route.estimatedTokens,
|
||||
});
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
|
||||
const abortOnDisconnect = () => {
|
||||
timeoutMs = getRequestTimeoutMs();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
const abortOnDisconnect = (source: string) => {
|
||||
if (!controller.signal.aborted && !res.writableEnded) {
|
||||
logger.info(
|
||||
'request.disconnect',
|
||||
'Aborting upstream request after local client disconnect',
|
||||
{
|
||||
profileName: profile.profileName,
|
||||
source,
|
||||
}
|
||||
);
|
||||
controller.abort();
|
||||
}
|
||||
};
|
||||
|
||||
req.on('aborted', abortOnDisconnect);
|
||||
req.on('close', abortOnDisconnect);
|
||||
res.on('close', abortOnDisconnect);
|
||||
req.on('aborted', () => abortOnDisconnect('req.aborted'));
|
||||
req.on('close', () => abortOnDisconnect('req.close'));
|
||||
req.socket?.on('close', () => abortOnDisconnect('req.socket.close'));
|
||||
res.on('close', () => abortOnDisconnect('res.close'));
|
||||
res.socket?.on('close', () => abortOnDisconnect('res.socket.close'));
|
||||
const disconnectPoll = setInterval(() => {
|
||||
if (
|
||||
req.destroyed ||
|
||||
res.destroyed ||
|
||||
req.socket?.destroyed === true ||
|
||||
res.socket?.destroyed === true
|
||||
) {
|
||||
abortOnDisconnect('poll.destroyed');
|
||||
}
|
||||
}, 50);
|
||||
|
||||
try {
|
||||
const upstreamResponse = await fetch(
|
||||
resolveOpenAIChatCompletionsUrl(profile.baseUrl),
|
||||
buildFetchInit(profile, upstreamBody, controller.signal, insecureDispatcher)
|
||||
resolveOpenAIChatCompletionsUrl(upstream.route.profile.baseUrl),
|
||||
buildFetchInit(upstream.route.profile, upstream.body, controller.signal, insecureDispatcher)
|
||||
);
|
||||
clearTimeout(timeout);
|
||||
clearInterval(disconnectPoll);
|
||||
logger.info('response.received', 'Received upstream response', {
|
||||
profileName: profile.profileName,
|
||||
routedProfileName: upstream.route.profile.profileName,
|
||||
status: upstreamResponse.status,
|
||||
});
|
||||
const response = await transformer.transform(upstreamResponse);
|
||||
await pipeWebResponseToNode(response, res);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
clearInterval(disconnectPoll);
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Unknown proxy error';
|
||||
logger.error('request.failed', 'Proxy message request failed', {
|
||||
profileName: profile.profileName,
|
||||
error: message,
|
||||
abort: error instanceof Error && error.name === 'AbortError',
|
||||
});
|
||||
const status =
|
||||
error instanceof Error && error.name === 'AbortError'
|
||||
? 502
|
||||
@@ -149,7 +216,7 @@ export async function handleProxyMessagesRequest(
|
||||
status,
|
||||
type,
|
||||
error instanceof Error && error.name === 'AbortError'
|
||||
? 'The upstream provider did not respond within 30 seconds'
|
||||
? `The upstream provider did not respond within ${formatTimeoutDuration(timeoutMs)}`
|
||||
: message
|
||||
),
|
||||
res
|
||||
|
||||
@@ -2,6 +2,7 @@ import * as http from 'http';
|
||||
import { Agent } from 'undici';
|
||||
import type { OpenAICompatProfileConfig } from '../profile-router';
|
||||
import { OPENAI_COMPAT_PROXY_SERVICE_NAME } from '../proxy-daemon-paths';
|
||||
import { createLogger } from '../../services/logging';
|
||||
import {
|
||||
handleProxyMessagesRequest,
|
||||
handleProxyModelsRequest,
|
||||
@@ -11,12 +12,19 @@ import { writeJson } from './http-helpers';
|
||||
|
||||
export interface OpenAICompatProxyServerOptions {
|
||||
profile: OpenAICompatProfileConfig;
|
||||
host?: string;
|
||||
port: number;
|
||||
authToken: string;
|
||||
insecure?: boolean;
|
||||
}
|
||||
|
||||
export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOptions): http.Server {
|
||||
const host = options.host?.trim() || '127.0.0.1';
|
||||
const logger = createLogger('proxy:openai-compat', {
|
||||
profileName: options.profile.profileName,
|
||||
host,
|
||||
port: options.port,
|
||||
});
|
||||
const insecureDispatcher = options.insecure
|
||||
? new Agent({ connect: { rejectUnauthorized: false } })
|
||||
: undefined;
|
||||
@@ -31,12 +39,31 @@ export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOpt
|
||||
writeJson(res, 200, {
|
||||
ok: true,
|
||||
service: OPENAI_COMPAT_PROXY_SERVICE_NAME,
|
||||
host,
|
||||
profile: options.profile.profileName,
|
||||
port: options.port,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (method === 'GET' && pathname === '/') {
|
||||
writeJson(res, 200, {
|
||||
ok: true,
|
||||
service: OPENAI_COMPAT_PROXY_SERVICE_NAME,
|
||||
bind: {
|
||||
host,
|
||||
port: options.port,
|
||||
},
|
||||
profile: {
|
||||
name: options.profile.profileName,
|
||||
provider: options.profile.provider,
|
||||
model: options.profile.model || null,
|
||||
},
|
||||
endpoints: ['/health', '/v1/messages', '/v1/models'],
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (method === 'GET' && pathname === '/v1/models') {
|
||||
if (!validateIncomingProxyAuth(req.headers, options.authToken)) {
|
||||
writeJson(res, 401, {
|
||||
@@ -63,13 +90,21 @@ export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOpt
|
||||
return;
|
||||
}
|
||||
|
||||
logger.warn('http.not_found', 'Rejected unknown proxy route', {
|
||||
method,
|
||||
pathname,
|
||||
});
|
||||
writeJson(res, 404, { error: 'Not found' });
|
||||
});
|
||||
|
||||
logger.info('server.start', 'OpenAI-compatible proxy server listening', {
|
||||
baseUrl: `http://${host}:${options.port}`,
|
||||
});
|
||||
server.on('close', () => {
|
||||
logger.info('server.stop', 'OpenAI-compatible proxy server stopped');
|
||||
void insecureDispatcher?.close();
|
||||
});
|
||||
|
||||
server.listen(options.port, '127.0.0.1');
|
||||
server.listen(options.port, host);
|
||||
return server;
|
||||
}
|
||||
|
||||
@@ -1,20 +1,97 @@
|
||||
import { translateAnthropicRequest } from '../../cursor/cursor-anthropic-translator';
|
||||
interface AnthropicThinking {
|
||||
type?: 'enabled' | 'disabled' | string;
|
||||
budget_tokens?: number;
|
||||
}
|
||||
|
||||
interface AnthropicTextBlock {
|
||||
type: 'text';
|
||||
text?: string;
|
||||
}
|
||||
|
||||
interface AnthropicImageBlock {
|
||||
type: 'image';
|
||||
source?: {
|
||||
type?: string;
|
||||
media_type?: string;
|
||||
data?: string;
|
||||
};
|
||||
}
|
||||
|
||||
interface AnthropicToolUseBlock {
|
||||
type: 'tool_use';
|
||||
id?: string;
|
||||
name?: string;
|
||||
input?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
interface AnthropicToolResultBlock {
|
||||
type: 'tool_result';
|
||||
tool_use_id?: string;
|
||||
content?: unknown;
|
||||
}
|
||||
|
||||
type AnthropicContentBlock =
|
||||
| AnthropicTextBlock
|
||||
| AnthropicImageBlock
|
||||
| AnthropicToolUseBlock
|
||||
| AnthropicToolResultBlock
|
||||
| { type: string; [key: string]: unknown };
|
||||
|
||||
interface AnthropicMessage {
|
||||
role?: 'user' | 'assistant' | string;
|
||||
content?: string | AnthropicContentBlock[];
|
||||
}
|
||||
|
||||
interface AnthropicProxyRequestShape {
|
||||
model?: unknown;
|
||||
system?: unknown;
|
||||
messages?: unknown;
|
||||
max_tokens?: unknown;
|
||||
temperature?: unknown;
|
||||
top_p?: unknown;
|
||||
stop_sequences?: unknown;
|
||||
metadata?: unknown;
|
||||
tools?: unknown;
|
||||
stream?: unknown;
|
||||
thinking?: AnthropicThinking;
|
||||
}
|
||||
|
||||
export interface ProxyOpenAIRequest extends ReturnType<typeof translateAnthropicRequest> {
|
||||
max_tokens?: number;
|
||||
temperature?: number;
|
||||
top_p?: number;
|
||||
stop?: string[];
|
||||
metadata?: Record<string, unknown>;
|
||||
interface OpenAITextPart {
|
||||
type: 'text';
|
||||
text: string;
|
||||
}
|
||||
|
||||
interface OpenAIImagePart {
|
||||
type: 'image_url';
|
||||
image_url: {
|
||||
url: string;
|
||||
};
|
||||
}
|
||||
|
||||
type OpenAIContentPart = OpenAITextPart | OpenAIImagePart;
|
||||
|
||||
interface OpenAIMessage {
|
||||
role: 'system' | 'user' | 'assistant' | 'tool';
|
||||
content: string | OpenAIContentPart[] | null;
|
||||
tool_call_id?: string;
|
||||
tool_calls?: Array<{
|
||||
id: string;
|
||||
type: 'function';
|
||||
function: {
|
||||
name: string;
|
||||
arguments: string;
|
||||
};
|
||||
}>;
|
||||
}
|
||||
|
||||
export interface ProxyOpenAIRequest {
|
||||
model?: string;
|
||||
stream: boolean;
|
||||
reasoning_effort?: string;
|
||||
reasoning?: {
|
||||
enabled: boolean;
|
||||
effort: string;
|
||||
};
|
||||
tools?: Array<{
|
||||
type: 'function';
|
||||
function: {
|
||||
@@ -23,6 +100,23 @@ export interface ProxyOpenAIRequest extends ReturnType<typeof translateAnthropic
|
||||
parameters: Record<string, unknown>;
|
||||
};
|
||||
}>;
|
||||
messages: OpenAIMessage[];
|
||||
max_tokens?: number;
|
||||
temperature?: number;
|
||||
top_p?: number;
|
||||
stop?: string[];
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
const TOOL_RESULT_SERIALIZATION_FALLBACK = '[unserializable content]';
|
||||
const TOOL_USE_ARGUMENTS_FALLBACK = '{}';
|
||||
|
||||
function assertObject(value: unknown, label: string): Record<string, unknown> {
|
||||
if (typeof value !== 'object' || value === null) {
|
||||
throw new Error(`${label} must be an object`);
|
||||
}
|
||||
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function asNumber(value: unknown): number | undefined {
|
||||
@@ -46,6 +140,90 @@ function asMetadata(value: unknown): Record<string, unknown> | undefined {
|
||||
: undefined;
|
||||
}
|
||||
|
||||
function safeJsonStringify(value: unknown, fallback: string): string {
|
||||
try {
|
||||
const serialized = JSON.stringify(value);
|
||||
return typeof serialized === 'string' ? serialized : fallback;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
function flattenTextContent(content: unknown, label: string): string {
|
||||
if (typeof content === 'string') {
|
||||
return content;
|
||||
}
|
||||
if (!Array.isArray(content)) {
|
||||
throw new Error(`${label} must be a string or content block array`);
|
||||
}
|
||||
|
||||
return content
|
||||
.map((block, index) => {
|
||||
const parsed = assertObject(block, `${label}[${index}]`);
|
||||
if (parsed.type !== 'text') {
|
||||
throw new Error(`${label}[${index}].type "${String(parsed.type)}" is not supported`);
|
||||
}
|
||||
return typeof parsed.text === 'string' ? parsed.text : '';
|
||||
})
|
||||
.join('\n');
|
||||
}
|
||||
|
||||
function toToolResultContent(content: unknown, label: string): string {
|
||||
if (content === undefined) {
|
||||
return '';
|
||||
}
|
||||
if (typeof content === 'string') {
|
||||
return content;
|
||||
}
|
||||
if (Array.isArray(content)) {
|
||||
return flattenTextContent(content, label);
|
||||
}
|
||||
return safeJsonStringify(content, TOOL_RESULT_SERIALIZATION_FALLBACK);
|
||||
}
|
||||
|
||||
function createFallbackToolId(messageIndex: number, blockIndex: number): string {
|
||||
return `toolu_proxy_fallback_${messageIndex}_${blockIndex}`;
|
||||
}
|
||||
|
||||
function toImagePart(block: AnthropicImageBlock, label: string): OpenAIImagePart {
|
||||
const source = block.source;
|
||||
if (!source || source.type !== 'base64' || !source.media_type || !source.data) {
|
||||
throw new Error(`${label}.source must be a base64 image payload`);
|
||||
}
|
||||
|
||||
return {
|
||||
type: 'image_url',
|
||||
image_url: {
|
||||
url: `data:${source.media_type};base64,${source.data}`,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function isImageBlock(block: AnthropicContentBlock): block is AnthropicImageBlock {
|
||||
return block.type === 'image';
|
||||
}
|
||||
|
||||
function isToolUseBlock(block: AnthropicContentBlock): block is AnthropicToolUseBlock {
|
||||
return block.type === 'tool_use';
|
||||
}
|
||||
|
||||
function isToolResultBlock(block: AnthropicContentBlock): block is AnthropicToolResultBlock {
|
||||
return block.type === 'tool_result';
|
||||
}
|
||||
|
||||
function flushUserContent(messages: OpenAIMessage[], parts: OpenAIContentPart[]): void {
|
||||
if (parts.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const onlyText = parts.every((part) => part.type === 'text');
|
||||
messages.push({
|
||||
role: 'user',
|
||||
content: onlyText ? parts.map((part) => (part as OpenAITextPart).text).join('\n') : [...parts],
|
||||
});
|
||||
parts.length = 0;
|
||||
}
|
||||
|
||||
function transformTools(value: unknown): ProxyOpenAIRequest['tools'] {
|
||||
if (!Array.isArray(value)) {
|
||||
return undefined;
|
||||
@@ -71,19 +249,179 @@ function transformTools(value: unknown): ProxyOpenAIRequest['tools'] {
|
||||
return tools.length > 0 ? tools : undefined;
|
||||
}
|
||||
|
||||
function mapThinkingToReasoning(
|
||||
thinking: AnthropicThinking | undefined
|
||||
): Pick<ProxyOpenAIRequest, 'reasoning' | 'reasoning_effort'> {
|
||||
if (!thinking || thinking.type === 'disabled') {
|
||||
return {};
|
||||
}
|
||||
|
||||
if (thinking.type !== 'enabled') {
|
||||
throw new Error('thinking.type must be "enabled" or "disabled"');
|
||||
}
|
||||
|
||||
const effort =
|
||||
typeof thinking.budget_tokens === 'number' && thinking.budget_tokens >= 8192
|
||||
? 'high'
|
||||
: 'medium';
|
||||
|
||||
return {
|
||||
reasoning_effort: effort,
|
||||
reasoning: {
|
||||
enabled: true,
|
||||
effort,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function transformMessages(messagesValue: unknown): OpenAIMessage[] {
|
||||
if (!Array.isArray(messagesValue)) {
|
||||
throw new Error('messages must be an array');
|
||||
}
|
||||
|
||||
const translatedMessages: OpenAIMessage[] = [];
|
||||
|
||||
messagesValue.forEach((message, messageIndex) => {
|
||||
const parsedMessage = assertObject(message, `messages[${messageIndex}]`) as AnthropicMessage;
|
||||
const role = parsedMessage.role;
|
||||
if (role !== 'user' && role !== 'assistant') {
|
||||
throw new Error(`messages[${messageIndex}].role must be "user" or "assistant"`);
|
||||
}
|
||||
|
||||
const content = parsedMessage.content;
|
||||
if (typeof content === 'string') {
|
||||
translatedMessages.push({ role, content });
|
||||
return;
|
||||
}
|
||||
|
||||
if (!Array.isArray(content)) {
|
||||
throw new Error(`messages[${messageIndex}].content must be a string or array`);
|
||||
}
|
||||
|
||||
const userParts: OpenAIContentPart[] = [];
|
||||
const assistantTextParts: string[] = [];
|
||||
const toolCalls: NonNullable<OpenAIMessage['tool_calls']> = [];
|
||||
let sawToolResult = false;
|
||||
|
||||
content.forEach((block, blockIndex) => {
|
||||
const parsed = assertObject(
|
||||
block,
|
||||
`messages[${messageIndex}].content[${blockIndex}]`
|
||||
) as AnthropicContentBlock;
|
||||
|
||||
if (parsed.type === 'text') {
|
||||
const text = typeof parsed.text === 'string' ? parsed.text : '';
|
||||
if (role === 'user') {
|
||||
userParts.push({ type: 'text', text });
|
||||
} else {
|
||||
assistantTextParts.push(text);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (isImageBlock(parsed)) {
|
||||
if (role !== 'user') {
|
||||
throw new Error(
|
||||
`messages[${messageIndex}].content[${blockIndex}] image requires user role`
|
||||
);
|
||||
}
|
||||
userParts.push(toImagePart(parsed, `messages[${messageIndex}].content[${blockIndex}]`));
|
||||
return;
|
||||
}
|
||||
|
||||
if (isToolUseBlock(parsed)) {
|
||||
if (role !== 'assistant') {
|
||||
throw new Error(
|
||||
`messages[${messageIndex}].content[${blockIndex}] tool_use requires assistant role`
|
||||
);
|
||||
}
|
||||
toolCalls.push({
|
||||
id:
|
||||
typeof parsed.id === 'string' && parsed.id.length > 0
|
||||
? parsed.id
|
||||
: createFallbackToolId(messageIndex, blockIndex),
|
||||
type: 'function',
|
||||
function: {
|
||||
name: typeof parsed.name === 'string' ? parsed.name : 'tool',
|
||||
arguments: safeJsonStringify(parsed.input ?? {}, TOOL_USE_ARGUMENTS_FALLBACK),
|
||||
},
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (isToolResultBlock(parsed)) {
|
||||
if (role !== 'user') {
|
||||
throw new Error(
|
||||
`messages[${messageIndex}].content[${blockIndex}] tool_result requires user role`
|
||||
);
|
||||
}
|
||||
if (typeof parsed.tool_use_id !== 'string' || parsed.tool_use_id.trim().length === 0) {
|
||||
throw new Error(
|
||||
`messages[${messageIndex}].content[${blockIndex}].tool_use_id must be a non-empty string`
|
||||
);
|
||||
}
|
||||
sawToolResult = true;
|
||||
flushUserContent(translatedMessages, userParts);
|
||||
translatedMessages.push({
|
||||
role: 'tool',
|
||||
tool_call_id: parsed.tool_use_id,
|
||||
content: toToolResultContent(
|
||||
parsed.content,
|
||||
`messages[${messageIndex}].content[${blockIndex}].content`
|
||||
),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`messages[${messageIndex}].content[${blockIndex}].type "${String(parsed.type)}" is not supported`
|
||||
);
|
||||
});
|
||||
|
||||
if (role === 'assistant') {
|
||||
translatedMessages.push({
|
||||
role: 'assistant',
|
||||
content: assistantTextParts.join('\n'),
|
||||
tool_calls: toolCalls.length > 0 ? toolCalls : undefined,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (userParts.length > 0 || !sawToolResult) {
|
||||
flushUserContent(translatedMessages, userParts);
|
||||
}
|
||||
});
|
||||
|
||||
return translatedMessages;
|
||||
}
|
||||
|
||||
export class ProxyRequestTransformer {
|
||||
transform(raw: unknown): ProxyOpenAIRequest {
|
||||
const translated = translateAnthropicRequest(raw);
|
||||
const source = (raw || {}) as AnthropicProxyRequestShape;
|
||||
const source = assertObject(raw || {}, 'request') as AnthropicProxyRequestShape;
|
||||
const messages = transformMessages(source.messages);
|
||||
const system = source.system;
|
||||
const allMessages =
|
||||
system !== undefined
|
||||
? [
|
||||
{ role: 'system', content: flattenTextContent(system, 'system') } as OpenAIMessage,
|
||||
...messages,
|
||||
]
|
||||
: messages;
|
||||
|
||||
return {
|
||||
...translated,
|
||||
model:
|
||||
typeof source.model === 'string' && source.model.trim().length > 0
|
||||
? source.model.trim()
|
||||
: undefined,
|
||||
stream: source.stream === true,
|
||||
messages: allMessages,
|
||||
max_tokens: asNumber(source.max_tokens),
|
||||
temperature: asNumber(source.temperature),
|
||||
top_p: asNumber(source.top_p),
|
||||
stop: asStringArray(source.stop_sequences),
|
||||
metadata: asMetadata(source.metadata),
|
||||
tools: transformTools(source.tools),
|
||||
...mapThinkingToReasoning(source.thinking),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,6 +40,19 @@ export interface CLIProxyVariantsConfig {
|
||||
[profileName: string]: CLIProxyVariantConfig;
|
||||
}
|
||||
|
||||
export interface OpenAICompatProxyRoutingConfig {
|
||||
default?: string;
|
||||
background?: string;
|
||||
think?: string;
|
||||
longContext?: string;
|
||||
webSearch?: string;
|
||||
longContextThreshold?: number;
|
||||
}
|
||||
|
||||
export interface OpenAICompatProxyConfig {
|
||||
routing?: OpenAICompatProxyRoutingConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* Main CCS configuration
|
||||
* Located at: ~/.ccs/config.json
|
||||
@@ -51,6 +64,8 @@ export interface Config {
|
||||
profile_targets?: Record<string, TargetType>;
|
||||
/** User-defined CLIProxy profile variants (optional) */
|
||||
cliproxy?: CLIProxyVariantsConfig;
|
||||
/** OpenAI-compatible local proxy configuration (optional) */
|
||||
proxy?: OpenAICompatProxyConfig;
|
||||
/** Legacy continuity inheritance mapping (profile -> source account) */
|
||||
continuity_inherit_from_account?: Record<string, string>;
|
||||
}
|
||||
|
||||
@@ -14,6 +14,8 @@ export type {
|
||||
ProfilesRegistry,
|
||||
CLIProxyVariantConfig,
|
||||
CLIProxyVariantsConfig,
|
||||
OpenAICompatProxyConfig,
|
||||
OpenAICompatProxyRoutingConfig,
|
||||
} from './config';
|
||||
export { isConfig, isSettings } from './config';
|
||||
|
||||
|
||||
@@ -673,6 +673,10 @@ router.put('/', (req: Request, res: Response): void => {
|
||||
currentConfig.cliproxy = mergeCliproxyConfig(currentConfig, config);
|
||||
}
|
||||
|
||||
if (config.proxy !== undefined) {
|
||||
currentConfig.proxy = config.proxy;
|
||||
}
|
||||
|
||||
if (config.preferences !== undefined) {
|
||||
currentConfig.preferences = config.preferences;
|
||||
}
|
||||
|
||||
@@ -68,20 +68,42 @@ describe('proxy command e2e', () => {
|
||||
'utf8'
|
||||
);
|
||||
|
||||
const started = runCli(['proxy', 'start', 'hf', '--port', String(port)]);
|
||||
const started = runCli(['proxy', 'start', 'hf', '--port', String(port), '--host', '127.0.0.1']);
|
||||
expect(started.status).toBe(0);
|
||||
|
||||
const status = runCli(['proxy', 'status']);
|
||||
expect(status.stdout).toContain(`Proxy running on port ${port}`);
|
||||
expect(status.stdout).toContain('Host: 127.0.0.1');
|
||||
expect(status.stdout).toContain('Profile: hf');
|
||||
|
||||
const activate = runCli(['proxy', 'activate', '--shell', 'bash']);
|
||||
expect(activate.stdout).toContain(`export ANTHROPIC_BASE_URL='http://127.0.0.1:${port}'`);
|
||||
expect(activate.stdout).toMatch(/export ANTHROPIC_AUTH_TOKEN='[a-f0-9]{48}'/);
|
||||
expect(activate.stdout).toContain("export DISABLE_TELEMETRY='1'");
|
||||
expect(activate.stdout).toContain("export DISABLE_COST_WARNINGS='1'");
|
||||
expect(activate.stdout).toContain("export API_TIMEOUT_MS='600000'");
|
||||
expect(activate.stdout).toContain("export NO_PROXY='127.0.0.1,localhost'");
|
||||
|
||||
const activateFish = runCli(['proxy', 'activate', '--fish']);
|
||||
expect(activateFish.stdout).toContain(`set -gx ANTHROPIC_BASE_URL 'http://127.0.0.1:${port}'`);
|
||||
|
||||
const health = await fetch(`http://127.0.0.1:${port}/health`);
|
||||
expect(health.status).toBe(200);
|
||||
|
||||
const info = await fetch(`http://127.0.0.1:${port}/`);
|
||||
expect(info.status).toBe(200);
|
||||
await expect(info.json()).resolves.toMatchObject({
|
||||
ok: true,
|
||||
service: 'ccs-openai-compat-proxy',
|
||||
bind: {
|
||||
host: '127.0.0.1',
|
||||
port,
|
||||
},
|
||||
profile: {
|
||||
name: 'hf',
|
||||
},
|
||||
});
|
||||
|
||||
const stopped = runCli(['proxy', 'stop']);
|
||||
expect(stopped.status).toBe(0);
|
||||
}, 35000);
|
||||
|
||||
@@ -0,0 +1,259 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
||||
import getPort from 'get-port';
|
||||
import * as fs from 'fs';
|
||||
import * as http from 'http';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { startOpenAICompatProxyServer } from '../../../src/proxy/server/proxy-server';
|
||||
import type { OpenAICompatProfileConfig } from '../../../src/proxy/profile-router';
|
||||
|
||||
let proxyServer: http.Server;
|
||||
let upstreamServer: http.Server;
|
||||
let upstreamSockets = new Set<import('net').Socket>();
|
||||
let upstreamPort: number;
|
||||
let proxyPort: number;
|
||||
let tempDir: string;
|
||||
let originalTimeoutEnv: string | undefined;
|
||||
let originalCcsHome: string | undefined;
|
||||
|
||||
async function startUpstream(
|
||||
handler: (req: http.IncomingMessage, res: http.ServerResponse) => Promise<void> | void
|
||||
): Promise<void> {
|
||||
upstreamServer = http.createServer((req, res) => {
|
||||
void Promise.resolve(handler(req, res));
|
||||
});
|
||||
upstreamServer.on('connection', (socket) => {
|
||||
upstreamSockets.add(socket);
|
||||
socket.on('close', () => {
|
||||
upstreamSockets.delete(socket);
|
||||
});
|
||||
});
|
||||
await new Promise<void>((resolve) =>
|
||||
upstreamServer.listen(upstreamPort, '127.0.0.1', () => resolve())
|
||||
);
|
||||
}
|
||||
|
||||
async function requestProxy(payload: unknown, signal?: AbortSignal): Promise<Response> {
|
||||
return fetch(`http://127.0.0.1:${proxyPort}/v1/messages`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-api-key': 'test-proxy-token',
|
||||
},
|
||||
body: JSON.stringify(payload),
|
||||
signal,
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
upstreamPort = await getPort();
|
||||
proxyPort = await getPort();
|
||||
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-edge-'));
|
||||
originalTimeoutEnv = process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS;
|
||||
originalCcsHome = process.env.CCS_HOME;
|
||||
process.env.CCS_HOME = tempDir;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
if (originalTimeoutEnv !== undefined) {
|
||||
process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS = originalTimeoutEnv;
|
||||
} else {
|
||||
delete process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS;
|
||||
}
|
||||
if (originalCcsHome !== undefined) {
|
||||
process.env.CCS_HOME = originalCcsHome;
|
||||
} else {
|
||||
delete process.env.CCS_HOME;
|
||||
}
|
||||
|
||||
if (proxyServer) {
|
||||
await new Promise<void>((resolve) => proxyServer.close(() => resolve()));
|
||||
}
|
||||
if (upstreamServer) {
|
||||
for (const socket of upstreamSockets) {
|
||||
socket.destroy();
|
||||
}
|
||||
upstreamSockets = new Set();
|
||||
await new Promise<void>((resolve) => upstreamServer.close(() => resolve()));
|
||||
}
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('openai proxy message edge cases', () => {
|
||||
async function startProxyWithHandler(
|
||||
handler: (req: http.IncomingMessage, res: http.ServerResponse) => Promise<void> | void
|
||||
) {
|
||||
await startUpstream(handler);
|
||||
const profile: OpenAICompatProfileConfig = {
|
||||
profileName: 'hf',
|
||||
settingsPath: '/tmp/hf.settings.json',
|
||||
baseUrl: `http://127.0.0.1:${upstreamPort}`,
|
||||
apiKey: 'hf_token',
|
||||
provider: 'generic-chat-completion-api',
|
||||
model: 'hf-model',
|
||||
};
|
||||
proxyServer = startOpenAICompatProxyServer({
|
||||
profile,
|
||||
port: proxyPort,
|
||||
authToken: 'test-proxy-token',
|
||||
});
|
||||
}
|
||||
|
||||
it('preserves rate-limit errors from the upstream provider', async () => {
|
||||
await startProxyWithHandler((_req, res) => {
|
||||
res.writeHead(429, {
|
||||
'Content-Type': 'application/json',
|
||||
'Retry-After': '9',
|
||||
});
|
||||
res.end(JSON.stringify({ error: { message: 'rate limited' } }));
|
||||
});
|
||||
|
||||
const response = await requestProxy({
|
||||
model: 'hf-model',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
|
||||
expect(response.status).toBe(429);
|
||||
expect(response.headers.get('retry-after')).toBe('9');
|
||||
await expect(response.json()).resolves.toMatchObject({
|
||||
type: 'error',
|
||||
error: {
|
||||
type: 'rate_limit_error',
|
||||
message: 'rate limited',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it('returns api_error when the upstream JSON response has no usable choices', async () => {
|
||||
await startProxyWithHandler((_req, res) => {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ id: 'chatcmpl_empty', choices: [] }));
|
||||
});
|
||||
|
||||
const response = await requestProxy({
|
||||
model: 'hf-model',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
|
||||
expect(response.status).toBe(502);
|
||||
await expect(response.json()).resolves.toMatchObject({
|
||||
type: 'error',
|
||||
error: {
|
||||
type: 'api_error',
|
||||
message: 'Failed to translate Cursor JSON response',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it('streams thinking deltas and chunked tool-call arguments back as Anthropic SSE', async () => {
|
||||
await startProxyWithHandler((_req, res) => {
|
||||
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
|
||||
res.write(
|
||||
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"role":"assistant"}}]}\n\n'
|
||||
);
|
||||
res.write(
|
||||
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"reasoning_content":"Need to search first."}}]}\n\n'
|
||||
);
|
||||
res.write(
|
||||
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"search","arguments":"{\\"q\\":\\""}}]}}]}\n\n'
|
||||
);
|
||||
res.write(
|
||||
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"docs\\"}"}}]}}]}\n\n'
|
||||
);
|
||||
res.write(
|
||||
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":8,"completion_tokens":6}}\n\n'
|
||||
);
|
||||
res.end('data: [DONE]\n\n');
|
||||
});
|
||||
|
||||
const response = await requestProxy({
|
||||
model: 'hf-model',
|
||||
stream: true,
|
||||
messages: [{ role: 'user', content: 'search docs' }],
|
||||
});
|
||||
|
||||
const body = await response.text();
|
||||
expect(response.status).toBe(200);
|
||||
expect(body).toContain('"type":"thinking_delta"');
|
||||
expect(body).toContain('"type":"tool_use"');
|
||||
expect(body).toContain('"partial_json":"{\\"q\\":\\""');
|
||||
expect(body).toContain('"partial_json":"docs\\"}"');
|
||||
expect(body).toContain('event: message_stop');
|
||||
});
|
||||
|
||||
it('returns a timeout error when the upstream does not respond in time', async () => {
|
||||
process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS = '50';
|
||||
await startProxyWithHandler(async (req, _res) => {
|
||||
await new Promise<void>((resolve) => req.on('close', () => resolve()));
|
||||
});
|
||||
|
||||
const response = await requestProxy({
|
||||
model: 'hf-model',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
|
||||
expect(response.status).toBe(502);
|
||||
await expect(response.json()).resolves.toMatchObject({
|
||||
type: 'error',
|
||||
error: {
|
||||
type: 'api_error',
|
||||
message: 'The upstream provider did not respond within 50ms',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it('aborts the upstream request when the client disconnects mid-flight', async () => {
|
||||
await startProxyWithHandler(() => {});
|
||||
|
||||
await new Promise<void>((resolve) => {
|
||||
const request = http.request(
|
||||
{
|
||||
hostname: '127.0.0.1',
|
||||
port: proxyPort,
|
||||
path: '/v1/messages',
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-api-key': 'test-proxy-token',
|
||||
},
|
||||
},
|
||||
() => resolve()
|
||||
);
|
||||
|
||||
request.write(
|
||||
JSON.stringify({
|
||||
model: 'hf-model',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
})
|
||||
);
|
||||
request.end();
|
||||
|
||||
setTimeout(() => {
|
||||
request.destroy(new Error('client aborted'));
|
||||
resolve();
|
||||
}, 50);
|
||||
});
|
||||
|
||||
const logPath = path.join(tempDir, '.ccs', 'logs', 'current.jsonl');
|
||||
await Promise.race([
|
||||
new Promise<void>((resolve, reject) => {
|
||||
const startedAt = Date.now();
|
||||
const timer = setInterval(() => {
|
||||
if (fs.existsSync(logPath)) {
|
||||
const content = fs.readFileSync(logPath, 'utf8');
|
||||
if (content.includes('"event":"request.disconnect"')) {
|
||||
clearInterval(timer);
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (Date.now() - startedAt > 1500) {
|
||||
clearInterval(timer);
|
||||
reject(new Error('proxy did not log disconnect cleanup'));
|
||||
}
|
||||
}, 50);
|
||||
}),
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -173,16 +173,44 @@ describe('openai proxy messages endpoint', () => {
|
||||
expect(body.error?.type).toBe('authentication_error');
|
||||
});
|
||||
|
||||
it('returns invalid_request_error for unsupported Anthropic content blocks', async () => {
|
||||
it('translates supported Anthropic image blocks into OpenAI image_url content', async () => {
|
||||
const response = await requestProxy({
|
||||
model: 'hf-model',
|
||||
messages: [{ role: 'user', content: [{ type: 'image' }] }],
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'text', text: 'Describe this' },
|
||||
{
|
||||
type: 'image',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: 'image/png',
|
||||
data: 'ZmFrZQ==',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
const body = (await response.json()) as { error?: { type?: string; message?: string } };
|
||||
|
||||
expect(response.status).toBe(400);
|
||||
expect(body.error?.type).toBe('invalid_request_error');
|
||||
expect(body.error?.message).toContain('is not supported');
|
||||
expect(response.status).toBe(200);
|
||||
const parsedUpstream = upstreamBody as {
|
||||
messages?: Array<{
|
||||
role: string;
|
||||
content: Array<{ type: string; text?: string; image_url?: { url?: string } }>;
|
||||
}>;
|
||||
};
|
||||
expect(parsedUpstream.messages?.[0]).toEqual({
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'text', text: 'Describe this' },
|
||||
{
|
||||
type: 'image_url',
|
||||
image_url: { url: 'data:image/png;base64,ZmFrZQ==' },
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it('accepts query strings and trailing slashes on the messages route', async () => {
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
||||
import getPort from 'get-port';
|
||||
import * as fs from 'fs';
|
||||
import * as http from 'http';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { startOpenAICompatProxyServer } from '../../../src/proxy/server/proxy-server';
|
||||
import type { OpenAICompatProfileConfig } from '../../../src/proxy/profile-router';
|
||||
|
||||
let originalCcsHome: string | undefined;
|
||||
let tempDir: string;
|
||||
let proxyServer: http.Server;
|
||||
let upstreamServers: http.Server[] = [];
|
||||
let proxyPort: number;
|
||||
|
||||
function startMockUpstream(
|
||||
port: number,
|
||||
hitLabel: string,
|
||||
hits: string[],
|
||||
bodies: Array<{ label: string; body: unknown }>
|
||||
): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const server = http.createServer(async (req, res) => {
|
||||
let body = '';
|
||||
for await (const chunk of req) {
|
||||
body += chunk.toString();
|
||||
}
|
||||
hits.push(hitLabel);
|
||||
bodies.push({ label: hitLabel, body: JSON.parse(body) });
|
||||
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
id: `chatcmpl_${hitLabel}`,
|
||||
model: hitLabel,
|
||||
choices: [
|
||||
{
|
||||
index: 0,
|
||||
message: { role: 'assistant', content: `Reply from ${hitLabel}` },
|
||||
finish_reason: 'stop',
|
||||
},
|
||||
],
|
||||
usage: { prompt_tokens: 2, completion_tokens: 3 },
|
||||
})
|
||||
);
|
||||
});
|
||||
upstreamServers.push(server);
|
||||
server.listen(port, '127.0.0.1', () => resolve());
|
||||
});
|
||||
}
|
||||
|
||||
function writeSettings(profileName: string, env: Record<string, string>): string {
|
||||
const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`);
|
||||
fs.writeFileSync(settingsPath, JSON.stringify({ env }, null, 2), 'utf8');
|
||||
return settingsPath;
|
||||
}
|
||||
|
||||
async function requestProxy(payload: unknown): Promise<Response> {
|
||||
return fetch(`http://127.0.0.1:${proxyPort}/v1/messages`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-api-key': 'test-proxy-token',
|
||||
},
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
originalCcsHome = process.env.CCS_HOME;
|
||||
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-routing-'));
|
||||
fs.mkdirSync(path.join(tempDir, '.ccs'), { recursive: true });
|
||||
process.env.CCS_HOME = tempDir;
|
||||
proxyPort = await getPort();
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(
|
||||
upstreamServers.map(
|
||||
(server) =>
|
||||
new Promise<void>((resolve) => {
|
||||
server.close(() => resolve());
|
||||
})
|
||||
)
|
||||
);
|
||||
upstreamServers = [];
|
||||
if (proxyServer) {
|
||||
await new Promise<void>((resolve) => proxyServer.close(() => resolve()));
|
||||
}
|
||||
if (originalCcsHome !== undefined) {
|
||||
process.env.CCS_HOME = originalCcsHome;
|
||||
} else {
|
||||
delete process.env.CCS_HOME;
|
||||
}
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('openai proxy request routing', () => {
|
||||
it('routes explicit profile:model selectors to the matching upstream profile', async () => {
|
||||
const primaryPort = await getPort();
|
||||
const secondaryPort = await getPort();
|
||||
const hits: string[] = [];
|
||||
const bodies: Array<{ label: string; body: unknown }> = [];
|
||||
await startMockUpstream(primaryPort, 'primary', hits, bodies);
|
||||
await startMockUpstream(secondaryPort, 'secondary', hits, bodies);
|
||||
|
||||
const primarySettings = writeSettings('hf', {
|
||||
ANTHROPIC_BASE_URL: `http://127.0.0.1:${primaryPort}`,
|
||||
ANTHROPIC_AUTH_TOKEN: 'hf_token',
|
||||
ANTHROPIC_MODEL: 'hf-default',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
});
|
||||
const secondarySettings = writeSettings('deepseek', {
|
||||
ANTHROPIC_BASE_URL: `http://127.0.0.1:${secondaryPort}`,
|
||||
ANTHROPIC_AUTH_TOKEN: 'deepseek_token',
|
||||
ANTHROPIC_MODEL: 'deepseek-chat',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
});
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(tempDir, '.ccs', 'config.json'),
|
||||
JSON.stringify({ profiles: { hf: primarySettings, deepseek: secondarySettings } }, null, 2),
|
||||
'utf8'
|
||||
);
|
||||
|
||||
const profile: OpenAICompatProfileConfig = {
|
||||
profileName: 'hf',
|
||||
settingsPath: primarySettings,
|
||||
baseUrl: `http://127.0.0.1:${primaryPort}`,
|
||||
apiKey: 'hf_token',
|
||||
provider: 'generic-chat-completion-api',
|
||||
model: 'hf-default',
|
||||
};
|
||||
proxyServer = startOpenAICompatProxyServer({
|
||||
profile,
|
||||
port: proxyPort,
|
||||
authToken: 'test-proxy-token',
|
||||
});
|
||||
|
||||
const response = await requestProxy({
|
||||
model: 'deepseek:deepseek-reasoner',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
expect(response.status).toBe(200);
|
||||
expect(await response.json()).toMatchObject({
|
||||
content: [{ type: 'text', text: 'Reply from secondary' }],
|
||||
});
|
||||
expect(hits).toEqual(['secondary']);
|
||||
expect(bodies[0]?.body).toMatchObject({ model: 'deepseek-reasoner' });
|
||||
});
|
||||
|
||||
it('routes thinking requests through the configured think scenario', async () => {
|
||||
const primaryPort = await getPort();
|
||||
const thinkPort = await getPort();
|
||||
const hits: string[] = [];
|
||||
const bodies: Array<{ label: string; body: unknown }> = [];
|
||||
await startMockUpstream(primaryPort, 'primary', hits, bodies);
|
||||
await startMockUpstream(thinkPort, 'thinker', hits, bodies);
|
||||
|
||||
const primarySettings = writeSettings('hf', {
|
||||
ANTHROPIC_BASE_URL: `http://127.0.0.1:${primaryPort}`,
|
||||
ANTHROPIC_AUTH_TOKEN: 'hf_token',
|
||||
ANTHROPIC_MODEL: 'hf-default',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
});
|
||||
const thinkSettings = writeSettings('thinker', {
|
||||
ANTHROPIC_BASE_URL: `http://127.0.0.1:${thinkPort}`,
|
||||
ANTHROPIC_AUTH_TOKEN: 'think_token',
|
||||
ANTHROPIC_MODEL: 'deepseek-reasoner',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
});
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(tempDir, '.ccs', 'config.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
profiles: { hf: primarySettings, thinker: thinkSettings },
|
||||
proxy: {
|
||||
routing: {
|
||||
think: 'thinker:deepseek-reasoner',
|
||||
},
|
||||
},
|
||||
},
|
||||
null,
|
||||
2
|
||||
),
|
||||
'utf8'
|
||||
);
|
||||
|
||||
const profile: OpenAICompatProfileConfig = {
|
||||
profileName: 'hf',
|
||||
settingsPath: primarySettings,
|
||||
baseUrl: `http://127.0.0.1:${primaryPort}`,
|
||||
apiKey: 'hf_token',
|
||||
provider: 'generic-chat-completion-api',
|
||||
model: 'hf-default',
|
||||
};
|
||||
proxyServer = startOpenAICompatProxyServer({
|
||||
profile,
|
||||
port: proxyPort,
|
||||
authToken: 'test-proxy-token',
|
||||
});
|
||||
|
||||
const response = await requestProxy({
|
||||
model: 'hf-default',
|
||||
thinking: { type: 'enabled', budget_tokens: 9000 },
|
||||
messages: [{ role: 'user', content: 'think hard' }],
|
||||
});
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(await response.json()).toMatchObject({
|
||||
content: [{ type: 'text', text: 'Reply from thinker' }],
|
||||
});
|
||||
expect(hits).toEqual(['thinker']);
|
||||
expect(bodies[0]?.body).toMatchObject({ model: 'deepseek-reasoner' });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,194 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { resolveProxyRequestRoute } from '../../../src/proxy/request-router';
|
||||
import { resolveOpenAICompatProfileConfig } from '../../../src/proxy/profile-router';
|
||||
import { loadSettings } from '../../../src/utils/config-manager';
|
||||
|
||||
let originalCcsHome: string | undefined;
|
||||
let tempDir: string;
|
||||
|
||||
function writeSettings(profileName: string, env: Record<string, string>): string {
|
||||
const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`);
|
||||
fs.writeFileSync(settingsPath, JSON.stringify({ env }, null, 2), 'utf8');
|
||||
return settingsPath;
|
||||
}
|
||||
|
||||
function buildProfile(profileName: string) {
|
||||
const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`);
|
||||
const profile = resolveOpenAICompatProfileConfig(
|
||||
profileName,
|
||||
settingsPath,
|
||||
loadSettings(settingsPath).env || {}
|
||||
);
|
||||
expect(profile).toBeTruthy();
|
||||
return profile!;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
originalCcsHome = process.env.CCS_HOME;
|
||||
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-router-'));
|
||||
fs.mkdirSync(path.join(tempDir, '.ccs'), { recursive: true });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalCcsHome !== undefined) {
|
||||
process.env.CCS_HOME = originalCcsHome;
|
||||
} else {
|
||||
delete process.env.CCS_HOME;
|
||||
}
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('resolveProxyRequestRoute', () => {
|
||||
beforeEach(() => {
|
||||
process.env.CCS_HOME = tempDir;
|
||||
|
||||
const profiles = {
|
||||
hf: writeSettings('hf', {
|
||||
ANTHROPIC_BASE_URL: 'https://router.huggingface.co/v1',
|
||||
ANTHROPIC_AUTH_TOKEN: 'hf_token',
|
||||
ANTHROPIC_MODEL: 'hf-default',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
}),
|
||||
deepseek: writeSettings('deepseek', {
|
||||
ANTHROPIC_BASE_URL: 'https://api.deepseek.com/v1',
|
||||
ANTHROPIC_AUTH_TOKEN: 'deepseek_token',
|
||||
ANTHROPIC_MODEL: 'deepseek-chat',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
}),
|
||||
thinker: writeSettings('thinker', {
|
||||
ANTHROPIC_BASE_URL: 'https://thinking.example.com/v1',
|
||||
ANTHROPIC_AUTH_TOKEN: 'think_token',
|
||||
ANTHROPIC_MODEL: 'deepseek-reasoner',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
}),
|
||||
search: writeSettings('search', {
|
||||
ANTHROPIC_BASE_URL: 'https://search.example.com/v1',
|
||||
ANTHROPIC_AUTH_TOKEN: 'search_token',
|
||||
ANTHROPIC_MODEL: 'sonar-pro',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
}),
|
||||
background: writeSettings('background', {
|
||||
ANTHROPIC_BASE_URL: 'https://background.example.com/v1',
|
||||
ANTHROPIC_AUTH_TOKEN: 'background_token',
|
||||
ANTHROPIC_MODEL: 'qwen-small',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
}),
|
||||
long: writeSettings('long', {
|
||||
ANTHROPIC_BASE_URL: 'https://long.example.com/v1',
|
||||
ANTHROPIC_AUTH_TOKEN: 'long_token',
|
||||
ANTHROPIC_MODEL: 'gemini-2.5-pro',
|
||||
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||
}),
|
||||
};
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(tempDir, '.ccs', 'config.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
profiles,
|
||||
proxy: {
|
||||
routing: {
|
||||
default: 'hf:hf-default',
|
||||
background: 'background:qwen-small',
|
||||
think: 'thinker:deepseek-reasoner',
|
||||
longContext: 'long:gemini-2.5-pro',
|
||||
longContextThreshold: 10,
|
||||
webSearch: 'search:sonar-pro',
|
||||
},
|
||||
},
|
||||
},
|
||||
null,
|
||||
2
|
||||
),
|
||||
'utf8'
|
||||
);
|
||||
});
|
||||
|
||||
it('uses explicit profile:model selectors', () => {
|
||||
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||
model: 'deepseek:deepseek-reasoner',
|
||||
stream: true,
|
||||
messages: [{ role: 'user', content: 'plan this' }],
|
||||
});
|
||||
|
||||
expect(route.profile.profileName).toBe('deepseek');
|
||||
expect(route.model).toBe('deepseek-reasoner');
|
||||
expect(route.source).toBe('explicit-profile');
|
||||
});
|
||||
|
||||
it('matches plain model ids to the profile that owns them', () => {
|
||||
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||
model: 'deepseek-chat',
|
||||
stream: true,
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
|
||||
expect(route.profile.profileName).toBe('deepseek');
|
||||
expect(route.model).toBe('deepseek-chat');
|
||||
expect(route.source).toBe('profile-model-match');
|
||||
});
|
||||
|
||||
it('routes thinking requests through the configured think scenario', () => {
|
||||
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||
model: 'hf-default',
|
||||
stream: true,
|
||||
reasoning_effort: 'high',
|
||||
reasoning: { enabled: true, effort: 'high' },
|
||||
messages: [{ role: 'user', content: 'work this out carefully' }],
|
||||
});
|
||||
|
||||
expect(route.profile.profileName).toBe('thinker');
|
||||
expect(route.model).toBe('deepseek-reasoner');
|
||||
expect(route.scenario).toBe('think');
|
||||
expect(route.source).toBe('scenario');
|
||||
});
|
||||
|
||||
it('routes long-context requests through the configured longContext scenario', () => {
|
||||
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||
model: 'hf-default',
|
||||
stream: true,
|
||||
messages: [{ role: 'user', content: 'x'.repeat(120) }],
|
||||
});
|
||||
|
||||
expect(route.profile.profileName).toBe('long');
|
||||
expect(route.model).toBe('gemini-2.5-pro');
|
||||
expect(route.scenario).toBe('longContext');
|
||||
expect(route.estimatedTokens).toBeGreaterThan(10);
|
||||
});
|
||||
|
||||
it('routes web_search tool requests through the configured webSearch scenario', () => {
|
||||
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||
model: 'hf-default',
|
||||
stream: true,
|
||||
messages: [{ role: 'user', content: 'search the docs' }],
|
||||
tools: [
|
||||
{
|
||||
type: 'function',
|
||||
function: {
|
||||
name: 'web_search',
|
||||
parameters: { type: 'object' },
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(route.profile.profileName).toBe('search');
|
||||
expect(route.model).toBe('sonar-pro');
|
||||
expect(route.scenario).toBe('webSearch');
|
||||
});
|
||||
|
||||
it('routes haiku requests through the configured background scenario', () => {
|
||||
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||
model: 'claude-3-haiku',
|
||||
stream: true,
|
||||
messages: [{ role: 'user', content: 'run this in the background' }],
|
||||
});
|
||||
|
||||
expect(route.profile.profileName).toBe('background');
|
||||
expect(route.model).toBe('qwen-small');
|
||||
expect(route.scenario).toBe('background');
|
||||
});
|
||||
});
|
||||
@@ -28,6 +28,7 @@ describe('ProxyRequestTransformer', () => {
|
||||
|
||||
expect(result.stream).toBe(true);
|
||||
expect(result.reasoning_effort).toBe('high');
|
||||
expect(result.reasoning).toEqual({ enabled: true, effort: 'high' });
|
||||
expect(result.max_tokens).toBe(1024);
|
||||
expect(result.temperature).toBe(0.2);
|
||||
expect(result.top_p).toBe(0.9);
|
||||
@@ -42,6 +43,43 @@ describe('ProxyRequestTransformer', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('translates base64 image blocks into OpenAI image_url parts', () => {
|
||||
const transformer = new ProxyRequestTransformer();
|
||||
const result = transformer.transform({
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'text', text: 'Describe this image' },
|
||||
{
|
||||
type: 'image',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: 'image/png',
|
||||
data: 'ZmFrZS1pbWFnZS1ieXRlcw==',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(result.messages).toEqual([
|
||||
{
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'text', text: 'Describe this image' },
|
||||
{
|
||||
type: 'image_url',
|
||||
image_url: {
|
||||
url: 'data:image/png;base64,ZmFrZS1pbWFnZS1ieXRlcw==',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('drops malformed optional fields but preserves the translated core request', () => {
|
||||
const transformer = new ProxyRequestTransformer();
|
||||
const result = transformer.transform({
|
||||
|
||||
Reference in New Issue
Block a user