* fix(content_filter.py): fix filter on toxic keywords
* feat: improve toxic/abusive language detection
* fix: additional improvements to nsfw filters
* feat: more improvements to nsfw filter
* feat(content_filter.json): add new australia specific nsfw content filter
ensure complete coverage for australia nsfw
* fix: cleanup policy templates
* fix(index.tsx): alert notice
* fix(index.tsx): add disclaimer notice
* feat(harmful_child_safety.yaml): new child safety content filter
ensure we catch inappropriate, child-specific content
* feat(policy_templates.json): add child safety and self harm filters
* fix(content_filter.py): improve racial bias filter to use a similar identifier + block word pattern and cover a wider range of ethnicities
* feat(policy_templates.json): add racial bias to nsfw policy template
* feat: add json content viewer
* fix(model_cost): add missing supports_system_messages and supports_tool_choice to bedrock/moonshotai.kimi-k2.5
* fix(streaming): ensure role=assistant is set on first streaming chunk via strip_role_from_delta
* fix(vertex_ai): ensure role=assistant on first streaming chunk for Llama models
Add VertexAILlama3StreamingHandler that injects role='assistant' into the
first streaming chunk delta when the Vertex AI Llama API omits it.
* fix(anthropic): filter unsupported JSON schema constraints for structured outputs
Fixes 400 error when using Anthropic models with structured outputs that have
min/max constraints.
The Anthropic API doesn't support these JSON schema constraints:
- minimum/maximum (numeric)
- exclusiveMinimum/exclusiveMaximum (numeric)
- minLength/maxLength (string)
- minItems/maxItems (array)
This mirrors the transformation done by the official Anthropic Python SDK.
See: https://platform.claude.com/docs/en/build-with-claude/structured-outputs#how-sdk-transformation-works
Adds tests for the schema filtering function.
* fix: update descriptions with removed constraint info in filter_anthropic_output_schema
Address review feedback: the function now appends removed constraint
information to the description field (matching Anthropic SDK behavior),
rather than silently dropping constraints.
---------
Co-authored-by: OpenClaw <openclaw@users.noreply.github.com>
* Add pipeline type definitions for guardrail pipelines
PipelineStep, GuardrailPipeline, PipelineStepResult, PipelineExecutionResult
with validation for actions (allow/block/next/modify_response) and modes.
* Export pipeline types from policy_engine types package
* Add optional pipeline field to Policy model
* Add pipeline executor for sequential guardrail execution
* Parse pipeline config in policy registry
* Add pipeline validation in policy validator
* Add pipeline resolution and managed guardrail tracking
* Resolve pipelines and exclude managed guardrails in pre-call
* Integrate pipeline execution into proxy pre_call_hook
* Add test guardrails for pipeline E2E testing
* Add example pipeline config YAML
* Add unit tests for pipeline type definitions
* Add unit tests for pipeline executor
* Add pipeline column to LiteLLM_PolicyTable schema
* Add pipeline field to policy CRUD request/response types
* Add pipeline support to policy DB CRUD operations
* Add PipelineStep and GuardrailPipeline TypeScript types
* Add Zapier-style pipeline flow builder UI component
* Integrate pipeline flow builder with mode toggle in policy form
* Add pipeline display section to policy info view
* Add unit tests for pipeline in policy CRUD types
* Refactor policy form to show mode picker first with icon cards
* Add full-screen FlowBuilderPage component for pipeline editing
* Wire up full-screen flow builder in PoliciesPanel with edit routing
* Restyle flow builder to match dev-tool UI aesthetic
* Restyle flow builder cards to match reference design
* Update step card to expanded layout with stacked ON PASS / ON FAIL sections
* Add end card to flow builder showing return to normal control flow
* Add PipelineTestRequest type for test-pipeline endpoint
* Export PipelineTestRequest from policy_engine types
* Add POST /policies/test-pipeline endpoint
* Add testPipelineCall networking function
* Add PipelineStepResult and PipelineTestResult types
* Add test pipeline panel to flow builder with run button and results display
* Fix pipeline executor: inject guardrail name into metadata so should_run_guardrail allows execution
* Update litellm/proxy/policy_engine/pipeline_executor.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/utils.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/policy_engine/policy_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/policy_engine/pipeline_executor.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Pyroscope: require PYROSCOPE_APP_NAME and PYROSCOPE_SERVER_ADDRESS, add UTF-8 locale hint
- No defaults for PYROSCOPE_APP_NAME or PYROSCOPE_SERVER_ADDRESS; fail at startup if unset when Pyroscope is enabled
- Set LANG/LC_ALL to C.UTF-8 when unset to reduce malformed_profile (invalid UTF-8) rejections
- Startup message suggests PYTHONUTF8=1 if server rejects profiles
- Simplify LITELLM_ENABLE_PYROSCOPE in config_settings; document Pyroscope env vars as required with no default
- Add pyroscope_profiling to sidebar (Alerting & Monitoring)
- pyproject.toml: pyroscope-io as required dep on non-Windows (marker), in proxy extra
* proxy: add PYROSCOPE_SAMPLE_RATE env, use verbose logging, fix int type
- Add optional PYROSCOPE_SAMPLE_RATE env (integer, no default)
- Pass sample_rate to pyroscope.configure() as int for pyroscope-io
- Replace print with verbose_proxy_logger (info/warning)
- Document PYROSCOPE_SAMPLE_RATE in config_settings.md
* Address Greptile PR feedback: Pyroscope optional, docs, tests, docstring
- pyproject.toml: mark pyroscope-io as optional=true (proxy extra only)
- Add docs/my-website/docs/proxy/pyroscope_profiling.md (fix broken sidebar link)
- Add tests/test_litellm/proxy/test_pyroscope.py for _init_pyroscope()
- proxy_server: fix _init_pyroscope docstring (required server/app name, sample rate as int)
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: populate identity fields in proxy admin JWT early-return path
When is_proxy_admin is True, the UserAPIKeyAuth early-return now includes
user_id, team_id, team_alias, team_metadata, org_id, and end_user_id
resolved from the JWT. Previously only user_role and parent_otel_span
were set, causing blank Team Name and Internal User in Request Logs UI.
* test: add unit tests for proxy admin JWT identity fields
* fix: add custom_body parameter to endpoint_func in create_pass_through_route
The bedrock_proxy_route calls `endpoint_func(custom_body=data)` to
pass a pre-parsed, SigV4-signed request body. However, the
`endpoint_func` closure created by `create_pass_through_route` does
not accept a `custom_body` keyword argument, causing:
TypeError: endpoint_func() got an unexpected keyword argument 'custom_body'
Add `custom_body: Optional[dict] = None` to both `endpoint_func`
definitions (adapter-based and URL-based). In the URL-based path,
when `custom_body` is provided by the caller, use it instead of
re-parsing the body from the raw request.
Fixes#16999
* Add tests for custom_body handling in create_pass_through_route
Address reviewer feedback on PR #20849:
- Document why the adapter-based endpoint_func accepts custom_body
for signature compatibility but does not forward it (the underlying
chat_completion_pass_through_endpoint does not support it).
- Add test_create_pass_through_route_custom_body_url_target: verifies
that when a caller (e.g. bedrock_proxy_route) supplies custom_body,
it takes precedence over the body parsed from the raw request.
- Add test_create_pass_through_route_no_custom_body_falls_back:
verifies that the default path (no custom_body) correctly uses the
request-parsed body, preserving existing behavior.
Both tests are fully mocked following the project's CONTRIBUTING.md
guidelines and the patterns established in the existing test file.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: themavik <themavik@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(bedrock): add DeepSeek V3.2 pricing and region support
* feat(bedrock): add minimax.minimax-m2.1 pricing and region support
* feat(bedrock): add moonshotai.kimi-k2.5 pricing and region support
* feat(bedrock): add qwen.qwen3-coder-next
pricing and region support
* add some sanity unit tests for the bedrock beta models added
* --amend
* resolve greptileai comments and suggestions
* docs: add reference to example_openai_endpoint repo for self-hosting fake OpenAI proxy (#21006)
- Updated benchmarks.md with a section on setting up fake OpenAI endpoints
- Updated load_test.md to mention the self-hosted option
- Updated load_test_advanced.md with a tip box about the example repo
Reference: https://github.com/BerriAI/example_openai_endpoint
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* MCP fixes
* fix(oldteams.tsx): show policies when creating
* fix(proxy/_types.py): ensure mcp rest endpoints can be called by virtual key
ensures UI works with virtual key testing mcp endpoints
* refactor: migrate get object permissions table logic to happen in user api key auth - allows functions to trust user api key object they receive has what they need
* fix(rest_endpoints.py): filter for allowed tools based on what key has access to
* fix(mcp_server_manager.py): ensure only allowed MCP's are returned to the user, via rest endpoints
* Guardrails - add toxic/abusive content filter guardrails
* fix(streaming): preserve usage data from post-finish_reason chunks in OpenAI-compatible streaming
Fixes#16112
OpenRouter and other OpenAI-compatible providers send a usage chunk after
the finish_reason='stop' chunk when stream_options.include_usage is True.
The OpenAIChatCompletionStreamingHandler.chunk_parser() was not passing
the usage field to ModelResponseStream, causing real token counts from the
provider to be lost and falling back to inaccurate estimates.
* fix: resolve merge conflict in test file
- Fix typo in test method name (extra space)
- Move test_prompt_cache_key_in_optional_params to its own class
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(vertex_ai): forward extra_body to completion transformation handler
The responses() function accepted extra_body as a named parameter but
did not pass it to response_api_handler when responses_api_provider_config
was None (completion transformation path), silently dropping it.
Also adds deep-merge support for extra_body in Vertex AI Gemini
transformation, so dict values like generationConfig are merged rather
than replaced.
* refactor(vertex_ai): extract _merge_extra_body to fix PLR0915 lint
Move the extra_body merge loop into a helper function to keep
_transform_request_body under the 50-statement limit.
* fix: allow Management keys to access user/daily/activity and team/daily/activity
* feat(vertex): surface trafficType via generic provider_specific_fields in Responses API
Extract Vertex AI's trafficType from usageMetadata in both streaming and
non-streaming paths, storing it in _hidden_params["provider_specific_fields"].
The Responses API transformation layer generically passes any
_hidden_params["provider_specific_fields"] dict to the ResponsesAPIResponse,
avoiding provider-specific logic in the bridge.
Also fix stream_chunk_builder to propagate _hidden_params from the last
streaming chunk to the rebuilt ModelResponse, ensuring provider metadata
survives the chunk→response rebuild.
---------
Co-authored-by: naaa760 <neh6a683@gmail.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
OAuth tokens (sk-ant-oat*) require Authorization: Bearer header per
Anthropic's OAuth specification, but were being sent via x-api-key
which Anthropic rejects with 'invalid x-api-key'.
- optionally_handle_anthropic_oauth: detect OAuth tokens in api_key
param (standard chat flow), not just Authorization header
- get_anthropic_headers: use Authorization: Bearer + required OAuth
headers for OAuth tokens, x-api-key for regular API keys
- Passthrough messages: skip x-api-key when Authorization is set
- Add oauth-2025-04-20 to beta headers whitelist config
Ishaan Jaffer