* add _transform_responses_api_function_call_to_chat_completion_message
* test_responses_api_with_tool_calls
* TestFunctionCallTransformation
* fixes for responses API testing google ai studio
* TestGoogleAIStudioResponsesAPITest
* test_responses_api_with_tool_calls
* test_responses_api_with_tool_calls
* test_basic_openai_responses_streaming_delete_endpoint
* fix(create_key_button.tsx): add prompts on UI
* feat(key_management_endpoints.py): support adding prompt to key via `/key/update`
* fix(key_info_view.tsx): show existing prompts on key in key_info_view.tsx
* fix(key_edit_view.tsx): UX - disable premium feature for non-premium users
prevent accidental clicking
* fix(create_key_button.tsx): disable premium features behind flag, prevent errors
* feat(prompts.tsx): add new ui component to view created prompts
enables viewing prompts created on config
* feat(prompt_info.tsx): add component for viewing the prompt information
* feat(prompt_endpoints.py): support converting dotprompt to json structure + accept json structure in promptmanager
allows prompt manager to work with api endpoints
* test(test_prompt_manager.py): add unit tests for json data input
* feat(dotprompt/__init__.py): add prompt data to dotpromptmanager
* fix(prompt_endpoints.py): working crud endpoints for prompt management
* feat(prompts/): support `prompt_file` for dotprompt
allows to precisely point to the prompt file a prompt should use
* feat(proxy/utils.py): resolve prompt id correctly
resolves user sent prompt id with internal prompt id
* feat(schema.prisma): initial pr with db schema for prompt management table
allows post endpoints to work with backend
* feat(prompt_endpoints.py): use db in patch_prompt endpoint
* feat(prompt_endpoints.py): use db for update_prompt endpoint
* feat(prompt_endpoints.py): use db on prompt delete endpoint
* build(schema.prisma): add prompt tale to schema.prisma in litellm-proxy-extras
* build(migration.sql): add new sql migration file
* fix(init_prompts.py): fix init
* feat(prompt_info_view.tsx): show the raw prompt template on ui
allows developer to know the prompt template they'll be calling
* feat(add_prompt_form.tsx): working ui add prompt flow
allows user to add prompts to litellm via ui
* build(ui/): styling fixes
* build(ui/): prompts.tsx
styling improvements
* fix(add_prompt_form.tsx): styling improvements
* build(prompts.tsx): styling improvements
* build(ui/): styling improvements
* build(ui/): fix ui error
* fix: fix ruff check
* docs: document new api params
* test: update tests
* fix(bedrock): prevent duplicate role assumption in EKS/IRSA environments
Fixes issue where AWS role assumption would fail in EKS/IRSA environments
when trying to assume the same role that's already being used.
The problem occurred when:
1. EKS/IRSA automatically assumes a role (e.g., LitellmRole)
2. LiteLLM tries to assume the same role again, causing AccessDenied errors
3. Different models with different roles would fail due to incorrect role context
Changes:
- Added check in _auth_with_aws_role() to detect if already using target role
- Skip role assumption if current identity matches target role
- Return current credentials instead of attempting duplicate assumption
- Added comprehensive test coverage for the fix
This ensures proper role chaining works in EKS/IRSA environments where:
- Service Account can assume Role A
- Role A can assume Role B for different models/accounts
Resolves the AccessDenied errors reported in bedrock usage scenarios.
* fix(bedrock): simplify role assumption for EKS/IRSA environments
Fixes AWS Bedrock role assumption in EKS/IRSA environments by properly
handling ambient credentials when no explicit credentials are provided.
The issue occurred because commit 197e7efa8f
introduced changes that broke role assumption in EKS/IRSA environments.
Changes:
- Simplified _auth_with_aws_role() to use ambient credentials when no
explicit AWS credentials are provided (aws_access_key_id and
aws_secret_access_key are both None)
- This allows web identity tokens in EKS/IRSA to work automatically
through boto3's credential chain
- Maintains backward compatibility for explicit credential scenarios
Added comprehensive test coverage:
- test_eks_irsa_ambient_credentials_used: Verifies ambient credentials work
- test_explicit_credentials_used_when_provided: Ensures explicit creds still work
- test_partial_credentials_still_use_ambient: Edge case handling
- test_cross_account_role_assumption: Multi-account scenarios
- test_role_assumption_with_custom_session_name: Custom session names
- test_role_assumption_ttl_calculation: TTL calculation verification
- test_role_assumption_error_handling: Error propagation
- test_multiple_role_assumptions_in_sequence: Sequential role assumptions
This fix ensures that in EKS/IRSA environments:
1. Service accounts can assume their initial role via web identity
2. That role can then assume other roles across accounts as configured
3. Different models can use different roles without conflicts
* fix(bedrock): add automatic IRSA detection for EKS environments
- Detect AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables
- Automatically use web identity token flow when IRSA is detected
- Read web identity token from file and pass to existing auth method
- Add test coverage for IRSA environment detection
- Fixes authentication errors in EKS with IRSA when no explicit credentials provided
* fix(bedrock): skip role assumption when IRSA role matches requested role
- Detect when AWS_ROLE_ARN environment variable matches the requested role
- Skip unnecessary role assumption when already running as the target role
- Use existing env vars authentication method for IRSA credentials
- Add test coverage for same-role IRSA scenario
- Fixes 'not authorized to perform: sts:AssumeRole' errors when trying to assume the same role
* fix(bedrock): use boto3's native IRSA support for cross-account role assumption
- Replace custom web identity token handling with boto3's built-in IRSA support
- boto3 automatically reads AWS_WEB_IDENTITY_TOKEN_FILE and assumes initial role
- Then use standard assume_role for cross-account access
- Update test to mock boto3 STS client instead of internal methods
- Fixes 'OIDC token could not be retrieved from secret manager' error
* fix(bedrock): improve IRSA error handling and add debug logging
- Add debug logging to show current identity and role assumption attempts
- Provide clearer error messages for trust policy issues
- Fix region handling in IRSA flow
- Re-raise exceptions instead of silently falling through
- This helps diagnose cross-account role assumption permission issues
* fix(bedrock): manually assume IRSA role with correct session name for cross-account scenarios
- When doing cross-account role assumption, manually assume the IRSA role first with the desired session name
- This ensures the session name in the assumed role ARN matches what's expected in trust policies
- For same-account scenarios, continue using boto3's automatic IRSA support
- Updated tests to handle the new flow
- This fixes the issue where cross-account trust policies require specific session names
* fix: Fix linting issues in base_aws_llm.py
- Fix f-string without placeholders (F541)
- Refactor _auth_with_aws_role to reduce statements count (PLR0915)
- Extract _handle_irsa_cross_account helper method
- Extract _handle_irsa_same_account helper method
- Extract _extract_credentials_and_ttl helper method
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(guardrails): Fix PANW Prisma AIRS post-call hook method name
- Changed async_post_call_hook to async_post_call_success_hook to match proxy calling convention
- Added event_hook parameter to initialization to ensure proper hook registration
- Fixes post-call response scanning for PANW Prisma AIRS guardrails
Resolves issue where post-call hooks were not being invoked due to method name mismatch.
* Update PANW Prisma AIRS tests to use correct method name
* fix(router.py): add acompletion_streaming_iterator inside router
allows router to catch errors mid-stream for fallbacks
Work for https://github.com/BerriAI/litellm/issues/6532
* fix(router.py): working mid-stream fallbacks
* fix(router.py): more iterations
* fix(router.py): working mid-stream fallbacks with fallbacks set on router
* fix(router.py): pass prior content back in new request as assistant prefix message
* fix(router.py): add a system prompt to help guide non-prefix supporting models to use the continued text correctly
* fix(common_utils.py): support converting `prefix: true` for non-prefix supporting models
* fix: reduce LOC in function
* test(test_router.py): add unit tests for new function
* test: add basic unit test
* fix(router.py): ensure return type of fallback stream is compatible with CustomStreamWrapper
prevent client code from breaking
* fix: cleanup
* test: update test
* fix: fix linting error
The test was failing because it was trying to patch MAX_LANGFUSE_INITIALIZED_CLIENTS
at the wrong path. The constant is imported from litellm.constants into the langfuse
module namespace, so we need to use patch.object on the imported module reference.
Changes:
- Import langfuse module explicitly for patching
- Use patch.object instead of patch string path
- This fixes the AttributeError that was causing CI failures
* ensure original client is disconnected when re-creating
* test_recreate_prisma_client_successful_disconnect
* test_recreate_prisma_client_successful_disconnect
* fix(proxy_cli.py): make use_prisma_migrate proxy default
Fixes https://github.com/BerriAI/litellm/issues/13046
Prisma migrate deploy prevents resetting db
* fix(auth_checks.py): resolve team only models while doing auth checks on model access groups
Fixes issue where key had access via an access group, but team only model could not be called
* test(test_router.py): add unit testing
* feat(provider_specific_fields.tsx): add aws sagemaker on UI
* feat: Add dot notation support for all JWT fields
- Updated all JWT field access methods to use get_nested_value for dot notation support
- Enhanced get_team_id to properly handle team_id_default fallback with nested fields
- Added comprehensive unit tests for nested JWT field access and edge cases
- Updated documentation to reflect dot notation support across all JWT fields
- Maintains full backward compatibility with existing flat field configurations
Supported fields with dot notation:
- team_id_jwt_field, team_ids_jwt_field, user_id_jwt_field
- user_email_jwt_field, org_id_jwt_field, object_id_jwt_field
- end_user_id_jwt_field (roles_jwt_field was already supported)
Example: user_id_jwt_field: 'user.sub' accesses token['user']['sub']
* fix: Add type annotations to resolve mypy errors
- Add explicit type annotation for team_ids variable in get_team_ids_from_jwt
- Add type ignore comment for sentinel object return in get_team_id
- Resolves mypy errors while maintaining functionality
* fix: Resolve mypy type error in get_team_ids_from_jwt
- Remove explicit List[str] type annotation that conflicts with get_nested_value return type
- Simplify return logic to use 'team_ids or []' ensuring always returns List[str]
- Fixes: Incompatible types in assignment (expression has type 'list[str] | None', variable has type 'list[str]')
* fix: Add proper type annotation for team_ids variable
- Use Optional[List[str]] type annotation to satisfy mypy requirements
- Resolves: Need type annotation for 'team_ids' [var-annotated]
- Maintains functionality while ensuring type safety
* refactor: remove outdated JWT unit tests and consolidate JWT-related functionality
- Deleted the test_jwt.py file as it contained outdated and redundant tests.
- Consolidated JWT-related tests into test_handle_jwt.py for better organization and maintainability.
- Updated tests to ensure proper functionality of JWT handling, including token validation and role mapping.
- Enhanced test coverage for JWT field access and nested claims handling.
* test: add comprehensive unit tests for JWT authentication
- Introduced a new test file `test_jwt.py` containing unit tests for JWT authentication.
- Implemented tests for loading configuration with custom role names, validating tokens, and handling team tokens.
- Enhanced coverage for JWT field access, nested claims, and role-based access control.
- Added fixtures for Prisma client and public JWT key generation to support testing.
- Ensured proper handling of valid and invalid tokens, including user and team scenarios.
* revert test_handle_jwt.py
* rename file
* test: remove outdated JWT nesting tests and add new nested field access tests
- Deleted the `test_jwt_nesting.py` file as it contained outdated tests.
- Introduced new tests in `test_handle_jwt.py` to verify nested JWT field access.
- Enhanced coverage for accessing nested values using dot notation and ensured backward compatibility with flat field names.
- Added tests for handling missing nested paths and appropriate default values.
- Improved handling of metadata prefixes in nested field access.
* restore file
* Fix: Add support for GOOGLE_API_KEY environment variables for Gemini API authentication
* added test cases
* incoperated feedback to make it more maintainable
* fix failed linting CI
- Fixed 'missing finish_reason for choice 1' error with reasoning_effort
- Anthropic sends multiple content blocks with different indices
- OpenAI expects all content in a single choice at index=0
- Added comprehensive tests for text-only, text+tool, and multiple tools
* Fix security vulnerability in list_team_v2 endpoint
- Add missing allowed_route_check_inside_route security check to list_team_v2
- Add @management_endpoint_wrapper decorator for consistency with list_team
- Add comprehensive tests to verify security checks work correctly
- Ensure non-admin users can only query their own teams
- Ensure admin users can query all teams
This fixes a security bug where non-admin users could potentially access
team information they shouldn't have access to through the list_team_v2
endpoint, which was missing the authorization check present in list_team.
* Fix test
* Test fixes
* Fixed test
* Restored invalid delete
* Revert
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(azure/chat/gpt_transformation.py): support api_version="preview"
Fixes https://github.com/BerriAI/litellm/issues/12945
* Fix anthropic passthrough logging handler model fallback for streaming requests (#13022)
* fix: anthropic passthrough logging handler model fallback for streaming requests
- Add fallback logic to retrieve model from logging_obj.model_call_details when request_body.model is empty
- Fixes issue #12933 where streaming requests to anthropic passthrough endpoints would crash due to missing model field
- Ensures downstream logging and cost calculation work correctly for all streaming scenarios
- Maintains backwards compatibility with existing non-streaming requests
* test: add minimal tests for anthropic passthrough logging handler model fallback
- Add unit tests for the model fallback logic in _handle_logging_anthropic_collected_chunks
- Test existing behavior when request_body.model is present
- Test fallback logic when request_body.model is empty but logging_obj.model_call_details has model
- Test edge cases where both sources are empty or missing
- Ensure backwards compatibility and graceful degradation
* fix(anthropic_passthrough_logging_handler.py): add provider to model name (accurate cost tracking)
* fix(anthropic_passthrough_logging_handler.py): don't reset custom llm provider, if already set
* fix: fix check
---------
Co-authored-by: Haggai Shachar <haggai.shachar@backline.ai>
Ishaan Jaff