* fix(access group): allow access group on mcp tool retrieval
* fix(test): fix broken tests and add test case for access group
* fix(mypy): fix typing issues
* fix proxy config
* fix(responses api): fix streaming ID consistency and tool format handling (#12640)
* fix(responses): ensure streaming chunk IDs use consistent encoding format
Fixes streaming ID inconsistency where streaming responses used raw provider IDs
while non-streaming responses used properly encoded IDs with provider context.
Changes:
- Updated LiteLLMCompletionStreamingIterator to accept provider context
- Added _encode_chunk_id() method using same logic as non-streaming responses
- Modified chunk transformation to encode all streaming item_ids with resp_ prefix
- Updated handlers to pass custom_llm_provider and litellm_metadata to streaming iterator
Impact:
- Streaming chunk IDs now format: resp_<base64_encoded_provider_context>
- Enables session continuity when using streaming response IDs as previous_response_id
- Allows provider detection and load balancing with streaming responses
- Maintains backward compatibility with existing streaming functionality
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(types): add explicit Optional[str] type annotation for model_id
This resolves MyPy type checking error where model_id could be None
but wasn't explicitly typed as Optional[str].
* fix(types): handle None case for litellm_metadata access
Prevents 'Item None has no attribute get' error by checking for None
before accessing litellm_metadata dictionary.
* test: add comprehensive tests for streaming ID consistency
Adds unit and E2E tests to verify streaming chunk IDs are properly encoded
with consistent format across streaming responses.
## Tests Added
### Unit Test (test_reasoning_content_transformation.py)
- `test_streaming_chunk_id_encoding()`: Validates the `_encode_chunk_id()` method
correctly encodes chunk IDs with `resp_` prefix and provider context
### E2E Tests (test_e2e_openai_responses_api.py)
- `test_streaming_id_consistency_across_chunks()`: Tests that all streaming chunk IDs
are properly encoded across multiple chunks in a real streaming response
- `test_streaming_response_id_as_previous_response_id()`: Tests the core use case -
using streaming response IDs for session continuity with `previous_response_id`
## Key Testing Approach
- Uses **Gemini** (non-OpenAI model) to test the transformation logic rather than
OpenAI passthrough, since the streaming ID consistency issue occurs when LiteLLM
transforms responses rather than just passing through to native OpenAI responses API
- Tests validate that streaming chunk IDs now use same encoding as non-streaming responses
- Verifies session continuity works with streaming responses
Addresses @ishaan-jaff's request for unit tests covering the streaming ID consistency fix.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(lint): remove unused imports in transformation.py
Removes unused imports to fix CI linting errors:
- GenericResponseOutputItem
- OutputFunctionToolCall
* test: remove E2E tests from openai_endpoints_tests
Remove streaming ID consistency E2E tests as requested by @ishaan-jaff.
Keep only the mock/unit test in test_reasoning_content_transformation.py
* revert: remove streaming chunk ID encoding to original behavior
This reverts the streaming chunk ID encoding changes to understand the original issue better.
Original behavior was:
- Streaming chunks: raw provider IDs
- Streaming final response: raw IDs (PROBLEM!)
- Non-streaming final response: encoded IDs (correct)
The real issue: streaming final response IDs were not encoded, breaking session continuity.
* fix(responses): encode streaming final response IDs to match OpenAI behavior
Fixes streaming ID inconsistency to match OpenAI's Responses API behavior:
- Streaming chunks: raw message IDs (like OpenAI's msg_xxx)
- Final response: encoded IDs (like OpenAI's resp_xxx)
This enables session continuity by ensuring streaming final response IDs
have the same encoded format as non-streaming responses, allowing them
to be used as previous_response_id in follow-up requests.
Changes:
- Add custom_llm_provider and litellm_metadata to LiteLLMCompletionStreamingIterator
- Update handlers to pass provider context to streaming iterator
- Apply _update_responses_api_response_id_with_model_id to final streaming response
- Keep streaming chunks as raw IDs to match OpenAI format
Impact:
- Session continuity works with streaming responses
- Load balancing can detect provider from streaming final response IDs
- Format matches OpenAI's Responses API exactly
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* test: update unit test to match correct OpenAI-compatible behavior
Updates the unit test to verify streaming chunk IDs are raw (not encoded)
to match OpenAI's responses API format:
- Streaming chunks: raw message IDs (like msg_xxx)
- Final response: encoded IDs (like resp_xxx)
This reflects the correct behavior implemented in the fix.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* cleanup
* TestBaseResponsesAPIStreamingIterator
---------
Co-authored-by: Javier de la Torre <jatorre@carto.com>
Co-authored-by: Claude <noreply@anthropic.com>
* (#13284) add avector_store_create to route_type which doesn't require model
* (#13284) exclude hidden params in metadata when create vector store
* (#13284) fix lint error
* (#13284) keep metadata None if metadata is None(not empty dict)
* (#13284) add test code
* (#13284) change test code name
* (#13284) add avector_store_search to route_type which doesn't require model
* fix unsupported operand type(s) for +=: 'NoneType' and 'str' on clientside auth creds for responses
* fix the client side auth to use correct metadata
* add more tests
* fix tests
* fix(route_checks.py): ensure disable llm api endpoints is correctly set
* fix(route_checks.py): raise httpexception
raise expected exceptions
* fix(router.py): handle team only wildcard models
fixes issue where team only wildcard models were not considered during auth checks
* fix(router.py): handle team only wildcard models
fixes issue where team only wildcard models were not considered during auth checks
* create OCI required files
* request and response conversion for non-streaming chat
* support tool calling with OCI generic API without streaming
* adaptation of api call for generic and cohere format
* include tool calls and responses in generic api and dropping support for cohere
* fix invalid content-length error
* support streaming for generic api
* fix auth error when using acompletion with streaming
* refactor: use base_llm_http_handler and include API type definitions
* update types and add type safety in different methods
* fix OCIFunction format
* create custom stream wrapper for decoding OCI stream
* remove unused files
* create unit tests for OCI
* lint the code
* remove manual test
* docs: update the docs to include OCI
* add _transform_responses_api_function_call_to_chat_completion_message
* test_responses_api_with_tool_calls
* TestFunctionCallTransformation
* fixes for responses API testing google ai studio
* TestGoogleAIStudioResponsesAPITest
* test_responses_api_with_tool_calls
* test_responses_api_with_tool_calls
* test_basic_openai_responses_streaming_delete_endpoint
* fix(create_key_button.tsx): add prompts on UI
* feat(key_management_endpoints.py): support adding prompt to key via `/key/update`
* fix(key_info_view.tsx): show existing prompts on key in key_info_view.tsx
* fix(key_edit_view.tsx): UX - disable premium feature for non-premium users
prevent accidental clicking
* fix(create_key_button.tsx): disable premium features behind flag, prevent errors
* feat(prompts.tsx): add new ui component to view created prompts
enables viewing prompts created on config
* feat(prompt_info.tsx): add component for viewing the prompt information
* feat(prompt_endpoints.py): support converting dotprompt to json structure + accept json structure in promptmanager
allows prompt manager to work with api endpoints
* test(test_prompt_manager.py): add unit tests for json data input
* feat(dotprompt/__init__.py): add prompt data to dotpromptmanager
* fix(prompt_endpoints.py): working crud endpoints for prompt management
* feat(prompts/): support `prompt_file` for dotprompt
allows to precisely point to the prompt file a prompt should use
* feat(proxy/utils.py): resolve prompt id correctly
resolves user sent prompt id with internal prompt id
* feat(schema.prisma): initial pr with db schema for prompt management table
allows post endpoints to work with backend
* feat(prompt_endpoints.py): use db in patch_prompt endpoint
* feat(prompt_endpoints.py): use db for update_prompt endpoint
* feat(prompt_endpoints.py): use db on prompt delete endpoint
* build(schema.prisma): add prompt tale to schema.prisma in litellm-proxy-extras
* build(migration.sql): add new sql migration file
* fix(init_prompts.py): fix init
* feat(prompt_info_view.tsx): show the raw prompt template on ui
allows developer to know the prompt template they'll be calling
* feat(add_prompt_form.tsx): working ui add prompt flow
allows user to add prompts to litellm via ui
* build(ui/): styling fixes
* build(ui/): prompts.tsx
styling improvements
* fix(add_prompt_form.tsx): styling improvements
* build(prompts.tsx): styling improvements
* build(ui/): styling improvements
* build(ui/): fix ui error
* fix: fix ruff check
* docs: document new api params
* test: update tests
* fix(bedrock): prevent duplicate role assumption in EKS/IRSA environments
Fixes issue where AWS role assumption would fail in EKS/IRSA environments
when trying to assume the same role that's already being used.
The problem occurred when:
1. EKS/IRSA automatically assumes a role (e.g., LitellmRole)
2. LiteLLM tries to assume the same role again, causing AccessDenied errors
3. Different models with different roles would fail due to incorrect role context
Changes:
- Added check in _auth_with_aws_role() to detect if already using target role
- Skip role assumption if current identity matches target role
- Return current credentials instead of attempting duplicate assumption
- Added comprehensive test coverage for the fix
This ensures proper role chaining works in EKS/IRSA environments where:
- Service Account can assume Role A
- Role A can assume Role B for different models/accounts
Resolves the AccessDenied errors reported in bedrock usage scenarios.
* fix(bedrock): simplify role assumption for EKS/IRSA environments
Fixes AWS Bedrock role assumption in EKS/IRSA environments by properly
handling ambient credentials when no explicit credentials are provided.
The issue occurred because commit 197e7efa8f
introduced changes that broke role assumption in EKS/IRSA environments.
Changes:
- Simplified _auth_with_aws_role() to use ambient credentials when no
explicit AWS credentials are provided (aws_access_key_id and
aws_secret_access_key are both None)
- This allows web identity tokens in EKS/IRSA to work automatically
through boto3's credential chain
- Maintains backward compatibility for explicit credential scenarios
Added comprehensive test coverage:
- test_eks_irsa_ambient_credentials_used: Verifies ambient credentials work
- test_explicit_credentials_used_when_provided: Ensures explicit creds still work
- test_partial_credentials_still_use_ambient: Edge case handling
- test_cross_account_role_assumption: Multi-account scenarios
- test_role_assumption_with_custom_session_name: Custom session names
- test_role_assumption_ttl_calculation: TTL calculation verification
- test_role_assumption_error_handling: Error propagation
- test_multiple_role_assumptions_in_sequence: Sequential role assumptions
This fix ensures that in EKS/IRSA environments:
1. Service accounts can assume their initial role via web identity
2. That role can then assume other roles across accounts as configured
3. Different models can use different roles without conflicts
* fix(bedrock): add automatic IRSA detection for EKS environments
- Detect AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables
- Automatically use web identity token flow when IRSA is detected
- Read web identity token from file and pass to existing auth method
- Add test coverage for IRSA environment detection
- Fixes authentication errors in EKS with IRSA when no explicit credentials provided
* fix(bedrock): skip role assumption when IRSA role matches requested role
- Detect when AWS_ROLE_ARN environment variable matches the requested role
- Skip unnecessary role assumption when already running as the target role
- Use existing env vars authentication method for IRSA credentials
- Add test coverage for same-role IRSA scenario
- Fixes 'not authorized to perform: sts:AssumeRole' errors when trying to assume the same role
* fix(bedrock): use boto3's native IRSA support for cross-account role assumption
- Replace custom web identity token handling with boto3's built-in IRSA support
- boto3 automatically reads AWS_WEB_IDENTITY_TOKEN_FILE and assumes initial role
- Then use standard assume_role for cross-account access
- Update test to mock boto3 STS client instead of internal methods
- Fixes 'OIDC token could not be retrieved from secret manager' error
* fix(bedrock): improve IRSA error handling and add debug logging
- Add debug logging to show current identity and role assumption attempts
- Provide clearer error messages for trust policy issues
- Fix region handling in IRSA flow
- Re-raise exceptions instead of silently falling through
- This helps diagnose cross-account role assumption permission issues
* fix(bedrock): manually assume IRSA role with correct session name for cross-account scenarios
- When doing cross-account role assumption, manually assume the IRSA role first with the desired session name
- This ensures the session name in the assumed role ARN matches what's expected in trust policies
- For same-account scenarios, continue using boto3's automatic IRSA support
- Updated tests to handle the new flow
- This fixes the issue where cross-account trust policies require specific session names
* fix: Fix linting issues in base_aws_llm.py
- Fix f-string without placeholders (F541)
- Refactor _auth_with_aws_role to reduce statements count (PLR0915)
- Extract _handle_irsa_cross_account helper method
- Extract _handle_irsa_same_account helper method
- Extract _extract_credentials_and_ttl helper method
---------
Co-authored-by: openhands <openhands@all-hands.dev>
