* add support of bearer token for bedrock integration
* fix linting issue
* fix type checking issue
* reoder arguments to address type checking issue
* switch to use get_secret_str to fetch env variable
Co-authored-by: 0x-fang <fanggong@amazon.com>
- Add azure/o3-deep-research model to pricing configuration
- Input cost: 0 per million tokens (1e-05 per token)
- Output cost: 0 per million tokens (4e-05 per token)
- Supports reasoning, web search, function calling, vision, and other advanced features
- Based on Azure AI Foundry Deep Research pricing documentation
Resolves#12318
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(auth_checks.py): resolve a model group alias when key has access to underlying model
Fixes LIT-293
* feat(anthropic/): add mock_response to anthropic /v1/messages
makes it easy to test fallback logic
* fix(router.py): support fallbacks on /v1/messages
adds working fallbacks on generic api route
* refactor(router.py): point _ageneric_api_call_with_fallbacks to updated function
* test: add unit test for new helper on router
* fix(router.py): use correct metadata variable name
* fix(router.py): use correct metadata field
* docs(config_settings.md): document new param
* fix(opentelemetry.py): initial commit adding support for OTEL_RESOURCE_ATTRIBUTES
* test: update unit tests
* feat(ui/): initial commit with revamped model hub
* fix(model_hub_table.tsx): generic 'supports_' rendering as capabilities
* feat(model_hub_table.tsx): indicate if a model is publicly available
* refactor(model_hub_table.tsx): refactor to use common model data table component
* fix(model_hub_table.tsx): fix box sizes
* fix(model_hub_table.tsx): enable selecting / deselecting models in columns
* feat(public_endpoints.py): initial commit adding `/public/model_hub` endpoint
enables sharing public models
* feat(public_endpoints/): instrumentation for public model hub route
* feat(proxy_server.py): support request access form for model groups
allows user to request access to a model
* refactor: use a dictionary of text + link instead of 1 hardcoded request access form
* fix(proxy_server.py): prevent duplicates in model_group info
* fix: fix linting error
* fix(__init__.py): fix linting error
* fix(db_spend_update_writer.py): fix db query
* fix(litellm_pre_call_utils.py): support passing anthropic-beta headers when 'forward_client_headers_to_llm_api' is True
allows user to pass along extra headers to vertex ai anthropic models
* docs(config_settings.md): update docs
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* change query type
* Update model_checks.py
* add _get_mcp_auth_header_from_headers
* test_process_mcp_request_with_custom_auth_header
* Using a different Authentication Header
* fix customize MCP Auth header name
* fix security - mcp
* fix(router.py): validate model provider before adding deployment to pattern
prevents routing on pattern match to invalid deployment
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* Fix: Properly close aiohttp client sessions to prevent resource leaks (#12107)
- Add close() method to BaseLLMAIOHTTPHandler to properly close aiohttp ClientSession
- Create async_client_cleanup module with utility functions to close all cached async clients
- Register automatic cleanup at exit via atexit hook
- Export close_litellm_async_clients() function for manual cleanup
- Add comprehensive tests to verify resource cleanup
This fixes the "Unclosed client session" and "Unclosed connector" warnings when using acompletion with Gemini and other models that use aiohttp.
Fixes#12107
* Fix: Remove unused import to satisfy linter
* Fix: Extend cleanup to handle AsyncHTTPHandler instances used by Gemini
The original implementation only cleaned up BaseLLMAIOHTTPHandler instances,
but Gemini/Vertex AI providers use AsyncHTTPHandler objects which contain
httpx clients with aiohttp transports.
This commit extends the cleanup function to:
- Handle AsyncHTTPHandler instances by accessing their internal client
- Close both the aiohttp transport and httpx client
- Add generic fallback for any objects with aclose method
This properly fixes the resource leak warnings for all provider types.
* fix(common_daily_activity.py): initial commit with working mock BE endpoint for mcp usage
* feat(ui/): show mcp server activity on UI
allows admin to know which mcp's are being used
* feat(common_daily_activity.py): return activity by key
* feat(ui/): show top api keys for a given model / mcp server
allow user to know which key is driving spend
* fix(common_daily_activity.py): use known mcp server names
* feat(server.py): log the namespaced tool name (includes server prefix)
allow accurate cost tracking
* feat(db_spend_update_writer.py): log by mcp_namespaced_tool_name
store aggregate daily activity by mcp_namespaced_tool_name
Enables cost / usage tracking by mcp tool name
* fix(server.py): add key/user metadata to mcp calls
* refactor(common_daily_activity.py): update to return mcp activity in API
* fix(common_daily_activity.py): handle empty key
* fix(common_daily_activity.py): track when api key is empty
* test(test_spend_management_endpoints.py): update tests
* fix: fix ui linting error
* fix: fix linting errors
* test: add missing key
* build(schema.prisma): add mcp tool tracking
* fix(migration.sql): add schema migration file
* feat(server.py): add request logging for mcp calls
enables storing the mcp calls
* fix(new_usage.tsx): fix linting errors
* fix: fix code qa errors
* fix(activity_metrics.tsx): fix ui linting errors post-merge
* fix(types/utils.py): fix linting error
* fix(server.py): always have name
Fix the header to highlight the website guardrailsai.com which is the real name/website of this provider.
Guardrails.ai leads to some different website, which is confusing.
```bash
LiteLLM:ERROR: server.py:370 - Error handling MCP request: Task group is not initialized. Make sure to use run().
Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/litellm/proxy/_experimental/mcp_server/server.py", line 368, in handle_streamable_http_mcp
await session_manager.handle_request(scope, receive, send)
File "/usr/lib/python3.13/site-packages/mcp/server/streamable_http_manager.py", line 137, in handle_request
raise RuntimeError("Task group is not initialized. Make sure to use run().")
RuntimeError: Task group is not initialized. Make sure to use run().
```
The use of asyncio.create_task with an infinite loop is not ideal for integration with FastAPI's lifespan events. Now correctly handles this by manually invoking the __aenter__ and __aexit__ methods of the session managers' async context managers within the application's lifespan.
* Enhance chart readability with short-form notation for large numbers (#11571)
* format y-axis value for total tokens
* format y-axis for the rest of the charts on model-activity
* revert changes for requests per day
* labels modified to plain text
* added plain text label for api_requests and spend
* minor
* improve keyname display (#11573)
* fix tags overlap
- no need to fix decimal points for valueFormatterSpend() bcz we're only going to use it for values on the x and y axis of a chart
- show only top 5 values of tags
- format the values on the x-axis
* format spend values to short form
* format spend to short form
* showing top 5 tags data more perceptively
* remove mock data
* remove console.log
* prettier
* prettier + remove unused functions
* Add _transform_input_image_item_to_image_item() to handle image transformations
* Update content transformation to process input_image items from Responses API format to Chat Completions format
