Commit Graph
148 Commits
Author SHA1 Message Date
Krish DholakiaandGitHub db498c1805 Fix team_member_budget update logic (#12843)
* fix(team_endpoints.py): always remove team member budget from updated_kv

this is not a field for the litellm team table

Prevents startup issue

* test(test_team_endpoints.py): add unit test to ensure 'team_member_budget' is never in update to table - separate logic

* refactor: cleanup
2025-07-21 22:06:29 -07:00
tanjiroandGitHub a752d7acc9 Fix SSO Logout | Create Unified Login Page with SSO and Username/Password Options (#12703) 2025-07-20 20:24:38 -07:00
Krish DholakiaandGitHub 0c461b2719 Litellm fix proxy unit testing p2 (#12779)
* test: update tests

* test: update test

* test: update unit tests
2025-07-19 16:35:05 -07:00
Cole McIntoshandGitHub b04b456bc2 fix(proxy): Fix Model Armor project_id initialization order (#12766)
When using Model Armor guardrail with explicit project_id in config,
the project_id was being overwritten to None due to incorrect
initialization order between ModelArmorGuardrail and VertexBase parent class.

This fix ensures that user-provided project_id is preserved by initializing
parent classes before setting instance attributes.

Fixes #12757
2025-07-19 15:20:40 -07:00
Jugal D. BhattandGitHub b443817a56 [Key Access] Litellm disabled callbacks for UI (#12769)
* add disabled callbacks to ui

* added body

* update edit settings

* add tests
2025-07-19 14:32:05 -07:00
Ishaan JaffandGitHub 5802a5bbe3 [Feat] LLM API Endpoint - Expose OpenAI Compatible /vector_stores/{vector_store_id}/search endpoint (#12749)
* fix _pass_through_endpoint_without_required_model

* add get_litellm_managed_vector_store_from_registry

* undo router change

* fix for using router + vector search methods

* add simple helper for _update_request_data_with_litellm_managed_vector_store_registry

* add vector_stores routes

* test_router_avector_store_search_passes_correct_args

* [Feat] UI - Allow clicking into Vector Stores (#12741)

* Add View Vector Store

* add /info for vector store

* fix updated_at

* allow easily testing the KB on litellm

* fix

* rename test

* test_init_vector_store_api_endpoints

* test_update_request_data_with_litellm_managed_vector_store_registry
2025-07-18 18:18:53 -07:00
Cole McIntoshandGitHub 506dc80b15 feat: integrate Google Cloud Model Armor guardrails (#12492)
* feat: integrate Google Cloud Model Armor guardrails (LIT-298)

- Add ModelArmorGuardrail class that extends CustomGuardrail and VertexBase
- Support for both pre-call (sanitizeUserPrompt) and post-call (sanitizeModelResponse) sanitization
- Integrate with existing Vertex AI authentication using VertexBase
- Add configuration model for Model Armor in guardrail types
- Register Model Armor in guardrail initializers and registry
- Include comprehensive test suite for Model Armor functionality
- Support content masking for both requests and responses
- Handle streaming responses with content sanitization

This integration allows LiteLLM to use Google Cloud Model Armor API for
content moderation and sanitization, providing similar functionality to
Bedrock Guards but using Google Cloud's security infrastructure.

* fix: remove unused imports flagged by ruff linter

- Remove unused asyncio import at top level (moved to local import where needed)
- Remove unused TextCompletionResponse import

* fix: remove additional unused imports

- Remove unused TextChoices import from line 34
- Remove duplicate asyncio import from line 384
- Replace asyncio.iscoroutine() with hasattr check for __await__

* fix: remove final unused imports

- Remove unused 'import sys' from line 9
- Remove unused 'StreamingChoices' from imports

* fix: remove unused import from model_armor.py

- Remove unused 'import os' from the top of the file

* fix: remove commented-out header from model_armor.py

- Eliminate unnecessary comments at the top of the file to improve code clarity.

* feat(guardrails): Add Model Armor UI support

- Convert model_armor.py to a directory structure with __init__.py for dynamic discovery
- Add get_config_model() method to ModelArmorGuardrail class for UI integration
- Add ui_friendly_name() to ModelArmorConfigModel returning "Google Cloud Model Armor"
- Remove manual registration from guardrail_registry.py to use dynamic discovery
- Model Armor now appears in the UI guardrails dropdown with proper configuration fields

This enables users to configure Model Armor guardrails through the LiteLLM UI interface.

* fix(guardrails): Fix undefined name 'GuardrailConfigModel' in Model Armor

- Import TYPE_CHECKING and GuardrailConfigModel from base module
- Fixes F821 linting error for undefined name in type annotation
- Follows same pattern as other guardrail implementations

* fix(guardrails): Fix Model Armor type errors and config model inheritance

- Create ModelArmorGuardrailConfigModel that properly inherits from GuardrailConfigModel base class
- Move config model to litellm/types/proxy/guardrails/guardrail_hooks/model_armor.py following convention
- Update get_config_model() to return the properly typed config model
- Remove ModelArmorConfigModel from LitellmParams inheritance chain
- Add template_id field to BaseLitellmParams instead

This fixes the mypy type errors and follows the same pattern as other guardrail implementations.

* fix(guardrails): Add missing Model Armor fields to BaseLitellmParams

- Add location, credentials, api_endpoint, and fail_on_error fields
- Fixes mypy errors about missing attributes in LitellmParams
- All Model Armor configuration parameters are now properly defined

* fix: Apply PR review feedback for Model Armor guardrail

- Move test file to tests/test_litellm/ for GitHub Actions
- Extract only last consecutive user messages to avoid context limits
- Use get_content_from_model_response helper for response extraction
- Handle non-ModelResponse types (e.g., TTS) gracefully
- Maintain newline separation for multi-part content

* refactor: Simplify message content extraction in ModelArmorGuardrail

- Removed the custom _extract_content_from_messages method.
- Integrated get_last_user_message helper for improved content extraction.
- Updated test to reflect changes in content formatting.

* refactor: Remove unused import in model_armor.py

- Deleted the unused import of AllMessageValues to clean up the codebase.

* Add unit tests for Model Armor guardrail functionality

- Implement tests for pre-call and post-call hooks, including content sanitization and blocking behavior.
- Validate error handling for API responses and credential management.
- Test streaming responses and handling of list content in user messages.
- Ensure proper assertions for API interactions and response sanitization.

* Add comprehensive test coverage for Model Armor guardrail

- Add test for requests with no messages field
- Add test for empty message content handling
- Add test for system/assistant-only messages
- Add test for fail_on_error=False behavior
- Add test for custom API endpoint configuration
- Add test for dictionary credentials (non-file path)
- Add test for action=NONE response handling
- Add test for missing sanitized_text field fallback
- Add test for non-text response types (TTS/image)
- Add test for auth token refresh behavior

All tests ensure robust edge case handling and proper error management.

* feat: improve Model Armor handling of non-ModelResponse types

- Add debug logging when skipping non-text responses (TTS, images, etc.)
- Improve docstring to clarify behavior for non-text responses
- Add test coverage for non-ModelResponse handling
- Ensure guardrail gracefully skips processing for response types it cannot handle
2025-07-18 12:51:09 -07:00
Krish DholakiaandGitHub 5004b915d1 Guardrails AI - support llmOutput based guardrails as pre-call hooks (#12674)
* build: move build_and_test to use prisma migrate

* fix(guardrails_ai.py): default to guardrail accepting 'llmOutput' as the input param

enables same guardrail to work for pre call and post call

* fix(__init__.py): set default value

* fix(guardrails_ai.py): updates

* fix: fix linting error
2025-07-18 11:31:30 -07:00
Jugal D. BhattandGitHub a112ec5b02 Health check app on separate port (#12718)
* add separate health app

* add new docs

* refactor

* fix colons

* Update config_settings.md

* refactor

* docs

* add unit test

* added supervisord

* remove app

* add supervisor conf

* Add markdown

* add video to md

* remove test

* docs build failure

* add to all docker files, change prod.md and add tests

* change dockerfiles

* remove extra file

* remove extra file

* remove extra file

* change apt->apk

* remove rdb file

* add fixed file
2025-07-18 11:17:15 -07:00
Cole McIntoshandGitHub 491555d32d fix(test_team_endpoints.py): fix AsyncMock error in test_new_team_with_object_permission (#12730)
Replace MagicMock with AsyncMock for litellm_teamtable.update to fix:
TypeError: object MagicMock can't be used in 'await' expression

The test was failing because it tried to await a MagicMock object.
Added AsyncMock for the update method to properly handle async operations.
2025-07-18 07:25:16 -07:00
Krish DholakiaandGitHub b77c9f5de2 fix(team_endpoints.py): ensure user id correctly added when new team … (#12719)
* fix(team_endpoints.py): ensure user id correctly added when new team created with user email as member

Fixes issue where user not correctly added to team on /team/new

* fix(internal_user_endpoints.py): make user email validation check case insensitive

Fixes issue where uppercase email was added even when lowercase email existed

* test: update test
2025-07-17 22:31:44 -07:00
Krish DholakiaandGitHub b515d051ff Litellm encrypt admin UI values (#12675)
* build: move build_and_test to use prisma migrate

* feat(proxy_setting_endpoints.py): encrypt env var before storing in db

Ensures env var can be read when loaded in from DB

Fixes issue when trying to add SSO from admin UI

* test: update tests
2025-07-17 22:24:58 -07:00
Jugal D. BhattandGitHub 4b09d0d517 [Liveness/Liveliness probe] add separate health app for liveness probes in files (#12669)
* add separate health app

* add new docs

* refactor

* fix colons

* Update config_settings.md

* refactor

* docs

* add unit test
2025-07-16 20:35:09 -07:00
Ishaan JaffandGitHub 43e7d5288f [Bug Fix] SCIM - add GET /ServiceProviderConfig (#12664)
* add scimv2 config endpoint

* test_get_service_provider_config
2025-07-16 15:43:25 -07:00
Jugal D. BhattandGitHub 83b0c4cba7 [MCP Gateway] Allow MCP sse and http to have namespaced url for better segregation LIT-304 (#12658)
* fix tools fetch for keys

* Add namespacing in url

* add test for namespacing url

* helper method

* fix test
2025-07-16 14:47:45 -07:00
Cole McIntoshandGitHub 79e4d77bcf fix: Handle circular references in spend tracking metadata JSON serialization (#12643)
* fix: Handle circular references in spend tracking metadata JSON serialization

- Fixes issue #12634 where circular references in metadata caused
  ValueError: Circular reference detected when logging spend data
- Adds _safe_json_dumps() function that detects and handles circular
  references by replacing them with placeholder strings
- Maintains full functionality for normal objects while preventing
  crashes from circular references
- Adds comprehensive tests for circular reference handling
- Critical fix for v1.74.3 stable release

* fix: Replace bare except clauses with specific Exception handling

- Fixes E722 linting errors in _safe_json_dumps function
- Maintains same error handling behavior while following best practices
- All tests continue to pass

* refactor: Use existing safe_dumps utility instead of custom implementation

- Replace custom _safe_json_dumps() with existing safe_dumps() from litellm_core_utils
- Remove duplicate code and leverage existing circular reference handling
- Update tests to use safe_dumps function
- Maintains same functionality while reducing code duplication
- All tests continue to pass
2025-07-16 07:24:41 -07:00
Marc AbramowitzandGitHub 40ccd2b70f Add "keys import" command to CLI (#12620)
* Add "keys import" command to CLI

E.g.:

```
litellm-proxy keys import \
  --source-base-url=https://old-litellm.company.com \
  --source-api-key=$LITELLM_KEY \
  --dry-run
```

* Add --created-since option

* Add tests

* Fix lint errors

* Fix lint issues

* Fix lint errors

* Fix response.raise_for_status not being a thing

* Fix a mypy error
2025-07-15 20:14:43 -07:00
Ishaan JaffandGitHub 46c950b17b [Bug Fix] Add swagger docs for LiteLLM /chat/completions, /embeddings, /responses (#12618)
* update ProxyChatCompletionRequest

* use custom OpenAPI schema

* use customize_openapi_schema

* add_llm_api_request_schema_body

* add embeddings and response spec

* fix add_llm_api_request_schema_body

* TestCustomOpenAPISpec

* fixes linting

* fix linting

* bump timeout
2025-07-15 13:37:22 -07:00
Ishaan JaffandGitHub d2fe8894c8 [Bug Fix] Include /mcp in list of available routes on proxy (#12612)
* create GetRoutes helper class

* update get routes

* TestGetRoutes

* fix get_routes_for_mounted_app
2025-07-15 09:24:48 -07:00
Krish DholakiaandGitHub 7c392475e6 Control Plane + Data Plane support (#12601)
* feat(route_checks.py): allow admin to disable proxy management endpoints on instance

useful for preventing multiple instances from doing admin actions

* docs(scaling_multiple_instances.md): add architecture doc on scaling multiple litellm instances

provide guidance on scaling proxy

* docs(scaling_multiple_instances.md): add doc on scaling across multiple regions for litellm

* fix(route_checks.py): allow disabling llm api endpoints on an instance

allows pure admin instance to exist

* refactor(enterprise/route_checks.py): refactor env var checks

* refactor: finish refactoring

* docs(control_plane_and_data_plane.md): refactor docs

* test: update tests
2025-07-14 21:31:56 -07:00
Ishaan JaffandGitHub 50b106be11 [MCP Gateway] Ensure we use the same param for specifying groups (#12561)
* just use 1 param for mcp groups

* fix just use 1 param for access groups

* test_get_tools_from_mcp_servers

* docs access groups

* group MCPs

* test fix

* fix screenshots on docs

* TestMCPAccessGroupsE2E

* update img

* fix MCP connect
2025-07-12 16:41:58 -07:00
Jugal D. BhattandGitHub 9fe9e1cd6e [MCP Gateway] Allow mcp access groups on test key and tool calls (#12529)
* added mcp tools on internal user and divide it by teams

* add support for server api call

* Added frontend for test key

* added tools used output

* fix ui for servers

* All servers to personal

* change columns format

* revert ui logic

* Added vertical align

* fix mapped tests

* fix lint

* fix lint

* remove extra file

* fix ui test

* comments fixes

* change query type

* change query type

* mcp acces group init

* add ability to change server display on ui through access groups

* Mcp access group names UI (#12486)

* Added ui changes to reflect mcp_access_groups

* fix edit mcp page

* change to string array (#12491)

* change to string array

* Remove print

* add ability to change server display on ui through access groups

* Litellm mcp access groups accesses (#12498)

* added mcp access groups for keys and teams

* added access groups above servers

* fixed ruff

* fixed mypy

* revert couple changes

* fix test

* fixed double asterisks

* Litellm mcp groups UI (#12522)

* add ui for teams

* fix object permissions

* fix mcp servers test object permission

* remove print

* add helper method

* add tests + remove logs

* add mcp access group servers to test key

* add mcp access group support for headers

* lint fix

* add tests and helper  function

* fixed test

* change list -> List

* tests
2025-07-12 10:36:42 -07:00
Krish DholakiaandGitHub d202ce229b Prevent writing default user setting updates to yaml (error in non-root env) + Use central team member budget when max_budget_in_team set on UI (#12533)
* fix(proxy_setting_endpoints.py): require store model in db is enabled for setting user default settings

* test(test_proxy_server.py): update test

* fix(reset_budget_job.py): initial commit adding reset budget logic for team members

* test: update unit testing

* test(test_proxy_budget_reset.py): validate team member budget was reset

* test(test_reset_budget_job.py): update unit tests

* test: update tests
2025-07-12 10:13:07 -07:00
Krish DholakiaandGitHub 749051105b Team Members - reset budget, if duration set + Prometheus - support tag based metrics (#12534)
* fix(internal_user_endpoints.py): initial commit removing logic to create new budget for new user if default max budget in team set

* feat(proxy_setting_endpoints.py): update team member budget when set via default internal user endpoint

removes need to create a unique budget per user

* feat(proxy_server.py): set team max member budget on startup, if set on config.yaml

* fix(prometheus.py): support custom tags for tracking on prometheus

Allows tracking user agent values on prometheus metrics

* test(test_internal_user_endpoints.py): fix test
2025-07-11 22:54:16 -07:00
Ishaan JaffandGitHub d931446a79 [MCP Gateway] Allow using stdio MCPs with LiteLLM (#12530)
* update types

* add STDIO to client

* allow using STDIO with MCP manager

* add Stdio-specific fields to schema.prisma

* fixes for MCP mgmt

* fix for adding stdio MCP to DB

* ui - allow adding stdio MCPs

* fix MCP server manager

* docs stdio MCP

* add_stdio_mcp.png

* new stdio tests

* allow adding MCPs through config.yaml

* fix tool test panel

* use TestMCPClient

* ui fixes for testing circle ci mcp
2025-07-11 20:21:02 -07:00
Jugal D. BhattandGitHub 38c9b69e8f [MCP Gateway] access group UI object permission fix (#12523)
* added mcp tools on internal user and divide it by teams

* add support for server api call

* Added frontend for test key

* added tools used output

* fix ui for servers

* All servers to personal

* change columns format

* revert ui logic

* Added vertical align

* fix mapped tests

* fix lint

* fix lint

* remove extra file

* fix ui test

* comments fixes

* change query type

* change query type

* mcp acces group init

* add ability to change server display on ui through access groups

* Mcp access group names UI (#12486)

* Added ui changes to reflect mcp_access_groups

* fix edit mcp page

* change to string array (#12491)

* change to string array

* Remove print

* add ability to change server display on ui through access groups

* Litellm mcp access groups accesses (#12498)

* added mcp access groups for keys and teams

* added access groups above servers

* fixed ruff

* fixed mypy

* revert couple changes

* fix test

* fixed double asterisks

* Litellm mcp groups UI (#12522)

* add ui for teams

* fix object permissions

* fix mcp servers test object permission

* remove print

* add helper method

* add tests + remove logs
2025-07-11 18:25:40 -07:00
Ishaan Jaff 7ac187b269 test_pre_call_hook_team_rpm_limits 2025-07-11 15:52:21 -07:00
Ishaan JaffandGitHub cb96f1d740 [Feat] - New guardrail - OpenAI Moderations API (#12519)
* add OpenAIModerationGuardrailConfigModel

* add OpenAIGuardrailBase

* _extract_response_text

* fixes for openai moderation guard

* docs OpenAI Moderation

* add OpenAIModerationGuardrailConfigModel

* use literal for model

* fix init openai moderation

* test_openai_moderation_guardrail_adds_to_litellm_callbacks

* show default on on UI

* fix code qa
2025-07-11 11:25:45 -07:00
Jugal D. BhattandGitHub 1b10dda2f4 Litellm mcp access group (#12514)
* added mcp tools on internal user and divide it by teams

* add support for server api call

* Added frontend for test key

* added tools used output

* fix ui for servers

* All servers to personal

* change columns format

* revert ui logic

* Added vertical align

* fix mapped tests

* fix lint

* fix lint

* remove extra file

* fix ui test

* comments fixes

* change query type

* change query type

* mcp acces group init

* add ability to change server display on ui through access groups

* Mcp access group names UI (#12486)

* Added ui changes to reflect mcp_access_groups

* fix edit mcp page

* change to string array (#12491)

* change to string array

* Remove print

* add ability to change server display on ui through access groups

* Litellm mcp access groups accesses (#12498)

* added mcp access groups for keys and teams

* added access groups above servers

* fixed ruff

* fixed mypy

* revert couple changes

* fix test

* fixed double asterisks
2025-07-11 07:50:18 -07:00
Krish DholakiaandGitHub 9d9562d0e8 Guardrails AI - pre-call + logging only guardrail (pii detection/competitor names) support (#12506)
* fix(guardrails_ai.py): initial commit adding pre-call hook support for guardrails ai

enables running user input through guardrails ai - if set

* feat(guardrails_ai.py): working pre call guardrail

enables pii detection to work via guardrails ai

* feat(guardrails_ai.py): support logging hook

enables masking input via guardrails ai on logging integrations

* test(test_guardrails_ai.py): add unit test for new input processing function
2025-07-10 21:41:02 -07:00
Jugal D. BhattandGitHub 0691ff8d13 Added validate payload error (#12494)
* Added validate payload error

* added logger

* added test case
2025-07-10 15:54:48 -07:00
Ishaan JaffandGitHub a0d6900223 [Bug Fix] fix parsing environment_variables from config.yaml (arize logger integration fix) (#12482)
* fix parsing environment_variables from config.yaml

* add tests
2025-07-10 09:15:07 -07:00
Krish DholakiaandGitHub 07e8609edb Resolve model group alias on Auth + /v1/messages Fallback support (#12440)
* fix(auth_checks.py): resolve a model group alias when key has access to underlying model

Fixes LIT-293

* feat(anthropic/): add mock_response to anthropic /v1/messages

makes it easy to test fallback logic

* fix(router.py): support fallbacks on /v1/messages

adds working fallbacks on generic api route

* refactor(router.py): point _ageneric_api_call_with_fallbacks to updated function

* test: add unit test for new helper on router

* fix(router.py): use correct metadata variable name

* fix(router.py): use correct metadata field

* docs(config_settings.md): document new param
2025-07-09 22:27:55 -07:00
Ishaan JaffandGitHub 17ea770ab5 [Feat] SSO - Allow users to run a custom sso login handler (#12465)
* fix return_ui_sso_redirect_response_after_sign_in

* add CustomSSOLoginHandler

* add handle_custom_ui_sso_sign_in

* fix handle_custom_ui_sso_sign_in

* fixes for /sso/key/generate

* add test_handle_custom_ui_sso_sign_in_success

* fixed for loading config

* add example custom handler

* test_custom_ui_sso_sign_in_handler_config_loading

* docs - custom sso

* move to enterprise folder

* EnterpriseCustomSSOHandler

* handle_custom_ui_sso_sign_in

* docs custom sso

* docs sso

* docs custom sso

* get_redirect_response_from_openid

* fix mock tests
2025-07-09 16:36:30 -07:00
Jugal D. BhattandGitHub a12bbe7da4 Litellm mcp internal users (#12458)
* added mcp tools on internal user and divide it by teams

* add support for server api call

* Added frontend for test key

* added tools used output

* fix ui for servers

* All servers to personal

* change columns format

* revert ui logic

* Added vertical align

* fix mapped tests

* fix lint

* fix lint

* remove extra file

* fix ui test

* comments fixes

* change query type

* change query type

* change query type

* Update model_checks.py
2025-07-09 16:30:51 -07:00
Ishaan JaffandGitHub afd382d09f [Feat] MCP Gateway - Allow customizing what client side header to use (#12460)
* add _get_mcp_auth_header_from_headers

* test_process_mcp_request_with_custom_auth_header

* Using a different Authentication Header

* fix customize MCP Auth header name
2025-07-09 12:30:20 -07:00
Krish DholakiaandGitHub c42880d771 fix(utils.py): rollback faulty security check on files (#12441)
* fix(utils.py): rollback faulty security check on files

Closes https://github.com/BerriAI/litellm/issues/11009

* fix(route_checks.py): add unit tests
2025-07-08 22:11:06 -07:00
Krish DholakiaandGitHub c4af2eb5e2 MCP - usage tracking (#12397)
* fix(common_daily_activity.py): initial commit with working mock BE endpoint for mcp usage

* feat(ui/): show mcp server activity on UI

allows admin to know which mcp's are being used

* feat(common_daily_activity.py): return activity by key

* feat(ui/): show top api keys for a given model / mcp server

allow user to know which key is driving spend

* fix(common_daily_activity.py): use known mcp server names

* feat(server.py): log the namespaced tool name (includes server prefix)

allow accurate cost tracking

* feat(db_spend_update_writer.py): log by mcp_namespaced_tool_name

store aggregate daily activity by mcp_namespaced_tool_name

Enables cost / usage tracking by mcp tool name

* fix(server.py): add key/user metadata to mcp calls

* refactor(common_daily_activity.py): update to return mcp activity in API

* fix(common_daily_activity.py): handle empty key

* fix(common_daily_activity.py): track when api key is empty

* test(test_spend_management_endpoints.py): update tests

* fix: fix ui linting error

* fix: fix linting errors

* test: add missing key

* build(schema.prisma): add mcp tool tracking

* fix(migration.sql): add schema migration file

* feat(server.py): add request logging for mcp calls

enables storing the mcp calls

* fix(new_usage.tsx): fix linting errors

* fix: fix code qa errors

* fix(activity_metrics.tsx): fix ui linting errors post-merge

* fix(types/utils.py): fix linting error

* fix(server.py): always have name
2025-07-08 22:08:16 -07:00
Ishaan JaffandGitHub d720b3d369 [Bug fix] Multiple API Keys Created on Startup When max_budget is Enabled (#12436)
* fix _add_proxy_budget_to_db

* fix if table_name is not None and table_name == "user":

* revert earlier change

* test_add_proxy_budget_to_db_only_creates_user_no_keys

* lint fix
2025-07-08 16:46:05 -07:00
Ishaan JaffandGitHub 5cad0dd94b [Feat] Add MCP Cost Tracking (#12385)
* fix MCP_TOOL_NAME_PREFIX

* test_mcp_cost_tracking

* init MCPCostCalculator

* add call_mcp_tool

* calculate_mcp_tool_call_cost

* add MCPServerCostInfo

* add mcp_server_cost_info

* add mcp_server_cost_info

* pass through litellm_logging_obj

* logged_standard_logging_payload

* fix logging MCP tool call

* test_mcp_cost_tracking_per_tool

* fix NewMCPServerRequest

* fix add_update_server

* add MCP info to schema.prisma

* fix create_mcp_server

* working custom cost per call

* fix mcp server cost

* fix MCPCostCalculator

* TestMCPCostCalculator
2025-07-07 21:08:10 -07:00
Ishaan JaffandGitHub 0a36f89009 [Bug Fix] s3 config.yaml file - ensure yaml safe load is used (#12373)
* use yaml safe load

* test_get_file_contents_from_s3_no_temp_file_creation
2025-07-07 14:42:33 -07:00
Krish DholakiaandGitHub bda9eecd45 Litellm dev 07 05 2025 p3 (#12349)
* refactor(aim.py): refactor to support adding aim guardrails on UI

* fix(base.py): add ui_friendly_name to config model

* feat(ui/): support loading new guardrails from backend api call

removes need to onboard each guardrail to ui

* fix: don't show optional params if not set and don't show ui_friendly_name (internal param0

* fix(ui/add_guardrail_form.tsx): ensure dynamic provider value is used

* fix(ui/): just one-time update the provider map dictionary

* fix(ui/): show masked api base / api key on guardrail update

* refactor(aporia_ai/): refactor to show on UI

* feat(aporia_ai/): add aporia ai guardrail to UI

* refactor(guardrails_ai/): refactor to add via UI

* refactor(lasso.py): refactor to enable adding lasso guardrails via UI

* feat(pangea.py): add pangea guardrail on UI

* feat(panw): add panw prisma airs through UI

* test: update tests

* fix: fix ruff linting error

* test: update tests

* fix: add missing docs

* fix: fix guardrail init

* fix: suppress linting errors

* fix(proxy_server.py): fix linting error
2025-07-05 18:44:00 -07:00
Krish DholakiaandGitHub e7f1fa26ab UI - Azure Content Guardrails (#12341)
* build(model_prices_and_context_window.json): remove 'supports_tool_choice' for specific mistral models

Closes https://github.com/BerriAI/litellm/issues/11750

* feat: initial commit adding cleaner ui for azure text moderation guardrails

* feat(guardrail_endpoints.py): add discoverable guardrail configs and improve converting base model to dict with types

* fix(guardrail_provider_fields.tsx): render from api endpoint correctly

* fix(guardrail_provider_fields.tsx): cleanup

* refactor(guardrail_endpoints.py): refactor to handle dictionaries with literal - allows multiselect

* feat(ui/): render dictionary with known keys correctly

* feat(ui/): render optional params on separate page

* style(ui/): style improvements to rendering optional params on the UI

* feat(azure/prompt_shield.py): add azure prompt shield back on UI

* fix(add_guardrail_form.tsx): fix form to handle updated api

* fix(guardrail_optional_params.tsx): ensure values are nested correctly for writing to api

* fix: fix linting error

* feat(text_moderation.py): handle str to int conversion

* fix(guardrail_info.tsx): only render pii settings if guardrail is presidio

* fix(guardrail_info.tsx): add guardrail specific fields to update settings

allows updating guardrail fields (e.g. severity threshold) post-create

* fix(guardrail_endpoints.py): set guardrail_id in guardrail object

ensures duplicate objects not created on guardrail update

* fix(guardrail_endpoints.py): allow provider specific fields to be updated on patch update

* refactor(guardrail_endpoints.py): remove duplicate info endpoint

* fix(guardrail_endpoints.py): mask sensitive keys on returning via guardrail `/info`

Prevent leaking keys

* fix(guardrail_optional_params.tsx): return numerical input when numerical component used

fixes issue where output was a str

* fix(guardrail_optional_params.tsx): render dict keys correctly

* fix(text_moderation.py): fix severity by category check

* fix(proxy/utils.py): check if guardrail should run for post call streaming hook

Prevents invalid guardrails from running if not requested

* test: fix import

* fix: fix linting error

* test: update test

* fix: fix tests

* fix: fix code qa errors

* fix(guardrail_endpoints.py): set max depth for function

* test: update recursive_detector.py

* test: update list

* build: merge main

* fix: fix ruff check errors
2025-07-05 10:19:29 -07:00
+1 df47008818 UI - Add Azure Content Safety Guardrails (improved UX) (#12330)
* build(model_prices_and_context_window.json): remove 'supports_tool_choice' for specific mistral models

Closes https://github.com/BerriAI/litellm/issues/11750

* feat: initial commit adding cleaner ui for azure text moderation guardrails

* feat(guardrail_endpoints.py): add discoverable guardrail configs and improve converting base model to dict with types

* fix(guardrail_provider_fields.tsx): render from api endpoint correctly

* fix(guardrail_provider_fields.tsx): cleanup

* refactor(guardrail_endpoints.py): refactor to handle dictionaries with literal - allows multiselect

* feat(ui/): render dictionary with known keys correctly

* feat(ui/): render optional params on separate page

* style(ui/): style improvements to rendering optional params on the UI

* feat(azure/prompt_shield.py): add azure prompt shield back on UI

* fix(add_guardrail_form.tsx): fix form to handle updated api

* fix(guardrail_optional_params.tsx): ensure values are nested correctly for writing to api

* fix: fix linting error

* test: update tests

* fix mapped tests (#12320)

* fix - use flush llm client cache

* faster mapped tests

* test_async_multiple_response_ids_routing

* fix tests

* test_ateam_member_update_admin_requires_premium

* regular mapped tests

* Revert "Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)"

This reverts commit 2c60c316ec.

* reset num workers

* check_for_litellm_module_deletion

* add test

* add code_qa_check_tests

* ci/cd new release

* docs(opentelemetry_integration.md): add otel debug flag to docs

help people debug their traces easily

* [Feat] Add failure logging support for s3 logger (#12299)

* add async_log_failure_event

* test_basic_s3_v2_logging_failure

* Comma separated spend and budget (#12317)

* all_keys_table, key_info_view, organizations

* teams

* the whole usage page

* teams page

* users table

* user info

* more commas

* some more commas

* Segregate MCP tools on connections using headers (#12296)

* Add get tools segregation

* add ui changes (#12302)

* resolve comments

* add mapped tests

* remove advanced settings (#12323)

* [Bug Fix] /generateContent API - Only pass supported params when using OpenAI models (#12297)

* fix - only pass GenericLiteLLMParams

* test_google_generate_content_with_openai

* Fix: Fix custom ca bundle support in aiohttp transport (#12281)

* Unify usage of get_ssl_configuration

* Fix doc

* [Feat] Add github co-pilot as a new LLM API provider (#12325)

* Litellm dev 03 05 2025 contributor prs (#9079)

* feat: add support for copilot provider

* test: add tests for github copilot

* chore: clean up github copilot authenticator

* test: add test for github copilot authenticator

* test: add test for github copilot for sonnet 3.7 thought model

* Fix #7629 - Add tzdata package to Dockerfile (#8915)

* Add tzdata package to Dockerfile

* Move tzdata to python requirement.txt

* feat: add support for copilot provider (#8577)

* feat: add support for copilot provider

* test: add tests for github copilot

* chore: clean up github copilot authenticator

* test: add test for github copilot authenticator

* test: add test for github copilot for sonnet 3.7 thought model

---------

Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>

* feat: add model information for copilot models

* fix: fix linting errors

* test: remove integration test for github_copilot + fix misisng mock

* fix: use print to make sure the logger message shown

* test: remove debug print

* fix lint (#11112)

* Add init files to make test directories Python packages and update import paths in test_token_counter.py (#11119)

* Update litellm/model_prices_and_context_window_backup.json

Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>

---------

Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>

* refactor github copilot

* test_github_copilot_transformation.py

* test_github_copilot_authenticator.py

* add GitHub Copilot

* fix order

* doc fix

---------

Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>

* add new env vars to docs

* ssl_cert_error (#12327)

* feat - add new banner on startup (#12328)

* [Security Bug Fix] Ensure only LLM API route fails get logged on Langfuse (and other loggers) (#12308)

* _is_proxy_only_llm_api_error

* test_proxy_only_error_true_for_llm_route

* add not on change

* Update tests/test_litellm/proxy/test_proxy_utils.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* add test_post_call_failure_hook_auth_error_key_info_route

* test fix _is_proxy_only_llm_api_error

* test_chat_completion_request_with_redaction

* test_post_call_failure_hook_auth_error_llm_api_route

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Ensure message redaction works for responses API logging (#12291)

* add fixes to choice implementation redaction

* add isInstance check on responses API

* change datadog to revert back

* change datadog to revert back

* fix type errors

* Redaction test changes

* Redaction test changes

* Redaction test changes - remove changes

* [Feat] UI - Allow Viewing/Editing Team Based Callbacks (#12329)

* add logging settings view on UI

* fix change ordering

* fix type error

* test fix - import loc

* [Feat] QA -  Arize Team based logging (#12331)

* add _get_tracer_with_dynamic_headers

* fix construct_dynamic_arize_headers

* [Feat] UI - Allow Viewing/Editing Team Based Callbacks (#12329)

* add logging settings view on UI

* fix change ordering

* add construct_dynamic_otel_headers for arize

* refactor common code

* test_construct_dynamic_arize_headers

* otel unit tests

* test_arize_dynamic_params

* test_arize_dynamic_headers_in_grpc_requests

* [Feat] UI - Allow setting Logging Callback Setting per Key (#12333)

* allow adding logging settings on a key

* add ability to edit key logging settings

* dont leak api keys on ui

* fix order

* add langsmith settings

* ui new build

* test_default_api_base

* fix new utils tests

* bump: version 1.73.7 → 1.74.0

* Add mcp server segregation comma separated support (#12326)

* add mcp comma separated support

* dont support legacy json array

* lint

* add live tail state to session storage (#12335)

* add aiml

* fix: fix unused imports

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: tanjiro <56165694+NANDINI-star@users.noreply.github.com>
Co-authored-by: Jugal D. Bhatt <55304795+jugaldb@users.noreply.github.com>
Co-authored-by: Joost van Doorn <joost.van.doorn@gmail.com>
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-05 09:24:54 -07:00
Ishaan JaffandGitHub 738db9336e [Feat] JWT - Sync user roles and team memberships when JWT Auth is used (#11994)
* add JWTLiteLLMRoleMap

* test_sync_user_role_and_teams

* add sync_user_role_and_teams

* test_sync_user_role_and_teams

* fix types

* Sync User Roles and Teams with IDP

* Add test for JWT role mapping to LiteLLM roles
2025-07-05 08:58:34 -07:00
Jugal D. BhattandGitHub 529f67f544 Add mcp server segregation comma separated support (#12326)
* add mcp comma separated support

* dont support legacy json array

* lint
2025-07-05 00:08:43 -07:00
ff7dd1756a [Security Bug Fix] Ensure only LLM API route fails get logged on Langfuse (and other loggers) (#12308)
* _is_proxy_only_llm_api_error

* test_proxy_only_error_true_for_llm_route

* add not on change

* Update tests/test_litellm/proxy/test_proxy_utils.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* add test_post_call_failure_hook_auth_error_key_info_route

* test fix _is_proxy_only_llm_api_error

* test_chat_completion_request_with_redaction

* test_post_call_failure_hook_auth_error_llm_api_route

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-04 14:42:42 -07:00
Ishaan JaffandGitHub 39955129f5 fix mapped tests (#12320)
* fix - use flush llm client cache

* faster mapped tests

* test_async_multiple_response_ids_routing

* fix tests

* test_ateam_member_update_admin_requires_premium

* regular mapped tests

* Revert "Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)"

This reverts commit 2c60c316ec.

* reset num workers
2025-07-04 10:04:43 -07:00
Ishaan Jaff 0835011388 test fix creds test 2025-07-03 22:01:52 -07:00
Ishaan Jaff 9c8619ed10 test fix 2025-07-03 21:16:26 -07:00