diff --git a/docs/my-website/docs/enterprise.md b/docs/my-website/docs/enterprise.md index 68611910d9..9101d8e375 100644 --- a/docs/my-website/docs/enterprise.md +++ b/docs/my-website/docs/enterprise.md @@ -4,7 +4,7 @@ import Image from '@theme/IdealImage'; For companies that need SSO, user management and professional support for LiteLLM Proxy :::info -Get free 7-day trial key [here](https://www.litellm.ai/#trial) +Get free 7-day trial key [here](https://www.litellm.ai/enterprise#trial) ::: ## Enterprise Features diff --git a/docs/my-website/docs/mcp.md b/docs/my-website/docs/mcp.md index 03de776a5c..d68416f116 100644 --- a/docs/my-website/docs/mcp.md +++ b/docs/my-website/docs/mcp.md @@ -42,7 +42,7 @@ LiteLLM supports the following MCP transports: style={{width: '80%', display: 'block', margin: '0'}} /> -### Adding a stdio MCP Server +#### Adding a stdio MCP Server For stdio MCP servers, select "Standard Input/Output (stdio)" as the transport type and provide the stdio configuration in JSON format: diff --git a/docs/my-website/docs/observability/datadog.md b/docs/my-website/docs/observability/datadog.md new file mode 100644 index 0000000000..7cd98d7269 --- /dev/null +++ b/docs/my-website/docs/observability/datadog.md @@ -0,0 +1,121 @@ +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# DataDog + +LiteLLM Supports logging to the following Datdog Integrations: +- `datadog` [Datadog Logs](https://docs.datadoghq.com/logs/) +- `datadog_llm_observability` [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) +- `ddtrace-run` [Datadog Tracing](#datadog-tracing) + + + + +We will use the `--config` to set `litellm.callbacks = ["datadog"]` this will log all successful LLM calls to DataDog + +**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` + +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: gpt-3.5-turbo +litellm_settings: + callbacks: ["datadog"] # logs llm success + failure logs on datadog + service_callback: ["datadog"] # logs redis, postgres failures on datadog +``` + + + + +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: gpt-3.5-turbo +litellm_settings: + callbacks: ["datadog_llm_observability"] # logs llm success logs on datadog +``` + + + + +**Step 2**: Set Required env variables for datadog + +```shell +DD_API_KEY="5f2d0f310***********" # your datadog API Key +DD_SITE="us5.datadoghq.com" # your datadog base url +DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to differentiate dev vs. prod deployments +``` + +**Step 3**: Start the proxy, make a test request + +Start proxy + +```shell +litellm --config config.yaml --debug +``` + +Test Request + +```shell +curl --location 'http://0.0.0.0:4000/chat/completions' \ + --header 'Content-Type: application/json' \ + --data '{ + "model": "gpt-3.5-turbo", + "messages": [ + { + "role": "user", + "content": "what llm are you" + } + ], + "metadata": { + "your-custom-metadata": "custom-field", + } +}' +``` + +Expected output on Datadog + + + +#### Datadog Tracing + +Use `ddtrace-run` to enable [Datadog Tracing](https://ddtrace.readthedocs.io/en/stable/installation_quickstart.html) on litellm proxy + +**DD Tracer** +Pass `USE_DDTRACE=true` to the docker run command. When `USE_DDTRACE=true`, the proxy will run `ddtrace-run litellm` as the `ENTRYPOINT` instead of just `litellm` + +**DD Profiler** + +Pass `USE_DDPROFILER=true` to the docker run command. When `USE_DDPROFILER=true`, the proxy will activate the [Datadog Profiler](https://docs.datadoghq.com/profiler/enabling/python/). This is useful for debugging CPU% and memory usage. + +We don't recommend using `USE_DDPROFILER` in production. It is only recommended for debugging CPU% and memory usage. + + +```bash +docker run \ + -v $(pwd)/litellm_config.yaml:/app/config.yaml \ + -e USE_DDTRACE=true \ + -e USE_DDPROFILER=true \ + -p 4000:4000 \ + ghcr.io/berriai/litellm:main-latest \ + --config /app/config.yaml --detailed_debug +``` + +### Set DD variables (`DD_SERVICE` etc) + +LiteLLM supports customizing the following Datadog environment variables + +| Environment Variable | Description | Default Value | Required | +|---------------------|-------------|---------------|----------| +| `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes | +| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes | +| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No | +| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No | +| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No | +| `DD_VERSION` | Version tag for your logs | "unknown" | ❌ No | +| `HOSTNAME` | Hostname tag for your logs | "" | ❌ No | +| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No | + diff --git a/docs/my-website/docs/observability/langfuse_integration.md b/docs/my-website/docs/observability/langfuse_integration.md index c1cf6165b3..a81336c5bc 100644 --- a/docs/my-website/docs/observability/langfuse_integration.md +++ b/docs/my-website/docs/observability/langfuse_integration.md @@ -212,6 +212,7 @@ The following parameters can be updated on a continuation of a trace by passing * `parent_observation_id` - Identifier for the parent observation, defaults to `None` * `prompt` - Langfuse prompt object used for the generation, defaults to `None` + Any other key value pairs passed into the metadata not listed in the above spec for a `litellm` completion will be added as a metadata key value pair for the generation. #### Disable Logging - Specific Calls diff --git a/docs/my-website/docs/proxy/admin_ui_sso.md b/docs/my-website/docs/proxy/admin_ui_sso.md index c1a641b4ff..c4c7423116 100644 --- a/docs/my-website/docs/proxy/admin_ui_sso.md +++ b/docs/my-website/docs/proxy/admin_ui_sso.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index 6e2c562436..de507c87b2 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -512,6 +512,7 @@ router_settings: | LAGO_API_KEY | API key for accessing Lago services | LANGFUSE_DEBUG | Toggle debug mode for Langfuse | LANGFUSE_FLUSH_INTERVAL | Interval for flushing Langfuse logs +| LANGFUSE_TRACING_ENVIRONMENT | Environment for Langfuse tracing | LANGFUSE_HOST | Host URL for Langfuse service | LANGFUSE_PUBLIC_KEY | Public key for Langfuse authentication | LANGFUSE_RELEASE | Release version of Langfuse integration diff --git a/docs/my-website/docs/proxy/guardrails/quick_start.md b/docs/my-website/docs/proxy/guardrails/quick_start.md index 6b90d273d9..824d4b241b 100644 --- a/docs/my-website/docs/proxy/guardrails/quick_start.md +++ b/docs/my-website/docs/proxy/guardrails/quick_start.md @@ -201,7 +201,7 @@ Follow this simple workflow to implement and tune guardrails: :::info -✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial) +✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial) ::: @@ -295,7 +295,7 @@ curl -i http://localhost:4000/v1/chat/completions \ :::info -✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial) +✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial) ::: @@ -380,7 +380,7 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g :::info -✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial) +✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial) ::: @@ -405,7 +405,7 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g :::info -✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial) +✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial) ::: @@ -465,7 +465,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ :::info -✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial) +✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial) ::: @@ -495,7 +495,7 @@ guardrails: :::info -✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial) +✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/jwt_auth_arch.md b/docs/my-website/docs/proxy/jwt_auth_arch.md index 6f591e5986..755d16c340 100644 --- a/docs/my-website/docs/proxy/jwt_auth_arch.md +++ b/docs/my-website/docs/proxy/jwt_auth_arch.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/logging.md b/docs/my-website/docs/proxy/logging.md index e1926d776e..b5b2b8bf50 100644 --- a/docs/my-website/docs/proxy/logging.md +++ b/docs/my-website/docs/proxy/logging.md @@ -1537,124 +1537,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ [**The standard logging object is logged on Azure Data Lake Storage**](../proxy/logging_spec) - -## DataDog - -LiteLLM Supports logging to the following Datdog Integrations: -- `datadog` [Datadog Logs](https://docs.datadoghq.com/logs/) -- `datadog_llm_observability` [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) -- `ddtrace-run` [Datadog Tracing](#datadog-tracing) - - - - -We will use the `--config` to set `litellm.callbacks = ["datadog"]` this will log all successful LLM calls to DataDog - -**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` - -```yaml -model_list: - - model_name: gpt-3.5-turbo - litellm_params: - model: gpt-3.5-turbo -litellm_settings: - callbacks: ["datadog"] # logs llm success + failure logs on datadog - service_callback: ["datadog"] # logs redis, postgres failures on datadog -``` - - - - -```yaml -model_list: - - model_name: gpt-3.5-turbo - litellm_params: - model: gpt-3.5-turbo -litellm_settings: - callbacks: ["datadog_llm_observability"] # logs llm success logs on datadog -``` - - - - -**Step 2**: Set Required env variables for datadog - -```shell -DD_API_KEY="5f2d0f310***********" # your datadog API Key -DD_SITE="us5.datadoghq.com" # your datadog base url -DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to differentiate dev vs. prod deployments -``` - -**Step 3**: Start the proxy, make a test request - -Start proxy - -```shell -litellm --config config.yaml --debug -``` - -Test Request - -```shell -curl --location 'http://0.0.0.0:4000/chat/completions' \ - --header 'Content-Type: application/json' \ - --data '{ - "model": "gpt-3.5-turbo", - "messages": [ - { - "role": "user", - "content": "what llm are you" - } - ], - "metadata": { - "your-custom-metadata": "custom-field", - } -}' -``` - -Expected output on Datadog - - - -#### Datadog Tracing - -Use `ddtrace-run` to enable [Datadog Tracing](https://ddtrace.readthedocs.io/en/stable/installation_quickstart.html) on litellm proxy - -**DD Tracer** -Pass `USE_DDTRACE=true` to the docker run command. When `USE_DDTRACE=true`, the proxy will run `ddtrace-run litellm` as the `ENTRYPOINT` instead of just `litellm` - -**DD Profiler** - -Pass `USE_DDPROFILER=true` to the docker run command. When `USE_DDPROFILER=true`, the proxy will activate the [Datadog Profiler](https://docs.datadoghq.com/profiler/enabling/python/). This is useful for debugging CPU% and memory usage. - -We don't recommend using `USE_DDPROFILER` in production. It is only recommended for debugging CPU% and memory usage. - - -```bash -docker run \ - -v $(pwd)/litellm_config.yaml:/app/config.yaml \ - -e USE_DDTRACE=true \ - -e USE_DDPROFILER=true \ - -p 4000:4000 \ - ghcr.io/berriai/litellm:main-latest \ - --config /app/config.yaml --detailed_debug -``` - -### Set DD variables (`DD_SERVICE` etc) - -LiteLLM supports customizing the following Datadog environment variables - -| Environment Variable | Description | Default Value | Required | -|---------------------|-------------|---------------|----------| -| `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes | -| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes | -| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No | -| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No | -| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No | -| `DD_VERSION` | Version tag for your logs | "unknown" | ❌ No | -| `HOSTNAME` | Hostname tag for your logs | "" | ❌ No | -| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No | - +## [Datadog](../observability/datadog) ## Lunary #### Step1: Install dependencies and set your environment variables diff --git a/docs/my-website/docs/proxy/pagerduty.md b/docs/my-website/docs/proxy/pagerduty.md index 70686deebd..281dabe274 100644 --- a/docs/my-website/docs/proxy/pagerduty.md +++ b/docs/my-website/docs/proxy/pagerduty.md @@ -8,7 +8,7 @@ import Image from '@theme/IdealImage'; [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/prometheus.md b/docs/my-website/docs/proxy/prometheus.md index a6cbe2bc38..b1aae1da7c 100644 --- a/docs/my-website/docs/proxy/prometheus.md +++ b/docs/my-website/docs/proxy/prometheus.md @@ -10,7 +10,7 @@ import Image from '@theme/IdealImage'; [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/request_headers.md b/docs/my-website/docs/proxy/request_headers.md index 79bcea2c86..7251262d3b 100644 --- a/docs/my-website/docs/proxy/request_headers.md +++ b/docs/my-website/docs/proxy/request_headers.md @@ -14,6 +14,8 @@ Special headers that are supported by LiteLLM. `anthropic-version` Optional[str]: The version of the Anthropic API to use. `anthropic-beta` Optional[str]: The beta version of the Anthropic API to use. + - For `/v1/messages` endpoint, this will always be forward the header to the underlying model. + - For `/chat/completions` endpoint, this will only be forwarded if `forward_client_headers_to_llm_api` is true. ## OpenAI Headers diff --git a/docs/my-website/docs/proxy/spend_logs_deletion.md b/docs/my-website/docs/proxy/spend_logs_deletion.md index 3738df5eaa..05627c0774 100644 --- a/docs/my-website/docs/proxy/spend_logs_deletion.md +++ b/docs/my-website/docs/proxy/spend_logs_deletion.md @@ -8,7 +8,7 @@ This walks through how to set the maximum retention period for spend logs. This [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/temporary_budget_increase.md b/docs/my-website/docs/proxy/temporary_budget_increase.md index de985eb9bd..00b1275030 100644 --- a/docs/my-website/docs/proxy/temporary_budget_increase.md +++ b/docs/my-website/docs/proxy/temporary_budget_increase.md @@ -16,7 +16,7 @@ Set temporary budget increase for a LiteLLM Virtual Key. Use this if you get ask [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/proxy/virtual_keys.md b/docs/my-website/docs/proxy/virtual_keys.md index 26ec69b30d..bf1090e585 100644 --- a/docs/my-website/docs/proxy/virtual_keys.md +++ b/docs/my-website/docs/proxy/virtual_keys.md @@ -527,7 +527,7 @@ This is an Enterprise feature. [Enterprise Pricing](https://www.litellm.ai/#pricing) -[Get free 7-day trial key](https://www.litellm.ai/#trial) +[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial) ::: diff --git a/docs/my-website/docs/response_api.md b/docs/my-website/docs/response_api.md index 26c0081be2..e64f922ac8 100644 --- a/docs/my-website/docs/response_api.md +++ b/docs/my-website/docs/response_api.md @@ -733,6 +733,68 @@ follow_up = client.responses.create( +## Calling non-Responses API endpoints (`/responses` to `/chat/completions` Bridge) + +LiteLLM allows you to call non-Responses API models via a bridge to LiteLLM's `/chat/completions` endpoint. This is useful for calling Anthropic, Gemini and even non-Responses API OpenAI models. + + +#### Python SDK Usage + +```python showLineNumbers title="SDK Usage" +import litellm +import os + +# Set API key +os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key" + +# Non-streaming response +response = litellm.responses( + model="anthropic/claude-3-5-sonnet-20240620", + input="Tell me a three sentence bedtime story about a unicorn.", + max_output_tokens=100 +) + +print(response) +``` + +#### LiteLLM Proxy Usage + +**Setup Config:** + +```yaml showLineNumbers title="Example Configuration" +model_list: +- model_name: anthropic-model + litellm_params: + model: anthropic/claude-3-5-sonnet-20240620 + api_key: os.environ/ANTHROPIC_API_KEY +``` + +**Start Proxy:** + +```bash showLineNumbers title="Start LiteLLM Proxy" +litellm --config /path/to/config.yaml + +# RUNNING on http://0.0.0.0:4000 +``` + +**Make Request:** + +```bash showLineNumbers title="non-Responses API Model Request" +curl http://localhost:4000/v1/responses \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "anthropic-model", + "input": "who is Michael Jordan" + }' +``` + + + + + + + ## Session Management - Non-OpenAI Models LiteLLM Proxy supports session management for non-OpenAI models. This allows you to store and fetch conversation history (state) in LiteLLM Proxy. diff --git a/docs/my-website/img/release_notes/mcp_tool_cost_tracking.png b/docs/my-website/img/release_notes/mcp_tool_cost_tracking.png new file mode 100644 index 0000000000..ef2f993da2 Binary files /dev/null and b/docs/my-website/img/release_notes/mcp_tool_cost_tracking.png differ diff --git a/docs/my-website/img/release_notes/model_hub_v2.png b/docs/my-website/img/release_notes/model_hub_v2.png new file mode 100644 index 0000000000..7731289cdb Binary files /dev/null and b/docs/my-website/img/release_notes/model_hub_v2.png differ diff --git a/docs/my-website/release_notes/v1.55.10/index.md b/docs/my-website/release_notes/v1.55.10/index.md index 2b5ce75cf0..46c4a1739c 100644 --- a/docs/my-website/release_notes/v1.55.10/index.md +++ b/docs/my-website/release_notes/v1.55.10/index.md @@ -28,7 +28,7 @@ import Image from '@theme/IdealImage'; :::info -Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial) +Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial) **No call needed** diff --git a/docs/my-website/release_notes/v1.74.3-stable/index.md b/docs/my-website/release_notes/v1.74.3-stable/index.md new file mode 100644 index 0000000000..7e4a5baa75 --- /dev/null +++ b/docs/my-website/release_notes/v1.74.3-stable/index.md @@ -0,0 +1,251 @@ +--- +title: "[PRE-RELEASE] v1.74.3-stable" +slug: "v1-74-3-stable" +date: 2025-07-12T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg + +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Deploy this version + + + + +``` showLineNumbers title="docker run litellm" +docker run \ +-e STORE_MODEL_IN_DB=True \ +-p 4000:4000 \ +ghcr.io/berriai/litellm:v1.74.3.rc.2 +``` + + + + +``` showLineNumbers title="pip install litellm" +pip install litellm==1.74.3rc2 +``` + + + + +--- + + +## New Models / Updated Models + +#### Pricing / Context Window Updates + +| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Type | +| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- | ---- | +| Xai | `xai/grok-4` | 256k | $3.00 | $15.00 | New | +| Xai | `xai/grok-4-0709` | 256k | $3.00 | $15.00 | New | +| Xai | `xai/grok-4-latest` | 256k | $3.00 | $15.00 | New | +| Mistral | `mistral/devstral-small-2507` | 128k | $0.1 | $0.3 | New | +| Mistral | `mistral/devstral-medium-2507` | 128k | $0.4 | $2 | New | +| Azure OpenAI | `azure/o3-deep-research` | 200k | $10 | $40 | New | + + +#### Features +- **[Xinference](../../docs/providers/xinference)** + - Image generation API support - [PR](https://github.com/BerriAI/litellm/pull/12439) +- **[Bedrock](../../docs/providers/bedrock)** + - API Key Auth support for AWS Bedrock API - [PR](https://github.com/BerriAI/litellm/pull/12495) +- **[🆕 Dashscope](../../docs/providers/dashscope)** + - New integration from Alibaba (enables qwen usage) - [PR](https://github.com/BerriAI/litellm/pull/12361) +- **[🆕 Bytez](../../docs/providers/bytez)** + - New /chat/completion integration - [PR](https://github.com/BerriAI/litellm/pull/12121) + +#### Bugs +- **[Github Copilot](../../docs/providers/github_copilot)** + - Fix API base url for Github Copilot - [PR](https://github.com/BerriAI/litellm/pull/12418) +- **[Bedrock](../../docs/providers/bedrock)** + - Ensure supported bedrock/converse/ params = bedrock/ params - [PR](https://github.com/BerriAI/litellm/pull/12466) + - Fix cache token cost calculation - [PR](https://github.com/BerriAI/litellm/pull/12488) +- **[XAI](../../docs/providers/xai)** + - ensure finish_reason includes tool calls when xai responses with tool calls - [PR](https://github.com/BerriAI/litellm/pull/12545) + +--- + +## LLM API Endpoints + +#### Features +- **[/completions](../../docs/text_completion)** + - Return ‘reasoning_content’ on streaming - [PR](https://github.com/BerriAI/litellm/pull/12377) +- **[/chat/completions](../../docs/completion/input)** + - Add 'thinking blocks' to stream chunk builder - [PR](https://github.com/BerriAI/litellm/pull/12395) +- **[/v1/messages](../../docs/anthropic_unified)** + - Fallbacks support - [PR](https://github.com/BerriAI/litellm/pull/12440) + - tool call handling for non-anthropic models (/v1/messages to /chat/completion bridge) - [PR](https://github.com/BerriAI/litellm/pull/12473) + +--- + +## [MCP Gateway](../../docs/mcp) + + + +#### Features +- **[Cost Tracking](../../docs/mcp#-mcp-cost-tracking)** + - Add Cost Tracking - [PR](https://github.com/BerriAI/litellm/pull/12385) + - Add usage tracking - [PR](https://github.com/BerriAI/litellm/pull/12397) + - Add custom cost configuration for each MCP tool - [PR](https://github.com/BerriAI/litellm/pull/12499) + - Add support for editing MCP cost per tool - [PR](https://github.com/BerriAI/litellm/pull/12501) + - Allow using custom post call MCP hook for cost tracking - [PR](https://github.com/BerriAI/litellm/pull/12469) +- **[Auth](../../docs/mcp#using-your-mcp-with-client-side-credentials)** + - Allow customizing what client side auth header to use - [PR](https://github.com/BerriAI/litellm/pull/12460) + - Raises error when MCP server header is malformed in the request - [PR](https://github.com/BerriAI/litellm/pull/12494) +- **[MCP Server](../../docs/mcp#adding-your-mcp)** + - Allow using stdio MCPs with LiteLLM (enables using Circle CI MCP w/ LiteLLM) - [PR](https://github.com/BerriAI/litellm/pull/12530), [Get Started](../../docs/mcp#adding-a-stdio-mcp-server) + +#### Bugs +- **General** + - Fix task group is not initialized error - [PR](https://github.com/BerriAI/litellm/pull/12411) s/o [@juancarlosm](https://github.com/juancarlosm) +- **[MCP Server](../../docs/mcp#adding-your-mcp)** + - Fix mcp tool separator to work with Claude code - [PR](https://github.com/BerriAI/litellm/pull/12430), [Get Started](../../docs/mcp#adding-your-mcp) + - Add validation to mcp server name to not allow "-" (enables namespaces to work) - [PR](https://github.com/BerriAI/litellm/pull/12515) + + +--- + +## Management Endpoints / UI + + + + +#### Features +- **Model Hub** + - new model hub table view - [PR](https://github.com/BerriAI/litellm/pull/12468) + - new /public/model_hub endpoint - [PR](https://github.com/BerriAI/litellm/pull/12468) + - Make Model Hub OSS - [PR](https://github.com/BerriAI/litellm/pull/12553) + - New ‘make public’ modal flow for showing proxy models on public model hub - [PR](https://github.com/BerriAI/litellm/pull/12555) +- **MCP** + - support for internal users to use and manage MCP servers - [PR](https://github.com/BerriAI/litellm/pull/12458) + - Adds UI support to add MCP access groups (similar to namespaces) - [PR](https://github.com/BerriAI/litellm/pull/12470) + - MCP Tool Testing Playground - [PR](https://github.com/BerriAI/litellm/pull/12520) + - Show cost config on root of MCP settings - [PR](https://github.com/BerriAI/litellm/pull/12526) +- **Test Key** + - Stick sessions - [PR](https://github.com/BerriAI/litellm/pull/12365) + - MCP Access Groups - allow mcp access groups - [PR](https://github.com/BerriAI/litellm/pull/12529) +- **Usage** + - Truncate long labels and improve tooltip in Top API Keys chart - [PR](https://github.com/BerriAI/litellm/pull/12371) + - Improve Chart Readability for Tag Usage - [PR](https://github.com/BerriAI/litellm/pull/12378) +- **Teams** + - Prevent navigation reset after team member operations - [PR](https://github.com/BerriAI/litellm/pull/12424) + - Team Members - reset budget, if duration set - [PR](https://github.com/BerriAI/litellm/pull/12534) + - Use central team member budget when max_budget_in_team set on UI - [PR](https://github.com/BerriAI/litellm/pull/12533) +- **SSO** + - Allow users to run a custom sso login handler - [PR](https://github.com/BerriAI/litellm/pull/12465) +- **Navbar** + - improve user dropdown UI with premium badge and cleaner layout - [PR](https://github.com/BerriAI/litellm/pull/12502) +- **General** + - Consistent layout for Create and Back buttons on all the pages - [PR](https://github.com/BerriAI/litellm/pull/12542) + - Align Show Password with Checkbox - [PR](https://github.com/BerriAI/litellm/pull/12538) + - Prevent writing default user setting updates to yaml (causes error in non-root env) - [PR](https://github.com/BerriAI/litellm/pull/12533) + +#### Bugs +- **Model Hub** + - fix duplicates in /model_group/info - [PR](https://github.com/BerriAI/litellm/pull/12468) +- **MCP** + - Fix UI not syncing MCP access groups properly with object permissions - [PR](https://github.com/BerriAI/litellm/pull/12523) + +--- + +## Logging / Guardrail Integrations + +#### Features +- **[Langfuse](../../docs/observability/langfuse_integration)** + - Version bump - [PR](https://github.com/BerriAI/litellm/pull/12376) + - LANGFUSE_TRACING_ENVIRONMENT support - [PR](https://github.com/BerriAI/litellm/pull/12376) +- **[Bedrock Guardrails](../../docs/proxy/guardrails/bedrock)** + - Raise Bedrock output text on 'BLOCKED' actions from guardrail - [PR](https://github.com/BerriAI/litellm/pull/12435) +- **[OTEL](../../docs/observability/opentelemetry_integration)** + - `OTEL_RESOURCE_ATTRIBUTES` support - [PR](https://github.com/BerriAI/litellm/pull/12468) +- **[Guardrails AI](../../docs/proxy/guardrails/guardrails_ai)** + - pre-call + logging only guardrail (pii detection/competitor names) support - [PR](https://github.com/BerriAI/litellm/pull/12506) +- **[Guardrails](../../docs/proxy/guardrails/quick_start)** + - [Enterprise] Support tag based mode for guardrails - [PR](https://github.com/BerriAI/litellm/pull/12508), [Get Started](../../docs/proxy/guardrails/quick_start#-tag-based-guardrail-modes) +- **[OpenAI Moderations API](../../docs/proxy/guardrails/openai_moderation)** + - New guardrail integration - [PR](https://github.com/BerriAI/litellm/pull/12519) +- **[Prometheus](../../docs/proxy/prometheus)** + - support tag based metrics (enables prometheus metrics for measuring roo-code/cline/claude code engagement) - [PR](https://github.com/BerriAI/litellm/pull/12534), [Get Started](../../docs/proxy/prometheus#custom-tags) +- **[Datadog LLM Observability](../../docs/observability/datadog)** + - Added `total_cost` field to track costs in DataDog LLM observability metrics - [PR](https://github.com/BerriAI/litellm/pull/12467) + +#### Bugs +- **[Prometheus](../../docs/proxy/prometheus)** + - Remove experimental `_by_tag` metrics (fixes cardinality issue) - [PR](https://github.com/BerriAI/litellm/pull/12395) +- **[Slack Alerting](../../docs/proxy/alerting)** + - Fix slack alerting for outage and region outage alerts - [PR](https://github.com/BerriAI/litellm/pull/12464), [Get Started](../../docs/proxy/alerting#region-outage-alerting--enterprise-feature) + +--- + +## Performance / Loadbalancing / Reliability improvements + +#### Bugs +- **[Responses API Bridge](../../docs/response_api#calling-non-responses-api-endpoints-responses-to-chatcompletions-bridge)** + - add image support for Responses API when falling back on Chat Completions - [PR](https://github.com/BerriAI/litellm/pull/12204) s/o [@ryan-castner](https://github.com/ryan-castner) +- **aiohttp** + - Properly close aiohttp client sessions to prevent resource leaks - [PR](https://github.com/BerriAI/litellm/pull/12251) +- **Router** + - don't add invalid deployment to router pattern match - [PR](https://github.com/BerriAI/litellm/pull/12459) + + +--- + +## General Proxy Improvements + +#### Bugs +- **S3** + - s3 config.yaml file - ensure yaml safe load is used - [PR](https://github.com/BerriAI/litellm/pull/12373) +- **Audit Logs** + - Add audit logs for model updates - [PR](https://github.com/BerriAI/litellm/pull/12396) +- **Startup** + - Multiple API Keys Created on Startup when max_budget is enabled - [PR](https://github.com/BerriAI/litellm/pull/12436) +- **Auth** + - Resolve model group alias on Auth (if user has access to underlying model, allow alias request to work) - [PR](https://github.com/BerriAI/litellm/pull/12440) +- **config.yaml** + - fix parsing environment_variables from config.yaml - [PR](https://github.com/BerriAI/litellm/pull/12482) +- **Security** + - Log hashed jwt w/ prefix instead of actual value - [PR](https://github.com/BerriAI/litellm/pull/12524) + +#### Features +- **MCP** + - Bump mcp version on docker img - [PR](https://github.com/BerriAI/litellm/pull/12362) +- **Request Headers** + - Forward ‘anthropic-beta’ header when forward_client_headers_to_llm_api is true - [PR](https://github.com/BerriAI/litellm/pull/12462) + +--- + +## New Contributors +* @kanaka made their first contribution in https://github.com/BerriAI/litellm/pull/12418 +* @juancarlosm made their first contribution in https://github.com/BerriAI/litellm/pull/12411 +* @DmitriyAlergant made their first contribution in https://github.com/BerriAI/litellm/pull/12356 +* @Rayshard made their first contribution in https://github.com/BerriAI/litellm/pull/12487 +* @minghao51 made their first contribution in https://github.com/BerriAI/litellm/pull/12361 +* @jdietzsch91 made their first contribution in https://github.com/BerriAI/litellm/pull/12488 +* @iwinux made their first contribution in https://github.com/BerriAI/litellm/pull/12473 +* @andresC98 made their first contribution in https://github.com/BerriAI/litellm/pull/12413 +* @EmaSuriano made their first contribution in https://github.com/BerriAI/litellm/pull/12509 +* @strawgate made their first contribution in https://github.com/BerriAI/litellm/pull/12528 +* @inf3rnus made their first contribution in https://github.com/BerriAI/litellm/pull/12121 + +## **[Git Diff](https://github.com/BerriAI/litellm/compare/v1.74.0-stable...v1.74.3-stable)** + diff --git a/docs/my-website/static/llms-full.txt b/docs/my-website/static/llms-full.txt index 30cc424f85..c64d417096 100644 --- a/docs/my-website/static/llms-full.txt +++ b/docs/my-website/static/llms-full.txt @@ -3424,7 +3424,7 @@ You can now set custom parameters (like success threshold) for your guardrails i info -Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial) +Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial) **No call needed** @@ -4107,7 +4107,7 @@ Use this to see the changes in the codebase. info -Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial) +Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial) **No call needed** @@ -4966,7 +4966,7 @@ Before adding a model you can test the connection to the LLM provider to verify info -Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial) +Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial) **No call needed** @@ -5815,7 +5815,7 @@ You can now set custom parameters (like success threshold) for your guardrails i info -Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial) +Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial) **No call needed** diff --git a/litellm/proxy/_experimental/out/model_hub_table.html b/litellm/proxy/_experimental/out/model_hub_table/index.html similarity index 100% rename from litellm/proxy/_experimental/out/model_hub_table.html rename to litellm/proxy/_experimental/out/model_hub_table/index.html diff --git a/litellm/proxy/_experimental/out/onboarding.html b/litellm/proxy/_experimental/out/onboarding.html deleted file mode 100644 index 5015fbdbd7..0000000000 --- a/litellm/proxy/_experimental/out/onboarding.html +++ /dev/null @@ -1 +0,0 @@ -LiteLLM Dashboard \ No newline at end of file