diff --git a/docs/my-website/docs/observability/helicone_integration.md b/docs/my-website/docs/observability/helicone_integration.md index 9b807b8d0f..27e972ddeb 100644 --- a/docs/my-website/docs/observability/helicone_integration.md +++ b/docs/my-website/docs/observability/helicone_integration.md @@ -1,3 +1,6 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + # Helicone - OSS LLM Observability Platform :::tip @@ -9,9 +12,68 @@ https://github.com/BerriAI/litellm [Helicone](https://helicone.ai/) is an open source observability platform that proxies your LLM requests and provides key insights into your usage, spend, latency and more. -## Using Helicone with LiteLLM +## Quick Start -LiteLLM provides `success_callbacks` and `failure_callbacks`, allowing you to easily log data to Helicone based on the status of your responses. + + + +Use just 1 line of code to instantly log your responses **across all providers** with Helicone: + +```python +import os +from litellm import completion + +## Set env variables +os.environ["HELICONE_API_KEY"] = "your-helicone-key" +os.environ["OPENAI_API_KEY"] = "your-openai-key" + +# Set callbacks +litellm.success_callback = ["helicone"] + +# OpenAI call +response = completion( + model="gpt-4o", + messages=[{"role": "user", "content": "Hi 👋 - I'm OpenAI"}], +) + +print(response) +``` + + + + +Add Helicone to your LiteLLM proxy configuration: + +```yaml title="config.yaml" +model_list: + - model_name: gpt-4 + litellm_params: + model: gpt-4 + api_key: os.environ/OPENAI_API_KEY + +# Add Helicone callback +litellm_settings: + success_callback: ["helicone"] + +# Set Helicone API key +environment_variables: + HELICONE_API_KEY: "your-helicone-key" +``` + +Start the proxy: +```bash +litellm --config config.yaml +``` + + + + +## Integration Methods + +There are two main approaches to integrate Helicone with LiteLLM: + +1. **Callbacks**: Log to Helicone while using any provider +2. **Proxy Mode**: Use Helicone as a proxy for advanced features ### Supported LLM Providers @@ -26,27 +88,16 @@ Helicone can log requests across [various LLM providers](https://docs.helicone.a - Replicate - And more -### Integration Methods +## Method 1: Using Callbacks -There are two main approaches to integrate Helicone with LiteLLM: +Log requests to Helicone while using any LLM provider directly. -1. Using callbacks -2. Using Helicone as a proxy - -Let's explore each method in detail. - -### Approach 1: Use Callbacks - -Use just 1 line of code to instantly log your responses **across all providers** with Helicone: - -```python -litellm.success_callback = ["helicone"] -``` - -Complete Code + + ```python import os +import litellm from litellm import completion ## Set env variables @@ -66,28 +117,78 @@ response = completion( print(response) ``` -### Approach 2: Use Helicone as a proxy + + + +```yaml title="config.yaml" +model_list: + - model_name: gpt-4 + litellm_params: + model: gpt-4 + api_key: os.environ/OPENAI_API_KEY + - model_name: claude-3 + litellm_params: + model: anthropic/claude-3-sonnet-20240229 + api_key: os.environ/ANTHROPIC_API_KEY + +# Add Helicone logging +litellm_settings: + success_callback: ["helicone"] + +# Environment variables +environment_variables: + HELICONE_API_KEY: "your-helicone-key" + OPENAI_API_KEY: "your-openai-key" + ANTHROPIC_API_KEY: "your-anthropic-key" +``` + +Start the proxy: +```bash +litellm --config config.yaml +``` + +Make requests to your proxy: +```python +import openai + +client = openai.OpenAI( + api_key="anything", # proxy doesn't require real API key + base_url="http://localhost:4000" +) + +response = client.chat.completions.create( + model="gpt-4", # This gets logged to Helicone + messages=[{"role": "user", "content": "Hello!"}] +) +``` + + + + +## Method 2: Using Helicone as a Proxy Helicone's proxy provides [advanced functionality](https://docs.helicone.ai/getting-started/proxy-vs-async) like caching, rate limiting, LLM security through [PromptArmor](https://promptarmor.com/) and more. -To use Helicone as a proxy for your LLM requests: + + -1. Set Helicone as your base URL via: litellm.api_base -2. Pass in Helicone request headers via: litellm.metadata - -Complete Code: +Set Helicone as your base URL and pass authentication headers: ```python import os import litellm from litellm import completion +# Configure LiteLLM to use Helicone proxy litellm.api_base = "https://oai.hconeai.com/v1" litellm.headers = { - "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", # Authenticate to send requests to Helicone API + "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", } -response = litellm.completion( +# Set your OpenAI API key +os.environ["OPENAI_API_KEY"] = "your-openai-key" + +response = completion( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "How does a court case get to the Supreme Court?"}] ) @@ -140,32 +241,112 @@ litellm.metadata = { Track multi-step and agentic LLM interactions using session IDs and paths: -```python -litellm.metadata = { - "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", # Authenticate to send requests to Helicone API - "Helicone-Session-Id": "session-abc-123", # The session ID you want to track - "Helicone-Session-Path": "parent-trace/child-trace", # The path of the session -} -``` - -- `Helicone-Session-Id`: Use this to specify the unique identifier for the session you want to track. This allows you to group related requests together. -- `Helicone-Session-Path`: This header defines the path of the session, allowing you to represent parent and child traces. For example, "parent/child" represents a child trace of a parent trace. - -By using these two headers, you can effectively group and visualize multi-step LLM interactions, gaining insights into complex AI workflows. - -### Retry and Fallback Mechanisms - -Set up retry mechanisms and fallback options: + + ```python +import litellm + +litellm.api_base = "https://oai.hconeai.com/v1" litellm.metadata = { - "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", # Authenticate to send requests to Helicone API - "Helicone-Retry-Enabled": "true", # Enable retry mechanism - "helicone-retry-num": "3", # Set number of retries - "helicone-retry-factor": "2", # Set exponential backoff factor - "Helicone-Fallbacks": '["gpt-3.5-turbo", "gpt-4"]', # Set fallback models + "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", + "Helicone-Session-Id": "session-abc-123", + "Helicone-Session-Path": "parent-trace/child-trace", } + +response = litellm.completion( + model="gpt-3.5-turbo", + messages=[{"role": "user", "content": "Start a conversation"}] +) ``` + + + +```python +import openai + +client = openai.OpenAI( + api_key="anything", + base_url="http://localhost:4000" +) + +# First request in session +response1 = client.chat.completions.create( + model="gpt-4", + messages=[{"role": "user", "content": "Hello"}], + extra_headers={ + "Helicone-Session-Id": "session-abc-123", + "Helicone-Session-Path": "conversation/greeting" + } +) + +# Follow-up request in same session +response2 = client.chat.completions.create( + model="gpt-4", + messages=[{"role": "user", "content": "Tell me more"}], + extra_headers={ + "Helicone-Session-Id": "session-abc-123", + "Helicone-Session-Path": "conversation/follow-up" + } +) +``` + + + + +- `Helicone-Session-Id`: Unique identifier for the session to group related requests +- `Helicone-Session-Path`: Hierarchical path to represent parent/child traces (e.g., "parent/child") + +## Retry and Fallback Mechanisms + + + + +```python +import litellm + +litellm.api_base = "https://oai.hconeai.com/v1" +litellm.metadata = { + "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", + "Helicone-Retry-Enabled": "true", + "helicone-retry-num": "3", + "helicone-retry-factor": "2", # Exponential backoff + "Helicone-Fallbacks": '["gpt-3.5-turbo", "gpt-4"]', +} + +response = litellm.completion( + model="gpt-4", + messages=[{"role": "user", "content": "Hello"}] +) +``` + + + + +```yaml title="config.yaml" +model_list: + - model_name: gpt-4 + litellm_params: + model: gpt-4 + api_key: os.environ/OPENAI_API_KEY + api_base: "https://oai.hconeai.com/v1" + +default_litellm_params: + headers: + Helicone-Auth: "Bearer ${HELICONE_API_KEY}" + Helicone-Retry-Enabled: "true" + helicone-retry-num: "3" + helicone-retry-factor: "2" + Helicone-Fallbacks: '["gpt-3.5-turbo", "gpt-4"]' + +environment_variables: + HELICONE_API_KEY: "your-helicone-key" + OPENAI_API_KEY: "your-openai-key" +``` + + + + > **Supported Headers** - For a full list of supported Helicone headers and their descriptions, please refer to the [Helicone documentation](https://docs.helicone.ai/getting-started/quick-start). > By utilizing these headers and metadata options, you can gain deeper insights into your LLM usage, optimize performance, and better manage your AI workflows with Helicone and LiteLLM. diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index ee857f700a..8847566eb9 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -93,6 +93,8 @@ callback_settings: general_settings: completion_model: string + store_prompts_in_spend_logs: boolean + forward_client_headers_to_llm_api: boolean disable_spend_logs: boolean # turn off writing each transaction to the db disable_master_key_return: boolean # turn off returning master key on UI (checked on '/user/info' endpoint) disable_retry_on_max_parallel_request_limit_error: boolean # turn off retries when max parallel request limit is reached @@ -121,6 +123,35 @@ general_settings: alerting: ["slack", "email"] alerting_threshold: 0 use_client_credentials_pass_through_routes: boolean # use client credentials for all pass through routes like "/vertex-ai", /bedrock/. When this is True Virtual Key auth will not be applied on these endpoints + +router_settings: + routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle" - RECOMMENDED for best performance + redis_host: # string + redis_password: # string + redis_port: # string + enable_pre_call_checks: true # bool - Before call is made check if a call is within model context window + allowed_fails: 3 # cooldown model if it fails > 1 call in a minute. + cooldown_time: 30 # (in seconds) how long to cooldown model if fails/min > allowed_fails + disable_cooldowns: True # bool - Disable cooldowns for all models + enable_tag_filtering: True # bool - Use tag based routing for requests + retry_policy: { # Dict[str, int]: retry policy for different types of exceptions + "AuthenticationErrorRetries": 3, + "TimeoutErrorRetries": 3, + "RateLimitErrorRetries": 3, + "ContentPolicyViolationErrorRetries": 4, + "InternalServerErrorRetries": 4 + } + allowed_fails_policy: { + "BadRequestErrorAllowedFails": 1000, # Allow 1000 BadRequestErrors before cooling down a deployment + "AuthenticationErrorAllowedFails": 10, # int + "TimeoutErrorAllowedFails": 12, # int + "RateLimitErrorAllowedFails": 10000, # int + "ContentPolicyViolationErrorAllowedFails": 15, # int + "InternalServerErrorAllowedFails": 20, # int + } + content_policy_fallbacks=[{"claude-2": ["my-fallback-model"]}] # List[Dict[str, List[str]]]: Fallback model for content policy violations + fallbacks=[{"claude-2": ["my-fallback-model"]}] # List[Dict[str, List[str]]]: Fallback model for all errors + ``` ### litellm_settings - Reference diff --git a/docs/my-website/docs/proxy/logging_spec.md b/docs/my-website/docs/proxy/logging_spec.md index a39a62318e..5166b86ae1 100644 --- a/docs/my-website/docs/proxy/logging_spec.md +++ b/docs/my-website/docs/proxy/logging_spec.md @@ -61,6 +61,11 @@ Inherits from `StandardLoggingUserAPIKeyMetadata` and adds: | `requester_metadata` | `Optional[dict]` | Additional requester metadata | | `vector_store_request_metadata` | `Optional[List[StandardLoggingVectorStoreRequest]]` | Vector store request metadata | | `requester_custom_headers` | Dict[str, str] | Any custom (`x-`) headers sent by the client to the proxy. | +| `prompt_management_metadata` | `Optional[StandardLoggingPromptManagementMetadata]` | Prompt management and versioning metadata | +| `mcp_tool_call_metadata` | `Optional[StandardLoggingMCPToolCall]` | MCP (Model Context Protocol) tool call information and cost tracking | +| `applied_guardrails` | `Optional[List[str]]` | List of applied guardrail names | +| `usage_object` | `Optional[dict]` | Raw usage object from the LLM provider | +| `cold_storage_object_key` | `Optional[str]` | S3/GCS object key for cold storage retrieval | | `guardrail_information` | `Optional[StandardLoggingGuardrailInformation]` | Guardrail information | @@ -145,4 +150,82 @@ A literal type with two possible values: | `duration` | `Optional[float]` | Duration of the guardrail in seconds | | `masked_entity_count` | `Optional[Dict[str, int]]` | Count of masked entities | +## StandardLoggingPromptManagementMetadata +Used for tracking prompt versioning and management information. + +| Field | Type | Description | +|-------|------|-------------| +| `prompt_id` | `str` | **Required**. Unique identifier for the prompt template or version | +| `prompt_variables` | `Optional[dict]` | Variables/parameters used in the prompt template (e.g., `{"user_name": "John", "context": "support"}`) | +| `prompt_integration` | `str` | **Required**. Integration or system managing the prompt (e.g., `"langfuse"`, `"promptlayer"`, `"custom"`) | + +## StandardLoggingMCPToolCall + +Used to track Model Context Protocol (MCP) tool calls within LiteLLM requests. This provides detailed logging for external tool integrations. + +| Field | Type | Description | +|-------|------|-------------| +| `name` | `str` | **Required**. The name of the tool being called (e.g., `"get_weather"`, `"search_database"`) | +| `arguments` | `dict` | **Required**. Arguments passed to the tool as key-value pairs | +| `result` | `Optional[dict]` | The response/result returned by the tool execution (populated by custom logging hooks) | +| `mcp_server_name` | `Optional[str]` | Name of the MCP server that handled the tool call (e.g., `"weather-service"`, `"database-connector"`) | +| `mcp_server_logo_url` | `Optional[str]` | URL for the MCP server's logo (used for UI display in LiteLLM dashboard) | +| `namespaced_tool_name` | `Optional[str]` | Fully qualified tool name including server prefix (e.g., `"deepwiki-mcp/get_page_content"`, `"github-mcp/create_issue"`) | +| `mcp_server_cost_info` | `Optional[MCPServerCostInfo]` | Cost tracking information for the tool call | + +### MCPServerCostInfo + +Cost tracking structure for MCP server tool calls: + +| Field | Type | Description | +|-------|------|-------------| +| `default_cost_per_query` | `Optional[float]` | Default cost in USD for any tool call to this MCP server | +| `tool_name_to_cost_per_query` | `Optional[Dict[str, float]]` | Per-tool cost mapping for granular pricing (e.g., `{"search": 0.01, "create": 0.05}`) | + +### Usage + +```python +# Basic MCP tool call metadata +mcp_tool_call = { + "name": "search_documents", + "arguments": { + "query": "machine learning tutorials", + "limit": 10, + "filter": "type:pdf" + }, + "mcp_server_name": "document-search-service", + "namespaced_tool_name": "docs-mcp/search_documents", + "mcp_server_cost_info": { + "default_cost_per_query": 0.02, + "tool_name_to_cost_per_query": { + "search_documents": 0.02, + "get_document": 0.01 + } + } +} + +# optional result field (via custom logging hooks) +mcp_tool_call_with_result = { + "name": "search_documents", + "arguments": { + "query": "machine learning tutorials", + "limit": 10, + "filter": "type:pdf" + }, + "result": { + "documents": [...], + "total_found": 42, + "search_time_ms": 150 + }, + "mcp_server_name": "document-search-service", + "namespaced_tool_name": "docs-mcp/search_documents", + "mcp_server_cost_info": { + "default_cost_per_query": 0.02, + "tool_name_to_cost_per_query": { + "search_documents": 0.02, + "get_document": 0.01 + } + } +} +``` \ No newline at end of file diff --git a/docs/my-website/docs/troubleshoot.md b/docs/my-website/docs/troubleshoot.md index b6a9c6a6b9..9d2b3757ee 100644 --- a/docs/my-website/docs/troubleshoot.md +++ b/docs/my-website/docs/troubleshoot.md @@ -2,7 +2,7 @@ [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) [Community Discord 💭](https://discord.gg/wuPM9dRgDw) -[Community Slack 💭](https://join.slack.com/share/enQtOTE0ODczMzk2Nzk4NC01YjUxNjY2YjBlYTFmNDRiZTM3NDFiYTM3MzVkODFiMDVjOGRjMmNmZTZkZTMzOWQzZGQyZWIwYjQ0MWExYmE3) +[Community Slack 💭](https://litellmossslack.slack.com/) Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬