mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-05 04:22:19 +00:00
fix(ollama): thread api_base to get_model_info + graceful fallback (#21970)
* auth_with_role_name add region_name arg for cross-account sts * update tests to include case with aws_region_name for _auth_with_aws_role * Only pass region_name to STS client when aws_region_name is set * Add optional aws_sts_endpoint to _auth_with_aws_role * Parametrize ambient-credentials test for no opts, region_name, and aws_sts_endpoint * consistently passing region and endpoint args into explicit credentials irsa * fix env var leakage * fix: bedrock openai-compatible imported-model should also have model arn encoded * feat: show proxy url in ModelHub (#21660) * fix(bedrock): correct modelInput format for Converse API batch models (#21656) * fix(proxy): add model_ids param to access group endpoints for precise deployment tagging (#21655) POST /access_group/new and PUT /access_group/{name}/update now accept an optional model_ids list that targets specific deployments by their unique model_id, instead of tagging every deployment that shares a model_name. When model_ids is provided it takes priority over model_names, giving API callers the same single-deployment precision that the UI already has via PATCH /model/{model_id}/update. Backward compatible: model_names continues to work as before. Closes #21544 * feat(proxy): add custom favicon support\n\nAdd ability to configure a custom favicon for the litellm proxy UI.\n\n- Add favicon_url field to UIThemeConfig model\n- Add LITELLM_FAVICON_URL env var support\n- Add /get_favicon endpoint to serve custom favicons\n- Update ThemeContext to dynamically set favicon\n- Add favicon URL input to UI theme settings page\n- Add comprehensive tests\n\nCloses #8323 (#21653) * fix(bedrock): prevent double UUID in create_file S3 key (#21650) In create_file for Bedrock, get_complete_file_url is called twice: once in the sync handler (generating UUID-1 for api_base) and once inside transform_create_file_request (generating UUID-2 for the actual S3 upload). The Bedrock provider correctly writes UUID-2 into litellm_params["upload_url"], but the sync handler unconditionally overwrites it with api_base (UUID-1). This causes the returned file_id to point to a non-existent S3 key. Fix: only set upload_url to api_base when transform_create_file_request has not already set it, preserving the Bedrock provider's value. Closes #21546 * feat(semantic-cache): support configurable vector dimensions for Qdrant (#21649) Add vector_size parameter to QdrantSemanticCache and expose it through the Cache facade as qdrant_semantic_cache_vector_size. This allows users to use embedding models with dimensions other than the default 1536, enabling cheaper/stronger models like Stella (1024d), bge-en-icl (4096d), voyage, cohere, etc. The parameter defaults to QDRANT_VECTOR_SIZE (env var or 1536) for backward compatibility. When creating new collections, the configured vector_size is used instead of the hardcoded constant. Closes #9377 * fix(utils): normalize camelCase thinking param keys to snake_case (#21762) Clients like OpenCode's @ai-sdk/openai-compatible send budgetTokens (camelCase) instead of budget_tokens in the thinking parameter, causing validation errors. Add early normalization in completion(). * feat: add optional digest mode for Slack alert types (#21683) Adds per-alert-type digest mode that aggregates duplicate alerts within a configurable time window and emits a single summary message with count, start/end timestamps. Configuration via general_settings.alert_type_config: alert_type_config: llm_requests_hanging: digest: true digest_interval: 86400 Digest key: (alert_type, request_model, api_base) Default interval: 24 hours Window type: fixed interval Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add blog_posts.json and local backup * feat: add GetBlogPosts utility with GitHub fetch and local fallback Adds GetBlogPosts class that fetches blog posts from GitHub with a 1-hour in-process TTL cache, validates the response, and falls back to the bundled blog_posts_backup.json on any network or validation failure. * test: add cache reset fixture and LITELLM_LOCAL_BLOG_POSTS test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add GET /public/litellm_blog_posts endpoint Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: log fallback warning in blog posts endpoint and tighten test * feat: add disable_show_blog to UISettings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add useUISettings and useDisableShowBlog hooks * fix: rename useUISettings to useUISettingsFlags to avoid naming collision * fix: use existing useUISettings hook in useDisableShowBlog to avoid cache duplication Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add BlogDropdown component with react-query and error/retry state Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: enforce 5-post limit in BlogDropdown and add cap test * fix: add retry, stable post key, enabled guard in BlogDropdown Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add BlogDropdown to navbar after Docs link * feat: add network_mock transport for benchmarking proxy overhead without real API calls Intercepts at httpx transport layer so the full proxy path (auth, routing, OpenAI SDK, response transformation) is exercised with zero-latency responses. Activated via `litellm_settings: { network_mock: true }` in proxy config. * Litellm dev 02 19 2026 p2 (#21871) * feat(ui/): new guardrails monitor 'demo mock representation of what guardrails monitor looks like * fix: ui updates * style(ui/): fix styling * feat: enable running ai monitor on individual guardrails * feat: add backend logic for guardrail monitoring * fix(guardrails/usage_endpoints.py): fix usage dashboard * fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo (#21754) * fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo * fix(budget): update stale docstring on get_budget_reset_time * fix: add missing return type annotations to iterator protocol methods in streaming_handler (#21750) * fix: add return type annotations to iterator protocol methods in streaming_handler Add missing return type annotations to __iter__, __aiter__, __next__, and __anext__ methods in CustomStreamWrapper and related classes. - __iter__(self) -> Iterator["ModelResponseStream"] - __aiter__(self) -> AsyncIterator["ModelResponseStream"] - __next__(self) -> "ModelResponseStream" - __anext__(self) -> "ModelResponseStream" Also adds AsyncIterator and Iterator to typing imports. Fixes issue with PLR0915 noqa comments and ensures proper type checking support. Related to: BerriAI/litellm#8304 * fix: add ruff PLR0915 noqa for files with too many statements * Add gollem Go agent framework cookbook example (#21747) Show how to use gollem, a production Go agent framework, with LiteLLM proxy for multi-provider LLM access including tool use and streaming. * fix: avoid mutating caller-owned dicts in SpendUpdateQueue aggregation (#21742) * fix(vertex_ai): enable context-1m-2025-08-07 beta header (#21870) * server root path regression doc * fixing syntax * fix: replace Zapier webhook with Google Form for survey submission (#21621) * Replace Zapier webhook with Google Form for survey submission * Add back error logging for survey submission debugging --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> * Revert "Merge pull request #21140 from BerriAI/litellm_perf_user_api_key_auth" This reverts commit0e1db3f7e4, reversing changes made to7e2d6f2355. * test_vertex_ai_gemini_2_5_pro_streaming * UI new build * fix rendering * ui new build * docs fix * docs fix * docs fix * docs fix * docs fix * docs fix * docs fix * docs fix * release note docs * docs * adding image * fix(vertex_ai): enable context-1m-2025-08-07 beta header The `context-1m-2025-08-07` Anthropic beta header was set to `null` for vertex_ai, causing it to be filtered out when users set `extra_headers: {anthropic-beta: context-1m-2025-08-07}`. This prevented using Claude's 1M context window feature via Vertex AI, resulting in `prompt is too long: 460500 tokens > 200000 maximum` errors. Fixes #21861 --------- Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> * Revert "fix(vertex_ai): enable context-1m-2025-08-07 beta header (#21870)" (#21876) This reverts commitbce078a796. * docs(ui): add pre-PR checklist to UI contributing guide Add testing and build verification steps per maintainer feedback from @yjiang-litellm. Contributors should run their related tests per-file and ensure npm run build passes before opening PRs. * Fix entries with fast and us/ * Add tests for fast and us * Add support for Priority PayGo for vertex ai and gemini * Add model pricing * fix: ensure arrival_time is set before calculating queue time * Fix: Anthropic model wildcard access issue * Add incident report * Add ability to see which model cost map is getting used * Fix name of title * Readd tpm limit * State management fixes for CheckBatchCost * Fix PR review comments * State management fixes for CheckBatchCost - Address greptile comments * fix mypy issues: * Add Noma guardrails v2 based on custom guardrails (#21400) * Fix code qa issues * Fix mypy issues * Fix mypy issues * Fix test_aaamodel_prices_and_context_window_json_is_valid * fix: update calendly on repo * fix(tests): use counter-based mock for time.time in prisma self-heal test The test used a fixed side_effect list for time.time(), but the number of calls varies by Python version, causing StopIteration on 3.12 and AssertionError on 3.14. Replace with an infinite counter-based callable and assert the timestamp was updated rather than checking for an exact value. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tests): use absolute path for model_prices JSON in validation test The test used a relative path 'litellm/model_prices_and_context_window.json' which only works when pytest runs from a specific working directory. Use os.path based on __file__ to resolve the path reliably. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update tests/test_litellm/test_utils.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(tests): use os.path instead of Path to avoid NameError Path is not imported at module level. Use os.path.join which is already available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * clean up mock transport: remove streaming, add defensive parsing * docs: add Google GenAI SDK tutorial (JS & Python) (#21885) * docs: add Google GenAI SDK tutorial for JS and Python Add tutorial for using Google's official GenAI SDK (@google/genai for JS, google-genai for Python) with LiteLLM proxy. Covers pass-through and native router endpoints, streaming, multi-turn chat, and multi-provider routing via model_group_alias. Also updates pass-through docs to use the new SDK replacing the deprecated @google/generative-ai. * fix(docs): correct Python SDK env var name in GenAI tutorial GOOGLE_GENAI_API_KEY does not exist in the google-genai SDK. The correct env var is GEMINI_API_KEY (or GOOGLE_API_KEY). Also note that the Python SDK has no base URL env var. * fix(docs): replace non-existent GOOGLE_GENAI_BASE_URL env var in interactions.md The Python google-genai SDK does not read GOOGLE_GENAI_BASE_URL. Use http_options={"base_url": "..."} in code instead. * docs: add network mock benchmarking section * docs: tweak benchmarks wording * fix: add auth headers and empty latencies guard to benchmark script * refactor: use method-level import for MockOpenAITransport * fix: guard print_aggregate against empty latencies * fix: add INCOMPLETE status to Interactions API enum and test Google added INCOMPLETE to the Interactions API OpenAPI spec status enum. Update both the Status3 enum in the SDK types and the test's expected values to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Guardrail Monitor - measure guardrail reliability in prod (#21944) * fix: fix log viewer for guardrail monitoring * feat(ui/): fix rendering logs per guardrail * fix: fix viewing logs on overview tab of guardrail * fix: log viewer * fix: fix naming to align with metric * docs: add performance & reliability section to v1.81.14 release notes * fix(tests): make RPM limit test sequential to avoid race condition Concurrent requests via run_in_executor + asyncio.gather caused a race condition where more requests slipped through the rate limiter than expected, leading to flaky test failures (e.g. 3 successes instead of 2 with rpm_limit=2). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: Singapore guardrail policies (PDPA + MAS AI Risk Management) (#21948) * feat: Singapore PDPA PII protection guardrail policy template Add Singapore Personal Data Protection Act (PDPA) guardrail support: Regex patterns (patterns.json): - sg_nric: NRIC/FIN detection ([STFGM] + 7 digits + checksum letter) - sg_phone: Singapore phone numbers (+65/0065/65 prefix) - sg_postal_code: 6-digit postal codes (contextual) - passport_singapore: Passport numbers (E/K + 7 digits, contextual) - sg_uen: Unique Entity Numbers (3 formats) - sg_bank_account: Bank account numbers (dash format, contextual) YAML policy templates (5 sub-guardrails): - sg_pdpa_personal_identifiers: s.13 Consent - sg_pdpa_sensitive_data: Advisory Guidelines - sg_pdpa_do_not_call: Part IX DNC Registry - sg_pdpa_data_transfer: s.26 overseas transfers - sg_pdpa_profiling_automated_decisions: Model AI Governance Framework Policy template entry in policy_templates.json with 9 guardrail definitions (4 regex-based + 5 YAML conditional keyword matching). Tests: - test_sg_patterns.py: regex pattern unit tests - test_sg_pdpa_guardrails.py: conditional keyword matching tests (100+ cases) * feat: MAS AI Risk Management Guidelines guardrail policy template Add Monetary Authority of Singapore (MAS) AI Risk Management Guidelines guardrail support for financial institutions: YAML policy templates (5 sub-guardrails): - sg_mas_fairness_bias: Blocks discriminatory financial AI (credit/loans/insurance by protected attributes) - sg_mas_transparency_explainability: Blocks opaque/unexplainable AI for consequential financial decisions - sg_mas_human_oversight: Blocks fully automated financial decisions without human-in-the-loop - sg_mas_data_governance: Blocks unauthorized sharing/mishandling of financial customer data - sg_mas_model_security: Blocks adversarial attacks, model poisoning, inversion on financial AI Policy template entry in policy_templates.json with 5 guardrail definitions. Aligned with MAS FEAT Principles, Project MindForge, and NIST AI RMF. Tests: - test_sg_mas_ai_guardrails.py: conditional keyword matching tests (100+ cases) * fix: address SG pattern review feedback - Update NRIC lowercase test for IGNORECASE runtime behavior - Add keyword context guard to sg_uen pattern to reduce false positives * docs: clarify MAS AIRM timeline references - Explicitly mark MAS AIRM as Nov 2025 consultation draft - Add 2018 qualifier for FEAT principles in MAS policy descriptions - Update MAS guardrail wording to avoid release-year ambiguity * chore: commit resolved MAS policy conflicts * test: * chore: * Add OpenAI Agents SDK tutorial with LiteLLM Proxy to docs (#21221) * Add OpenAI Agents SDK tutorial to docs * Update OpenAI Agents SDK tutorial to use LiteLLM environment variables * Enhance OpenAI Agents SDK tutorial with built-in LiteLLM extension details and updated configuration steps. Adjust section headings for clarity and improve the flow of information regarding model setup and usage. * adjust blog posts to fetch from github first * feat(videos): add variant parameter to video content download (#21955) openai videos models support the features to download variants. See more details here: https://developers.openai.com/api/docs/guides/video-generation#use-image-references. Plumb variant (e.g. "thumbnail", "spritesheet") through the full video content download chain: avideo_content → video_content → video_content_handler → transform_video_content_request. OpenAI appends ?variant=<value> to the GET URL; other providers accept the parameter in their signature but ignore it. * fixing path * adjust blog post path * Revert duplicate issue checker to text-based matching, remove duplicate PR workflow Remove the Claude Code-powered duplicate PR detection workflow and revert the duplicate issue checker back to wow-actions/potential-duplicates with text similarity matching. * ui changes * adding tests * adjust default aggregation threshold * fix(videos): pass api_key from litellm_params to video remix handlers (#21965) video_remix_handler and async_video_remix_handler were not falling back to litellm_params.api_key when the api_key parameter was None, causing Authorization: Bearer None to be sent to the provider. This matches the pattern already used by async_video_generation_handler. * adding testing coverage + fixing flaky tests * fix(ollama): thread api_base through get_model_info and add graceful fallback When users pass api_base to litellm.completion() for Ollama, the model info fetch (context window, function_calling support) was ignoring the user's api_base and only reading OLLAMA_API_BASE env var or defaulting to localhost:11434. This caused confusing errors in logs when Ollama runs on a remote server. Thread api_base from litellm_params through the get_model_info call chain so OllamaConfig.get_model_info() uses the correct server. Also return safe defaults instead of raising when the server is unreachable. Fixes #21967 --------- Co-authored-by: An Tang <ta@stripe.com> Co-authored-by: janfrederickk <75388864+janfrederickk@users.noreply.github.com> Co-authored-by: Zhenting Huang <3061613175@qq.com> Co-authored-by: Darien Kindlund <darien@kindlund.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Ryan Crabbe <rcrabbe@berkeley.edu> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: LeeJuOh <56071126+LeeJuOh@users.noreply.github.com> Co-authored-by: Monesh Ram <31161039+WhoisMonesh@users.noreply.github.com> Co-authored-by: Trevor Prater <trevor.prater@gmail.com> Co-authored-by: The Mavik <179817126+themavik@users.noreply.github.com> Co-authored-by: Edwin Isac <33712823+edwiniac@users.noreply.github.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Harshit Jain <harshitjain0562@gmail.com> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: Ephrim Stanley <ephrim.stanley@point72.com> Co-authored-by: TomAlon <tom@noma.security> Co-authored-by: Julio Quinteros Pro <jquinter@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: Ron Zhong <ron-zhong@hotmail.com> Co-authored-by: Arindam Majumder <109217591+Arindam200@users.noreply.github.com> Co-authored-by: Lei Nie <lenie@quora.com>
This commit is contained in:
Cesar Garcia
co-authored by
An Tang
janfrederickk
Zhenting Huang
Darien Kindlund
Claude Opus 4.6
yuneng-jiang
Ryan Crabbe
Krish Dholakia
LeeJuOh
Monesh Ram
Trevor Prater
The Mavik
Edwin Isac
milan-berri
Ishaan Jaff
Sameer Kankute
Harshit Jain
Harshit Jain
Ephrim Stanley
TomAlon
Julio Quinteros Pro
greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
ryan-crabbe
Ron Zhong
Arindam Majumder
Lei Nie
An Tang
janfrederickk
Zhenting Huang
Darien Kindlund
Claude Opus 4.6
yuneng-jiang
Ryan Crabbe
Krish Dholakia
LeeJuOh
Monesh Ram
Trevor Prater
The Mavik
Edwin Isac
milan-berri
Ishaan Jaff
Sameer Kankute
Harshit Jain
Harshit Jain
Ephrim Stanley
TomAlon
Julio Quinteros Pro
greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
ryan-crabbe
Ron Zhong
Arindam Majumder
Lei Nie
parent
470444febc
commit
9495f4e941
@@ -1,7 +1,7 @@
|
||||
blank_issues_enabled: true
|
||||
contact_links:
|
||||
- name: Schedule Demo
|
||||
url: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
|
||||
url: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions
|
||||
about: Speak directly with Krrish and Ishaan, the founders, to discuss issues, share feedback, or explore improvements for LiteLLM
|
||||
- name: Discord
|
||||
url: https://discord.com/invite/wuPM9dRgDw
|
||||
|
||||
@@ -2,47 +2,28 @@ name: Check Duplicate Issues
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [opened]
|
||||
types: [opened, edited]
|
||||
|
||||
jobs:
|
||||
check-duplicates:
|
||||
if: github.event.action == 'opened'
|
||||
check-duplicate:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
contents: read
|
||||
steps:
|
||||
- name: Install Claude Code
|
||||
run: npm install -g @anthropic-ai/claude-code
|
||||
|
||||
- name: Check duplicates
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.LITELLM_VIRTUAL_KEY }}
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.LITELLM_BASE_URL }}
|
||||
- name: Check for potential duplicates
|
||||
uses: wow-actions/potential-duplicates@v1
|
||||
with:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PROMPT: |
|
||||
A new issue has been created in the ${{ github.repository }} repository.
|
||||
label: potential-duplicate
|
||||
threshold: 0.6
|
||||
reaction: eyes
|
||||
comment: |
|
||||
**⚠️ Potential duplicate detected**
|
||||
|
||||
Issue number: ${{ github.event.issue.number }}
|
||||
This issue appears similar to existing issue(s):
|
||||
{{#issues}}
|
||||
- [#{{number}}]({{html_url}}) - {{title}} ({{accuracy}}% similar)
|
||||
{{/issues}}
|
||||
|
||||
Lookup this issue with gh issue view ${{ github.event.issue.number }} --repo ${{ github.repository }}.
|
||||
|
||||
Search through existing issues (excluding #${{ github.event.issue.number }}) to find potential duplicates.
|
||||
|
||||
Use gh issue list --repo ${{ github.repository }} with relevant search terms from the new issue's title and description. Try multiple keyword combinations to search broadly. Check both open and recently closed issues.
|
||||
|
||||
Consider:
|
||||
1. Similar titles or descriptions
|
||||
2. Same error messages or symptoms
|
||||
3. Related functionality or components
|
||||
4. Similar feature requests
|
||||
|
||||
If you find potential duplicates, post a SINGLE comment on issue #${{ github.event.issue.number }} using gh issue comment ${{ github.event.issue.number }} --repo ${{ github.repository }} with this format:
|
||||
|
||||
_This comment was generated by an LLM and may be inaccurate._
|
||||
|
||||
This issue might be a duplicate of existing issues. Please check:
|
||||
- #[issue_number]: [brief description of similarity]
|
||||
|
||||
If you find NO duplicates, do NOT post any comment. Stay silent.
|
||||
run: claude -p "$PROMPT" --model sonnet --max-turns 10 --allowedTools "Bash(gh issue *)"
|
||||
Please review the linked issue(s) to see if they address your concern. If this is not a duplicate, please provide additional context to help us understand the difference.
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
name: Check Duplicate PRs
|
||||
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
jobs:
|
||||
check-duplicates:
|
||||
if: |
|
||||
github.event.pull_request.user.login != 'ishaan-jaff' &&
|
||||
github.event.pull_request.user.login != 'krrishdholakia' &&
|
||||
github.event.pull_request.user.login != 'actions-user' &&
|
||||
!endsWith(github.event.pull_request.user.login, '[bot]')
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Install Claude Code
|
||||
run: npm install -g @anthropic-ai/claude-code
|
||||
|
||||
- name: Check duplicates
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.LITELLM_VIRTUAL_KEY }}
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.LITELLM_BASE_URL }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PROMPT: |
|
||||
A new PR has been opened in the ${{ github.repository }} repository.
|
||||
|
||||
PR number: ${{ github.event.pull_request.number }}
|
||||
|
||||
Lookup this PR with gh pr view ${{ github.event.pull_request.number }} --repo ${{ github.repository }}.
|
||||
|
||||
Search through existing open PRs (excluding #${{ github.event.pull_request.number }}) to find potential duplicates.
|
||||
|
||||
Use gh pr list --repo ${{ github.repository }} with relevant search terms from the new PR's title and description. Try multiple keyword combinations to search broadly. Check both open and recently closed PRs.
|
||||
|
||||
Consider:
|
||||
1. Similar titles or descriptions
|
||||
2. Same bug fix or feature being implemented
|
||||
3. Related functionality or components
|
||||
4. Overlapping code changes (same files or areas)
|
||||
|
||||
If you find potential duplicates, post a SINGLE comment on PR #${{ github.event.pull_request.number }} using gh pr comment ${{ github.event.pull_request.number }} --repo ${{ github.repository }} with this format:
|
||||
|
||||
_This comment was generated by an LLM and may be inaccurate._
|
||||
|
||||
This PR might be a duplicate of existing PRs. Please check:
|
||||
- #[pr_number]: [brief description of similarity]
|
||||
|
||||
If you find NO duplicates, do NOT post any comment. Stay silent.
|
||||
run: claude -p "$PROMPT" --model sonnet --max-turns 10 --allowedTools "Bash(gh pr *)"
|
||||
@@ -123,7 +123,7 @@ if __name__ == "__main__":
|
||||
+ docker_run_command
|
||||
+ "\n\n"
|
||||
+ "### Don't want to maintain your internal proxy? get in touch 🎉"
|
||||
+ "\nHosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
|
||||
+ "\nHosted Proxy Alpha: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions"
|
||||
+ "\n\n"
|
||||
+ "## Load Test LiteLLM Proxy Results"
|
||||
+ "\n\n"
|
||||
|
||||
@@ -399,7 +399,7 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature
|
||||
# Enterprise
|
||||
For companies that need better security, user management and professional support
|
||||
|
||||
[Talk to founders](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Talk to founders](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
This covers:
|
||||
- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):**
|
||||
|
||||
@@ -178,4 +178,4 @@ Benchmark Results for 'When will BerriAI IPO?':
|
||||
```
|
||||
|
||||
## Support
|
||||
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
|
||||
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# Gollem Go Agent Framework with LiteLLM
|
||||
|
||||
A working example showing how to use [gollem](https://github.com/fugue-labs/gollem), a production-grade Go agent framework, with LiteLLM as a proxy gateway. This lets Go developers access 100+ LLM providers through a single proxy while keeping compile-time type safety for tools and structured output.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Start LiteLLM Proxy
|
||||
|
||||
```bash
|
||||
# Simple start with a single model
|
||||
litellm --model gpt-4o
|
||||
|
||||
# Or with the example config for multi-provider access
|
||||
litellm --config proxy_config.yaml
|
||||
```
|
||||
|
||||
### 2. Run the examples
|
||||
|
||||
```bash
|
||||
# Install Go dependencies
|
||||
go mod tidy
|
||||
|
||||
# Basic agent
|
||||
go run ./basic
|
||||
|
||||
# Agent with type-safe tools
|
||||
go run ./tools
|
||||
|
||||
# Streaming responses
|
||||
go run ./streaming
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
The included `proxy_config.yaml` sets up three providers through LiteLLM:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-4o # OpenAI
|
||||
- model_name: claude-sonnet # Anthropic
|
||||
- model_name: gemini-pro # Google Vertex AI
|
||||
```
|
||||
|
||||
Switch providers in Go by changing a single string — no code changes needed:
|
||||
|
||||
```go
|
||||
model := openai.NewLiteLLM("http://localhost:4000",
|
||||
openai.WithModel("gpt-4o"), // OpenAI
|
||||
// openai.WithModel("claude-sonnet"), // Anthropic
|
||||
// openai.WithModel("gemini-pro"), // Google
|
||||
)
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### `basic/` — Basic Agent
|
||||
|
||||
Connects gollem to LiteLLM and runs a simple prompt. Demonstrates the `NewLiteLLM` constructor and basic agent creation.
|
||||
|
||||
### `tools/` — Type-Safe Tools
|
||||
|
||||
Shows gollem's compile-time type-safe tool framework working through LiteLLM's tool-use passthrough. The tool parameters are Go structs with JSON tags — the schema is generated automatically at compile time.
|
||||
|
||||
### `streaming/` — Streaming Responses
|
||||
|
||||
Real-time token streaming using Go 1.23+ range-over-function iterators, proxied through LiteLLM's SSE passthrough.
|
||||
|
||||
## How It Works
|
||||
|
||||
Gollem's `openai.NewLiteLLM()` constructor creates an OpenAI-compatible provider pointed at your LiteLLM proxy. Since LiteLLM speaks the OpenAI API protocol, everything works out of the box:
|
||||
|
||||
- **Chat completions** — standard request/response
|
||||
- **Tool use** — LiteLLM passes tool definitions and calls through transparently
|
||||
- **Streaming** — Server-Sent Events proxied through LiteLLM
|
||||
- **Structured output** — JSON schema response format works with supporting models
|
||||
|
||||
```
|
||||
Go App (gollem) → LiteLLM Proxy → OpenAI / Anthropic / Google / ...
|
||||
```
|
||||
|
||||
## Why Use This?
|
||||
|
||||
- **Type-safe Go**: Compile-time type checking for tools, structured output, and agent configuration — no runtime surprises
|
||||
- **Single proxy, many models**: Switch between OpenAI, Anthropic, Google, and 100+ other providers by changing a model name string
|
||||
- **Zero-dependency core**: gollem's core has no external dependencies — just stdlib
|
||||
- **Single binary deployment**: `go build` produces one binary, no pip/venv/Docker needed
|
||||
- **Cost tracking & rate limiting**: LiteLLM handles cost tracking, rate limits, and fallbacks at the proxy layer
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# Required for providers you want to use (set in LiteLLM config or env)
|
||||
export OPENAI_API_KEY="sk-..."
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
|
||||
# Optional: point to a non-default LiteLLM proxy
|
||||
export LITELLM_PROXY_URL="http://localhost:4000"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Connection errors?**
|
||||
- Make sure LiteLLM is running: `litellm --model gpt-4o`
|
||||
- Check the URL is correct (default: `http://localhost:4000`)
|
||||
|
||||
**Model not found?**
|
||||
- Verify the model name matches what's configured in LiteLLM
|
||||
- Run `curl http://localhost:4000/models` to see available models
|
||||
|
||||
**Tool calls not working?**
|
||||
- Ensure the underlying model supports tool use (GPT-4o, Claude, Gemini)
|
||||
- Check LiteLLM logs for any provider-specific errors
|
||||
|
||||
## Learn More
|
||||
|
||||
- [gollem GitHub](https://github.com/fugue-labs/gollem)
|
||||
- [gollem API Reference](https://pkg.go.dev/github.com/fugue-labs/gollem/core)
|
||||
- [LiteLLM Proxy Docs](https://docs.litellm.ai/docs/simple_proxy)
|
||||
- [LiteLLM Supported Models](https://docs.litellm.ai/docs/providers)
|
||||
@@ -0,0 +1,41 @@
|
||||
// Basic gollem agent connected to a LiteLLM proxy.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// litellm --model gpt-4o # start proxy in another terminal
|
||||
// go run ./basic
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"github.com/fugue-labs/gollem/core"
|
||||
"github.com/fugue-labs/gollem/provider/openai"
|
||||
)
|
||||
|
||||
func main() {
|
||||
proxyURL := "http://localhost:4000"
|
||||
if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
|
||||
proxyURL = u
|
||||
}
|
||||
|
||||
// Connect to LiteLLM proxy. NewLiteLLM creates an OpenAI-compatible
|
||||
// provider pointed at the given URL.
|
||||
model := openai.NewLiteLLM(proxyURL,
|
||||
openai.WithModel("gpt-4o"), // any model name configured in LiteLLM
|
||||
)
|
||||
|
||||
// Create and run a simple agent.
|
||||
agent := core.NewAgent[string](model,
|
||||
core.WithSystemPrompt[string]("You are a helpful assistant. Be concise."),
|
||||
)
|
||||
|
||||
result, err := agent.Run(context.Background(), "Explain quantum computing in two sentences.")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println(result.Output)
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
|
||||
|
||||
go 1.25.1
|
||||
|
||||
require github.com/fugue-labs/gollem v0.1.0
|
||||
@@ -0,0 +1,2 @@
|
||||
github.com/fugue-labs/gollem v0.1.0 h1:QexYnvkb44QZFEljgAePqMIGZjgsbk0Y5GJ2jYYgfa8=
|
||||
github.com/fugue-labs/gollem v0.1.0/go.mod h1:htW1YO81uysSKVOkYJtxhGCFrzm+36HBFxEWuECoHKQ=
|
||||
@@ -0,0 +1,16 @@
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-20250514
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
- model_name: gemini-pro
|
||||
litellm_params:
|
||||
model: vertex_ai/gemini-2.0-flash
|
||||
vertex_project: my-project
|
||||
vertex_location: us-central1
|
||||
@@ -0,0 +1,56 @@
|
||||
// Streaming responses from gollem through LiteLLM.
|
||||
//
|
||||
// Uses Go 1.23+ range-over-function iterators for real-time token
|
||||
// streaming via LiteLLM's SSE passthrough.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// litellm --model gpt-4o
|
||||
// go run ./streaming
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"github.com/fugue-labs/gollem/core"
|
||||
"github.com/fugue-labs/gollem/provider/openai"
|
||||
)
|
||||
|
||||
func main() {
|
||||
proxyURL := "http://localhost:4000"
|
||||
if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
|
||||
proxyURL = u
|
||||
}
|
||||
|
||||
model := openai.NewLiteLLM(proxyURL,
|
||||
openai.WithModel("gpt-4o"),
|
||||
)
|
||||
|
||||
agent := core.NewAgent[string](model)
|
||||
|
||||
// RunStream returns a streaming result that yields tokens as they arrive.
|
||||
stream, err := agent.RunStream(context.Background(), "Write a haiku about distributed systems")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// StreamText yields text chunks in real-time.
|
||||
// The boolean argument controls whether deltas (true) or accumulated
|
||||
// text (false) is returned.
|
||||
fmt.Print("Response: ")
|
||||
for text, err := range stream.StreamText(true) {
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Print(text)
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
// After streaming completes, the final response is available.
|
||||
resp := stream.Response()
|
||||
fmt.Printf("\nTokens used: input=%d, output=%d\n",
|
||||
resp.Usage.InputTokens, resp.Usage.OutputTokens)
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
// Gollem agent with type-safe tools through LiteLLM.
|
||||
//
|
||||
// The tool parameters are Go structs — gollem generates the JSON schema
|
||||
// automatically at compile time. LiteLLM passes tool definitions through
|
||||
// transparently to the underlying provider.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// litellm --model gpt-4o
|
||||
// go run ./tools
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"github.com/fugue-labs/gollem/core"
|
||||
"github.com/fugue-labs/gollem/provider/openai"
|
||||
)
|
||||
|
||||
// WeatherParams defines the tool's input schema via struct tags.
|
||||
// The JSON schema is generated at compile time — no runtime reflection needed.
|
||||
type WeatherParams struct {
|
||||
City string `json:"city" description:"City name to get weather for"`
|
||||
Unit string `json:"unit,omitempty" description:"Temperature unit: celsius or fahrenheit"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
proxyURL := "http://localhost:4000"
|
||||
if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
|
||||
proxyURL = u
|
||||
}
|
||||
|
||||
model := openai.NewLiteLLM(proxyURL,
|
||||
openai.WithModel("gpt-4o"),
|
||||
)
|
||||
|
||||
// Define a type-safe tool. The function signature enforces correct types.
|
||||
weatherTool := core.FuncTool[WeatherParams](
|
||||
"get_weather",
|
||||
"Get current weather for a city",
|
||||
func(ctx context.Context, p WeatherParams) (string, error) {
|
||||
unit := p.Unit
|
||||
if unit == "" {
|
||||
unit = "fahrenheit"
|
||||
}
|
||||
// In production, call a real weather API here.
|
||||
return fmt.Sprintf("Weather in %s: 72°F (22°C), sunny", p.City), nil
|
||||
},
|
||||
)
|
||||
|
||||
agent := core.NewAgent[string](model,
|
||||
core.WithTools[string](weatherTool),
|
||||
core.WithSystemPrompt[string]("You are a helpful weather assistant. Use the get_weather tool to answer weather questions."),
|
||||
)
|
||||
|
||||
result, err := agent.Run(context.Background(), "What's the weather like in San Francisco and Tokyo?")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println(result.Output)
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
slug: anthropic-wildcard-model-access-incident
|
||||
title: "Incident Report: Wildcard Blocking New Models After Cost Map Reload"
|
||||
date: 2026-02-23T10:00:00
|
||||
authors:
|
||||
- name: Sameer Kankute
|
||||
title: SWE @ LiteLLM (LLM Translation)
|
||||
url: https://www.linkedin.com/in/sameer-kankute/
|
||||
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
|
||||
- name: Krrish Dholakia
|
||||
title: "CEO, LiteLLM"
|
||||
url: https://www.linkedin.com/in/krish-d/
|
||||
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
|
||||
- name: Ishaan Jaff
|
||||
title: "CTO, LiteLLM"
|
||||
url: https://www.linkedin.com/in/reffajnaahsi/
|
||||
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
|
||||
tags: [incident-report, proxy, auth, model-access]
|
||||
hide_table_of_contents: false
|
||||
---
|
||||
|
||||
**Date:** Feb 23, 2026
|
||||
**Duration:** ~3 hours
|
||||
**Severity:** High (for users with provider wildcard access rules)
|
||||
**Status:** Resolved
|
||||
|
||||
## Summary
|
||||
|
||||
When a new Anthropic model (e.g. `claude-sonnet-4-6`) was added to the LiteLLM model cost map and a cost map reload was triggered, requests to the new model were rejected with:
|
||||
|
||||
```
|
||||
key not allowed to access model. This key can only access models=['anthropic/*']. Tried to access claude-sonnet-4-6.
|
||||
```
|
||||
|
||||
The reload updated `litellm.model_cost` correctly but never re-ran `add_known_models()`, so `litellm.anthropic_models` (the in-memory set used by the wildcard resolver) remained stale. The new model was invisible to the `anthropic/*` wildcard even though the cost map knew about it.
|
||||
|
||||
- **LLM calls:** All requests to newly-added Anthropic models were blocked with a 401.
|
||||
- **Existing models:** Unaffected — only models missing from the stale provider set were impacted.
|
||||
- **Other providers:** Same bug class existed for any provider wildcard (e.g. `openai/*`, `gemini/*`).
|
||||
|
||||
{/* truncate */}
|
||||
|
||||
---
|
||||
|
||||
## Background
|
||||
|
||||
LiteLLM supports provider-level wildcard access rules. When an admin configures a key or team with `models=['anthropic/*']`, any model whose provider resolves to `anthropic` should be allowed. The resolution happens in `_model_custom_llm_provider_matches_wildcard_pattern`:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["1. Request arrives for claude-sonnet-4-6"] --> B["2. Auth check: can this key call this model?
|
||||
proxy/auth/auth_checks.py"]
|
||||
B --> C["3. Key has models=['anthropic/*']
|
||||
→ wildcard match attempted"]
|
||||
C --> D["4. get_llm_provider('claude-sonnet-4-6')
|
||||
checks litellm.anthropic_models set"]
|
||||
D -->|"model IN set"| E["5a. ✅ Provider = 'anthropic'
|
||||
→ 'anthropic/claude-sonnet-4-6' matches 'anthropic/*'"]
|
||||
D -->|"model NOT IN set"| F["5b. ❌ Provider unknown
|
||||
→ exception raised → wildcard returns False"]
|
||||
E --> G["6. Request allowed"]
|
||||
F --> H["6. 401: key not allowed to access model"]
|
||||
|
||||
style E fill:#d4edda,stroke:#28a745
|
||||
style F fill:#f8d7da,stroke:#dc3545
|
||||
style H fill:#f8d7da,stroke:#dc3545
|
||||
style D fill:#fff3cd,stroke:#ffc107
|
||||
```
|
||||
|
||||
`litellm.anthropic_models` is a Python `set` populated at import time by `add_known_models()`. It is the source `get_llm_provider()` consults to map a bare model name like `claude-sonnet-4-6` to the provider string `"anthropic"`.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
`add_known_models()` is called **once** at module import time. Both reload paths in `proxy_server.py` updated `litellm.model_cost` with the fresh map but never called `add_known_models()` again:
|
||||
|
||||
```python
|
||||
# Before the fix — both reload paths looked like this:
|
||||
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
|
||||
litellm.model_cost = new_model_cost_map # ✅ cost map updated
|
||||
_invalidate_model_cost_lowercase_map() # ✅ cache cleared
|
||||
# ❌ add_known_models() never called
|
||||
# → litellm.anthropic_models still has the old set
|
||||
# → new model not in the set
|
||||
# → get_llm_provider() raises for the new model
|
||||
# → wildcard match returns False
|
||||
# → 401 for every request to the new model
|
||||
```
|
||||
|
||||
The gap existed in two places:
|
||||
1. `_check_and_reload_model_cost_map` — the periodic automatic reload (every 10 s)
|
||||
2. The `/reload/model_cost_map` admin endpoint — the manual reload
|
||||
|
||||
**Timeline:**
|
||||
|
||||
1. New model (`claude-sonnet-4-6`) added to `model_prices_and_context_window.json`
|
||||
2. Admin triggers cost map reload via UI → `litellm.model_cost` updated
|
||||
3. Users with `anthropic/*` wildcard keys attempt requests to `claude-sonnet-4-6`
|
||||
4. `get_llm_provider('claude-sonnet-4-6')` raises → wildcard returns False → 401
|
||||
5. Admin reloads cost map again — same result (root cause not addressed)
|
||||
6. ~3 hours of investigation → root cause identified → fix deployed
|
||||
|
||||
---
|
||||
|
||||
## The Fix
|
||||
|
||||
After each reload, `add_known_models()` is called with the freshly fetched map passed explicitly. Passing the map directly (rather than relying on the module-level reference) removes any ambiguity about which dict is iterated:
|
||||
|
||||
```python
|
||||
# After the fix — both reload paths now do:
|
||||
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
|
||||
litellm.model_cost = new_model_cost_map
|
||||
_invalidate_model_cost_lowercase_map()
|
||||
litellm.add_known_models(model_cost_map=new_model_cost_map) # ✅ sets repopulated
|
||||
```
|
||||
|
||||
`add_known_models()` was also updated to accept an optional explicit map so callers cannot accidentally iterate a stale module-level reference:
|
||||
|
||||
```python
|
||||
# Before
|
||||
def add_known_models():
|
||||
for key, value in model_cost.items(): # reads module global — ambiguous after reload
|
||||
...
|
||||
|
||||
# After
|
||||
def add_known_models(model_cost_map: Optional[Dict] = None):
|
||||
_map = model_cost_map if model_cost_map is not None else model_cost
|
||||
for key, value in _map.items(): # always iterates the map you just fetched
|
||||
...
|
||||
```
|
||||
|
||||
After the fix, the provider sets (`anthropic_models`, `open_ai_chat_completion_models`, etc.) are always consistent with `litellm.model_cost` immediately after every reload. New models become accessible via wildcard rules without any proxy restart.
|
||||
|
||||
---
|
||||
|
||||
## Remediation
|
||||
|
||||
| # | Action | Status | Code |
|
||||
|---|---|---|---|
|
||||
| 1 | Call `add_known_models(model_cost_map=...)` in the periodic reload path | ✅ Done | [`proxy_server.py#L4393`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L4393) |
|
||||
| 2 | Call `add_known_models(model_cost_map=...)` in the `/reload/model_cost_map` endpoint | ✅ Done | [`proxy_server.py#L11904`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L11904) |
|
||||
| 3 | Update `add_known_models()` to accept an explicit map parameter | ✅ Done | [`__init__.py#L617`](https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py#L617) |
|
||||
| 4 | Regression test: `add_known_models(model_cost_map=...)` populates provider sets | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
|
||||
| 5 | Regression test: `anthropic/*` wildcard grants/denies access correctly after reload | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
|
||||
|
||||
---
|
||||
@@ -5,6 +5,44 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.
|
||||
|
||||
## Setting Up Benchmarking with Network Mock
|
||||
|
||||
The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider.
|
||||
|
||||
**1. Create a proxy config:**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: db-openai-endpoint
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: "sk-fake-key"
|
||||
api_base: "https://api.openai.com"
|
||||
|
||||
litellm_settings:
|
||||
network_mock: true
|
||||
callbacks: []
|
||||
num_retries: 0
|
||||
request_timeout: 30
|
||||
|
||||
general_settings:
|
||||
master_key: "sk-1234"
|
||||
```
|
||||
|
||||
**2. Start the proxy:**
|
||||
|
||||
```bash
|
||||
litellm --config benchmark_config.yaml --port 4000 --num_workers 8
|
||||
```
|
||||
|
||||
**3. Run the benchmark script:**
|
||||
|
||||
```bash
|
||||
python scripts/benchmark_mock.py --requests 2000 --max-concurrent 200 --runs 3
|
||||
```
|
||||
|
||||
This measures pure proxy overhead on the hot path without any network latency to a real or fake provider.
|
||||
|
||||
## Setting Up a Fake OpenAI Endpoint
|
||||
|
||||
For load testing and benchmarking, you can use a fake OpenAI proxy server. LiteLLM provides:
|
||||
|
||||
@@ -297,6 +297,7 @@ litellm.cache = Cache(
|
||||
similarity_threshold=0.7, # similarity threshold for cache hits, 0 == no similarity, 1 = exact matches, 0.5 == 50% similarity
|
||||
qdrant_quantization_config ="binary", # can be one of 'binary', 'product' or 'scalar' quantizations that is supported by qdrant
|
||||
qdrant_semantic_cache_embedding_model="text-embedding-ada-002", # this model is passed to litellm.embedding(), any litellm.embedding() model is supported here
|
||||
qdrant_semantic_cache_vector_size=1536, # vector size for the embedding model, must match the dimensionality of the embedding model used
|
||||
)
|
||||
|
||||
response1 = completion(
|
||||
@@ -635,6 +636,7 @@ def __init__(
|
||||
qdrant_quantization_config: Optional[str] = None,
|
||||
qdrant_semantic_cache_embedding_model="text-embedding-ada-002",
|
||||
|
||||
qdrant_semantic_cache_vector_size: Optional[int] = None,
|
||||
**kwargs
|
||||
):
|
||||
```
|
||||
|
||||
@@ -79,7 +79,27 @@ cp -r out/* ../../litellm/proxy/_experimental/out/
|
||||
|
||||
Then restart the proxy and access the UI at `http://localhost:4000/ui`
|
||||
|
||||
## 4. Submitting a PR
|
||||
## 4. Pre-PR Checklist
|
||||
|
||||
Before submitting your pull request, make sure the following pass locally from `ui/litellm-dashboard/`:
|
||||
|
||||
**Run tests related to your changes:**
|
||||
|
||||
```bash
|
||||
npx vitest run src/components/path/to/YourComponent.test.tsx
|
||||
```
|
||||
|
||||
Tests are co-located with components (e.g., `TeamInfo.tsx` → `TeamInfo.test.tsx`). If you add a new component, add a corresponding `.test.tsx` file next to it.
|
||||
|
||||
**Run the build:**
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
These map to the `ui_tests` and `ui_build` CI checks.
|
||||
|
||||
## 5. Submitting a PR
|
||||
|
||||
1. Create a new branch for your changes:
|
||||
```bash
|
||||
|
||||
@@ -4,7 +4,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
:::info
|
||||
- ✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
|
||||
- Who is Enterprise for? Companies giving access to 100+ users **OR** 10+ AI use-cases. If you're not sure, [get in touch with us](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) to discuss your needs.
|
||||
- Who is Enterprise for? Companies giving access to 100+ users **OR** 10+ AI use-cases. If you're not sure, [get in touch with us](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) to discuss your needs.
|
||||
:::
|
||||
|
||||
For companies that need SSO, user management and professional support for LiteLLM Proxy
|
||||
@@ -36,7 +36,7 @@ Manage Yourself - you can deploy our Docker Image or build a custom image from o
|
||||
|
||||
### What’s the cost of the Self-Managed Enterprise edition?
|
||||
|
||||
Self-Managed Enterprise deployments require our team to understand your exact needs. [Get in touch with us to learn more](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
Self-Managed Enterprise deployments require our team to understand your exact needs. [Get in touch with us to learn more](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
|
||||
### How does deployment with Enterprise License work?
|
||||
@@ -106,7 +106,7 @@ Professional Support can assist with LLM/Provider integrations, deployment, upgr
|
||||
|
||||
Pricing is based on usage. We can figure out a price that works for your team, on the call.
|
||||
|
||||
[**Contact Us to learn more**](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[**Contact Us to learn more**](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
:::info
|
||||
|
||||
This is an Enterprise only endpoint [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
This is an Enterprise only endpoint [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -130,13 +130,12 @@ Point the Google GenAI SDK to LiteLLM Proxy:
|
||||
|
||||
```python showLineNumbers title="Google GenAI SDK with LiteLLM Proxy"
|
||||
from google import genai
|
||||
import os
|
||||
|
||||
# Point SDK to LiteLLM Proxy
|
||||
os.environ["GOOGLE_GENAI_BASE_URL"] = "http://localhost:4000"
|
||||
os.environ["GEMINI_API_KEY"] = "sk-1234" # Your LiteLLM API key
|
||||
|
||||
client = genai.Client()
|
||||
client = genai.Client(
|
||||
api_key="sk-1234", # Your LiteLLM API key
|
||||
http_options={"base_url": "http://localhost:4000"},
|
||||
)
|
||||
|
||||
# Create an interaction
|
||||
interaction = client.interactions.create(
|
||||
@@ -151,12 +150,11 @@ print(interaction.outputs[-1].text)
|
||||
|
||||
```python showLineNumbers title="Google GenAI SDK Streaming"
|
||||
from google import genai
|
||||
import os
|
||||
|
||||
os.environ["GOOGLE_GENAI_BASE_URL"] = "http://localhost:4000"
|
||||
os.environ["GEMINI_API_KEY"] = "sk-1234"
|
||||
|
||||
client = genai.Client()
|
||||
client = genai.Client(
|
||||
api_key="sk-1234", # Your LiteLLM API key
|
||||
http_options={"base_url": "http://localhost:4000"},
|
||||
)
|
||||
|
||||
for chunk in client.interactions.create_stream(
|
||||
model="gemini/gemini-2.5-flash",
|
||||
|
||||
@@ -6,7 +6,7 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -35,26 +35,25 @@ curl 'http://0.0.0.0:4000/gemini/v1beta/models/gemini-1.5-flash:countTokens?key=
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="js" label="Google AI Node.js SDK">
|
||||
<TabItem value="js" label="Google GenAI JS SDK">
|
||||
|
||||
```javascript
|
||||
const { GoogleGenerativeAI } = require("@google/generative-ai");
|
||||
const { GoogleGenAI } = require("@google/genai");
|
||||
|
||||
const modelParams = {
|
||||
model: 'gemini-pro',
|
||||
};
|
||||
|
||||
const requestOptions = {
|
||||
baseUrl: 'http://localhost:4000/gemini', // http://<proxy-base-url>/gemini
|
||||
};
|
||||
|
||||
const genAI = new GoogleGenerativeAI("sk-1234"); // litellm proxy API key
|
||||
const model = genAI.getGenerativeModel(modelParams, requestOptions);
|
||||
const ai = new GoogleGenAI({
|
||||
apiKey: "sk-1234", // litellm proxy API key
|
||||
httpOptions: {
|
||||
baseUrl: "http://localhost:4000/gemini", // http://<proxy-base-url>/gemini
|
||||
},
|
||||
});
|
||||
|
||||
async function main() {
|
||||
try {
|
||||
const result = await model.generateContent("Explain how AI works");
|
||||
console.log(result.response.text());
|
||||
const response = await ai.models.generateContent({
|
||||
model: "gemini-2.5-flash",
|
||||
contents: "Explain how AI works",
|
||||
});
|
||||
console.log(response.text);
|
||||
} catch (error) {
|
||||
console.error('Error:', error);
|
||||
}
|
||||
@@ -63,12 +62,13 @@ async function main() {
|
||||
// For streaming responses
|
||||
async function main_streaming() {
|
||||
try {
|
||||
const streamingResult = await model.generateContentStream("Explain how AI works");
|
||||
for await (const chunk of streamingResult.stream) {
|
||||
console.log('Stream chunk:', JSON.stringify(chunk));
|
||||
const response = await ai.models.generateContentStream({
|
||||
model: "gemini-2.5-flash",
|
||||
contents: "Explain how AI works",
|
||||
});
|
||||
for await (const chunk of response) {
|
||||
process.stdout.write(chunk.text);
|
||||
}
|
||||
const aggregatedResponse = await streamingResult.response;
|
||||
console.log('Aggregated response:', JSON.stringify(aggregatedResponse));
|
||||
} catch (error) {
|
||||
console.error('Error:', error);
|
||||
}
|
||||
@@ -321,29 +321,28 @@ curl 'http://0.0.0.0:4000/gemini/v1beta/models/gemini-1.5-flash:generateContent?
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="js" label="Google AI Node.js SDK">
|
||||
<TabItem value="js" label="Google GenAI JS SDK">
|
||||
|
||||
```javascript
|
||||
const { GoogleGenerativeAI } = require("@google/generative-ai");
|
||||
const { GoogleGenAI } = require("@google/genai");
|
||||
|
||||
const modelParams = {
|
||||
model: 'gemini-pro',
|
||||
};
|
||||
|
||||
const requestOptions = {
|
||||
baseUrl: 'http://localhost:4000/gemini', // http://<proxy-base-url>/gemini
|
||||
customHeaders: {
|
||||
"tags": "gemini-js-sdk,pass-through-endpoint"
|
||||
}
|
||||
};
|
||||
|
||||
const genAI = new GoogleGenerativeAI("sk-1234");
|
||||
const model = genAI.getGenerativeModel(modelParams, requestOptions);
|
||||
const ai = new GoogleGenAI({
|
||||
apiKey: "sk-1234",
|
||||
httpOptions: {
|
||||
baseUrl: "http://localhost:4000/gemini", // http://<proxy-base-url>/gemini
|
||||
headers: {
|
||||
"tags": "gemini-js-sdk,pass-through-endpoint",
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
async function main() {
|
||||
try {
|
||||
const result = await model.generateContent("Explain how AI works");
|
||||
console.log(result.response.text());
|
||||
const response = await ai.models.generateContent({
|
||||
model: "gemini-2.5-flash",
|
||||
contents: "Explain how AI works",
|
||||
});
|
||||
console.log(response.text);
|
||||
} catch (error) {
|
||||
console.error('Error:', error);
|
||||
}
|
||||
|
||||
@@ -438,6 +438,59 @@ curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \
|
||||
|
||||
- `event_message` *str*: A human-readable description of the event.
|
||||
|
||||
### Digest Mode (Reducing Alert Noise)
|
||||
|
||||
By default, LiteLLM sends a separate Slack message for **every** alert event. For high-frequency alert types like `llm_requests_hanging` or `llm_too_slow`, this can produce hundreds of duplicate messages per day.
|
||||
|
||||
**Digest mode** aggregates duplicate alerts within a configurable time window and emits a single summary message with the total count and time range.
|
||||
|
||||
#### Configuration
|
||||
|
||||
Use `alert_type_config` in `general_settings` to enable digest mode per alert type:
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
alerting: ["slack"]
|
||||
alert_type_config:
|
||||
llm_requests_hanging:
|
||||
digest: true
|
||||
digest_interval: 86400 # 24 hours (default)
|
||||
llm_too_slow:
|
||||
digest: true
|
||||
digest_interval: 3600 # 1 hour
|
||||
llm_exceptions:
|
||||
digest: true
|
||||
# uses default interval (86400 seconds / 24 hours)
|
||||
```
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `digest` | bool | `false` | Enable digest mode for this alert type |
|
||||
| `digest_interval` | int | `86400` (24h) | Time window in seconds. Alerts are aggregated within this interval. |
|
||||
|
||||
#### How It Works
|
||||
|
||||
1. When an alert fires for a digest-enabled type, it is **grouped** by `(alert_type, request_model, api_base)` instead of being sent immediately
|
||||
2. A counter tracks how many times the alert fires within the interval
|
||||
3. When the interval expires, a **single summary message** is sent:
|
||||
|
||||
```
|
||||
Alert type: `llm_requests_hanging` (Digest)
|
||||
Level: `Medium`
|
||||
Start: `2026-02-19 03:27:39`
|
||||
End: `2026-02-20 03:27:39`
|
||||
Count: `847`
|
||||
|
||||
Message: `Requests are hanging - 600s+ request time`
|
||||
Request Model: `gemini-2.5-flash`
|
||||
API Base: `None`
|
||||
```
|
||||
|
||||
#### Limitations
|
||||
|
||||
- **Per-instance**: Digest state is held in memory per proxy instance. If you run multiple instances (e.g., Cloud Run with autoscaling), each instance maintains its own digest and emits its own summary.
|
||||
- **Not durable**: If an instance is terminated before the digest interval expires, the aggregated alerts for that instance are lost.
|
||||
|
||||
## Region-outage alerting (✨ Enterprise feature)
|
||||
|
||||
:::info
|
||||
|
||||
@@ -22,6 +22,8 @@ litellm_settings:
|
||||
This ensures that all budget resets happen at midnight in your specified timezone rather than in UTC.
|
||||
If no timezone is specified, UTC will be used by default.
|
||||
|
||||
Any valid [IANA timezone string](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) is supported (powered by Python's `zoneinfo` module). DST transitions are handled automatically.
|
||||
|
||||
Common timezone values:
|
||||
|
||||
- `UTC` - Coordinated Universal Time
|
||||
|
||||
@@ -340,6 +340,7 @@ litellm_settings:
|
||||
qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
|
||||
qdrant_collection_name: test_collection
|
||||
qdrant_quantization_config: binary
|
||||
qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
|
||||
similarity_threshold: 0.8 # similarity threshold for semantic cache
|
||||
```
|
||||
|
||||
|
||||
@@ -73,6 +73,7 @@ litellm_settings:
|
||||
qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
|
||||
qdrant_collection_name: test_collection
|
||||
qdrant_quantization_config: binary
|
||||
qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
|
||||
similarity_threshold: 0.8 # similarity threshold for semantic cache
|
||||
|
||||
# Optional - S3 Cache Settings
|
||||
|
||||
@@ -161,7 +161,7 @@ Use this when you want non-proxy admins to access `/spend` endpoints
|
||||
|
||||
:::info
|
||||
|
||||
Schedule a [meeting with us to get your Enterprise License](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
Schedule a [meeting with us to get your Enterprise License](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -203,7 +203,7 @@ After regenerating the key, the user will receive an email notification with:
|
||||
|
||||
:::info
|
||||
|
||||
Customizing Email Branding is an Enterprise Feature [Get in touch with us for a Free Trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
Customizing Email Branding is an Enterprise Feature [Get in touch with us for a Free Trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
|
||||
# ✨ Enterprise Features
|
||||
:::tip
|
||||
|
||||
To get a license, get in touch with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
To get a license, get in touch with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -139,7 +139,7 @@ curl -i http://localhost:4000/v1/chat/completions \
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -409,7 +409,7 @@ curl -i -X POST http://localhost:4000/v1/chat/completions \
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -59,7 +59,7 @@ curl -i http://localhost:4000/v1/chat/completions \
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -6,6 +6,108 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
Use [Noma Security](https://noma.security/) to protect your LLM applications with comprehensive AI content moderation and safety guardrails.
|
||||
|
||||
:::warning Deprecated: `guardrail: noma` (Legacy)
|
||||
`guardrail: noma` is deprecated and users should migrate to `guardrail: noma_v2`.
|
||||
The legacy `guardrail: noma` API will no longer be supported after March 31, 2026.
|
||||
|
||||
For easier migration of existing integrations, keep `guardrail: noma` and set `use_v2: true`.
|
||||
With `use_v2: true`, requests route to `noma_v2`; `monitor_mode` and `block_failures` still apply, while `anonymize_input` is ignored.
|
||||
:::
|
||||
|
||||
## Noma v2 guardrails (Recommended)
|
||||
|
||||
### Quick Start
|
||||
|
||||
```yaml showLineNumbers title="litellm config.yaml"
|
||||
guardrails:
|
||||
- guardrail_name: "noma-v2-guard"
|
||||
litellm_params:
|
||||
guardrail: noma_v2
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/NOMA_API_KEY
|
||||
api_base: os.environ/NOMA_API_BASE
|
||||
```
|
||||
|
||||
If you want to migrate gradually without changing guardrail names yet:
|
||||
|
||||
```yaml showLineNumbers title="litellm config.yaml"
|
||||
guardrails:
|
||||
- guardrail_name: "noma-guard"
|
||||
litellm_params:
|
||||
guardrail: noma
|
||||
use_v2: true
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/NOMA_API_KEY
|
||||
api_base: os.environ/NOMA_API_BASE
|
||||
```
|
||||
|
||||
### Supported Params
|
||||
|
||||
- **`guardrail`**: Use `noma_v2` (recommended), or `noma` with `use_v2: true` for migration
|
||||
- **`mode`**: `pre_call`, `post_call`, `during_call`, `pre_mcp_call`, `during_mcp_call`
|
||||
- **`api_key`**: Noma API key (required for Noma SaaS, optional for self-managed deployments)
|
||||
- **`api_base`**: Noma API base URL (defaults to `https://api.noma.security/`)
|
||||
- **`application_id`**: Application identifier. If omitted, v2 checks dynamic `extra_body.application_id`, then configured/env `application_id`; otherwise it is omitted.
|
||||
- **`monitor_mode`**: If `true`, runs in monitor-only mode without blocking (defaults to `false`)
|
||||
- **`block_failures`**: If `true`, fail-closed on guardrail technical failures (defaults to `true`)
|
||||
- **`use_v2`**: Migration toggle when `guardrail: noma` is used
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```shell
|
||||
export NOMA_API_KEY="your-api-key-here"
|
||||
export NOMA_API_BASE="https://api.noma.security/" # Optional
|
||||
export NOMA_APPLICATION_ID="my-app" # Optional
|
||||
export NOMA_MONITOR_MODE="false" # Optional
|
||||
export NOMA_BLOCK_FAILURES="true" # Optional
|
||||
```
|
||||
|
||||
### Multiple Guardrails
|
||||
|
||||
Apply different v2 configurations for input and output:
|
||||
|
||||
```yaml showLineNumbers title="litellm config.yaml"
|
||||
guardrails:
|
||||
- guardrail_name: "noma-v2-input"
|
||||
litellm_params:
|
||||
guardrail: noma_v2
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/NOMA_API_KEY
|
||||
|
||||
- guardrail_name: "noma-v2-output"
|
||||
litellm_params:
|
||||
guardrail: noma_v2
|
||||
mode: "post_call"
|
||||
api_key: os.environ/NOMA_API_KEY
|
||||
```
|
||||
|
||||
### Pass Additional Parameters
|
||||
|
||||
This is supported in v2 via `extra_body`.
|
||||
Currently, `noma_v2` consumes dynamic `application_id`.
|
||||
|
||||
```shell showLineNumbers title="Curl Request"
|
||||
curl 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "gpt-4o-mini",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Hello, how are you?"
|
||||
}
|
||||
],
|
||||
"guardrails": {
|
||||
"noma-v2-guard": {
|
||||
"extra_body": {
|
||||
"application_id": "my-specific-app-id"
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
## Noma guardrails (Legacy)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Define Guardrails on your LiteLLM config.yaml
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
:::info
|
||||
|
||||
You need a LiteLLM License to unlock this feature. [Grab time](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat), to get one today!
|
||||
You need a LiteLLM License to unlock this feature. [Grab time](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions), to get one today!
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -1109,7 +1109,7 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
@@ -1194,7 +1194,7 @@ Log LLM Logs/SpendLogs to [Google Cloud Storage PubSub Topic](https://cloud.goog
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
@@ -1497,7 +1497,7 @@ Log LLM Logs to [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azur
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ LiteLLM tracks changes to the following entities and actions:
|
||||
|
||||
:::tip
|
||||
|
||||
Requires Enterprise License, Get in touch with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
Requires Enterprise License, Get in touch with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ Use this if you want to use an Oauth2.0 token to make `/chat`, `/embeddings` req
|
||||
|
||||
:::info
|
||||
|
||||
This is an Enterprise Feature - [get in touch with us if you want a free trial to test if this feature meets your needs]((https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat))
|
||||
This is an Enterprise Feature - [get in touch with us if you want a free trial to test if this feature meets your needs]((https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions))
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -47,7 +47,7 @@ export LITELLM_LOG="ERROR"
|
||||
|
||||
:::info
|
||||
|
||||
Need Help or want dedicated support ? Talk to a founder [here]: (https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
Need Help or want dedicated support ? Talk to a founder [here]: (https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
:::info
|
||||
|
||||
Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat).
|
||||
Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions).
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -215,7 +215,7 @@ LiteLLM Proxy supports team-based tag routing, allowing you to associate specifi
|
||||
|
||||
:::info
|
||||
|
||||
This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ Team 3 -> Disabled Logging (for GDPR compliance)
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
@@ -248,7 +248,7 @@ Use the `/key/generate` or `/key/update` endpoints to add logging callbacks to a
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
This is an Enterprise feature.
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Use JWT's to auth admins / users / projects into the proxy.
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
|
||||
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -82,7 +82,7 @@ Benchmark Results for 'When will BerriAI IPO?':
|
||||
+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
|
||||
```
|
||||
## Support
|
||||
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
|
||||
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
@@ -0,0 +1,406 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Google GenAI SDK with LiteLLM
|
||||
|
||||
Use Google's official GenAI SDK (JavaScript/TypeScript and Python) with any LLM provider through LiteLLM Proxy.
|
||||
|
||||
The Google GenAI SDK (`@google/genai` for JS, `google-genai` for Python) provides a native interface for calling Gemini models. By pointing it to LiteLLM, you can use the same SDK with OpenAI, Anthropic, Bedrock, Azure, Vertex AI, or any other provider — while keeping the native Gemini request/response format.
|
||||
|
||||
## Why Use LiteLLM with Google GenAI SDK?
|
||||
|
||||
**Developer Benefits:**
|
||||
- **Universal Model Access**: Use any LiteLLM-supported model (Anthropic, OpenAI, Vertex AI, Bedrock, etc.) through the Google GenAI SDK interface
|
||||
- **Higher Rate Limits & Reliability**: Load balance across multiple models and providers to avoid hitting individual provider limits, with fallbacks to ensure you get responses even if one provider fails
|
||||
|
||||
**Proxy Admin Benefits:**
|
||||
- **Centralized Management**: Control access to all models through a single LiteLLM proxy instance without giving developers API keys to each provider
|
||||
- **Budget Controls**: Set spending limits and track costs across all SDK usage
|
||||
- **Logging & Observability**: Track all requests with cost tracking, logging, and analytics
|
||||
|
||||
| Feature | Supported | Notes |
|
||||
|---------|-----------|-------|
|
||||
| Cost Tracking | ✅ | All models on `/generateContent` endpoint |
|
||||
| Logging | ✅ | Works across all integrations |
|
||||
| Streaming | ✅ | `streamGenerateContent` supported |
|
||||
| Virtual Keys | ✅ | Use LiteLLM keys instead of Google keys |
|
||||
| Load Balancing | ✅ | Via native router endpoints |
|
||||
| Fallbacks | ✅ | Via native router endpoints |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install the SDK
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="js" label="JavaScript/TypeScript">
|
||||
|
||||
```bash
|
||||
npm install @google/genai
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python">
|
||||
|
||||
```bash
|
||||
pip install google-genai
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### 2. Start LiteLLM Proxy
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gemini-2.5-flash
|
||||
litellm_params:
|
||||
model: gemini/gemini-2.5-flash
|
||||
api_key: os.environ/GEMINI_API_KEY
|
||||
```
|
||||
|
||||
```bash
|
||||
litellm --config config.yaml
|
||||
```
|
||||
|
||||
### 3. Call the SDK through LiteLLM
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="js" label="JavaScript/TypeScript">
|
||||
|
||||
```javascript title="index.js" showLineNumbers
|
||||
const { GoogleGenAI } = require("@google/genai");
|
||||
|
||||
const ai = new GoogleGenAI({
|
||||
apiKey: "sk-1234", // LiteLLM virtual key (not a Google key)
|
||||
httpOptions: {
|
||||
baseUrl: "http://localhost:4000/gemini", // LiteLLM proxy URL
|
||||
},
|
||||
});
|
||||
|
||||
async function main() {
|
||||
const response = await ai.models.generateContent({
|
||||
model: "gemini-2.5-flash",
|
||||
contents: "Explain how AI works",
|
||||
});
|
||||
console.log(response.text);
|
||||
}
|
||||
|
||||
main();
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python">
|
||||
|
||||
```python title="main.py" showLineNumbers
|
||||
from google import genai
|
||||
|
||||
client = genai.Client(
|
||||
api_key="sk-1234", # LiteLLM virtual key (not a Google key)
|
||||
http_options={"base_url": "http://localhost:4000/gemini"}, # LiteLLM proxy URL
|
||||
)
|
||||
|
||||
response = client.models.generate_content(
|
||||
model="gemini-2.5-flash",
|
||||
contents="Explain how AI works",
|
||||
)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="curl" label="curl">
|
||||
|
||||
```bash
|
||||
curl "http://localhost:4000/gemini/v1beta/models/gemini-2.5-flash:generateContent?key=sk-1234" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-X POST \
|
||||
-d '{
|
||||
"contents": [{
|
||||
"parts": [{"text": "Explain how AI works"}]
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Streaming
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="js" label="JavaScript/TypeScript">
|
||||
|
||||
```javascript title="streaming.js" showLineNumbers
|
||||
const { GoogleGenAI } = require("@google/genai");
|
||||
|
||||
const ai = new GoogleGenAI({
|
||||
apiKey: "sk-1234",
|
||||
httpOptions: {
|
||||
baseUrl: "http://localhost:4000/gemini",
|
||||
},
|
||||
});
|
||||
|
||||
async function main() {
|
||||
const response = await ai.models.generateContentStream({
|
||||
model: "gemini-2.5-flash",
|
||||
contents: "Write a short poem about the ocean",
|
||||
});
|
||||
|
||||
for await (const chunk of response) {
|
||||
process.stdout.write(chunk.text);
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python">
|
||||
|
||||
```python title="streaming.py" showLineNumbers
|
||||
from google import genai
|
||||
|
||||
client = genai.Client(
|
||||
api_key="sk-1234",
|
||||
http_options={"base_url": "http://localhost:4000/gemini"},
|
||||
)
|
||||
|
||||
response = client.models.generate_content_stream(
|
||||
model="gemini-2.5-flash",
|
||||
contents="Write a short poem about the ocean",
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
print(chunk.text, end="")
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Multi-turn Chat
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="js" label="JavaScript/TypeScript">
|
||||
|
||||
```javascript title="chat.js" showLineNumbers
|
||||
const { GoogleGenAI } = require("@google/genai");
|
||||
|
||||
const ai = new GoogleGenAI({
|
||||
apiKey: "sk-1234",
|
||||
httpOptions: {
|
||||
baseUrl: "http://localhost:4000/gemini",
|
||||
},
|
||||
});
|
||||
|
||||
async function main() {
|
||||
const chat = ai.chats.create({
|
||||
model: "gemini-2.5-flash",
|
||||
});
|
||||
|
||||
const response1 = await chat.sendMessage({ message: "I have 2 dogs and 3 cats." });
|
||||
console.log(response1.text);
|
||||
|
||||
const response2 = await chat.sendMessage({ message: "How many pets is that in total?" });
|
||||
console.log(response2.text);
|
||||
}
|
||||
|
||||
main();
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python">
|
||||
|
||||
```python title="chat.py" showLineNumbers
|
||||
from google import genai
|
||||
|
||||
client = genai.Client(
|
||||
api_key="sk-1234",
|
||||
http_options={"base_url": "http://localhost:4000/gemini"},
|
||||
)
|
||||
|
||||
chat = client.chats.create(model="gemini-2.5-flash")
|
||||
|
||||
response1 = chat.send_message("I have 2 dogs and 3 cats.")
|
||||
print(response1.text)
|
||||
|
||||
response2 = chat.send_message("How many pets is that in total?")
|
||||
print(response2.text)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Advanced: Use Any Model with the GenAI SDK
|
||||
|
||||
By default, the GenAI SDK talks to Gemini models. But with LiteLLM's router, you can route GenAI SDK requests to **any provider** — Anthropic, OpenAI, Bedrock, etc.
|
||||
|
||||
This works by using `model_group_alias` to map Gemini model names to your desired provider models. LiteLLM handles the format translation internally.
|
||||
|
||||
:::info
|
||||
|
||||
For this to work, point the SDK `baseUrl` to `http://localhost:4000` (without `/gemini`). This routes requests through LiteLLM's native Google endpoints, which go through the router and support model aliasing.
|
||||
|
||||
:::
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="anthropic" label="Anthropic">
|
||||
|
||||
Route `gemini-2.5-flash` requests to Claude Sonnet:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-20250514
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
router_settings:
|
||||
model_group_alias: {"gemini-2.5-flash": "claude-sonnet"}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="openai" label="OpenAI">
|
||||
|
||||
Route `gemini-2.5-flash` requests to GPT-4o:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o-model
|
||||
litellm_params:
|
||||
model: gpt-4o
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
router_settings:
|
||||
model_group_alias: {"gemini-2.5-flash": "gpt-4o-model"}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="bedrock" label="Bedrock">
|
||||
|
||||
Route `gemini-2.5-flash` requests to Claude on Bedrock:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: bedrock-claude
|
||||
litellm_params:
|
||||
model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
aws_region_name: us-east-1
|
||||
|
||||
router_settings:
|
||||
model_group_alias: {"gemini-2.5-flash": "bedrock-claude"}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="multi" label="Multi-Provider Load Balancing">
|
||||
|
||||
Load balance across Anthropic and OpenAI:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: my-model
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-20250514
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
- model_name: my-model
|
||||
litellm_params:
|
||||
model: gpt-4o
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
router_settings:
|
||||
model_group_alias: {"gemini-2.5-flash": "my-model"}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
Then use the SDK with `baseUrl` pointing to LiteLLM (without `/gemini`):
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="js" label="JavaScript/TypeScript">
|
||||
|
||||
```javascript title="any_model.js" showLineNumbers
|
||||
const { GoogleGenAI } = require("@google/genai");
|
||||
|
||||
const ai = new GoogleGenAI({
|
||||
apiKey: "sk-1234",
|
||||
httpOptions: {
|
||||
baseUrl: "http://localhost:4000", // No /gemini — goes through the router
|
||||
},
|
||||
});
|
||||
|
||||
async function main() {
|
||||
// This calls Claude/GPT-4o/Bedrock under the hood via model_group_alias
|
||||
const response = await ai.models.generateContent({
|
||||
model: "gemini-2.5-flash",
|
||||
contents: "Hello from any model!",
|
||||
});
|
||||
console.log(response.text);
|
||||
}
|
||||
|
||||
main();
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python">
|
||||
|
||||
```python title="any_model.py" showLineNumbers
|
||||
from google import genai
|
||||
|
||||
client = genai.Client(
|
||||
api_key="sk-1234",
|
||||
http_options={"base_url": "http://localhost:4000"}, # No /gemini
|
||||
)
|
||||
|
||||
# This calls Claude/GPT-4o/Bedrock under the hood via model_group_alias
|
||||
response = client.models.generate_content(
|
||||
model="gemini-2.5-flash",
|
||||
contents="Hello from any model!",
|
||||
)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Pass-through vs Native Router Endpoints
|
||||
|
||||
LiteLLM offers two ways to handle GenAI SDK requests:
|
||||
|
||||
| | Pass-through (`/gemini`) | Native Router (`/`) |
|
||||
|---|---|---|
|
||||
| **baseUrl** | `http://localhost:4000/gemini` | `http://localhost:4000` |
|
||||
| **Models** | Gemini only | Any provider via `model_group_alias` |
|
||||
| **Translation** | None — proxies directly to Google | Translates internally |
|
||||
| **Cost Tracking** | ✅ | ✅ |
|
||||
| **Virtual Keys** | ✅ | ✅ |
|
||||
| **Load Balancing** | ❌ | ✅ |
|
||||
| **Fallbacks** | ❌ | ✅ |
|
||||
| **Best for** | Simple Gemini proxy | Multi-provider routing |
|
||||
|
||||
## Environment Variable Configuration
|
||||
|
||||
You can also configure the SDK via environment variables instead of code:
|
||||
|
||||
```bash
|
||||
# For JavaScript SDK (@google/genai)
|
||||
export GOOGLE_GEMINI_BASE_URL="http://localhost:4000/gemini"
|
||||
export GEMINI_API_KEY="sk-1234"
|
||||
|
||||
# For Python SDK (google-genai)
|
||||
# Note: The Python SDK does not support a base URL env var.
|
||||
# Configure it in code with http_options={"base_url": "..."} instead.
|
||||
export GEMINI_API_KEY="sk-1234"
|
||||
```
|
||||
|
||||
This is especially useful for tools built on top of the GenAI SDK (like [Gemini CLI](./litellm_gemini_cli.md)).
|
||||
|
||||
## Related Resources
|
||||
|
||||
- [Gemini CLI with LiteLLM](./litellm_gemini_cli.md)
|
||||
- [Google AI Studio Pass-Through](../pass_through/google_ai_studio)
|
||||
- [Google ADK with LiteLLM](./google_adk.md)
|
||||
- [LiteLLM Proxy Quick Start](../proxy/quick_start)
|
||||
- [`@google/genai` npm package](https://www.npmjs.com/package/@google/genai)
|
||||
- [`google-genai` PyPI package](https://pypi.org/project/google-genai/)
|
||||
@@ -0,0 +1,373 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# OpenAI Agents SDK with LiteLLM
|
||||
|
||||
Use OpenAI's Agents SDK with any LLM provider through LiteLLM Proxy.
|
||||
|
||||
This tutorial shows you how to build AI agents using the OpenAI Agents SDK with support for multiple LLM providers through LiteLLM.
|
||||
|
||||
## Overview
|
||||
|
||||
The OpenAI Agents SDK provides a high-level interface for building AI agents. By integrating with LiteLLM, you can:
|
||||
|
||||
- Use multiple LLM providers (Bedrock, Azure, Vertex AI, etc.) with the same agent code
|
||||
- Switch easily between models from different providers
|
||||
- Connect to a LiteLLM proxy for centralized model management
|
||||
|
||||
:::tip Built-in LiteLLM Extension
|
||||
|
||||
The OpenAI Agents SDK includes an official LiteLLM extension (`LitellmModel`) that works without a proxy. If you don't need centralized proxy features (cost tracking, rate limiting, load balancing), you can use it directly:
|
||||
|
||||
```python
|
||||
from agents import Agent, Runner
|
||||
from agents.extensions.models.litellm_model import LitellmModel
|
||||
|
||||
|
||||
agent = Agent(
|
||||
name="Assistant",
|
||||
instructions="You are a helpful assistant.",
|
||||
model=LitellmModel(model="anthropic/claude-sonnet-4-20250514"),
|
||||
)
|
||||
|
||||
result = Runner.run_sync(agent, "Hello!")
|
||||
print(result.final_output)
|
||||
```
|
||||
|
||||
See the [Docs](https://openai.github.io/openai-agents-python/models/litellm/) for more details. The rest of this tutorial focuses on the **proxy-based approach** for teams that need centralized model management.
|
||||
|
||||
:::
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python environment setup
|
||||
- API keys for your LLM providers
|
||||
- Basic understanding of LLMs and agent concepts
|
||||
|
||||
## Installation
|
||||
|
||||
```bash showLineNumbers title="Install dependencies"
|
||||
pip install openai-agents litellm
|
||||
```
|
||||
|
||||
## 1. Start LiteLLM Proxy
|
||||
|
||||
Configure and start the LiteLLM proxy with the models you want to use:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: bedrock-claude-sonnet-4
|
||||
litellm_params:
|
||||
model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
|
||||
aws_region_name: "us-east-1"
|
||||
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: "openai/gpt-4o"
|
||||
|
||||
- model_name: claude-sonnet-4
|
||||
litellm_params:
|
||||
model: "anthropic/claude-sonnet-4-20250514"
|
||||
|
||||
- model_name: bedrock-claude-haiku
|
||||
litellm_params:
|
||||
model: "bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0"
|
||||
aws_region_name: "us-east-1"
|
||||
|
||||
- model_name: bedrock-nova-premier
|
||||
litellm_params:
|
||||
model: "bedrock/amazon.nova-premier-v1:0"
|
||||
aws_region_name: "us-east-1"
|
||||
```
|
||||
|
||||
```bash
|
||||
litellm --config config.yaml
|
||||
```
|
||||
|
||||
Required environment variables:
|
||||
|
||||
| Variable | Value | Description |
|
||||
|----------|-------|-------------|
|
||||
| `LITELLM_BASE_URL` | `http://localhost:4000` | LiteLLM proxy URL |
|
||||
| `LITELLM_API_KEY` | `sk-1234` | Your LiteLLM API key (not your provider's key) |
|
||||
|
||||
## 2. Setting Up Environment
|
||||
|
||||
Import the necessary libraries and configure your LiteLLM proxy connection:
|
||||
|
||||
```python showLineNumbers title="Setup environment"
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from agents import (
|
||||
Agent,
|
||||
Model,
|
||||
ModelProvider,
|
||||
OpenAIChatCompletionsModel,
|
||||
RunConfig,
|
||||
Runner,
|
||||
function_tool,
|
||||
set_tracing_disabled,
|
||||
)
|
||||
|
||||
# Point to LiteLLM proxy
|
||||
BASE_URL = os.getenv("LITELLM_BASE_URL") or "http://localhost:4000"
|
||||
API_KEY = os.getenv("LITELLM_API_KEY") or "sk-1234"
|
||||
|
||||
# Define model constants for cleaner code
|
||||
MODEL_BEDROCK_SONNET = "bedrock-claude-sonnet-4"
|
||||
MODEL_BEDROCK_HAIKU = "bedrock-claude-haiku"
|
||||
MODEL_GPT_4O = "gpt-4o"
|
||||
|
||||
# Create the OpenAI client pointed at LiteLLM
|
||||
client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
|
||||
|
||||
# Disable tracing since we're not using OpenAI's platform directly
|
||||
set_tracing_disabled(disabled=True)
|
||||
```
|
||||
|
||||
## 3. Create a Custom Model Provider
|
||||
|
||||
The Agents SDK uses a `ModelProvider` to resolve model names. Create a custom provider that routes all requests through LiteLLM:
|
||||
|
||||
```python showLineNumbers title="Custom LiteLLM model provider"
|
||||
class LiteLLMModelProvider(ModelProvider):
|
||||
def get_model(self, model_name: str | None) -> Model:
|
||||
return OpenAIChatCompletionsModel(
|
||||
model=model_name or MODEL_BEDROCK_SONNET,
|
||||
openai_client=client,
|
||||
)
|
||||
|
||||
|
||||
LITELLM_MODEL_PROVIDER = LiteLLMModelProvider()
|
||||
```
|
||||
|
||||
## 4. Define a Simple Tool
|
||||
|
||||
Create a tool that your agent can use:
|
||||
|
||||
```python showLineNumbers title="Weather tool implementation"
|
||||
@function_tool
|
||||
def get_weather(city: str) -> str:
|
||||
"""Retrieves the current weather report for a specified city.
|
||||
|
||||
Args:
|
||||
city: The name of the city (e.g., "New York", "London", "Tokyo").
|
||||
|
||||
Returns:
|
||||
A string containing the weather information for the city.
|
||||
"""
|
||||
print(f"[debug] getting weather for {city}")
|
||||
|
||||
mock_weather_db = {
|
||||
"new york": "The weather in New York is sunny with a temperature of 25°C.",
|
||||
"london": "It's cloudy in London with a temperature of 15°C.",
|
||||
"tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
|
||||
}
|
||||
|
||||
city_normalized = city.lower()
|
||||
|
||||
if city_normalized in mock_weather_db:
|
||||
return mock_weather_db[city_normalized]
|
||||
else:
|
||||
return f"Sorry, I don't have weather information for '{city}'."
|
||||
```
|
||||
|
||||
## 5. Using Different Models with Agents
|
||||
|
||||
### 5.1 Using Bedrock Models
|
||||
|
||||
```python showLineNumbers title="Bedrock model via LiteLLM proxy"
|
||||
async def test_bedrock_agent():
|
||||
print("\n--- Testing Bedrock Claude Agent ---")
|
||||
|
||||
agent = Agent(
|
||||
name="weather_agent_bedrock",
|
||||
instructions="You are a helpful weather assistant powered by Claude. "
|
||||
"Use the 'get_weather' tool for city weather requests. "
|
||||
"Present information clearly.",
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
result = await Runner.run(
|
||||
agent,
|
||||
"What's the weather in Tokyo?",
|
||||
run_config=RunConfig(
|
||||
model_provider=LITELLM_MODEL_PROVIDER,
|
||||
model="bedrock-claude-sonnet-4", # Uses the model name from your LiteLLM config
|
||||
),
|
||||
)
|
||||
print(f"<<< Agent Response: {result.final_output}")
|
||||
|
||||
|
||||
asyncio.run(test_bedrock_agent())
|
||||
```
|
||||
|
||||
### 5.2 Using OpenAI Models
|
||||
|
||||
```python showLineNumbers title="OpenAI model via LiteLLM proxy"
|
||||
async def test_openai_agent():
|
||||
print("\n--- Testing OpenAI GPT Agent ---")
|
||||
|
||||
agent = Agent(
|
||||
name="weather_agent_gpt",
|
||||
instructions="You are a helpful weather assistant powered by GPT-4o. "
|
||||
"Use the 'get_weather' tool for city weather requests. "
|
||||
"Present information clearly.",
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
result = await Runner.run(
|
||||
agent,
|
||||
"What's the weather in London?",
|
||||
run_config=RunConfig(
|
||||
model_provider=LITELLM_MODEL_PROVIDER,
|
||||
model="gpt-4o", # Uses the model name from your LiteLLM config
|
||||
),
|
||||
)
|
||||
print(f"<<< Agent Response: {result.final_output}")
|
||||
|
||||
|
||||
asyncio.run(test_openai_agent())
|
||||
```
|
||||
|
||||
### 5.3 Using Anthropic Models
|
||||
|
||||
```python showLineNumbers title="Anthropic model via LiteLLM proxy"
|
||||
async def test_anthropic_agent():
|
||||
print("\n--- Testing Anthropic Claude Agent ---")
|
||||
|
||||
agent = Agent(
|
||||
name="weather_agent_claude",
|
||||
instructions="You are a helpful weather assistant powered by Claude. "
|
||||
"Use the 'get_weather' tool for city weather requests. "
|
||||
"Present information clearly.",
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
result = await Runner.run(
|
||||
agent,
|
||||
"What's the weather in New York?",
|
||||
run_config=RunConfig(
|
||||
model_provider=LITELLM_MODEL_PROVIDER,
|
||||
model="claude-sonnet-4", # Uses the model name from your LiteLLM config
|
||||
),
|
||||
)
|
||||
print(f"<<< Agent Response: {result.final_output}")
|
||||
|
||||
|
||||
asyncio.run(test_anthropic_agent())
|
||||
```
|
||||
|
||||
## 6. Complete Working Example
|
||||
|
||||
Here's a full end-to-end script you can copy and run:
|
||||
|
||||
```python showLineNumbers title="complete_agent.py"
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from agents import (
|
||||
Agent,
|
||||
Model,
|
||||
ModelProvider,
|
||||
OpenAIChatCompletionsModel,
|
||||
RunConfig,
|
||||
Runner,
|
||||
function_tool,
|
||||
set_tracing_disabled,
|
||||
)
|
||||
|
||||
# Point to LiteLLM proxy
|
||||
BASE_URL = os.getenv("LITELLM_BASE_URL") or "http://localhost:4000"
|
||||
API_KEY = os.getenv("LITELLM_API_KEY") or "sk-1234"
|
||||
MODEL_NAME = os.getenv("MODEL_NAME") or "bedrock-claude-sonnet-4"
|
||||
|
||||
client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
|
||||
set_tracing_disabled(disabled=True)
|
||||
|
||||
|
||||
class LiteLLMModelProvider(ModelProvider):
|
||||
def get_model(self, model_name: str | None) -> Model:
|
||||
return OpenAIChatCompletionsModel(
|
||||
model=model_name or MODEL_NAME,
|
||||
openai_client=client,
|
||||
)
|
||||
|
||||
|
||||
LITELLM_MODEL_PROVIDER = LiteLLMModelProvider()
|
||||
|
||||
|
||||
@function_tool
|
||||
def get_weather(city: str) -> str:
|
||||
"""Retrieves the current weather report for a specified city."""
|
||||
print(f"[debug] getting weather for {city}")
|
||||
|
||||
mock_weather_db = {
|
||||
"new york": "The weather in New York is sunny with a temperature of 25°C.",
|
||||
"london": "It's cloudy in London with a temperature of 15°C.",
|
||||
"tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
|
||||
}
|
||||
|
||||
city_normalized = city.lower()
|
||||
if city_normalized in mock_weather_db:
|
||||
return mock_weather_db[city_normalized]
|
||||
else:
|
||||
return f"Sorry, I don't have weather information for '{city}'."
|
||||
|
||||
|
||||
async def main():
|
||||
agent = Agent(
|
||||
name="Assistant",
|
||||
instructions="You are a helpful weather assistant. "
|
||||
"Use the 'get_weather' tool for city weather requests. "
|
||||
"Present information clearly and concisely.",
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
# Run with the default model (bedrock-claude-sonnet-4)
|
||||
result = await Runner.run(
|
||||
agent,
|
||||
"What's the weather in Tokyo?",
|
||||
run_config=RunConfig(model_provider=LITELLM_MODEL_PROVIDER),
|
||||
)
|
||||
print(result.final_output)
|
||||
|
||||
# Switch to a different model by passing model in RunConfig
|
||||
result = await Runner.run(
|
||||
agent,
|
||||
"What's the weather in London?",
|
||||
run_config=RunConfig(
|
||||
model_provider=LITELLM_MODEL_PROVIDER,
|
||||
model="gpt-4o",
|
||||
),
|
||||
)
|
||||
print(result.final_output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Why Use LiteLLM with Agents SDK?
|
||||
|
||||
| Feature | Benefit |
|
||||
|---------|---------|
|
||||
| **Multi-Provider** | Use the same agent code with OpenAI, Bedrock, Azure, Vertex AI, etc. |
|
||||
| **Cost Tracking** | Track spending across all agent conversations |
|
||||
| **Rate Limiting** | Set budgets and limits on agent usage |
|
||||
| **Load Balancing** | Distribute requests across multiple API keys or regions |
|
||||
| **Fallbacks** | Automatically retry with different models if one fails |
|
||||
|
||||
## Related Resources
|
||||
|
||||
- [OpenAI Agents SDK Documentation](https://openai.github.io/openai-agents-python/)
|
||||
- [LiteLLM Proxy Quick Start](../proxy/quick_start)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 538 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 625 KiB |
@@ -96,6 +96,39 @@ The Compliance Playground lets you test any guardrail against our pre-built eval
|
||||
|
||||
---
|
||||
|
||||
## Performance & Reliability — Up to 13% Lower Latency
|
||||
|
||||
<Image img={require('../img/release_notes/v1_81_14_perf.png')} />
|
||||
|
||||
This release cuts latency across all percentiles through 20+ micro-optimizations across logging, cost calculation, routing, and connection management. See [benchmarking](../../docs/benchmarks) for more info about how to benchmark yourself.
|
||||
|
||||
- **Mean latency:** 78.4 ms → **70.3 ms** (−10.3%)
|
||||
- **p50 latency:** 64.8 ms → **57.3 ms** (−11.7%)
|
||||
- **p99 latency:** 288.9 ms → **250.0 ms** (−13.4%)
|
||||
|
||||
**Streaming Connection Pool Fix**
|
||||
|
||||
Fixed a 3-fold connection leak that caused TCP connection starvation under streaming workloads: the aiohttp transport wasn't closing connections, no `finally` blocks were calling close on disconnect, and a Uvicorn bug prevented disconnect signaling. [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Client Disconnects] --> B[Stream Abandoned]
|
||||
B --> C{Connection cleaned up?}
|
||||
C -->|Before| D["❌ No — connection leaked"]
|
||||
C -->|After| E["✅ Yes — connection returned to pool"]
|
||||
```
|
||||
|
||||
**Redis Connection Pool Reliability**
|
||||
|
||||
Fixed 4 separate connection pool bugs to make how we use Redis more reliable. The most important change was on pools being leaked on cache expiry and the other fixes are detailed here in [PR #21717](https://github.com/BerriAI/litellm/pull/21717).
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Cache Entry Expires] --> B{Pool cleanup?}
|
||||
B -->|Before| C["❌ New untracked pool created — leaked"]
|
||||
B -->|After| D["✅ Pool closed on eviction"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## New Providers and Endpoints
|
||||
@@ -438,6 +471,7 @@ The Compliance Playground lets you test any guardrail against our pre-built eval
|
||||
|
||||
- Fix Redis connection pool reliability — prevent connection exhaustion under load - [PR #21717](https://github.com/BerriAI/litellm/pull/21717)
|
||||
- Fix Prisma connection self-heal for auth and runtime reconnection (reverted, will be re-introduced with fixes) - [PR #21706](https://github.com/BerriAI/litellm/pull/21706)
|
||||
- Close streaming connections to prevent connection pool exhaustion - [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
|
||||
- Make `PodLockManager.release_lock` atomic compare-and-delete - [PR #21226](https://github.com/BerriAI/litellm/pull/21226)
|
||||
|
||||
---
|
||||
|
||||
@@ -166,6 +166,7 @@ const sidebars = {
|
||||
"tutorials/cursor_integration",
|
||||
"tutorials/github_copilot_integration",
|
||||
"tutorials/litellm_gemini_cli",
|
||||
"tutorials/google_genai_sdk",
|
||||
"tutorials/litellm_qwen_code_cli",
|
||||
"tutorials/openai_codex"
|
||||
]
|
||||
@@ -180,6 +181,7 @@ const sidebars = {
|
||||
slug: "/agent_sdks"
|
||||
},
|
||||
items: [
|
||||
"tutorials/openai_agents_sdk",
|
||||
"tutorials/claude_agent_sdk",
|
||||
"tutorials/copilotkit_sdk",
|
||||
"tutorials/google_adk",
|
||||
@@ -419,6 +421,7 @@ const sidebars = {
|
||||
"proxy/dynamic_rate_limit",
|
||||
"proxy/rate_limit_tiers",
|
||||
"proxy/temporary_budget_increase",
|
||||
"proxy/budget_reset_and_tz",
|
||||
],
|
||||
},
|
||||
"proxy/caching",
|
||||
|
||||
@@ -7,7 +7,7 @@ With regard to the BerriAI Software:
|
||||
This software and associated documentation files (the "Software") may only be
|
||||
used in production, if you (and any entity that you represent) have agreed to,
|
||||
and are in compliance with, the BerriAI Subscription Terms of Service, available
|
||||
via [call](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) or email (info@berri.ai) (the "Enterprise Terms"), or other
|
||||
via [call](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) or email (info@berri.ai) (the "Enterprise Terms"), or other
|
||||
agreement governing the use of the Software, as agreed by you and BerriAI,
|
||||
and otherwise have a valid BerriAI Enterprise license for the
|
||||
correct number of user seats. Subject to the foregoing sentence, you are free to
|
||||
|
||||
@@ -4,6 +4,6 @@ Code in this folder is licensed under a commercial license. Please review the [L
|
||||
|
||||
**These features are covered under the LiteLLM Enterprise contract**
|
||||
|
||||
👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat?month=2024-02)
|
||||
👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions?month=2024-02)
|
||||
|
||||
See all Enterprise Features here 👉 [Docs](https://docs.litellm.ai/docs/proxy/enterprise)
|
||||
|
||||
@@ -16,6 +16,10 @@ from litellm_enterprise.types.enterprise_callbacks.send_emails import (
|
||||
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm.caching.caching import DualCache
|
||||
from litellm.constants import (
|
||||
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
|
||||
EMAIL_BUDGET_ALERT_TTL,
|
||||
)
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.integrations.email_templates.email_footer import EMAIL_FOOTER
|
||||
from litellm.integrations.email_templates.key_created_email import (
|
||||
@@ -24,14 +28,14 @@ from litellm.integrations.email_templates.key_created_email import (
|
||||
from litellm.integrations.email_templates.key_rotated_email import (
|
||||
KEY_ROTATED_EMAIL_TEMPLATE,
|
||||
)
|
||||
from litellm.integrations.email_templates.user_invitation_email import (
|
||||
USER_INVITATION_EMAIL_TEMPLATE,
|
||||
)
|
||||
from litellm.integrations.email_templates.templates import (
|
||||
MAX_BUDGET_ALERT_EMAIL_TEMPLATE,
|
||||
SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
|
||||
TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
|
||||
)
|
||||
from litellm.integrations.email_templates.user_invitation_email import (
|
||||
USER_INVITATION_EMAIL_TEMPLATE,
|
||||
)
|
||||
from litellm.proxy._types import (
|
||||
CallInfo,
|
||||
InvitationNew,
|
||||
@@ -41,10 +45,6 @@ from litellm.proxy._types import (
|
||||
)
|
||||
from litellm.secret_managers.main import get_secret_bool
|
||||
from litellm.types.integrations.slack_alerting import LITELLM_LOGO_URL
|
||||
from litellm.constants import (
|
||||
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
|
||||
EMAIL_BUDGET_ALERT_TTL,
|
||||
)
|
||||
|
||||
|
||||
class BaseEmailLogger(CustomLogger):
|
||||
@@ -121,10 +121,16 @@ class BaseEmailLogger(CustomLogger):
|
||||
)
|
||||
|
||||
# Check if API key should be included in email
|
||||
include_api_key = get_secret_bool(secret_name="EMAIL_INCLUDE_API_KEY", default_value=True)
|
||||
include_api_key = get_secret_bool(
|
||||
secret_name="EMAIL_INCLUDE_API_KEY", default_value=True
|
||||
)
|
||||
if include_api_key is None:
|
||||
include_api_key = True # Default to True if not set
|
||||
key_token_display = send_key_created_email_event.virtual_key if include_api_key else "[Key hidden for security - retrieve from dashboard]"
|
||||
key_token_display = (
|
||||
send_key_created_email_event.virtual_key
|
||||
if include_api_key
|
||||
else "[Key hidden for security - retrieve from dashboard]"
|
||||
)
|
||||
|
||||
email_html_content = KEY_CREATED_EMAIL_TEMPLATE.format(
|
||||
email_logo_url=email_params.logo_url,
|
||||
@@ -162,10 +168,16 @@ class BaseEmailLogger(CustomLogger):
|
||||
)
|
||||
|
||||
# Check if API key should be included in email
|
||||
include_api_key = get_secret_bool(secret_name="EMAIL_INCLUDE_API_KEY", default_value=True)
|
||||
include_api_key = get_secret_bool(
|
||||
secret_name="EMAIL_INCLUDE_API_KEY", default_value=True
|
||||
)
|
||||
if include_api_key is None:
|
||||
include_api_key = True # Default to True if not set
|
||||
key_token_display = send_key_rotated_email_event.virtual_key if include_api_key else "[Key hidden for security - retrieve from dashboard]"
|
||||
key_token_display = (
|
||||
send_key_rotated_email_event.virtual_key
|
||||
if include_api_key
|
||||
else "[Key hidden for security - retrieve from dashboard]"
|
||||
)
|
||||
|
||||
email_html_content = KEY_ROTATED_EMAIL_TEMPLATE.format(
|
||||
email_logo_url=email_params.logo_url,
|
||||
@@ -201,7 +213,9 @@ class BaseEmailLogger(CustomLogger):
|
||||
)
|
||||
|
||||
# Format budget values
|
||||
soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
|
||||
soft_budget_str = (
|
||||
f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
|
||||
)
|
||||
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
|
||||
max_budget_info = ""
|
||||
if event.max_budget is not None:
|
||||
@@ -231,13 +245,13 @@ class BaseEmailLogger(CustomLogger):
|
||||
"""
|
||||
# Collect all recipient emails
|
||||
recipient_emails: List[str] = []
|
||||
|
||||
|
||||
# Add additional alert emails from team metadata.soft_budget_alert_emails
|
||||
if hasattr(event, "alert_emails") and event.alert_emails:
|
||||
for email in event.alert_emails:
|
||||
if email and email not in recipient_emails: # Avoid duplicates
|
||||
recipient_emails.append(email)
|
||||
|
||||
|
||||
# If no recipients found, skip sending
|
||||
if not recipient_emails:
|
||||
verbose_proxy_logger.warning(
|
||||
@@ -268,7 +282,9 @@ class BaseEmailLogger(CustomLogger):
|
||||
)
|
||||
|
||||
# Format budget values
|
||||
soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
|
||||
soft_budget_str = (
|
||||
f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
|
||||
)
|
||||
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
|
||||
max_budget_info = ""
|
||||
if event.max_budget is not None:
|
||||
@@ -286,7 +302,7 @@ class BaseEmailLogger(CustomLogger):
|
||||
base_url=email_params.base_url,
|
||||
email_support_contact=email_params.support_contact,
|
||||
)
|
||||
|
||||
|
||||
# Send email to all recipients
|
||||
await self.send_email(
|
||||
from_email=self.DEFAULT_LITELLM_EMAIL,
|
||||
@@ -313,11 +329,17 @@ class BaseEmailLogger(CustomLogger):
|
||||
|
||||
# Format budget values
|
||||
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
|
||||
max_budget_str = f"${event.max_budget}" if event.max_budget is not None else "N/A"
|
||||
|
||||
max_budget_str = (
|
||||
f"${event.max_budget}" if event.max_budget is not None else "N/A"
|
||||
)
|
||||
|
||||
# Calculate percentage and alert threshold
|
||||
percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
|
||||
alert_threshold_str = f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}" if event.max_budget is not None else "N/A"
|
||||
alert_threshold_str = (
|
||||
f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}"
|
||||
if event.max_budget is not None
|
||||
else "N/A"
|
||||
)
|
||||
|
||||
email_html_content = MAX_BUDGET_ALERT_EMAIL_TEMPLATE.format(
|
||||
email_logo_url=email_params.logo_url,
|
||||
@@ -382,7 +404,10 @@ class BaseEmailLogger(CustomLogger):
|
||||
# For non-team alerts, require either max_budget or soft_budget
|
||||
if user_info.max_budget is None and user_info.soft_budget is None:
|
||||
return
|
||||
if user_info.soft_budget is not None and user_info.spend >= user_info.soft_budget:
|
||||
if (
|
||||
user_info.soft_budget is not None
|
||||
and user_info.spend >= user_info.soft_budget
|
||||
):
|
||||
# Generate cache key based on event type and identifier
|
||||
# Use appropriate ID based on event_group to ensure unique cache keys per entity type
|
||||
if user_info.event_group == Litellm_EntityType.TEAM:
|
||||
@@ -395,7 +420,7 @@ class BaseEmailLogger(CustomLogger):
|
||||
# For KEY and other types, use token or user_id
|
||||
_id = user_info.token or user_info.user_id or "default_id"
|
||||
_cache_key = f"email_budget_alerts:soft_budget_crossed:{_id}"
|
||||
|
||||
|
||||
# Check if we've already sent this alert
|
||||
result = await _cache.async_get_cache(key=_cache_key)
|
||||
if result is None:
|
||||
@@ -420,14 +445,14 @@ class BaseEmailLogger(CustomLogger):
|
||||
event_group=user_info.event_group,
|
||||
alert_emails=user_info.alert_emails,
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
# Use team-specific function for team alerts, otherwise use standard function
|
||||
if user_info.event_group == Litellm_EntityType.TEAM:
|
||||
await self.send_team_soft_budget_alert_email(webhook_event)
|
||||
else:
|
||||
await self.send_soft_budget_alert_email(webhook_event)
|
||||
|
||||
|
||||
# Cache the alert to prevent duplicate sends
|
||||
await _cache.async_set_cache(
|
||||
key=_cache_key,
|
||||
@@ -444,20 +469,27 @@ class BaseEmailLogger(CustomLogger):
|
||||
# For max_budget_alert, check if we've already sent an alert
|
||||
if type == "max_budget_alert":
|
||||
if user_info.max_budget is not None and user_info.spend is not None:
|
||||
alert_threshold = user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
|
||||
|
||||
alert_threshold = (
|
||||
user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
|
||||
)
|
||||
|
||||
# Only alert if we've crossed the threshold but haven't exceeded max_budget yet
|
||||
if user_info.spend >= alert_threshold and user_info.spend < user_info.max_budget:
|
||||
if (
|
||||
user_info.spend >= alert_threshold
|
||||
and user_info.spend < user_info.max_budget
|
||||
):
|
||||
# Generate cache key based on event type and identifier
|
||||
_id = user_info.token or user_info.user_id or "default_id"
|
||||
_cache_key = f"email_budget_alerts:max_budget_alert:{_id}"
|
||||
|
||||
|
||||
# Check if we've already sent this alert
|
||||
result = await _cache.async_get_cache(key=_cache_key)
|
||||
if result is None:
|
||||
# Calculate percentage
|
||||
percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
|
||||
|
||||
percentage = int(
|
||||
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100
|
||||
)
|
||||
|
||||
# Create WebhookEvent for max budget alert
|
||||
event_message = f"Max Budget Alert - {percentage}% of Maximum Budget Reached"
|
||||
webhook_event = WebhookEvent(
|
||||
@@ -478,10 +510,10 @@ class BaseEmailLogger(CustomLogger):
|
||||
projected_spend=user_info.projected_spend,
|
||||
event_group=user_info.event_group,
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
await self.send_max_budget_alert_email(webhook_event)
|
||||
|
||||
|
||||
# Cache the alert to prevent duplicate sends
|
||||
await _cache.async_set_cache(
|
||||
key=_cache_key,
|
||||
@@ -525,9 +557,14 @@ class BaseEmailLogger(CustomLogger):
|
||||
unused_custom_fields = []
|
||||
|
||||
# Function to safely get custom value or default
|
||||
def get_custom_or_default(custom_value: Optional[str], default_value: str, field_name: str) -> str:
|
||||
if custom_value is not None: # Only check premium if trying to use custom value
|
||||
def get_custom_or_default(
|
||||
custom_value: Optional[str], default_value: str, field_name: str
|
||||
) -> str:
|
||||
if (
|
||||
custom_value is not None
|
||||
): # Only check premium if trying to use custom value
|
||||
from litellm.proxy.proxy_server import premium_user
|
||||
|
||||
if premium_user is not True:
|
||||
unused_custom_fields.append(field_name)
|
||||
return default_value
|
||||
@@ -536,38 +573,48 @@ class BaseEmailLogger(CustomLogger):
|
||||
|
||||
# Get parameters, falling back to defaults if custom values aren't allowed
|
||||
logo_url = get_custom_or_default(custom_logo, LITELLM_LOGO_URL, "logo URL")
|
||||
support_contact = get_custom_or_default(custom_support, self.DEFAULT_SUPPORT_EMAIL, "support contact")
|
||||
base_url = os.getenv("PROXY_BASE_URL", "http://0.0.0.0:4000") # Not a premium feature
|
||||
signature = get_custom_or_default(custom_signature, EMAIL_FOOTER, "email signature")
|
||||
support_contact = get_custom_or_default(
|
||||
custom_support, self.DEFAULT_SUPPORT_EMAIL, "support contact"
|
||||
)
|
||||
base_url = os.getenv(
|
||||
"PROXY_BASE_URL", "http://0.0.0.0:4000"
|
||||
) # Not a premium feature
|
||||
signature = get_custom_or_default(
|
||||
custom_signature, EMAIL_FOOTER, "email signature"
|
||||
)
|
||||
|
||||
# Get custom subject template based on email event type
|
||||
if email_event == EmailEvent.new_user_invitation:
|
||||
subject_template = get_custom_or_default(
|
||||
custom_subject_invitation,
|
||||
self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.new_user_invitation],
|
||||
"invitation subject template"
|
||||
"invitation subject template",
|
||||
)
|
||||
elif email_event == EmailEvent.virtual_key_created:
|
||||
subject_template = get_custom_or_default(
|
||||
custom_subject_key_created,
|
||||
self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.virtual_key_created],
|
||||
"key created subject template"
|
||||
"key created subject template",
|
||||
)
|
||||
elif email_event == EmailEvent.virtual_key_rotated:
|
||||
custom_subject_key_rotated = os.getenv("EMAIL_SUBJECT_KEY_ROTATED", None)
|
||||
subject_template = get_custom_or_default(
|
||||
custom_subject_key_rotated,
|
||||
self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.virtual_key_rotated],
|
||||
"key rotated subject template"
|
||||
"key rotated subject template",
|
||||
)
|
||||
else:
|
||||
subject_template = "LiteLLM: {event_message}"
|
||||
|
||||
subject = subject_template.format(event_message=event_message) if event_message else "LiteLLM Notification"
|
||||
subject = (
|
||||
subject_template.format(event_message=event_message)
|
||||
if event_message
|
||||
else "LiteLLM Notification"
|
||||
)
|
||||
|
||||
recipient_email: Optional[
|
||||
str
|
||||
] = user_email or await self._lookup_user_email_from_db(user_id=user_id)
|
||||
recipient_email: Optional[str] = (
|
||||
user_email or await self._lookup_user_email_from_db(user_id=user_id)
|
||||
)
|
||||
if recipient_email is None:
|
||||
raise ValueError(
|
||||
f"User email not found for user_id: {user_id}. User email is required to send email."
|
||||
@@ -585,11 +632,9 @@ class BaseEmailLogger(CustomLogger):
|
||||
warning_msg = (
|
||||
f"Email sent with default values instead of custom values for: {fields_str}. "
|
||||
"This is an Enterprise feature. To use custom email fields, please upgrade to LiteLLM Enterprise. "
|
||||
"Schedule a meeting here: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
|
||||
)
|
||||
verbose_proxy_logger.warning(
|
||||
f"{warning_msg}"
|
||||
"Schedule a meeting here: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions"
|
||||
)
|
||||
verbose_proxy_logger.warning(f"{warning_msg}")
|
||||
|
||||
return EmailParams(
|
||||
logo_url=logo_url,
|
||||
@@ -636,44 +681,49 @@ class BaseEmailLogger(CustomLogger):
|
||||
if not user_id:
|
||||
verbose_proxy_logger.debug("No user_id provided for invitation link")
|
||||
return base_url
|
||||
|
||||
|
||||
if not await self._is_prisma_client_available():
|
||||
return base_url
|
||||
|
||||
|
||||
# Wait for any concurrent invitation creation to complete
|
||||
await self._wait_for_invitation_creation()
|
||||
|
||||
|
||||
# Get or create invitation
|
||||
invitation = await self._get_or_create_invitation(user_id)
|
||||
if not invitation:
|
||||
verbose_proxy_logger.warning(f"Failed to get/create invitation for user_id: {user_id}")
|
||||
verbose_proxy_logger.warning(
|
||||
f"Failed to get/create invitation for user_id: {user_id}"
|
||||
)
|
||||
return base_url
|
||||
|
||||
|
||||
return self._construct_invitation_link(invitation.id, base_url)
|
||||
|
||||
async def _is_prisma_client_available(self) -> bool:
|
||||
"""Check if Prisma client is available"""
|
||||
from litellm.proxy.proxy_server import prisma_client
|
||||
|
||||
|
||||
if prisma_client is None:
|
||||
verbose_proxy_logger.debug("Prisma client not found. Unable to lookup invitation")
|
||||
verbose_proxy_logger.debug(
|
||||
"Prisma client not found. Unable to lookup invitation"
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
async def _wait_for_invitation_creation(self) -> None:
|
||||
"""
|
||||
Wait for any concurrent invitation creation to complete.
|
||||
|
||||
|
||||
The UI calls /invitation/new to generate the invitation link.
|
||||
We wait to ensure any pending invitation creation is completed.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
async def _get_or_create_invitation(self, user_id: str):
|
||||
"""
|
||||
Get existing invitation or create a new one for the user
|
||||
|
||||
|
||||
Returns:
|
||||
Invitation object with id attribute, or None if failed
|
||||
"""
|
||||
@@ -681,31 +731,41 @@ class BaseEmailLogger(CustomLogger):
|
||||
create_invitation_for_user,
|
||||
)
|
||||
from litellm.proxy.proxy_server import prisma_client
|
||||
|
||||
|
||||
if prisma_client is None:
|
||||
verbose_proxy_logger.error("Prisma client is None in _get_or_create_invitation")
|
||||
verbose_proxy_logger.error(
|
||||
"Prisma client is None in _get_or_create_invitation"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
try:
|
||||
# Try to get existing invitation
|
||||
existing_invitations = await prisma_client.db.litellm_invitationlink.find_many(
|
||||
where={"user_id": user_id},
|
||||
order={"created_at": "desc"},
|
||||
existing_invitations = (
|
||||
await prisma_client.db.litellm_invitationlink.find_many(
|
||||
where={"user_id": user_id},
|
||||
order={"created_at": "desc"},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if existing_invitations and len(existing_invitations) > 0:
|
||||
verbose_proxy_logger.debug(f"Found existing invitation for user_id: {user_id}")
|
||||
verbose_proxy_logger.debug(
|
||||
f"Found existing invitation for user_id: {user_id}"
|
||||
)
|
||||
return existing_invitations[0]
|
||||
|
||||
|
||||
# Create new invitation if none exists
|
||||
verbose_proxy_logger.debug(f"Creating new invitation for user_id: {user_id}")
|
||||
verbose_proxy_logger.debug(
|
||||
f"Creating new invitation for user_id: {user_id}"
|
||||
)
|
||||
return await create_invitation_for_user(
|
||||
data=InvitationNew(user_id=user_id),
|
||||
user_api_key_dict=UserAPIKeyAuth(user_id=user_id),
|
||||
)
|
||||
|
||||
|
||||
except Exception as e:
|
||||
verbose_proxy_logger.error(f"Error getting/creating invitation for user_id {user_id}: {e}")
|
||||
verbose_proxy_logger.error(
|
||||
f"Error getting/creating invitation for user_id {user_id}: {e}"
|
||||
)
|
||||
return None
|
||||
|
||||
def _construct_invitation_link(self, invitation_id: str, base_url: str) -> str:
|
||||
|
||||
@@ -13,6 +13,9 @@ if TYPE_CHECKING:
|
||||
from litellm.router import Router
|
||||
|
||||
|
||||
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
|
||||
|
||||
|
||||
class CheckBatchCost:
|
||||
def __init__(
|
||||
self,
|
||||
@@ -27,6 +30,25 @@ class CheckBatchCost:
|
||||
self.prisma_client: PrismaClient = prisma_client
|
||||
self.llm_router: Router = llm_router
|
||||
|
||||
async def _get_user_info(self, batch_id, user_id) -> dict:
|
||||
"""
|
||||
Look up user email and key alias by user_id for enriching the S3 callback metadata.
|
||||
Returns a dict with user_api_key_user_email and user_api_key_alias (both may be None).
|
||||
"""
|
||||
try:
|
||||
user_row = await self.prisma_client.db.litellm_usertable.find_unique(
|
||||
where={"user_id": user_id}
|
||||
)
|
||||
if user_row is None:
|
||||
return {}
|
||||
return {
|
||||
"user_api_key_user_email": getattr(user_row, "user_email", None),
|
||||
"user_api_key_alias": getattr(user_row, "user_alias", None),
|
||||
}
|
||||
except Exception as e:
|
||||
verbose_proxy_logger.error(f"CheckBatchCost: could not look up user {user_id} for batch {batch_id}: {e}")
|
||||
return {}
|
||||
|
||||
async def check_batch_cost(self):
|
||||
"""
|
||||
Check if the batch JOB has been tracked.
|
||||
@@ -48,10 +70,12 @@ class CheckBatchCost:
|
||||
get_model_id_from_unified_batch_id,
|
||||
)
|
||||
|
||||
# Look for all batches that have not yet been processed by CheckBatchCost
|
||||
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
|
||||
where={
|
||||
"status": {"in": ["validating", "in_progress", "finalizing"]},
|
||||
"file_purpose": "batch",
|
||||
"batch_processed" : False,
|
||||
"status": {"not_in": ["failed", "expired", "cancelled"]}
|
||||
}
|
||||
)
|
||||
completed_jobs = []
|
||||
@@ -107,6 +131,21 @@ class CheckBatchCost:
|
||||
f"Batch ID: {batch_id} is complete, tracking cost and usage"
|
||||
)
|
||||
|
||||
# aretrieve_batch is called with the raw provider batch ID, so response.id
|
||||
# is the raw provider value (e.g. "batch_20260223-0518.234"). We need the
|
||||
# unified base64 ID in the S3 log so downstream consumers can correlate it
|
||||
# back to the batch they submitted via the proxy.
|
||||
#
|
||||
# CheckBatchCost builds its own LiteLLMLogging object (logging_obj below) and
|
||||
# calls async_success_handler(result=response) directly. That handler calls
|
||||
# _build_standard_logging_payload(response, ...) which reads response.id at
|
||||
# that point — so setting response.id here is sufficient.
|
||||
#
|
||||
# The HTTP endpoint does this substitution via the managed files hook
|
||||
# (async_post_call_success_hook). CheckBatchCost bypasses that hook entirely,
|
||||
# so we do it explicitly here.
|
||||
response.id = job.unified_object_id
|
||||
|
||||
# This background job runs as default_user_id, so going through the HTTP endpoint
|
||||
# would trigger check_managed_file_id_access and get 403. Instead, extract the raw
|
||||
# provider file ID and call afile_content directly with deployment credentials.
|
||||
@@ -171,11 +210,21 @@ class CheckBatchCost:
|
||||
function_id=str(uuid.uuid4()),
|
||||
)
|
||||
|
||||
creator_user_id = job.created_by
|
||||
user_info = await self._get_user_info(batch_id, job.created_by)
|
||||
|
||||
logging_obj.update_environment_variables(
|
||||
litellm_params={
|
||||
# set the user-agent header so that S3 callback consumers can easily identify CheckBatchCost callbacks
|
||||
"proxy_server_request": {
|
||||
"headers": {
|
||||
"user-agent": CHECK_BATCH_COST_USER_AGENT,
|
||||
}
|
||||
},
|
||||
"metadata": {
|
||||
"user_api_key_user_id": job.created_by or "default-user-id",
|
||||
}
|
||||
"user_api_key_user_id": creator_user_id,
|
||||
**user_info,
|
||||
},
|
||||
},
|
||||
optional_params={},
|
||||
)
|
||||
@@ -191,8 +240,7 @@ class CheckBatchCost:
|
||||
completed_jobs.append(job)
|
||||
|
||||
if len(completed_jobs) > 0:
|
||||
# mark the jobs as complete
|
||||
await self.prisma_client.db.litellm_managedobjecttable.update_many(
|
||||
where={"id": {"in": [job.id for job in completed_jobs]}},
|
||||
data={"status": "complete"},
|
||||
data={"batch_processed": True, "status": "complete"},
|
||||
)
|
||||
|
||||
@@ -1086,11 +1086,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
||||
self, file_id: str
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Find batches in non-terminal states that reference this file.
|
||||
|
||||
Non-terminal states: validating, in_progress, finalizing
|
||||
Terminal states: completed, complete, failed, expired, cancelled
|
||||
|
||||
Find batches that reference this file and still need cost tracking.
|
||||
Find batches that are in non-terminal state and have not yet been processed by CheckBatchCost.
|
||||
Args:
|
||||
file_id: The unified file ID to check
|
||||
|
||||
@@ -1121,7 +1118,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
||||
batches = await self.prisma_client.db.litellm_managedobjecttable.find_many(
|
||||
where={
|
||||
"file_purpose": "batch",
|
||||
"status": {"in": ["validating", "in_progress", "finalizing"]},
|
||||
"batch_processed": False,
|
||||
"status": {"not_in": ["failed", "expired", "cancelled"]}
|
||||
},
|
||||
take=MAX_MATCHES_TO_RETURN,
|
||||
order={"created_at": "desc"},
|
||||
@@ -1205,7 +1203,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
||||
|
||||
error_message += (
|
||||
f"To delete this file before complete cost tracking, please delete or cancel the referencing batch(es) first. "
|
||||
f"Alternatively, wait for all batches to complete processing."
|
||||
f"Alternatively, wait for all batches to complete and for cost to be computed (batch_processed=true)."
|
||||
)
|
||||
|
||||
raise HTTPException(
|
||||
|
||||
-2
@@ -1,2 +0,0 @@
|
||||
-- This is an empty migration.
|
||||
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
-- CreateTable
|
||||
CREATE TABLE "LiteLLM_DailyGuardrailMetrics" (
|
||||
"guardrail_id" TEXT NOT NULL,
|
||||
"date" TEXT NOT NULL,
|
||||
"requests_evaluated" BIGINT NOT NULL DEFAULT 0,
|
||||
"passed_count" BIGINT NOT NULL DEFAULT 0,
|
||||
"blocked_count" BIGINT NOT NULL DEFAULT 0,
|
||||
"flagged_count" BIGINT NOT NULL DEFAULT 0,
|
||||
"avg_score" DOUBLE PRECISION,
|
||||
"avg_latency_ms" DOUBLE PRECISION,
|
||||
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMP(3) NOT NULL,
|
||||
|
||||
CONSTRAINT "LiteLLM_DailyGuardrailMetrics_pkey" PRIMARY KEY ("guardrail_id","date")
|
||||
);
|
||||
|
||||
-- CreateTable
|
||||
CREATE TABLE "LiteLLM_DailyPolicyMetrics" (
|
||||
"policy_id" TEXT NOT NULL,
|
||||
"date" TEXT NOT NULL,
|
||||
"requests_evaluated" BIGINT NOT NULL DEFAULT 0,
|
||||
"passed_count" BIGINT NOT NULL DEFAULT 0,
|
||||
"blocked_count" BIGINT NOT NULL DEFAULT 0,
|
||||
"flagged_count" BIGINT NOT NULL DEFAULT 0,
|
||||
"avg_score" DOUBLE PRECISION,
|
||||
"avg_latency_ms" DOUBLE PRECISION,
|
||||
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMP(3) NOT NULL,
|
||||
|
||||
CONSTRAINT "LiteLLM_DailyPolicyMetrics_pkey" PRIMARY KEY ("policy_id","date")
|
||||
);
|
||||
|
||||
-- CreateTable
|
||||
CREATE TABLE "LiteLLM_SpendLogGuardrailIndex" (
|
||||
"request_id" TEXT NOT NULL,
|
||||
"guardrail_id" TEXT NOT NULL,
|
||||
"policy_id" TEXT,
|
||||
"start_time" TIMESTAMP(3) NOT NULL,
|
||||
|
||||
CONSTRAINT "LiteLLM_SpendLogGuardrailIndex_pkey" PRIMARY KEY ("request_id","guardrail_id")
|
||||
);
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "LiteLLM_DailyGuardrailMetrics_date_idx" ON "LiteLLM_DailyGuardrailMetrics"("date");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "LiteLLM_DailyGuardrailMetrics_guardrail_id_idx" ON "LiteLLM_DailyGuardrailMetrics"("guardrail_id");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "LiteLLM_DailyPolicyMetrics_date_idx" ON "LiteLLM_DailyPolicyMetrics"("date");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "LiteLLM_DailyPolicyMetrics_policy_id_idx" ON "LiteLLM_DailyPolicyMetrics"("policy_id");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "LiteLLM_SpendLogGuardrailIndex_guardrail_id_start_time_idx" ON "LiteLLM_SpendLogGuardrailIndex"("guardrail_id", "start_time");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "LiteLLM_SpendLogGuardrailIndex_policy_id_start_time_idx" ON "LiteLLM_SpendLogGuardrailIndex"("policy_id", "start_time");
|
||||
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
-- Add batch_processed column to LiteLLM_ManagedObjectTable
|
||||
-- Set to true by CheckBatchCost after cost has been computed for a completed batch
|
||||
ALTER TABLE "LiteLLM_ManagedObjectTable" ADD COLUMN "batch_processed" BOOLEAN NOT NULL DEFAULT false;
|
||||
@@ -813,6 +813,7 @@ model LiteLLM_ManagedObjectTable { // for batches or finetuning jobs which use t
|
||||
file_object Json // Stores the OpenAIFileObject
|
||||
file_purpose String // either 'batch' or 'fine-tune'
|
||||
status String? // check if batch cost has been tracked
|
||||
batch_processed Boolean @default(false) // set to true by CheckBatchCost after cost is computed
|
||||
created_at DateTime @default(now())
|
||||
created_by String?
|
||||
updated_at DateTime @updatedAt
|
||||
@@ -866,6 +867,54 @@ model LiteLLM_GuardrailsTable {
|
||||
updated_at DateTime @updatedAt
|
||||
}
|
||||
|
||||
// Daily guardrail metrics for usage dashboard (one row per guardrail per day)
|
||||
model LiteLLM_DailyGuardrailMetrics {
|
||||
guardrail_id String // logical id; may not FK if guardrail from config
|
||||
date String // YYYY-MM-DD
|
||||
requests_evaluated BigInt @default(0)
|
||||
passed_count BigInt @default(0)
|
||||
blocked_count BigInt @default(0)
|
||||
flagged_count BigInt @default(0)
|
||||
avg_score Float?
|
||||
avg_latency_ms Float?
|
||||
created_at DateTime @default(now())
|
||||
updated_at DateTime @updatedAt
|
||||
|
||||
@@id([guardrail_id, date])
|
||||
@@index([date])
|
||||
@@index([guardrail_id])
|
||||
}
|
||||
|
||||
// Daily policy metrics for usage dashboard (one row per policy per day)
|
||||
model LiteLLM_DailyPolicyMetrics {
|
||||
policy_id String
|
||||
date String // YYYY-MM-DD
|
||||
requests_evaluated BigInt @default(0)
|
||||
passed_count BigInt @default(0)
|
||||
blocked_count BigInt @default(0)
|
||||
flagged_count BigInt @default(0)
|
||||
avg_score Float?
|
||||
avg_latency_ms Float?
|
||||
created_at DateTime @default(now())
|
||||
updated_at DateTime @updatedAt
|
||||
|
||||
@@id([policy_id, date])
|
||||
@@index([date])
|
||||
@@index([policy_id])
|
||||
}
|
||||
|
||||
// Index for fast "last N logs for guardrail/policy" from SpendLogs
|
||||
model LiteLLM_SpendLogGuardrailIndex {
|
||||
request_id String
|
||||
guardrail_id String
|
||||
policy_id String? // set when run as part of a policy pipeline
|
||||
start_time DateTime
|
||||
|
||||
@@id([request_id, guardrail_id])
|
||||
@@index([guardrail_id, start_time])
|
||||
@@index([policy_id, start_time])
|
||||
}
|
||||
|
||||
// Prompt table for storing prompt configurations
|
||||
model LiteLLM_PromptTable {
|
||||
id String @id @default(uuid())
|
||||
|
||||
+8
-2
@@ -339,6 +339,10 @@ model_cost_map_url: str = os.getenv(
|
||||
"LITELLM_MODEL_COST_MAP_URL",
|
||||
"https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json",
|
||||
)
|
||||
blog_posts_url: str = os.getenv(
|
||||
"LITELLM_BLOG_POSTS_URL",
|
||||
"https://raw.githubusercontent.com/BerriAI/litellm/main/litellm/blog_posts.json",
|
||||
)
|
||||
anthropic_beta_headers_url: str = os.getenv(
|
||||
"LITELLM_ANTHROPIC_BETA_HEADERS_URL",
|
||||
"https://raw.githubusercontent.com/BerriAI/litellm/main/litellm/anthropic_beta_headers_config.json",
|
||||
@@ -405,6 +409,7 @@ disable_aiohttp_trust_env: bool = (
|
||||
force_ipv4: bool = (
|
||||
False # when True, litellm will force ipv4 for all LLM requests. Some users have seen httpx ConnectionError when using ipv6.
|
||||
)
|
||||
network_mock: bool = False # When True, use mock transport — no real network calls
|
||||
|
||||
####### STOP SEQUENCE LIMIT #######
|
||||
disable_stop_sequence_limit: bool = False # when True, stop sequence limit is disabled
|
||||
@@ -614,8 +619,9 @@ def is_openai_finetune_model(key: str) -> bool:
|
||||
return key.startswith("ft:") and not key.count(":") > 1
|
||||
|
||||
|
||||
def add_known_models():
|
||||
for key, value in model_cost.items():
|
||||
def add_known_models(model_cost_map: Optional[Dict] = None):
|
||||
_map = model_cost_map if model_cost_map is not None else model_cost
|
||||
for key, value in _map.items():
|
||||
if value.get("litellm_provider") == "openai" and not is_openai_finetune_model(
|
||||
key
|
||||
):
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"posts": [
|
||||
{
|
||||
"title": "Incident Report: SERVER_ROOT_PATH regression broke UI routing",
|
||||
"description": "How a single line removal caused UI 404s for all deployments using SERVER_ROOT_PATH, and the tests we added to prevent it from happening again.",
|
||||
"date": "2026-02-21",
|
||||
"url": "https://docs.litellm.ai/blog/server-root-path-incident"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -108,6 +108,7 @@ class Cache:
|
||||
qdrant_collection_name: Optional[str] = None,
|
||||
qdrant_quantization_config: Optional[str] = None,
|
||||
qdrant_semantic_cache_embedding_model: str = "text-embedding-ada-002",
|
||||
qdrant_semantic_cache_vector_size: Optional[int] = None,
|
||||
# GCP IAM authentication parameters
|
||||
gcp_service_account: Optional[str] = None,
|
||||
gcp_ssl_ca_certs: Optional[str] = None,
|
||||
@@ -207,6 +208,7 @@ class Cache:
|
||||
similarity_threshold=similarity_threshold,
|
||||
quantization_config=qdrant_quantization_config,
|
||||
embedding_model=qdrant_semantic_cache_embedding_model,
|
||||
vector_size=qdrant_semantic_cache_vector_size,
|
||||
)
|
||||
elif type == LiteLLMCacheType.LOCAL:
|
||||
self.cache = InMemoryCache()
|
||||
|
||||
@@ -31,6 +31,7 @@ class QdrantSemanticCache(BaseCache):
|
||||
quantization_config=None,
|
||||
embedding_model="text-embedding-ada-002",
|
||||
host_type=None,
|
||||
vector_size=None,
|
||||
):
|
||||
import os
|
||||
|
||||
@@ -53,6 +54,7 @@ class QdrantSemanticCache(BaseCache):
|
||||
raise Exception("similarity_threshold must be provided, passed None")
|
||||
self.similarity_threshold = similarity_threshold
|
||||
self.embedding_model = embedding_model
|
||||
self.vector_size = vector_size if vector_size is not None else QDRANT_VECTOR_SIZE
|
||||
headers = {}
|
||||
|
||||
# check if defined as os.environ/ variable
|
||||
@@ -138,7 +140,7 @@ class QdrantSemanticCache(BaseCache):
|
||||
new_collection_status = self.sync_client.put(
|
||||
url=f"{self.qdrant_api_base}/collections/{self.collection_name}",
|
||||
json={
|
||||
"vectors": {"size": QDRANT_VECTOR_SIZE, "distance": "Cosine"},
|
||||
"vectors": {"size": self.vector_size, "distance": "Cosine"},
|
||||
"quantization_config": quantization_params,
|
||||
},
|
||||
headers=self.headers,
|
||||
|
||||
@@ -242,9 +242,13 @@ REDIS_DAILY_END_USER_SPEND_UPDATE_BUFFER_KEY = (
|
||||
REDIS_DAILY_AGENT_SPEND_UPDATE_BUFFER_KEY = "litellm_daily_agent_spend_update_buffer"
|
||||
REDIS_DAILY_TAG_SPEND_UPDATE_BUFFER_KEY = "litellm_daily_tag_spend_update_buffer"
|
||||
MAX_REDIS_BUFFER_DEQUEUE_COUNT = int(os.getenv("MAX_REDIS_BUFFER_DEQUEUE_COUNT", 100))
|
||||
MAX_SIZE_IN_MEMORY_QUEUE = int(os.getenv("MAX_SIZE_IN_MEMORY_QUEUE", 2000))
|
||||
# Bounds asyncio.Queue() instances (log queues, spend update queues, etc.) to prevent unbounded memory growth
|
||||
LITELLM_ASYNCIO_QUEUE_MAXSIZE = int(os.getenv("LITELLM_ASYNCIO_QUEUE_MAXSIZE", 1000))
|
||||
# Aggregation threshold: default to 80% of the asyncio queue maxsize so the check can always trigger.
|
||||
# Must be < LITELLM_ASYNCIO_QUEUE_MAXSIZE; if set higher the aggregation logic will never fire.
|
||||
MAX_SIZE_IN_MEMORY_QUEUE = int(
|
||||
os.getenv("MAX_SIZE_IN_MEMORY_QUEUE", int(LITELLM_ASYNCIO_QUEUE_MAXSIZE * 0.8))
|
||||
)
|
||||
MAX_IN_MEMORY_QUEUE_FLUSH_COUNT = int(
|
||||
os.getenv("MAX_IN_MEMORY_QUEUE_FLUSH_COUNT", 1000)
|
||||
)
|
||||
|
||||
@@ -480,6 +480,7 @@ def cost_per_token( # noqa: PLR0915
|
||||
model=model_without_prefix,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
usage=usage_block,
|
||||
service_tier=service_tier,
|
||||
)
|
||||
elif custom_llm_provider == "anthropic":
|
||||
return anthropic_cost_per_token(model=model, usage=usage_block)
|
||||
@@ -500,7 +501,9 @@ def cost_per_token( # noqa: PLR0915
|
||||
model=model, usage=usage_block, response_time_ms=response_time_ms
|
||||
)
|
||||
elif custom_llm_provider == "gemini":
|
||||
return gemini_cost_per_token(model=model, usage=usage_block)
|
||||
return gemini_cost_per_token(
|
||||
model=model, usage=usage_block, service_tier=service_tier
|
||||
)
|
||||
elif custom_llm_provider == "deepseek":
|
||||
return deepseek_cost_per_token(model=model, usage=usage_block)
|
||||
elif custom_llm_provider == "perplexity":
|
||||
@@ -704,6 +707,36 @@ def _get_response_model(completion_response: Any) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
_GEMINI_TRAFFIC_TYPE_TO_SERVICE_TIER: dict = {
|
||||
# ON_DEMAND_PRIORITY maps to "priority" — selects input_cost_per_token_priority, etc.
|
||||
"ON_DEMAND_PRIORITY": "priority",
|
||||
# FLEX / BATCH maps to "flex" — selects input_cost_per_token_flex, etc.
|
||||
"FLEX": "flex",
|
||||
"BATCH": "flex",
|
||||
# ON_DEMAND is standard pricing — no service_tier suffix applied
|
||||
"ON_DEMAND": None,
|
||||
}
|
||||
|
||||
|
||||
def _map_traffic_type_to_service_tier(traffic_type: Optional[str]) -> Optional[str]:
|
||||
"""
|
||||
Map a Gemini usageMetadata.trafficType value to a LiteLLM service_tier string.
|
||||
|
||||
This allows the same `_priority` / `_flex` cost-key suffix logic used for
|
||||
OpenAI/Azure to work for Gemini and Vertex AI models.
|
||||
|
||||
trafficType values seen in practice
|
||||
------------------------------------
|
||||
ON_DEMAND -> standard pricing (service_tier = None)
|
||||
ON_DEMAND_PRIORITY -> priority pricing (service_tier = "priority")
|
||||
FLEX / BATCH -> batch/flex pricing (service_tier = "flex")
|
||||
"""
|
||||
if traffic_type is None:
|
||||
return None
|
||||
service_tier = _GEMINI_TRAFFIC_TYPE_TO_SERVICE_TIER.get(traffic_type.upper())
|
||||
return service_tier
|
||||
|
||||
|
||||
def _get_usage_object(
|
||||
completion_response: Any,
|
||||
) -> Optional[Usage]:
|
||||
@@ -1145,6 +1178,20 @@ def completion_cost( # noqa: PLR0915
|
||||
"custom_llm_provider", custom_llm_provider or None
|
||||
)
|
||||
region_name = hidden_params.get("region_name", region_name)
|
||||
|
||||
# For Gemini/Vertex AI responses, trafficType is stored in
|
||||
# provider_specific_fields. Map it to the service_tier used
|
||||
# by the cost key lookup (_priority / _flex suffixes) so that
|
||||
# ON_DEMAND_PRIORITY requests are billed at priority prices.
|
||||
if service_tier is None:
|
||||
provider_specific = (
|
||||
hidden_params.get("provider_specific_fields") or {}
|
||||
)
|
||||
raw_traffic_type = provider_specific.get("traffic_type")
|
||||
if raw_traffic_type:
|
||||
service_tier = _map_traffic_type_to_service_tier(
|
||||
raw_traffic_type
|
||||
)
|
||||
else:
|
||||
if model is None:
|
||||
raise ValueError(
|
||||
|
||||
@@ -172,4 +172,6 @@ Team Alias: `{hanging_request_data.team_alias}`"""
|
||||
level="Medium",
|
||||
alert_type=AlertType.llm_requests_hanging,
|
||||
alerting_metadata=hanging_request_data.alerting_metadata or {},
|
||||
request_model=hanging_request_data.model,
|
||||
api_base=hanging_request_data.api_base,
|
||||
)
|
||||
|
||||
@@ -70,6 +70,7 @@ class SlackAlerting(CustomBatchLogger):
|
||||
] = None, # if user wants to separate alerts to diff channels
|
||||
alerting_args={},
|
||||
default_webhook_url: Optional[str] = None,
|
||||
alert_type_config: Optional[Dict[str, dict]] = None,
|
||||
**kwargs,
|
||||
):
|
||||
if alerting_threshold is None:
|
||||
@@ -92,6 +93,12 @@ class SlackAlerting(CustomBatchLogger):
|
||||
self.hanging_request_check = AlertingHangingRequestCheck(
|
||||
slack_alerting_object=self,
|
||||
)
|
||||
self.alert_type_config: Dict[str, AlertTypeConfig] = {}
|
||||
if alert_type_config:
|
||||
for key, val in alert_type_config.items():
|
||||
self.alert_type_config[key] = AlertTypeConfig(**val) if isinstance(val, dict) else val
|
||||
self.digest_buckets: Dict[str, DigestEntry] = {}
|
||||
self.digest_lock = asyncio.Lock()
|
||||
super().__init__(**kwargs, flush_lock=self.flush_lock)
|
||||
|
||||
def update_values(
|
||||
@@ -102,6 +109,7 @@ class SlackAlerting(CustomBatchLogger):
|
||||
alert_to_webhook_url: Optional[Dict[AlertType, Union[List[str], str]]] = None,
|
||||
alerting_args: Optional[Dict] = None,
|
||||
llm_router: Optional[Router] = None,
|
||||
alert_type_config: Optional[Dict[str, dict]] = None,
|
||||
):
|
||||
if alerting is not None:
|
||||
self.alerting = alerting
|
||||
@@ -116,6 +124,9 @@ class SlackAlerting(CustomBatchLogger):
|
||||
if not self.periodic_started:
|
||||
asyncio.create_task(self.periodic_flush())
|
||||
self.periodic_started = True
|
||||
if alert_type_config is not None:
|
||||
for key, val in alert_type_config.items():
|
||||
self.alert_type_config[key] = AlertTypeConfig(**val) if isinstance(val, dict) else val
|
||||
|
||||
if alert_to_webhook_url is not None:
|
||||
# update the dict
|
||||
@@ -284,6 +295,8 @@ class SlackAlerting(CustomBatchLogger):
|
||||
level="Low",
|
||||
alert_type=AlertType.llm_too_slow,
|
||||
alerting_metadata=alerting_metadata,
|
||||
request_model=model,
|
||||
api_base=api_base,
|
||||
)
|
||||
|
||||
async def async_update_daily_reports(
|
||||
@@ -1354,13 +1367,15 @@ Model Info:
|
||||
|
||||
return False
|
||||
|
||||
async def send_alert(
|
||||
async def send_alert( # noqa: PLR0915
|
||||
self,
|
||||
message: str,
|
||||
level: Literal["Low", "Medium", "High"],
|
||||
alert_type: AlertType,
|
||||
alerting_metadata: dict,
|
||||
user_info: Optional[WebhookEvent] = None,
|
||||
request_model: Optional[str] = None,
|
||||
api_base: Optional[str] = None,
|
||||
**kwargs,
|
||||
):
|
||||
"""
|
||||
@@ -1376,6 +1391,8 @@ Model Info:
|
||||
Parameters:
|
||||
level: str - Low|Medium|High - if calls might fail (Medium) or are failing (High); Currently, no alerts would be 'Low'.
|
||||
message: str - what is the alert about
|
||||
request_model: Optional[str] - model name for digest grouping
|
||||
api_base: Optional[str] - api base for digest grouping
|
||||
"""
|
||||
if self.alerting is None:
|
||||
return
|
||||
@@ -1413,6 +1430,44 @@ Model Info:
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
# Check if digest mode is enabled for this alert type
|
||||
alert_type_name_str = getattr(alert_type, "value", str(alert_type))
|
||||
_atc = self.alert_type_config.get(alert_type_name_str)
|
||||
if _atc is not None and _atc.digest:
|
||||
# Resolve webhook URL for this alert type (needed for digest entry)
|
||||
if (
|
||||
self.alert_to_webhook_url is not None
|
||||
and alert_type in self.alert_to_webhook_url
|
||||
):
|
||||
_digest_webhook: Optional[Union[str, List[str]]] = self.alert_to_webhook_url[alert_type]
|
||||
elif self.default_webhook_url is not None:
|
||||
_digest_webhook = self.default_webhook_url
|
||||
else:
|
||||
_digest_webhook = os.getenv("SLACK_WEBHOOK_URL", None)
|
||||
if _digest_webhook is None:
|
||||
raise ValueError("Missing SLACK_WEBHOOK_URL from environment")
|
||||
|
||||
digest_key = f"{alert_type_name_str}:{request_model or ''}:{api_base or ''}"
|
||||
|
||||
async with self.digest_lock:
|
||||
now = datetime.now()
|
||||
if digest_key in self.digest_buckets:
|
||||
self.digest_buckets[digest_key]["count"] += 1
|
||||
self.digest_buckets[digest_key]["last_time"] = now
|
||||
else:
|
||||
self.digest_buckets[digest_key] = DigestEntry(
|
||||
alert_type=alert_type_name_str,
|
||||
request_model=request_model or "",
|
||||
api_base=api_base or "",
|
||||
first_message=message,
|
||||
level=level,
|
||||
count=1,
|
||||
start_time=now,
|
||||
last_time=now,
|
||||
webhook_url=_digest_webhook,
|
||||
)
|
||||
return # Suppress immediate alert; will be emitted by _flush_digest_buckets
|
||||
|
||||
# Get the current timestamp
|
||||
current_time = datetime.now().strftime("%H:%M:%S")
|
||||
_proxy_base_url = os.getenv("PROXY_BASE_URL", None)
|
||||
@@ -1488,6 +1543,72 @@ Model Info:
|
||||
await asyncio.gather(*tasks)
|
||||
self.log_queue.clear()
|
||||
|
||||
async def _flush_digest_buckets(self):
|
||||
"""Flush any digest buckets whose interval has expired.
|
||||
|
||||
For each expired bucket, formats a digest summary message and
|
||||
appends it to the log_queue for delivery via the normal batching path.
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
now = datetime.now()
|
||||
flushed_keys: List[str] = []
|
||||
|
||||
async with self.digest_lock:
|
||||
for key, entry in self.digest_buckets.items():
|
||||
alert_type_name = entry["alert_type"]
|
||||
_atc = self.alert_type_config.get(alert_type_name)
|
||||
if _atc is None:
|
||||
continue
|
||||
elapsed = (now - entry["start_time"]).total_seconds()
|
||||
if elapsed < _atc.digest_interval:
|
||||
continue
|
||||
|
||||
# Build digest summary message
|
||||
start_ts = entry["start_time"].strftime("%H:%M:%S")
|
||||
end_ts = entry["last_time"].strftime("%H:%M:%S")
|
||||
start_date = entry["start_time"].strftime("%Y-%m-%d")
|
||||
end_date = entry["last_time"].strftime("%Y-%m-%d")
|
||||
formatted_message = (
|
||||
f"Alert type: `{alert_type_name}` (Digest)\n"
|
||||
f"Level: `{entry['level']}`\n"
|
||||
f"Start: `{start_date} {start_ts}`\n"
|
||||
f"End: `{end_date} {end_ts}`\n"
|
||||
f"Count: `{entry['count']}`\n\n"
|
||||
f"Message: {entry['first_message']}"
|
||||
)
|
||||
_proxy_base_url = os.getenv("PROXY_BASE_URL", None)
|
||||
if _proxy_base_url is not None:
|
||||
formatted_message += f"\n\nProxy URL: `{_proxy_base_url}`"
|
||||
|
||||
payload = {"text": formatted_message}
|
||||
headers = {"Content-type": "application/json"}
|
||||
webhook_url = entry["webhook_url"]
|
||||
|
||||
if isinstance(webhook_url, list):
|
||||
for url in webhook_url:
|
||||
self.log_queue.append(
|
||||
{"url": url, "headers": headers, "payload": payload, "alert_type": alert_type_name}
|
||||
)
|
||||
else:
|
||||
self.log_queue.append(
|
||||
{"url": webhook_url, "headers": headers, "payload": payload, "alert_type": alert_type_name}
|
||||
)
|
||||
flushed_keys.append(key)
|
||||
|
||||
for key in flushed_keys:
|
||||
del self.digest_buckets[key]
|
||||
|
||||
async def periodic_flush(self):
|
||||
"""Override base periodic_flush to also flush digest buckets."""
|
||||
while True:
|
||||
await asyncio.sleep(self.flush_interval)
|
||||
try:
|
||||
await self._flush_digest_buckets()
|
||||
except Exception as e:
|
||||
verbose_proxy_logger.debug(f"Error flushing digest buckets: {str(e)}")
|
||||
await self.flush_queue()
|
||||
|
||||
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
|
||||
"""Log deployment latency"""
|
||||
try:
|
||||
|
||||
@@ -587,9 +587,10 @@ class CustomGuardrail(CustomLogger):
|
||||
elif "litellm_metadata" in request_data:
|
||||
_append_guardrail_info(request_data["litellm_metadata"])
|
||||
else:
|
||||
verbose_logger.warning(
|
||||
"unable to log guardrail information. No metadata found in request_data"
|
||||
)
|
||||
# Ensure guardrail info is always logged (e.g. proxy may not have set
|
||||
# metadata yet). Attach to "metadata" so spend log / standard logging see it.
|
||||
request_data["metadata"] = {}
|
||||
_append_guardrail_info(request_data["metadata"])
|
||||
|
||||
async def apply_guardrail(
|
||||
self,
|
||||
|
||||
@@ -8,8 +8,9 @@ duration_in_seconds is used in diff parts of the code base, example
|
||||
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from datetime import datetime, timedelta, timezone, tzinfo
|
||||
from typing import Optional, Tuple
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
|
||||
def _extract_from_regex(duration: str) -> Tuple[int, str]:
|
||||
@@ -116,7 +117,7 @@ def get_next_standardized_reset_time(
|
||||
- Next reset time at a standardized interval in the specified timezone
|
||||
"""
|
||||
# Set up timezone and normalize current time
|
||||
current_time, timezone = _setup_timezone(current_time, timezone_str)
|
||||
current_time, tz = _setup_timezone(current_time, timezone_str)
|
||||
|
||||
# Parse duration
|
||||
value, unit = _parse_duration(duration)
|
||||
@@ -131,7 +132,7 @@ def get_next_standardized_reset_time(
|
||||
|
||||
# Handle different time units
|
||||
if unit == "d":
|
||||
return _handle_day_reset(current_time, base_midnight, value, timezone)
|
||||
return _handle_day_reset(current_time, base_midnight, value, tz)
|
||||
elif unit == "h":
|
||||
return _handle_hour_reset(current_time, base_midnight, value)
|
||||
elif unit == "m":
|
||||
@@ -147,22 +148,13 @@ def get_next_standardized_reset_time(
|
||||
|
||||
def _setup_timezone(
|
||||
current_time: datetime, timezone_str: str = "UTC"
|
||||
) -> Tuple[datetime, timezone]:
|
||||
) -> Tuple[datetime, tzinfo]:
|
||||
"""Set up timezone and normalize current time to that timezone."""
|
||||
try:
|
||||
if timezone_str is None:
|
||||
tz = timezone.utc
|
||||
tz: tzinfo = timezone.utc
|
||||
else:
|
||||
# Map common timezone strings to their UTC offsets
|
||||
timezone_map = {
|
||||
"US/Eastern": timezone(timedelta(hours=-4)), # EDT
|
||||
"US/Pacific": timezone(timedelta(hours=-7)), # PDT
|
||||
"Asia/Kolkata": timezone(timedelta(hours=5, minutes=30)), # IST
|
||||
"Asia/Bangkok": timezone(timedelta(hours=7)), # ICT (Indochina Time)
|
||||
"Europe/London": timezone(timedelta(hours=1)), # BST
|
||||
"UTC": timezone.utc,
|
||||
}
|
||||
tz = timezone_map.get(timezone_str, timezone.utc)
|
||||
tz = ZoneInfo(timezone_str)
|
||||
except Exception:
|
||||
# If timezone is invalid, fall back to UTC
|
||||
tz = timezone.utc
|
||||
@@ -190,7 +182,7 @@ def _parse_duration(duration: str) -> Tuple[Optional[int], Optional[str]]:
|
||||
|
||||
|
||||
def _handle_day_reset(
|
||||
current_time: datetime, base_midnight: datetime, value: int, timezone: timezone
|
||||
current_time: datetime, base_midnight: datetime, value: int, tz: tzinfo
|
||||
) -> datetime:
|
||||
"""Handle day-based reset times."""
|
||||
# Handle zero value - immediate expiration
|
||||
@@ -215,7 +207,7 @@ def _handle_day_reset(
|
||||
minute=0,
|
||||
second=0,
|
||||
microsecond=0,
|
||||
tzinfo=timezone,
|
||||
tzinfo=tz,
|
||||
)
|
||||
else:
|
||||
next_reset = datetime(
|
||||
@@ -226,7 +218,7 @@ def _handle_day_reset(
|
||||
minute=0,
|
||||
second=0,
|
||||
microsecond=0,
|
||||
tzinfo=timezone,
|
||||
tzinfo=tz,
|
||||
)
|
||||
return next_reset
|
||||
else: # Custom day value - next interval is value days from current
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
"""
|
||||
Pulls the latest LiteLLM blog posts from GitHub.
|
||||
|
||||
Falls back to the bundled local backup on any failure.
|
||||
GitHub JSON URL is configured via litellm.blog_posts_url (or LITELLM_BLOG_POSTS_URL env var).
|
||||
|
||||
Disable remote fetching entirely:
|
||||
export LITELLM_LOCAL_BLOG_POSTS=True
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from importlib.resources import files
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
from pydantic import BaseModel
|
||||
|
||||
from litellm import verbose_logger
|
||||
|
||||
BLOG_POSTS_TTL_SECONDS: int = 3600 # 1 hour
|
||||
|
||||
|
||||
class BlogPost(BaseModel):
|
||||
title: str
|
||||
description: str
|
||||
date: str
|
||||
url: str
|
||||
|
||||
|
||||
class BlogPostsResponse(BaseModel):
|
||||
posts: List[BlogPost]
|
||||
|
||||
|
||||
class GetBlogPosts:
|
||||
"""
|
||||
Fetches, validates, and caches LiteLLM blog posts.
|
||||
|
||||
Mirrors the structure of GetModelCostMap:
|
||||
- Fetches from GitHub with a 5-second timeout
|
||||
- Validates the response has a non-empty ``posts`` list
|
||||
- Caches the result in-process for BLOG_POSTS_TTL_SECONDS (1 hour)
|
||||
- Falls back to the bundled local backup on any failure
|
||||
"""
|
||||
|
||||
_cached_posts: Optional[List[Dict[str, str]]] = None
|
||||
_last_fetch_time: float = 0.0
|
||||
|
||||
@staticmethod
|
||||
def load_local_blog_posts() -> List[Dict[str, str]]:
|
||||
"""Load the bundled local backup blog posts."""
|
||||
content = json.loads(
|
||||
files("litellm")
|
||||
.joinpath("blog_posts.json")
|
||||
.read_text(encoding="utf-8")
|
||||
)
|
||||
return content.get("posts", [])
|
||||
|
||||
@staticmethod
|
||||
def fetch_remote_blog_posts(url: str, timeout: int = 5) -> dict:
|
||||
"""
|
||||
Fetch blog posts JSON from a remote URL.
|
||||
|
||||
Returns the parsed response. Raises on network/parse errors.
|
||||
"""
|
||||
response = httpx.get(url, timeout=timeout)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
@staticmethod
|
||||
def validate_blog_posts(data: Any) -> bool:
|
||||
"""Return True if data is a dict with a non-empty ``posts`` list."""
|
||||
if not isinstance(data, dict):
|
||||
verbose_logger.warning(
|
||||
"LiteLLM: Blog posts response is not a dict (type=%s). "
|
||||
"Falling back to local backup.",
|
||||
type(data).__name__,
|
||||
)
|
||||
return False
|
||||
posts = data.get("posts")
|
||||
if not isinstance(posts, list) or len(posts) == 0:
|
||||
verbose_logger.warning(
|
||||
"LiteLLM: Blog posts response has no valid 'posts' list. "
|
||||
"Falling back to local backup.",
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def get_blog_posts(cls, url: str) -> List[Dict[str, str]]:
|
||||
"""
|
||||
Return the blog posts list.
|
||||
|
||||
Uses the in-process cache if within BLOG_POSTS_TTL_SECONDS.
|
||||
Fetches from ``url`` otherwise, falling back to local backup on failure.
|
||||
"""
|
||||
if os.getenv("LITELLM_LOCAL_BLOG_POSTS", "").lower() == "true":
|
||||
return cls.load_local_blog_posts()
|
||||
|
||||
now = time.time()
|
||||
cached = cls._cached_posts
|
||||
if cached is not None and (now - cls._last_fetch_time) < BLOG_POSTS_TTL_SECONDS:
|
||||
return cached
|
||||
|
||||
try:
|
||||
data = cls.fetch_remote_blog_posts(url)
|
||||
except Exception as e:
|
||||
verbose_logger.warning(
|
||||
"LiteLLM: Failed to fetch blog posts from %s: %s. "
|
||||
"Falling back to local backup.",
|
||||
url,
|
||||
str(e),
|
||||
)
|
||||
return cls.load_local_blog_posts()
|
||||
|
||||
if not cls.validate_blog_posts(data):
|
||||
return cls.load_local_blog_posts()
|
||||
|
||||
posts = data["posts"]
|
||||
cls._cached_posts = posts
|
||||
cls._last_fetch_time = now
|
||||
return posts
|
||||
|
||||
|
||||
def get_blog_posts(url: str) -> List[Dict[str, str]]:
|
||||
"""Public entry point — returns the blog posts list."""
|
||||
return GetBlogPosts.get_blog_posts(url=url)
|
||||
@@ -11,6 +11,7 @@ export LITELLM_LOCAL_MODEL_COST_MAP=True
|
||||
import json
|
||||
import os
|
||||
from importlib.resources import files
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
@@ -151,6 +152,37 @@ class GetModelCostMap:
|
||||
return response.json()
|
||||
|
||||
|
||||
class ModelCostMapSourceInfo:
|
||||
"""Tracks the source of the currently loaded model cost map."""
|
||||
|
||||
source: str = "local" # "local" or "remote"
|
||||
url: Optional[str] = None
|
||||
is_env_forced: bool = False
|
||||
fallback_reason: Optional[str] = None
|
||||
|
||||
|
||||
# Module-level singleton tracking the source of the current cost map
|
||||
_cost_map_source_info = ModelCostMapSourceInfo()
|
||||
|
||||
|
||||
def get_model_cost_map_source_info() -> dict:
|
||||
"""
|
||||
Return metadata about where the current model cost map was loaded from.
|
||||
|
||||
Returns a dict with:
|
||||
- source: "local" or "remote"
|
||||
- url: the remote URL attempted (or None for local-only)
|
||||
- is_env_forced: True if LITELLM_LOCAL_MODEL_COST_MAP=True forced local usage
|
||||
- fallback_reason: human-readable reason if remote failed and local was used
|
||||
"""
|
||||
return {
|
||||
"source": _cost_map_source_info.source,
|
||||
"url": _cost_map_source_info.url,
|
||||
"is_env_forced": _cost_map_source_info.is_env_forced,
|
||||
"fallback_reason": _cost_map_source_info.fallback_reason,
|
||||
}
|
||||
|
||||
|
||||
def get_model_cost_map(url: str) -> dict:
|
||||
"""
|
||||
Public entry point — returns the model cost map dict.
|
||||
@@ -166,8 +198,15 @@ def get_model_cost_map(url: str) -> dict:
|
||||
# Note: can't use get_secret_bool here — this runs during litellm.__init__
|
||||
# before litellm._key_management_settings is set.
|
||||
if os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", "").lower() == "true":
|
||||
_cost_map_source_info.source = "local"
|
||||
_cost_map_source_info.url = None
|
||||
_cost_map_source_info.is_env_forced = True
|
||||
_cost_map_source_info.fallback_reason = None
|
||||
return GetModelCostMap.load_local_model_cost_map()
|
||||
|
||||
_cost_map_source_info.url = url
|
||||
_cost_map_source_info.is_env_forced = False
|
||||
|
||||
try:
|
||||
content = GetModelCostMap.fetch_remote_model_cost_map(url)
|
||||
except Exception as e:
|
||||
@@ -177,6 +216,8 @@ def get_model_cost_map(url: str) -> dict:
|
||||
url,
|
||||
str(e),
|
||||
)
|
||||
_cost_map_source_info.source = "local"
|
||||
_cost_map_source_info.fallback_reason = f"Remote fetch failed: {str(e)}"
|
||||
return GetModelCostMap.load_local_model_cost_map()
|
||||
|
||||
# Validate using cached count (cheap int comparison, no file I/O)
|
||||
@@ -189,6 +230,10 @@ def get_model_cost_map(url: str) -> dict:
|
||||
"Using local backup instead. url=%s",
|
||||
url,
|
||||
)
|
||||
_cost_map_source_info.source = "local"
|
||||
_cost_map_source_info.fallback_reason = "Remote data failed integrity validation"
|
||||
return GetModelCostMap.load_local_model_cost_map()
|
||||
|
||||
_cost_map_source_info.source = "remote"
|
||||
_cost_map_source_info.fallback_reason = None
|
||||
return content
|
||||
|
||||
@@ -4709,6 +4709,7 @@ class StandardLoggingPayloadSetup:
|
||||
custom_pricing: Optional[bool],
|
||||
custom_llm_provider: Optional[str],
|
||||
init_response_obj: Union[Any, BaseModel, dict],
|
||||
api_base: Optional[str] = None,
|
||||
) -> StandardLoggingModelInformation:
|
||||
model_cost_name = _select_model_name_for_cost_calc(
|
||||
model=None,
|
||||
@@ -4723,7 +4724,9 @@ class StandardLoggingPayloadSetup:
|
||||
else:
|
||||
try:
|
||||
_model_cost_information = litellm.get_model_info(
|
||||
model=model_cost_name, custom_llm_provider=custom_llm_provider
|
||||
model=model_cost_name,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
api_base=api_base,
|
||||
)
|
||||
model_cost_information = StandardLoggingModelInformation(
|
||||
model_map_key=model_cost_name,
|
||||
@@ -5236,6 +5239,7 @@ def get_standard_logging_object_payload(
|
||||
custom_pricing=custom_pricing,
|
||||
custom_llm_provider=kwargs.get("custom_llm_provider"),
|
||||
init_response_obj=init_response_obj,
|
||||
api_base=litellm_params.get("api_base"),
|
||||
)
|
||||
response_cost: float = kwargs.get("response_cost", 0) or 0.0
|
||||
|
||||
|
||||
@@ -200,8 +200,14 @@ def _get_token_base_cost(
|
||||
## CHECK IF ABOVE THRESHOLD
|
||||
# Optimization: collect threshold keys first to avoid sorting all model_info keys.
|
||||
# Most models don't have threshold pricing, so we can return early.
|
||||
# Exclude service_tier-specific variants (e.g. input_cost_per_token_above_200k_tokens_priority)
|
||||
# so that the threshold detection loop only processes standard keys. The
|
||||
# service_tier-specific above-threshold key is resolved later via _get_service_tier_cost_key.
|
||||
threshold_keys = [
|
||||
k for k in model_info if k.startswith("input_cost_per_token_above_")
|
||||
k
|
||||
for k in model_info
|
||||
if k.startswith("input_cost_per_token_above_")
|
||||
and not any(k.endswith(f"_{st.value}") for st in ServiceTier)
|
||||
]
|
||||
if not threshold_keys:
|
||||
return (
|
||||
@@ -224,14 +230,34 @@ def _get_token_base_cost(
|
||||
1000 if "k" in threshold_str else 1
|
||||
)
|
||||
if usage.prompt_tokens > threshold:
|
||||
# Prefer a service_tier-specific above-threshold key when available,
|
||||
# e.g. input_cost_per_token_priority_above_200k_tokens for Gemini
|
||||
# ON_DEMAND_PRIORITY. Falls back to the standard key automatically
|
||||
# via _get_cost_per_unit's service_tier fallback logic.
|
||||
tiered_input_key = (
|
||||
_get_service_tier_cost_key(
|
||||
f"input_cost_per_token_above_{threshold_str}_tokens",
|
||||
service_tier,
|
||||
)
|
||||
if service_tier
|
||||
else key
|
||||
)
|
||||
prompt_base_cost = cast(
|
||||
float, _get_cost_per_unit(model_info, key, prompt_base_cost)
|
||||
float, _get_cost_per_unit(model_info, tiered_input_key, prompt_base_cost)
|
||||
)
|
||||
tiered_output_key = (
|
||||
_get_service_tier_cost_key(
|
||||
f"output_cost_per_token_above_{threshold_str}_tokens",
|
||||
service_tier,
|
||||
)
|
||||
if service_tier
|
||||
else f"output_cost_per_token_above_{threshold_str}_tokens"
|
||||
)
|
||||
completion_base_cost = cast(
|
||||
float,
|
||||
_get_cost_per_unit(
|
||||
model_info,
|
||||
f"output_cost_per_token_above_{threshold_str}_tokens",
|
||||
tiered_output_key,
|
||||
completion_base_cost,
|
||||
),
|
||||
)
|
||||
@@ -517,6 +543,7 @@ def _calculate_input_cost(
|
||||
cache_read_cost: float,
|
||||
cache_creation_cost: float,
|
||||
cache_creation_cost_above_1hr: float,
|
||||
service_tier: Optional[str] = None,
|
||||
) -> float:
|
||||
"""
|
||||
Calculates the input cost for a given model, prompt tokens, and completion tokens.
|
||||
@@ -528,8 +555,11 @@ def _calculate_input_cost(
|
||||
|
||||
### AUDIO COST
|
||||
if prompt_tokens_details["audio_tokens"]:
|
||||
audio_cost_key = _get_service_tier_cost_key(
|
||||
"input_cost_per_audio_token", service_tier
|
||||
)
|
||||
prompt_cost += calculate_cost_component(
|
||||
model_info, "input_cost_per_audio_token", prompt_tokens_details["audio_tokens"]
|
||||
model_info, audio_cost_key, prompt_tokens_details["audio_tokens"]
|
||||
)
|
||||
|
||||
### IMAGE TOKEN COST
|
||||
@@ -659,6 +689,7 @@ def generic_cost_per_token( # noqa: PLR0915
|
||||
cache_read_cost=cache_read_cost,
|
||||
cache_creation_cost=cache_creation_cost,
|
||||
cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
|
||||
service_tier=service_tier,
|
||||
)
|
||||
|
||||
## CALCULATE OUTPUT COST
|
||||
|
||||
@@ -6,7 +6,17 @@ import logging
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from typing import Any, Callable, Dict, List, Optional, Union, cast
|
||||
from typing import (
|
||||
Any,
|
||||
AsyncIterator,
|
||||
Callable,
|
||||
Dict,
|
||||
Iterator,
|
||||
List,
|
||||
Optional,
|
||||
Union,
|
||||
cast,
|
||||
)
|
||||
|
||||
import anyio
|
||||
import httpx
|
||||
@@ -151,10 +161,10 @@ class CustomStreamWrapper:
|
||||
self.is_function_call = self.check_is_function_call(logging_obj=logging_obj)
|
||||
self.created: Optional[int] = None
|
||||
|
||||
def __iter__(self):
|
||||
def __iter__(self) -> Iterator["ModelResponseStream"]:
|
||||
return self
|
||||
|
||||
def __aiter__(self):
|
||||
def __aiter__(self) -> AsyncIterator["ModelResponseStream"]:
|
||||
return self
|
||||
|
||||
async def aclose(self):
|
||||
@@ -1726,7 +1736,7 @@ class CustomStreamWrapper:
|
||||
model_response.choices[0].finish_reason = "tool_calls"
|
||||
return model_response
|
||||
|
||||
def __next__(self): # noqa: PLR0915
|
||||
def __next__(self) -> "ModelResponseStream": # noqa: PLR0915
|
||||
cache_hit = False
|
||||
if (
|
||||
self.custom_llm_provider is not None
|
||||
@@ -1748,7 +1758,7 @@ class CustomStreamWrapper:
|
||||
chunk = next(self.completion_stream)
|
||||
if chunk is not None and chunk != b"":
|
||||
print_verbose(
|
||||
f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk}; custom_llm_provider: {self.custom_llm_provider}"
|
||||
f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk.decode('utf-8', errors='replace') if isinstance(chunk, bytes) else chunk}; custom_llm_provider: {self.custom_llm_provider}"
|
||||
)
|
||||
response: Optional[ModelResponseStream] = self.chunk_creator(
|
||||
chunk=chunk
|
||||
@@ -1900,7 +1910,7 @@ class CustomStreamWrapper:
|
||||
|
||||
return self.completion_stream
|
||||
|
||||
async def __anext__(self): # noqa: PLR0915
|
||||
async def __anext__(self) -> "ModelResponseStream": # noqa: PLR0915
|
||||
cache_hit = False
|
||||
if (
|
||||
self.custom_llm_provider is not None
|
||||
@@ -1996,9 +2006,7 @@ class CustomStreamWrapper:
|
||||
else:
|
||||
chunk = next(self.completion_stream)
|
||||
if chunk is not None and chunk != b"":
|
||||
processed_chunk: Optional[
|
||||
ModelResponseStream
|
||||
] = self.chunk_creator(chunk=chunk)
|
||||
processed_chunk = self.chunk_creator(chunk=chunk)
|
||||
if processed_chunk is None:
|
||||
continue
|
||||
|
||||
|
||||
@@ -5,10 +5,50 @@ Helper util for handling anthropic-specific cost calculation
|
||||
|
||||
from typing import TYPE_CHECKING, Optional, Tuple
|
||||
|
||||
from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
|
||||
from litellm.litellm_core_utils.llm_cost_calc.utils import (
|
||||
_get_token_base_cost,
|
||||
_parse_prompt_tokens_details,
|
||||
calculate_cache_writing_cost,
|
||||
generic_cost_per_token,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.types.utils import ModelInfo, Usage
|
||||
import litellm
|
||||
|
||||
|
||||
def _compute_cache_only_cost(model_info: "ModelInfo", usage: "Usage") -> float:
|
||||
"""
|
||||
Return only the cache-related portion of the prompt cost (cache read + cache write).
|
||||
|
||||
These costs must NOT be scaled by geo/speed multipliers because the old
|
||||
explicit ``fast/`` model entries carried unchanged cache rates while
|
||||
multiplying only the regular input/output token costs.
|
||||
"""
|
||||
if usage.prompt_tokens_details is None:
|
||||
return 0.0
|
||||
|
||||
prompt_tokens_details = _parse_prompt_tokens_details(usage)
|
||||
_, _, cache_creation_cost, cache_creation_cost_above_1hr, cache_read_cost = (
|
||||
_get_token_base_cost(model_info=model_info, usage=usage)
|
||||
)
|
||||
|
||||
cache_cost = float(prompt_tokens_details["cache_hit_tokens"]) * cache_read_cost
|
||||
|
||||
if (
|
||||
prompt_tokens_details["cache_creation_tokens"]
|
||||
or prompt_tokens_details["cache_creation_token_details"] is not None
|
||||
):
|
||||
cache_cost += calculate_cache_writing_cost(
|
||||
cache_creation_tokens=prompt_tokens_details["cache_creation_tokens"],
|
||||
cache_creation_token_details=prompt_tokens_details[
|
||||
"cache_creation_token_details"
|
||||
],
|
||||
cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
|
||||
cache_creation_cost=cache_creation_cost,
|
||||
)
|
||||
|
||||
return cache_cost
|
||||
|
||||
|
||||
def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
|
||||
@@ -22,20 +62,34 @@ def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
|
||||
Returns:
|
||||
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
|
||||
"""
|
||||
model_with_prefix = model
|
||||
|
||||
# First, prepend inference_geo if present
|
||||
if hasattr(usage, "inference_geo") and usage.inference_geo and usage.inference_geo.lower() not in ["global", "not_available"]:
|
||||
model_with_prefix = f"{usage.inference_geo}/{model_with_prefix}"
|
||||
|
||||
# Then, prepend speed if it's "fast"
|
||||
if hasattr(usage, "speed") and usage.speed == "fast":
|
||||
model_with_prefix = f"fast/{model_with_prefix}"
|
||||
|
||||
prompt_cost, completion_cost = generic_cost_per_token(
|
||||
model=model_with_prefix, usage=usage, custom_llm_provider="anthropic"
|
||||
model=model, usage=usage, custom_llm_provider="anthropic"
|
||||
)
|
||||
|
||||
# Apply provider_specific_entry multipliers for geo/speed routing
|
||||
try:
|
||||
model_info = litellm.get_model_info(model=model, custom_llm_provider="anthropic")
|
||||
provider_specific_entry: dict = model_info.get("provider_specific_entry") or {}
|
||||
|
||||
multiplier = 1.0
|
||||
if (
|
||||
hasattr(usage, "inference_geo")
|
||||
and usage.inference_geo
|
||||
and usage.inference_geo.lower() not in ["global", "not_available"]
|
||||
):
|
||||
multiplier *= provider_specific_entry.get(
|
||||
usage.inference_geo.lower(), 1.0
|
||||
)
|
||||
if hasattr(usage, "speed") and usage.speed == "fast":
|
||||
multiplier *= provider_specific_entry.get("fast", 1.0)
|
||||
|
||||
if multiplier != 1.0:
|
||||
cache_cost = _compute_cache_only_cost(model_info=model_info, usage=usage)
|
||||
prompt_cost = (prompt_cost - cache_cost) * multiplier + cache_cost
|
||||
completion_cost *= multiplier
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return prompt_cost, completion_cost
|
||||
|
||||
|
||||
|
||||
@@ -118,10 +118,11 @@ class BaseVideoConfig(ABC):
|
||||
api_base: str,
|
||||
litellm_params: GenericLiteLLMParams,
|
||||
headers: dict,
|
||||
variant: Optional[str] = None,
|
||||
) -> Tuple[str, Dict]:
|
||||
"""
|
||||
Transform the video content request into a URL and data/params
|
||||
|
||||
|
||||
Returns:
|
||||
Tuple[str, Dict]: (url, params) for the video content request
|
||||
"""
|
||||
|
||||
@@ -202,52 +202,84 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
|
||||
return optional_params
|
||||
|
||||
|
||||
# Providers whose InvokeModel body uses the Converse API format
|
||||
# (messages + inferenceConfig + image blocks). Nova is the primary
|
||||
# example; add others here as they adopt the same schema.
|
||||
CONVERSE_INVOKE_PROVIDERS = ("nova",)
|
||||
|
||||
def _map_openai_to_bedrock_params(
|
||||
self,
|
||||
openai_request_body: Dict[str, Any],
|
||||
provider: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Transform OpenAI request body to Bedrock-compatible modelInput parameters using existing transformation logic
|
||||
Transform OpenAI request body to Bedrock-compatible modelInput
|
||||
parameters using existing transformation logic.
|
||||
|
||||
Routes to the correct per-provider transformation so that the
|
||||
resulting dict matches the InvokeModel body that Bedrock expects
|
||||
for batch inference.
|
||||
"""
|
||||
from litellm.types.utils import LlmProviders
|
||||
|
||||
_model = openai_request_body.get("model", "")
|
||||
messages = openai_request_body.get("messages", [])
|
||||
|
||||
# Use existing Anthropic transformation logic for Anthropic models
|
||||
optional_params = {
|
||||
k: v
|
||||
for k, v in openai_request_body.items()
|
||||
if k not in ["model", "messages"]
|
||||
}
|
||||
|
||||
# --- Anthropic: use existing AmazonAnthropicClaudeConfig ---
|
||||
if provider == LlmProviders.ANTHROPIC:
|
||||
from litellm.llms.bedrock.chat.invoke_transformations.anthropic_claude3_transformation import (
|
||||
AmazonAnthropicClaudeConfig,
|
||||
)
|
||||
|
||||
anthropic_config = AmazonAnthropicClaudeConfig()
|
||||
|
||||
# Extract optional params (everything except model and messages)
|
||||
optional_params = {k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
|
||||
mapped_params = anthropic_config.map_openai_params(
|
||||
|
||||
config = AmazonAnthropicClaudeConfig()
|
||||
mapped_params = config.map_openai_params(
|
||||
non_default_params={},
|
||||
optional_params=optional_params,
|
||||
model=_model,
|
||||
drop_params=False
|
||||
drop_params=False,
|
||||
)
|
||||
|
||||
# Transform using existing Anthropic logic
|
||||
bedrock_params = anthropic_config.transform_request(
|
||||
return config.transform_request(
|
||||
model=_model,
|
||||
messages=messages,
|
||||
optional_params=mapped_params,
|
||||
litellm_params={},
|
||||
headers={}
|
||||
headers={},
|
||||
)
|
||||
|
||||
return bedrock_params
|
||||
else:
|
||||
# For other providers, use basic mapping
|
||||
bedrock_params = {
|
||||
"messages": messages,
|
||||
**{k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
|
||||
}
|
||||
return bedrock_params
|
||||
# --- Converse API providers (e.g. Nova): use AmazonConverseConfig
|
||||
# to correctly convert image_url blocks to Bedrock image format
|
||||
# and wrap inference params inside inferenceConfig. ---
|
||||
if provider in self.CONVERSE_INVOKE_PROVIDERS:
|
||||
from litellm.llms.bedrock.chat.converse_transformation import (
|
||||
AmazonConverseConfig,
|
||||
)
|
||||
|
||||
converse_config = AmazonConverseConfig()
|
||||
mapped_params = converse_config.map_openai_params(
|
||||
non_default_params=optional_params,
|
||||
optional_params={},
|
||||
model=_model,
|
||||
drop_params=False,
|
||||
)
|
||||
return converse_config.transform_request(
|
||||
model=_model,
|
||||
messages=messages,
|
||||
optional_params=mapped_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
# --- All other providers: passthrough (OpenAI-compatible models
|
||||
# like openai.gpt-oss-*, qwen, deepseek, etc.) ---
|
||||
return {
|
||||
"messages": messages,
|
||||
**optional_params,
|
||||
}
|
||||
|
||||
def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
|
||||
self, openai_jsonl_content: List[Dict[str, Any]]
|
||||
|
||||
@@ -3014,8 +3014,11 @@ class BaseLLMHTTPHandler:
|
||||
raise ValueError(f"Unsupported transformed_request type: {type(transformed_request)}")
|
||||
|
||||
# Store the upload URL in litellm_params for the transformation method
|
||||
# Honour the URL already set by transform_create_file_request (e.g. Bedrock pre-signed S3 uploads),
|
||||
# fall back to api_base for providers that do not set it.
|
||||
litellm_params_with_url = dict(litellm_params)
|
||||
litellm_params_with_url["upload_url"] = api_base
|
||||
if "upload_url" not in litellm_params:
|
||||
litellm_params_with_url["upload_url"] = api_base
|
||||
|
||||
return provider_config.transform_create_file_response(
|
||||
model=None,
|
||||
@@ -5397,6 +5400,7 @@ class BaseLLMHTTPHandler:
|
||||
api_key: Optional[str] = None,
|
||||
client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
|
||||
_is_async: bool = False,
|
||||
variant: Optional[str] = None,
|
||||
) -> Union[bytes, Coroutine[Any, Any, bytes]]:
|
||||
"""
|
||||
Handle video content download requests.
|
||||
@@ -5412,6 +5416,7 @@ class BaseLLMHTTPHandler:
|
||||
extra_headers=extra_headers,
|
||||
api_key=api_key,
|
||||
client=client,
|
||||
variant=variant,
|
||||
)
|
||||
|
||||
if client is None or not isinstance(client, HTTPHandler):
|
||||
@@ -5443,6 +5448,7 @@ class BaseLLMHTTPHandler:
|
||||
api_base=api_base,
|
||||
litellm_params=litellm_params,
|
||||
headers=headers,
|
||||
variant=variant,
|
||||
)
|
||||
|
||||
try:
|
||||
@@ -5485,6 +5491,7 @@ class BaseLLMHTTPHandler:
|
||||
extra_headers: Optional[Dict[str, Any]] = None,
|
||||
api_key: Optional[str] = None,
|
||||
client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
|
||||
variant: Optional[str] = None,
|
||||
) -> bytes:
|
||||
"""
|
||||
Async version of the video content download handler.
|
||||
@@ -5519,6 +5526,7 @@ class BaseLLMHTTPHandler:
|
||||
api_base=api_base,
|
||||
litellm_params=litellm_params,
|
||||
headers=headers,
|
||||
variant=variant,
|
||||
)
|
||||
|
||||
try:
|
||||
@@ -5594,7 +5602,7 @@ class BaseLLMHTTPHandler:
|
||||
sync_httpx_client = client
|
||||
|
||||
headers = video_remix_provider_config.validate_environment(
|
||||
api_key=api_key,
|
||||
api_key=api_key or litellm_params.get("api_key", None),
|
||||
headers=extra_headers or {},
|
||||
model="",
|
||||
)
|
||||
@@ -5676,7 +5684,7 @@ class BaseLLMHTTPHandler:
|
||||
async_httpx_client = client
|
||||
|
||||
headers = video_remix_provider_config.validate_environment(
|
||||
api_key=api_key,
|
||||
api_key=api_key or litellm_params.get("api_key", None),
|
||||
headers=extra_headers or {},
|
||||
model="",
|
||||
)
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
"""
|
||||
Mock httpx transport that returns valid OpenAI ChatCompletion responses.
|
||||
|
||||
Activated via `litellm_settings: { network_mock: true }`.
|
||||
Intercepts at the httpx transport layer — the lowest point before bytes hit the wire —
|
||||
so the full proxy -> router -> OpenAI SDK -> httpx path is exercised.
|
||||
"""
|
||||
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
from typing import Tuple
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-built response templates
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _mock_id() -> str:
|
||||
return f"chatcmpl-mock-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
def _chat_completion_json(model: str) -> dict:
|
||||
"""Return a minimal valid ChatCompletion object."""
|
||||
return {
|
||||
"id": _mock_id(),
|
||||
"object": "chat.completion",
|
||||
"created": int(time.time()),
|
||||
"model": model,
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "Mock response",
|
||||
},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 1,
|
||||
"completion_tokens": 1,
|
||||
"total_tokens": 2,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Transport
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_JSON_HEADERS = {
|
||||
"content-type": "application/json",
|
||||
}
|
||||
|
||||
|
||||
class MockOpenAITransport(httpx.AsyncBaseTransport, httpx.BaseTransport):
|
||||
"""
|
||||
httpx transport that returns canned OpenAI ChatCompletion responses.
|
||||
|
||||
Supports both async (AsyncOpenAI) and sync (OpenAI) SDK paths.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _parse_request(request: httpx.Request) -> Tuple[str, bool]:
|
||||
"""Extract model from the request body."""
|
||||
try:
|
||||
body = json.loads(request.content)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return ("mock-model", False)
|
||||
model = body.get("model", "mock-model")
|
||||
return (model, False)
|
||||
|
||||
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
|
||||
model, _ = self._parse_request(request)
|
||||
body = json.dumps(_chat_completion_json(model)).encode()
|
||||
return httpx.Response(
|
||||
status_code=200,
|
||||
headers=_JSON_HEADERS,
|
||||
content=body,
|
||||
)
|
||||
|
||||
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||
model, _ = self._parse_request(request)
|
||||
body = json.dumps(_chat_completion_json(model)).encode()
|
||||
return httpx.Response(
|
||||
status_code=200,
|
||||
headers=_JSON_HEADERS,
|
||||
content=body,
|
||||
)
|
||||
@@ -4,13 +4,15 @@ This file is used to calculate the cost of the Gemini API.
|
||||
Handles the context caching for Gemini API.
|
||||
"""
|
||||
|
||||
from typing import TYPE_CHECKING, Tuple
|
||||
from typing import TYPE_CHECKING, Optional, Tuple
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.types.utils import ModelInfo, Usage
|
||||
|
||||
|
||||
def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
|
||||
def cost_per_token(
|
||||
model: str, usage: "Usage", service_tier: Optional[str] = None
|
||||
) -> Tuple[float, float]:
|
||||
"""
|
||||
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
|
||||
|
||||
@@ -19,7 +21,7 @@ def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
|
||||
from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
|
||||
|
||||
return generic_cost_per_token(
|
||||
model=model, usage=usage, custom_llm_provider="gemini"
|
||||
model=model, usage=usage, custom_llm_provider="gemini", service_tier=service_tier
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -393,10 +393,11 @@ class GeminiVideoConfig(BaseVideoConfig):
|
||||
api_base: str,
|
||||
litellm_params: GenericLiteLLMParams,
|
||||
headers: dict,
|
||||
variant: Optional[str] = None,
|
||||
) -> Tuple[str, Dict]:
|
||||
"""
|
||||
Transform the video content request for Veo API.
|
||||
|
||||
|
||||
For Veo, we need to:
|
||||
1. Get operation status to extract video URI
|
||||
2. Return download URL for the video
|
||||
|
||||
@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, Any, AsyncIterator, Iterator, List, Optional,
|
||||
from httpx._models import Headers, Response
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm._logging import verbose_logger, verbose_proxy_logger
|
||||
from litellm.litellm_core_utils.prompt_templates.common_utils import (
|
||||
get_str_from_messages,
|
||||
)
|
||||
@@ -223,7 +223,9 @@ class OllamaConfig(BaseConfig):
|
||||
or get_secret_str("OLLAMA_API_KEY")
|
||||
)
|
||||
|
||||
def get_model_info(self, model: str) -> ModelInfoBase:
|
||||
def get_model_info(
|
||||
self, model: str, api_base: Optional[str] = None
|
||||
) -> ModelInfoBase:
|
||||
"""
|
||||
curl http://localhost:11434/api/show -d '{
|
||||
"name": "mistral"
|
||||
@@ -231,7 +233,11 @@ class OllamaConfig(BaseConfig):
|
||||
"""
|
||||
if model.startswith("ollama/") or model.startswith("ollama_chat/"):
|
||||
model = model.split("/", 1)[1]
|
||||
api_base = get_secret_str("OLLAMA_API_BASE") or "http://localhost:11434"
|
||||
api_base = (
|
||||
api_base
|
||||
or get_secret_str("OLLAMA_API_BASE")
|
||||
or "http://localhost:11434"
|
||||
)
|
||||
api_key = self.get_api_key()
|
||||
headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
|
||||
|
||||
@@ -242,8 +248,21 @@ class OllamaConfig(BaseConfig):
|
||||
headers=headers,
|
||||
)
|
||||
except Exception as e:
|
||||
raise Exception(
|
||||
f"OllamaError: Error getting model info for {model}. Set Ollama API Base via `OLLAMA_API_BASE` environment variable. Error: {e}"
|
||||
verbose_logger.debug(
|
||||
"OllamaError: Could not get model info for %s from %s. Error: %s",
|
||||
model,
|
||||
api_base,
|
||||
e,
|
||||
)
|
||||
return ModelInfoBase(
|
||||
key=model,
|
||||
litellm_provider="ollama",
|
||||
mode="chat",
|
||||
input_cost_per_token=0.0,
|
||||
output_cost_per_token=0.0,
|
||||
max_tokens=None,
|
||||
max_input_tokens=None,
|
||||
max_output_tokens=None,
|
||||
)
|
||||
|
||||
model_info = response.json()
|
||||
|
||||
@@ -205,6 +205,11 @@ class BaseOpenAILLM:
|
||||
if litellm.aclient_session is not None:
|
||||
return litellm.aclient_session
|
||||
|
||||
if getattr(litellm, "network_mock", False):
|
||||
from litellm.llms.custom_httpx.mock_transport import MockOpenAITransport
|
||||
|
||||
return httpx.AsyncClient(transport=MockOpenAITransport())
|
||||
|
||||
# Get unified SSL configuration
|
||||
ssl_config = get_ssl_configuration()
|
||||
|
||||
@@ -225,6 +230,11 @@ class BaseOpenAILLM:
|
||||
if litellm.client_session is not None:
|
||||
return litellm.client_session
|
||||
|
||||
if getattr(litellm, "network_mock", False):
|
||||
from litellm.llms.custom_httpx.mock_transport import MockOpenAITransport
|
||||
|
||||
return httpx.Client(transport=MockOpenAITransport())
|
||||
|
||||
# Get unified SSL configuration
|
||||
ssl_config = get_ssl_configuration()
|
||||
|
||||
|
||||
@@ -172,18 +172,22 @@ class OpenAIVideoConfig(BaseVideoConfig):
|
||||
api_base: str,
|
||||
litellm_params: GenericLiteLLMParams,
|
||||
headers: dict,
|
||||
variant: Optional[str] = None,
|
||||
) -> Tuple[str, Dict]:
|
||||
"""
|
||||
Transform the video content request for OpenAI API.
|
||||
|
||||
|
||||
OpenAI API expects the following request:
|
||||
- GET /v1/videos/{video_id}/content
|
||||
- GET /v1/videos/{video_id}/content?variant=thumbnail
|
||||
"""
|
||||
original_video_id = extract_original_video_id(video_id)
|
||||
|
||||
|
||||
# Construct the URL for video content download
|
||||
url = f"{api_base.rstrip('/')}/{original_video_id}/content"
|
||||
|
||||
if variant is not None:
|
||||
url = f"{url}?variant={variant}"
|
||||
|
||||
# No additional data needed for GET content request
|
||||
data: Dict[str, Any] = {}
|
||||
|
||||
|
||||
@@ -310,10 +310,11 @@ class RunwayMLVideoConfig(BaseVideoConfig):
|
||||
api_base: str,
|
||||
litellm_params: GenericLiteLLMParams,
|
||||
headers: dict,
|
||||
variant: Optional[str] = None,
|
||||
) -> Tuple[str, Dict]:
|
||||
"""
|
||||
Transform the video content request for RunwayML API.
|
||||
|
||||
|
||||
RunwayML doesn't have a separate content download endpoint.
|
||||
The video URL is returned in the task output field.
|
||||
We'll retrieve the task and extract the video URL.
|
||||
|
||||
@@ -224,6 +224,7 @@ def cost_per_token(
|
||||
model: str,
|
||||
custom_llm_provider: str,
|
||||
usage: Usage,
|
||||
service_tier: Optional[str] = None,
|
||||
) -> Tuple[float, float]:
|
||||
"""
|
||||
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
|
||||
@@ -233,6 +234,8 @@ def cost_per_token(
|
||||
- custom_llm_provider: str, either "vertex_ai-*" or "gemini"
|
||||
- prompt_tokens: float, the number of input tokens
|
||||
- completion_tokens: float, the number of output tokens
|
||||
- service_tier: optional tier derived from Gemini trafficType
|
||||
("priority" for ON_DEMAND_PRIORITY, "flex" for FLEX/batch).
|
||||
|
||||
Returns:
|
||||
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
|
||||
@@ -266,4 +269,5 @@ def cost_per_token(
|
||||
model=model,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
usage=usage,
|
||||
service_tier=service_tier,
|
||||
)
|
||||
|
||||
@@ -455,6 +455,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
|
||||
api_base: str,
|
||||
litellm_params: GenericLiteLLMParams,
|
||||
headers: dict,
|
||||
variant: Optional[str] = None,
|
||||
) -> Tuple[str, Dict]:
|
||||
"""
|
||||
Transform the video content request for Veo API.
|
||||
|
||||
@@ -147,6 +147,7 @@ from litellm.utils import (
|
||||
token_counter,
|
||||
validate_and_fix_openai_messages,
|
||||
validate_and_fix_openai_tools,
|
||||
validate_and_fix_thinking_param,
|
||||
validate_chat_completion_tool_choice,
|
||||
validate_openai_optional_params,
|
||||
)
|
||||
@@ -1103,6 +1104,8 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
tool_choice = validate_chat_completion_tool_choice(tool_choice=tool_choice)
|
||||
# validate optional params
|
||||
stop = validate_openai_optional_params(stop=stop)
|
||||
# normalize camelCase thinking keys (e.g. budgetTokens -> budget_tokens)
|
||||
thinking = validate_and_fix_thinking_param(thinking=thinking)
|
||||
|
||||
######### unpacking kwargs #####################
|
||||
args = locals()
|
||||
|
||||
@@ -8295,37 +8295,6 @@
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"us/claude-sonnet-4-6": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
|
||||
"input_cost_per_token": 3.3e-06,
|
||||
"input_cost_per_token_above_200k_tokens": 6.6e-06,
|
||||
"litellm_provider": "anthropic",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 64000,
|
||||
"max_tokens": 64000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.65e-05,
|
||||
"output_cost_per_token_above_200k_tokens": 2.475e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346,
|
||||
"inference_geo": "us"
|
||||
},
|
||||
"claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 3.75e-06,
|
||||
"cache_read_input_token_cost": 3e-07,
|
||||
@@ -8517,100 +8486,11 @@
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"fast/claude-opus-4-6": {
|
||||
"cache_creation_input_token_cost": 6.25e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 1.25e-05,
|
||||
"cache_creation_input_token_cost_above_1hr": 1e-05,
|
||||
"cache_read_input_token_cost": 5e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
|
||||
"input_cost_per_token": 3e-05,
|
||||
"input_cost_per_token_above_200k_tokens": 1e-05,
|
||||
"litellm_provider": "anthropic",
|
||||
"max_input_tokens": 1000000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 0.00015,
|
||||
"output_cost_per_token_above_200k_tokens": 3.75e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"us/claude-opus-4-6": {
|
||||
"cache_creation_input_token_cost": 6.875e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 1.375e-05,
|
||||
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
|
||||
"cache_read_input_token_cost": 5.5e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens": 1.1e-06,
|
||||
"input_cost_per_token": 5.5e-06,
|
||||
"input_cost_per_token_above_200k_tokens": 1.1e-05,
|
||||
"litellm_provider": "anthropic",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.75e-05,
|
||||
"output_cost_per_token_above_200k_tokens": 4.125e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"fast/us/claude-opus-4-6": {
|
||||
"cache_creation_input_token_cost": 6.875e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 1.375e-05,
|
||||
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
|
||||
"cache_read_input_token_cost": 5.5e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens": 1.1e-06,
|
||||
"input_cost_per_token": 3e-05,
|
||||
"input_cost_per_token_above_200k_tokens": 1.1e-05,
|
||||
"litellm_provider": "anthropic",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 0.00015,
|
||||
"output_cost_per_token_above_200k_tokens": 4.125e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
"tool_use_system_prompt_tokens": 346,
|
||||
"provider_specific_entry": {
|
||||
"us": 1.1,
|
||||
"fast": 6.0
|
||||
}
|
||||
},
|
||||
"claude-opus-4-6-20260205": {
|
||||
"cache_creation_input_token_cost": 6.25e-06,
|
||||
@@ -8641,69 +8521,11 @@
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"fast/claude-opus-4-6-20260205": {
|
||||
"cache_creation_input_token_cost": 6.25e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 1.25e-05,
|
||||
"cache_creation_input_token_cost_above_1hr": 1e-05,
|
||||
"cache_read_input_token_cost": 5e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
|
||||
"input_cost_per_token": 3e-05,
|
||||
"input_cost_per_token_above_200k_tokens": 1e-05,
|
||||
"litellm_provider": "anthropic",
|
||||
"max_input_tokens": 1000000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 0.00015,
|
||||
"output_cost_per_token_above_200k_tokens": 3.75e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"us/claude-opus-4-6-20260205": {
|
||||
"cache_creation_input_token_cost": 6.875e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 1.375e-05,
|
||||
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
|
||||
"cache_read_input_token_cost": 5.5e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens": 1.1e-06,
|
||||
"input_cost_per_token": 5.5e-06,
|
||||
"input_cost_per_token_above_200k_tokens": 1.1e-05,
|
||||
"litellm_provider": "anthropic",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.75e-05,
|
||||
"output_cost_per_token_above_200k_tokens": 4.125e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
"tool_use_system_prompt_tokens": 346,
|
||||
"provider_specific_entry": {
|
||||
"us": 1.1,
|
||||
"fast": 6.0
|
||||
}
|
||||
},
|
||||
"claude-sonnet-4-20250514": {
|
||||
"deprecation_date": "2026-05-14",
|
||||
@@ -14768,7 +14590,14 @@
|
||||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3.1-pro-preview": {
|
||||
"cache_read_input_token_cost": 2e-07,
|
||||
@@ -14819,7 +14648,14 @@
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_url_context": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3.1-pro-preview-customtools": {
|
||||
"cache_read_input_token_cost": 2e-07,
|
||||
@@ -14919,7 +14755,14 @@
|
||||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"vertex_ai/gemini-3-flash-preview": {
|
||||
"cache_read_input_token_cost": 5e-08,
|
||||
@@ -14963,7 +14806,12 @@
|
||||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 9e-07,
|
||||
"input_cost_per_audio_token_priority": 1.8e-06,
|
||||
"output_cost_per_token_priority": 5.4e-06,
|
||||
"cache_read_input_token_cost_priority": 9e-08,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"vertex_ai/gemini-3.1-pro-preview": {
|
||||
"cache_read_input_token_cost": 2e-07,
|
||||
@@ -15014,7 +14862,14 @@
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_url_context": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"vertex_ai/gemini-3.1-pro-preview-customtools": {
|
||||
"cache_read_input_token_cost": 2e-07,
|
||||
@@ -15065,7 +14920,14 @@
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_url_context": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-2.5-pro-exp-03-25": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
@@ -16860,6 +16722,8 @@
|
||||
"cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
|
||||
"input_cost_per_token": 1.25e-06,
|
||||
"input_cost_per_token_above_200k_tokens": 2.5e-06,
|
||||
"input_cost_per_token_priority": 1.25e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 2.5e-06,
|
||||
"litellm_provider": "gemini",
|
||||
"max_audio_length_hours": 8.4,
|
||||
"max_audio_per_prompt": 1,
|
||||
@@ -16873,8 +16737,11 @@
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1e-05,
|
||||
"output_cost_per_token_above_200k_tokens": 1.5e-05,
|
||||
"output_cost_per_token_priority": 1e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 1.5e-05,
|
||||
"rpm": 2000,
|
||||
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
|
||||
"supports_service_tier": true,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/completions"
|
||||
@@ -16979,7 +16846,14 @@
|
||||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 800000
|
||||
"tpm": 800000,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3-flash-preview": {
|
||||
"cache_read_input_token_cost": 5e-08,
|
||||
@@ -17027,7 +16901,12 @@
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true,
|
||||
"tpm": 800000
|
||||
"tpm": 800000,
|
||||
"input_cost_per_token_priority": 9e-07,
|
||||
"input_cost_per_audio_token_priority": 1.8e-06,
|
||||
"output_cost_per_token_priority": 5.4e-06,
|
||||
"cache_read_input_token_cost_priority": 9e-08,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3.1-pro-preview": {
|
||||
"cache_read_input_token_cost": 2e-07,
|
||||
@@ -17078,7 +16957,14 @@
|
||||
"supports_web_search": true,
|
||||
"supports_url_context": true,
|
||||
"supports_native_streaming": true,
|
||||
"tpm": 800000
|
||||
"tpm": 800000,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3.1-pro-preview-customtools": {
|
||||
"cache_read_input_token_cost": 2e-07,
|
||||
@@ -17129,7 +17015,14 @@
|
||||
"supports_web_search": true,
|
||||
"supports_url_context": true,
|
||||
"supports_native_streaming": true,
|
||||
"tpm": 800000
|
||||
"tpm": 800000,
|
||||
"input_cost_per_token_priority": 3.6e-06,
|
||||
"input_cost_per_token_above_200k_tokens_priority": 7.2e-06,
|
||||
"output_cost_per_token_priority": 2.16e-05,
|
||||
"output_cost_per_token_above_200k_tokens_priority": 3.24e-05,
|
||||
"cache_read_input_token_cost_priority": 3.6e-07,
|
||||
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3-flash-preview": {
|
||||
"cache_read_input_token_cost": 5e-08,
|
||||
@@ -17175,7 +17068,12 @@
|
||||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"input_cost_per_token_priority": 9e-07,
|
||||
"input_cost_per_audio_token_priority": 1.8e-06,
|
||||
"output_cost_per_token_priority": 5.4e-06,
|
||||
"cache_read_input_token_cost_priority": 9e-08,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-2.5-pro-exp-03-25": {
|
||||
"cache_read_input_token_cost": 0.0,
|
||||
@@ -37749,4 +37647,4 @@
|
||||
"notes": "DuckDuckGo Instant Answer API is free and does not require an API key."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2454,5 +2454,367 @@
|
||||
"Injection Protection"
|
||||
],
|
||||
"estimated_latency_ms": 1
|
||||
},
|
||||
{
|
||||
"id": "pdpa-singapore",
|
||||
"title": "Singapore PDPA \u2014 Personal Data Protection",
|
||||
"description": "Singapore Personal Data Protection Act (PDPA) compliance. Covers 5 obligation areas: personal identifier collection (s.13 Consent), sensitive data profiling (Advisory Guidelines), Do Not Call Registry violations (Part IX), overseas data transfers (s.26), and automated profiling without human oversight (Model AI Governance Framework). Also includes regex-based PII detection for NRIC/FIN, Singapore phone numbers, postal codes, passports, UEN, and bank account numbers. Zero-cost keyword-based detection.",
|
||||
"icon": "ShieldCheckIcon",
|
||||
"iconColor": "text-red-500",
|
||||
"iconBg": "bg-red-50",
|
||||
"guardrails": [
|
||||
"pdpa-sg-pii-identifiers",
|
||||
"pdpa-sg-contact-information",
|
||||
"pdpa-sg-financial-data",
|
||||
"pdpa-sg-business-identifiers",
|
||||
"pdpa-sg-personal-identifiers",
|
||||
"pdpa-sg-sensitive-data",
|
||||
"pdpa-sg-do-not-call",
|
||||
"pdpa-sg-data-transfer",
|
||||
"pdpa-sg-profiling-automated-decisions"
|
||||
],
|
||||
"complexity": "High",
|
||||
"guardrailDefinitions": [
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-pii-identifiers",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"patterns": [
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "sg_nric",
|
||||
"action": "MASK"
|
||||
},
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "passport_singapore",
|
||||
"action": "MASK"
|
||||
}
|
||||
],
|
||||
"pattern_redaction_format": "[{pattern_name}_REDACTED]"
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Masks Singapore NRIC/FIN and passport numbers for PDPA compliance"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-contact-information",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"patterns": [
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "sg_phone",
|
||||
"action": "MASK"
|
||||
},
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "sg_postal_code",
|
||||
"action": "MASK"
|
||||
},
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "email",
|
||||
"action": "MASK"
|
||||
}
|
||||
],
|
||||
"pattern_redaction_format": "[{pattern_name}_REDACTED]"
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Masks Singapore phone numbers, postal codes, and email addresses"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-financial-data",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"patterns": [
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "sg_bank_account",
|
||||
"action": "MASK"
|
||||
},
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "credit_card",
|
||||
"action": "MASK"
|
||||
}
|
||||
],
|
||||
"pattern_redaction_format": "[{pattern_name}_REDACTED]"
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Masks Singapore bank account numbers and credit card numbers"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-business-identifiers",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"patterns": [
|
||||
{
|
||||
"pattern_type": "prebuilt",
|
||||
"pattern_name": "sg_uen",
|
||||
"action": "MASK"
|
||||
}
|
||||
],
|
||||
"pattern_redaction_format": "[UEN_REDACTED]"
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Masks Singapore Unique Entity Numbers (business registration)"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-personal-identifiers",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_pdpa_personal_identifiers",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_pdpa_personal_identifiers.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "PDPA s.13 \u2014 Blocks unauthorized collection, harvesting, or extraction of Singapore personal identifiers (NRIC/FIN, SingPass, passports)"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-sensitive-data",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_pdpa_sensitive_data",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_pdpa_sensitive_data.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "PDPA Advisory Guidelines \u2014 Blocks profiling or inference of sensitive personal data categories (race, religion, health, politics) for Singapore residents"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-do-not-call",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_pdpa_do_not_call",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_pdpa_do_not_call.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "PDPA Part IX \u2014 Blocks generation of unsolicited marketing lists and DNC Registry bypass attempts for Singapore phone numbers"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-data-transfer",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_pdpa_data_transfer",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_pdpa_data_transfer.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "PDPA s.26 \u2014 Blocks unprotected overseas transfer of Singapore personal data without adequate safeguards"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "pdpa-sg-profiling-automated-decisions",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_pdpa_profiling_automated_decisions",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_pdpa_profiling_automated_decisions.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "PDPA + Model AI Governance Framework \u2014 Blocks automated profiling and decision-making about Singapore residents without human oversight"
|
||||
}
|
||||
}
|
||||
],
|
||||
"templateData": {
|
||||
"policy_name": "pdpa-singapore",
|
||||
"description": "Singapore PDPA compliance policy. Covers personal identifier protection (s.13), sensitive data profiling (Advisory Guidelines), Do Not Call Registry (Part IX), overseas data transfers (s.26), and automated profiling (Model AI Governance Framework). Includes regex-based PII detection for NRIC/FIN, phone numbers, postal codes, passports, UEN, and bank accounts.",
|
||||
"guardrails_add": [
|
||||
"pdpa-sg-pii-identifiers",
|
||||
"pdpa-sg-contact-information",
|
||||
"pdpa-sg-financial-data",
|
||||
"pdpa-sg-business-identifiers",
|
||||
"pdpa-sg-personal-identifiers",
|
||||
"pdpa-sg-sensitive-data",
|
||||
"pdpa-sg-do-not-call",
|
||||
"pdpa-sg-data-transfer",
|
||||
"pdpa-sg-profiling-automated-decisions"
|
||||
],
|
||||
"guardrails_remove": []
|
||||
},
|
||||
"tags": [
|
||||
"PII Protection",
|
||||
"Regulatory",
|
||||
"Singapore"
|
||||
],
|
||||
"estimated_latency_ms": 1
|
||||
},
|
||||
{
|
||||
"id": "mas-ai-risk-management",
|
||||
"title": "Singapore MAS \u2014 AI Risk Management for Financial Institutions",
|
||||
"description": "Monetary Authority of Singapore (MAS) AI Risk Management for Financial Institutions alignment. Covers 5 enforceable obligation areas: fairness & bias in financial decisions, transparency & explainability of AI models, human oversight for consequential actions, data governance for financial customer data, and model security against adversarial attacks. Based on Guidelines on Artificial Intelligence Risk Management (MAS), and aligned with the 2018 FEAT Principles and Project MindForge. Zero-cost keyword-based detection.",
|
||||
"icon": "ShieldCheckIcon",
|
||||
"iconColor": "text-blue-600",
|
||||
"iconBg": "bg-blue-50",
|
||||
"guardrails": [
|
||||
"mas-sg-fairness-bias",
|
||||
"mas-sg-transparency-explainability",
|
||||
"mas-sg-human-oversight",
|
||||
"mas-sg-data-governance",
|
||||
"mas-sg-model-security"
|
||||
],
|
||||
"complexity": "High",
|
||||
"guardrailDefinitions": [
|
||||
{
|
||||
"guardrail_name": "mas-sg-fairness-bias",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_mas_fairness_bias",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_mas_fairness_bias.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Guidelines on Artificial Intelligence Risk Management (MAS) — Blocks discriminatory AI practices in financial services that score, deny, or price based on protected attributes (race, religion, age, gender, nationality)"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "mas-sg-transparency-explainability",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_mas_transparency_explainability",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_mas_transparency_explainability.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Guidelines on Artificial Intelligence Risk Management (MAS) — Blocks deployment of opaque or unexplainable AI systems for consequential financial decisions"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "mas-sg-human-oversight",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_mas_human_oversight",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_mas_human_oversight.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Guidelines on Artificial Intelligence Risk Management (MAS) — Blocks fully automated financial AI decisions without human-in-the-loop for consequential actions (loans, claims, trading)"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "mas-sg-data-governance",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_mas_data_governance",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_mas_data_governance.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Guidelines on Artificial Intelligence Risk Management (MAS) — Blocks unauthorized sharing, exposure, or mishandling of financial customer data without proper governance and data lineage"
|
||||
}
|
||||
},
|
||||
{
|
||||
"guardrail_name": "mas-sg-model-security",
|
||||
"litellm_params": {
|
||||
"guardrail": "litellm_content_filter",
|
||||
"mode": "pre_call",
|
||||
"categories": [
|
||||
{
|
||||
"category": "sg_mas_model_security",
|
||||
"category_file": "litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/policy_templates/sg_mas_model_security.yaml",
|
||||
"enabled": true,
|
||||
"action": "BLOCK",
|
||||
"severity_threshold": "medium"
|
||||
}
|
||||
]
|
||||
},
|
||||
"guardrail_info": {
|
||||
"description": "Guidelines on Artificial Intelligence Risk Management (MAS) — Blocks adversarial attacks, model poisoning, inversion, and exfiltration attempts targeting financial AI systems"
|
||||
}
|
||||
}
|
||||
],
|
||||
"templateData": {
|
||||
"policy_name": "mas-ai-risk-management",
|
||||
"description": "Guidelines on Artificial Intelligence Risk Management (MAS) for Financial Institutions alignment. Covers fairness & bias, transparency & explainability, human oversight, data governance, and model security. Aligned with the 2018 FEAT Principles, Project MindForge, and NIST AI RMF.",
|
||||
"guardrails_add": [
|
||||
"mas-sg-fairness-bias",
|
||||
"mas-sg-transparency-explainability",
|
||||
"mas-sg-human-oversight",
|
||||
"mas-sg-data-governance",
|
||||
"mas-sg-model-security"
|
||||
],
|
||||
"guardrails_remove": []
|
||||
},
|
||||
"tags": [
|
||||
"Financial Services",
|
||||
"Regulatory",
|
||||
"Singapore"
|
||||
],
|
||||
"estimated_latency_ms": 1
|
||||
}
|
||||
]
|
||||
|
||||
@@ -23,11 +23,6 @@ model_list:
|
||||
|
||||
|
||||
guardrails:
|
||||
- guardrail_name: mcp-user-permissions
|
||||
litellm_params:
|
||||
guardrail: mcp_end_user_permission
|
||||
mode: pre_call
|
||||
default_on: true
|
||||
- guardrail_name: "airline-competitor-intent"
|
||||
guardrail_id: "airline-competitor-intent"
|
||||
litellm_params:
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user