3 Commits

Include renames

Author	SHA1	Message	Date
Mateo WangandGitHub	2c733c00f5	chore(ci): modernize model references in tests and configs (#27856) ... * test: modernize models used in CircleCI e2e test suites Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo, claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current equivalents across the e2e_openai_endpoints and proxy_e2e_anthropic_messages_tests CircleCI jobs. - gpt-4o -> gpt-5.5 (responses API e2e tests) - gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config) - gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning, still actively fine-tunable) - gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 / gpt-5-mini - bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001 (also aligning oai_misc_config model_name with what test_bedrock_batches_api.py actually requests) - bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15) -> claude-sonnet-4-5-20250929 * test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5 Greptile/Cursor flagged that after the previous commit, the bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5 (both pointed to claude-sonnet-4-5-20250929). Rename to bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID (us.anthropic.claude-sonnet-4-6, already in the litellm model registry) so the alias name matches the underlying model version. * test: modernize models across remaining CI-mounted configs & tests Expands the modernization sweep to all CircleCI-mounted proxy configs and to test directories where the model literal is a fixture/route key (not the test's subject). Config changes: - proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 / gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump text-embedding-ada-002 underlying to text-embedding-3-small. User- facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.) preserved for backward compatibility with tests. - simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml: bump gpt-3.5-turbo underlying to gpt-5-mini. - pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet / claude-3-haiku entries replaced with claude-sonnet-4-5 / claude- haiku-4-5 / claude-opus-4-7. - oai_misc_config.yaml: align alias name with the gpt-5-mini rename. Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4- 20250514 retire 2026-06-15): - tests/llm_translation/test_anthropic_completion.py: bump 3 references + paired Vertex AI ID to claude-sonnet-4-5. - tests/llm_translation/test_optional_params.py: bump 2 references. - tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py and test_bedrock_anthropic_messages_test.py: bump router fixtures using the deprecated model IDs. - tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py: modernize docstring examples. - tests/test_end_users.py: update references to renamed alias. * test: modernize placeholder model literals in router_unit_tests Mass replace_all on fixture/placeholder model literals across the router_unit_tests/ suite (model name is a routing key / label, not the test subject). Sub-agent sweep so far — additional commits will follow for logging_callback_tests/, enterprise/, top-level tests/test_*.py, and other CI-mounted dirs. Mappings applied: - gpt-3.5-turbo -> gpt-5-mini - gpt-4 (bare) -> gpt-5.5 - gpt-4o (bare) -> gpt-5 - text-embedding-ada-002 -> text-embedding-3-small - claude-3-sonnet-20240229 / claude-3-opus-20240229 / claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 -> claude-sonnet-4-5-20250929 / claude-opus-4-7 / claude-haiku-4-5-20251001 as appropriate Explicitly preserved: - gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current - gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals) - JSONL batch body literals - Mock LLM response model fields (must match upstream) - Fake/mock identifiers * test: modernize placeholder model literals across remaining CI suites Sub-agent sweep across logging_callback_tests/, guardrails_tests/, enterprise/, pass_through_unit_tests/, otel_tests/, llm_responses_api_testing/, batches_tests/, spend_tracking_tests/, litellm_utils_tests/, unified_google_tests/, and a few top-level tests/test_*.py files where the model literal is a fixture or placeholder (router model_list, mock standard logging payload, mock callback data) rather than the test's subject. Mappings applied (see scope notes below): - gpt-3.5-turbo -> gpt-5-mini - gpt-4 (bare) -> gpt-5.5 - gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5 is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex / gpt-5-mini exist) - gpt-4o-mini (bare) -> gpt-5-mini - text-embedding-ada-002 -> text-embedding-3-small - claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929 - claude-3-opus-20240229 -> claude-opus-4-7 - claude-3-haiku-20240307 -> claude-haiku-4-5-20251001 - claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929 - claude-3-7-sonnet-20250219 -> claude-sonnet-4-6 - gemini-1.5-flash -> gemini-2.5-flash - gemini-1.5-pro -> gemini-2.5-pro Explicitly preserved (not modernized): - llm_translation/ tests where model is the SUBJECT (provider-specific translation/transformation logic). Only the deprecated 20250514 references were already bumped in a prior commit. - Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges documented by the sub-agent). - Bedrock model IDs in test_health_check.py path-stripping tests. - JSONL batch request bodies and mock LLM response bodies (must match upstream literal). - Langfuse expected-request-body JSON fixtures (cost values are exact- match-asserted; changing the model would shift response_cost). - gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI equivalent). - Top-level tests calling the proxy through user-facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases in proxy_server_config.yaml stay; only the underlying model was bumped. - tests/test_gpt5_azure_temperature_support.py (the test's whole point is model-name handling). - Fake / mock / openai/fake identifiers. Notable side fixes: - test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini), resolving a latent inconsistency. - proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5` (bare gpt-5 is not a valid OpenAI alias). - test_batches_logging_unit_tests.py: explicit_models list entries kept distinct (gpt-5-mini + gpt-5.5) after bulk rename. * test: fix CI failures from model modernization sweep CI surfaced 4 categories of regression from the bulk modernization: 1. Azure deployment names are customer-specific. Reverted: - tests/litellm_utils_tests/test_health_check.py: azure/text- embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure account does not have a text-embedding-3-small deployment). - tests/logging_callback_tests/test_custom_callback_router.py: same revert for two router fixtures driving aembedding. 2. gpt-5 family does not accept temperature != 1. Tests that pass a custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern non-reasoning OpenAI mini that still accepts temperature/logprobs): - tests/logging_callback_tests/test_datadog.py - tests/logging_callback_tests/test_langsmith_unit_test.py - tests/logging_callback_tests/test_otel_logging.py 3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to gpt-5.5 (a reasoning model that rejects logprobs). The proxy test tests/test_openai_endpoints.py::test_chat_completion_streaming exercises logprobs/top_logprobs through that alias. Bumped the underlying model to gpt-4.1 (non-reasoning, still modern). 4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with hardcoded model="gpt-4o" and a model-specific spend value. Reverted the litellm.acompletion calls in the test to model="gpt-4o" so the fixture's exact-match assertions still hold. 5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py: anthropic.messages.create routing to openai/gpt-5-mini returned an empty content[0] with max_tokens=100 (reasoning-token consumption). Swapped to openai/gpt-4.1-mini. * test: fix Assistants API model + 2 cursor[bot] review nits 1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5 isn't accepted by the /v1/assistants endpoint ("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants- API-supported, non-reasoning). 2. example_config_yaml/pass_through_config.yaml: the previous sweep bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the Sonnet tier intact. (Cursor bugbot review.) 3. example_config_yaml/simple_config.yaml: model_name was left as gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which muddles the "simple" example. Make both sides gpt-5-mini so the most basic example is a straight 1:1 mapping again. (Cursor bugbot review.) * fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models tests/test_openai_endpoints.py::test_completion calls the proxy alias "gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with custom temperature / logprobs / the legacy /v1/completions endpoint. The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini, which are reasoning models that reject temperature != 1 and don't expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini (modern non-reasoning OpenAI models) instead — keeps user-facing aliases preserved while picking a current underlying that still supports the parameters/endpoints the tests exercise.	2026-05-15 15:44:28 -07:00
Ishaan Jaffer	e8461b5b97	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
3 people	b1b96ff3cf	[Perf] Alexsander fixes round 2 - Oct 18th (#15695) ... * perf(router): Optimize prompt management model check with early exit Add early return for models without '/' to avoid expensive get_model_list() calls for 99% of standard model requests (gpt-4, claude-3, etc). - Refactor _is_prompt_management_model() with "/" check before model lookup - Add unit tests to verify optimization doesn't break detection * perf(caching): optimize Redis batch cache operations and reduce unnecessary queries This commit introduces several performance optimizations to the Redis caching layer: **DualCache Improvements (dual_cache.py):** 1. Increase batch cache size limit from 100 to 1000 - Allows for larger batch operations, reducing Redis round-trips 2. Throttle repeated Redis queries for cache misses - Update last_redis_batch_access_time for ALL queried keys, including those with None values - Prevents excessive Redis queries for frequently-accessed non-existent keys 3. Add early exit optimization - Short-circuit when redis_result is None or contains only None values - Avoids unnecessary processing when no cache hits are found 4. Optimize key lookup performance - Replace O(n) keys.index() calls with O(1) dict lookup via key_to_index mapping - Reduces algorithmic complexity in batch operations 5. Streamline cache updates - Combine result updates and in-memory cache updates in single loop - Only cache non-None values to avoid polluting in-memory cache **CooldownCache Improvements (cooldown_cache.py):** 1. Enhanced early return logic - Check if all values in results are None, not just if results is None - Prevents unnecessary iteration when no valid cooldown data exists These changes significantly improve Redis caching performance, especially for: - High-throughput batch operations - Scenarios with frequent cache misses - Large-scale deployments with many concurrent requests * fix: remove unnecessary test * refactor: move default_max_redis_batch_cache_size to constants - Add DEFAULT_MAX_REDIS_BATCH_CACHE_SIZE constant (default: 1000) - Update DualCache to use constant from constants.py - Document new environment variable in config_settings.md * fix: only use in memory cache when set * fix(router): improve prompt management model detection with smart early return The previous early return optimization in _is_prompt_management_model() was checking if the model name parameter contained '/' and returning False if it didn't. This broke detection for model aliases (e.g., 'chatbot_actions') that don't have '/' in their name but map to prompt management models (e.g., 'langfuse/openai-gpt-3.5-turbo'). Changed the early return logic to only exit early when: - Model name contains '/' AND - The prefix is NOT a known prompt management provider This maintains the performance optimization for 99% of direct model calls (avoiding expensive get_model_list lookups) while correctly handling: - Direct prompt management calls (e.g., 'langfuse/model') - Model aliases without '/' (e.g., 'chatbot_actions') - Regular models with/without '/' (e.g., 'gpt-3.5-turbo', 'openai/gpt-4') Fixes test: test_router_prompt_management_factory * perf(router): optimize _pre_call_checks with shallow copy (1400x faster) Replace deepcopy with list() in _pre_call_checks - runs on every request. Only pops from list, never modifies deployment dicts, so shallow copy is safe. Performance: 1400x faster on hot path Impact: 2-5x overall throughput improvement for routing workloads Tests: Added regression test to ensure no mutation + filtering works * perf(router): replace deepcopy with shallow copy for default deployment Replace expensive copy.deepcopy() with shallow copy for default_deployment in _common_checks_available_deployment() hot path. Changes: - Use dict.copy() for top-level deployment dict - Use dict.copy() for nested litellm_params dict - Only the 'model' field is modified, so deep recursion is unnecessary Impact: - 100x+ faster for default deployment path (every request when used) - deepcopy recursively traverses entire object tree - Shallow copy only copies two dict levels (exactly what's needed) Test coverage: - Added regression test to verify deployment isolation - Ensures returned deployments don't mutate original default_deployment - Validates multiple concurrent requests get independent copies * perf(router): remove unnecessary dict copy in completion hot paths Remove unnecessary deployment['litellm_params'].copy() in _completion and _acompletion functions. The dict is only read and spread into a new dict, never modified, making the defensive copy wasteful. Changes: - Remove .copy() in _completion (sync hot path) - Remove .copy() in _acompletion (async hot path) Impact: - Every completion request (highest traffic endpoints) - Eliminates unnecessary dict allocation and copy on every call - Dict spreading already creates new dict, so no mutation possible Test coverage: - Added tests verifying deployment params unchanged after calls - Tests both sync and async completion paths - Validates optimization doesn't introduce mutations * perf(router): optimize deployment filtering in pre-call checks Replace O(n²) list pop pattern with O(n) set-based filtering in _pre_call_checks() to improve routing performance under high load. Changes: - Use set() instead of list for invalid_model_indices tracking - Replace reversed list.pop() loop with single-pass list comprehension - Eliminate redundant list→set conversion overhead Impact: - Hot path optimization: runs on every request through the router - ~2-5x faster filtering when many deployments fail validation - Most beneficial with 50+ deployments per model group or high invalidation rates (rate limits, context window exceeded) Technical details: Old: O(k²) where k = invalid deployments (pop shifts remaining elements) New: O(n) single pass with O(1) set membership checks * add: memory profiler feat(proxy): Add configurable GC thresholds and enhance memory debugging endpoints - Add PYTHON_GC_THRESHOLD env var to configure garbage collection thresholds - Add POST /debug/memory/gc/configure endpoint for runtime GC tuning - Enhance memory debugging endpoints with better structure and explanations - Add comprehensive router and cache memory tracking - Include worker PID in all debug responses for multi-worker debugging * refactor: reduce complexity in get_memory_details endpoint Extract 6 helper functions from get_memory_details to fix linter error PLR0915 (too many statements). Improves maintainability while preserving functionality. * fix(router): remove incorrect early exit in _is_prompt_management_model Removes early exit optimization that checked model_name prefix instead of the actual litellm_params model. This incorrectly returned False for custom model aliases that map to prompt management providers. Example: "my-langfuse-prompt/test_id" -> "langfuse_prompt/actual_id" The method now correctly checks the underlying model's prefix. Fixes test_is_prompt_management_model_optimization * fix(proxy): add explicit type annotations to debug_utils dictionaries Resolved 6 mypy type errors in proxy/common_utils/debug_utils.py by adding explicit Dict[str, Any] annotations to dictionary variables where mypy was incorrectly inferring narrow types. This allows the dictionaries to accept different value types (strings, nested dicts) for error handling and various return structures. Fixed: - Line 246: caches dictionary in get_memory_summary() - Line 371: cache_stats dictionary in _get_cache_memory_stats() - Line 439: litellm_router_memory dictionary in _get_router_memory_stats() * fix(proxy): fix Python 3.8 compatibility in debug_utils type annotations - Replace tuple[...], list[...] with Tuple[...], List[...] from typing - Replace Dict | None with Optional[Dict] for Python 3.8 compatibility - Add missing imports: List, Optional, Tuple to typing imports Fixes TypeError: 'type' object is not subscriptable in Python 3.8 --------- Co-authored-by: AlexsanderHamir <alexsanderhamirgomesbaptista@gmail.com>	2025-10-18 11:12:00 -07:00