Budget checks on API keys, teams, and team members were not enforced in
multi-pod deployments because user_api_key_cache is intentionally
in-memory-only. Each pod tracked spend independently, so with N pods
the effective budget was N × max_budget.
Introduces a separate spend_counter_cache (DualCache wired to
redis_usage_cache) with atomic increment/read helpers:
- increment_spend_counters(): awaited in cost callback (not create_task)
to update both in-memory and Redis before the next auth check
- get_current_spend(): reads Redis first (cross-pod authoritative),
falls back to in-memory, then to cached object .spend from DB
Budget check functions (_virtual_key_max_budget_check,
_team_max_budget_check, _check_team_member_budget) now read spend via
get_current_spend() instead of cached object .spend fields.
When Redis is not configured, falls back to in-memory-only counters
(same as current single-instance behavior).
Fixes#23714
* Include user_email in new user creation within get_user_object
Enhance the get_user_object function to include user_email in the parameters when creating a new user. This change is accompanied by a new test to verify that user_email is correctly included during the upsert process.
* Improve error handling in test_get_user_object by logging exceptions
Updated the test_get_user_object_upsert_includes_user_email function to log exceptions when they occur, enhancing the visibility of potential issues during testing. This change helps in diagnosing failures related to the mock LiteLLM_UserTable.
Replaces the skip_route_check approach from PR #22662 with a configurable
opt-in flag. By default, common_checks() is not run for custom auth flows,
preserving backwards compatibility with pre-#22164 behavior.
Users who want budget/team/route enforcement on custom auth can enable it:
general_settings:
custom_auth_run_common_checks: true
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Custom user-added routes (e.g. /ldap/ngs/ready) used with Depends(user_api_key_auth) were being rejected as admin-only after _run_post_custom_auth_checks was introduced in commit 14badde13c.
The route authorization check in common_checks is designed for LiteLLM's own management routes. Custom auth flows that add their own routes should be trusted since the custom auth function already validated the request. Budget and expiry checks still run.
Add skip_route_check parameter to common_checks() and pass skip_route_check=True from _run_post_custom_auth_checks() to skip route authorization while preserving budget/team/model checks.
Regression test added: test_common_checks_skip_route_check_for_custom_auth
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Warn when budget lookup fails; cache won't populate
- Add _log_budget_lookup_failure helper in auth_checks.py
- Log at WARNING in get_user_object, get_team_object, get_key_object
when DB lookups fail (schema mismatch, etc.)
- Add schema migration hint for prisma/db errors
- Add dry-run test for _log_budget_lookup_failure
* fix: skip budget lookup failure log for expected user-not-found case
Avoid logging 'cache will not be populated' when the user simply doesn't
exist - not caching is correct behavior in that case. Only log for
unexpected errors (schema, DB, etc.) where the message is meaningful.
* fix(ui_sso.py): fix form action on login when sso is enabled
* fix: multiple fixes - fix resetting env var in proxy config + add key to exception message on key decryption
fixes issue where env vars would be reset
* refactor(proxy_server.py): cleanup redundant decryption line
* fix(proxy_setting_endpoints.py): show saved ui access mode
allows admin to know what they'd previously stored in db
* fix(auth_checks.py): resolve a model group alias when key has access to underlying model
Fixes LIT-293
* feat(anthropic/): add mock_response to anthropic /v1/messages
makes it easy to test fallback logic
* fix(router.py): support fallbacks on /v1/messages
adds working fallbacks on generic api route
* refactor(router.py): point _ageneric_api_call_with_fallbacks to updated function
* test: add unit test for new helper on router
* fix(router.py): use correct metadata variable name
* fix(router.py): use correct metadata field
* docs(config_settings.md): document new param
michelligabriele