* fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in bedrock KB tests
* fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in test_router
* fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in test_router_retries
* fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in test_router_timeout
SpendLogsMetadata added new fields (user_api_key, status, error_information,
etc.) that weren't in the expected spend_logs_payload.json fixture, causing
test_async_gcs_pub_sub_v1 to fail.
Adds a new config option to exclude specific fields from StandardLoggingPayload
before any callback receives it. This provides a general approach to control
what data is logged across ALL integrations (S3, GCS, Datadog, etc.).
## Changes
1. **litellm/__init__.py**: Added new global setting
`standard_logging_payload_excluded_fields: Optional[List[str]] = None`
2. **litellm/integrations/custom_logger.py**: Modified
`redact_standard_logging_payload_from_model_call_details()` to:
- Remove specified fields entirely from the StandardLoggingPayload
- Works alongside existing `turn_off_message_logging` feature
- Excluded fields take precedence (removed rather than redacted)
3. **tests/**: Added comprehensive test suite with 17 tests covering:
- Single/multiple field exclusion
- Interaction with turn_off_message_logging
- Original payload immutability
- Config loading via setattr (proxy pattern)
- Edge cases (empty list, non-existent fields, None standard_logging_object)
## Usage
```yaml
litellm_settings:
success_callback: ["s3"]
standard_logging_payload_excluded_fields: ["response", "messages"]
```
This removes the `response` and `messages` fields from logs before any
callback processes them, reducing log size and improving privacy compliance.
## Available Fields
The fields match StandardLoggingPayload TypedDict keys including:
- messages, response (large payload fields)
- metadata, hidden_params, model_parameters
- error_str, error_information
- And all other StandardLoggingPayload fields
Closes the need for per-integration flags like `s3_log_response`.
* perf: Optimize get_standard_logging_metadata with set intersection
- Cache StandardLoggingMetadata.__annotations__.keys() as module-level frozenset
- Use set intersection to iterate only keys present in both metadata and supported keys
- Single lookup for user_api_key instead of 3 separate .get() calls
Results:
- get_standard_logging_metadata: 1.55s → 1.41s (9.2% faster)
* test: add unit tests for get_standard_logging_metadata non-string user_api_key handling
Adds log_format parameter supporting json_array (default), ndjson, and single formats. NDJSON format enables webhook integrations like Sumo Logic to parse individual log records at ingest time. Defaults to json_array for backward compatibility.
* fix: enforce team member budget check in common_checks
- Add missing team member budget validation in common_checks() function
- Checks team membership budget when team key is used
- Raises BudgetExceededError when team member spend exceeds max_budget_in_team
- Follows same pattern as other budget checks (team, user, end_user)
- Uses cached get_team_membership() for performance
- Fix AttributeError in lowest_tpm_rpm.py
- Add null check for model_info before accessing .get() method
- Prevents 'NoneType' object has no attribute 'get' error
- Add unit tests for team member budget enforcement
- Test budget exceeded scenario
- Test within budget scenario
- Test edge cases (no budget, no membership, personal keys)
- Tests run without requiring proxy server
Fixes failing test: test_users_in_team_budget
* fix: mock get_async_httpx_client in test_langsmith_key_based_logging
- Mock get_async_httpx_client to return a mock AsyncHTTPHandler instance
- Fixes test failure where mock_post was never called
- LangsmithLogger creates its own httpx client instance via get_async_httpx_client,
so we need to mock the factory function rather than the class method
- Use MagicMock for response.raise_for_status (sync method) instead of AsyncMock
* fix: resolve linting errors (PLR0915, F401)
- Remove unused imports (datetime, ServiceLoggerPayload) from arize_phoenix.py
- Extract health ping setup logic from RedisCache.__init__ to reduce statement count
- Extract team member budget check from common_checks to reduce statement count
* fix: resolve type errors in ChatCompletionToolCallChunk construction
- Cast type field to Literal['function'] to satisfy TypedDict requirements
- Ensure arguments field is explicitly str type to match TypedDict signature
- Fixes pyright errors for incompatible types in transformation.py
- Add agent_id: null to expected JSON to match actual payload structure
- Fixes test_async_gcs_pub_sub_v1 test failure
- agent_id is an optional field in SpendLogsPayload that is always included (as null when not provided)
* feat(langfuse): Add support for custom masking function
Allow users to pass a custom masking function via metadata to selectively
redact sensitive data (credit cards, emails, PII) before sending to Langfuse.
Usage:
```python
def mask_pii(data):
if isinstance(data, str):
data = re.sub(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b', '[CARD]', data)
return data
litellm.completion(
model="gpt-4",
messages=[...],
metadata={"langfuse_masking_function": mask_pii}
)
```
* fix(langfuse): Isolate masking function from other logging integrations
Extract langfuse_masking_function from metadata early in the flow and store
it in a dedicated key (_langfuse_masking_function) that only the Langfuse
logger knows to look for. This prevents the callable from leaking to other
logging integrations (Datadog, S3, etc.) which would serialize it as
"<function at 0x...>".
Changes:
- scrub_sensitive_keys_in_metadata() now extracts and stores the function
- Langfuse logger looks in the dedicated key first, falls back to metadata
- Added tests to verify isolation works correctly
* test: add failing tests for organization budget enforcement bug
Add comprehensive tests exposing that organization-level budgets are
retrieved but never enforced during request authentication. Tests verify:
1. Basic org budget exceeded scenario (team under budget, org over)
2. Multiple teams collectively exceeding org budget
3. Organization budget fields exist but are never checked
4. Inconsistency between team budget enforcement (works) and org (doesn't)
Tests intentionally fail to document the bug. Will be fixed in next commit.
Related to organization_max_budget not being enforced in auth_checks.py
* fix: enforce organization budget in auth checks
Add organization budget enforcement to common_checks() in auth_checks.py.
Previously, organization_max_budget was retrieved from DB but never checked,
allowing teams to collectively exceed their organization's budget limit.
Changes:
- Add _organization_max_budget_check() function following team budget pattern
- Call org budget check after team budget check in common_checks()
- Add "organization_budget" to budget_alerts type literals
- Update tests to verify org budget is enforced
Budget hierarchy is now properly enforced:
Organization Budget (hard ceiling)
└─ Team Budget (sub-allocation)
└─ Team Member Budget (per-user within team)
└─ Key Budget (per-key)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: add organization_id to budget alerts, fix enum comparison and linting of newly added code
- Add organization_id field to CallInfo class for better alert context
- Include organization_id in budget alerts (token, soft, team, org)
- Fix event_group enum comparison (was comparing enum to string)
- Add OrganizationBudgetAlert class for organization budget alerting
- Add organization_budget to test parameterizations
- Apply Black formatting to slack_alerting.py
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(generic_api_callback.py): make generic api OSS + support multiple generic API's
Enables https://github.com/BerriAI/litellm/pull/17094#discussion_r2562832967
* feat(callback_utils.py): support custom generic api callbacks
* feat(generic_api_callback.py): support specifying which event types to run the generic api for
* fix(litellm_logging.py): log system prompt for anthropic messages
* feat(generic_api_callback.py): support generic api compatible api's - e.g. rubrik agent cloud
* docs(sidebars.js): document new OSS generic api
* docs(generic_api.md): document new OSS Generic API
* docs(custom_webhook_api.md): document custom webhook api integration tutorial
* docs(custom_webhook_api.md): cleanup
* docs(custom_webhook_api.md): document what get's logged to custom webhook api
* Refactor: Pass callback config to GenericAPILogger
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
* Fix: Handle empty messages list in logging payload
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
* Checkpoint before follow-up message
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
* feat: Cache GenericAPILogger instances to improve performance
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Fixes#16486
## Problem
Callbacks configured via litellm.success_callback (e.g., PostHog, LangSmith)
were not being invoked for litellm.acompletion() in short-lived scripts.
The callbacks worked correctly for synchronous completions but async
completions would queue callbacks that were lost when the script exited
before GLOBAL_LOGGING_WORKER could process them.
Root cause: asyncio.run() closes the event loop immediately after the
async function completes, preventing the background worker from processing
queued callbacks.
## Solution
Implemented a two-level atexit handler approach:
1. GLOBAL_LOGGING_WORKER atexit handler (logging_worker.py):
- Processes remaining callbacks from queue before exit
- Creates new event loop to run pending coroutines synchronously
- Applies time and iteration limits to prevent blocking shutdown
2. Integration-specific atexit handlers (posthog.py as example):
- Flushes internal queue to external service
- Uses synchronous HTTP client for reliable delivery
- Each integration needs its own handler due to varying sync APIs
## Changes
- litellm/litellm_core_utils/logging_worker.py:
- Added _flush_on_exit() method
- Registered atexit handler in __init__
- Processes up to MAX_ITERATIONS_TO_CLEAR_QUEUE events
- Time-limited to MAX_TIME_TO_CLEAR_QUEUE seconds
- litellm/integrations/posthog.py:
- Added _flush_on_exit() method
- Registered atexit handler in __init__
- Groups events by credentials for batch sending
- Uses sync_client for reliable HTTP delivery
- tests/logging_callback_tests/test_posthog.py:
- Added test_async_callback_atexit_handler_exists()
- Added test_posthog_atexit_flushes_internal_queue()
- Added test_sync_callback_not_affected_by_atexit()
## Testing
- All existing tests pass
- Manual end-to-end testing confirms fix:
- Async events now arrive in PostHog
- Sync events continue working (no regression)
- Unit tests verify atexit handlers registered and functional
## Impact
- Fixes async callback delivery for ALL integrations using GLOBAL_LOGGING_WORKER
- No breaking changes - only adds missing functionality
- Sync path unchanged - no performance impact
Ryan Crabbe