* fix(agentcore): handle JSON responses from agents using sync return
BedrockAgentCoreApp agents that use synchronous `return` (instead of
async `yield`) respond with Content-Type: application/json instead of
text/event-stream. The streaming parser only handles SSE format, silently
discarding the JSON body and returning empty content to the client.
This adds Content-Type detection in both sync and async streaming
wrappers — when application/json is received, the response is parsed
and converted to a single-chunk stream. Also extends _parse_json_response
with a fallback chain supporting multiple agent response schemas (standard
AgentCore, Strands framework, plain string, raw JSON fallback).
* fix(agentcore): add dict-type guard to _parse_json_response
Prevent AttributeError when json.loads() returns a non-dict
(e.g. JSON array or primitive) by adding an isinstance check
at the top of _parse_json_response. Non-dict values fall back
to raw JSON string content.
* fix(agentcore): handle malformed JSON and split streaming chunks
- Wrap json.loads() in try/except in both sync and async streaming
wrappers so malformed JSON bodies raise a structured BedrockError
instead of a raw JSONDecodeError
- Split the JSON-fallback streaming path into two chunks (content
chunk with finish_reason=None, then stop sentinel with empty delta)
to match the SSE path convention
* fix(agentcore): catch IO errors in streaming JSON path + async error test
- Broaden except clause to catch both json.JSONDecodeError and IO-level
exceptions (httpx.ReadError, etc.) from response.read()/aread(), so
all failures surface as structured BedrockError
- Add async malformed-JSON test to mirror the sync test coverage
* feat(proxy): add Prisma DB pool and engine health metrics to Prometheus
Add a PrismaMetricsCollector that periodically queries pg_stat_activity
and the Prisma engine process to expose connection pool and engine health
as Prometheus gauges/counters. Auto-enabled when prometheus_system is in
service_callback.
New metrics:
- litellm_db_pool_active_connections (Gauge)
- litellm_db_pool_idle_connections (Gauge)
- litellm_db_pool_total_connections (Gauge)
- litellm_db_pool_waiting_connections (Gauge)
- litellm_db_engine_up (Gauge)
- litellm_db_engine_restarts_total (Counter)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review feedback
- Only increment engine_restarts counter on heavy reconnects (engine
actually dead), not lightweight network-blip reconnects
- Fix potential KeyError in _get_or_create_gauge/counter fallback path
when REGISTRY._names_to_collectors is absent
- Rename litellm_db_pool_waiting_connections to
litellm_db_pool_lock_waiting_connections to clarify it measures lock
contention, not pool slot queuing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: warn when prometheus_system enabled but watchdog disabled
Log a warning when users have prometheus_system in service_callback
but PRISMA_HEALTH_WATCHDOG_ENABLED=false, since DB pool and engine
metrics won't be collected in that configuration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: retrigger CI checks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: use labeled gauge for DB pool connection metrics
Replace 3 separate pool gauges (active, idle, total) with a single
`litellm_db_pool_connections` gauge using a `state` label. This is more
Prometheus-idiomatic and exposes all pg_stat_activity states (active,
idle, idle in transaction, etc.) without ambiguity about what "total"
includes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review — stale labels and fallback re-registration
- Zero out known pg_stat_activity states that are absent from the current
query result, preventing stale gauge values from persisting.
- Simplify _get_or_create_gauge/counter by removing the fallback loop
that could re-register an already-registered metric (ValueError).
- Add test for stale label clearing across collection cycles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: include "unknown" in _PG_STATES for stale label clearing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: collect immediately on start and consolidate into single query
- Move sleep to end of loop so metrics appear on /metrics immediately
after startup instead of after a 30s delay.
- Combine pool state and lock waiting queries into a single SQL query
using conditional aggregation, halving per-cycle DB overhead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prevent tight spin loop on collection error
Move asyncio.sleep outside the try/except so it always executes even
when _collect_engine_health() or _collect_pool_metrics() raises.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add multiprocess_mode to _get_or_create_gauge initialization
- Include `multiprocess_mode` parameter to properly support multiprocessing in Gauge creation.
- Ensure consistent behavior for labeled and unlabeled Gauges.
* fix: handle invalid env var and document watchdog prerequisite
- Add try/except ValueError for PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
to prevent proxy startup crash on non-numeric values (e.g. "30s")
- Document that DB metrics require both prometheus_system callback and
PRISMA_HEALTH_WATCHDOG_ENABLED=true
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use defensive null coalescing for query_raw row values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add invalid env var fallback test and fix mock signature
- Add test for non-numeric PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
- Add **kwargs to mock _patched_get_or_create_gauge for forward compat
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When both enable_jwt_auth and enable_oauth2_auth are True, the proxy now
routes tokens based on their format:
- JWT tokens (3 dot-separated parts) -> JWT auth handler
- Opaque tokens -> OAuth2 auth handler
This enables using JWT for human users and OAuth2 for M2M (machine) clients
on the same LiteLLM instance. Previously, enabling OAuth2 would intercept
all tokens on LLM API routes before JWT auth could run.
When only one auth method is enabled, behavior is unchanged (backward compatible).
gpt-5.4-pro and gpt-5.4-pro-2026-03-05 do not support the
/v1/chat/completions endpoint — OpenAI returns a 404 with
"This is not a chat model". These models are responses-only,
like o3-pro and o1-pro.
Changes:
- Set mode from "chat" to "responses" for both model entries
- Update supported_endpoints to ["/v1/responses", "/v1/batch"]
- Add regression test for responses API bridge routing
FixesBerriAI/litellm#23014
AgentsPanel no longer renders AgentCardGrid — it uses a Table directly.
The two tests that looked for agent-card-grid are updated to instead verify
the Actions column header appears for admins and is absent for non-admins.
When calling get_users() directly (not via FastAPI), Query() defaults
are not resolved. Pass organization_ids=None explicitly to avoid
'Query' object has no attribute 'split' error.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The `health_check` Query() call was missing its closing parenthesis,
causing a SyntaxError that blocked all proxy_server imports.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_get_users_*: pass proxy admin user_api_key_dict since get_users
now calls _authorize_user_list_request which checks user_role
- test_validate_team_member_add_permissions_non_admin: set
organization_id on mock team since _is_user_org_admin_for_team
accesses it
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- _authorize_and_filter_teams now uses Prisma WHERE clause
(organization_id IN ...) instead of fetching the full team table
- Add missing Switch import in agents.tsx (pre-existing build error)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract _authorize_user_list_request and _authorize_and_filter_teams
helpers to reduce statement count in get_users and list_team. Remove
unused _is_user_team_admin import.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Thread org objects {organization_id, organization_alias} instead of bare IDs from
users/page.tsx → view_users.tsx → CreateUserButton so the selector can show aliases
- Replace single-select org dropdown with multi-select; always shown when organizationIds
is non-null; disabled/pre-selected for single-org admins; displays "Alias (id)"
- handleCreate: maps organization_ids → organizations before POST, removes redundant
organizationMemberAddCall (backend _add_user_to_organizations handles it)
- _user_is_org_admin: also checks organizations list field in addition to singular
organization_id so /user/new succeeds for org admins
- Add 5 backend unit tests for _user_is_org_admin and 2 frontend tests for new form behavior
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Org admins and team admins opening the invite-user modal could not see
the 4 global proxy roles because GET /user/available_roles has no
request body, so the org-admin route check (which requires
organization_id in the payload) always returned False and blocked them.
Add /user/available_roles to self_managed_routes so the route-access
check passes for any authenticated user. The endpoint's existing
Depends(user_api_key_auth) still requires a valid API key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: enforce x-litellm-trace-id in header, if required
* feat: update spend for agent
* refactor: update agent table to follow similar format as other entities - also add a spend column - allows us to see spend of an agent
* fix: cleanup ui
* feat: return spend on agent endpoints
* feat: scope pr
* feat(agents/): support budgets + rate limiting on agents + agent sessions
* fix: address PR review feedback
- Add missing tpm_limit, rpm_limit, session_tpm_limit, session_rpm_limit
columns to root schema.prisma to match proxy and extras schemas
- Add backwards-compatible fallback to key metadata for max_iterations
so existing users don't silently lose enforcement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: qa'ed RPM limiting on agents
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add health check toggle to agents page
Backend:
- Add health_check query parameter to GET /v1/agents endpoint
- When health_check=true, performs concurrent GET requests to each agent's
URL and filters out agents with unreachable URLs (5s timeout)
- Agents returning HTTP <500 are considered healthy; 5xx and connection
errors mark agents as unhealthy
UI:
- Add Health Check toggle (Switch) to agents panel header
- Toggle triggers re-fetch with health_check=true, filtering the agent list
- Icon color changes (green/gray) to indicate toggle state
- Tooltip explains behavior: 'only agents with reachable URLs are shown'
Networking:
- Update getAgentsList to accept optional healthCheck boolean parameter
Tests:
- Backend: 9 new tests covering health check filtering, _check_agent_url_health
helper (no URL, 200, 404, 500, connection error cases)
- UI: 3 new tests verifying toggle renders, initial fetch without health check,
and fetch with health check after toggle click
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* fix: fix greptile comment re: security issue
* fix: fix based on greptile feedback
* fix: align health check tests with implementation
- Rename test_should_return_unhealthy_when_no_url to
test_should_return_healthy_when_no_url (implementation returns
healthy=True for agents without a URL)
- Patch get_async_httpx_client instead of httpx.AsyncClient so mocks
actually intercept the HTTP calls made by _check_agent_url_health
- Remove unnecessary __aenter__/__aexit__ context-manager mocks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: undo _experimental/out renames from cherry-pick
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update litellm/proxy/agent_endpoints/endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(docker): bump tar/minimatch/pypdf for CVE fixes + harden SBOM patching
- Bump tar 7.5.8→7.5.10, minimatch 10.2.1→10.2.4, pypdf 6.6.2→6.7.3
- Add sed-based SBOM metadata patching with properly indented find/sed
- Add npm package manager cleanup (apk del / apt-get purge) to remove
stale SBOM entries from image scanners
- Scope || true to only apk del via brace grouping { ... || true; }
- Guard npm root -g with non-empty assertion to prevent silent failures
- Scope minimatch sed regex to ^10.x to avoid matching other major versions
Addresses: CVE-2026-27903, CVE-2026-27904, GHSA-qffp-2rhf-9h96, CVE-2026-27888
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(docker): scope find to /usr/local/lib /usr/lib, drop autoremove
- Replace `find /` with `find /usr/local/lib /usr/lib` to avoid
traversing /proc, /sys, /dev during SBOM metadata patching
- Remove `apt-get autoremove -y` from Debian-based Dockerfiles to
prevent nodejs from being removed as an auto-installed dependency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
UI sends organization_ids=org1,org2,org3 and backend parses the
comma-separated string, matching the existing pattern used elsewhere.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Backend authorization: enforce server-side org scoping on /user/list.
Non-proxy-admin callers must be org_admin; their permitted org IDs are
fetched from the DB and intersected with any requested org filter.
- Race condition: gate userListQuery on org data being loaded for
non-proxy-admin users (undefined = loading, null = no filter needed).
- Silent failure: surface organizationMemberAddCall errors via
NotificationsManager instead of only console.error.
- Multi-org: org admins managing multiple orgs can choose which org to
add the new user to via a dropdown (defaults to first org).
- Change organization_id param to List[str] (repeated query params)
instead of comma-separated string.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updates two tests in TeamInfo.test.tsx that were asserting for "X Members" text that was removed from the TeamVirtualKeysTable component in commit ec4ef9c924.
- Changed test "should display X Members in Virtual Keys tab when navigated to" to assert for pagination controls instead
- Changed test "should show Filters and pagination controls in Virtual Keys tab" to wait for Filters button instead of member count
All 29 tests now pass.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Extract _resolve_org_filter_for_user_search and _resolve_team_org_filter
helpers from ui_view_users to bring complexity under the ruff C901 limit.
- Fix misleading error when a team admin searches but team has no org:
now says "this team is not part of an organization" instead of the
generic "only admins can search" message.
- Add distinct errors for team-not-found and not-a-team-admin cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ryan Crabbe