mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-07 09:12:20 +00:00
milan-berri
5119b9462f
* feat(arize): enrich OpenInference attributes for better span rendering
Pure rendering enhancements to the Arize / Arize Phoenix integration. No
existing attribute keys or values are removed or overwritten; every new
emit is independently try/except-wrapped and fires only when its source
data is present so existing behavior is preserved.
What this adds
- Coerce non-dict response objects (e.g. httpx.Response from passthrough
routes) via JSON decode so id/model/usage extraction stops crashing
with "'Response' object has no attribute 'get'". Dicts and Pydantic
objects with .get pass through unchanged.
- Set OPENINFERENCE_SPAN_KIND defensively early so a downstream failure
can't blank the kind; the original late write (incl. TOOL upgrade) is
preserved.
- Add "passthrough" keyword to _infer_open_inference_span_kind so
allm_passthrough_route / llm_passthrough_route resolve to LLM instead
of UNKNOWN.
- Emit cache token breakdown: LLM_TOKEN_COUNT_PROMPT_DETAILS_CACHE_READ /
_CACHE_WRITE / _AUDIO. Sources covered: OpenAI prompt_tokens_details
and Anthropic / Bedrock cache_{read,creation}_input_tokens.
- Render assistant tool_calls on both input and output messages via
MESSAGE_TOOL_CALLS.* (Pydantic-aware, handles ModelResponse choices).
Tool-result input messages also get MESSAGE_TOOL_CALL_ID and
MESSAGE_NAME.
- Render multimodal list-shaped content via MESSAGE_CONTENTS.* (OpenAI
image_url, Anthropic source.{media_type,data} as data: URI). Legacy
MESSAGE_CONTENT write is unchanged.
- Emit SESSION_ID (end_user_id / trace_id), USER_ID (only when not
already set by optional_params.user or model_params.user), and
litellm.{team_id,team_alias,key_alias} from StandardLoggingPayload
metadata.
- Emit llm.response.cost as float from StandardLoggingPayload.response_cost.
- Bedrock / Anthropic passthrough normalization: extract input from
additional_args.complete_input_dict and output from the coerced
provider response so INPUT_VALUE / OUTPUT_VALUE / LLM_INPUT_MESSAGES /
LLM_OUTPUT_MESSAGES are populated. Only runs when call_type contains
"passthrough" / "pass_through".
Tests
- 15 new unit tests covering each addition plus explicit regression
guards (USER_ID overwrite protection, passthrough normalizer scope,
coerce identity for dicts/.get-bearing objects, no spurious cache
emits).
- Existing test_arize_set_attributes count bumped from 26 to 27 to
account for the additional defensive span.kind write (same value,
written twice).
- tests/test_litellm/integrations/arize/: 70 passed (55 baseline + 15
new). tests/test_litellm/integrations/test_opentelemetry.py: 221
passed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(arize): collapse additive try/except blocks into _safe_emit helper
The additive attribute emitters all share the same shape: run a callable,
swallow any exception to debug log so it cannot blank the span. Hoisting
that pattern into a single _safe_emit(label, fn, *args, **kwargs) helper
removes 5 repeated try/except blocks. Behavior unchanged; arize test
suite still passes (70/70).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): emit cost under canonical llm.cost.total key
Arize's "Total Cost" column reads the OpenInference-standard
`llm.cost.total` attribute. The previous custom `llm.response.cost`
key never surfaced in the trace list. Now emits both keys (canonical +
legacy) so renderers + any existing consumers both work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): keep span.kind=LLM for tool-using completions + render tool_calls in Output
A chat completion that passes `tools=[...]` or returns `tool_calls` is still
an LLM call per the OpenInference spec — TOOL is reserved for actual tool
execution. The previous override demoted these to TOOL, breaking Arize's
LLM-scoped dashboards/evals and skewing token/cost analytics for any
tool-using traffic.
Additionally, when an assistant response had no text content but did
request tool calls, `output.value` was set to the empty string so Arize's
"Output" pane rendered blank. Now serializes the tool_calls into a compact
JSON summary in `output.value` (the structured `MESSAGE_TOOL_CALLS.*`
attributes are still emitted unchanged).
Cleanups:
- extract `_get_tool_calls` and `_normalize_tool_call` helpers,
deduplicating the dict-vs-Pydantic + function-dict logic across
`_set_choice_outputs`, `_emit_message_tool_calls`, and the new
`_summarize_tool_calls_for_output`.
- drop redundant late `OPENINFERENCE_SPAN_KIND` write — the defensive
early write is now the single source of truth.
- remove a dead local re-import of `MessageAttributes`/`SpanAttributes`.
Tests: 73 pass (added regression guard asserting span.kind stays LLM for
completions that pass tools AND return tool_calls; existing call_count
assertion restored to 26).
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(arize): tighten cleanup — fold _get_tool_calls into _safe_get
Two tiny cleanups, no behavior change:
- collapse `_get_tool_calls` to use `_safe_get`, removing a 7-line
hand-rolled dict-vs-attribute fallback that duplicated existing logic.
- trim the `_set_choice_outputs` tool-call summary comment from 4 lines
to 2 (was over-explaining).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): address Greptile review — drop session_id=trace_id fallback, remove dead code, fix Black
Three Greptile-flagged issues + the Black formatting CI failure.
1. SESSION_ID no longer falls back to trace_id. Previously every span
without an explicit `user_api_key_end_user_id` would have its
session.id set to the per-request trace_id, which creates one
distinct "session" per request and breaks Arize's Session-grouping
analytics. Now SESSION_ID is emitted only when an explicit end-user
identifier exists, and the trace_id is emitted under its own
`litellm.trace_id` key so spans remain filterable by trace.
2. Removed dead `ArizeOTELAttributes.set_response_output_messages`
override. Confirmed zero callers in the entire repo (the live path
is `_set_choice_outputs` via `_set_response_attributes`). The
override was preexisting dead code, but the expansion of
`_set_choice_outputs` in this PR made the divergence misleading.
3. Removed permanently-dead first branch in cache_write detection.
`_safe_get(prompt_token_details, "cache_creation_tokens")` looks
for a key that neither OpenAI's `prompt_tokens_details` nor
Anthropic's payload ever exposes. Now reads straight off `usage`
for `cache_creation_input_tokens`.
4. Reformatted both files under Black 26.3.1 (the version CI uses
via `uv sync --frozen`). Local previously used 24.10.0.
Tests: 74/74 pass in the arize suite (added
`test_arize_does_not_use_trace_id_as_session_id_fallback`).
Combined arize + opentelemetry suite: 295/295 pass.
End-to-end verified live: tool-call still emits `span.kind=LLM` and
JSON tool_calls in `output.value`; `session.id` is now correctly
unset when no end_user_id is provided; `litellm.trace_id` is
populated; Bedrock passthrough input/output unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): gate passthrough prompt export on message redaction
- Skip the complete_input_dict bridge in _maybe_normalize_passthrough when
should_redact_message_logging() is true, so enabling redaction no longer
leaks raw passthrough prompts into Arize (Veria security finding).
- Split passthrough input/output rendering into helpers to satisfy PLR0915.
- Remove dead call_type assignment (F841).
Validated live against a Bedrock passthrough proxy exporting to Arize:
non-redacted renders the real prompt on litellm_request; global
turn_off_message_logging yields input.value=redacted-by-litellm with the
raw_gen_ai_request child span suppressed and no SSN/marker leakage.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>