mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-06 20:25:29 +00:00
docs(advisor): move supported providers to top, focus how it works on litellm native loop
This commit is contained in:
@@ -12,51 +12,6 @@ The advisor tool is in beta. Include `anthropic-beta: advisor-tool-2026-03-01` i
|
||||
|
||||
:::
|
||||
|
||||
## How it works
|
||||
|
||||
LiteLLM handles the advisor tool differently depending on the provider.
|
||||
|
||||
**Anthropic API (native):** the advisor tool definition is forwarded as-is to Anthropic. Anthropic runs the advisor sub-inference server-side, inside the same `/v1/messages` call. No extra round-trips.
|
||||
|
||||
**All other providers (OpenAI, Bedrock, Vertex, Groq, Mistral, …):** LiteLLM implements the orchestration loop itself via `AdvisorOrchestrationHandler`. The advisor tool is translated into a regular function tool the provider understands. When the executor calls it, LiteLLM intercepts, makes a separate sub-call to the advisor model (always `claude-opus-4-6`), injects the advice, and re-calls the executor — all transparently inside your original request.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Client request — /messages or /chat/completions\ntools includes advisor_20260301"] --> B{"provider?"}
|
||||
|
||||
B -->|anthropic| C["Forward to Anthropic API\nadvisor runs server-side natively\nno extra round-trips"]
|
||||
|
||||
B -->|"openai / bedrock / vertex\ngroq / mistral / any other"| D["AdvisorOrchestrationHandler\nintercepts request"]
|
||||
|
||||
D --> E["Translate advisor_20260301\ninto a regular function tool"]
|
||||
|
||||
E --> F["EXECUTOR CALL\ne.g. openai/gpt-4.1-mini\nreceives synthetic advisor fn tool"]
|
||||
|
||||
F --> G{"executor\nstop_reason?"}
|
||||
|
||||
G -->|"tool_use name=advisor"| H{"iteration >\nmax_uses?"}
|
||||
|
||||
H -->|no| I["ADVISOR SUB-CALL\nclaude-opus-4-6\nno tools — full transcript forwarded"]
|
||||
|
||||
I --> J["Inject advisor advice\nas tool_result into message history"]
|
||||
|
||||
J --> F
|
||||
|
||||
H -->|yes| K["Raise AdvisorMaxIterationsError\n(caller can catch and handle)"]
|
||||
|
||||
G -->|"end_turn / other\nno advisor call"| L["Return clean final response\nno advisor blocks exposed to caller"]
|
||||
|
||||
C --> L
|
||||
```
|
||||
|
||||
**Key properties of the non-native path:**
|
||||
|
||||
- Executor always called non-streaming; streaming is emulated via `FakeAnthropicMessagesStreamIterator` on the final response.
|
||||
- The advisor sub-call uses no tools and receives the full conversation transcript.
|
||||
- `advisor_tool_result` blocks are stripped from message history before each executor call — providers like OpenAI never see Anthropic-specific block types.
|
||||
- `max_uses` is a hard cap: once exceeded, `AdvisorMaxIterationsError` is raised. Callers can catch this or set a high enough limit.
|
||||
- `max_uses=0` disables the advisor entirely — the first call raises immediately.
|
||||
|
||||
## Supported Providers
|
||||
|
||||
| Provider | Chat Completions API | Messages API | Notes |
|
||||
@@ -67,6 +22,41 @@ flowchart TD
|
||||
| **Google Vertex AI** | ✅ | ✅ | LiteLLM orchestration loop |
|
||||
| **Groq / Mistral / others** | ✅ | ✅ | LiteLLM orchestration loop |
|
||||
|
||||
## How it works (LiteLLM native orchestration)
|
||||
|
||||
For non-Anthropic providers, LiteLLM implements the advisor loop itself. The API you call is identical — LiteLLM handles everything transparently.
|
||||
|
||||
When a request arrives with an `advisor_20260301` tool and a non-Anthropic provider, `AdvisorOrchestrationHandler` intercepts it. It translates the advisor tool into a regular function tool the provider understands, then runs an orchestration loop:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Your request\ntools: advisor_20260301\nmodel: e.g. openai/gpt-4.1-mini"] --> B["AdvisorOrchestrationHandler\ntranslates advisor → regular fn tool"]
|
||||
|
||||
B --> C["EXECUTOR CALL\nopenai / bedrock / vertex / etc."]
|
||||
|
||||
C --> D{"executor calls\nadvisor tool?"}
|
||||
|
||||
D -->|"yes — tool_use\nname=advisor"| E{"max_uses\nexceeded?"}
|
||||
|
||||
E -->|no| F["ADVISOR SUB-CALL\nclaude-opus-4-6\nfull transcript forwarded\nno tools"]
|
||||
|
||||
F --> G["Inject advice as\ntool_result into history"]
|
||||
|
||||
G --> C
|
||||
|
||||
E -->|yes| H["AdvisorMaxIterationsError"]
|
||||
|
||||
D -->|"no — end_turn\nor other stop reason"| I["Clean final response\nno advisor blocks in output"]
|
||||
```
|
||||
|
||||
**What LiteLLM does for you:**
|
||||
|
||||
- Strips `advisor_20260301` from the outgoing request — the provider only sees a standard function tool named `advisor`
|
||||
- When the executor calls it, intercepts before the result reaches you, runs the advisor sub-call, and injects the advice
|
||||
- Strips any `advisor_tool_result` / `server_tool_use` blocks from message history on re-send so non-Anthropic providers never see Anthropic-specific types
|
||||
- Wraps the final response in an SSE stream if you requested `stream=True`
|
||||
- Enforces `max_uses` as a hard cap — `AdvisorMaxIterationsError` is raised if exceeded; `max_uses=0` disables the advisor entirely
|
||||
|
||||
## Model Compatibility
|
||||
|
||||
The executor and advisor models must form a valid pair. Currently the only supported advisor model is `claude-opus-4-6`.
|
||||
|
||||
Reference in New Issue
Block a user