docs(advisor): add how it works section with mermaid diagram + non-native provider table

2026-07-21 10:21:08 +00:00 · 2026-04-11 18:23:33 -07:00
parent 9be7b4c07c
commit a8bc7bfcd4
1 changed files with 83 additions and 6 deletions
@@ -12,14 +12,60 @@ The advisor tool is in beta. Include `anthropic-beta: advisor-tool-2026-03-01` i

 :::

+## How it works
+
+LiteLLM handles the advisor tool differently depending on the provider.
+
+**Anthropic API (native):** the advisor tool definition is forwarded as-is to Anthropic. Anthropic runs the advisor sub-inference server-side, inside the same `/v1/messages` call. No extra round-trips.
+
+**All other providers (OpenAI, Bedrock, Vertex, Groq, Mistral, …):** LiteLLM implements the orchestration loop itself via `AdvisorOrchestrationHandler`. The advisor tool is translated into a regular function tool the provider understands. When the executor calls it, LiteLLM intercepts, makes a separate sub-call to the advisor model (always `claude-opus-4-6`), injects the advice, and re-calls the executor — all transparently inside your original request.
+
+```mermaid
+flowchart TD
+    A["Client request — /messages or /chat/completions\ntools includes advisor_20260301"] --> B{"provider?"}
+
+    B -->|anthropic| C["Forward to Anthropic API\nadvisor runs server-side natively\nno extra round-trips"]
+
+    B -->|"openai / bedrock / vertex\ngroq / mistral / any other"| D["AdvisorOrchestrationHandler\nintercepts request"]
+
+    D --> E["Translate advisor_20260301\ninto a regular function tool"]
+
+    E --> F["EXECUTOR CALL\ne.g. openai/gpt-4.1-mini\nreceives synthetic advisor fn tool"]
+
+    F --> G{"executor\nstop_reason?"}
+
+    G -->|"tool_use name=advisor"| H{"iteration >\nmax_uses?"}
+
+    H -->|no| I["ADVISOR SUB-CALL\nclaude-opus-4-6\nno tools — full transcript forwarded"]
+
+    I --> J["Inject advisor advice\nas tool_result into message history"]
+
+    J --> F
+
+    H -->|yes| K["Raise AdvisorMaxIterationsError\n(caller can catch and handle)"]
+
+    G -->|"end_turn / other\nno advisor call"| L["Return clean final response\nno advisor blocks exposed to caller"]
+
+    C --> L
+```
+
+**Key properties of the non-native path:**
+
+- Executor always called non-streaming; streaming is emulated via `FakeAnthropicMessagesStreamIterator` on the final response.
+- The advisor sub-call uses no tools and receives the full conversation transcript.
+- `advisor_tool_result` blocks are stripped from message history before each executor call — providers like OpenAI never see Anthropic-specific block types.
+- `max_uses` is a hard cap: once exceeded, `AdvisorMaxIterationsError` is raised. Callers can catch this or set a high enough limit.
+- `max_uses=0` disables the advisor entirely — the first call raises immediately.
+
 ## Supported Providers

-| Provider | Chat Completions API | Messages API |
-|----------|---------------------|--------------|
-| **Anthropic API** | ✅ | ✅ |
-| **Azure Anthropic** | ❌ (coming soon) | ❌ (coming soon) |
-| **Google Cloud Vertex AI** | ❌ (coming soon) | ❌ (coming soon) |
-| **Amazon Bedrock** | ❌ (coming soon) | ❌ (coming soon) |
+| Provider | Chat Completions API | Messages API | Notes |
+|----------|---------------------|--------------|-------|
+| **Anthropic API** | ✅ | ✅ | Native — runs server-side |
+| **OpenAI / Azure OpenAI** | ✅ | ✅ | LiteLLM orchestration loop |
+| **Amazon Bedrock** | ✅ | ✅ | LiteLLM orchestration loop |
+| **Google Vertex AI** | ✅ | ✅ | LiteLLM orchestration loop |
+| **Groq / Mistral / others** | ✅ | ✅ | LiteLLM orchestration loop |

 ## Model Compatibility

@@ -305,6 +351,37 @@ response = client.beta.messages.create(
 print(response)
 ```

+#### Non-Anthropic Provider (LiteLLM orchestration loop)
+
+```python showLineNumbers title="Advisor Tool with OpenAI executor"
+import asyncio
+import litellm
+
+async def main():
+    # executor: openai/gpt-4.1-mini  |  advisor: claude-opus-4-6
+    # LiteLLM runs the orchestration loop automatically
+    response = await litellm.anthropic.messages.acreate(
+        model="openai/gpt-4.1-mini",
+        messages=[
+            {"role": "user", "content": "Implement a Python LRU cache with O(1) get and put."}
+        ],
+        tools=[
+            {
+                "type": "advisor_20260301",
+                "name": "advisor",
+                "model": "claude-opus-4-6",
+                "max_uses": 3,
+            }
+        ],
+        max_tokens=1024,
+        custom_llm_provider="openai",
+    )
+    # Final response is clean — no advisor tool_use blocks
+    print(response["content"][0]["text"])
+
+asyncio.run(main())
+```
+
 ---

 ## Response Structure