diff --git a/docs/my-website/docs/completion/anthropic_advisor_tool.md b/docs/my-website/docs/completion/anthropic_advisor_tool.md index 3cb87ffdf3..bff0ba2687 100644 --- a/docs/my-website/docs/completion/anthropic_advisor_tool.md +++ b/docs/my-website/docs/completion/anthropic_advisor_tool.md @@ -12,14 +12,60 @@ The advisor tool is in beta. Include `anthropic-beta: advisor-tool-2026-03-01` i ::: +## How it works + +LiteLLM handles the advisor tool differently depending on the provider. + +**Anthropic API (native):** the advisor tool definition is forwarded as-is to Anthropic. Anthropic runs the advisor sub-inference server-side, inside the same `/v1/messages` call. No extra round-trips. + +**All other providers (OpenAI, Bedrock, Vertex, Groq, Mistral, …):** LiteLLM implements the orchestration loop itself via `AdvisorOrchestrationHandler`. The advisor tool is translated into a regular function tool the provider understands. When the executor calls it, LiteLLM intercepts, makes a separate sub-call to the advisor model (always `claude-opus-4-6`), injects the advice, and re-calls the executor — all transparently inside your original request. + +```mermaid +flowchart TD + A["Client request — /messages or /chat/completions\ntools includes advisor_20260301"] --> B{"provider?"} + + B -->|anthropic| C["Forward to Anthropic API\nadvisor runs server-side natively\nno extra round-trips"] + + B -->|"openai / bedrock / vertex\ngroq / mistral / any other"| D["AdvisorOrchestrationHandler\nintercepts request"] + + D --> E["Translate advisor_20260301\ninto a regular function tool"] + + E --> F["EXECUTOR CALL\ne.g. openai/gpt-4.1-mini\nreceives synthetic advisor fn tool"] + + F --> G{"executor\nstop_reason?"} + + G -->|"tool_use name=advisor"| H{"iteration >\nmax_uses?"} + + H -->|no| I["ADVISOR SUB-CALL\nclaude-opus-4-6\nno tools — full transcript forwarded"] + + I --> J["Inject advisor advice\nas tool_result into message history"] + + J --> F + + H -->|yes| K["Raise AdvisorMaxIterationsError\n(caller can catch and handle)"] + + G -->|"end_turn / other\nno advisor call"| L["Return clean final response\nno advisor blocks exposed to caller"] + + C --> L +``` + +**Key properties of the non-native path:** + +- Executor always called non-streaming; streaming is emulated via `FakeAnthropicMessagesStreamIterator` on the final response. +- The advisor sub-call uses no tools and receives the full conversation transcript. +- `advisor_tool_result` blocks are stripped from message history before each executor call — providers like OpenAI never see Anthropic-specific block types. +- `max_uses` is a hard cap: once exceeded, `AdvisorMaxIterationsError` is raised. Callers can catch this or set a high enough limit. +- `max_uses=0` disables the advisor entirely — the first call raises immediately. + ## Supported Providers -| Provider | Chat Completions API | Messages API | -|----------|---------------------|--------------| -| **Anthropic API** | ✅ | ✅ | -| **Azure Anthropic** | ❌ (coming soon) | ❌ (coming soon) | -| **Google Cloud Vertex AI** | ❌ (coming soon) | ❌ (coming soon) | -| **Amazon Bedrock** | ❌ (coming soon) | ❌ (coming soon) | +| Provider | Chat Completions API | Messages API | Notes | +|----------|---------------------|--------------|-------| +| **Anthropic API** | ✅ | ✅ | Native — runs server-side | +| **OpenAI / Azure OpenAI** | ✅ | ✅ | LiteLLM orchestration loop | +| **Amazon Bedrock** | ✅ | ✅ | LiteLLM orchestration loop | +| **Google Vertex AI** | ✅ | ✅ | LiteLLM orchestration loop | +| **Groq / Mistral / others** | ✅ | ✅ | LiteLLM orchestration loop | ## Model Compatibility @@ -305,6 +351,37 @@ response = client.beta.messages.create( print(response) ``` +#### Non-Anthropic Provider (LiteLLM orchestration loop) + +```python showLineNumbers title="Advisor Tool with OpenAI executor" +import asyncio +import litellm + +async def main(): + # executor: openai/gpt-4.1-mini | advisor: claude-opus-4-6 + # LiteLLM runs the orchestration loop automatically + response = await litellm.anthropic.messages.acreate( + model="openai/gpt-4.1-mini", + messages=[ + {"role": "user", "content": "Implement a Python LRU cache with O(1) get and put."} + ], + tools=[ + { + "type": "advisor_20260301", + "name": "advisor", + "model": "claude-opus-4-6", + "max_uses": 3, + } + ], + max_tokens=1024, + custom_llm_provider="openai", + ) + # Final response is clean — no advisor tool_use blocks + print(response["content"][0]["text"]) + +asyncio.run(main()) +``` + --- ## Response Structure