mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-10 22:24:51 +00:00
docs add section on bedrock provider to show how to use KBs
This commit is contained in:
@@ -0,0 +1,144 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# Bedrock Knowledge Bases
|
||||
|
||||
AWS Bedrock Knowledge Bases allows you to connect your LLM's to your organization's data, letting your models retrieve and reference information specific to your business.
|
||||
|
||||
| Property | Details |
|
||||
|----------|---------|
|
||||
| Description | Bedrock Knowledge Bases connects your data to LLM's, enabling them to retrieve and reference your organization's information in their responses. |
|
||||
| Provider Route on LiteLLM | `bedrock` in the litellm vector_store_registry |
|
||||
| Provider Doc | [AWS Bedrock Knowledge Bases ↗](https://aws.amazon.com/bedrock/knowledge-bases/) |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### LiteLLM Python SDK
|
||||
|
||||
```python showLineNumbers title="Example using LiteLLM Python SDK"
|
||||
import os
|
||||
import litellm
|
||||
|
||||
from litellm.vector_stores.vector_store_registry import VectorStoreRegistry, LiteLLM_ManagedVectorStore
|
||||
|
||||
# Init vector store registry with your Bedrock Knowledge Base
|
||||
litellm.vector_store_registry = VectorStoreRegistry(
|
||||
vector_stores=[
|
||||
LiteLLM_ManagedVectorStore(
|
||||
vector_store_id="YOUR_KNOWLEDGE_BASE_ID", # KB ID from AWS Bedrock
|
||||
custom_llm_provider="bedrock"
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
# Make a completion request using your Knowledge Base
|
||||
response = await litellm.acompletion(
|
||||
model="anthropic/claude-3-5-sonnet",
|
||||
messages=[{"role": "user", "content": "What does our company policy say about remote work?"}],
|
||||
tools=[
|
||||
{
|
||||
"type": "file_search",
|
||||
"vector_store_ids": ["YOUR_KNOWLEDGE_BASE_ID"]
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### LiteLLM Proxy
|
||||
|
||||
#### 1. Configure your vector_store_registry
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="config-yaml" label="config.yaml">
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: claude-3-5-sonnet
|
||||
litellm_params:
|
||||
model: anthropic/claude-3-5-sonnet
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
vector_store_registry:
|
||||
- vector_store_name: "bedrock-company-docs"
|
||||
litellm_params:
|
||||
vector_store_id: "YOUR_KNOWLEDGE_BASE_ID"
|
||||
custom_llm_provider: "bedrock"
|
||||
vector_store_description: "Bedrock Knowledge Base for company documents"
|
||||
vector_store_metadata:
|
||||
source: "Company internal documentation"
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="litellm-ui" label="LiteLLM UI">
|
||||
|
||||
On the LiteLLM UI, Navigate to Experimental > Vector Stores > Create Vector Store. On this page you can create a vector store with a name, vector store id and credentials.
|
||||
|
||||
<Image
|
||||
img={require('../../img/kb_2.png')}
|
||||
style={{width: '50%'}}
|
||||
/>
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
#### 2. Make a request with vector_store_ids parameter
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="Curl">
|
||||
|
||||
```bash
|
||||
curl http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $LITELLM_API_KEY" \
|
||||
-d '{
|
||||
"model": "claude-3-5-sonnet",
|
||||
"messages": [{"role": "user", "content": "What does our company policy say about remote work?"}],
|
||||
"tools": [
|
||||
{
|
||||
"type": "file_search",
|
||||
"vector_store_ids": ["YOUR_KNOWLEDGE_BASE_ID"]
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="openai-sdk" label="OpenAI Python SDK">
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
# Initialize client with your LiteLLM proxy URL
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:4000",
|
||||
api_key="your-litellm-api-key"
|
||||
)
|
||||
|
||||
# Make a completion request with vector_store_ids parameter
|
||||
response = client.chat.completions.create(
|
||||
model="claude-3-5-sonnet",
|
||||
messages=[{"role": "user", "content": "What does our company policy say about remote work?"}],
|
||||
tools=[
|
||||
{
|
||||
"type": "file_search",
|
||||
"vector_store_ids": ["YOUR_KNOWLEDGE_BASE_ID"]
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
Futher Reading Vector Stores:
|
||||
- [Always on Vector Stores](https://docs.litellm.ai/docs/completion/knowledgebase#always-on-for-a-model)
|
||||
- [Listing available vector stores on litellm proxy](https://docs.litellm.ai/docs/completion/knowledgebase#listing-available-vector-stores)
|
||||
- [How LiteLLM Vector Stores Work](https://docs.litellm.ai/docs/completion/knowledgebase#how-it-works)
|
||||
@@ -2,7 +2,7 @@ import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Bedrock
|
||||
# Bedrock Guardrails
|
||||
|
||||
LiteLLM supports Bedrock guardrails via the [Bedrock ApplyGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html).
|
||||
|
||||
|
||||
@@ -208,7 +208,14 @@ const sidebars = {
|
||||
},
|
||||
"providers/anthropic",
|
||||
"providers/aws_sagemaker",
|
||||
"providers/bedrock",
|
||||
{
|
||||
type: "category",
|
||||
label: "Bedrock",
|
||||
items: [
|
||||
"providers/bedrock",
|
||||
"providers/bedrock_vector_store",
|
||||
]
|
||||
},
|
||||
"providers/litellm_proxy",
|
||||
"providers/mistral",
|
||||
"providers/codestral",
|
||||
|
||||
Reference in New Issue
Block a user