mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-18 04:28:19 +00:00
Add CompactifAI provider documentation and config
- Create comprehensive provider documentation with usage examples - Cover basic completion, streaming, async, and function calling - Document AWS Marketplace subscription and API key setup process - Include proxy configuration and advanced parameter examples - Add error handling examples and model information - Update website sidebar to include CompactifAI in provider list - Update README.md with CompactifAI provider reference
This commit is contained in:
Tim Elfrink
@@ -316,6 +316,7 @@ curl 'http://0.0.0.0:4000/key/generate' \
|
||||
| [google AI Studio - gemini](https://docs.litellm.ai/docs/providers/gemini) | ✅ | ✅ | ✅ | ✅ | | |
|
||||
| [mistral ai api](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | ✅ | ✅ | ✅ | |
|
||||
| [cloudflare AI Workers](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ | ✅ | | |
|
||||
| [CompactifAI](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | ✅ | ✅ | ✅ | | |
|
||||
| [cohere](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | ✅ | ✅ | |
|
||||
| [anthropic](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | ✅ | | |
|
||||
| [empower](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# CompactifAI
|
||||
https://docs.compactif.ai/
|
||||
|
||||
CompactifAI offers highly compressed versions of leading language models, delivering up to **70% lower inference costs**, **4x throughput gains**, and **low-latency inference** with minimal quality loss (<5%). CompactifAI's OpenAI-compatible API makes integration straightforward, enabling developers to build ultra-efficient, scalable AI applications with superior concurrency and resource efficiency.
|
||||
|
||||
| Property | Details |
|
||||
|-------|-------|
|
||||
| Description | CompactifAI offers compressed versions of leading language models with up to 70% cost reduction and 4x throughput gains |
|
||||
| Provider Route on LiteLLM | `compactifai/` (add this prefix to the model name - e.g. `compactifai/llama-2-7b-compressed`) |
|
||||
| Provider Doc | [CompactifAI ↗](https://docs.compactif.ai/) |
|
||||
| API Endpoint for Provider | https://api.compactif.ai/v1 |
|
||||
| Supported Endpoints | `/chat/completions`, `/completions` |
|
||||
|
||||
## Supported OpenAI Parameters
|
||||
|
||||
CompactifAI is fully OpenAI-compatible and supports the following parameters:
|
||||
|
||||
```
|
||||
"stream",
|
||||
"stop",
|
||||
"temperature",
|
||||
"top_p",
|
||||
"max_tokens",
|
||||
"presence_penalty",
|
||||
"frequency_penalty",
|
||||
"logit_bias",
|
||||
"user",
|
||||
"response_format",
|
||||
"seed",
|
||||
"tools",
|
||||
"tool_choice",
|
||||
"parallel_tool_calls",
|
||||
"extra_headers"
|
||||
```
|
||||
|
||||
## API Key Setup
|
||||
|
||||
CompactifAI API keys are available through AWS Marketplace subscription:
|
||||
|
||||
1. Subscribe via [AWS Marketplace](https://aws.amazon.com/marketplace)
|
||||
2. Complete subscription verification (24-hour review process)
|
||||
3. Access MultiverseIAM dashboard with provided credentials
|
||||
4. Retrieve your API key from the dashboard
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["COMPACTIFAI_API_KEY"] = "your-api-key"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
os.environ['COMPACTIFAI_API_KEY'] = "your-api-key"
|
||||
|
||||
response = completion(
|
||||
model="compactifai/llama-2-7b-compressed",
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello from LiteLLM!"}
|
||||
],
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="proxy" label="Proxy">
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: llama-2-compressed
|
||||
litellm_params:
|
||||
model: compactifai/llama-2-7b-compressed
|
||||
api_key: os.environ/COMPACTIFAI_API_KEY
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Streaming
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
os.environ['COMPACTIFAI_API_KEY'] = "your-api-key"
|
||||
|
||||
response = completion(
|
||||
model="compactifai/llama-2-7b-compressed",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a short story"}
|
||||
],
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
print(chunk)
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Parameters
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
|
||||
response = completion(
|
||||
model="compactifai/llama-2-7b-compressed",
|
||||
messages=[{"role": "user", "content": "Explain quantum computing"}],
|
||||
temperature=0.7,
|
||||
max_tokens=500,
|
||||
top_p=0.9,
|
||||
stop=["Human:", "AI:"]
|
||||
)
|
||||
```
|
||||
|
||||
### Function Calling
|
||||
|
||||
CompactifAI supports OpenAI-compatible function calling:
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
|
||||
functions = [
|
||||
{
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather information",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = completion(
|
||||
model="compactifai/llama-2-7b-compressed",
|
||||
messages=[{"role": "user", "content": "What's the weather in San Francisco?"}],
|
||||
tools=[{"type": "function", "function": f} for f in functions],
|
||||
tool_choice="auto"
|
||||
)
|
||||
```
|
||||
|
||||
### Async Usage
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from litellm import acompletion
|
||||
|
||||
async def async_call():
|
||||
response = await acompletion(
|
||||
model="compactifai/llama-2-7b-compressed",
|
||||
messages=[{"role": "user", "content": "Hello async world!"}]
|
||||
)
|
||||
return response
|
||||
|
||||
# Run async function
|
||||
response = asyncio.run(async_call())
|
||||
print(response)
|
||||
```
|
||||
|
||||
## Available Models
|
||||
|
||||
CompactifAI offers compressed versions of popular models. Use the `/models` endpoint to get the latest list:
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
headers = {"Authorization": f"Bearer {your_api_key}"}
|
||||
response = httpx.get("https://api.compactif.ai/v1/models", headers=headers)
|
||||
models = response.json()
|
||||
```
|
||||
|
||||
Common model formats:
|
||||
- `compactifai/llama-2-7b-compressed`
|
||||
- `compactifai/mistral-7b-compressed`
|
||||
- `compactifai/codellama-7b-compressed`
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Cost Efficient**: Up to 70% lower inference costs compared to standard models
|
||||
- **High Performance**: 4x throughput gains with minimal quality loss (<5%)
|
||||
- **Low Latency**: Optimized for fast response times
|
||||
- **Drop-in Replacement**: Full OpenAI API compatibility
|
||||
- **Scalable**: Superior concurrency and resource efficiency
|
||||
|
||||
## Error Handling
|
||||
|
||||
CompactifAI returns standard OpenAI-compatible error responses:
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
from litellm.exceptions import AuthenticationError, RateLimitError
|
||||
|
||||
try:
|
||||
response = completion(
|
||||
model="compactifai/llama-2-7b-compressed",
|
||||
messages=[{"role": "user", "content": "Hello"}]
|
||||
)
|
||||
except AuthenticationError:
|
||||
print("Invalid API key")
|
||||
except RateLimitError:
|
||||
print("Rate limit exceeded")
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://docs.compactif.ai/
|
||||
- LinkedIn: [MultiverseComputing](https://www.linkedin.com/company/multiversecomputing)
|
||||
- Analysis: [Artificial Analysis Provider Comparison](https://artificialanalysis.ai/providers/compactifai)
|
||||
@@ -451,6 +451,7 @@ const sidebars = {
|
||||
"providers/elevenlabs",
|
||||
"providers/fireworks_ai",
|
||||
"providers/clarifai",
|
||||
"providers/compactifai",
|
||||
"providers/vllm",
|
||||
"providers/llamafile",
|
||||
"providers/infinity",
|
||||
|
||||
Reference in New Issue
Block a user