tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-07-04 23:07:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

(docs) using openai compatible endpoints

Browse Source
This commit is contained in:
ishaan-jaff
2024-02-24 18:46:20 -08:00
parent 94046657be
commit a4d2d1c4e6
1 changed files with 27 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/docs/providers/openai_compatible.md
+27 -32
View File
@@ -6,42 +6,37 @@ To call models hosted behind an openai proxy, make 2 changes:
2. **Do NOT** add anything additional to the base url e.g. `/v1/embedding`. LiteLLM uses the openai-client to make these calls, and that automatically adds the relevant endpoints.
## Usage
## Usage - completion
```python
import litellm
import os
response = litellm.completion(
model="openai/mistral, # add `openai/` prefix to model so litellm knows to route to OpenAI
api_key="sk-1234", # api key to your openai compatible endpoint
api_base="http://0.0.0.0:8000", # set API Base of your Custom OpenAI Endpoint
messages=[
{
"role": "user",
"content": "Hey, how's it going?",
}
],
)
print(response)
```
## Usage - embedding
```python
import litellm
from litellm import embedding
litellm.set_verbose = True
import os
litellm_proxy_endpoint = "http://0.0.0.0:8000"
bearer_token = "sk-1234"
CHOSEN_LITE_LLM_EMBEDDING_MODEL = "openai/GPT-J 6B - Sagemaker Text Embedding (Internal)"
litellm.set_verbose = False
print(litellm_proxy_endpoint)
response = embedding(
model = CHOSEN_LITE_LLM_EMBEDDING_MODEL, # add `openai/` prefix to model so litellm knows to route to OpenAI
api_key=bearer_token,
api_base=litellm_proxy_endpoint, # set API Base of your Custom OpenAI Endpoint
input=["good morning from litellm"],
api_version='2023-07-01-preview'
response = litellm.embedding(
model="openai/GPT-J", # add `openai/` prefix to model so litellm knows to route to OpenAI
api_key="sk-1234", # api key to your openai compatible endpoint
api_base="http://0.0.0.0:8000", # set API Base of your Custom OpenAI Endpoint
input=["good morning from litellm"]
)
print('================================================')
print(len(response.data[0]['embedding']))
print(response)
```