tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-08 22:25:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

(docs) setting cache paras on proxy + Openai client

Browse Source
This commit is contained in:
ishaan-jaff
2024-02-22 13:17:32 -08:00
parent ede57d4ac2
commit 580f9e3097
1 changed files with 16 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/docs/proxy/caching.md
+16 -10
View File
@@ -238,9 +238,11 @@ chat_completion = client.chat.completions.create(
}
],
model="gpt-3.5-turbo",
cache={
"no-cache": True # will not return a cached response
}
extra_body = { # OpenAI python accepts extra args in extra_body
cache: {
"no-cache": True # will not return a cached response
}
}
)
```
@@ -264,9 +266,11 @@ chat_completion = client.chat.completions.create(
}
],
model="gpt-3.5-turbo",
cache={
"ttl": 600 # caches response for 10 minutes
}
extra_body = { # OpenAI python accepts extra args in extra_body
cache: {
"ttl": 600 # caches response for 10 minutes
}
}
)
```
@@ -288,13 +292,15 @@ chat_completion = client.chat.completions.create(
}
],
model="gpt-3.5-turbo",
cache={
"s-maxage": 600 # only get responses cached within last 10 minutes
}
extra_body = { # OpenAI python accepts extra args in extra_body
cache: {
"s-maxage": 600 # only get responses cached within last 10 minutes
}
}
)
```
## Supported `cache_params`
## Supported `cache_params` on proxy config.yaml
```yaml
cache_params: