mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-23 22:27:10 +00:00
(docs) hf viewing log probs
This commit is contained in:
@@ -139,6 +139,72 @@ os.environ["HUGGINGFACE_API_KEY"] = ""
|
||||
os.environ["HUGGINGFACE_API_BASE"] = ""
|
||||
```
|
||||
|
||||
### Viewing Log probs
|
||||
|
||||
#### Using `decoder_input_details` - OpenAI `echo`
|
||||
The `echo` param is supported by OpenAI Completions - Use `litellm.text_completion()` for this
|
||||
```python
|
||||
from litellm import text_completion
|
||||
response = text_completion(
|
||||
model="huggingface/bigcode/starcoder",
|
||||
prompt="good morning",
|
||||
max_tokens=10, logprobs=10,
|
||||
echo=True
|
||||
)
|
||||
```
|
||||
|
||||
#### Output
|
||||
```json
|
||||
{
|
||||
"id":"chatcmpl-3fc71792-c442-4ba1-a611-19dd0ac371ad",
|
||||
"object":"text_completion",
|
||||
"created":1698801125.936519,
|
||||
"model":"bigcode/starcoder",
|
||||
"choices":[
|
||||
{
|
||||
"text":", I'm going to make you a sand",
|
||||
"index":0,
|
||||
"logprobs":{
|
||||
"tokens":[
|
||||
"good",
|
||||
" morning",
|
||||
",",
|
||||
" I",
|
||||
"'m",
|
||||
" going",
|
||||
" to",
|
||||
" make",
|
||||
" you",
|
||||
" a",
|
||||
" s",
|
||||
"and"
|
||||
],
|
||||
"token_logprobs":[
|
||||
"None",
|
||||
-14.96875,
|
||||
-2.2285156,
|
||||
-2.734375,
|
||||
-2.0957031,
|
||||
-2.0917969,
|
||||
-0.09429932,
|
||||
-3.1132812,
|
||||
-1.3203125,
|
||||
-1.2304688,
|
||||
-1.6201172,
|
||||
-0.010292053
|
||||
]
|
||||
},
|
||||
"finish_reason":"length"
|
||||
}
|
||||
],
|
||||
"usage":{
|
||||
"completion_tokens":9,
|
||||
"prompt_tokens":2,
|
||||
"total_tokens":11
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Models with Prompt Formatting
|
||||
For models with special prompt templates (e.g. Llama2), we format the prompt to fit their template.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user