tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-07-15 06:19:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
553b993473e1cfc60fae5dfee0ef38fd0c052d7b
litellm/cookbook/LiteLLM_HuggingFace.ipynb
T
ishaan-jaff 361bf02e53 update cookbook
2023-09-27 20:19:38 -07:00

29 KiB
Vendored
Raw Blame History

LiteLLM HuggingFace

Docs for huggingface: https://docs.litellm.ai/docs/providers/huggingface

In [ ]:
!pip install litellm

HuggingFace TGI Model - Deployed Inference Endpoints

Steps to use

  • set api_base to your deployed api base
  • Add the huggingface/ prefix to your model so litellm knows it's a huggingface Deployed Inference Endpoint
In [9]:
import os
import litellm

os.environ["HUGGINGFACE_API_KEY"] = ""

# TGI model: Call https://huggingface.co/glaiveai/glaive-coder-7b
# add the 'huggingface/' prefix to the model to set huggingface as the provider
# set api base to your deployed api endpoint from hugging face
response = litellm.completion(
    model="huggingface/glaiveai/glaive-coder-7b",
    messages=[{ "content": "Hello, how are you?","role": "user"}],
    api_base="https://wjiegasee9bmqke2.us-east-1.aws.endpoints.huggingface.cloud"
)
print(response)
{
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "\n\nI am doing well, thank you for asking. How about you?\nI am doing",
        "role": "assistant",
        "logprobs": -8.9481967812
      }
    }
  ],
  "id": "chatcmpl-74dc9d89-3916-47ce-9bea-b80e66660f77",
  "created": 1695871068.8413374,
  "model": "glaiveai/glaive-coder-7b",
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 18,
    "total_tokens": 24
  }
}

HuggingFace Non TGI/Non Conversational Model - Deployed Inference Endpoints

  • set api_base to your deployed api base
  • Add the huggingface/ prefix to your model so litellm knows it's a huggingface Deployed Inference Endpoint
In [6]:
import os
import litellm

os.environ["HUGGINGFACE_API_KEY"] = ""
#  model: https://huggingface.co/roneneldan/TinyStories-3M
# add the 'huggingface/' prefix to the model to set huggingface as the provider
# set api base to your deployed api endpoint from hugging face
response = litellm.completion(
            model="huggingface/roneneldan/TinyStories-3M",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            api_base="https://p69xlsj6rpno5drq.us-east-1.aws.endpoints.huggingface.cloud",
        )
print(response)

{
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Hello, how are you? I have a surprise for you. I have a surprise for you.",
        "role": "assistant",
        "logprobs": null
      }
    }
  ],
  "id": "chatcmpl-6035abd6-7753-4a7d-ba0a-8193522e23cf",
  "created": 1695871015.0468287,
  "model": "roneneldan/TinyStories-3M",
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 20,
    "total_tokens": 26
  }
}

Hugging Face Free Inference API

When API base is not set it defaults to sending requests to https://api-inference.huggingface.co/models/

In order to use litellm to call hugging face inference api llms

  • Copy the model name from hugging face
  • set model = "huggingface/<model-name>"

Example set model=huggingface/bigcode/starcoder to call bigcode/starcoder

https://huggingface.co/bigcode/starcoder

In [ ]:
import os
import litellm

os.environ["HUGGINGFACE_API_KEY"] = ""

# Call https://huggingface.co/bigcode/starcoder
# add the 'huggingface/' prefix to the model to set huggingface as the provider
response = litellm.completion(
    model="huggingface/bigcode/starcoder",
    messages=[{ "content": "Hello, how are you?","role": "user"}]
)
print(response)


# Call https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf
response = litellm.completion(
    model="huggingface/codellama/CodeLlama-34b-Instruct-hf",
    messages=[{ "content": "Hello, how are you?","role": "user"}]
)
print(response)
{
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": " I am fine, thank you. And you?')\nprint(result)\n\n# 2",
        "role": "assistant",
        "logprobs": null
      }
    }
  ],
  "id": "chatcmpl-982e4cd0-9779-4108-9f7e-d6cbf9b71516",
  "created": 1695835548.2239568,
  "model": "bigcode/starcoder",
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 17,
    "total_tokens": 23
  }
}
{
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Hello! I'm doing well, thank you for asking. It's nice to meet you",
        "role": "assistant",
        "logprobs": null
      }
    }
  ],
  "id": "chatcmpl-6622d64d-e9fc-4a46-9ca7-b2d011f6968c",
  "created": 1695835549.2932954,
  "model": "codellama/CodeLlama-34b-Instruct-hf",
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 18,
    "total_tokens": 30
  }
}

HuggingFace - Deployed Inference Endpoints + Streaming

Set stream = True

In [ ]:
import os
import litellm

os.environ["HUGGINGFACE_API_KEY"] = ""

# Call https://huggingface.co/glaiveai/glaive-coder-7b
# add the 'huggingface/' prefix to the model to set huggingface as the provider
# set api base to your deployed api endpoint from hugging face
response = litellm.completion(
    model="huggingface/aws-glaive-coder-7b-0998",
    messages=[{ "content": "Hello, how are you?","role": "user"}],
    api_base="https://wjiegasee9bmqke2.us-east-1.aws.endpoints.huggingface.cloud",
    stream=True
)
print(response)

for chunk in response:
  print(chunk)
<litellm.utils.CustomStreamWrapper object at 0x7d1364efa650>
data json: {'token': {'id': 13, 'text': '\n', 'logprob': -1.4355469, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "\n",
        "role": "assistant"
      }
    }
  ],
  "id": "chatcmpl-b581bf7e-e20d-46fd-9ca0-b38870db3f3c",
  "created": 1695837652,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 13, 'text': '\n', 'logprob': -1.9277344, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "\n"
      }
    }
  ],
  "id": "chatcmpl-49c7b630-ec07-4390-ae22-bbb068ac66aa",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 29902, 'text': 'I', 'logprob': -1.4570312, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "I"
      }
    }
  ],
  "id": "chatcmpl-6b1635f3-810a-4976-b603-2c47a9525fff",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 626, 'text': ' am', 'logprob': -0.70703125, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " am"
      }
    }
  ],
  "id": "chatcmpl-dcfad593-6c02-4f4c-abdb-3027ccda80e1",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 2599, 'text': ' doing', 'logprob': -1.0107422, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " doing"
      }
    }
  ],
  "id": "chatcmpl-fcdd4076-7907-44f9-8ebf-a462b90e076c",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 1532, 'text': ' well', 'logprob': -0.43603516, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " well"
      }
    }
  ],
  "id": "chatcmpl-c3e521f8-5cec-4a65-908a-f6678a635806",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 29892, 'text': ',', 'logprob': -0.08898926, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": ","
      }
    }
  ],
  "id": "chatcmpl-c32355fb-94cc-43c8-9213-71ff387dc636",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 6452, 'text': ' thank', 'logprob': -0.19006348, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " thank"
      }
    }
  ],
  "id": "chatcmpl-0ea8172e-adcf-4bcc-b919-df675d3def85",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 366, 'text': ' you', 'logprob': -0.0012788773, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " you"
      }
    }
  ],
  "id": "chatcmpl-9c9fc627-fec9-454e-a630-6f8a2291b662",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 363, 'text': ' for', 'logprob': -0.026885986, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " for"
      }
    }
  ],
  "id": "chatcmpl-fd32fd74-6cac-4ddb-83d1-205810ab897a",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 6721, 'text': ' asking', 'logprob': -0.035705566, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " asking"
      }
    }
  ],
  "id": "chatcmpl-2ff7ab0a-d574-4e83-804f-8505be94712a",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 29889, 'text': '.', 'logprob': -0.07635498, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "."
      }
    }
  ],
  "id": "chatcmpl-488d6bba-23bf-4383-bced-3cc0fbad3b17",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 1128, 'text': ' How', 'logprob': -0.46557617, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " How"
      }
    }
  ],
  "id": "chatcmpl-078c7ce3-e748-4fd7-bf11-e1389e23e0ef",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 1048, 'text': ' about', 'logprob': -0.068359375, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " about"
      }
    }
  ],
  "id": "chatcmpl-d6b1e355-edf1-4486-8567-d4b5bbfd7d74",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 366, 'text': ' you', 'logprob': -0.0006146431, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " you"
      }
    }
  ],
  "id": "chatcmpl-d2633371-0235-4a31-9621-4a9ec9b587a8",
  "created": 1695837653,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 29973, 'text': '?', 'logprob': -0.0001667738, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "?"
      }
    }
  ],
  "id": "chatcmpl-70ccf462-e915-465a-bb86-46f5a244451e",
  "created": 1695837654,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 13, 'text': '\n', 'logprob': -0.03363037, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "\n"
      }
    }
  ],
  "id": "chatcmpl-31d947e4-71df-4954-9d1a-8e68464da879",
  "created": 1695837654,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 29902, 'text': 'I', 'logprob': -0.17321777, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "I"
      }
    }
  ],
  "id": "chatcmpl-af84f782-a682-4660-86ba-da1ad71f5c93",
  "created": 1695837654,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 626, 'text': ' am', 'logprob': -0.38891602, 'special': False}, 'generated_text': None, 'details': None}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": " am"
      }
    }
  ],
  "id": "chatcmpl-3d4e01aa-4d56-4b98-be8f-8f9a6a4e0856",
  "created": 1695837654,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
data json: {'token': {'id': 2599, 'text': ' doing', 'logprob': -0.4243164, 'special': False}, 'generated_text': '\n\nI am doing well, thank you for asking. How about you?\nI am doing', 'details': {'finish_reason': 'length', 'generated_tokens': 20, 'seed': None}}
{
  "object": "chat.completion.chunk",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "delta": {
        "content": " doing"
      }
    }
  ],
  "id": "chatcmpl-18583fad-c957-432e-9a62-5620620271a2",
  "created": 1695837654,
  "model": "aws-glaive-coder-7b-0998",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}
In [ ]:
Reference in New Issue View Git Blame Copy Permalink