---
title: HuggingFace Inference API
description: 'Configure Danswer to use HuggingFace APIs'
---

Refer to [Model Configs](https://docs.danswer.dev/gen_ai_configs/overview#model-configs) for how to set the
environment variables for your particular deployment.

To use the HuggingFace Inference APIs, you must sign up for a `Pro Account` to get an API Key
1. After signing up for `Pro Account`, go to your user settings:

![HFSettings](/images/gen_ai/HFSettings.png)

2. Copy the `User Access Token`

![HFAccessToken](/images/gen_ai/HFAccessToken.png)

## Terminology Disambiguation
In Danswer, a "chat" model refers to using a series of messages with attached roles such as "system", "assistant", or
"user". This is consistent with the OpenAI terminology.

HuggingFace uses "chat" in reference to a model that is finetuned to follow instructions.
They use "conversational" to mean the equivalent of OpenAI's "chat".


## Set Danswer to use `Llama-2-70B` via next-token generation prompting
- INTERNAL_MODEL_VERSION=huggingface-client-completion
- GEN_AI_MODEL_VERSION=meta-llama/Llama-2-70b-chat-hf
    - Note, "chat" in the above refers to instruction-fine-tuning
    - As of Aug 2023, this model does not support conversational prompting
- GEN_AI_API_KEY=&lt;your-huggingface-access-token&gt;
    - You can also leave this unset and set it later via the UI


## Set Danswer to use `Llama-2-70B` via chat (conversational) prompting
- INTERNAL_MODEL_VERSION=huggingface-client-chat-completion
- GEN_AI_MODEL_VERSION=meta-llama/Llama-2-70b-hf
    - As of Aug 2023, only the non-instruction-finetuned model support conversational prompting
- GEN_AI_API_KEY=&lt;your-huggingface-access-token&gt;
    - You can also leave this unset and set it later via the UI
