---
title: LLM Model via REST API
description: 'Configure Danswer to use a custom Model Server'
---

Danswer can also make requests to an arbitrary model server via rest requests. Optionally an access token can be passed
in. To customize the request format and handling of the response, it may be necessary to update/rebuild the Danswer
containers.

## Extending Danswer to be compatible with your custom model server
There is a default implementation which follows the HuggingFace `Inference Endpoint` API.

If instead, calling to a custom model server, there's a very minimal interface to be implemented with examples found
[here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/direct_qa/request_model.py#L31).

If going with a custom implementation, be sure to register it in the code where `ModelHostType` is mentioned.
Refer to the code [here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/configs/constants.py#L49).

## Blog on using Danswer with a self hosted `Llama-2-13B-chat-GGML`
- [https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183](https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183)
- This demo uses Google Colab to access a free GPU but this is not suitable for long term deployments

## Set Danswer to use an LLM behind a REST API
There is an offering from HuggingFace called "Inference Endpoints" where users can rent dedicated hardware and host
HuggingFace compatible models behind a REST API.
Danswer works out of the box with any text-generation HuggingFace models hosted this way.
- INTERNAL_MODEL_VERSION=request-completion
- GEN_AI_HOST_TYPE=huggingface
    - or reference your custom class
- GEN_AI_ENDPOINT=&lt;your-huggingface-inference-endpoint-url&gt;

