mirror of
https://github.com/tiennm99/danswer-docs.git
synced 2026-08-09 22:21:50 +00:00
32 lines
1.7 KiB
Plaintext
32 lines
1.7 KiB
Plaintext
---
|
|
title: LLM Model via REST API
|
|
description: 'Configure Danswer to use a custom Model Server'
|
|
---
|
|
|
|
Danswer can also make requests to an arbitrary model server via rest requests. Optionally an access token can be passed
|
|
in. To customize the request format and handling of the response, it may be necessary to update/rebuild the Danswer
|
|
containers.
|
|
|
|
## Extending Danswer to be compatible with your custom model server
|
|
There is a default implementation which follows the HuggingFace `Inference Endpoint` API.
|
|
|
|
If instead, calling to a custom model server, there's a very minimal interface to be implemented with examples found
|
|
[here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/direct_qa/request_model.py#L31).
|
|
|
|
If going with a custom implementation, be sure to register it in the code where `ModelHostType` is mentioned.
|
|
Refer to the code [here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/configs/constants.py#L49).
|
|
|
|
## Blog on using Danswer with a self hosted `Llama-2-13B-chat-GGML`
|
|
- [https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183](https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183)
|
|
- This demo uses Google Colab to access a free GPU but this is not suitable for long term deployments
|
|
|
|
## Set Danswer to use an LLM behind a REST API
|
|
There is an offering from HuggingFace called "Inference Endpoints" where users can rent dedicated hardware and host
|
|
HuggingFace compatible models behind a REST API.
|
|
Danswer works out of the box with any text-generation HuggingFace models hosted this way.
|
|
- INTERNAL_MODEL_VERSION=request-completion
|
|
- GEN_AI_HOST_TYPE=huggingface
|
|
- or reference your custom class
|
|
- GEN_AI_ENDPOINT=<your-huggingface-inference-endpoint-url>
|
|
|