mirror of
https://github.com/tiennm99/danswer-docs.git
synced 2026-09-10 06:20:23 +00:00
59 lines
3.5 KiB
Plaintext
59 lines
3.5 KiB
Plaintext
---
|
|
title: Overview
|
|
description: 'Overview of the Generative AI functionality in Danswer'
|
|
---
|
|
|
|
## What are Generative AI (LLM) models used for?
|
|
The Large Language Models are used to interpret the contents from the most relevant documents retrieved via Search.
|
|
These models extract out the useful knowledge from your documents and generates the **AI Answer**.
|
|
|
|
## What is the default LLM?
|
|
By default, Danswer uses `GPT-3.5-Turbo` from OpenAI. This is the most accessible model/hosting service since it does
|
|
not require any access approval process like `GPT-4` or `Llama2` variants.
|
|
OpenAI also hosts the models behind an API which makes it easy to use and much more cost efficient than hosting a model
|
|
yourself on dedicated hardware.
|
|
|
|
## Why would you want to use a different model
|
|
- Use a more powerful model despite higher cost (such as GPT-4)
|
|
- Use a hosting service with a different data retention policy
|
|
- Currently OpenAI and Azure OpenAI retain data for 30 days for monitoring against misuse
|
|
- Host the model yourself for complete control and flexibility
|
|
- The Gen AI is the only feature in Danswer that reaches out to third party controlled service
|
|
- There are options (see below) to avoid this entirely but at the cost of performance
|
|
- Use a different model perhaps finetuned or better suited for a particular domain of interest
|
|
|
|
## What are the options
|
|
- Other OpenAI models such as `gpt-4` or `text-davinci-003`
|
|
- Azure OpenAI. The same models as OpenAI but via Azure Cloud for better security guarantees.
|
|
- As of Aug 2023, requires approval via this [form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOFA5Qk1UWDRBMjg0WFhPMkIzTzhKQ1dWNyQlQCN0PWcu)
|
|
- GPT4All and all compatible models.
|
|
- This runs the compressed LLMs in memory within Danswer removing the dependency on an externally hosted model
|
|
- This approach requires a beefy system or a significant sacrifice in performance.
|
|
- HuggingFace Hosted Inference API
|
|
- HuggingFace does not store prompts/tokens but does store logs.
|
|
- Requires a `Pro Account` subscription.
|
|
- Any LLM interfaced via REST API
|
|
- Intended for self hosted models on-prem or in any cloud provider such as AWS, Azure, GCP, etc.
|
|
- May require minor extension/rebuilding of Danswer containers to conform to the expected API for the model.
|
|
|
|
|
|
## Danswer Model configs
|
|
All Danswer Gen AI configs are done through deployment environment variables. For `docker compose` this means
|
|
overwriting the default values in the .env file during deployment. For `Kubernetes` this means updating the service
|
|
deployment yaml files (specifically, the api_server and background services).
|
|
|
|
The environment variables that impact the Gen AI models are as follows:
|
|
- **INTERNAL_MODEL_VERSION**: controls which of the options mentioned above is being used
|
|
- **GEN_AI_API_KEY**: If the Generative AI model requires an API key for access, otherwise can leave blank
|
|
- **GEN_AI_MODEL_VERSION**: Which model version to use.
|
|
- _Only relevant for OpenAI and GPT4All_
|
|
- **GEN_AI_ENDPOINT**: What is the endpoint for hitting the model server.
|
|
- _Only relevant for REST API LLM models_
|
|
- **GEN_AI_HOST_TYPE**: What is the expected way to construct the request for the model endpoint.
|
|
- _Only relevant for REST API LLM models_
|
|
|
|
See the next sections for specifics on how to configure the different options.
|
|
|
|
As always, don't hesitate to reach out to the [Danswer team](https://docs.danswer.dev/contact_us) if you have any
|
|
questions or issues.
|