Opensource Models (#6)

2026-07-12 13:04:48 +00:00 · 2023-08-12 18:44:13 -07:00
parent bd25f30cde
commit 4df7327a56
12 changed files with 169 additions and 5 deletions
@@ -13,7 +13,7 @@ Note: This Connector is relatively involved to set up.
 - This guide is for users to try it locally for personal accounts.

 - If you are using this for an organization, the steps differ slightly.
-    - Please reach out to [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com) OR `@Yuhong Sun` / `@Chris Weaver` in [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1w76msxmd-HJHLe3KNFIAIzk_0dSOKaQ) if you need help.
+    - Please reach out to [founders@danswer.ai](mailto:founders@danswer.ai) OR `@Yuhong Sun` / `@Chris Weaver` in [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1w76msxmd-HJHLe3KNFIAIzk_0dSOKaQ) if you need help.

 ### Authorization
    1. Create a **Google Cloud Project**
@@ -33,7 +33,7 @@ Note: This Connector is relatively involved to set up.
        - Under **APIs & services**, select the **OAuth consent screen** tab
        - If you don't have a **Google Organization** select **External** for **User Type**
        - Call the app Danswer (or whatever you want)
-        - For the required emails, use any email of your choice or danswer.dev@gmail.com
+        - For the required emails, use any email of your choice or `founders@danswer.ai`
        if you wish for the Danswer team to help handle issues.
        - Click **SAVE AND CONTINUE**

@@ -4,7 +4,7 @@ description: 'We would love to hear from you!'
 ---

 ### Please don't hesitate to reach out!
-Shoot us a message at [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com)
+Shoot us a message at [founders@danswer.ai](mailto:founders@danswer.ai)

 Or join our [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1u3h3ke3b-VGh1idW19R8oiNRiKBYv2w)
 or [Discord](https://discord.gg/TDJ59cGV2X)
@@ -0,0 +1,7 @@
+---
+title: Azure OpenAI GPT
+description: 'Configure Danswer to use GPT models on Azure'
+---
+
+This is currently a work in progress tracked by:
+https://github.com/danswer-ai/danswer/issues/165
@@ -0,0 +1,40 @@
+---
+title: HuggingFace Inference API
+description: 'Configure Danswer to use HuggingFace APIs'
+---
+
+Refer to [Model Configs](https://docs.danswer.dev/gen_ai_configs/overview#model-configs) for how to set the
+environment variables for your particular deployment.
+
+To use the HuggingFace Inference APIs, you must sign up for a `Pro Account` to get an API Key
+1. After signing up for `Pro Account`, go to your user settings:
+
+![HFSettings](/images/gen_ai/HFSettings.png)
+
+2. Copy the `User Access Token`
+
+![HFAccessToken](/images/gen_ai/HFAccessToken.png)
+
+## Terminology Disambiguation
+In Danswer, a "chat" model refers to using a series of messages with attached roles such as "system", "assistant", or
+"user". This is consistent with the OpenAI terminology.
+
+HuggingFace uses "chat" in reference to a model that is finetuned to follow instructions.
+They use "conversational" to mean the equivalent of OpenAI's "chat".
+
+
+## Set Danswer to use `Llama-2-70B` via next-token generation prompting
+- INTERNAL_MODEL_VERSION=huggingface-client-completion
+- GEN_AI_MODEL_VERSION=meta-llama/Llama-2-70b-chat-hf
+    - Note, "chat" in the above refers to instruction-fine-tuning
+    - As of Aug 2023, this model does not support conversational prompting
+- GEN_AI_API_KEY=&lt;your-huggingface-access-token&gt;
+    - You can also leave this unset and set it later via the UI
+
+
+## Set Danswer to use `Llama-2-70B` via chat (conversational) prompting
+- INTERNAL_MODEL_VERSION=huggingface-client-chat-completion
+- GEN_AI_MODEL_VERSION=meta-llama/Llama-2-70b-hf
+    - As of Aug 2023, only the non-instruction-finetuned model support conversational prompting
+- GEN_AI_API_KEY=&lt;your-huggingface-access-token&gt;
+    - You can also leave this unset and set it later via the UI
@@ -0,0 +1,21 @@
+---
+title: OpenAI Options
+description: 'Configure Danswer to use different OpenAI models'
+---
+
+Refer to [Model Configs](https://docs.danswer.dev/gen_ai_configs/overview#model-configs) for how to set the
+environment variables for your particular deployment.
+
+The default Danswer model is GPT-3.5-Turbo so if you wish to use this, there's no need to change anything.
+
+## Set Danswer to use GPT-4
+- INTERNAL_MODEL_VERSION=openai-chat-completion
+- GEN_AI_MODEL_VERSION=gpt-4
+- GEN_AI_API_KEY=&lt;your-gpt4-compatible-key&gt;
+    - You can also leave this unset and set it later via the UI
+
+## Set Danswer to use Text-Davinci-003
+- INTERNAL_MODEL_VERSION=openai-completion
+- GEN_AI_MODEL_VERSION=text-davinci-003
+- GEN_AI_API_KEY=&lt;your-openai-key&gt;
+    - You can also leave this unset and set it later via the UI
@@ -0,0 +1,58 @@
+---
+title: Overview
+description: 'Overview of the Generative AI functionality in Danswer'
+---
+
+## What are Generative AI (LLM) models used for?
+The Large Language Models are used to interpret the contents from the most relevant documents retrieved via Search.
+These models extract out the useful knowledge from your documents and generates the **AI Answer**.
+
+## What is the default LLM?
+By default, Danswer uses `GPT-3.5-Turbo` from OpenAI. This is the most accessible model/hosting service since it does
+not require any access approval process like `GPT-4` or `Llama2` variants.
+OpenAI also hosts the models behind an API which makes it easy to use and much more cost efficient than hosting a model
+yourself on dedicated hardware.
+
+## Why would you want to use a different model
+- Use a more powerful model despite higher cost (such as GPT-4)
+- Use a hosting service with a different data retention policy
+    - Currently OpenAI and Azure OpenAI retain data for 30 days for monitoring against misuse
+- Host the model yourself for complete control and flexibility
+    - The Gen AI is the only feature in Danswer that reaches out to third party controlled service
+    - There are options (see below) to avoid this entirely but at the cost of performance
+- Use a different model perhaps finetuned or better suited for a particular domain of interest
+
+## What are the options
+- Other OpenAI models such as `gpt-4` or `text-davinci-003`
+- Azure OpenAI. The same models as OpenAI but via Azure Cloud for better security guarantees.
+    - As of Aug 2023, requires approval via this [form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOFA5Qk1UWDRBMjg0WFhPMkIzTzhKQ1dWNyQlQCN0PWcu)
+- GPT4All and all compatible models.
+    - This runs the compressed LLMs in memory within Danswer removing the dependency on an externally hosted model
+    - This approach requires a beefy system or a significant sacrifice in performance.
+- HuggingFace Hosted Inference API
+    - HuggingFace does not store prompts/tokens but does store logs.
+    - Requires a `Pro Account` subscription.
+- Any LLM interfaced via REST API
+    - Intended for self hosted models on-prem or in any cloud provider such as AWS, Azure, GCP, etc.
+    - May require minor extension/rebuilding of Danswer containers to conform to the expected API for the model.
+
+
+## Model configs
+All Danswer Gen AI configs are done through deployment environment variables. For `docker compose` this means
+overwriting the default values in the .env file during deployment. For `Kubernetes` this means updating the service
+deployment yaml files (specifically, the api_server and background services).
+
+The environment variables that impact the Gen AI models are as follows:
+- **INTERNAL_MODEL_VERSION**: controls which of the options mentioned above is being used
+- **GEN_AI_API_KEY**: If the Generative AI model requires an API key for access, otherwise can leave blank
+- **GEN_AI_MODEL_VERSION**: Which model version to use.
+    - _Only relevant for OpenAI and GPT4All_
+- **GEN_AI_ENDPOINT**: What is the endpoint for hitting the model server.
+    - _Only relevant for REST API LLM models_
+- **GEN_AI_HOST_TYPE**: What is the expected way to construct the request for the model endpoint.
+    - _Only relevant for REST API LLM models_
+
+See the next sections for specifics on how to configure the different options.
+
+As always, don't hesitate to reach out to the [Danswer team](https://docs.danswer.dev/contact_us) if you have any
+questions or issues.
@@ -0,0 +1,28 @@
+---
+title: LLM Model via REST API
+description: 'Configure Danswer to use a custom Model Server'
+---
+
+Danswer can also make requests to an arbitrary model server via rest requests. Optionally an access token can be passed
+in. To customize the request format and handling of the response, it may be necessary to update/rebuild the Danswer
+containers.
+
+## Extending Danswer to be compatible with your custom model server
+There is a default implementation which follows the HuggingFace `Inference Endpoint` API.
+
+If instead, calling to a custom model server, there's a very minimal interface to be implemented with examples found
+[here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/direct_qa/request_model.py#L31).
+
+If going with a custom implementation, be sure to register it in the code where `ModelHostType` is mentioned.
+Refer to the code [here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/configs/constants.py#L49).
+
+## Blog on using Danswer with a self hosted `Llama-2-13B-chat-GGML`
+- [https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183](https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183)
+- This demo uses Google Colab to access a free GPU but this is not suitable for long term deployments
+
+## Set Danswer to use the LLM model server
+- INTERNAL_MODEL_VERSION=request-completion
+- GEN_AI_HOST_TYPE=colab-demo
+    - or reference your custom class
+- GEN_AI_ENDPOINT=&lt;your-model-endpoint-url&gt;
+
@@ -26,9 +26,9 @@ Under OAuth Consent screen select either Internal or External

 On the next page:
 - Provide an app name (can go with `Danswer`)
- Provide any email you own (or danswer.dev@gmail.com if you want us to handle questions from your Danswer users)
+- Provide any email you own (or founders@danswer.ai if you want us to handle questions from your Danswer users)
 - Upload the Danswer logo (or leave blank)
- The **Developer contact information** can be any email you own (or again, danswer.dev@gmail.com)
+- The **Developer contact information** can be any email you own (or again, founders@danswer.ai)

 ![GoogleApp](/images/google_oauth_setup/GoogleApp.png)

@@ -36,6 +36,16 @@
        "google_oauth_setup",
        "advanced_setup",
        "slack_bot_setup",
+        {
+          "group": "Gen AI Configs",
+          "pages": [
+            "gen_ai_configs/overview",
+            "gen_ai_configs/open_ai",
+            "gen_ai_configs/azure",
+            "gen_ai_configs/huggingface",
+            "gen_ai_configs/rest_api"
+          ]
+        },
        "contact_us"
      ]
    },