diff --git a/connectors/google_drive.mdx b/connectors/google_drive.mdx index a7e7afc..019444c 100644 --- a/connectors/google_drive.mdx +++ b/connectors/google_drive.mdx @@ -13,7 +13,7 @@ Note: This Connector is relatively involved to set up. - This guide is for users to try it locally for personal accounts. - If you are using this for an organization, the steps differ slightly. - - Please reach out to [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com) OR `@Yuhong Sun` / `@Chris Weaver` in [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1w76msxmd-HJHLe3KNFIAIzk_0dSOKaQ) if you need help. + - Please reach out to [founders@danswer.ai](mailto:founders@danswer.ai) OR `@Yuhong Sun` / `@Chris Weaver` in [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1w76msxmd-HJHLe3KNFIAIzk_0dSOKaQ) if you need help. ### Authorization 1. Create a **Google Cloud Project** @@ -33,7 +33,7 @@ Note: This Connector is relatively involved to set up. - Under **APIs & services**, select the **OAuth consent screen** tab - If you don't have a **Google Organization** select **External** for **User Type** - Call the app Danswer (or whatever you want) - - For the required emails, use any email of your choice or danswer.dev@gmail.com + - For the required emails, use any email of your choice or `founders@danswer.ai` if you wish for the Danswer team to help handle issues. - Click **SAVE AND CONTINUE** diff --git a/contact_us.mdx b/contact_us.mdx index aa21f0d..010bfbb 100644 --- a/contact_us.mdx +++ b/contact_us.mdx @@ -4,7 +4,7 @@ description: 'We would love to hear from you!' --- ### Please don't hesitate to reach out! -Shoot us a message at [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com) +Shoot us a message at [founders@danswer.ai](mailto:founders@danswer.ai) Or join our [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1u3h3ke3b-VGh1idW19R8oiNRiKBYv2w) or [Discord](https://discord.gg/TDJ59cGV2X) diff --git a/gen_ai_configs/azure.mdx b/gen_ai_configs/azure.mdx new file mode 100644 index 0000000..346595e --- /dev/null +++ b/gen_ai_configs/azure.mdx @@ -0,0 +1,7 @@ +--- +title: Azure OpenAI GPT +description: 'Configure Danswer to use GPT models on Azure' +--- + +This is currently a work in progress tracked by: +https://github.com/danswer-ai/danswer/issues/165 diff --git a/gen_ai_configs/huggingface.mdx b/gen_ai_configs/huggingface.mdx new file mode 100644 index 0000000..d84d41b --- /dev/null +++ b/gen_ai_configs/huggingface.mdx @@ -0,0 +1,40 @@ +--- +title: HuggingFace Inference API +description: 'Configure Danswer to use HuggingFace APIs' +--- + +Refer to [Model Configs](https://docs.danswer.dev/gen_ai_configs/overview#model-configs) for how to set the +environment variables for your particular deployment. + +To use the HuggingFace Inference APIs, you must sign up for a `Pro Account` to get an API Key +1. After signing up for `Pro Account`, go to your user settings: + +![HFSettings](/images/gen_ai/HFSettings.png) + +2. Copy the `User Access Token` + +![HFAccessToken](/images/gen_ai/HFAccessToken.png) + +## Terminology Disambiguation +In Danswer, a "chat" model refers to using a series of messages with attached roles such as "system", "assistant", or +"user". This is consistent with the OpenAI terminology. + +HuggingFace uses "chat" in reference to a model that is finetuned to follow instructions. +They use "conversational" to mean the equivalent of OpenAI's "chat". + + +## Set Danswer to use `Llama-2-70B` via next-token generation prompting +- INTERNAL_MODEL_VERSION=huggingface-client-completion +- GEN_AI_MODEL_VERSION=meta-llama/Llama-2-70b-chat-hf + - Note, "chat" in the above refers to instruction-fine-tuning + - As of Aug 2023, this model does not support conversational prompting +- GEN_AI_API_KEY=<your-huggingface-access-token> + - You can also leave this unset and set it later via the UI + + +## Set Danswer to use `Llama-2-70B` via chat (conversational) prompting +- INTERNAL_MODEL_VERSION=huggingface-client-chat-completion +- GEN_AI_MODEL_VERSION=meta-llama/Llama-2-70b-hf + - As of Aug 2023, only the non-instruction-finetuned model support conversational prompting +- GEN_AI_API_KEY=<your-huggingface-access-token> + - You can also leave this unset and set it later via the UI diff --git a/gen_ai_configs/open_ai.mdx b/gen_ai_configs/open_ai.mdx new file mode 100644 index 0000000..d7739c2 --- /dev/null +++ b/gen_ai_configs/open_ai.mdx @@ -0,0 +1,21 @@ +--- +title: OpenAI Options +description: 'Configure Danswer to use different OpenAI models' +--- + +Refer to [Model Configs](https://docs.danswer.dev/gen_ai_configs/overview#model-configs) for how to set the +environment variables for your particular deployment. + +The default Danswer model is GPT-3.5-Turbo so if you wish to use this, there's no need to change anything. + +## Set Danswer to use GPT-4 +- INTERNAL_MODEL_VERSION=openai-chat-completion +- GEN_AI_MODEL_VERSION=gpt-4 +- GEN_AI_API_KEY=<your-gpt4-compatible-key> + - You can also leave this unset and set it later via the UI + +## Set Danswer to use Text-Davinci-003 +- INTERNAL_MODEL_VERSION=openai-completion +- GEN_AI_MODEL_VERSION=text-davinci-003 +- GEN_AI_API_KEY=<your-openai-key> + - You can also leave this unset and set it later via the UI diff --git a/gen_ai_configs/overview.mdx b/gen_ai_configs/overview.mdx new file mode 100644 index 0000000..584fd11 --- /dev/null +++ b/gen_ai_configs/overview.mdx @@ -0,0 +1,58 @@ +--- +title: Overview +description: 'Overview of the Generative AI functionality in Danswer' +--- + +## What are Generative AI (LLM) models used for? +The Large Language Models are used to interpret the contents from the most relevant documents retrieved via Search. +These models extract out the useful knowledge from your documents and generates the **AI Answer**. + +## What is the default LLM? +By default, Danswer uses `GPT-3.5-Turbo` from OpenAI. This is the most accessible model/hosting service since it does +not require any access approval process like `GPT-4` or `Llama2` variants. +OpenAI also hosts the models behind an API which makes it easy to use and much more cost efficient than hosting a model +yourself on dedicated hardware. + +## Why would you want to use a different model +- Use a more powerful model despite higher cost (such as GPT-4) +- Use a hosting service with a different data retention policy + - Currently OpenAI and Azure OpenAI retain data for 30 days for monitoring against misuse +- Host the model yourself for complete control and flexibility + - The Gen AI is the only feature in Danswer that reaches out to third party controlled service + - There are options (see below) to avoid this entirely but at the cost of performance +- Use a different model perhaps finetuned or better suited for a particular domain of interest + +## What are the options +- Other OpenAI models such as `gpt-4` or `text-davinci-003` +- Azure OpenAI. The same models as OpenAI but via Azure Cloud for better security guarantees. + - As of Aug 2023, requires approval via this [form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOFA5Qk1UWDRBMjg0WFhPMkIzTzhKQ1dWNyQlQCN0PWcu) +- GPT4All and all compatible models. + - This runs the compressed LLMs in memory within Danswer removing the dependency on an externally hosted model + - This approach requires a beefy system or a significant sacrifice in performance. +- HuggingFace Hosted Inference API + - HuggingFace does not store prompts/tokens but does store logs. + - Requires a `Pro Account` subscription. +- Any LLM interfaced via REST API + - Intended for self hosted models on-prem or in any cloud provider such as AWS, Azure, GCP, etc. + - May require minor extension/rebuilding of Danswer containers to conform to the expected API for the model. + + +## Model configs +All Danswer Gen AI configs are done through deployment environment variables. For `docker compose` this means +overwriting the default values in the .env file during deployment. For `Kubernetes` this means updating the service +deployment yaml files (specifically, the api_server and background services). + +The environment variables that impact the Gen AI models are as follows: +- **INTERNAL_MODEL_VERSION**: controls which of the options mentioned above is being used +- **GEN_AI_API_KEY**: If the Generative AI model requires an API key for access, otherwise can leave blank +- **GEN_AI_MODEL_VERSION**: Which model version to use. + - _Only relevant for OpenAI and GPT4All_ +- **GEN_AI_ENDPOINT**: What is the endpoint for hitting the model server. + - _Only relevant for REST API LLM models_ +- **GEN_AI_HOST_TYPE**: What is the expected way to construct the request for the model endpoint. + - _Only relevant for REST API LLM models_ + +See the next sections for specifics on how to configure the different options. + +As always, don't hesitate to reach out to the [Danswer team](https://docs.danswer.dev/contact_us) if you have any +questions or issues. diff --git a/gen_ai_configs/rest_api.mdx b/gen_ai_configs/rest_api.mdx new file mode 100644 index 0000000..bd756c3 --- /dev/null +++ b/gen_ai_configs/rest_api.mdx @@ -0,0 +1,28 @@ +--- +title: LLM Model via REST API +description: 'Configure Danswer to use a custom Model Server' +--- + +Danswer can also make requests to an arbitrary model server via rest requests. Optionally an access token can be passed +in. To customize the request format and handling of the response, it may be necessary to update/rebuild the Danswer +containers. + +## Extending Danswer to be compatible with your custom model server +There is a default implementation which follows the HuggingFace `Inference Endpoint` API. + +If instead, calling to a custom model server, there's a very minimal interface to be implemented with examples found +[here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/direct_qa/request_model.py#L31). + +If going with a custom implementation, be sure to register it in the code where `ModelHostType` is mentioned. +Refer to the code [here](https://github.com/danswer-ai/danswer/blob/main/backend/danswer/configs/constants.py#L49). + +## Blog on using Danswer with a self hosted `Llama-2-13B-chat-GGML` +- [https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183](https://medium.com/@yuhongsun96/host-a-llama-2-api-on-gpu-for-free-a5311463c183) +- This demo uses Google Colab to access a free GPU but this is not suitable for long term deployments + +## Set Danswer to use the LLM model server +- INTERNAL_MODEL_VERSION=request-completion +- GEN_AI_HOST_TYPE=colab-demo + - or reference your custom class +- GEN_AI_ENDPOINT=<your-model-endpoint-url> + diff --git a/google_oauth_setup.mdx b/google_oauth_setup.mdx index 144d49f..e5a299c 100644 --- a/google_oauth_setup.mdx +++ b/google_oauth_setup.mdx @@ -26,9 +26,9 @@ Under OAuth Consent screen select either Internal or External On the next page: - Provide an app name (can go with `Danswer`) -- Provide any email you own (or danswer.dev@gmail.com if you want us to handle questions from your Danswer users) +- Provide any email you own (or founders@danswer.ai if you want us to handle questions from your Danswer users) - Upload the Danswer logo (or leave blank) -- The **Developer contact information** can be any email you own (or again, danswer.dev@gmail.com) +- The **Developer contact information** can be any email you own (or again, founders@danswer.ai) ![GoogleApp](/images/google_oauth_setup/GoogleApp.png) diff --git a/images/connectors/guru/GuruAuth.png b/images/connectors/guru/GuruAuth.png new file mode 100644 index 0000000..8ead5ad Binary files /dev/null and b/images/connectors/guru/GuruAuth.png differ diff --git a/images/gen_ai/HFAccessToken.png b/images/gen_ai/HFAccessToken.png new file mode 100644 index 0000000..134c753 Binary files /dev/null and b/images/gen_ai/HFAccessToken.png differ diff --git a/images/gen_ai/HFSettings.png b/images/gen_ai/HFSettings.png new file mode 100644 index 0000000..d573181 Binary files /dev/null and b/images/gen_ai/HFSettings.png differ diff --git a/mint.json b/mint.json index c3617f1..5b660d1 100644 --- a/mint.json +++ b/mint.json @@ -36,6 +36,16 @@ "google_oauth_setup", "advanced_setup", "slack_bot_setup", + { + "group": "Gen AI Configs", + "pages": [ + "gen_ai_configs/overview", + "gen_ai_configs/open_ai", + "gen_ai_configs/azure", + "gen_ai_configs/huggingface", + "gen_ai_configs/rest_api" + ] + }, "contact_us" ] },