commit 1d981713f7de5a9b77d9b8293c66b44e1e5b0bfe Author: Yuhong Sun Date: Wed Jul 12 19:40:43 2023 -0700 Making repo public diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..314f162 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023 Yuhong Sun, Chris Weaver + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..f1f05f0 --- /dev/null +++ b/README.md @@ -0,0 +1,32 @@ +

+ +

+ +# Danswer Docs +This repo generates the setup guide for Danswer found at [https://docs.danswer.dev/](https://docs.danswer.dev/). + +It uses Mintlify which is a no-code documentation generation tool. +Instructions on setting up local preview are included below. + +More info on Mintlify found [here](https://mintlify.com/). + +### Set up Mintlify +Install the [Mintlify CLI](https://www.npmjs.com/package/mintlify) to preview the documentation changes locally. + +To install, use the following command (requires node >= v19.0.0) +``` +npm i -g mintlify +``` + +Run the following command at the root of your documentation (where mint.json is) +``` +mintlify dev +``` + +### Publishing Changes +Changes are automatically deployed to production after merging to main. + +### Troubleshooting +- Mintlify dev isn't running - Run `mintlify install` it'll re-install dependencies. +- Page loads as a 404 - Make sure you are running in a folder with `mint.json` +- Mintlify Docs - https://mintlify.com/docs/introduction diff --git a/advanced_setup.mdx b/advanced_setup.mdx new file mode 100644 index 0000000..297407a --- /dev/null +++ b/advanced_setup.mdx @@ -0,0 +1,39 @@ +--- +title: Enterprise Setup +description: '[WIP] How to set up for an organization' +--- + +Below, we discuss the simplest possible way to setup Danswer within your enterprise. We know that every company does things +differently, and we'd love to help you figure out the ideal setup for your enterprise! Feel free to reach out to us via +the links on our [Contact Us](https://docs.danswer.dev/contact_us) page. + +## Getting a Machine + +The first step is to get a machine hosted in your VPC to run Danswer on. +- For AWS, this would be an EC2 instance. +- For GCP, this would be a Google Compute Engine instance. +- For Azure, this would be an Azure Virtual Machine instance. +- For DigitalOcean, this would be a Droplet. +- ... + +For resourcing, we recommend a machine with 8vCPU and +16GB memory (although 4vCPU and 8GB memory can work in a pinch). With this basic setup, all components needed to +operate Danswer will run on this single machine. + +## Setting Up the Domain + +Modify your internal DNS to include entries which point to the private IPv4 address for the instance created above. + +## Starting Danswer + +1. SSH into the machine setup in the `Getting a Machine` section. +2. If not already setup, install docker following the relevant guide here: https://docs.docker.com/engine/install/ +3. Clone the [Danswer](https://github.com/danswer-ai/danswer) repo: + `git clone https://github.com/danswer-ai/danswer.git` +4. Navigate to **danswer/deployment/docker_compose** +5. Bring up your docker engine and run: + - To pull images from DockerHub and run Danswer: + - `docker compose -f docker-compose.dev.yml -p danswer-stack up -d --pull always --force-recreate` + - Alternatively, to build the containers from source and start Danswer, run: + - `docker compose -f docker-compose.dev.yml -p danswer-stack up -d --build --force-recreate` + - This may take 15+ minutes depending on your internet speed. diff --git a/connectors/confluence.mdx b/connectors/confluence.mdx new file mode 100644 index 0000000..c80877d --- /dev/null +++ b/connectors/confluence.mdx @@ -0,0 +1,25 @@ +--- +title: Confluence Connector +description: 'Access knowledge from your company Wiki' +--- + +## How it works +The Confluence connector pulls in all pages and comments from the specified spaces every **10** minutes.. +- Depending on demand, Danswer may support indexing specific Confluence pages or directories in the future. + +## Setting up +### Authorization + 1. Follow the guide described [here](https://docs.searchunify.com/Content/Content-Sources/Atlassian-Jira-Confluence-Authentication-Create-API-Token.htm) to get an access token. + +### Indexing + 1. Navigate to the Admin Dashboard and select the **Confluence** Connector + 2. In **Step 1**, provide the **Access Token** and the **Username** it belongs to: + +![ConfluenceConnector](/images/connectors/confluence/ConfluenceAuth.png) + + 3. For each space to index, provide a single URL to any page in the space. For + example, it might look like the following: + +![ConfluenceConnector](/images/connectors/confluence/ConfluenceIndex.png) + + 4. Click the `Connect` button! All your Confluence pages in that space will now be pulled into Danswer every **10** minutes. diff --git a/connectors/file.mdx b/connectors/file.mdx new file mode 100644 index 0000000..8e317b8 --- /dev/null +++ b/connectors/file.mdx @@ -0,0 +1,39 @@ +--- +title: File Connector +description: 'Access knowledge from Local Files' +--- + +## How it works +The File Connector indexes user uploaded files. + - Currently supports `.txt` files or `.zip` files containing `.txt` files. + - Optional metadata line supports linking to other internal tools by URL + +The metadata line takes the format of: +`#DANSWER_METADATA={"link": ""}` where `` is replaced with a URL. + +This may be something like `https://internaltool.yourcompany.com` + +Example: +``` +#DANSWER_METADATA={"link": "https://www.danswereng.com/captcha"} +How to set up captcha +Follow the example below to set up a captcha +like you saw when you visited this page! +By including a captcha, this page is able to +prevent web scrapers from reading it. +``` +As we can see, there are many web pages or internal tools that aren't directly scrape-able. +In addition to handling local file uploads, the file connector is offered as an option for these tools which may have APIs for accessing the contents. + +## Setting up +### Authorization + - No external auth flows required. + - Admins can upload files and make them available to everyone + - [WIP] Admins or normal users will be able to upload files via personal connectors and make them accessible for just themselves. + +### Indexing + 1. Navigate to the Admin Dashboard and select the **File** Connector. + 2. Select a `.txt` file or a `.zip` file and click + +![WebConnector](/images/connectors/file/FileConnector.png) + diff --git a/connectors/github.mdx b/connectors/github.mdx new file mode 100644 index 0000000..aff8c7a --- /dev/null +++ b/connectors/github.mdx @@ -0,0 +1,27 @@ +--- +title: GitHub Connector +description: 'Access knowledge from your Repositories' +--- + +## How it works +The Github Connector picks up all of the Pull Requests in a repository including the Tile and Summary. +- It will index both Open and Closed PRs. +- Includes certain other metadata such as the PR's URL, creator, etc. + +## Setting up +### Authorization + 1. This Connector uses a GitHub Access Token. + - [This guide](https://docs.github.com/en/enterprise-server@3.4/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) shows the following steps in detail. + 2. Log in to GitHub. + 3. In the upper right corner, expand your profile and click on **Settings** + 4. On the bottom, go to **Developer settings** -> **Personal access tokens** -> **Tokens (classic)** + 5. Click on **Generate new token** + - Be sure to give the Token `repo` access so that it can access the PRs. + - Set any expiration time. A new token will have to be provided to Danswer to continue updating the Danswer + index once this one expires. + +### Indexing + 1. Navigate to the Admin Dashboard and select the **GitHub** Connector + 2. For [https://github.com/danswer-ai/danswer](https://github.com/danswer-ai/danswer), it would look like: + +![GithubConnector](/images/connectors/github/GithubConnector.png) \ No newline at end of file diff --git a/connectors/google_drive.mdx b/connectors/google_drive.mdx new file mode 100644 index 0000000..a7e7afc --- /dev/null +++ b/connectors/google_drive.mdx @@ -0,0 +1,70 @@ +--- +title: Google Drive Connector +description: 'Access knowledge from your Files' +--- + +## How it works +The Google Drive connector ingests your drive documents. + - Currently supports Google Docs, Sheets, and PDF files. + +## Setting up +Note: This Connector is relatively involved to set up. + +- This guide is for users to try it locally for personal accounts. + +- If you are using this for an organization, the steps differ slightly. + - Please reach out to [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com) OR `@Yuhong Sun` / `@Chris Weaver` in [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1w76msxmd-HJHLe3KNFIAIzk_0dSOKaQ) if you need help. + +### Authorization + 1. Create a **Google Cloud Project** + - [https://console.cloud.google.com/projectcreate](https://console.cloud.google.com/projectcreate) + + 2. Enable the **Google Drive API** + - On the left panel, open **APIs & services** + - Go to **Enabled APIs and services** + - On the top click **+ENABLE APIS AND SERVICES** + - Search for **Google Drive API** and click **ENABLE** + - Alternatively visit this [link](https://console.cloud.google.com/flows/enableapi?apiid=drive.googleapis.com), + select your project and enable the **Google Drive API** + +![GoogleDriveEnableAPI](/images/connectors/google_drive/GoogleDriveEnableAPI.png) + + 3. Set up the OAuth consent screen + - Under **APIs & services**, select the **OAuth consent screen** tab + - If you don't have a **Google Organization** select **External** for **User Type** + - Call the app Danswer (or whatever you want) + - For the required emails, use any email of your choice or danswer.dev@gmail.com + if you wish for the Danswer team to help handle issues. + - Click **SAVE AND CONTINUE** + + 4. Set up Scopes + - Add the scope `.../auth/drive.readonly` for `Google Drive API` +![GoogleDriveScopes](/images/connectors/google_drive/GoogleDriveScopes.png) + + 5. Set up Test users + - This is only applicable for users without a Google Organization. + - Typically for a company, Danswer would be set up as an internal app so this step would not apply. + - Add at least one test user email. Only the email accounts added here will be allowed to run the OAuth + flow to index new documents. + - Click **SAVE AND CONTINUE**, review the changes and click **BACK TO DASHBOARD** + + 6. Create Credentials + - Go to the **Credentials** tab and select **+ CREATE CREDENTIALS** -> **OAuth client ID** +![GoogleDriveCredential](/images/connectors/google_drive/GoogleDriveCredential.png) + - Choose **Web application** and give it some name like `DanswerConnector` + - Add a **Authorized JavaScript origins** for `http://localhost:3000` (or `https://` if you have setup Danswer for production use) + - Add a **Authorized redirect URIs** for `http://localhost:3000/admin/connectors/google-drive/auth/callback` (or `https:///admin/connectors/google-drive/auth/callback` if you have setup Danswer for production use) +![GoogleDriveScopes](/images/connectors/google_drive/DriveCredentials.png) + + - Click create and on the right hand side next to **Client secret**, there is an option to download the + credentials as a JSON. Download the JSON for use in the next step. +![GoogleDriveScopes](/images/connectors/google_drive/DriveDownloadCredentials.png) + +### Indexing + 1. Navigate to the Admin Dashboard and select the **Google Drive** Connector. + 2. Select the credentials JSON from **step 6** above as shown in image below, then click the **Upload** button to save your credentials into Danswer. + 3. Click on **Authenticate with Google Drive** + - Completing the OAuth flow will allow Danswer to index the docs that the user has read access to. + 4. Click the `Add` button! All Google Drive documents accessible by the user who completed step (3) will now be pulled into Danswer every **10** minutes. + +![GoogleDriveConnectorPage](/images/connectors/google_drive/GoogleDriveConnectorPage.png) \ No newline at end of file diff --git a/connectors/jira.mdx b/connectors/jira.mdx new file mode 100644 index 0000000..7180f1d --- /dev/null +++ b/connectors/jira.mdx @@ -0,0 +1,24 @@ +--- +title: Jira Connector +description: 'Access the latest issue and project updates from Jira' +--- + +## How it works +The Jira connector pulls in all tickets from the specified projects every **10** minutes. + +## Setting up +### Authorization + 1. Follow the guide described [here](https://docs.searchunify.com/Content/Content-Sources/Atlassian-Jira-Confluence-Authentication-Create-API-Token.htm) to get an access token. + +### Indexing + 1. Navigate to the Admin Dashboard and select the **Jira** Connector + 2. In **Step 1**, provide the **Access Token** and the **Username** it belongs to: + +![JiraConnectorAuth](/images/connectors/jira/JiraAuth.png) + + 3. For each project you want to index, provide a single URL to any page in the project. For + example, it might look like the following: + +![ConfluenceConnector](/images/connectors/jira/JiraIndex.png) + + 4. Click the `Connect` button! All your Jira tickets in that space will now be pulled into Danswer every **10** minutes. \ No newline at end of file diff --git a/connectors/overview.mdx b/connectors/overview.mdx new file mode 100644 index 0000000..020a2c1 --- /dev/null +++ b/connectors/overview.mdx @@ -0,0 +1,30 @@ +--- +title: Overview +description: 'Basic Information about Connectors' +--- + +## What are Connectors +Connectors hook up Danswer to your data sources so that answers are grounded in your organization's knowledge. + +### Connectors help you +- **Choose Sources** so you can include only the data you want indexed. +- **Configure Access** so Danswer can securely access data with your permission. +- **Set Up Fetching** options to keep Danswer's answers up to date. + + +## Monitoring Connectors +Open the `Connectors Dashboard` (accessible from the profile icon on the top right) +At the top there is a `Status` page which shows which sources have been indexed and the status of the indexing job. + +![IndexStatus](/images/connectors/IndexStatus.png) + + +## Missing a Connector? +Is there a Connector that would be useful to you? Let us know via Email or Github Issue. + +Contact us at [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com). + +Please check if the connector you're interested in is already mentioned on the Danswer +[GitHub Issues](https://github.com/danswer-ai/danswer/issues) page. + +Don't forget to thumbs up the issues requesting the connectors that you would like to see built next! diff --git a/connectors/slack.mdx b/connectors/slack.mdx new file mode 100644 index 0000000..778f729 --- /dev/null +++ b/connectors/slack.mdx @@ -0,0 +1,62 @@ +--- +title: Slack Connector +description: 'Access knowledge from your Messages' +--- + +## How it works +The Slack connector indexes all public channels for a given workspace. +- Upcoming: Support for private channels by tagging/adding the Danswer Slack Bot to private channels. + + +## Setting up +### Authorization +1. **Note: You must be an admin of the Slack workspace to set up the connector** +2. Navigate and sign in to [https://api.slack.com/apps](https://api.slack.com/apps). +3. Create a new Slack app: + - Click the **Create New App** button in the top right. + - Select **From an app manifest** option. + - Select the relevant workspace from the dropdown and click **Next**. + +![SlackApp](/images/connectors/slack/SlackApp.png) + +4. Copy the following manifest into the text box +``` +display_information: + name: danswer-read-only +features: + bot_user: + display_name: DanswerReadOnly + always_online: false +oauth_config: + scopes: + bot: + - channels:history + - channels:read + - groups:history + - channels:join + - im:history + - users:read +settings: + org_deploy_enabled: false + socket_mode_enabled: false + token_rotation_enabled: false +``` + +5. Click the **Create** button. +6. In the app page, navigate to the **OAuth & Permissions** tab under the **Features** header. +7. Copy the **Bot User OAuth Token**, this will be used to access Slack. + +![SlackBotToken](/images/connectors/slack/SlackBotToken.png) + + +### Indexing +1. Navigate to the Connector Dashboard and select the **Slack** Connector. +2. Place the **Bot User OAuth Token** under **Step 1 Provide Credentials** + +![SlackConnectorSetting](/images/connectors/slack/SlackConnectorSetting.png) + +3. Set the **Workspace ID** (see below) and click **Connect**. + +![SlackId](/images/connectors/slack/SlackId.png) + +__Note:__ The first indexing pulls all of the public channels and takes longer than future updates. \ No newline at end of file diff --git a/connectors/web.mdx b/connectors/web.mdx new file mode 100644 index 0000000..32f9da5 --- /dev/null +++ b/connectors/web.mdx @@ -0,0 +1,24 @@ +--- +title: Web Connector +description: 'Access knowledge from Web Pages' +--- + +## How it works +The Web Connector scrapes sites based on a base URL. + - It only indexes files from the same domain and containing the same base path. + - It will index pages reachable via hyperlinks from the base URL. + - The text contents are cleaned up via some heuristics and some metadata such as the page Title is extracted. + +## Setting up +### Authorization + - As long as the page is reachable, no additional authorization is necessary. + +### Indexing + 1. Navigate to the Admin Dashboard and select the **Web** Connector. + 2. Input the base URL to index and click on Index. + +![WebConnector](/images/connectors/web/WebConnector.png) + +To see the status of the indexing, visit the Connectors Status page (top left). + +![WebConnectorStatus](/images/connectors/web/WebConnectorStatus.png) diff --git a/contact_us.mdx b/contact_us.mdx new file mode 100644 index 0000000..fefdb93 --- /dev/null +++ b/contact_us.mdx @@ -0,0 +1,12 @@ +--- +title: Contact Us +description: 'We would love to hear from you!' +--- + +### Please don't hesitate to reach out! +Shoot us a message at [danswer.dev@gmail.com](mailto:danswer.dev@gmail.com) + +Or join our [Slack](https://join.slack.com/t/danswer/shared_invite/zt-1u3h3ke3b-VGh1idW19R8oiNRiKBYv2w) +or [Discord](https://discord.gg/s6qETAVZ) + +![Danswer](/logo/Danswer.png) \ No newline at end of file diff --git a/favicon.png b/favicon.png new file mode 100644 index 0000000..e9a7cf5 Binary files /dev/null and b/favicon.png differ diff --git a/images/connectors/IndexStatus.png b/images/connectors/IndexStatus.png new file mode 100644 index 0000000..2fa2008 Binary files /dev/null and b/images/connectors/IndexStatus.png differ diff --git a/images/connectors/confluence/ConfluenceAuth.png b/images/connectors/confluence/ConfluenceAuth.png new file mode 100644 index 0000000..f4874e1 Binary files /dev/null and b/images/connectors/confluence/ConfluenceAuth.png differ diff --git a/images/connectors/confluence/ConfluenceIndex.png b/images/connectors/confluence/ConfluenceIndex.png new file mode 100644 index 0000000..a3442bc Binary files /dev/null and b/images/connectors/confluence/ConfluenceIndex.png differ diff --git a/images/connectors/file/FileConnector.png b/images/connectors/file/FileConnector.png new file mode 100644 index 0000000..fab332d Binary files /dev/null and b/images/connectors/file/FileConnector.png differ diff --git a/images/connectors/github/GithubConnector.png b/images/connectors/github/GithubConnector.png new file mode 100644 index 0000000..322735b Binary files /dev/null and b/images/connectors/github/GithubConnector.png differ diff --git a/images/connectors/google_drive/DriveCredentials.png b/images/connectors/google_drive/DriveCredentials.png new file mode 100644 index 0000000..781b372 Binary files /dev/null and b/images/connectors/google_drive/DriveCredentials.png differ diff --git a/images/connectors/google_drive/DriveDownloadCredentials.png b/images/connectors/google_drive/DriveDownloadCredentials.png new file mode 100644 index 0000000..e7771e2 Binary files /dev/null and b/images/connectors/google_drive/DriveDownloadCredentials.png differ diff --git a/images/connectors/google_drive/GoogleDriveConnectorPage.png b/images/connectors/google_drive/GoogleDriveConnectorPage.png new file mode 100644 index 0000000..2c1d435 Binary files /dev/null and b/images/connectors/google_drive/GoogleDriveConnectorPage.png differ diff --git a/images/connectors/google_drive/GoogleDriveCredential.png b/images/connectors/google_drive/GoogleDriveCredential.png new file mode 100644 index 0000000..21f526c Binary files /dev/null and b/images/connectors/google_drive/GoogleDriveCredential.png differ diff --git a/images/connectors/google_drive/GoogleDriveEnableAPI.png b/images/connectors/google_drive/GoogleDriveEnableAPI.png new file mode 100644 index 0000000..934c53d Binary files /dev/null and b/images/connectors/google_drive/GoogleDriveEnableAPI.png differ diff --git a/images/connectors/google_drive/GoogleDriveScopes.png b/images/connectors/google_drive/GoogleDriveScopes.png new file mode 100644 index 0000000..6ee3b98 Binary files /dev/null and b/images/connectors/google_drive/GoogleDriveScopes.png differ diff --git a/images/connectors/jira/JiraAuth.png b/images/connectors/jira/JiraAuth.png new file mode 100644 index 0000000..54583c7 Binary files /dev/null and b/images/connectors/jira/JiraAuth.png differ diff --git a/images/connectors/jira/JiraIndex.png b/images/connectors/jira/JiraIndex.png new file mode 100644 index 0000000..0550374 Binary files /dev/null and b/images/connectors/jira/JiraIndex.png differ diff --git a/images/connectors/slack/SlackApp.png b/images/connectors/slack/SlackApp.png new file mode 100644 index 0000000..28cfc22 Binary files /dev/null and b/images/connectors/slack/SlackApp.png differ diff --git a/images/connectors/slack/SlackBotToken.png b/images/connectors/slack/SlackBotToken.png new file mode 100644 index 0000000..fba745b Binary files /dev/null and b/images/connectors/slack/SlackBotToken.png differ diff --git a/images/connectors/slack/SlackConnectorSetting.png b/images/connectors/slack/SlackConnectorSetting.png new file mode 100644 index 0000000..ba1968e Binary files /dev/null and b/images/connectors/slack/SlackConnectorSetting.png differ diff --git a/images/connectors/slack/SlackId.png b/images/connectors/slack/SlackId.png new file mode 100644 index 0000000..bcf4d30 Binary files /dev/null and b/images/connectors/slack/SlackId.png differ diff --git a/images/connectors/web/WebConnector.png b/images/connectors/web/WebConnector.png new file mode 100644 index 0000000..842b062 Binary files /dev/null and b/images/connectors/web/WebConnector.png differ diff --git a/images/connectors/web/WebConnectorStatus.png b/images/connectors/web/WebConnectorStatus.png new file mode 100644 index 0000000..4010ef1 Binary files /dev/null and b/images/connectors/web/WebConnectorStatus.png differ diff --git a/images/quickstart/DanswerConnectors.png b/images/quickstart/DanswerConnectors.png new file mode 100644 index 0000000..e861c35 Binary files /dev/null and b/images/quickstart/DanswerConnectors.png differ diff --git a/images/quickstart/DanswerSampleQA.png b/images/quickstart/DanswerSampleQA.png new file mode 100644 index 0000000..fe5cc0b Binary files /dev/null and b/images/quickstart/DanswerSampleQA.png differ diff --git a/images/quickstart/DanswerWebConnector.png b/images/quickstart/DanswerWebConnector.png new file mode 100644 index 0000000..ecb1fc8 Binary files /dev/null and b/images/quickstart/DanswerWebConnector.png differ diff --git a/images/slack_bot/DanswerBotInstall.png b/images/slack_bot/DanswerBotInstall.png new file mode 100644 index 0000000..3267894 Binary files /dev/null and b/images/slack_bot/DanswerBotInstall.png differ diff --git a/images/slack_bot/SlackBot.png b/images/slack_bot/SlackBot.png new file mode 100644 index 0000000..960fbf2 Binary files /dev/null and b/images/slack_bot/SlackBot.png differ diff --git a/introduction.mdx b/introduction.mdx new file mode 100644 index 0000000..cda358f --- /dev/null +++ b/introduction.mdx @@ -0,0 +1,51 @@ +--- +title: Introduction +description: 'Learn about Danswer' +--- + +## What is Danswer + +[Danswer](https://github.com/danswer-ai/danswer) is an OpenSource Enterprise Question Answering Tool. + +### **But what is that?** 🤔 + + + +**What do you mean by Question Answer?** + - Direct QA allows you to ask questions in simple language and receive instant, accurate responses. + - Unlock knowledge from private data sources whether that be company internal documents or personal sources you want + to add. + - Every answer is backed by quotes and reference documents so you can always trust what you get back. + +**What about OpenSource?** + - The project is licensed under the MIT License, do anything you'd like! + - Danswer is built to be easily hosted by anyone, all you need to setup are the data connectors. + +**How about Enterprise?** + - Danswer has user authentication with document level access control. + - Danswer provides connectors to common enterprise tools such as Slack, Google Drive, GitHub etc. + + +## Main Features 💃 + - Direct QA powered by Generative AI models with answers backed by quotes and source links. + - Intelligent Document Retrieval (Semantic Search/Reranking pipeline) using the latest LLMs. + - An AI Helper backed by a custom Deep Learning model to interpret user intent. + - User authentication with document level access management. + - Connectors to Slack, GitHub, GoogleDrive, Confluence, local files, and web scrapping, with more to come. + - Management Dashboard to manage connectors and set up features such as live update fetching. + - One line Docker Compose deployment of all services to host Danswer anywhere. + + +## Upcoming + - Chat/Conversation support. + - Support custom endpoints for Generative AI models or even self-host options. + - Template to easily build custom connectors. + - Personalized search \ No newline at end of file diff --git a/logo/Danswer.png b/logo/Danswer.png new file mode 100644 index 0000000..e9a7cf5 Binary files /dev/null and b/logo/Danswer.png differ diff --git a/logo/dark.png b/logo/dark.png new file mode 100644 index 0000000..2277d94 Binary files /dev/null and b/logo/dark.png differ diff --git a/logo/light.png b/logo/light.png new file mode 100644 index 0000000..2277d94 Binary files /dev/null and b/logo/light.png differ diff --git a/mint.json b/mint.json new file mode 100644 index 0000000..dc7e715 --- /dev/null +++ b/mint.json @@ -0,0 +1,60 @@ +{ + "$schema": "https://mintlify.com/schema.json", + "name": "Danswer Documentation", + "logo": { + "dark": "/logo/dark.png", + "light": "/logo/light.png" + }, + "favicon": "/favicon.png", + "colors": { + "primary": "#0069ED", + "light": "#4D9CFF", + "dark": "#0050B4" + }, + "topbarCtaButton": { + "type": "github", + "url": "https://github.com/danswer-ai/danswer" + }, + "anchors": [ + { + "name": "Slack", + "icon": "slack", + "url": "https://join.slack.com/t/danswer/shared_invite/zt-1u3h3ke3b-VGh1idW19R8oiNRiKBYv2w" + }, + { + "name": "Discord", + "icon": "discord", + "url": "https://discord.gg/TDJ59cGV2X" + } + ], + "navigation": [ + { + "group": "Welcome to Danswer", + "pages": [ + "introduction", + "quickstart", + "advanced_setup", + "slack_bot_setup", + "contact_us" + ] + }, + { + "group": "Connectors", + "pages": [ + "connectors/overview", + "connectors/web", + "connectors/slack", + "connectors/github", + "connectors/google_drive", + "connectors/confluence", + "connectors/jira", + "connectors/file" + ] + } + ], + "footerSocials": { + "github": "https://github.com/danswer-ai/danswer", + "twitter": "https://twitter.com/DanswerAI", + "linkedin": "https://linkedin.com/company/danswerai" + } +} diff --git a/quickstart.mdx b/quickstart.mdx new file mode 100644 index 0000000..4594fd0 --- /dev/null +++ b/quickstart.mdx @@ -0,0 +1,54 @@ +--- +title: "Quickstart" +description: "How to deploy Danswer on your local machine" +--- + +## Requirements + - git + - docker with compose (docker version >= 1.13.0) + + +## Setup +**This quickstart guide covers setting up Danswer for local execution** + 1. Clone the [Danswer](https://github.com/danswer-ai/danswer) repo: + `git clone https://github.com/danswer-ai/danswer.git` + 2. Navigate to **danswer/deployment/docker_compose** + 4. Bring up your docker engine and run: + - To pull images from DockerHub and run Danswer: + - `docker compose -f docker-compose.dev.yml -p danswer-stack up -d --pull always --force-recreate` + - Alternatively, to build the containers from source and start Danswer, run: + - `docker compose -f docker-compose.dev.yml -p danswer-stack up -d --build --force-recreate` + - These commands are also used to redeploy if any **.env** variables are updated + - This may take 15+ minutes depending on your internet speed. + 5. Danswer will now be running on http://localhost:3000. + + +## OpenAI API Key +**Note:** On the initial visit, Danswer will prompt for an OpenAI API key. +Without this Danswer will be able to provide search functionalities but not direct Question Answering. + +You can get an OpenAI API key at: +[https://platform.openai.com/account/api-keys](https://platform.openai.com/account/api-keys) + + +## Indexing Documents +**This quickstart guide will index a publicly accessible website as this requires no additional authorization setup** + 1. Navigate to the top right of Danswer's home screen and select **Admin Panel** + +![Connectors](/images/quickstart/DanswerConnectors.png) + + 2. In the Web Connector dashboard, pick any base URL to index. + - This will index all pages under that base URL that is reachable from hyperlinks. + - You can check the indexing status page to monitor the progress. + +![WebConnector](/images/quickstart/DanswerWebConnector.png) + + 3. After the pages are indexed, you can now navigate back to the homepage and start asking questions and getting + answers! 🥳 + +![SampleQA](/images/quickstart/DanswerSampleQA.png) + + +## Shutting Down + 1. `docker compose -f docker-compose.dev.yml -p danswer-stack down` + - add `-v` at the end to delete the volumes (containing users and indexed documents) \ No newline at end of file diff --git a/slack_bot_setup.mdx b/slack_bot_setup.mdx new file mode 100644 index 0000000..8471f46 --- /dev/null +++ b/slack_bot_setup.mdx @@ -0,0 +1,96 @@ +--- +title: Slack Bot Setup +description: "How to set up a Slack bot to automatically answer questions" +--- + +## How it works + +Danswer will connect to your Slack instance via Web Sockets, and listen for relevant messages. + +Slack will only send us messages for channels where DanswerBot has been manually added to. For all +channel messages (not messages inside a thread), Danswer will consider the question, search through +all available documents, and try to come up with an answer. If it finds an answer, it will respond to +the message in the **thread** as follows: + +![SlackBot](/images/slack_bot/SlackBot.png) + +Since we are using Web Sockets, Danswer is able to initiate the connection. This means that this is able +to work even if you are running Danswer inside a firewall protected VPC. + +## Setting up + +### Authorization + +**Note:** You must be an admin of the Slack workspace to set up the Slack bot. + +1. Navigate and sign in to [https://api.slack.com/apps](https://api.slack.com/apps). +2. Create a new Slack app: + - Click the **Create New App** button in the top right. + - Select **From an app manifest** option. + - Select the relevant workspace from the dropdown and click **Next**. + +![SlackApp](/images/connectors/slack/SlackApp.png) + +3. Copy the following manifest into the text box + +``` +display_information: + name: DanswerBot + description: I help answer questions! +features: + bot_user: + display_name: DanswerBot + always_online: true +oauth_config: + scopes: + bot: + - channels:history + - channels:read + - groups:history + - channels:join + - app_mentions:read + - chat:write +settings: + event_subscriptions: + bot_events: + - app_mention + - message.channels + interactivity: + is_enabled: true + org_deploy_enabled: false + socket_mode_enabled: true + token_rotation_enabled: false +``` + +5. Click the **Create** button. +6. Generate an `App-level Token`, following the guide [here](https://api.slack.com/apis/connections/socket#token). +Add the `connections:write` scope to the app-level token and click **Generate**. +Copy this somewhere safe for now, as you will need this later to initiate the Web Socket-based connection to Slack. +7. In the app page, navigate to the **OAuth & Permissions** tab under the **Features** header. +8. Under **OAuth Tokens for Your Workspace**, select `Install to Workspace` and Allow the app. +![DanswerBotInstall](/images/slack_bot/DanswerBotInstall.png) +9. Copy the **Bot User OAuth Token**, this will be needed later to enable us to respond to messages as DanswerBot. + +### Setting Env variables + +**Note:** Currently, Slack bot setup is only suppported out of the box for the docker compose based deployment. +Please contact us if you are deploying Danswer in some other way, we'd be happy to help you set up the Slack bot! + +1. [OPTIONAL] SSH into the box where you are hosting Danswer. If running locally, ignore this step. +2. Navigate to `danswer/deployment/docker_compose. +3. Open up the `.env` file using your favorite editor and add the following two lines: +``` +DANSWER_BOT_SLACK_APP_TOKEN= +DANSWER_BOT_SLACK_BOT_TOKEN= +``` +4. Restart the background service with one of the following: +``` +# if running locally +docker compose -f docker-compose.dev.yml -p danswer-stack up background -d --force-recreate + +# if running on a machine in the cloud for production use +docker compose -f docker-compose.prod.yml -p danswer-stack up background -d --build --force-recreate +``` +5. Wait a few minutes for the background component to restart, and voila you're done! +You should be able to add DanswerBot to a channel of your choosing, +ask a question, and get a response!