From ecc6790a234c85296ee083968fa284f90436705e Mon Sep 17 00:00:00 2001
From: ishaan-jaff <ishaanjaffer0324@gmail.com>
Date: Sat, 11 Nov 2023 12:08:10 -0800
Subject: [PATCH] (docs) fast eval with proxy

---
 docs/my-website/docs/simple_proxy.md | 56 ++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/docs/my-website/docs/simple_proxy.md b/docs/my-website/docs/simple_proxy.md
index b8fb634b12..3eda0c6c51 100644
--- a/docs/my-website/docs/simple_proxy.md
+++ b/docs/my-website/docs/simple_proxy.md
@@ -243,6 +243,62 @@ python gpt4_eval.py -q '../evaluation_set/flask_evaluation.jsonl'
 ```
 </TabItem>
 
+<TabItem value="FastEval" label="Fast Eval">
+
+**Step 1: Start the local proxy**
+see supported models [here](https://docs.litellm.ai/docs/simple_proxy)
+```shell
+$ litellm --model huggingface/bigcode/starcoder
+```
+
+**Step 2: Set OpenAI API Base & Key**
+```shell
+$ export OPENAI_API_BASE=http://0.0.0.0:8000
+```
+
+Set this to anything since the proxy has the credentials
+```shell
+export OPENAI_API_KEY=anything
+```
+
+**Step 3 Run with FastEval** 
+
+**Clone FastEval**
+```shell
+# Clone this repository, make it the current working directory
+git clone --depth 1 https://github.com/FastEval/FastEval.git
+cd FastEval
+```
+
+**Set API Base on FastEval**
+
+On FastEval make the following **2 line code change** to set `OPENAI_API_BASE`
+
+https://github.com/FastEval/FastEval/pull/90/files
+```python
+try:
+    api_base = os.environ["OPENAI_API_BASE"] #changed: read api base from .env
+    if api_base == None:
+        api_base = "https://api.openai.com/v1"
+    response = await self.reply_two_attempts_with_different_max_new_tokens(
+        conversation=conversation,
+        api_base=api_base, # #changed: pass api_base
+        api_key=os.environ["OPENAI_API_KEY"],
+        temperature=temperature,
+        max_new_tokens=max_new_tokens,
+```
+
+**Run FastEval**
+Set `-b` to the benchmark you want to run. Possible values are `mt-bench`, `human-eval-plus`, `ds1000`, `cot`, `cot/gsm8k`, `cot/math`, `cot/bbh`, `cot/mmlu` and `custom-test-data`
+
+Since LiteLLM provides an OpenAI compatible proxy `-t` and `-m` don't need to change
+`-t` will remain openai
+`-m` will remain gpt-3.5
+
+```shell
+./fasteval -b human-eval-plus -t openai -m gpt-3.5-turbo
+```
+</TabItem>
 <TabItem value="mlflow" label="ML Flow Eval">
 
 MLflow provides an API `mlflow.evaluate()` to help evaluate your LLMs https://mlflow.org/docs/latest/llms/llm-evaluate/index.html