litellm/cookbook/LiteLLM_Comparing_LLMs.ipynb

Comparing LLMs on a Test Set using LiteLLM

LiteLLM allows you to use any LLM as a drop in replacement for gpt-3.5-turbo

This notebook walks through how you can compare GPT-4 vs Claude-2 on a given test set using litellm

In [ ]:

!pip install litellm

In [16]:

from litellm import completion

# init your test set questions
questions = [
    "how do i call completion() using LiteLLM",
    "does LiteLLM support VertexAI",
    "how do I set my keys on replicate llama2?",
]


# set your prompt
prompt = """
You are a coding assistant helping users using litellm.
litellm is a light package to simplify calling OpenAI, Azure, Cohere, Anthropic, Huggingface API Endpoints. It manages:

"""

In [18]:

import os
os.environ['OPENAI_API_KEY'] = ""
os.environ['ANTHROPIC_API_KEY'] = ""

Calling gpt-3.5-turbo and claude-2 on the same questions

LiteLLM completion() allows you to call all LLMs in the same format

In [ ]:

results = [] # for storing results

models = ['gpt-3.5-turbo', 'claude-2'] # define what models you're testing, see: https://docs.litellm.ai/docs/providers
for question in questions:
    row = [question]
    for model in models:
      print("Calling:", model, "question:", question)
      response = completion( # using litellm.completion
            model=model,
            messages=[
                {'role': 'system', 'content': prompt},
                {'role': 'user', 'content': question}
            ]
      )
      answer = response.choices[0].message['content']
      row.append(answer)
      print(print("Calling:", model, "answer:", answer))

    results.append(row) # save results

Visualizing Results

In [15]:

# Create a table to visualize results
import pandas as pd

columns = ['Question'] + models
df = pd.DataFrame(results, columns=columns)

df

Out [15]:

	Question	gpt-3.5-turbo	claude-2
0	how do i call completion() using LiteLLM	To call the `completion()` function using Lite...	Here is how you can call the completion() met...
1	does LiteLLM support VertexAI	Yes, LiteLLM does support Google Cloud Vertex ...	Unfortunately, LiteLLM does not currently sup...
2	how do I set my keys on replicate llama2?	To set your keys on Replicate Llama2, follow t...	Here are the steps to set your API keys on Re...