litellm/cookbook/LiteLLM_AB_TestLLMs.ipynb

LiteLLM A/B Testing LLMs in production

• LiteLLM allows you to use 100+ LLMs as a drop in replacement for gpt-3.5-turbo

This tutorial walks through how to use LiteLLM to easily A/B Test LLMs in production

Example 1: A/B Test GPT-4 & GPT-3.5

Step 1

👉 Get your id from here: https://admin.litellm.ai/

In [4]:

from litellm import completion_with_split_tests
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = ""


# define a dict of model id and % of requests for model
# see models here: https://docs.litellm.ai/docs/providers
split_per_model = {
	"gpt-4": 0.3,
	"gpt-3.5-turbo": 0.7
}

messages = [{ "content": "Hello, how are you?","role": "user"}]

completion_with_split_tests(messages=messages, use_client=True,
   id="91fad14a-8c0f-4e99-8eaa-68245435aa80") # [Optional Set your own ID]

Out [4]:

last_fetched_at: 1693624804.2941535

<OpenAIObject chat.completion id=chatcmpl-7uBT4QHc8BAoZKkU7JoH4ahmXvu0M at 0x7c2895c9e890> JSON: {
  "id": "chatcmpl-7uBT4QHc8BAoZKkU7JoH4ahmXvu0M",
  "object": "chat.completion",
  "created": 1693624806,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I'm an AI, so I don't have emotions, but I'm here to assist you. How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 29,
    "total_tokens": 42
  }
}

A/B Test GPT-4 and Claude-2

In [5]:

from litellm import completion_with_split_tests
import os

## set ENV variables
os.environ["ANTHROPIC_API_KEY"] = ""

# define a dict of model id and % of requests for model
split_per_model = {
	"gpt-4": 0.3,
	"claude-2": 0.7
}

messages = [{ "content": "Hello, how are you?","role": "user"}]


completion_with_split_tests(messages=messages, use_client=True,
   id="91fad14a-8c0f-4e99-8eaa-68245435aa80") # [Optional Set your own ID]

Out [5]:

last_fetched_at: 1693624809.3467667

<OpenAIObject chat.completion id=chatcmpl-7uBTA6gotsTksvCU7GffJ64ybfHUw at 0x7c28aa288630> JSON: {
  "id": "chatcmpl-7uBTA6gotsTksvCU7GffJ64ybfHUw",
  "object": "chat.completion",
  "created": 1693624812,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "As an AI, I don't have feelings, but I'm here and ready to assist you. How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 27,
    "total_tokens": 40
  }
}