[feature] ConfidentAI logging enabled for proxy and sdk (#10649)

* async success implemented * fail async event * sync events added * docs added * docs added * test added * style * test * . * lock file genrated due to tenacity change * mypy errors * resolved comments * resolved comments * resolved comments * resolved comments * style * style * resolved comments
2026-08-21 08:26:34 +00:00 · 2025-05-23 11:40:48 -07:00
parent e9b7059af4
commit 8da898c55e
13 changed files with 622 additions and 4933 deletions
@@ -0,0 +1,55 @@
+import Image from '@theme/IdealImage';
+
+# 🔭 DeepEval - Open-Source Evals with Tracing
+
+### What is DeepEval?
+[DeepEval](https://deepeval.com) is an open-source evaluation framework for LLMs ([Github](https://github.com/confident-ai/deepeval)). 
+
+### What is Confident AI?
+
+[Confident AI](https://documentation.confident-ai.com) (the ***deepeval*** platfrom) offers an Observatory for teams to trace and monitor LLM applications. Think Datadog for LLM apps. The observatory allows you to:
+
+- Detect and debug issues in your LLM applications in real-time
+- Search and analyze historical generation data with powerful filters
+- Collect human feedback on model responses
+- Run evaluations to measure and improve performance
+- Track costs and latency to optimize resource usage
+
+<Image img={require('../../img/deepeval_dashboard.png')} />
+
+### Quickstart
+
+```python
+import os
+import time
+import litellm
+
+
+os.environ['OPENAI_API_KEY']='<your-openai-api-key>'
+os.environ['CONFIDENT_API_KEY']='<your-confident-api-key>'
+
+litellm.success_callback = ["deepeval"]
+litellm.failure_callback = ["deepeval"]
+
+try:
+    response = litellm.completion(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "user", "content": "What's the weather like in San Francisco?"}
+        ],
+    )
+except Exception as e:
+    print(e)
+
+print(response)
+```
+
+:::info
+You can obtain your `CONFIDENT_API_KEY` by logging into [Confident AI](https://app.confident-ai.com/project) platform. 
+:::
+
+## Support & Talk with Deepeval team
+- [Confident AI Docs 📝](https://documentation.confident-ai.com)
+- [Platform 🚀](https://confident-ai.com)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Support ✉️ support@confident-ai.com
@@ -11,6 +11,7 @@ Log Proxy input, output, and exceptions using:
 - GCS, s3, Azure (Blob) Buckets
 - Lunary
 - MLflow
+- Deepeval
 - Custom Callbacks - Custom code and API endpoints
 - Langsmith
 - DataDog
@@ -1182,7 +1183,58 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
 '
 ```

+## Deepeval
+LiteLLM supports logging on [Confidential AI](https://documentation.confident-ai.com/) (The Deepeval Platform):

+### Usage:
+1. Add `deepeval` in the LiteLLM `config.yaml`
+
+```yaml
+model_list:
+  - model_name: gpt-4o
+    litellm_params:
+      model: gpt-4o
+litellm_settings:
+  success_callback: ["deepeval"]
+  failure_callback: ["deepeval"]
+```
+
+2. Set your environment variables in `.env` file. 
+```shell
+CONFIDENT_API_KEY=<your-api-key>
+```
+:::info
+You can obtain your `CONFIDENT_API_KEY` by logging into [Confident AI](https://app.confident-ai.com/project) platform. 
+:::
+
+3. Start your proxy server:
+```shell
+litellm --config config.yaml --debug
+```
+
+4. Make a request:
+```shell
+curl -X POST 'http://0.0.0.0:4000/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+    "model": "gpt-3.5-turbo",
+    "messages": [
+      {
+        "role": "system",
+        "content": "You are a helpful math tutor. Guide the user through the solution step by step."
+      },
+      {
+        "role": "user",
+        "content": "how can I solve 8x + 7 = -23"
+      }
+    ]
+}'
+```
+
+5. Check trace on platform: 
+
+<Image img={require('../../img/deepeval_visible_trace.png')} />

 ## s3 Buckets

@@ -460,6 +460,7 @@ const sidebars = {
        "observability/agentops_integration",
        "observability/langfuse_integration",
        "observability/lunary_integration",
+        "observability/deepeval_integration",
        "observability/mlflow",
        "observability/gcs_bucket_integration",
        "observability/langsmith_integration",