mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-03 17:11:24 +00:00
arc53-machine
4e1002f5e0
Covers what is recorded, where it is shown, the TRACES_* settings, the gen_ai.* spans and metrics, content capture, and the limits of exporting spans when the request ends.
194 lines
7.7 KiB
Plaintext
194 lines
7.7 KiB
Plaintext
---
|
||
title: Observability
|
||
description: Send traces, metrics, and logs from DocsGPT to any OpenTelemetry-compatible backend (Axiom, Honeycomb, Grafana, Datadog, Jaeger, etc.).
|
||
---
|
||
|
||
import { Callout } from 'nextra/components'
|
||
|
||
# Observability
|
||
|
||
DocsGPT bundles the OpenTelemetry SDK and auto-instrumentation packages
|
||
in `docsgpt/requirements.txt` — they install with the rest of the
|
||
backend deps. Telemetry is **off by default**; opt in by prefixing the
|
||
launch command with `opentelemetry-instrument` and setting OTLP env
|
||
vars.
|
||
|
||
Auto-instrumentation covers Flask, Starlette, Celery, SQLAlchemy,
|
||
psycopg, Redis, requests, and Python logging. Agent runs, LLM calls,
|
||
tool calls and retrieval are recorded by DocsGPT itself and exported as
|
||
OpenTelemetry GenAI spans — see [Execution traces](#execution-traces).
|
||
|
||
## Enabling
|
||
|
||
Set these env vars in your `.env` (or compose `environment:` block):
|
||
|
||
```bash
|
||
OTEL_SDK_DISABLED=false
|
||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector.example.com
|
||
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<token>
|
||
OTEL_TRACES_EXPORTER=otlp
|
||
OTEL_METRICS_EXPORTER=otlp
|
||
OTEL_LOGS_EXPORTER=otlp
|
||
OTEL_PYTHON_LOG_CORRELATION=true
|
||
OTEL_RESOURCE_ATTRIBUTES=service.name=docsgpt-backend,deployment.environment=prod
|
||
```
|
||
|
||
Then prefix the process command with `opentelemetry-instrument`. The
|
||
simplest way is a compose override (no image rebuild):
|
||
|
||
```yaml
|
||
# deployment/docker-compose.override.yaml
|
||
services:
|
||
backend:
|
||
command: >
|
||
opentelemetry-instrument gunicorn -w 1 -k uvicorn_worker.UvicornWorker
|
||
--bind 0.0.0.0:7091 --config docsgpt/gunicorn_conf.py
|
||
docsgpt.asgi:asgi_app
|
||
environment:
|
||
- OTEL_SERVICE_NAME=docsgpt-backend
|
||
worker:
|
||
command: opentelemetry-instrument celery -A docsgpt.app.celery worker -l INFO -B
|
||
environment:
|
||
- OTEL_SERVICE_NAME=docsgpt-celery-worker
|
||
```
|
||
|
||
For local dev, prepend `dotenv run --` so the `OTEL_*` vars from `.env`
|
||
reach `opentelemetry-instrument` before it boots the SDK:
|
||
|
||
```bash
|
||
dotenv run -- opentelemetry-instrument flask --app docsgpt/app.py run --port=7091
|
||
dotenv run -- opentelemetry-instrument celery -A docsgpt.app.celery worker -l INFO --pool=solo
|
||
```
|
||
|
||
|
||
<Callout type="info" emoji="ℹ️">
|
||
Logs are exported in-process when `OTEL_LOGS_EXPORTER=otlp` is set —
|
||
`docsgpt/core/logging_config.py` detects the flag and preserves
|
||
the OTEL log handler. Without it, `logging` writes only to stdout.
|
||
</Callout>
|
||
|
||
## Execution traces
|
||
|
||
Every request records an **execution trace**: a timed tree of the steps
|
||
behind it. Traces are recorded for chat turns (`/stream`, `/api/answer`,
|
||
`/v1/chat/completions`, including each round of a tool-approval pause),
|
||
scheduled and webhook runs, workflows, the research agent, `/api/search`,
|
||
the MCP `search_docs` tool, and graph builds.
|
||
|
||
| Step | Recorded when |
|
||
| --- | --- |
|
||
| `invoke_agent` | An agent (or a workflow node's agent) runs |
|
||
| `chat` | An LLM call made during the request, including retries, fallbacks, query rephrasing, prescreening, history compression and guardrail judges |
|
||
| `execute_tool` | A tool is executed, paused for approval, denied or skipped |
|
||
| `retrieval` | A retriever or the multi-source dispatcher searches |
|
||
| `embeddings` | The query is embedded |
|
||
| `search` | One source is searched |
|
||
| `rerank` | Prescreening filters retrieved chunks |
|
||
| `guardrail` | A guardrail calls a remote check or fires |
|
||
| `step` | A workflow node or research phase runs |
|
||
|
||
Traces are stored in the `request_traces` table and shown in the app: open
|
||
**Settings → Logs** (or an agent's **Logs** tab), expand an entry and choose
|
||
**View trace** to see a waterfall of every step with its timing, tokens,
|
||
cost and details.
|
||
|
||
Stored traces keep short previews — tool arguments and results, retrieved
|
||
chunk titles and snippets, rephrased queries, answer excerpts — truncated
|
||
and with secret-named fields redacted. Full prompts are never stored. When a
|
||
guardrail fires during a request, every preview is dropped from its trace.
|
||
|
||
```bash
|
||
TRACES_ENABLED=true # record traces at all
|
||
TRACES_CAPTURE_CONTENT=true # keep previews in stored traces
|
||
TRACES_PREVIEW_CHARS=2000 # characters kept per preview
|
||
TRACES_MAX_SPANS=500 # steps kept per trace; the rest are counted
|
||
TRACES_RETENTION_DAYS=30 # a daily task deletes older traces
|
||
TRACES_OTEL_EXPORT=true # also export traces as OTel GenAI spans
|
||
```
|
||
|
||
### GenAI spans and metrics
|
||
|
||
When DocsGPT runs under `opentelemetry-instrument`, each finished trace is
|
||
also exported as spans that follow the
|
||
[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):
|
||
`invoke_agent {agent}`, `chat {model}`, `execute_tool {tool}`,
|
||
`embeddings {model}` and `retrieval`, with attributes such as
|
||
`gen_ai.provider.name`, `gen_ai.request.model`,
|
||
`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`,
|
||
`gen_ai.usage.cache_read.input_tokens`, `gen_ai.conversation.id`,
|
||
`gen_ai.agent.id` and `gen_ai.tool.name`. DocsGPT-specific details use the
|
||
`docsgpt.*` prefix (`docsgpt.request_id`, `docsgpt.token_source`,
|
||
`docsgpt.cache_hit`, `docsgpt.ttft_ms`, ...). The trace's root span is a
|
||
child of the request's HTTP server span, and the stored trace keeps the
|
||
OTel trace id so you can move between the two.
|
||
|
||
Two metrics are recorded for every model call:
|
||
`gen_ai.client.token.usage` and `gen_ai.client.operation.duration`.
|
||
|
||
Backends that understand the GenAI conventions — Langfuse
|
||
(`/api/public/otel`), Arize Phoenix, Datadog LLM Observability, Grafana —
|
||
render these as LLM traces with token and cost views.
|
||
|
||
Prompt and tool content is **not** exported by default, because the OTLP
|
||
backend may be a third party. Opt in with the standard variable:
|
||
|
||
```bash
|
||
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY
|
||
```
|
||
|
||
This adds the same redacted previews the app stores (for example
|
||
`gen_ai.tool.call.arguments` and `gen_ai.tool.call.result`).
|
||
|
||
<Callout type="info" emoji="ℹ️">
|
||
GenAI spans are exported when the request finishes, with their original
|
||
timestamps. Consequences: a long research run appears only when it ends;
|
||
HTTP and database spans made during a step sit beside the step rather than
|
||
under it; and log records carry the request's span ids, not the step's.
|
||
</Callout>
|
||
|
||
The GenAI conventions are still in development upstream, so attribute names
|
||
may change in later releases.
|
||
|
||
## Backend examples
|
||
|
||
### Axiom
|
||
|
||
```bash
|
||
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.axiom.co
|
||
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20xaat-XXXX,X-Axiom-Dataset=docsgpt
|
||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||
```
|
||
|
||
`%20` is the URL-encoded space between `Bearer` and the token. Create
|
||
the dataset in the Axiom UI before sending.
|
||
|
||
### Self-hosted OTLP collector / Jaeger / Tempo
|
||
|
||
```bash
|
||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
|
||
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
|
||
```
|
||
|
||
### Honeycomb / Grafana Cloud / Datadog
|
||
|
||
Each vendor publishes a single-line `OTEL_EXPORTER_OTLP_ENDPOINT` plus
|
||
`OTEL_EXPORTER_OTLP_HEADERS` recipe — drop them in alongside the
|
||
service-name override.
|
||
|
||
## Caveats
|
||
|
||
- The Dockerfile uses `gunicorn -w 1`. If you raise worker count, move
|
||
SDK init into a `post_worker_init` hook to avoid one-thread-per-process
|
||
exporter contention.
|
||
- `asgi.py` wraps Flask in Starlette's `WSGIMiddleware`. Both
|
||
instrumentors are installed, so each request produces a Starlette
|
||
span enclosing a Flask span. Drop
|
||
`opentelemetry-instrumentation-flask` from `requirements.txt` if the
|
||
duplication is noisy.
|
||
- OTEL packages add ~50 MB to the image. They install on every build —
|
||
the runtime cost is zero unless you set `opentelemetry-instrument` on
|
||
the command and set the OTLP env vars.
|
||
- The OTEL exporter ecosystem currently caps `protobuf` at `<7`, so the
|
||
backend runs on protobuf 6.x. This will catch up in a future OTEL
|
||
release. |