deepeval-tracing

---
name: deepeval-tracing
description: >
  Instrument an AI application with DeepEval's native tracing so its behavior
  is visible in Confident AI. TRIGGER when the user wants to add DeepEval
  tracing or @observe to an LLM app, agent, RAG pipeline, or chatbot; wire a
  framework, model-provider, or vector-database integration (LangGraph,
  LangChain, OpenAI Agents, LlamaIndex, Pydantic AI, CrewAI, and others);
  choose between a native integration and manual instrumentation; set span
  types, tags, or metadata; or send DeepEval-SDK traces to Confident AI's
  Observatory. DO NOT TRIGGER for building DeepEval pytest eval suites,
  datasets, goldens, metrics, or deepeval test run (use the `deepeval` skill),
  or for raw OpenTelemetry / OTLP export without the deepeval package (use the
  `deepeval-otel` skill). This skill is purely DeepEval-SDK instrumentation —
  producing well-formed traces, not running evals.
license: Apache-2.0
metadata:
  author: Confident AI
  version: "1.0.0"
  category: observability
  tags: "deepeval, tracing, observe, instrumentation, integrations, spans, confident-ai"
  compatibility: "Python with `pip install deepeval`. Instrumentation uses the DeepEval SDK (`deepeval.tracing`). Sending traces to Confident AI requires `deepeval login` or an exported `CONFIDENT_API_KEY`."
---

# DeepEval Tracing

Use this skill to instrument an **AI application** — an LLM app, agent, RAG
pipeline, or chatbot — with **DeepEval's native tracing** so its execution is
visible span by span in **Confident AI's Observatory**. The work is: pick a
supported integration when one exists, fall back to manual `@observe`
otherwise, give each span a meaningful type, and add tags and metadata.

This skill stops at producing well-formed traces. Attaching evaluation metrics
and running evals is the `deepeval` skill's job.

## Scope: AI Applications Only

Instrument only the AI parts of the system — agent loops and planning, LLM
calls, retrieval / vector search, and tool calls. The span types (`llm`,
`retriever`, `tool`, `agent`) describe AI components. Do not trace non-AI
software (web servers, CRUD backends, infrastructure). If the target has no
LLM, agent, retrieval, or tool-calling component, this skill does not apply.

## When to Use vs the `deepeval` and `deepeval-otel` Skills

- **This skill (`deepeval-tracing`)** — instrument an app with the DeepEval SDK
  (`@observe`, framework integrations) so traces reach Confident AI.
- **`deepeval` skill** — build pytest eval suites: datasets, metrics, traced
  evals, `deepeval test run`, iteration. It runs evals *against* an app this
  skill instrumented.
- **`deepeval-otel` skill** — instrument with the vendor-neutral OpenTelemetry
  SDK instead of the DeepEval SDK (raw OTLP, including non-Python apps).

The three are complementary. If unsure between this skill and `deepeval-otel`:
use this one when the app is Python and you want the DeepEval SDK; use
`deepeval-otel` when you want raw OpenTelemetry or the app is not Python.

## Prerequisites

- An AI application in Python with `pip install deepeval`.
- For traces to reach Confident AI: `deepeval login`, or an exported
  `CONFIDENT_API_KEY` (preferred for CI and non-interactive runs).

## Workflow

1. Confirm the target is an AI application (it has LLM calls, an agent loop,
   retrieval, or tool calls). If it has none of these, stop — this skill does
   not apply.
2. Detect the framework, model provider, agent SDK, and vector database in use.
3. Read `references/integrations.md` and the exact integration doc for what was
   detected. Prefer a native integration over manual instrumentation.
4. If no native integration fits, instrument manually with `@observe`. Read
   `references/tracing.md`.
5. Give each span a meaningful `type` (`llm`, `retriever`, `tool`, `agent`) and
   capture inputs/outputs.
6. Add trace-level tags and metadata where they help diagnose failure patterns.
   Never trace secrets, credentials, or raw sensitive data.
7. Confirm `deepeval login` or `CONFIDENT_API_KEY`, then verify traces appear
   in the Confident AI Observatory.

## Core Principles

1. Instrument AI components only — `llm`, `retriever`, `tool`, `agent` spans.
   Never trace non-AI software.
2. Prefer a supported integration over manual `@observe`. Manual tracing is the
   fallback for unsupported frameworks and app-owned wrapper boundaries.
3. Read the exact integration doc before writing tracing code.
4. Give spans meaningful types; let names default to function names unless
   there is a strong reason to override.
5. Never trace secrets, credentials, API keys, or raw sensitive user data.
6. Producing traces is the scope. Attaching metrics and running evals belong to
   the `deepeval` skill; raw OpenTelemetry export belongs to `deepeval-otel`.

## References

| Topic | File |
| --- | --- |
| Manual instrumentation: `@observe`, span types, tags, metadata | `references/tracing.md` |
| Integration selection rule and framework / model / vector-DB doc index | `references/integrations.md` |

More agent context in confident-ai/deepeval

2 other files this repository gives its agents.