xtuner-trace

---
name: xtuner-trace
description: Use when the user wants to instrument, inspect, or debug XTuner traces to reconstruct a sample/request execution path, follow cross-process call chains, or identify latency hotspots, bottleneck stages, and abnormal paths from span data.
---

# XTuner Trace

Use this skill for the current trace layer in this repository: OpenTelemetry runtime setup, the basic public API, local setup scripts, and the trace viewer.

## Current Boundary

Keep this package infrastructure-only:

- Runtime/configuration: `xtuner/v1/rl/trace/runtime.py`
- OTel SDK adapter: `xtuner/v1/rl/trace/otel_utils.py`
- Public facade: `xtuner/v1/rl/trace/__init__.py`
- Basic API: `xtuner/v1/rl/trace/api.py`
- Rollout starter preset: `xtuner/v1/rl/trace/rollout_api.py`
- Local trace tooling and viewer: `recipe/trace/`

Do not add rollout, agent, judger, Ray remote, HTTP proxy, reward, status, or session-server business semantics to `api.py`, `runtime.py`, or `otel_utils.py`. Keep rollout-specific starter behavior inside `rollout_api.py` and gated by `TraceConfig.enable_rollout_trace`.

## Viewer Choice

Choose the viewer by the observation question:

- Use `recipe/trace/viewer` when the user wants rollout-oriented inspection, such as comparing samples within a training step by stage duration, status, reward/filter metadata, or recorded sample call chain.
- Use Jaeger when the question is not sample/step-centric, such as inspecting raw spans, service boundaries, cross-process causality, backend/client calls, collector/export behavior, or custom instrumentation that does not emit the rollout/sample attributes expected by the recipe viewer.

The recipe viewer reads `traces.jsonl` and adds XTuner rollout/sample aggregation. Jaeger remains the raw trace drill-down surface for the same traced run.

## Black-Box Agent Trace Routing

When the trace request involves an agent, first ask whether this is a black-box
agent trace. In this mode, XTuner exposes an OpenAI-compatible inference endpoint
to the agent through `SessionServer`; the agent runs its native loop and calls
that endpoint for LLM generation. Tool execution, environment interaction, and
observation handling happen inside the agent runtime and are not directly
instrumented by XTuner.

If the answer is yes, or the request mentions trajectory reconstruction,
`SessionServer` request gaps, `record_synthetic_span(...)`,
`reconstructed_agent_run`, `reconstructed_tool_call.N`, XTuner viewer plus
Jaeger display of recovered tool calls, read
[references/blackbox-agent-trace.md](references/blackbox-agent-trace.md) before
planning or editing.

## Basic API

These are the interfaces defined in `xtuner/v1/rl/trace/api.py` and re-exported
from `xtuner.v1.rl.trace`:

- `trace_span(name, attributes=None, parent_carrier=None)`
- `trace_function(name=None, attributes=None)`
- `trace_event(name, attributes=None)`
- `set_trace_attributes(attributes)`
- `inject_trace_context(carrier=None)`

## Runtime API

Use these `xtuner/v1/rl/trace/runtime.py` interfaces for explicit trace
runtime setup:

- `TraceConfig`
- `configure_trace(...)`
- `close_trace()`

## Rollout Starter Preset

`TraceConfig(enable_rollout_trace=True)` enables the built-in rollout starter
trace. Its helpers live in `xtuner/v1/rl/trace/rollout_api.py`:

- `trace_rollout_endpoint(...)`
- `trace_rollout_remote(...)`

Keep this layer narrow: it may depend on `RolloutState` and Ray call boundaries,
but it must not leak those dependencies into the basic API.

Rollout endpoint `xtuner.span_name_path` is a sample call chain, not just the
current process context stack. `trace_rollout_endpoint(...)` should append the
endpoint span name to an internal trace-only chain carried in
`RolloutState.extra_fields`, pass that state through remote rollout boundaries,
and clean the internal field at the outermost endpoint. For example, after a
sample generates and then judges, `judger.run` should record a path like
`single_turn_agent_loop.run -> rollout.controller.generate -> rollout.worker.generate -> judger.run`
even though the local active context at judge time may only be
`single_turn_agent_loop.run -> judger.run`. Keep this behavior in
`rollout_api.py` and viewer tests; do not add rollout call-chain semantics to
the basic trace API.

## Add Trace Workflow

When adding trace instrumentation to an XTuner run:

1. Ask for or locate the launch script and training config before editing.
2. In the launch script, source `recipe/trace/setup_trace.sh` when
   `XTUNER_TRACE_ENABLED=1`.
3. In the training config, add `TraceConfig` and set `enabled=True`; set
   `xtuner_viewer_enabled=True` when the user needs interactive sample/step inspection. Set
   `enable_rollout_trace=True` only when the user wants the built-in rollout
   starter trace.
4. Before adding any `trace_span(...)` instrumentation, you must ask the user which
   stages they want to observe and which metrics each stage should expose.
   Do not infer default stages unless the user explicitly asks you to choose.
5. Add `trace_span(...)` only around the user-confirmed observed stages. Put fields known at span
   start in initial attributes, and update runtime or final fields with
   `set_trace_attributes(...)`.
6. For cross-process or request boundaries, inject a carrier with
   `inject_trace_context(...)` and pass it to downstream
   `trace_span(..., parent_carrier=carrier)`.
7. Keep transport-specific propagation at the caller boundary; do not move
   rollout, agent, judger, Ray, or HTTP semantics into the basic trace package.
8. Ensure the main training log includes the trace output path, XTuner viewer URL,
   and equivalent manual viewer command when the XTuner viewer is enabled.

## Guardrails

- Do not recreate `trace_utils.py`, `context_propagation.py`, span-name registries, or business attribute builders under `xtuner/v1/rl/trace`.
- Do not import `RolloutState`, agent item classes, judgers, rollout workers, Ray actors, aiohttp clients, or trainer configs from the basic trace package.
- Do not call OpenTelemetry SDK directly from business code; use the basic API only when trace instrumentation is explicitly requested.
- Do not record prompts, responses, full configs, secrets, raw headers, stack traces, or large payloads as attributes.
- Viewer stage grouping must come from span attributes such as `xtuner.stage`, `stage`, or `stage.name`; otherwise fall back to the raw span name.

For concrete patterns, read [references/trace-patterns.md](references/trace-patterns.md) before editing trace code.

## Verification

Use focused checks:

- `PYTHONPATH=. python -m compileall -q xtuner/v1/rl/trace recipe/trace`
- `PYTHONPATH=. python -m unittest discover -s tests/rl -p 'test_trace*.py' -v` when the trace tests exist in the worktree.
- `git diff --check`

More agent context in InternLM/xtuner

10 other files this repository gives its agents.