Trace Eval Loop (Session ATIF)

Repeatable export → analyze → diagnose → improve → re-export cycle for session-exported ATIF-v1.7 trajectories.

Optimize tool/harness contracts (schema, handler, response/hints, prompt assembly, export fidelity). Don't care: model, serving engine.

Relationship to sibling skills

Session API has no ATIF export endpoint. Obtain ATIF via UI session export (format: 'atif') or an already-exported file the user provides. See docs/guides/session-export-formats.md.

Hard rules

• Separate measured fact, trace interpretation, and hypothesis.
• Keyword hits in observation text are heuristics, not verified tool failures.
• Prefer the narrowest owning layer (schema → handler → response/hints → prompt → execution policy → export/telemetry). Do not paper over a contract bug with a prompt tip.
• One primary causal change per cycle when implementing. Ask before editing git-tracked product files.
• Artifacts under .libragent/work/trace-eval/<cycle-id>/ unless the user says otherwise.
• Do not run pnpm refactor:validate unless explicitly requested.
• No harness/tool contract broken → report the tool pattern and stop.

Cycle

Default max cycles: 3 (override if user says). Freeze cycle-id like 20261007-session-export-1.

1. Freeze the eval contract

Write contract.md:

• Source session id / title (if known), git revision, assistant id
• ATIF file path(s) and how they were obtained (UI export vs copy)
• User goal / success criteria for the original session (from first user step)
• In-scope tools / out-of-scope (model choice never in scope)
• Pass/fail for this eval (e.g. “no repeated identical tool call ≥3”, “no oversized observation without pagination hint”)

2. Inventory and run the analyzer

Accept one file, many files, or a directory of *_trajectory.json / trajectory.json:

python .agents/skills/trace-eval-loop/scripts/analyze_atif_trace.py \
  path/to/session_trajectory.json \
  --output .libragent/work/trace-eval/<cycle-id>/summary.json \
  --markdown .libragent/work/trace-eval/<cycle-id>/summary.md

Multiple inputs:

python .agents/skills/trace-eval-loop/scripts/analyze_atif_trace.py \
  path/a_trajectory.json path/b_trajectory.json exports/ \
  --output .libragent/work/trace-eval/<cycle-id>/summary.json

Copy or symlink the raw ATIF into the cycle dir as trajectory.json (or keep original names listed in contract.md).

Before deep interpretation, flag:

• missing/schema_version ≠ ATIF-v1.7
• empty steps / empty-trajectory placeholder message
• orphan tool observations
• absent final_metrics token fields
• unequal coverage when comparing baseline vs candidate exports

3. Reconstruct and classify

For each trajectory (and for representative slices when many):

1. Rebuild intent → tool call → observation → next tool from steps (not memory).
2. Find the first material divergence from an efficient path.
3. Count: invalid/missing args, adjacent repeats, heuristic error observations, oversized observations, unknown tools, skipped verification, unrecovered errors.
4. Read references/evidence-model.md before assigning root cause.

4. Map symptom → owning layer

Use critique-builtin-tool, lean-builtin-tool-auditor, or refactor-builtin-tool when evidence points at builtin implementation details.

5. Rank improvements

Write tasks.md using references/improvement-backlog.md.

Each item: evidence pointer (step_id / tool_call_id), owning paths, minimal change, re-eval acceptance check, confidence (high/medium/low).

Prefer repeated, high-confidence contract defects. Keep low-confidence ideas in a backlog — do not convert every correlation into a work item.

6. Act (only if user approved patching)

1. Apply the smallest fix for the top task (or user-named task).
2. Focused tests (pnpm rust:test --test … / targeted lint).
3. Re-run the same user scenario (or comparable session), re-export ATIF.
4. Bump cycle-id, analyze again, write delta.md.

Planning-only requests: stop after analysis.md + tasks.md and summarize.

7. Record

.libragent/work/trace-eval/<cycle-id>/
├── contract.md
├── trajectory.json          # or originals + manifest
├── summary.json
├── summary.md
├── analysis.md
├── tasks.md
└── delta.md                 # from cycle 2+

Use references/report-template.md for analysis.md. End with adopt | defer | iterate | inconclusive.

Stop conditions

Stop and report instead of changing code when:

• ATIF is invalid / empty / incomparable across cycles
• telemetry/export is insufficient to localize the issue
• the only evidence is hidden reasoning_content
• variance exceeds the observed difference (multi-trace)
• the tool pattern is clear but no schema/handler/response/export contract is broken