agentleFS
fritzprixfritzprix/libr-agent/.agents/skills/jobs-trace-analyzer/SKILL.md

jobs-trace-analyzer

Skill12 starsChanged 6 days agoMITView on GitHub

Analyze benchmark jobs (Terminal-Bench, Harbor) in jobs/, inspect ATIF trajectory logs, tool call errors, prompt issues, timeout/circuit-breaker failures, and verifier results to identify harness bottlenecks in LibrAgent and generate evidence-based improvement plans. Trigger phrases: 'jobs 트레이스 분석', 'benchmark trace analysis', 'jobs 디렉토리 분석', '하네스 분석', '성적 감점 요인 분석', 'jobs-trace-analyzer'.

The file

Files
SKILL.md

Jobs Trace & Harness Bottleneck Analyzer

Analyze benchmark run trajectories stored under jobs/<job_timestamp>/ to pinpoint harness bottlenecks in LibrAgent (Builtin Tools, Prompts, Circuit Breakers, Timeouts, Context Compaction) that degrade scores, and formulate evidence-backed improvement plans.

Scope: tool sequences, observations, harness layers. Don't care: model, serving engine. No harness contract broken → pattern and stop.

Relationship to harbor-harness-improvement-loop

This skill is a lighter inventory / categorization pass over jobs/<run>/.

ConcernUse
Quick pass-rate, failure buckets, tool frequencyjobs-trace-analyzer (this skill)
Evidence-backed BM → fix → rerun cyclesharbor-harness-improvement-loop

Harbor’s analyzer is stricter: verifier/reward.txt as reward SSOT, ATIF-first metrics, and explicit heuristic ≠ verified failure. Prefer that skill when proposing product changes. Do not treat this script’s categories as causal proof.

Workflow

1. Job Inventory & Automatic Trace Analysis

python .agents/skills/jobs-trace-analyzer/scripts/analyze_jobs_trace.py \
  jobs/<job_timestamp> \
  --output .libragent/work/trace-analysis/<job_timestamp>-analysis.md

Optional: --verbose prints per-task tool name lists.

The script reports:

  • Total trials, pass rate (%), failed trial counts
  • Failure categories (see below)
  • Tool call frequency (ATIF-first when metadata trajectory is empty)
  • heuristic_error_observations (keyword hits in ATIF observation text — not verified tool failures)

Failure categories

CategoryMeaning
SUCCESSReward ≥ 1.0
TIMEOUT_EXCEEDEDException type contains Timeout
NETWORK_API_ERRORConnection / API network exception
TOOL_EXECUTION_ERRORStructured tool error fields only (rare; not observation heuristics)
INITIALIZATION_OR_EARLY_ABORTZero turns
AGENT_LOOP_OR_STUCK≥3 repeated adjacent identical tool calls (loop detected)
HIGH_TURN_COUNTTurns > 30 without repeated-call evidence (inspect manually)
VERIFIER_FAILED_WRONG_STATEDefault failed bucket

Tool extraction priority

  1. agent_result.metadata.trajectory tool calls when present
  2. Else ATIF agent/trajectory.json → steps[].tool_calls[].function_name
  3. Else metadata.tool_calls_count (count only; names unavailable)

Reward prefers verifier/reward.txt, then verifier_result.rewards.reward.


2. Deep-Dive Trajectory Inspection (ATIF & Task Logs)

For each failed trial, inspect jobs/<job_timestamp>/<task_name>/:

  1. result.json: rewards, exception_info / exception.txt, metadata turns
  2. agent/trajectory.json (ATIF): first divergence turn
  3. verifier/reward.txt & trial.log: why verification failed

3. Harness Layer Mapping

Don't care (never diagnose, compare, or blame): model, serving engine. In this harness project, tests evaluate the harness itself. If no harness contract (tool schema, handler, prompt assembly, circuit breaker, adapter) is broken, report the tool pattern and stop.

SymptomPrimary Owning LayerTarget Code
Tool parameter / schema errorBuiltin MCP toolssrc-tauri/src/mcp/builtin/
Repeated tool loopCircuit breaker / natural recoverysrc-tauri/src/agent/llm/circuit_breaker.rs, natural_recovery.rs
Timeout / unfinishedTimeout multipliers / shellsrc-tauri/src/mcp/builtin/workspace/utils.rs
Missing verification / premature endPromptssrc-tauri/src/agent/prompt/
Context loss / truncationCompactionsrc-tauri/src/agent/llm/completion/request/compact.rs

4. Formulate Evidence-Based Improvement Plan

For each issue: symptom + trajectory evidence → root cause → smallest intervention → verification (pnpm refactor:validate, cargo test, re-run slice). Escalate to harbor-harness-improvement-loop for controlled BM cycles.


5. Report Output Location

Save reports under:

.libragent/work/trace-analysis/<job_name>-analysis.md

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.