jobs-trace-analyzer
Skill12 starsChanged 6 days agoMITView on GitHub
Analyze benchmark jobs (Terminal-Bench, Harbor) in jobs/, inspect ATIF trajectory logs, tool call errors, prompt issues, timeout/circuit-breaker failures, and verifier results to identify harness bottlenecks in LibrAgent and generate evidence-based improvement plans. Trigger phrases: 'jobs 트레이스 분석', 'benchmark trace analysis', 'jobs 디렉토리 분석', '하네스 분석', '성적 감점 요인 분석', 'jobs-trace-analyzer'.
In this repository51 other files it gives its agents.
- agents.mdAGENTS.md
- .agents/skills/react-best-practices/AGENTS.mdAGENTS.md
- .agents/skills/react-native-skills/AGENTS.mdAGENTS.md
- .agents/skills/react-view-transitions/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- agent-initSkill
- agy-delegateSkill
- audit-docsSkill
- code-audit-expertSkill
- computer-diagnosisSkill
- create-builtin-toolSkill
- create-prSkill
- critique-builtin-toolSkill
- doc-auditorSkill
- docxSkill
- extract-log-debugSkill
- feature-refactor-proposalSkill
- gatekeeper-loopSkill
- harbor-harness-improvement-loopSkill
- harness-eval-loopSkill
- i18n-language-standardsSkill
- lean-builtin-tool-auditorSkill
- libragent-qualitySkill
- libr-delegateSkill
- libr-tool-e2e-harness-loopSkill
- modular-large-file-refactorSkill
- performance-auditSkill
- pptxSkill
- producer-loopSkill
- vercel-react-best-practicesSkill
- vercel-react-native-skillsSkill
- vercel-react-view-transitionsSkill
- readme-showcase-syncSkill
- refactor-builtin-toolSkill
- release-head-self-e2eSkill
- release-managerSkill
- review-local-changesSkill
- review-pr-commentsSkill
- science-searchSkill
- setup-wizardSkill
- skill-creatorSkill
- soul-awakeningSkill
- sqlite-analyzerSkill
- sync-pending-approvalsSkill
- tool-creatorSkill
- trace-analyzerSkill
- trace-eval-loopSkill
- vitepress-doc-syncSkill
- worker-loopSkill
- x-cliSkill
The file
Files
Jobs Trace & Harness Bottleneck Analyzer
Analyze benchmark run trajectories stored under jobs/<job_timestamp>/ to
pinpoint harness bottlenecks in LibrAgent (Builtin Tools, Prompts, Circuit
Breakers, Timeouts, Context Compaction) that degrade scores, and formulate
evidence-backed improvement plans.
Scope: tool sequences, observations, harness layers. Don't care: model, serving engine. No harness contract broken → pattern and stop.
Relationship to harbor-harness-improvement-loop
This skill is a lighter inventory / categorization pass over jobs/<run>/.
| Concern | Use |
| Quick pass-rate, failure buckets, tool frequency | jobs-trace-analyzer (this skill) |
| Evidence-backed BM → fix → rerun cycles | harbor-harness-improvement-loop |
Harbor’s analyzer is stricter: verifier/reward.txt as reward SSOT, ATIF-first
metrics, and explicit heuristic ≠ verified failure. Prefer that skill when
proposing product changes. Do not treat this script’s categories as causal proof.
Workflow
1. Job Inventory & Automatic Trace Analysis
python .agents/skills/jobs-trace-analyzer/scripts/analyze_jobs_trace.py \
jobs/<job_timestamp> \
--output .libragent/work/trace-analysis/<job_timestamp>-analysis.mdOptional: --verbose prints per-task tool name lists.
The script reports:
- Total trials, pass rate (%), failed trial counts
- Failure categories (see below)
- Tool call frequency (ATIF-first when metadata trajectory is empty)
heuristic_error_observations(keyword hits in ATIF observation text — not verified tool failures)
Failure categories
| Category | Meaning |
SUCCESS | Reward ≥ 1.0 |
TIMEOUT_EXCEEDED | Exception type contains Timeout |
NETWORK_API_ERROR | Connection / API network exception |
TOOL_EXECUTION_ERROR | Structured tool error fields only (rare; not observation heuristics) |
INITIALIZATION_OR_EARLY_ABORT | Zero turns |
AGENT_LOOP_OR_STUCK | ≥3 repeated adjacent identical tool calls (loop detected) |
HIGH_TURN_COUNT | Turns > 30 without repeated-call evidence (inspect manually) |
VERIFIER_FAILED_WRONG_STATE | Default failed bucket |
Tool extraction priority
agent_result.metadata.trajectorytool calls when present- Else ATIF
agent/trajectory.json→steps[].tool_calls[].function_name - Else
metadata.tool_calls_count(count only; names unavailable)
Reward prefers verifier/reward.txt, then verifier_result.rewards.reward.
2. Deep-Dive Trajectory Inspection (ATIF & Task Logs)
For each failed trial, inspect jobs/<job_timestamp>/<task_name>/:
result.json: rewards,exception_info/exception.txt, metadata turnsagent/trajectory.json(ATIF): first divergence turnverifier/reward.txt&trial.log: why verification failed
3. Harness Layer Mapping
Don't care (never diagnose, compare, or blame): model, serving engine. In this harness project, tests evaluate the harness itself. If no harness contract (tool schema, handler, prompt assembly, circuit breaker, adapter) is broken, report the tool pattern and stop.
| Symptom | Primary Owning Layer | Target Code |
| Tool parameter / schema error | Builtin MCP tools | src-tauri/src/mcp/builtin/ |
| Repeated tool loop | Circuit breaker / natural recovery | src-tauri/src/agent/llm/circuit_breaker.rs, natural_recovery.rs |
| Timeout / unfinished | Timeout multipliers / shell | src-tauri/src/mcp/builtin/workspace/utils.rs |
| Missing verification / premature end | Prompts | src-tauri/src/agent/prompt/ |
| Context loss / truncation | Compaction | src-tauri/src/agent/llm/completion/request/compact.rs |
4. Formulate Evidence-Based Improvement Plan
For each issue: symptom + trajectory evidence → root cause → smallest
intervention → verification (pnpm refactor:validate, cargo test, re-run
slice). Escalate to harbor-harness-improvement-loop for controlled BM cycles.
5. Report Output Location
Save reports under:
.libragent/work/trace-analysis/<job_name>-analysis.md
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

