file-run-issues
Skill212 starsChanged 7 months agoApache-2.0View on GitHub
Aggregate a retort run's findings.jsonl into a machine-readable assessment.json summary with severity counts, penalty score, requirement coverage, and top findings.
In this repository6 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- compare-runsSkill
- diagnose-failed-runSkill
- evaluate-runSkill
- run-summarySkill
- update-optimal-blogSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 1 is distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 28
Read the file line-by-line.
- Very specific: line 68
If total requirements cannot be determined, set requirement_coverage to null.
- Very specific: line 76
If stack.json is absent or has no model/agent field, use "unknown".
- Very specific: line 80
Write atomically (via .tmp rename):
- Very specific: line 112
If --dry-run was specified, print the JSON to stdout and skip the file write.
- Very specific: line 126
If assessment.json is absent, FindingsScorer returns 0.5 (neutral) rather than failing.
- Very specific: line 131
Write assessment.json with all-zero severity counts, penalty_score 1.0, empty top_findings.
- Very specific: line 134
Use model: "unknown". Do not abort.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
File Run Issues
Overview
evaluate-run produces a findings.jsonl per run — one JSON object per observation. This skill aggregates those findings into assessment.json, a compact machine-readable summary used by FindingsScorer and downstream reporting. It does not create beads issues or GitHub issues.
Parameters
- run_dir (required): Same path
evaluate-runused, e.g.experiment-1/runs/language=rust_model=opus_tooling=beads/rep2/ - min_severity (optional, default:
info): Skip findings below this severity when computing counts. Options:critical,high,medium,low,info - dry_run (optional, default:
false): Print the assessment JSON without writing assessment.json
Steps
1. Load findings
test -f {run_dir}/findings.jsonl || { echo "no findings.jsonl — run evaluate-run first"; exit 1; }Read the file line-by-line. Each line is one finding with at minimum these fields:
{"id": "R3", "kind": "requirement_missing", "severity": "high", "title": "...", "evidence": "...", "suggestion": "..."}2. Filter by severity
Drop findings whose severity is below min_severity. The severity ordering from highest to lowest is: critical, high, medium, low, info.
3. Count severities
Count how many findings fall into each severity bucket:
{"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1}4. Compute penalty_score
start = 1.0
subtract: critical * 0.25 + high * 0.10 + medium * 0.03 + low * 0.01
clamp result to [0.0, 1.0]A run with no findings scores 1.0. A run with one critical finding scores 0.75. A run with four critical findings scores 0.0 (clamped).
5. Collect top findings
Select the top 5 findings by severity (critical first, then high, medium, low, info). Within the same severity level, preserve the original order from findings.jsonl. Include all fields from the original finding object.
6. Compute requirement_coverage
Count findings with kind in requirement_missing or requirement_partial — these represent requirements the agent did not fully implement. Estimate total requirements from R<N> IDs present in findings plus any implemented ones (inferred from evaluation.md if available, otherwise estimate from the highest R-number seen).
requirement_coverage = implemented_count / total_requirementsIf total requirements cannot be determined, set requirement_coverage to null.
7. Read model from stack.json
cat {run_dir}/stack.json | jq -r '.model // .agent // "unknown"'If stack.json is absent or has no model/agent field, use "unknown".
8. Write assessment.json
Write atomically (via .tmp rename):
{
"severity_counts": {"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1},
"penalty_score": 0.67,
"top_findings": [...],
"requirement_coverage": 0.75,
"model": "haiku",
"evaluated_at": "2026-04-18T21:00:00Z"
}Constraints:
- You MUST write atomically — write to
{run_dir}/assessment.json.tmpthen rename to{run_dir}/assessment.json. evaluated_atMUST be an ISO 8601 UTC timestamp.penalty_scoreMUST be rounded to 4 decimal places.requirement_coverageMAY benullif total requirements cannot be determined.
9. Emit a summary
Print a terminal-readable summary:
Assessment written to {run_dir}/assessment.json
Severity counts: critical=0 high=2 medium=5 low=3 info=1
Penalty score: 0.6700 (1.0 = clean, 0.0 = critical failures)
Req coverage: 75.0%
Model: haiku
Top finding: [high] No pagination support on GET /booksIf --dry-run was specified, print the JSON to stdout and skip the file write.
Constraints Summary
- You MUST NOT create beads issues, GitHub issues, or any external tracker records.
- You MUST write assessment.json atomically.
- You MUST be safe to re-run repeatedly — re-running overwrites assessment.json with fresh aggregation.
- You MUST respect
--dry-runby only printing what would be written. - You MUST finish quickly — this is aggregation only, no LLM calls, no network calls.
Interaction with retort
FindingsScorerinsrc/retort/scoring/scorers/findings.pyreads{run_dir}/assessment.jsonand returnspenalty_scoredirectly.- The retort CLI MAY invoke this skill automatically after
evaluate-runcompletes. - If
assessment.jsonis absent,FindingsScorerreturns 0.5 (neutral) rather than failing.
Troubleshooting
findings.jsonl is empty
- Write assessment.json with all-zero severity counts, penalty_score 1.0, empty top_findings.
stack.json is missing
- Use
model: "unknown". Do not abort.
assessment.json.tmp rename fails (permissions)
- Fall back to direct write. Log a warning.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

