agentleFS
adriancoadrianco/retort/skills/file-run-issues/SKILL.md

file-run-issues

Skill212 starsChanged 7 months agoApache-2.0View on GitHub

Aggregate a retort run's findings.jsonl into a machine-readable assessment.json summary with severity counts, penalty score, requirement coverage, and top findings.

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 1 is distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Read the file line-by-line.

    line 28
  2. Very specific:

    If total requirements cannot be determined, set requirement_coverage to null.

    line 68
  3. Very specific:

    If stack.json is absent or has no model/agent field, use "unknown".

    line 76
  4. Very specific:

    Write atomically (via .tmp rename):

    line 80
  5. Very specific:

    If --dry-run was specified, print the JSON to stdout and skip the file write.

    line 112
  6. Very specific:

    If assessment.json is absent, FindingsScorer returns 0.5 (neutral) rather than failing.

    line 126
  7. Very specific:

    Write assessment.json with all-zero severity counts, penalty_score 1.0, empty top_findings.

    line 131
  8. Very specific:

    Use model: "unknown". Do not abort.

    line 134

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

File Run Issues

Overview

evaluate-run produces a findings.jsonl per run — one JSON object per observation. This skill aggregates those findings into assessment.json, a compact machine-readable summary used by FindingsScorer and downstream reporting. It does not create beads issues or GitHub issues.

Parameters

  • run_dir (required): Same path evaluate-run used, e.g. experiment-1/runs/language=rust_model=opus_tooling=beads/rep2/
  • min_severity (optional, default: info): Skip findings below this severity when computing counts. Options: critical, high, medium, low, info
  • dry_run (optional, default: false): Print the assessment JSON without writing assessment.json

Steps

1. Load findings

test -f {run_dir}/findings.jsonl || { echo "no findings.jsonl — run evaluate-run first"; exit 1; }

Read the file line-by-line. Each line is one finding with at minimum these fields:

{"id": "R3", "kind": "requirement_missing", "severity": "high", "title": "...", "evidence": "...", "suggestion": "..."}

2. Filter by severity

Drop findings whose severity is below min_severity. The severity ordering from highest to lowest is: critical, high, medium, low, info.

3. Count severities

Count how many findings fall into each severity bucket:

{"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1}

4. Compute penalty_score

start = 1.0
subtract: critical * 0.25 + high * 0.10 + medium * 0.03 + low * 0.01
clamp result to [0.0, 1.0]

A run with no findings scores 1.0. A run with one critical finding scores 0.75. A run with four critical findings scores 0.0 (clamped).

5. Collect top findings

Select the top 5 findings by severity (critical first, then high, medium, low, info). Within the same severity level, preserve the original order from findings.jsonl. Include all fields from the original finding object.

6. Compute requirement_coverage

Count findings with kind in requirement_missing or requirement_partial — these represent requirements the agent did not fully implement. Estimate total requirements from R<N> IDs present in findings plus any implemented ones (inferred from evaluation.md if available, otherwise estimate from the highest R-number seen).

requirement_coverage = implemented_count / total_requirements

If total requirements cannot be determined, set requirement_coverage to null.

7. Read model from stack.json

cat {run_dir}/stack.json | jq -r '.model // .agent // "unknown"'

If stack.json is absent or has no model/agent field, use "unknown".

8. Write assessment.json

Write atomically (via .tmp rename):

{
  "severity_counts": {"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1},
  "penalty_score": 0.67,
  "top_findings": [...],
  "requirement_coverage": 0.75,
  "model": "haiku",
  "evaluated_at": "2026-04-18T21:00:00Z"
}

Constraints:

  • You MUST write atomically — write to {run_dir}/assessment.json.tmp then rename to {run_dir}/assessment.json.
  • evaluated_at MUST be an ISO 8601 UTC timestamp.
  • penalty_score MUST be rounded to 4 decimal places.
  • requirement_coverage MAY be null if total requirements cannot be determined.

9. Emit a summary

Print a terminal-readable summary:

Assessment written to {run_dir}/assessment.json
  Severity counts: critical=0 high=2 medium=5 low=3 info=1
  Penalty score:   0.6700  (1.0 = clean, 0.0 = critical failures)
  Req coverage:    75.0%
  Model:           haiku
  Top finding:     [high] No pagination support on GET /books

If --dry-run was specified, print the JSON to stdout and skip the file write.

Constraints Summary

  • You MUST NOT create beads issues, GitHub issues, or any external tracker records.
  • You MUST write assessment.json atomically.
  • You MUST be safe to re-run repeatedly — re-running overwrites assessment.json with fresh aggregation.
  • You MUST respect --dry-run by only printing what would be written.
  • You MUST finish quickly — this is aggregation only, no LLM calls, no network calls.

Interaction with retort

  • FindingsScorer in src/retort/scoring/scorers/findings.py reads {run_dir}/assessment.json and returns penalty_score directly.
  • The retort CLI MAY invoke this skill automatically after evaluate-run completes.
  • If assessment.json is absent, FindingsScorer returns 0.5 (neutral) rather than failing.

Troubleshooting

findings.jsonl is empty

  • Write assessment.json with all-zero severity counts, penalty_score 1.0, empty top_findings.

stack.json is missing

  • Use model: "unknown". Do not abort.

assessment.json.tmp rename fails (permissions)

  • Fall back to direct write. Log a warning.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.