diagnose-run
Skill71 starsChanged 36 days agoMITView on GitHub
Investigate a pipeline run or a suspected defect in this system, using saved analysis artifacts. Use before rating any defect, when a verdict looks wrong, or when a batch degrades. Establishes incidence from real data first.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- appsec-foundry/appsec-advisortheir own version, first seen Oct 17, 2026
- matt-whitaker/claude-team-exampletheir own version, first seen Oct 11, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- appsec-foundry/appsec-advisor★ 18same name
- matt-whitaker/claude-team-example★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository11 other files it gives its agents.
- AGENTS.mdAGENTS.md
- add-agentSkill
- analyze-tickerSkill
- diagnose-runSkill
- maintain-agent-metadataSkill
- run-pipelineSkill
- run-testsSkill
- add-agentSkill
- analyze-tickerSkill
- run-testsSkill
- validation-batchSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 45
Read the ledger, not the prose.
- Very specific: line 71
Before concluding that a code change caused a slowdown, compare token_usage.by_tier against a prior run: vendor queueing dominates …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Diagnosing a run
Measure before you rate
Do not rate a defect you have not measured. Nearly every question about this system is answerable in under a minute against the saved artifacts, and the answer routinely inverts the priority order: about half the plausible-looking findings in this repo's history turned out to be inert, and half the real ones were invisible from reading the source.
grep -rl 'SOME_FLAG' results/ | wc -l # how often does this actually fire?
ls results/*_analysis.json | wc -l # corpus sizeresults/ is operator-local: it accumulates as you run analyses, and a fresh clone
has none. Every command here assumes you have run the pipeline at least once.
Report incidence as a fraction of the corpus, not as an anecdote. "Fires on 6 of 12 tickers" and "fires on 2 of 4,621 artifacts" call for opposite responses.
What a saved artifact carries
results/*_analysis.json is the richest persisted payload. The fields worth reaching
for first:
| Field | Answers |
run_summary | verdict, debate rounds, optional failures, model provenance, cost |
artifact_statuses.<field> | did this node produce a usable artifact, and if not, error_kind |
red_flags | the persisted flag ledger — authoritative, unlike re-derived prose |
analysis_snapshot, decision_trace | the provenance contract; a rejected one blocks publication |
token_usage.by_tier / by_model / by_provider | cost, and whether flex was actually served |
prediction_snapshot | what the run predicted, for later outcome comparison |
prompts_metadata | which prompt versions produced this |
Read the ledger, not the prose. A flag re-derived from a report is the thing the
provenance work exists to prevent; the persisted red_flags list is what the run
actually recorded.
Batch-level tools
poetry run python scripts/eval_longitudinal_compare.py --run-log <path>
poetry run python scripts/cost_report.py --baseline <dir> --candidate <dir>scripts/eval_longitudinal_compare.py breaks DNS failures out by operation and host and
groups classified failures by provider — use it so a genuinely new network dependency
does not hide inside a familiar-looking one. Pass the analysis log, not a file you
teed the wrapper script into: the latter contains only the wrapper's own echo lines
and no event records, so scanning it finds nothing by construction.
scripts/cost_report.py is read-only and diffs two run sets, which is how you attribute a
cost change to a lever rather than to noise.
Attributing a change
Verdicts flip between BUY and HOLD run-to-run on identical data, so a verdict-level A/B is noise-dominated. Attribute changes through the data layer — which source won a field, which flags fired, which artifacts failed — and use repeats.
Before concluding that a code change caused a slowdown, compare token_usage.by_tier
against a prior run: vendor queueing dominates wall-clock time here.
Degradation is quiet by design
An optional cross-check seat failing is publishable, so a run can be reported OK
while producing no cross-check at all. That makes the stock look safer, because the
flags that seat would have raised simply never appear. Check run_summary for
optional failures before comparing risk tallies across runs.
Full history: docs/PROVENANCE.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

