improve-agent
Skill1 starsChanged 3 months agoMITView on GitHub
Turn an AI-agent failure or behavior change into a reproducible improvement case with governed datasets, raw executions, deterministic and optional LLM evaluators, explicit acceptance gates, baseline/candidate comparison, human review, promoted regression tests, immutable champion history, a standalone local dashboard, and repeated-trial statistical reliability analysis. Use when evaluating or comparing agent versions, investigating failed conversations or tool calls, validating prompt/model/routing/tool changes, checking multi-agent behavior, preserving fixed incidents as permanent tests, tracking accepted agent versions over time, or checking whether an observed improvement is repeatable rather than agent/judge noise.
5 other projects use a version of this.
Adopted by
Most recently taken up by
- stempeck/agentfactorytheir own version, first seen Nov 14, 2026
- agno-agi/contexttheir own version, first seen Nov 8, 2026
- duvoai/skillstheir own version, first seen Oct 2, 2026
- farzanmrz/meta-devtheir own version, first seen Jul 6, 2026
- subarnasaikia/kaggle-skillstheir own version, first seen Apr 25, 2026
Other versions5 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 5
- agno-agi/context★ 16same name
- duvoai/skills★ 8same name
- stempeck/agentfactory★ 3same name
- subarnasaikia/kaggle-skills★ 2same name
- farzanmrz/meta-dev★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
The file
Show its rules and text
We split this skill into the rules it gives an agent
16 rules. 0 are common, 10 are distinctive, and 6 are very specific to this project.
0
Common
Most of the 5 edited versions we read keep these.
10
Distinctive
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Create evidence for an agent behavior change.
- Very specific: line 29
Read references/runner-protocol.md before creating or executing a runner.
- Very specific: line 39
Use scripts/openai_judge.py as the optional OpenAI implementation.
- Distinctive: line 54
After explicit acceptance, promote selected improved cases:
- Very specific: line 129
No evaluator runs here. aik judge reads evidence aik evaluate already produced and maps it onto the weighted criteria, hard rules and …
- Distinctive: line 232
Never hide regressions behind a better average score.
- Distinctive: line 233
Keep raw evidence separate from evaluated evidence.
- Distinctive: line 235
Treat evaluator errors as missing evidence, not agent failures.
- Very specific: line 238
Prefer deterministic contracts for tool counts, routing, confirmations, forbidden data, and turn limits.
- Very specific: line 239
Use the failure categories in references/failure-taxonomy.md when diagnosing cases.
- Distinctive: line 240
Never deploy an agent or mutate a production system as part of this workflow.
- Distinctive: line 241
Treat runner commands as project code with the current user's permissions.
- Distinctive: line 242
Never promote tests before a human explicitly accepts the candidate.
- Very specific: line 243
Never hand holdout cases, results, or reasons to a prompt optimizer.
- Distinctive: line 244
Never treat an unpriced or untimed case as free or instant.
- Distinctive: line 245
Never rank a variant on the Pareto front before it clears the acceptance policy.
“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Improve Agent
Create evidence for an agent behavior change. Treat datasets and before/after runs as permanent artifacts, not temporary output.
Workflow
- Locate the repository root and the target
improvements/anddatasets/regression.jsonlpaths. Initialize one improvement case:
python .agents/skills/improve-agent/scripts/init_case.py <case-name>- Complete
case.yamlwith the problem, hypothesis, intended change, and risks. - Define the dataset metadata, stable test IDs, behavioral expectations, risk, source, split, and status. Read
references/data-format.mdandreferences/dataset-construction.md. - Define
evaluation.jsonbefore executing the agent. Combine hard behavioral contracts with an optional semantic rubric. Readreferences/evaluation.md. Populate raw baseline and candidate evidence on the same cases. Import evidence, or execute a project-owned runner:
python .agents/skills/improve-agent/scripts/execute_cases.py improvements/<case-name> --stage baseline -- python path/to/runner.py python .agents/skills/improve-agent/scripts/execute_cases.py improvements/<case-name> --stage candidate -- python path/to/runner.pyRead
references/runner-protocol.mdbefore creating or executing a runner. Inspect untrusted runner code first. Use--resumeafter interruption,--overwritefor an intentional rerun, and--idor--tagto select cases.For LangGraph projects, read
references/langgraph-adapter.mdand usescripts/langgraph_runner.py.Evaluate each stage. Omit the command for deterministic-only evaluation, or pass an external judge after
--:python aik.py evaluate improvements/<case-name> --stage baseline python aik.py evaluate improvements/<case-name> --stage candidate -- python path/to/judge.pyUse
scripts/openai_judge.pyas the optional OpenAI implementation. AIK core does not depend on OpenAI or any other provider.Validate the case:
python .agents/skills/improve-agent/scripts/validate_case.py improvements/<case-name>Generate comparison data, acceptance recommendation, and human review:
python .agents/skills/improve-agent/scripts/run_review.py improvements/<case-name>- Review every regression, evaluator error, and non-comparable case. Treat
eligibleas ready for human approval, never as automatic acceptance. - Record the human decision in
decision.yaml. - After explicit acceptance, promote selected improved cases:
python .agents/skills/improve-agent/scripts/promote_tests.py improvements/<case-name> datasets/regression.jsonl --status improvedChampion History and Dashboard
Once a campaign is evaluated and reviewed, preserve it as immutable history and see progress over time without opening raw JSON:
python aik.py history init improvements/<case-name> --stage baseline --label "Initial version" --agent-version v1
python aik.py history record improvements/<case-name> --stage candidate --label "Fix" --agent-version v2
python aik.py history promote <run-id> # only after decision.yaml is accepted and the recommendation is eligible
python aik.py history list
python aik.py history validate
python aik.py dashboardHistory recording only archives evidence Sprint 4 already produced; it never reruns
an agent. history init runs once per history root and sets the fixed anchor and the
initial champion together. history record never moves the champion; only explicit
history promote does, and only when the archived run is hash-verified, its role is
candidate, it is not stale, its dataset/evaluator cohort matches the champion (or a
recorded cohort_anchor rerun bridges a dataset/evaluator change), its recommendation
is eligible, and its human decision is accepted. aik.py dashboard writes
aik-history/dashboard.json and dashboard.html, a standalone local report with
level, delta vs champion/anchor, slope, regressions, evaluator scores, coverage, and a
full timeline — coverage growth is never presented as quality improvement, and an
incompatible cohort never produces a fabricated delta. Read
references/history-dashboard.md for the full command reference, comparability
rules, metric formulas, and security notes.
Statistical Reliability
A single baseline/candidate run cannot tell you whether an improvement is repeatable or just agent/LLM-judge noise. Repeat the same cases and let AIK measure it:
python aik.py run improvements/<case-name> --stage baseline --repetitions 5 -- RUNNER_COMMAND
python aik.py run improvements/<case-name> --stage candidate --repetitions 5 -- RUNNER_COMMAND
python aik.py evaluate improvements/<case-name> --stage baseline --trials
python aik.py evaluate improvements/<case-name> --stage candidate --trials
python aik.py reliability improvements/<case-name>
python aik.py review improvements/<case-name>--repetitions defaults to 1 and is fully backward compatible at that default.
Above 1, every (case_id, repetition) executes independently through the same
external-process runner protocol and is evaluated by the same evaluators used for a
single run — there is no separate evaluation engine. aik reliability reads that
evidence and writes reliability.json: pass rate, Wilson confidence interval, a
deterministic seeded bootstrap for the baseline/candidate delta and P(candidate >
baseline), flaky cases, and evaluator version incompatibilities, using only the
Python standard library. Missing or errored observations are never converted into a
zero score. Add an optional reliability section to evaluation.json to gate
acceptance on this evidence — without it, behavior is unchanged from Sprint 4/5. With
it, insufficient repetitions or a probability below the configured threshold can only
push a recommendation toward manual_review/rejected, never toward eligible: a
statistical result can never override a deterministic rejection. aik history
archives reliability.json like any other artifact (hashed, tamper-detected), so an
unrepeatable candidate cannot become champion. Read
references/statistical-reliability.md for the exact formulas, config schema, and
promotion interaction.
Domain Verdict on One Case
The rest of AIK judges a version. A business reviewer needs a verdict on one case:
python aik.py judge improvements/<case-name> --case-id dossier-4711
python aik.py judge improvements/<case-name> # every caseNo evaluator runs here. aik judge reads evidence aik evaluate already produced
and maps it onto the weighted criteria, hard rules and verdict vocabulary declared in
a judge block of evaluation.json (aik validate checks it, including that every
referenced evaluator exists).
A hard rule is a policy limit, not a quality signal: one violation forces reject
regardless of the score, and a hard rule that could not be checked forces
insufficient_evidence rather than passing by default. confidence is the share of
criterion weight that produced a definite status — coverage, not a probability that
the verdict is right, and never a model's self-reported certainty. With abstention
enabled, confidence below the floor yields insufficient_evidence instead of a guess.
A criterion with no definite status is excluded from the score, not counted as zero.
Read references/domain-judge.md.
Agreement With Experts
python aik.py agreement improvements/<case-name>Reads expert_decisions.jsonl (one line per expert per case, using the judge
vocabulary), judges every case fresh, and reports Cohen's kappa with a deterministic
seeded bootstrap interval.
Report the kappa against the human ceiling: aik agreement also computes every
expert-vs-expert pair, because a judge cannot be more right than the humans are
consistent. With a single rater there is no ceiling and the report says so rather
than implying the number means something alone.
Contested cases (experts disagree) are excluded and listed — they mark an ambiguous
policy, not a wrong judge. Abstentions are excluded and reported with the remaining
coverage, because a judge that abstains on everything hard scores well on what is
left. Kappa is null with a reason, never 1.0, when chance agreement is total.
Read references/agreement.md.
Objectives and the Constrained Pareto Front
Quality alone cannot decide whether a task can move to a cheaper model. Rank evaluated variants on quality, cost and latency together:
python aik.py pareto improvements/<case-name> \
--variant "gpt-4o=improvements/<case-name>/gpt-4o.evaluated.jsonl" \
--variant "mini=improvements/<case-name>/mini.evaluated.jsonl"Feasibility is decided before anything is ranked. Each variant first runs through the
same deterministic acceptance policy the review uses, then through the optional
absolute gates in the objectives block of evaluation.json
(minimum_pass_rate, maximum_cost_per_case_usd, maximum_latency_p95_ms,
require_cost, require_latency). A rejected or manual_review variant never
reaches the front, however cheap it is — an unconstrained front would happily
surface a variant that is very cheap and fails a critical case.
cost.total_usd per case comes from the runner (protocol 1.2); AIK measures
execution.duration_ms itself. A case with no cost is unpriced, never free: a variant
with any unpriced case is not_comparable and is excluded from the ranking rather
than ranked on an invented number. All variants must share one dataset fingerprint.
cheapest_feasible is a recommendation for a reviewer, never an automatic migration.
Read references/objectives-and-pareto.md.
Optimizer Feedback
A prompt optimizer improves by reading why a case failed, so any case it sees stops measuring generalization. Export optimizer input only through:
python aik.py feedback improvements/<case-name> --stage candidateThis writes feedback.json with per-case diagnostic text for the optimizer-visible
splits (development, regression, challenge) and never the holdout split —
there is no flag to include it. The holdout appears only as a count. Every export
appends a private content seal to optimizer-log.jsonl, so the number of rounds
run against a dataset stays visible, and a holdout changed mid-optimization is
detected and reported as a warning. The command refuses to run on a legacy dataset,
on a dataset with no holdout cases, or when a case is missing its split.
AIK does not sandbox the optimizer: it makes the honest path the default and leaves
an audit trail. Read references/optimizer-feedback.md.
To run an optimizer rather than export for one, declare an optimization block in
evaluation.json (which components it may rewrite, and the candidate budget) and:
python aik.py optimize improvements/<case-name> -- python path/to/my_optimizer.pyAIK builds the feedback payload itself, so the holdout cannot reach the optimizer,
and validates what comes back: no unknown component, no over-budget candidate list,
no empty text. Candidates are written as text under optimization/candidates/ with a
proposal.json. Nothing is executed, evaluated or promoted by this command — run the
candidates through aik run/aik evaluate, then aik pareto or aik review.
examples/optimizers/rule_optimizer.py runs offline with no dependency;
examples/optimizers/gepa_optimizer.py is the optional GEPA implementation. Read
references/optimizer-protocol.md.
Evaluation Rules
- Classify
failed -> passedasimproved. - Classify
passed -> failedasregressed. - Classify equal pass/fail outcomes as
unchanged; preserve metric deltas for review. - Classify missing baseline or candidate results as
not_comparable. - Never hide regressions behind a better average score.
- Keep raw evidence separate from evaluated evidence.
- Define evaluator versions and acceptance thresholds before running the candidate.
- Treat evaluator errors as missing evidence, not agent failures.
- Label LLM-estimated satisfaction as predicted satisfaction; do not present it as real user feedback.
- Require stable result IDs across runs. When dataset, evaluator, or trial metadata is managed outside this Sprint 1 format, verify it before interpreting the comparison.
- Prefer deterministic contracts for tool counts, routing, confirmations, forbidden data, and turn limits. Read
references/behavioral-contracts.mdwhen specifying contracts. - Use the failure categories in
references/failure-taxonomy.mdwhen diagnosing cases. - Never deploy an agent or mutate a production system as part of this workflow.
- Treat runner commands as project code with the current user's permissions. AIK does not sandbox them.
- Never promote tests before a human explicitly accepts the candidate.
- Never hand
holdoutcases, results, or reasons to a prompt optimizer. Useaik feedback. - Never treat an unpriced or untimed case as free or instant; it is unmeasured evidence.
- Never rank a variant on the Pareto front before it clears the acceptance policy.
- Redact secrets and personal or internal identifiers before sharing artifacts.
Minimum Deliverables
An improvement case is complete only when it contains:
case.yaml
test_cases.jsonl
dataset.json
evaluation.json
baseline.raw.jsonl
baseline.evaluated.jsonl
candidate.raw.jsonl
candidate.evaluated.jsonl
comparison.json
review.html
decision.yamlReport the counts of improved, regressed, unchanged, and not_comparable, then link the human-readable review.html.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

