agentleFS
Sign inStart free
AnastasiyaWAnastasiyaW/claude-code-config/skills/development/harness-feedback/SKILL.md

harness-feedback

Skill213 starsChanged 3 days agoMITView on GitHub

Use when an agent says a test, VM, proof, evaluator, or release gate is overloaded, too strict, blocking staging, or causing false positives; split checks by profile, measure the burden, preserve high-risk evidence, and verify the smallest corrected workflow. Do not use for ordinary test selection, a single test failure, or a full security audit without a harness-scope question.

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 5 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Treat "the harness is too strict" as an engineering finding, not as permission to disable a safety check.

    line 8
  2. Distinctive:

    Use these profiles unless the project has a more specific, documented contract:

    line 14
  3. Very specific:

    Use the deterministic harness-load-advisor.py signal as an intake event.

    line 39
  4. Distinctive:

    Do not write "overkill" and move on.

    line 45
  5. Distinctive:

    Do not replace a misplaced gate with retries, sleeps, or a bypass marker.

    line 64
  6. Distinctive:

    Do not infer overload from one slow run.

    line 65

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Harness Feedback

Treat "the harness is too strict" as an engineering finding, not as permission to disable a safety check. Find the boundary that owns the mismatch and move the check to the narrowest profile that actually needs its evidence.

Profiles

Use these profiles unless the project has a more specific, documented contract:

ProfilePurposeTypical blocking checks
staging-smokeFast proof that the changed build starts and the critical path worksbuild, focused regression, one stable smoke/contract check
security-proofProve an adversarial or trust-boundary claimhostile tests, source/collector proof, fresh-context evaluator
release-attestationProve the exact releasable artifact and its identitysigning, Authenticode/tool identity, installer/package checks
nightly-stressFind intermittent and capacity failuresrace, stress, AV/OS matrix, long-running evals

staging-smoke must not require signing, production credentials, a release certificate, or a long VM stress run. security-proof may run on an unsigned staging build when its claim is source or runtime behavior. A release check may remain blocking for release promotion without becoming a per-edit gate.

Feedback Loop

For every overload signal, record:

  1. requested profile and change boundary;
  2. gate that blocked or dominated the run;
  3. command, elapsed time, failure count, and evidence actually produced;
  4. whether the gate was relevant, duplicated, flaky, or misplaced;
  5. the smallest profile split or deletion of duplicate coverage;
  6. a before/after run of the affected profile and a fresh review of the rule.

Use the deterministic harness-load-advisor.py signal as an intake event. It stores metadata outside the repository and forces the final report to name the mismatch. Durable policy changes belong in Git; raw session traces do not.

Required Report

Do not write "overkill" and move on. Report:

Harness feedback: OVERLOAD | CLEAR
Requested profile: staging-smoke | security-proof | release-attestation | nightly-stress
Mis-scoped gate: <name>
Evidence: <command, result, elapsed time, or explicit missing proof>
Correction: <profile split or rule change>
Verification: <before/after commands and result>
Residual risk: <what remains intentionally gated and where>

Gotchas

  • A fresh evaluator is an independence control, not a release-signing check.
  • A VM can be a reusable execution environment without forcing release identity checks into every VM smoke.
  • A green fast gate does not prove release readiness; a red release-only gate does not invalidate a staging smoke unless the staging claim depends on it.
  • Do not replace a misplaced gate with retries, sleeps, or a bypass marker.
  • Do not infer overload from one slow run; distinguish environment failure from a profile contract error.

Troubleshooting

SymptomLikely causeAction
Staging smoke asks for signingRelease gate leaked into staging profilesplit release-attestation and run the smoke on the unsigned staging artifact
Security proof blocks on a production VMRuntime environment and release identity are coupledkeep the VM, remove release-only assertions from the security profile
Same gate fails repeatedlyWrong scope, flaky boundary, or missing fixtureclassify the failure and add a focused reproducer; never silently retry
Agent says "tests passed" with no profileEvidence contract is incompleterequire the report fields above and the exact command/result
Fix removes a safety checkCausal ownership was not tracedrestore the check, document the narrower boundary, and re-verify it there

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free