disparate-impact-audit
Skill0 starsChanged 4 months agoMITView on GitHub
Audit any automated decision, scoring, screening, or ranking system for who bears its errors — with disaggregated error rates, proxy/construct-validity checks, appeal-suppression, monoculture effects, and rubber-stamp human review. Use whenever a model or algorithm makes or shapes consequential decisions about people: hiring, lending, benefits, healthcare, fraud flags, risk scores, content moderation, insurance, admissions, policing. Triggers on "is this algorithm fair," "audit this model," "AI screening/scoring/decision system," bias review, disparate-impact or four-fifths questions, and before deploying any system that says yes/no about a person. Do not accept a single aggregate accuracy number for these systems.
In this repository11 other files it gives its agents.
- analogy-checkSkill
- change-my-mindSkill
- disaggregateSkill
- grade-the-evidenceSkill
- hype-checkSkill
- prose-surgeonSkill
- protect-the-skillSkill
- scenario-branchSkill
- steelmanSkill
- validate-the-claimSkill
- value-framesSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 7 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 18
When a system decides things about people, one number — "94% accurate" — is a trap.
- Distinctive: line 29
Run all six. Each is a distinct way these systems harm people that aggregate accuracy conceals.
- Distinctive: line 40
Ask: what is the label actually recording?
- Distinctive: line 47
When denials are appealed, they're often overturned at high rates (80%+ in some settings) — but only a small fraction are ever appealed …
- Distinctive: line 52
Ask: how many deciders share this model, and does a rejection here follow the person elsewhere?
- Distinctive: line 64
Check appeal reversal rate and appeal *access* by group.
- Distinctive: line 112
Construct validity won't yield to more data: If the label encodes bias, a bigger model reproduces it more efficiently.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Disparate-Impact Audit
When a system decides things about people, one number — "94% accurate" — is a trap. It hides whose 6% the errors fall on, and automated errors have a consistent habit of concentrating on the people least able to absorb or contest them. A fraud system flags 48,000 people with a 93% error rate. A benefits algorithm weights nationality and pushes families into homelessness. A hiring model rejects the same applicant at every employer using it. This skill is the playbook for finding those failures before deployment — and it starts from a rule: a system that decides about people is not evaluated by its average.
The six checks
Run all six. Each is a distinct way these systems harm people that aggregate accuracy conceals.
1. Disaggregated error rates — whose errors?
- Break false-positive and false-negative rates out by group (race, gender, age, disability, income, language, geography). The aggregate can look fine while one group's error rate is multiples higher — e.g. facial-recognition false-positive rates varying by two orders of magnitude across demographic pairings.
- Apply the four-fifths rule as a screen: if a protected group's selection (or adverse) rate is below 80% of the most-favored group's, that's a legal red flag. Pooling across all decisions can wash this out — check per decision/posting, not just in aggregate.
- False positives and false negatives are not symmetric in harm. Name which one this system inflicts and on whom (a missed fraud flag vs. a wrongful denial land on different people).
2. Construct validity — is it measuring the thing, or a proxy for it?
This is the deepest check, and the one more data won't fix.
- Systems often claim to measure a target (maltreatment risk, job fit, health need) but actually measure administrative proxies that encode historical surveillance and bias: benefits receipt, prior contact with the system, diagnosis codes. A parent flagged "high risk" because their life is more surveilled by the state, not because they pose more risk.
- Ask: what is the label actually recording? If the training label is "was investigated" rather than "was actually harmful," the model reproduces who got investigated and calls it risk. That's a construct-validity failure, not a tuning problem — it will not resolve with more compute.
3. The rubber-stamp check — is "human in the loop" real?
- A human reviewer is the standard safeguard and it frequently fails silently: reviewers mirror the algorithm's recommendation (up to ~90% in some studies), or get a review window too short to think (seconds per case). That's automation with a human alibi, not human judgment.
- Test it: do humans override the system enough to matter, and — critically — do their overrides reduce disparity? In at least one benchmark case, caseworkers overriding the algorithm reduced racial disparity, meaning the humans were the fairness mechanism and the algorithm the bias source. Measure override rate and its direction.
4. Appeal suppression — who gets to contest, and who actually does?
- When denials are appealed, they're often overturned at high rates (80%+ in some settings) — but only a small fraction are ever appealed (low double digits). That gap means many wrong decisions stand simply because contesting them is hard.
- The decisive missing number is usually appeal rate disaggregated by income/race/education. Those least able to navigate an appeal absorb the most uncontested errors. If that data doesn't exist, its absence is itself a finding — flag it as the highest-value thing to measure.
5. Monoculture — correlated rejection at scale
- When many deciders use the same vendor/model, its bias isn't diluted across independent judgments — it's replicated. An applicant rejected by one becomes an applicant rejected by all of them. Per-employer bias can look small while system-wide a group is locked out everywhere at once (e.g. a slice of applicants rejected by every job using the same screener).
- Ask: how many deciders share this model, and does a rejection here follow the person elsewhere? Shared scores that persist across time and employers turn one biased judgment into a structural barrier.
6. Benchmark-vs-deployment & feedback loops
- Vendor fairness claims need the same skepticism as any performance claim (see validate-the-claim): commissioned "no bias found" studies vs. independent audits reaching the opposite result.
- Watch for feedback loops: a system trained on enforcement data directs enforcement, which generates more of the same data, which confirms the system. The output can't validate the model when the output shaped the training set.
Procedure
- Identify the decision the system makes and its consequence for the person.
- Demand disaggregated FP/FN rates by group; apply four-fifths per-decision.
- Interrogate the training label — target construct or biased proxy?
- Measure human-override rate and whether overrides cut or track disparity.
- Check appeal reversal rate and appeal access by group; flag if the disaggregation is missing.
- Map monoculture — how many deciders share the model, does rejection follow the person.
- Independent (not commissioned) validation; check for feedback loops.
- Verdict: deploy with named guardrails | pilot with instrumentation | do-not-deploy — and the specific data still needed.
Output template
SYSTEM: <what it decides> | CONSEQUENCE TO PERSON: <what a wrong call costs them>
1. ERROR RATES BY GROUP: <FP/FN disaggregated; four-fifths per-decision result>
2. CONSTRUCT VALIDITY: <target vs. proxy — what's the label actually recording?>
3. HUMAN REVIEW: <override rate; do overrides reduce or mirror disparity?>
4. APPEALS: <reversal rate; access by group; missing disaggregation?>
5. MONOCULTURE: <how many deciders share it; does rejection follow the person?>
6. VALIDATION/LOOPS: <independent audit? feedback loop present?>
MISSING DATA THAT MATTERS MOST: <the highest-value unmeasured thing>
VERDICT: deploy-with-guardrails | pilot | do-not-deploy — <why>Example (compressed)
System: AI résumé screener used by 150+ employers via one vendor.
1. ERROR RATES: aggregate adverse-impact ~zero — but disaggregated per posting,
a quarter of Black applicants and a sixth of Asian applicants hit roles
crossing the four-fifths threshold. Pooling hid it.
2. CONSTRUCT VALIDITY: "predicts good hire" trained on past hiring outcomes —
i.e. past hiring bias. Label records who got hired before, not who performs.
3. HUMAN REVIEW: recruiters mirror the AI's ranking up to ~90% of the time.
The "human in the loop" is a rubber stamp.
4. APPEALS: applicants can't appeal or even see the score. No contest path.
5. MONOCULTURE: shared model + scores that persist for months → an applicant
rejected once is rejected across every employer using it. Correlated lockout,
not distributed chance.
6. VALIDATION: vendor's own audit says "fair"; independent audit at scale found
the per-posting disparities above.
MISSING DATA: per-posting adverse-impact monitoring by group, published.
VERDICT: do-not-deploy as-is. At minimum: per-posting four-fifths monitoring,
real appeal path, and breaking score persistence across employers.Gotchas
- Aggregate fairness is not fairness. Per-decision and per-group is where these systems fail; the pooled number is designed to look clean.
- Construct validity won't yield to more data. If the label encodes bias, a bigger model reproduces it more efficiently. This is the check people most often skip.
- "Human in the loop" is a claim to test, not a safeguard to assume. Measure the override rate and its direction.
- Pairs with disaggregate (its dimensions drive the group breakdown), validate-the-claim (vendor fairness studies), and value-frames (distributive-justice frame on who bears the error).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

