agentleFS
natexai2026natexai2026/2030-skills/skills/disparate-impact-audit/SKILL.md

disparate-impact-audit

Skill0 starsChanged 4 months agoMITView on GitHub

Audit any automated decision, scoring, screening, or ranking system for who bears its errors — with disaggregated error rates, proxy/construct-validity checks, appeal-suppression, monoculture effects, and rubber-stamp human review. Use whenever a model or algorithm makes or shapes consequential decisions about people: hiring, lending, benefits, healthcare, fraud flags, risk scores, content moderation, insurance, admissions, policing. Triggers on "is this algorithm fair," "audit this model," "AI screening/scoring/decision system," bias review, disparate-impact or four-fifths questions, and before deploying any system that says yes/no about a person. Do not accept a single aggregate accuracy number for these systems.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 7 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When a system decides things about people, one number — "94% accurate" — is a trap.

    line 18
  2. Distinctive:

    Run all six. Each is a distinct way these systems harm people that aggregate accuracy conceals.

    line 29
  3. Distinctive:

    Ask: what is the label actually recording?

    line 40
  4. Distinctive:

    When denials are appealed, they're often overturned at high rates (80%+ in some settings) — but only a small fraction are ever appealed …

    line 47
  5. Distinctive:

    Ask: how many deciders share this model, and does a rejection here follow the person elsewhere?

    line 52
  6. Distinctive:

    Check appeal reversal rate and appeal *access* by group.

    line 64
  7. Distinctive:

    Construct validity won't yield to more data: If the label encodes bias, a bigger model reproduces it more efficiently.

    line 112

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Disparate-Impact Audit

When a system decides things about people, one number — "94% accurate" — is a trap. It hides whose 6% the errors fall on, and automated errors have a consistent habit of concentrating on the people least able to absorb or contest them. A fraud system flags 48,000 people with a 93% error rate. A benefits algorithm weights nationality and pushes families into homelessness. A hiring model rejects the same applicant at every employer using it. This skill is the playbook for finding those failures before deployment — and it starts from a rule: a system that decides about people is not evaluated by its average.

The six checks

Run all six. Each is a distinct way these systems harm people that aggregate accuracy conceals.

1. Disaggregated error rates — whose errors?

  • Break false-positive and false-negative rates out by group (race, gender, age, disability, income, language, geography). The aggregate can look fine while one group's error rate is multiples higher — e.g. facial-recognition false-positive rates varying by two orders of magnitude across demographic pairings.
  • Apply the four-fifths rule as a screen: if a protected group's selection (or adverse) rate is below 80% of the most-favored group's, that's a legal red flag. Pooling across all decisions can wash this out — check per decision/posting, not just in aggregate.
  • False positives and false negatives are not symmetric in harm. Name which one this system inflicts and on whom (a missed fraud flag vs. a wrongful denial land on different people).

2. Construct validity — is it measuring the thing, or a proxy for it?

This is the deepest check, and the one more data won't fix.

  • Systems often claim to measure a target (maltreatment risk, job fit, health need) but actually measure administrative proxies that encode historical surveillance and bias: benefits receipt, prior contact with the system, diagnosis codes. A parent flagged "high risk" because their life is more surveilled by the state, not because they pose more risk.
  • Ask: what is the label actually recording? If the training label is "was investigated" rather than "was actually harmful," the model reproduces who got investigated and calls it risk. That's a construct-validity failure, not a tuning problem — it will not resolve with more compute.

3. The rubber-stamp check — is "human in the loop" real?

  • A human reviewer is the standard safeguard and it frequently fails silently: reviewers mirror the algorithm's recommendation (up to ~90% in some studies), or get a review window too short to think (seconds per case). That's automation with a human alibi, not human judgment.
  • Test it: do humans override the system enough to matter, and — critically — do their overrides reduce disparity? In at least one benchmark case, caseworkers overriding the algorithm reduced racial disparity, meaning the humans were the fairness mechanism and the algorithm the bias source. Measure override rate and its direction.

4. Appeal suppression — who gets to contest, and who actually does?

  • When denials are appealed, they're often overturned at high rates (80%+ in some settings) — but only a small fraction are ever appealed (low double digits). That gap means many wrong decisions stand simply because contesting them is hard.
  • The decisive missing number is usually appeal rate disaggregated by income/race/education. Those least able to navigate an appeal absorb the most uncontested errors. If that data doesn't exist, its absence is itself a finding — flag it as the highest-value thing to measure.

5. Monoculture — correlated rejection at scale

  • When many deciders use the same vendor/model, its bias isn't diluted across independent judgments — it's replicated. An applicant rejected by one becomes an applicant rejected by all of them. Per-employer bias can look small while system-wide a group is locked out everywhere at once (e.g. a slice of applicants rejected by every job using the same screener).
  • Ask: how many deciders share this model, and does a rejection here follow the person elsewhere? Shared scores that persist across time and employers turn one biased judgment into a structural barrier.

6. Benchmark-vs-deployment & feedback loops

  • Vendor fairness claims need the same skepticism as any performance claim (see validate-the-claim): commissioned "no bias found" studies vs. independent audits reaching the opposite result.
  • Watch for feedback loops: a system trained on enforcement data directs enforcement, which generates more of the same data, which confirms the system. The output can't validate the model when the output shaped the training set.

Procedure

  1. Identify the decision the system makes and its consequence for the person.
  2. Demand disaggregated FP/FN rates by group; apply four-fifths per-decision.
  3. Interrogate the training label — target construct or biased proxy?
  4. Measure human-override rate and whether overrides cut or track disparity.
  5. Check appeal reversal rate and appeal access by group; flag if the disaggregation is missing.
  6. Map monoculture — how many deciders share the model, does rejection follow the person.
  7. Independent (not commissioned) validation; check for feedback loops.
  8. Verdict: deploy with named guardrails | pilot with instrumentation | do-not-deploy — and the specific data still needed.

Output template

SYSTEM: <what it decides> | CONSEQUENCE TO PERSON: <what a wrong call costs them>

1. ERROR RATES BY GROUP: <FP/FN disaggregated; four-fifths per-decision result>
2. CONSTRUCT VALIDITY: <target vs. proxy — what's the label actually recording?>
3. HUMAN REVIEW: <override rate; do overrides reduce or mirror disparity?>
4. APPEALS: <reversal rate; access by group; missing disaggregation?>
5. MONOCULTURE: <how many deciders share it; does rejection follow the person?>
6. VALIDATION/LOOPS: <independent audit? feedback loop present?>

MISSING DATA THAT MATTERS MOST: <the highest-value unmeasured thing>
VERDICT: deploy-with-guardrails | pilot | do-not-deploy — <why>

Example (compressed)

System: AI résumé screener used by 150+ employers via one vendor.

1. ERROR RATES: aggregate adverse-impact ~zero — but disaggregated per posting,
   a quarter of Black applicants and a sixth of Asian applicants hit roles
   crossing the four-fifths threshold. Pooling hid it.
2. CONSTRUCT VALIDITY: "predicts good hire" trained on past hiring outcomes —
   i.e. past hiring bias. Label records who got hired before, not who performs.
3. HUMAN REVIEW: recruiters mirror the AI's ranking up to ~90% of the time.
   The "human in the loop" is a rubber stamp.
4. APPEALS: applicants can't appeal or even see the score. No contest path.
5. MONOCULTURE: shared model + scores that persist for months → an applicant
   rejected once is rejected across every employer using it. Correlated lockout,
   not distributed chance.
6. VALIDATION: vendor's own audit says "fair"; independent audit at scale found
   the per-posting disparities above.

MISSING DATA: per-posting adverse-impact monitoring by group, published.
VERDICT: do-not-deploy as-is. At minimum: per-posting four-fifths monitoring,
   real appeal path, and breaking score persistence across employers.

Gotchas

  • Aggregate fairness is not fairness. Per-decision and per-group is where these systems fail; the pooled number is designed to look clean.
  • Construct validity won't yield to more data. If the label encodes bias, a bigger model reproduces it more efficiently. This is the check people most often skip.
  • "Human in the loop" is a claim to test, not a safeguard to assume. Measure the override rate and its direction.
  • Pairs with disaggregate (its dimensions drive the group breakdown), validate-the-claim (vendor fairness studies), and value-frames (distributive-justice frame on who bears the error).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.