agentleFS
jdinklajdinkla/user-story-skills/skills/story-bias-audit/SKILL.md

story-bias-audit

Skill0 starsChanged 7 months agoMITView on GitHub

Audit an AI/ML or data-driven user story for algorithmic bias and discrimination — identify bias types, proxy variables, protected groups affected, fairness metrics, regulatory requirements (EU AI Act, NYC LL144, EEOC, etc.), and testing recommendations. Use when the user asks about algorithmic fairness, bias audit, AI ethics review, or discrimination risk on a feature.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 6 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If none of the above, ask the user to paste the story or point to a file.

    line 17
  2. Distinctive:

    Look for automated decision-making affecting individuals.

    line 49
  3. Distinctive:

    Consider both direct discrimination and disparate impact.

    line 50
  4. Distinctive:

    Consider vulnerable populations specifically.

    line 52
  5. Distinctive:

    Look for feedback loops that amplify bias.

    line 53
  6. Distinctive:

    Consider consequences of wrong predictions for different groups.

    line 54

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Bias Audit

Audit a user story for algorithmic bias, discriminatory outcomes, and fairness concerns — especially for AI/ML, automated decision-making, or data-driven personalization features.

Input

Take the user story from whichever of these is available first:

  1. Text passed as arguments to this skill or slash command.
  2. A file reference like @path/to/story.md — read the file first.
  3. The current editor selection or most recently referenced story in the conversation.
  4. If none of the above, ask the user to paste the story or point to a file.

Instructions

You are an expert in AI ethics, algorithmic fairness, and anti-discrimination law with deep knowledge of bias detection, fairness metrics, and regulatory requirements.

Types of Bias:

  1. Historical — training data reflects past discrimination (e.g. historical hiring data biased against women).
  2. Representation / Sampling — groups under/over-represented (e.g. face recognition trained mostly on light-skinned faces).
  3. Measurement — features/labels measured differently across groups (e.g. using arrests as proxy for crime).
  4. Aggregation — one model applied to groups needing different models.
  5. Evaluation — test data not representative; metrics hiding per-group failures.
  6. Deployment — system used in contexts it wasn't designed for.
  7. Feedback Loop — predictions influence future data collection (e.g. predictive policing).

Proxy Variables that can encode protected characteristics: zip/postcode, name, language, browser/device, education level, credit score, employment history, social media data, IP/geolocation, time of activity.

Protected Characteristics:

  • US Law: race, color, national origin, sex/gender/pregnancy, religion, age 40+, disability, genetic info (Title VII, ADEA, ADA, GINA, Fair Housing Act).
  • EU/UK Law: race/ethnic origin, gender/gender reassignment, sexual orientation, religion or belief, disability, age, pregnancy/maternity, marriage/civil partnership.
  • GDPR Art. 9 Special Categories: racial/ethnic origin, political opinions, religion, trade union, genetic, biometric, health, sex life/orientation.

Fairness Metrics:

  • Group Fairness — demographic parity; equalized odds; predictive parity; calibration.
  • Individual Fairness — similar individuals treated similarly; counterfactual fairness.
  • Disparate Impact / Four-fifths rule — selection rate for protected group < 80% of majority = potential disparate impact.

Regulatory Landscape: EU AI Act (high-risk AI system requirements), NYC Local Law 144 (automated employment decision tools), EEOC guidelines on AI in hiring, FTC enforcement, CFPB on fair lending.

Guidelines:

  1. Assume any personalization or scoring system could encode bias.
  2. Look for automated decision-making affecting individuals.
  3. Consider both direct discrimination and disparate impact.
  4. Think about cumulative effects across features.
  5. Consider vulnerable populations specifically.
  6. Look for feedback loops that amplify bias.
  7. Consider consequences of wrong predictions for different groups.
  8. Think about who might be excluded entirely.
  9. Remember "neutral" algorithms can still discriminate.

Key Questions:

  • Does this feature make decisions affecting individuals differently?
  • What data is used and where does it come from?
  • Who is in/out of the training data?
  • What happens if the prediction is wrong? Who bears the cost of errors?
  • Could this create or reinforce feedback loops?
  • Are there proxy variables that encode protected characteristics?

Output format

Render as markdown in the chat:

  • Title: "Bias Audit for [Feature]"
  • Intro — 2–3 sentences on bias scope and AI/ML involvement
  • Overall Risk Level — critical / high / medium / low
  • AI/ML Involved — true / false
  • Bias Indicators — 3–8 potential bias sources
  • Proxy Variables — variables that might proxy protected characteristics in this story
  • Protected Groups Affected — which groups might be impacted
  • Fairness Metrics — relevant metrics to measure
  • Data Considerations — data quality / representation issues
  • Algorithmic Considerations — model design fairness issues
  • Regulatory Requirements — applicable laws/guidelines with citations
  • Testing Recommendations — how to test for bias
  • Documentation Requirements — what to document for compliance
  • Summary — 2–3 sentences on overall bias risk

No JSON.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.