agentleFS
datahub-projectdatahub-project/datahub/.claude/skills/datahub-test-taxonomy/SKILL.md

datahub-test-taxonomy

Skill13k starsChanged 3 months agoApache-2.0View on GitHub

Maintain the smart-testing test taxonomy: tag new or changed tests and drop deleted ones, tag any tests that have no tags yet, regenerate all test tags (optionally for one framework), or regenerate the taxonomy from the rubric. Use after adding, modifying, or deleting smoke-test pytest, Playwright, or metadata-ingestion tests, or when the taxonomy needs a refresh.

Adopted by

Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data

Week of Sep 20: 0 new projectsWeek of Sep 27: 0 new projectsWeek of Oct 4: 0 new projectsWeek of Oct 11: 0 new projectsWeek of Oct 18: 0 new projectsWeek of Oct 25: 0 new projectsWeek of Nov 1: 0 new projectsWeek of Nov 8: 0 new projects
Sep 2026Oct 2026Nov 2026
New projects taking a version of this file each week, over the 8 weeks our latest run read: only 0 in all, so this is a hint, not a trend.

Who's talking about datahub-project/datahub

The biggest 5 of 64 mentions linking to the repository that our crawl found. Each line is their own headline.

See all 64

Talked about on GitHub, YouTube, Hacker News, Bluesky and Stack Overflow (64 times).

On GitHub · 50

45 more on GitHub

On YouTube · 11

6 more on YouTube

On Stack Overflow · 1

The file

Show its rules and text

We split this skill into the rules it gives an agent

13 rules. About 0 are common, 6 are distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Only pytest tags drive CI today.

    line 37
  2. Distinctive:

    Every file listing, pruning, validation, and enum generation goes through the script, so the model only does the judgement work: reading …

    line 54
  3. Distinctive:

    Only process frameworks that have files.

    line 91
  4. Very specific:

    Run tags.py prune --write for those frameworks, which removes deleted tests' entries and dead globs.

    line 93
  5. Distinctive:

    Read it and derive its tags per references/tagging.md.

    line 97
  6. Distinctive:

    If an existing glob already covers it with the right tags, leave the file alone.

    line 98
  7. Distinctive:

    If several files in one directory share the same tags and that directory has no entries yet, one glob is fine.

    line 101
  8. Very specific:

    Run tags.py validate and tags.py untagged.

    line 106
  9. Very specific:

    Run tags.py validate and tags.py untagged.

    line 125
  10. Very specific:

    Follow references/taxonomy.md to write .github/smart-testing/taxonomy.yaml and, if this repository has one, taxonomy.overlay.yaml.

    line 131
  11. Very specific:

    Run tags.py validate across all frameworks.

    line 137
  12. Very specific:

    Run ./gradlew :smoke-test:stackFreePytest, which covers the taxonomy/enum sync test.

    line 140
  13. Distinctive:

    Don't commit: the user reviews the diff first.

    line 145

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test taxonomy

Smart test selection matches a PR's tags against per-test tags. This skill keeps the tags and the taxonomy behind them current. It edits files only: it never runs the tests it tags, and it never commits.

Files

FileRole
.github/smart-testing/taxonomy.yamlThe common vocabulary for every framework.
.github/smart-testing/taxonomy.overlay.yamlOptional tags for repo-only features.
.github/smart-testing/taxonomy-rubric.mdThe criteria a good taxonomy meets.
smoke-test/smart_testing/taxonomy.pyEnum mirror of the taxonomy (generated block).
smoke-test/smart_testing/taxonomy_overlay.pyMirror of the overlay tags (generated).
references/tagging.mdHow to tag one test file.
references/taxonomy.mdHow to regenerate the taxonomy.
FrameworkTag filePaths relative toTest files
pytestsmoke-test/smart_testing/test_tags.yamlsmoke-test/tests/e2e/**/test_*.py, tests/e2e/**/*_test.py
playwright.github/smart-testing/playwright/test_tags.yamlrepo roote2e-test/ui/playwright/tests/**/*.spec.ts
ingestion.github/smart-testing/ingestion/test_tags.yamlrepo rootmetadata-ingestion/tests/**/{test_*,*_test}.py

Only pytest tags drive CI today. The others are maintained for when their selectors land. scripts/tags.py holds the real definitions (FRAMEWORKS).

Base and overlay

The base files (taxonomy.yaml, each test_tags.yaml) are shared by repositories that run the same tests, so they hold only tags and tests every one of them has. Each may have an optional overlay next to it (taxonomy.overlay.yaml, test_tags.overlay.yaml) for features and tests only this repository has. Every consumer merges base and overlay.

  • An entry goes in the overlay when its test file, or any tag it needs, is overlay-only. Otherwise it goes in the base.
  • A new tag goes in the overlay when the feature it names only exists here.
  • tags.py validate fails when a base entry uses an overlay tag.

Helper script

Every file listing, pruning, validation, and enum generation goes through the script, so the model only does the judgement work: reading tests and choosing tags.

uv run -q --with pyyaml==6.0.3 python ${CLAUDE_SKILL_DIR}/scripts/tags.py <command> [--framework F ...]
CommandDoes
changed [--base B]This branch's added, modified, and deleted test files, as JSON.
listAll test files, grouped by directory.
untaggedTest files that no entry matches.
prune [--write]Entries matching no file (deleted paths, dead globs). --write removes them, comments intact.
validateEvery tag is in the taxonomy under its dimension, and every entry matches a file. Exits 1 on failure.
enums [--write]Regenerate taxonomy.py's enum block and taxonomy_overlay.py from the taxonomy yaml files.

changed works with or without a PR, on any target branch. The base branch is --base, else the open PR's base, else the repository's default branch. It diffs the merge base against the working tree and adds untracked files. That covers committed and uncommitted edits, and excludes anything that landed on the base after the branch forked.

Modes

Parse $ARGUMENTS: the first word is the mode (default tag), then flags. --framework is repeatable and takes pytest, playwright or ingestion.

tag: new, changed and deleted tests (default)

With --untagged, the files to tag are every test that no entry matches, whether or not this branch touched it. This is how existing gaps get backfilled. --base doesn't apply.

  1. Get the files:

    • By default, run tags.py changed (plus --base/--framework if given).
    • With --untagged, run tags.py untagged (plus --framework) and treat every listed file as added.

    Only process frameworks that have files. If none do, say so and stop.

  2. Run tags.py prune --write for those frameworks, which removes deleted tests' entries and dead globs.
  3. Handle each added or modified file:

    • Read it and derive its tags per references/tagging.md.
    • If an existing glob already covers it with the right tags, leave the file alone.
    • Otherwise add an exact entry, or update the existing one, in the base or the overlay (see Base and overlay).
    • If several files in one directory share the same tags and that directory has no entries yet, one glob is fine.

    With many files (say more than 25), batch them to subagents the way retag does.

  4. Run tags.py validate and tags.py untagged. Fix what they report for the files you touched.

retag: regenerate all tags

  1. Frameworks are --framework, or all three if it's absent. For each, run tags.py list.
  2. Split the directories into batches of about 25 test files. Give each batch to a subagent (Agent tool, model: "opus"), running them in parallel. Each subagent gets:

    • the path of references/tagging.md, which it must read first
    • its file list
    • the framework's path convention

    It returns YAML entries only, with globs where a whole directory shares tags.

  3. Rebuild the tests: sections of the tag file and its overlay from the merged entries, split per Base and overlay, with one # ── <dir>/ ── section per top-level directory in sorted order. Keep the file headers and always_run exactly as they are.
  4. Run tags.py validate and tags.py untagged. Both must come back clean. Re-tag anything they flag.
  5. Report the number of entries and tags per dimension, before and after.

taxonomy: regenerate the taxonomy

  1. Follow references/taxonomy.md to write .github/smart-testing/taxonomy.yaml and, if this repository has one, taxonomy.overlay.yaml.
  2. Report which tags were added, removed and renamed, and which definitions changed, with rationale.
  3. Run tags.py enums --write.
  4. Run tags.py validate across all frameworks. For each entry that uses a removed or renamed tag, re-tag the files it covers with references/tagging.md. Then validate again.
  5. Run ./gradlew :smoke-test:stackFreePytest, which covers the taxonomy/enum sync test.

Finish

Summarize what changed, per file. List any untaggable tests and any taxonomy gaps you hit. Don't commit: the user reviews the diff first.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.