GitHub file history

Pulls the full git history of chosen files from many public GitHub repos, one TSV per file.

Each file becomes one TSV (gzipped by default) with a row per commit that touched it, plus a manifest. The repo list can come from a plain file, a GitHub search export, a random draw from a census list, or the agentleFS public catalog. It needs no GitHub token: it uses public git clone and raw.githubusercontent.com only, and neither counts against the REST API rate limit.

Example output (the smoke test below, 2026-10-11; base64 bodies cut short):

$ bash $SKILL/scripts/collect.sh $SKILL/examples/smoke.txt /tmp/gfh-smoke 4
listed 3  files collected 4  errors 1
$ zcat /tmp/gfh-smoke/vers/jshttp_mime-types__README.md.tsv.gz | head -2
80cdcb346c66fb39ab17be88a0a1031ffaee678f	2026-01-08T15:03:44-06:00	0	IyBtaW1lLXR5cGVzCgpbIVtOUE0g...
3aabd6896010e1d2c0c75d5cea36f90cd403b623	2025-11-25T11:12:26+01:00	0	IyBtaW1lLXR5cGVzCgpbIVtOUE0g...

Use it when:

• you want to see how a kind of file changes over time across projects: READMEs, package.json, Dockerfiles, CI workflows, agent instruction files, editor rules
• you need input for text-rule-trends

Requirements

• bash, git 2.22+ (partial clone and %(trailers)), gzip, base64, awk, timeout
• Python 3.8+, standard library only (for make_list.py)
• Network access to github.com, and to raw.githubusercontent.com for --probe. The catalog source also reads agentlefs.com.

Inputs

A list file with one wanted file per line, as owner/repo/PATTERN. A pattern is a path or a glob:

A glob is matched against every path that ever existed on the default branch. So it also finds files that were later deleted or renamed. Each match becomes its own file history.

You can write the list by hand, or build it with make_list.py from one of four sources.

Steps

Run the scripts by their path and write outputs to a folder of your own, never inside the skill folder. Below, SKILL is this skill's folder.

1. Make the list

# a plain list of repos (owner/repo or GitHub URLs, one per line)
python3 $SKILL/scripts/make_list.py file repos.txt --path README.md --path package.json > list.txt

# a GitHub search export: gh search repos ... --json fullName, a saved /search/repositories
# response, or a CSV with full_name / repo_name / nameWithOwner
python3 $SKILL/scripts/make_list.py search repos.json --path '.cursor/rules/*.mdc' > list.txt

# a random draw from any census list, keeping only files that exist today
python3 $SKILL/scripts/make_list.py census census.csv.gz --n 1000 --seed 1 \
    --path Dockerfile --probe --sampled drawn.txt > list.txt

# the agentleFS public catalog: its own agent files, or its repos crossed with --path
python3 $SKILL/scripts/make_list.py catalog --names CLAUDE.md AGENTS.md > list.txt
python3 $SKILL/scripts/make_list.py catalog --path package.json --limit 50 > list.txt

• --probe sends a HEAD request to https://raw.githubusercontent.com/<repo>/HEAD/<path> and keeps the pairs that answer 200. It works for literal paths only. Globs pass through. Non-404 errors are retried 4 times, then printed as ERR <code> <repo> <path>. --jobs (default 16) sets how many probes run at once.
• census reads a CSV with a repo_name column (gzipped or not; the header may come after junk lines) or one owner/repo per line. Lines that are not a valid owner/repo are dropped and counted on stderr. It draws with random.Random(seed).sample. --sampled keeps the drawn repos, so you have the denominator on record.
• No census ships with this skill. Make one from GH Archive's free hourly event files:

  python3 $SKILL/scripts/gharchive_census.py --start 2026-10-01T00 --hours 720 --every 24 > census.txt

  That reads one hour a day for 30 days and keeps every repo with a push. One hour is about 18 MB and gave 39,397 repos (2026-10-01 12:00 UTC, tested 2026-10-11). Any list of public repos works instead, in either format above. Your list decides what "random" means, so write down how you made it.
• catalog reads https://agentlefs.com/context/sitemap.xml. It lists agent instruction files in public repos with 1,000+ stars. Without --path, it lists the catalog's files (root only unless --any-depth, filtered by --names). With --path, it is a source of popular repos.

2. Collect histories

bash $SKILL/scripts/collect.sh list.txt data/mysample 8

collect.sh groups the list by repo, so each repo is cloned once. For each repo, history.sh makes a bare, blob-less clone (git clone --bare --filter=blob:none). It expands any globs against the history, lists the commits that touched each file, and fetches only those blobs, in batches of 500 per git fetch rather than one round trip each. If a blob still cannot be read, that file is not written and ERR blob is logged, so a failed fetch never looks like a deletion, and a re-run tries the file again. It is safe to re-run. Finished files are skipped, and literal paths already on disk skip the clone.

One repo on its own: bash $SKILL/scripts/history.sh owner/repo OUTDIR README.md 'docs/*.md'.

Settings, as environment variables:

• CLONE_TIMEOUT (seconds, default 300) limits each clone. Huge repos can need more.
• HISTORY_FORMAT=tsv writes plain .tsv files instead of .tsv.gz, for stores that hold text only.

Outputs

data/<sample>/
  manifest.tsv            key <TAB> owner/repo <TAB> path   (one line per file collected)
  err.log                 always written; "ERR clone <repo>", "ERR nohistory <repo> <pattern>", or
                          "ERR blob <repo> <path> <sha>" when a version could not be fetched
  vers/<key>.tsv.gz       one per file; key = <owner>_<repo>__<path with / as _>

Each vers row is one commit that touched the file, in git log order, with four tab-separated fields. That order is newest first by commit graph, not strictly by author date: author dates can go backwards (rebases, cherry-picks, wrong clocks, mixed UTC offsets). Sort by the parsed date if you need date order; text-rule-trends does.

text-rule-trends reads this format, .tsv.gz or .tsv.

Examples

README and package.json in five repos (examples/readme-package/)

repos.txt names five public repos. list.txt was made from it:

python3 $SKILL/scripts/make_list.py file $SKILL/examples/readme-package/repos.txt \
    --path package.json --path '[Rr][Ee][Aa][Dd][Mm][Ee]*' > list.txt
bash $SKILL/scripts/collect.sh list.txt /tmp/readme-package 5
# listed 10  files collected 12  errors 1

On 2026-10-11 this took 3.4 seconds. The README glob found 7 files in 5 repos, including renames: expressjs/cors had README.markdown and then README.md, and pallets/itsdangerous had README, then README.rst, then README.md. The one error is pallets/itsdangerous, a Python project with no package.json. Row counts grow as the repos get new commits.

Smoke test (examples/smoke.txt)

Two README histories, one glob, and one repo that does not exist. A few seconds:

bash $SKILL/scripts/collect.sh $SKILL/examples/smoke.txt /tmp/gfh-smoke 4
# listed 3  files collected 4  errors 1

The glob jshttp/mime-types/*.md found HISTORY.md, README.md and SOURCES.md (2026-10-11). The error is ERR clone nobody-here/not-a-repo-xyz. Every vers row should have 4 fields, and each body should decode with base64 -d.

Worked example. These skills were first written for the agentleFS article "CLAUDE.md best practices" (https://agentlefs.com/blog/claude-md-best-practices), which studied 5,786 CLAUDE.md and AGENTS.md files. Its data, settings and step-by-step recipe live outside the skills, in the article-claude-md-best-practices folder (RECIPE.md and data/).

Limits and caveats

• Your repo list decides what your sample means. A random draw from a census is only as random as the census. The catalog covers popular repos (1,000+ stars, licences that allow redistribution), which are not typical repos. Write down where the list came from.
• The catalog sitemap has a cap of 20,000 URLs, most stars first. It was at the cap on 2026-10-11. The script prints a note when the cap is reached. The catalog changes every week.
• --probe only sees files that exist today on the default branch. Deleted or renamed files are missed. Globs are not probed, so they find deleted files too, but they cost a clone per repo.
• History follows one path, with git's default history simplification. A file that only ever lived on a side branch that was merged has no history that way; for those (and only those) history.sh retries with --full-history, so they are collected instead of logged as nohistory. Histories that the default finds are never changed. A rename looks like a deletion plus a new file. A glob that matches both names gives you both files. Dates are author dates, which people can set to anything.
• Keys can collide in rare cases, because / and _ both become _ (a/b_c and a_b/c). Paths with spaces are not supported.
• Rate limits. Clones and raw HEAD requests need no token, but GitHub throttles heavy use. Keep --jobs and the collect parallelism modest (8 to 24). api.github.com is not used. If you add API calls of your own, read the token from GITHUB_TOKEN and never print it.
• Speed. Blobs are fetched in batches, so the README example (12 files, about 1,200 versions) took 3.4 seconds on 2026-10-11. Blob-less clones still download every commit and tree, so huge monorepos can take minutes each.
• Licences. Collecting public files to study them is one thing. Redistributing their text is another. Only republish file contents (including these TSVs) for repos whose licence allows it.

Credits

• GH Archive (https://www.gharchive.org), by Ilya Grigorik: its free hourly files of public GitHub events are what gharchive_census.py reads to build a census of active repos. Nothing from it ships with this skill. Its front page states no licence for the data (checked 2026-10-11); the events are GitHub's public timeline, under GitHub's terms.
• git partial clone (git clone --filter=blob:none, git 2.22+): blob-less clones, with only the needed blobs fetched in batches, are what make many repos cheap. git (GPL-2.0) is called as an installed program, not bundled.
• File contents collected belong to their repos' authors, under each repo's licence (see Limits).