[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8},"actionData","errors","root",{"_77":78},"routes/registry-package",{"_9":10,"_11":12,"_13":14,"_15":16,"_17":18,"_19":20},"pkg",{"_26":27,"_28":29,"_30":31,"_32":33,"_34":35,"_22":23,"_36":37,"_38":39,"_40":41,"_42":43,"_24":25,"_44":45},"comments",[],"versions",[21],"selectedFile","examples/readme-package/repos.txt","canonical","https://agentlefs.com/registry/github-file-history/examples/readme-package/repos.txt","historical",false,{"_22":23,"_24":25},"version",1,"publishedAt","2026-10-12T22:40:50.300Z","slug","github-file-history","kind","skill","title","GitHub file history","summary","Collect every version of any file across public GitHub repos, from a list, a search or a GH Archive census.","publisher","agentleFS","body","# GitHub file history\n\nPulls the full git history of chosen files from many public GitHub repos, one TSV per file.\n\nEach file becomes one TSV (gzipped by default) with a row per commit that touched it, plus a\nmanifest. The repo list can come from a plain file, a GitHub search export, a random draw from a\ncensus list, or the agentleFS public catalog. It needs no GitHub token: it uses public\n`git clone` and `raw.githubusercontent.com` only, and neither counts against the REST API rate\nlimit.\n\nExample output (the smoke test below, 2026-10-11; base64 bodies cut short):\n\n```\n$ bash $SKILL/scripts/collect.sh $SKILL/examples/smoke.txt /tmp/gfh-smoke 4\nlisted 3 files collected 4 errors 1\n$ zcat /tmp/gfh-smoke/vers/jshttp_mime-types__README.md.tsv.gz | head -2\n80cdcb346c66fb39ab17be88a0a1031ffaee678f\t2026-01-08T15:03:44-06:00\t0\tIyBtaW1lLXR5cGVzCgpbIVtOUE0g...\n3aabd6896010e1d2c0c75d5cea36f90cd403b623\t2025-11-25T11:12:26+01:00\t0\tIyBtaW1lLXR5cGVzCgpbIVtOUE0g...\n```\n\nUse it when:\n- you want to see how a kind of file changes over time across projects: READMEs, package.json,\n Dockerfiles, CI workflows, agent instruction files, editor rules\n- you need input for `text-rule-trends`\n\n## Requirements\n\n- `bash`, `git` 2.22+ (partial clone and `%(trailers)`), `gzip`, `base64`, `awk`, `timeout`\n- Python 3.8+, standard library only (for `make_list.py`)\n- Network access to `github.com`, and to `raw.githubusercontent.com` for `--probe`.\n The `catalog` source also reads `agentlefs.com`.\n\n## Inputs\n\nA list file with one wanted file per line, as `owner/repo/PATTERN`. A pattern is a path or a glob:\n\n| pattern | matches |\n|---|---|\n| `README.md` | that path only |\n| `*.md` | any `.md` file at the root |\n| `.cursor/rules/*.mdc` | `.mdc` files in that folder (not below it) |\n| `docs/**/*.md` | `.md` files anywhere under `docs/` |\n| `[Rr][Ee][Aa][Dd][Mm][Ee]*` | README, readme.md, README.rst, ... |\n\nA glob is matched against every path that ever existed on the default branch. So it also finds\nfiles that were later deleted or renamed. Each match becomes its own file history.\n\nYou can write the list by hand, or build it with `make_list.py` from one of four sources.\n\n## Steps\n\nRun the scripts by their path and write outputs to a folder of your own, never inside the skill\nfolder. Below, `SKILL` is this skill's folder.\n\n### 1. Make the list\n\n```bash\n# a plain list of repos (owner/repo or GitHub URLs, one per line)\npython3 $SKILL/scripts/make_list.py file repos.txt --path README.md --path package.json > list.txt\n\n# a GitHub search export: gh search repos ... --json fullName, a saved /search/repositories\n# response, or a CSV with full_name / repo_name / nameWithOwner\npython3 $SKILL/scripts/make_list.py search repos.json --path '.cursor/rules/*.mdc' > list.txt\n\n# a random draw from any census list, keeping only files that exist today\npython3 $SKILL/scripts/make_list.py census census.csv.gz --n 1000 --seed 1 \\\n --path Dockerfile --probe --sampled drawn.txt > list.txt\n\n# the agentleFS public catalog: its own agent files, or its repos crossed with --path\npython3 $SKILL/scripts/make_list.py catalog --names CLAUDE.md AGENTS.md > list.txt\npython3 $SKILL/scripts/make_list.py catalog --path package.json --limit 50 > list.txt\n```\n\n- `--probe` sends a HEAD request to `https://raw.githubusercontent.com//HEAD/` and\n keeps the pairs that answer 200. It works for literal paths only. Globs pass through.\n Non-404 errors are retried 4 times, then printed as `ERR

  `.\n  `--jobs` (default 16) sets how many probes run at once.\n- `census` reads a CSV with a `repo_name` column (gzipped or not; the header may come after junk\n  lines) or one `owner/repo` per line. Lines that are not a valid `owner/repo` are dropped and\n  counted on stderr. It draws with `random.Random(seed).sample`. `--sampled` keeps the drawn\n  repos, so you have the denominator on record.\n- **No census ships with this skill.** Make one from GH Archive's free hourly event files:\n  ```bash\n  python3 $SKILL/scripts/gharchive_census.py --start 2026-10-01T00 --hours 720 --every 24 > census.txt\n  ```\n  That reads one hour a day for 30 days and keeps every repo with a push. One hour is about\n  18 MB and gave 39,397 repos (2026-10-01 12:00 UTC, tested 2026-10-11). Any list of public\n  repos works instead, in either format above. Your list decides what \"random\" means, so write\n  down how you made it.\n- `catalog` reads `https://agentlefs.com/context/sitemap.xml`. It lists agent instruction files\n  in public repos with 1,000+ stars. Without `--path`, it lists the catalog's files (root only\n  unless `--any-depth`, filtered by `--names`). With `--path`, it is a source of popular repos.\n\n### 2. Collect histories\n\n```bash\nbash $SKILL/scripts/collect.sh list.txt data/mysample 8\n```\n\n`collect.sh` groups the list by repo, so each repo is cloned once. For each repo, `history.sh`\nmakes a bare, blob-less clone (`git clone --bare --filter=blob:none`). It expands any globs\nagainst the history, lists the commits that touched each file, and fetches only those blobs, in\nbatches of 500 per `git fetch` rather than one round trip each. If a blob still cannot be read,\nthat file is not written and `ERR blob` is logged, so a failed fetch never looks like a deletion,\nand a re-run tries the file again.\nIt is safe to re-run. Finished files are skipped, and literal paths already on disk skip the clone.\n\nOne repo on its own: `bash $SKILL/scripts/history.sh owner/repo OUTDIR README.md 'docs/*.md'`.\n\nSettings, as environment variables:\n- `CLONE_TIMEOUT` (seconds, default 300) limits each clone. Huge repos can need more.\n- `HISTORY_FORMAT=tsv` writes plain `.tsv` files instead of `.tsv.gz`, for stores that hold text only.\n\n## Outputs\n\n```\ndata//\n  manifest.tsv            key  owner/repo  path   (one line per file collected)\n  err.log                 always written; \"ERR clone \", \"ERR nohistory  \", or\n                          \"ERR blob   \" when a version could not be fetched\n  vers/.tsv.gz       one per file; key = ___\n```\n\nEach `vers` row is one commit that touched the file, in `git log` order, with four tab-separated\nfields. That order is newest first by commit graph, not strictly by author date: author dates can\ngo backwards (rebases, cherry-picks, wrong clocks, mixed UTC offsets). Sort by the parsed date\nif you need date order; `text-rule-trends` does.\n\n| field | meaning |\n|---|---|\n| sha | commit id |\n| author_date | ISO 8601 author date with offset, e.g. `2026-03-31T14:14:16-07:00` |\n| claude_coauthor | 1 if a `Co-authored-by` trailer mentions \"claude\", else 0 |\n| body | the file's content after that commit, base64 (no newlines). Empty if the commit deleted it |\n\n`text-rule-trends` reads this format, `.tsv.gz` or `.tsv`.\n\n## Examples\n\n### README and package.json in five repos (`examples/readme-package/`)\n\n`repos.txt` names five public repos. `list.txt` was made from it:\n\n```bash\npython3 $SKILL/scripts/make_list.py file $SKILL/examples/readme-package/repos.txt \\\n    --path package.json --path '[Rr][Ee][Aa][Dd][Mm][Ee]*' > list.txt\nbash $SKILL/scripts/collect.sh list.txt /tmp/readme-package 5\n# listed 10  files collected 12  errors 1\n```\n\nOn 2026-10-11 this took 3.4 seconds. The README glob found 7 files in 5 repos, including\nrenames: `expressjs/cors` had `README.markdown` and then `README.md`, and `pallets/itsdangerous`\nhad `README`, then `README.rst`, then `README.md`. The one error is `pallets/itsdangerous`, a\nPython project with no `package.json`. Row counts grow as the repos get new commits.\n\n### Smoke test (`examples/smoke.txt`)\n\nTwo README histories, one glob, and one repo that does not exist. A few seconds:\n\n```bash\nbash $SKILL/scripts/collect.sh $SKILL/examples/smoke.txt /tmp/gfh-smoke 4\n# listed 3  files collected 4  errors 1\n```\n\nThe glob `jshttp/mime-types/*.md` found `HISTORY.md`, `README.md` and `SOURCES.md` (2026-10-11).\nThe error is `ERR clone nobody-here/not-a-repo-xyz`. Every `vers` row should have 4 fields, and\neach body should decode with `base64 -d`.\n\n**Worked example.** These skills were first written for the agentleFS article \"CLAUDE.md best\npractices\" (https://agentlefs.com/blog/claude-md-best-practices), which studied 5,786 CLAUDE.md and\nAGENTS.md files. Its data, settings and step-by-step recipe live outside the skills, in the\n`article-claude-md-best-practices` folder (`RECIPE.md` and `data/`).\n\n## Limits and caveats\n\n- **Your repo list decides what your sample means.** A random draw from a census is only as\n  random as the census. The catalog covers popular repos (1,000+ stars, licences that allow\n  redistribution), which are not typical repos. Write down where the list came from.\n- **The catalog sitemap has a cap of 20,000 URLs**, most stars first. It was at the cap on\n  2026-10-11. The script prints a note when the cap is reached. The catalog changes every week.\n- **`--probe` only sees files that exist today on the default branch.** Deleted or renamed files\n  are missed. Globs are not probed, so they find deleted files too, but they cost a clone per repo.\n- **History follows one path, with git's default history simplification.** A file that only ever lived on a side branch that was merged has no history\n  that way; for those (and only those) `history.sh` retries with `--full-history`, so they are\n  collected instead of logged as nohistory. Histories that the default finds are never changed.\n  A rename looks like a deletion plus a new file. A glob that matches\n  both names gives you both files. Dates are author dates, which people can set to anything.\n- **Keys can collide** in rare cases, because `/` and `_` both become `_` (`a/b_c` and `a_b/c`).\n  Paths with spaces are not supported.\n- **Rate limits.** Clones and raw HEAD requests need no token, but GitHub throttles heavy use.\n  Keep `--jobs` and the collect parallelism modest (8 to 24). `api.github.com` is not used. If you\n  add API calls of your own, read the token from `GITHUB_TOKEN` and never print it.\n- **Speed.** Blobs are fetched in batches, so the README example (12 files, about 1,200 versions)\n  took 3.4 seconds on 2026-10-11. Blob-less clones still download every commit and tree, so huge\n  monorepos can take minutes each.\n- **Licences.** Collecting public files to study them is one thing. Redistributing their text is\n  another. Only republish file contents (including these TSVs) for repos whose licence allows it.\n\n## Credits\n\n- **GH Archive** (https://www.gharchive.org), by Ilya Grigorik: its free hourly files of public\n  GitHub events are what `gharchive_census.py` reads to build a census of active repos. Nothing\n  from it ships with this skill. Its front page states no licence for the data (checked\n  2026-10-11); the events are GitHub's public timeline, under GitHub's terms.\n- **git partial clone** (`git clone --filter=blob:none`, git 2.22+): blob-less clones, with only\n  the needed blobs fetched in batches, are what make many repos cheap. git (GPL-2.0) is called as\n  an installed program, not bundled.\n- File contents collected belong to their repos' authors, under each repo's licence (see Limits).\n\n","fields",{},"files",[50,51,52,53,54,55,56,57,58],"approvedBy","frdchang@gmail.com","standing",{"_46":47,"_48":47,"_49":-5},"chains",0,"ratings","meanByChain",{"_59":75,"_61":76},{"_59":74,"_61":37},{"_59":72,"_61":73},{"_59":16,"_61":71},{"_59":69,"_61":70},{"_59":67,"_61":68},{"_59":65,"_61":66},{"_59":63,"_61":64},{"_59":60,"_61":62},"path","scripts/make_list.py","content","#!/usr/bin/env python3\n\"\"\"Build the list collect.sh reads: one owner/repo/PATTERN per line.\n\nPick one source of repos, then say which paths you want with --path (repeatable; a literal\npath or a glob such as '.cursor/rules/*.mdc'). Every repo is crossed with every --path.\n\nSources:\n  file FILE              one owner/repo per line (a GitHub URL or an owner/repo/path line is\n                         cut down to owner/repo). Lines starting with # are skipped.\n  search FILE            a GitHub search export: the JSON of `gh search repos ... --json fullName`,\n                         a saved REST response from /search/repositories ({\"items\": [...]}), a\n                         JSON list of names, or a CSV with a full_name, fullName, nameWithOwner,\n                         repo_name or repo column.\n  census FILE --n N      a random draw of N repos from any census list (CSV with a repo_name\n                         column, gzipped or not, or one owner/repo per line), with\n                         random.Random(--seed).sample. --sampled writes the drawn repos, so the\n                         denominator is on record.\n  catalog                the agentleFS public catalog sitemap (https://agentlefs.com/context/sitemap.xml).\n                         It lists agent instruction files (AGENTS.md, CLAUDE.md, SKILL.md, llms.txt)\n                         in public repos with 1,000+ stars whose licence allows redistribution.\n                         Without --path it lists the catalog's own files (root files only, unless\n                         --any-depth; --names filters by file name). With --path it is just a\n                         source of repos.\n\nOptions:\n  --path PATTERN         a path or glob to collect in every repo (repeatable)\n  --probe                keep only (repo, path) pairs whose file exists today on the default\n                         branch, by a HEAD request to raw.githubusercontent.com (no token, no API\n                         rate limit). Literal paths only; globs are kept as they are.\n  --jobs 16              probes at once. Non-404 errors are retried 4 times, then reported on\n                         stderr as \"ERR   \".\n  --limit N              stop after N output lines\n\nExamples:\n  python3 make_list.py file repos.txt --path README.md --path package.json > list.txt\n  python3 make_list.py census census.csv.gz --n 30000 --seed 1 --path CLAUDE.md --path AGENTS.md \\\\\n      --probe --sampled drawn.txt > random.txt\n  python3 make_list.py catalog --names CLAUDE.md AGENTS.md > popular.txt\n  python3 make_list.py search repos.json --path '.cursor/rules/*.mdc' > list.txt\n\"\"\"\nimport argparse\nimport concurrent.futures as cf\nimport csv\nimport gzip\nimport io\nimport json\nimport random\nimport re\nimport sys\nimport time\nimport urllib.error\nimport urllib.request\n\nSITEMAP = \"https://agentlefs.com/context/sitemap.xml\"\nUA = {\"User-Agent\": \"github-file-history/1.0\"}\nNAME = re.compile(r\"^[A-Za-z0-9-]+/[A-Za-z0-9._-]+$\")\nNAME_COLS = (\"full_name\", \"fullName\", \"nameWithOwner\", \"repo_name\", \"repo\")\n\n\ndef opener(path):\n    return gzip.open(path, \"rt\", encoding=\"utf-8\", errors=\"replace\") if path.endswith(\".gz\") else open(path, encoding=\"utf-8\", errors=\"replace\")\n\n\ndef as_repo(s):\n    s = s.strip()\n    s = re.sub(r\"^(https?://)?(www\\.)?github\\.com/\", \"\", s)\n    s = re.sub(r\"\\.git$\", \"\", s)\n    parts = [p for p in s.split(\"/\") if p]\n    name = \"/\".join(parts[:2]) if len(parts) >= 2 and not s.startswith(\"#\") else None\n    return name if name and NAME.match(name) else None\n\n\ndef from_file(path):\n    with opener(path) as f:\n        return [r for r in (as_repo(l) for l in f) if r]\n\n\ndef from_search(path):\n    with opener(path) as f:\n        text = f.read()\n    try:\n        data = json.loads(text)\n    except ValueError:\n        start = min((text.find(c) for c in NAME_COLS if c in text), default=0)\n        start = text.rfind(\"\\n\", 0, start) + 1\n        rows = csv.DictReader(io.StringIO(text[start:]))\n        col = next(c for c in NAME_COLS if c in (rows.fieldnames or []))\n        return [r for r in (as_repo(row[col] or \"\") for row in rows) if r]\n    if isinstance(data, dict):\n        data = data.get(\"items\", data.get(\"repositories\", []))\n    out = []\n    for item in data:\n        name = item if isinstance(item, str) else next((item[c] for c in NAME_COLS if c in item), None)\n        if name is None and isinstance(item, dict) and \"owner\" in item and \"name\" in item:\n            owner = item[\"owner\"][\"login\"] if isinstance(item[\"owner\"], dict) else item[\"owner\"]\n            name = f\"{owner}/{item['name']}\"\n        r = as_repo(name or \"\")\n        if r:\n            out.append(r)\n    return out\n\n\ndef read_census(path):\n    \"\"\"Repo names from a census: a CSV with a repo_name column (the header may follow junk lines),\n    or one owner/repo per line. Names that are not owner/repo are dropped and counted.\"\"\"\n    with opener(path) as f:\n        lines = f.read().splitlines()\n    header = next((i for i, l in enumerate(lines[:100]) if \"repo_name\" in next(csv.reader([l]), [])), None)\n    if header is not None:\n        names = [(r.get(\"repo_name\") or \"\").strip() for r in csv.DictReader(io.StringIO(\"\\n\".join(lines[header:])))]\n    else:\n        names = [l.strip() for l in lines if l.strip() and not l.startswith(\"#\")]\n    good = [n for n in names if NAME.match(n)]\n    if len(good) < len(names):\n        bad = [n for n in names if not NAME.match(n)]\n        print(f\"census: dropped {len(bad)} lines that are not owner/repo, e.g. {bad[0][:60]!r}\", file=sys.stderr)\n    if not good:\n        sys.exit(f\"{path}: no owner/repo names found. Use a CSV with a repo_name column, or one owner/repo per line.\")\n    return good\n\n\ndef from_catalog(src, names, any_depth):\n    if src.startswith(\"http\"):\n        xml = urllib.request.urlopen(urllib.request.Request(src, headers=UA), timeout=60).read().decode(\"utf-8\")\n    else:\n        xml = open(src, encoding=\"utf-8\").read()\n    locs = re.findall(r\"https://agentlefs\\.com/context/([^<]+)\", xml)\n    total = xml.count(\"\")\n    out, seen = [], set()\n    for p in locs:\n        parts = p.split(\"/\")\n        if len(parts) < 3 or (not any_depth and len(parts) != 3):\n            continue\n        if names and parts[-1] not in names:\n            continue\n        if p not in seen:\n            seen.add(p)\n            out.append(p)\n    print(f\"sitemap urls {total}, files kept {len(out)}\", file=sys.stderr)\n    if total >= 20000:\n        print(\"note: the sitemap is at its 20,000-URL cap, so the lowest-starred files may be missing\", file=sys.stderr)\n    return out\n\n\ndef probe(repo, path, tries=4):\n    url = f\"https://raw.githubusercontent.com/{repo}/HEAD/{path}\"\n    code = None\n    for k in range(tries):\n        try:\n            with urllib.request.urlopen(urllib.request.Request(url, method=\"HEAD\", headers=UA), timeout=15) as r:\n                return r.status\n        except urllib.error.HTTPError as e:\n            if e.code == 404:\n                return 404\n            code = e.code\n        except Exception as e:  # timeouts, resets\n            code = type(e).__name__\n        time.sleep(2 ** k)\n    return code\n\n\ndef is_glob(p):\n    return any(c in p for c in \"*?[\")\n\n\ndef main():\n    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n    ap.add_argument(\"source\", choices=[\"file\", \"search\", \"census\", \"catalog\"])\n    ap.add_argument(\"input\", nargs=\"?\", help=\"the file to read (catalog: optional local sitemap)\")\n    ap.add_argument(\"--path\", action=\"append\", default=[], help=\"path or glob to collect in every repo\")\n    ap.add_argument(\"--n\", type=int, help=\"census: repos to draw\")\n    ap.add_argument(\"--seed\", type=int, default=1)\n    ap.add_argument(\"--sampled\", help=\"census: also write the drawn repos here\")\n    ap.add_argument(\"--names\", nargs=\"+\", help=\"catalog: keep these file names\")\n    ap.add_argument(\"--any-depth\", action=\"store_true\", help=\"catalog: also files below the repo root\")\n    ap.add_argument(\"--probe\", action=\"store_true\")\n    ap.add_argument(\"--jobs\", type=int, default=16)\n    ap.add_argument(\"--limit\", type=int, default=0)\n    a = ap.parse_args()\n\n    if a.source == \"catalog\":\n        files = from_catalog(a.input or SITEMAP, a.names, a.any_depth)\n        if not a.path:\n            pairs = [(\"/\".join(p.split(\"/\")[:2]), \"/\".join(p.split(\"/\")[2:])) for p in files]\n            repos = None\n        else:\n            repos = list(dict.fromkeys(\"/\".join(p.split(\"/\")[:2]) for p in files))\n    else:\n        if not a.input:\n            sys.exit(f\"{a.source} needs an input file\")\n        if a.source == \"file\":\n            repos = from_file(a.input)\n        elif a.source == \"search\":\n            repos = from_search(a.input)\n        else:\n            if not a.n:\n                sys.exit(\"census needs --n\")\n            names = read_census(a.input)\n            repos = random.Random(a.seed).sample(names, min(a.n, len(names)))\n            print(f\"census {len(names)} repos, drew {len(repos)} (seed {a.seed})\", file=sys.stderr)\n            if a.sampled:\n                open(a.sampled, \"w\").write(\"\".join(r + \"\\n\" for r in repos))\n        repos = list(dict.fromkeys(repos)) if a.source != \"census\" else repos\n    if repos is not None:\n        if not a.path:\n            sys.exit(\"say which files to collect with --path\")\n        pairs = [(r, p) for r in repos for p in a.path]\n\n    if a.probe:\n        todo = [(r, p) for r, p in pairs if not is_glob(p)]\n        ok, errs = set(), 0\n        with cf.ThreadPoolExecutor(a.jobs) as ex:\n            for (r, p), code in zip(todo, ex.map(lambda j: probe(*j), todo)):\n                if code == 200:\n                    ok.add((r, p))\n                elif code != 404:\n                    print(f\"ERR {code} {r} {p}\", file=sys.stderr)\n                    errs += 1\n        print(f\"probed {len(todo)}, found {len(ok)}, errors {errs}\", file=sys.stderr)\n        pairs = [(r, p) for r, p in pairs if is_glob(p) or (r, p) in ok]\n    if a.limit:\n        pairs = pairs[: a.limit]\n    sys.stdout.write(\"\".join(f\"{r}/{p}\\n\" for r, p in pairs))\n\n\nif __name__ == \"__main__\":\n    main()\n","scripts/history.sh","#!/usr/bin/env bash\n# history.sh OWNER/REPO OUTDIR PATTERN [PATTERN ...]\n#\n# One bare, blob-less clone of a public repo, then the full history of every file that matches\n# a PATTERN. A pattern is a literal path (README.md, docs/intro.md) or a glob:\n#   *   any characters except /      ?  one character except /\n#   **  any characters including /   [abc]  one of a set\n# A glob is matched against every path that ever existed in the history, so files that were\n# deleted or renamed away are found too.\n#\n# For each file it writes OUTDIR/vers/___.tsv.gz, one row per\n# commit that touched the file, in git log order (newest first):\n#   sha  author_date (ISO 8601)  claude_coauthor (0/1)  base64(file body)\n# A commit that deleted the file has an empty body. Set HISTORY_FORMAT=tsv for plain .tsv files.\n#\n# It prints one manifest line per file written or already present: key  owner/repo  path.\n# Errors go to stderr: \"ERR clone \", \"ERR nohistory  \", or \"ERR blob \n#  \" when a version's content could not be fetched (that file is not written).\n# Each file's rows come from its own single-path git log, so a file's history is the same\n# whether you asked for it by name or through a glob.\nset -uo pipefail\nrepo=${1:?owner/repo}; out=${2:?output dir}; shift 2\n[ $# -gt 0 ] || { echo \"usage: history.sh OWNER/REPO OUTDIR PATTERN...\" >&2; exit 2; }\nmkdir -p \"$out/vers\"\nout=$(cd \"$out\" && pwd)\next=tsv.gz; [ \"${HISTORY_FORMAT:-gz}\" = tsv ] && ext=tsv\nkeyof() { local p=$1; echo \"${repo//\\//_}__${p//\\//_}\"; }\nisglob() { case \"$1\" in *[\\*\\?\\[]*) return 0;; *) return 1;; esac; }\n\n# Literal paths that are already collected need no clone.\nneed=0\nfor pat in \"$@\"; do\n  if isglob \"$pat\"; then need=1\n  elif [ -s \"$out/vers/$(keyof \"$pat\").$ext\" ]; then printf '%s\\t%s\\t%s\\n' \"$(keyof \"$pat\")\" \"$repo\" \"$pat\"\n  else need=1; fi\ndone\n[ $need = 1 ] || exit 0\n\nd=$(mktemp -d)\ntrap 'rm -rf \"$d\" 2>/dev/null' EXIT\nif ! timeout \"${CLONE_TIMEOUT:-300}\" git clone -q --bare --filter=blob:none \"https://github.com/$repo.git\" \"$d/g\" 2>/dev/null; then\n  echo \"ERR clone $repo\" >&2\n  exit 0\nfi\ncd \"$d/g\" || exit 0\n\n# glob -> anchored ERE\nglob_re() {\n  local g=$1 r=\"\" c i\n  for ((i = 0; i < ${#g}; i++)); do\n    c=${g:i:1}\n    if [ \"$c\" = \"*\" ] && [ \"${g:i+1:1}\" = \"*\" ]; then\n      if [ \"${g:i+2:1}\" = \"/\" ]; then r+=\"(.*/)?\"; i=$((i + 2)); else r+=\".*\"; i=$((i + 1)); fi\n    elif [ \"$c\" = \"*\" ]; then r+=\"[^/]*\"\n    elif [ \"$c\" = \"?\" ]; then r+=\"[^/]\"\n    elif [ \"$c\" = \"[\" ]; then r+=\"[\"\n    elif [ \"$c\" = \"]\" ]; then r+=\"]\"\n    elif [[ \"$c\" == [.^\\$+\\(\\)\\{\\}\\|\\\\] ]]; then r+=\"\\\\$c\"\n    else r+=\"$c\"; fi\n  done\n  echo \"^$r\\$\"\n}\n\npaths=()\nfor pat in \"$@\"; do\n  if isglob \"$pat\"; then\n    re=$(glob_re \"$pat\")\n    mapfile -t found < <(git log --format= --name-only --no-renames 2>/dev/null | grep -E \"$re\" | sort -u)\n    [ ${#found[@]} -gt 0 ] || { echo \"ERR nohistory $repo $pat\" >&2; continue; }\n    paths+=(\"${found[@]}\")\n  else\n    paths+=(\"$pat\")\n  fi\ndone\n\nzero=0000000000000000000000000000000000000000\n[ ${#paths[@]} -gt 0 ] || exit 0          # every pattern was a glob that matched nothing (already logged)\nprintf '%s\\n' \"${paths[@]}\" | awk '!seen[$0]++' > \"$d/paths.txt\"\n\n# Pass 1: the commits that touched each file, and the blob each one left.\nn=0\n: > \"$d/todo.tsv\"\nwhile IFS= read -r path <&3; do\n  key=$(keyof \"$path\")\n  dest=\"$out/vers/$key.$ext\"\n  if [ -s \"$dest\" ]; then printf '%s\\t%s\\t%s\\n' \"$key\" \"$repo\" \"$path\"; continue; fi\n  n=$((n + 1))\n  # One line per (commit, blob of PATH after the commit). --raw gives the new blob id.\n  # Default history simplification first (git log's usual view of one path). Only if that finds\n  # nothing, e.g. a file that lived only on a merged side branch, retry with --full-history.\n  for mode in \"\" \"--full-history\"; do\n    git log $mode --format='C%x09%H%x09%aI%x09%(trailers:key=Co-authored-by,valueonly,separator=;)' \\\n        --raw --no-abbrev --no-renames -- \"$path\" 2>/dev/null \\\n      | awk -F'\\t' -v f=\"$path\" '\n          /^C\\t/ { sha=$2; date=$3; co=(tolower($4) ~ /claude/) ? 1 : 0; next }\n          /^:/   { split($1, a, \" \"); if ($2 == f) print sha \"\\t\" date \"\\t\" co \"\\t\" a[4] }' > \"$d/c$n.tsv\"\n    [ -s \"$d/c$n.tsv\" ] && break\n  done\n  if [ ! -s \"$d/c$n.tsv\" ]; then\n    echo \"ERR nohistory $repo $path\" >&2\n    continue\n  fi\n  printf '%s\\t%s\\n' \"$n\" \"$path\" >> \"$d/todo.tsv\"\ndone 3< \"$d/paths.txt\"\n\n# Pass 2: fetch every needed blob in a few batches instead of one round trip per blob.\n# If the server refuses a batch, git falls back to fetching each blob when it is read.\ncat \"$d\"/c[0-9]*.tsv 2>/dev/null | cut -f4 | grep -v \"^$zero\\$\" | sort -u > \"$d/oids.txt\"\nif [ -s \"$d/oids.txt\" ]; then\n  xargs -n 500 git -c fetch.negotiationAlgorithm=noop fetch -q --no-tags --no-write-fetch-head \\\n      --recurse-submodules=no --filter=blob:none origin < \"$d/oids.txt\" >/dev/null 2>&1 || true\nfi\n\n# Pass 3: write each file's rows. A blob that cannot be read is an error, never a deletion:\n# the file is left out and logged as \"ERR blob\", so a re-run tries it again.\nwhile IFS=$'\\t' read -r i path <&3; do\n  key=$(keyof \"$path\")\n  dest=\"$out/vers/$key.$ext\"\n  ok=1\n  while IFS=$'\\t' read -r sha date co oid; do\n    if [ \"$oid\" = \"$zero\" ]; then b=\"\"\n    elif ! b=$(git cat-file blob \"$oid\" 2>/dev/null | base64 -w0; exit \"${PIPESTATUS[0]}\"); then\n      echo \"ERR blob $repo $path $sha\" >&2; ok=0; break\n    fi\n    printf '%s\\t%s\\t%s\\t%s\\n' \"$sha\" \"$date\" \"$co\" \"$b\"\n  done < \"$d/c$i.tsv\" > \"$d/rows.tsv\"\n  [ $ok = 1 ] || continue\n  if [ \"$ext\" = tsv ]; then mv \"$d/rows.tsv\" \"$dest\"; else gzip -n < \"$d/rows.tsv\" > \"$dest.tmp\" && mv \"$dest.tmp\" \"$dest\"; fi\n  printf '%s\\t%s\\t%s\\n' \"$key\" \"$repo\" \"$path\"\ndone 3< \"$d/todo.tsv\"\n","scripts/gharchive_census.py","#!/usr/bin/env python3\n\"\"\"Build a census of active public repos from GH Archive's free hourly files.\n\nGH Archive (https://www.gharchive.org) publishes every public GitHub event, one gzipped JSON file\nper hour, at https://data.gharchive.org/YYYY-MM-DD-H.json.gz. No account or token is needed.\nThis keeps the names of repos with at least one event of the chosen types in the chosen hours.\n\nUsage:\n  python3 gharchive_census.py --start 2026-10-01T00 --hours 24 [--every 1]\n                              [--types PushEvent] > census.txt\n\n  --start   first hour, UTC, as YYYY-MM-DDTHH\n  --hours   how many hours to read from --start\n  --every   read one hour in N (e.g. --hours 720 --every 24: one hour a day for 30 days)\n  --types   event types to count (default PushEvent: repos someone pushed to)\n\nOutput: one owner/repo per line, sorted, unique. make_list.py census reads it.\nOne hour is about 18 MB to download and gives about 40,000 repos (2026-10-01 12:00 UTC gave 39,397).\nA failed hour is reported on stderr and skipped.\n\"\"\"\nimport argparse\nimport gzip\nimport io\nimport json\nimport re\nimport sys\nimport urllib.request\nfrom datetime import datetime, timedelta\n\nNAME = re.compile(r\"^[A-Za-z0-9-]+/[A-Za-z0-9._-]+$\")\n\n\ndef main():\n    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n    ap.add_argument(\"--start\", required=True)\n    ap.add_argument(\"--hours\", type=int, default=1)\n    ap.add_argument(\"--every\", type=int, default=1)\n    ap.add_argument(\"--types\", nargs=\"+\", default=[\"PushEvent\"])\n    a = ap.parse_args()\n    t0 = datetime.strptime(a.start, \"%Y-%m-%dT%H\")\n    types, repos, read = set(a.types), set(), 0\n    for h in range(0, a.hours, a.every):\n        t = t0 + timedelta(hours=h)\n        url = f\"https://data.gharchive.org/{t:%Y-%m-%d}-{t.hour}.json.gz\"\n        try:\n            raw = urllib.request.urlopen(urllib.request.Request(url, headers={\"User-Agent\": \"github-file-history/1.0\"}), timeout=120).read()\n        except Exception as e:  # missing hours exist in the archive\n            print(f\"ERR {url} {e}\", file=sys.stderr)\n            continue\n        with gzip.open(io.BytesIO(raw), \"rt\", encoding=\"utf-8\", errors=\"replace\") as f:\n            for line in f:\n                try:\n                    e = json.loads(line)\n                except ValueError:\n                    continue\n                if e.get(\"type\") in types:\n                    name = (e.get(\"repo\") or {}).get(\"name\", \"\")\n                    if NAME.match(name):\n                        repos.add(name)\n        read += 1\n        print(f\"{url}: {len(repos)} repos so far\", file=sys.stderr)\n    sys.stdout.write(\"\".join(r + \"\\n\" for r in sorted(repos)))\n    print(f\"read {read} hours, {len(repos)} repos\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n","scripts/collect.sh","#!/usr/bin/env bash\n# collect.sh LIST OUTDIR [JOBS]\n#\n# LIST: one line per wanted file as owner/repo/PATTERN, where PATTERN is a path or a glob\n# (README.md, package.json, .cursor/rules/*.mdc, docs/**/*.md). Blank lines and lines starting\n# with # are ignored. Paths with spaces are not supported. make_list.py writes this file.\n#\n# Lines are grouped by repo, so each repo is cloned once however many patterns it has.\n# history.sh runs JOBS repos at a time (default 8).\n# Writes OUTDIR/manifest.tsv (key, repo, path: one line per file collected) and OUTDIR/err.log.\n# Safe to re-run: finished files are skipped, and literal paths already on disk need no clone.\nset -uo pipefail\nlist=${1:?list of owner/repo/pattern}; out=${2:?output dir}; jobs=${3:-8}\nhere=$(cd \"$(dirname \"$0\")\" && pwd)\nmkdir -p \"$out/vers\"\nout=$(cd \"$out\" && pwd)\n: > \"$out/err.log\"\n\n# one line per repo: owner/repo pattern1 pattern2 ...\ngrep -v -E '^[[:space:]]*(#|$)' \"$list\" | awk '{\n    n = split($0, a, \"/\"); repo = a[1] \"/\" a[2]; pat = substr($0, length(repo) + 2)\n    if (!(repo in seen)) { seen[repo] = 1; order[++k] = repo }\n    pats[repo] = pats[repo] \" \" pat\n  } END { for (i = 1; i <= k; i++) print order[i] pats[order[i]] }' > \"$out/.repos\"\n\nexport HISTORY_OUT=\"$out\" HISTORY_SH=\"$here/history.sh\"\n# xargs -L1 passes each repo line as arguments; history.sh prints manifest lines on stdout.\nxargs -P \"$jobs\" -L 1 bash -c 'bash \"$HISTORY_SH\" \"$0\" \"$HISTORY_OUT\" \"$@\"' < \"$out/.repos\" \\\n  > \"$out/.manifest\" 2>>\"$out/err.log\"\nsort -u \"$out/.manifest\" > \"$out/manifest.tsv\"\nrm -f \"$out/.manifest\" \"$out/.repos\"\n\nlisted=$(grep -c -v -E '^[[:space:]]*(#|$)' \"$list\")\necho \"listed $listed  files collected $(wc -l < \"$out/manifest.tsv\")  errors $(grep -c \"^ERR\" \"$out/err.log\")\"\n","examples/smoke.txt","# smoke test: two README histories, one glob, and one repo that does not exist\nchalk/chalk/readme.md\njshttp/mime-types/*.md\nnobody-here/not-a-repo-xyz/README.md\n","# five public repos for the README / package.json example\nchalk/chalk\nexpressjs/cors\nsindresorhus/ky\npallets/itsdangerous\njshttp/mime-types\n","examples/readme-package/list.txt","chalk/chalk/package.json\nchalk/chalk/[Rr][Ee][Aa][Dd][Mm][Ee]*\nexpressjs/cors/package.json\nexpressjs/cors/[Rr][Ee][Aa][Dd][Mm][Ee]*\nsindresorhus/ky/package.json\nsindresorhus/ky/[Rr][Ee][Aa][Dd][Mm][Ee]*\npallets/itsdangerous/package.json\npallets/itsdangerous/[Rr][Ee][Aa][Dd][Mm][Ee]*\njshttp/mime-types/package.json\njshttp/mime-types/[Rr][Ee][Aa][Dd][Mm][Ee]*\n","SKILL.md","LICENSE","MIT License\n\nCopyright (c) 2026 agentleFS\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","clerkState",{"_79":80},"__internal_clerk_state",{"_81":82,"_83":84,"_85":86,"_87":86,"_88":20,"_89":86,"_90":86,"_91":-7,"_92":-7,"_93":-7,"_94":-7,"_95":96,"_97":-7,"_98":-7,"_99":-7,"_100":-7,"_101":-7,"_102":20,"_103":20,"_104":20},"__clerk_ssr_state",{"_114":115,"_116":-5,"_117":-5,"_118":-5,"_119":-5,"_120":-5,"_121":-5,"_122":-5,"_123":-5,"_124":-5,"_125":-5,"_126":127,"_128":129,"_130":131,"_106":20},"__publishableKey","pk_live_Y2xlcmsuYWdlbnRsZWZzLmNvbSQ","__proxyUrl","","__domain","__isSatellite","__signInUrl","__signUpUrl","__signInForceRedirectUrl","__signUpForceRedirectUrl","__signInFallbackRedirectUrl","__signUpFallbackRedirectUrl","__clerk_debug",{"_105":20,"_106":20,"_107":86,"_108":109,"_110":86,"_111":84,"_112":20,"_113":86},"__clerkJSUrl","__clerkJSVersion","__clerkUIUrl","__clerkUIVersion","__prefetchUI","__telemetryDisabled","__telemetryDebug","__unsafeDisableDevelopmentModeConsoleWarning","isSignedIn","isAuthenticated","proxyUrl","reason","session-token-and-uat-missing","message","publishableKey","isSatellite","domain","tokenType","session_token","sessionClaims","sessionId","sessionStatus","userId","actor","orgId","orgRole","orgSlug","orgPermissions","factorVerificationAge","getToken",["SingleFetchFallback"],"has",["SingleFetchFallback"],"debug",["SingleFetchFallback"]]