agentleFS
joshtunejoshtune/mx-workflow/skills/hallucination-check/SKILL.md

hallucination-check

Skill1 starsChanged 3 months agoMITView on GitHub

Catch invented APIs, fabricated function signatures, made-up library methods, non-existent imports, and uninstalled packages in code — cross-referenced against the dependencies that are actually installed. Use after an AI assistant generates or edits code that calls external libraries, to verify every reference is real before trusting it.

1 other project uses a version of this.

Adopted by

Most recently taken up by

  • Other versions1 project uses a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 1

    1. blamechris/chroxy★ 8same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository5 other files it gives its agents.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Default to the changes under review: the working-tree diff plus untracked new files (new files are where hallucinated code most often …

    line 17
  2. Very specific:

    Prefer the installed copy (lockfile version, node_modules, site-packages) over assumptions.

    line 30
  3. Distinctive:

    When a reference genuinely can't be resolved (no type defs, dynamic export), report it UNVERIFIED — do not call it a hallucination.

    line 44

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- Standalone Skill derived from the mx-workflow plugin command `commands/hallucination-check.md` (github.com/joshtune/mx-workflow). If you maintain both, keep them in sync. -->

Hallucination Check

AI generators confidently invent things that don't exist: methods that were never on a library, signatures with the wrong arguments, type imports for symbols that aren't exported, whole packages that aren't installed. This skill hunts those down by cross-referencing every external reference against the code that is actually installed and present.

It is read-only and evidence-based — every flag must point at a real lookup ("parseAsync is not an export of zod@3.23.8"), never a guess.

1. Scope the code to check

Default to the changes under review: the working-tree diff plus untracked new files (new files are where hallucinated code most often hides). If the user names files or a diff, use that. Check the changed lines, not the whole repo.

2. Establish the dependency surface

Ground every lookup in what's actually installed, not in registry knowledge:

EcosystemManifestInstalled truth
JS/TSpackage.jsonnode_modules/<pkg> — exports/main, .d.ts type defs
Pythonpyproject.toml / requirements.txtpip show <pkg>, site-packages, __init__.py
Gogo.modgo list -m all, module cache
RustCargo.tomlcargo metadata, registry src

Prefer the installed copy (lockfile version, node_modules, site-packages) over assumptions. A hallucination check that guesses is itself hallucinating.

3. Extract references from the code

Collect, from the changed lines: imports (module + named symbols), member access on imported modules (pkg.method(...)), type references from imports, and internal references (calls to functions/classes/types that should be defined in this repo).

4. Verify each reference

  • Package exists? In the manifest AND present in node_modules/site-packages? Imported-but-absent → flag (and say whether it's a hallucination vs a missing install).
  • Symbol exported? Grep the installed package's entry/type defs for the named export. { foo } imported but not exported → flag.
  • Method/attribute exists? For pkg.method(...), check the installed type defs/source for the member. Missing → flag.
  • Signature plausible? Where type defs exist, sanity-check arg count/shape; mismatch → flag (lower confidence than a missing symbol).
  • Internal symbol defined? Grep the repo for the definition; called-but-never-defined → flag.

When a reference genuinely can't be resolved (no type defs, dynamic export), report it UNVERIFIED — do not call it a hallucination.

5. Report

HALLUCINATION CHECK
===================
Deps: <N> packages resolved · <M> references checked
VERDICT: CLEAN / SUSPECT (<n> likely, <u> unverified)

LIKELY HALLUCINATIONS
[HIGH]   src/auth.ts:12 — `verifyJwt` is not an export of `jose@5.2.0`
                          (installed exports: jwtVerify, SignJWT, …). Did you mean `jwtVerify`?
[HIGH]   src/db.ts:40   — package `pg-promise` imported but not in package.json or node_modules
[MEDIUM] src/api.ts:88  — `client.batchGet(...)` — no `batchGet` on @aws-sdk/client-dynamodb@3.x

UNVERIFIED (resolve manually)
[ ? ]    src/x.ts:5 — `plugin.run` — dynamic export, no type defs

INTERNAL
[HIGH]   src/checkout.ts:30 — `calculateTax(...)` called but not defined anywhere in repo

Report CLEAN only when every resolvable reference checked out. Always separate confident hallucinations from UNVERIFIED ones — never inflate the count.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.