agentleFS
Sign inStart free
andrasatandrasat/pi-llm-self-verify/skills/self-verify/SKILL.md

self-verify

Skill0 starsChanged 3 months agoMITView on GitHub

Generate independent candidate solutions and select the strongest one with a fine-grained logprob verifier. Use for high-stakes or ambiguous tasks, or when the user asks to self-verify or pick the best solution.

  • Reads credentials

3 other projects use a version of this.

Adopted by

Most recently taken up by

  • Other versions3 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 3

    1. m16khb-org/issueops★ 6same name
    2. pfeff/claude-skills★ 3same name
    3. Skydooooog0221/claude-code-skills★ 0same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repositoryOne other file it gives its agents.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 0 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use the self_verify tool for a best-of-N selection pass before implementing a high-stakes task.

    line 11
  2. Very specific:

    Use /self-verify <task> to run it directly.

    line 27
  3. Very specific:

    Every run stores its problem, candidates, winner, and scores under .pi/verify/.

    line 54

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Self-Verify

Use the self_verify tool for a best-of-N selection pass before implementing a high-stakes task. It generates candidates with read-only pi subprocesses, then scores them through a verifier backend that returns token logprobs. The parent tool writes ignored .pi/verify selection artifacts. The backend can be DeepSeek, Vertex Gemini, or any compatible OpenAI-style endpoint.

The result is advisory. The winning candidate still needs normal inspection, implementation, tests, and independent review. Self-verify is never a release gate and never authorizes committing, pushing, or closing an issue by itself.

Defaults

  • 3 candidates
  • 2 repeated evaluations per criterion
  • 1 tournament pivot
  • correctness and verification criteria

Use /self-verify <task> to run it directly. With no argument, the command uses the most recent user task when one is available.

Verifier setup

Choose one backend:

  • DeepSeek: /login deepseek or DEEPSEEK_API_KEY; default model deepseek-v4-flash.
  • OpenAI-compatible: set VERIFIER_BASE_URL, VERIFIER_MODEL, and VERIFIER_API_KEY for OpenAI, Together, Cerebras, OpenRouter, Fireworks, vLLM, or SGLang.
  • Vertex Gemini: set VERIFIER_PROVIDER=vertex, VERIFIER_MODEL=gemini-2.5-flash, GOOGLE_CLOUD_PROJECT, and either VERTEX_API_KEY or ADC.

Useful controls:

  • VERIFIER_TOP_LOGPROBS (1-20, default 20; use 5 for many Fireworks deployments)
  • VERIFIER_MAX_TOKENS
  • VERIFIER_EVALS and VERIFIER_PIVOTS
  • VERIFIER_AUTH_PROVIDER for an explicitly selected Pi API-key credential

Non-secret verify settings are candidates, evals, pivots, model, provider, topLogprobs, maxTokens, effort, project, location, and thinkingBudget; see the README for the complete JSON template. Endpoint, auth-provider, and credential values stay in Pi auth or the environment. The selected model must return usable token-level top alternatives for both score tags; literal score text and chosen-token-only responses are rejected. A normal text judge is not equivalent.

Every run stores its problem, candidates, winner, and scores under .pi/verify/. These artifacts can contain sensitive task text; keep them ignored or disable/clean them according to your project's data policy.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free