prime-verifiers
Skill8 starsChanged 8 months agoMITView on GitHub
Prime Intellect verifiers & PrimeRL — build RL environments, rubrics, and reward functions for LLM post-training. Covers environment classes, tool integration, GEPA prompt optimization, and async distributed RL training.
- Installs packages
In this repository23 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- validate-skillsSkill
- alchemi-toolkit-opsSkill
- autoresearch-setupSkill
- bionemoSkill
- biosimspaceSkill
- cuda-quantumSkill
- cutileSkill
- daskSkill
- flashinferSkill
- materials-projectSkill
- mpiSkill
- nemotronSkill
- omniverse-simreadySkill
- physicsnemoSkill
- polarsSkill
- qiskitSkill
- raySkill
- rdkitSkill
- slurmSkill
- tinkerSkill
- tritonSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 69
Every environment must expose a load_environment() function:
- Very specific: line 311
Validate with
prime eval run: before publishing
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Prime Intellect Verifiers & PrimeRL
Build RL environments, rubrics, and reward functions for LLM post-training using Prime Intellect's verifiers library and PrimeRL distributed training framework.
Official Sources:
Architecture Overview
Three integrated components form the stack:
| Component | Purpose | Key API |
| verifiers | Environment definitions, rubrics, evaluation | vf.Environment, vf.Rubric, vf.ToolEnv |
| prime-rl | Async distributed RL training (1000+ GPUs) | Orchestrator, Trainer, Inference |
| prime CLI | Workspace setup, sandboxes, publishing | prime lab setup, prime env init |
Quick Start
# Install verifiers
pip install verifiers
# Or with all extras
pip install "verifiers[all]"
# Set up a lab workspace
prime lab setup
# Create a new environment
prime env init my-env
cd environments/my-envCore Concepts
Environments
Everything is an Environment. An environment encapsulates: dataset + interaction harness + reward rubric.
Environment class hierarchy:
| Class | Use Case |
SingleTurnEnv | One prompt, one response, score it |
MultiTurnEnv | Multi-step agent interactions |
ToolEnv | OpenAI-compatible tool calling |
MCPEnv | Tools via MCP servers |
StatefulToolEnv | Per-rollout state (hidden args, dynamic tools) |
SandboxEnv | Containerized bash shell execution |
PythonEnv | Persistent Python REPL in sandbox |
EnvGroup | Combine multiple environments for multi-task training |
Environment Contract
Every environment must expose a load_environment() function:
import verifiers as vf
def load_environment(
split: str = "train",
seed: int | None = None,
system_prompt: str | None = None,
**kwargs,
) -> vf.Environment:
dataset = ... # HF Dataset with prompt/answer/info columns
rubric = vf.Rubric(
funcs=[reward_fn],
weights=[1.0],
parser=vf.XMLParser(fields=["answer"]),
)
return vf.SingleTurnEnv(
dataset=dataset,
rubric=rubric,
system_prompt=system_prompt or "You are a helpful assistant.",
)Rubrics and Reward Functions
Rubrics define how completions are scored. Reward functions use argument injection by name:
def exact_match(completion: str, answer: str, parser: vf.XMLParser) -> float:
"""Score 1.0 if parsed answer matches expected."""
parsed = parser(completion)
return 1.0 if parsed.get("answer", "").strip() == answer.strip() else 0.0
def length_penalty(completion: str) -> float:
"""Penalize overly long responses."""
return max(0.0, 1.0 - len(completion) / 5000)
rubric = vf.Rubric(
funcs=[exact_match, length_penalty],
weights=[0.8, 0.2],
parser=vf.XMLParser(fields=["answer"]),
)Available injectable arguments:
completion— model output textanswer— ground truth from datasetinfo— metadata dict from datasetparser— the rubric's parser instancejudge— LLM judge client (forJudgeRubric)state— rollout state dict (for stateful envs)messages— full conversation history
Built-in rubric types:
Rubric— weighted combination of reward functionsJudgeRubric— LLM-as-judge scoringMathRubric— symbolic math verification viamath-verifyRubricGroup— combines multiple rubrics in parallel
Parsers
Extract structured fields from model output:
# XML-style parsing: <answer>42</answer>
parser = vf.XMLParser(fields=["answer", "reasoning"])
# Think/answer parsing: <think>...</think>\n<answer>...</answer>
parser = vf.ThinkParser(fields=["answer"])
# Optional thinking: works with or without <think> tags
parser = vf.MaybeThinkParser(fields=["answer"])Tool Environments
import verifiers as vf
async def calculator(expression: str) -> str:
"""Evaluate a mathematical expression."""
try:
result = eval(expression)
return str(result)
except Exception as e:
return f"Error: {e}"
def load_environment(**kwargs) -> vf.Environment:
dataset = ...
rubric = vf.Rubric(funcs=[reward_fn], parser=parser)
return vf.ToolEnv(
dataset=dataset,
tools=[calculator],
rubric=rubric,
max_turns=10,
)Stateful Tool Environments
For tools that need per-rollout hidden state:
class MyEnv(vf.StatefulToolEnv):
def setup_state(self, prompt, answer, info):
"""Initialize per-rollout state."""
return {"secret": info["hidden_value"], "attempts": 0}
def update_tool_args(self, state):
"""Inject state into tool calls as hidden args."""
return {"check_answer": {"_secret": state["secret"]}}Rollout Lifecycle
For custom multi-turn environments:
class MyMultiTurnEnv(vf.MultiTurnEnv):
def env_response(self, messages, state):
"""Generate environment response after each model turn."""
last = messages[-1]["content"]
if "DONE" in last:
raise vf.stop() # End the rollout
return f"Feedback: {self.evaluate(last)}"
@vf.cleanup
def cleanup(self, messages, state):
"""Run after rollout ends (success or error)."""
return messages # Can modify final messages
@vf.teardown
def teardown(self, messages, state):
"""Run after everything, for resource cleanup."""
passCLI Workflow
# Set up workspace (downloads AGENTS.md, skills, templates)
prime lab setup
# Create new environment from template
prime env init math-solver
# Install environment locally
prime env install math-solver
# Run evaluation
prime eval run math-solver --endpoint openai://gpt-4o --num-samples 100
# Optimize system prompt with GEPA
prime gepa run math-solver --rounds 5
# Publish to Environments Hub
prime env pushEvaluation
# Local eval with any OpenAI-compatible endpoint
prime eval run my-env \
--endpoint openai://gpt-4o \
--num-samples 200 \
--split test
# With custom system prompt
prime eval run my-env \
--endpoint vllm://localhost:8000 \
--system-prompt "Think step by step."GEPA (Genetic-Pareto Prompt Optimization)
Iteratively optimize system prompts without gradient training:
prime gepa run my-env \
--endpoint openai://gpt-4o \
--rounds 10 \
--population 8
# Uses a teacher LLM to reflect on eval results
# and propose improved system promptsPrimeRL Training
Three disaggregated components for distributed RL:
Orchestrator (CPU) ──→ Trainer (GPU, FSDP2) ──→ Inference (vLLM)
│ │ │
├─ Schedules rollouts ├─ GRPO/PPO/RLOO ├─ OpenAI-compat API
├─ Assembles batches ├─ Custom model impls ├─ Hot weight reload
└─ Hosts env servers └─ LoRA support └─ update_weights()Supported RL objectives: GRPO, GSPO, OPO, RLOO, CISPO, PPO
Training config pattern:
model: Qwen/Qwen3-8B
objective: grpo
group_size: 8
max_turns: 10
environment: my-envEnvironment Publishing
Package structure for publishable environments:
my-env/
├── my_env/
│ ├── __init__.py
│ └── env.py # load_environment() here
├── pyproject.toml # [tool.verifiers.eval] config
└── README.mdpyproject.toml:
[project]
name = "my-env"
dependencies = ["verifiers>=0.1.10"]
[tool.verifiers.eval]
default_num_samples = 100
default_endpoint = "openai://gpt-4o"
[tool.hatch.build]
packages = ["my_env"]Best Practices
- Always use
prime env initto scaffold environments — never create manually - Validate with
prime eval runbefore publishing - Use
vf.ensure_keys()to validate required API keys at environment load time - Choose the right env class:
ToolEnvfor stateless tools,StatefulToolEnvfor per-rollout state,SandboxEnvfor code execution - Keep reward functions pure — no side effects, deterministic when possible
- Use group reward functions for GRPO-style advantage centering (accept
completionsplural, return lists) - Test locally first:
prime eval run my-env --endpoint openai://gpt-4o --num-samples 10
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

