Evals Plugin

The Plugin CLI entrypoint is uv run nemo evals.

Purpose

Use this skill to choose an evaluation interface and metric, validate a minimal example, submit a NeMo Helix evaluation job, and retrieve its results.

Inputs

Establish these inputs before building an evaluation:

• Evaluation interface: dataset-driven, task-driven, or retrieval-driven evaluation
• Execution interface: standalone SDK evaluation or a durable NeMo Helix job.
• Pass/fail dataset examples: the smallest representative pass and failure cases.
• Metrics: the behaviors to score and the template fields they consume.
• Target: no target for offline scoring, or the model, agent, runner, or precomputed trials that produce outputs.

Instructions

1. Clarify whether the input is dataset-driven rows, task-driven agent work, or a BEIR retrieval corpus. If the user wants the full embed/rerank fine-tune recipe, hand off to nemo-retrieval-recipes and use this skill only to run and debug retrieve-eval.
2. Choose the simplest metric that measures the requested behavior. Prefer deterministic metrics when possible.
3. Build a tiny smoke case with one expected pass and one expected failure. For a Kubernetes taskset run, the oracle run in Harbor on Kubernetes is the smoke case.
4. Validate metric behavior with the standalone SDK. Inspect row-level output, aggregate count and nan_count, and coverage before interpreting a mean.
5. Fix field mappings, prompts, parsers, or task definitions before scaling.
6. Submit the platform job only after the input and scoring shape works.

Read Metric Selection before choosing a metric for a rubric, RAG workflow, or tool-calling evaluation.

Choose the execution interface

Default to the plugin-specific job commands for durable platform evaluation: nemo evals evaluate, nemo evals agent-evaluate, and nemo evals retrieve-eval. The old nemo evals ... run and nemo evals ... submit CLI verbs are not part of the supported evaluator job surface. The one exception is nemo evals runs submit, the supported path for Harbor taskset runs on Kubernetes. For fast metric iteration without the platform, use the standalone nhx_evals_sdk.Evaluator instead.

• Read SDK Execution for datasets, targets, configuration, field mapping, job lifecycle, and custom metric packaging.
• Read Stored Resources for persisted definitions and result queries.

Limitations

• api_key_secret is an environment-variable name standalone but a NeMo Platform secret name on submit. See API Auth.
• HTTP 409 from a submission often means a referenced platform secret is missing, not a duplicate job. Read the response body.
• intent is grader metadata and is never shown to the agent; only inputs reaches it.
• Metric templates use item.* for dataset rows but reference.*, sample.*, and inputs.* in agent evaluation.
• Metric progress can reach 100 percent before the platform job is terminal. Always call job.wait_until_done() before retrieving results or downloading artifacts.

CLI Interface

Prerequisites

All commands in this file assume that the shell's working directory is the root of the NVIDIA-NeMo/nemo-helix repository.

In a NeMo Helix repository checkout, run commands through the workspace:

# confirms plugin readiness and lists the registered evaluator jobs.
uv run nemo evals info
# lists available metric names; add a metric name to print its schema.
uv run nemo evals metric-types
# next three commands print the dataset-driven, task-driven, and retrieval job input and
# output schemas - can be very large, use with caution to avoid filling up the context window.
uv run nemo evals evaluate explain
uv run nemo evals agent-evaluate explain
uv run nemo evals retrieve-eval explain

When the skill and plugin are installed, use the installed nemo command without assuming a repository root or manually activating .venv.

Resolve bundled assets relative to this skill directory. In this repository the canonical path is skills/nemo-evals-plugin; an installed skill may live under a different skills root.

Bundled assets

Available Scripts

In this repository, NeMo uses the displayed workspace command:

uv run --frozen python skills/nemo-evals-plugin/scripts/generate_example_specs.py --check

Do not assume a client-specific run_script() helper; use the displayed uv run command.

Examples

Dataset-driven evaluation examples

• Follow Validate standalone, then submit to the platform. for the two-row pass/fail smoke test and its CLI submission.
• Follow Map noncanonical fields when dataset columns need field_mapping.
• Follow Getting job results for submission, terminal waiting, result retrieval, and artifact download.
• Follow Store a metric, task, and taskset for reusable definitions, and Query persisted results for result lookup.

Task-driven agent evaluation examples

Standalone SDK evaluation

Use AgentEvaluator().run(...) for standalone task-driven SDK evaluation. Its target can be a Model, a GenericAgent, or a direct AgentTaskRunner.

Platform job evaluation

Use the plugin agent-evaluate job for platform task evaluation. Its target is a ModelTarget, AgentTarget, FabricRunnerTarget, HarborRunnerTarget, or GymRunnerTarget; alternatively provide precomputed trials. Provide exactly one of target or trials.

A FabricRunnerTarget's source is an inline config ({"kind": "fabric", "source": {"config": {...}}}) or a registered agent (nemo agents register): {"kind": "fabric", "source": {"agent": "<name>"}}. A HarborRunnerTarget's source takes the same registered-agent shape ({"kind": "harbor", "source": {"agent": "<name>"}}) to run it inside each task container, next to a built-in agent ({"name": "oracle"}) or your own ({"import_path": "pkg:Agent"}). A GymRunnerTarget's source takes it too ({"kind": "gym", "source": {"agent": "<name>"}, "resources_server": "<name>"}), running it on the sandboxed Gym host with its default model on Gym's policy server; the job stages a wheels-v1 package with the harness, so sandboxed execution is required. At submit time the service resolves the agent exactly as a deployment would — models bound to the workspace Inference Gateway, no credentials in the spec — and runs it fresh for every trial; it never calls an existing deployment. An optional environment (the same spec nemo agents deploy takes) redirects the agent's declared MCP servers to mocks, adds process env, and binds secrets by reference. Only nemo-agents-spec-v1 agents resolve; a different model is a different agent, so there is no model override.

Submission accepts inline tasks or a stored TasksetRef. Stored tasksets are resolved in the target workspace.

Kubernetes taskset run

To run a Harbor task suite on a Kubernetes deployment, use nemo evals runs submit, not an agent-evaluate job. Read Harbor on Kubernetes for the oracle smoke run, openhands with a model (closed-book opt-in), Experiments, results and retries.

Read Agent Evaluation for inline tasks, TasksetRef, concurrency, fail-fast behavior, sparse-output denominators, result artifacts, and runner configuration, including Gym FileSet and sandbox requirements.

Prepare Fabric in a repository checkout

Fabric runner examples and tests need the optional harness adapters and the matching Relay gateway:

uv sync --frozen --package nhx-evals-sdk --extra fabric --inexact
script/dev-install-fabric.sh

The install script downloads the checksum-verified nemo-relay binary that matches the locked Python bindings. Add its reported directory to PATH, then use uv run --frozen --no-sync ... for Fabric checks so uv does not remove the optional adapters.

Output Format

Report a completed platform evaluation in this form:

Job: <job-name>
Status: <terminal-status>
Metrics: <metric-names>
Mean: <aggregate-mean>
Artifacts: <downloaded result or artifact location>
Errors: <error messages>

Read specialized references

• Read Evaluator API Auth before using a model, agent, remote metric, or durable submission.
• Read LLM Judge before writing judge scores, prompts, or parsers.

Troubleshooting

Read Evaluator troubleshooting when schema, authentication, job, result, or runner behavior fails.

Follow security best practices

Never print, serialize, or commit secret values. Store only environment-variable names or platform secret references in specs and examples.