agentleFS
Sign inStart free
NVIDIA-NeMoNVIDIA-NeMo/RL/skills/testing/SKILL.md

testing

Skill2.1k starsChanged 20 days agoApache-2.0View on GitHub

Testing conventions for NeMo-RL. Covers Ray actor coverage pragmas, nightly test requirements, and recipe naming rules.

  • Reads credentials

We split this skill into the rules it gives an agent

4 rules. 0 are common, 1 is distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 171 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Place # pragma: no cover on the class or def line:

    line 13
  2. Distinctive:

    When adding support for a new model, add a corresponding nightly test consisting of:

    line 30
  3. Very specific:

    Place in the appropriate domain subdirectory (examples/configs/recipes/llm/ or examples/configs/recipes/vlm/).

    line 34
  4. Very specific:

    Create a shell script in the matching domain (tests/test_suites/llm/ or tests/test_suites/vlm/).

    line 38

“Common” is measured: kept by at least 6 in 10 of the 171 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Testing Conventions

Coverage and Ray Actors

For any source file under nemo_rl/*.py that defines a class or function decorated with @ray.remote, add a coverage pragma because these run in separate Ray processes and are not reliably tracked by coverage.

Place # pragma: no cover on the class or def line:

import ray

@ray.remote  # pragma: no cover
class RolloutActor:
    def run(self) -> None:
        ...

@ray.remote  # pragma: no cover
def remote_eval(batch):
    ...

Nightly Tests for New Model Support

When adding support for a new model, add a corresponding nightly test consisting of:

1. Recipe YAML under examples/configs/recipes/

Place in the appropriate domain subdirectory (examples/configs/recipes/llm/ or examples/configs/recipes/vlm/). Name it following the recipe naming rules below.

2. Driver script under tests/test_suites/

Create a shell script in the matching domain (tests/test_suites/llm/ or tests/test_suites/vlm/). Source any common environment (e.g., common.env) and invoke the training entrypoint with uv run ... --config <path-to-yaml>. Match the driver script filename to the YAML base name with .sh.

3. Add to nightly list

Append the driver script path (relative to tests/test_suites/) to @tests/test_suites/nightly.txt.

Recipe Naming Rules

LLM Pattern

<algo>-<model>-<nodes>n<gpus>g-<strategy-and-params>[-modifiers][-long][.vN].(yaml|sh)
  • algo: sft, dpo, grpo, etc.
  • model: llama3.1-8b-instruct, qwen2.5-7b-instruct, etc.
  • nodes/gpus: 1n8g, 4n8g, 8n8g
  • strategy-and-params: automodel, automodel-ep8, megatron, megatron-ep16tp2cp2
  • modifiers (optional): sp, actckpt, fp8, noncolocated, quick
  • -long (optional): long-running recipe
  • .vN (optional): version suffix for convergence-impacting changes

Examples:

sft-llama3.1-8b-1n8g-automodel-long.yaml
grpo-llama3.1-8b-instruct-1n8g-megatron-fp8.yaml
grpo-qwen2.5-7b-instruct-4n8g-automodel-tp4sp.v3.yaml

VLM Pattern

vlm_<algo>-<model>-<nodes>n<gpus>g-<strategy>[-modifiers][.vN].(yaml|sh)

Directory Placement

examples/configs/recipes/
  llm/<name>.yaml
  vlm/<name>.yaml

tests/test_suites/
  llm/<name>.sh
  vlm/<name>.sh
  nightly.txt

Research projects

Research unit and functional tests are discovered under research/*/tests/. CI syncs each project's runtime dependencies and test group before executing them. Research suite scripts belong under research/<project>/tests/test_suites/, with matching recipes under research/<project>/configs/recipes/. List scripts by their repository-relative paths in the root suite lists (nightly.txt, disabled.txt, etc.). Script/recipe accounting includes both core and research directories, preserving each research project's identity.

Download now (ZIP)

Trending

Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data

Week of Aug 30: 0 new projectsWeek of Sep 6: 0 new projectsWeek of Sep 13: 0 new projectsWeek of Sep 20: 0 new projectsWeek of Sep 27: 0 new projectsWeek of Oct 4: 0 new projectsWeek of Oct 11: 0 new projectsWeek of Oct 18: 0 new projects
Aug 2026Sep 2026Oct 2026
New projects taking a version of this file each week, over the 8 weeks our latest run read: only 0 in all, so this is a hint, not a trend.

Who's talking about NVIDIA-NeMo/RL

The biggest 5 of 52 mentions linking to the repository that our crawl found. Each line is their own headline.

See all 52

Talked about on GitHub and Bluesky (52 times).

On GitHub · 50

45 more on GitHub

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free