CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

SOL ExecBench is NVIDIA's GPU kernel evaluation and benchmarking framework. It evaluates custom GPU kernel solutions (PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++) for correctness and performance by comparing them against reference implementations.

Build & Run Commands

# Install dependencies (uses uv package manager)
uv sync --all-groups

# Run the CLI
uv run sol-execbench <problem_dir> --solution solution.json

# Run all tests
uv run pytest tests/

# Run a single test file
uv run pytest tests/sol_execbench/core/data/test_definition.py

# Run tests matching a keyword
uv run pytest tests/ -k "test_correctness"

# Run sample integration tests (requires CUDA GPU)
uv run pytest tests/sol_execbench/samples/

# Lint and format
uv run ruff check .
uv run ruff format .

Architecture

The codebase has three layers:

1. CLI (cli/main.py) — Click-based CLI entry point (sol-execbench). Parses args, invokes the driver, displays Rich tables with results.
2. Driver (driver/) — Orchestrates evaluation. ProblemPackager stages problem files (definition, workloads, solution sources) into a temp directory, then runs compilation and evaluation as subprocesses. The eval driver template (driver/templates/eval_driver.py, ~800 lines) is the self-contained script that runs inside the subprocess — it imports torch/triton, loads inputs, executes kernels, computes correctness/performance, and emits JSONL traces to stdout.
3. Core (core/) — Two sub-packages:
   • core/data/ — Pydantic v2 models for all data types: Definition (kernel specs, tensor shapes), Solution (source files, build specs), Workload (input generation), Trace (evaluation results with correctness/performance).
   • core/bench/ — Benchmarking utilities: CUDA event timing, numerical correctness computation, GPU clock locking, reward hack detection (detects timing manipulation, stream injection, monkey-patching).

Key execution flow

CLI → ProblemPackager stages files to temp dir → subprocess compiles C++/CUDA (if needed) → subprocess runs eval_driver.py → eval_driver outputs JSONL traces to stdout → CLI parses traces and displays results.

Two calling conventions

Kernel solutions use either destination-passing style (DPS, modifies pre-allocated outputs in-place) or return-value style (returns outputs directly). The eval driver handles both.

Test Markers & Fixtures

• @pytest.mark.requires_torch_cuda — auto-skipped when no CUDA GPU available
• @pytest.mark.cpp — compiles C++/CUDA extensions (slow)
• @requires_sm100 — skips on GPUs below sm_100 (Blackwell). Use this for cuTile tests.
• tmp_cache_dir fixture — isolated build cache per test via SOLEXECBENCH_CACHE_PATH

Style

Ruff for linting and formatting. Rule E741 is ignored. Exclude data/ and tests/data/ from lint.

Commits

Format: #<Issue> - <Title> with DCO sign-off (git commit -s).