NeMo-Agent-Toolkit / nat-tests

---
description: Follow these rules when the user's request involves writing, creating, or modifying tests for NeMo Agent Toolkit
globs:
alwaysApply: false
---

# Testing Guidelines for NeMo Agent Toolkit

Follow these rules when writing, creating, or modifying tests for NeMo Agent Toolkit.

## Referenced Documentation

- **Running Tests Guide**: [running-tests.md](mdc:docs/source/resources/running-tests.md) - Complete guide for running unit and integration tests
- **Test Fixtures**: [plugin.py](mdc:packages/nvidia_nat_test/src/nat/test/plugin.py) - Test fixtures
- **Test Utilities**: [utils.py](mdc:packages/nvidia_nat_test/src/nat/test/utils.py) - Test utilities
- **Docker Services**: [docker-compose.services.yml](mdc:tests/test_data/docker-compose.services.yml) - Services for integration testing

## General Testing Rules

All tests in NeMo Agent Toolkit use pytest. See the general coding guidelines for basic testing requirements.

### Unit Tests

- Use `pytest` for all unit tests
- Name test files `test_*.py`
- Mock external services with `pytest_httpserver` or `unittest.mock`
- Maintain ≥ 80% code coverage
- Do NOT add `@pytest.mark.asyncio` to any test - async tests are automatically detected and run by the async runner

### Fixtures
- Use `@pytest.fixture(name="fixture_name")` decorator pattern
- Fixtures that are used in multiple tests should be defined in the `packages/nvidia_nat_test/src/nat/test/plugin.py` file, fixtures specific to a single package should be defined in a `conftest.py` file in the `tests` directory of the package, and fixtures that are specific to a single test can be defined in the test file itself.

#### Available Fixtures

**API Keys**: `nvidia_api_key`, `openai_api_key`, `tavily_api_key`, `mem0_api_key`

**Services**: `milvus_uri`, `redis_url`, `mysql_connection_info`, `phoenix_url`

**Directories**: `root_repo_dir`, `examples_dir` (NAT repo only)

**Mocked Objects**: `mock_httpx_async_client`, `mock_httpx_sync_client`, `mock_builder`

There are many more fixtures available, refer to the [plugin.py](mdc:packages/nvidia_nat_test/src/nat/test/plugin.py) file and any `conftest.py` files defined in the file hierarchy of the current test file for the complete list.

### Integration Tests

For workflows that require actual LLM services or external services, follow the integration testing guidelines:

**See**: [Integration Testing Guidelines](mdc:.cursor/rules/nat-tests/integration-tests.mdc)

Key requirements:

- Use `@pytest.mark.slow` and `@pytest.mark.integration` decorators
- Use API key fixtures from `nvidia-nat-test` package
- Use `locate_example_config()` and `run_workflow()` utilities
- Service fixtures ensure services are running before tests execute

### Test LLM (nat_test_llm)

For deterministic testing without requiring actual LLM API calls, use the `nat_test_llm`:

**See**: [Test LLM Guidelines](mdc:.cursor/rules/nat-tests/nat-test-llm.mdc)

Key features:

- Stub LLM responses with predictable sequences
- No API keys or external services required
- Configurable artificial latency for testing timing scenarios
- Works with all framework wrappers (LangChain, LlamaIndex, CrewAI, etc.)

### Running Tests

Always use the user's current virtual environment when running tests.

Always include a package subdirectory or multiple subdirectories when invoking pytest.

```bash
# Unit tests only (default)
pytest packages/nvidia_nat_core

# Include slow tests
pytest --run_slow packages/nvidia_nat_core packages/nvidia_nat_langchain

# Include integration tests
pytest --run_integration packages/nvidia_nat_core

# All tests
pytest --run_slow --run_integration packages/nvidia_nat_langchain
```

## Related Rules

- **Integration Tests**: [integration-tests.mdc](mdc:.cursor/rules/nat-tests/integration-tests.mdc) - Detailed integration testing guidelines
- **Test LLM**: [nat-test-llm.mdc](mdc:.cursor/rules/nat-tests/nat-test-llm.mdc) - Using nat_test_llm to simulate deterministic LLM responses
- **General Guidelines**: [general.mdc](mdc:.cursor/rules/general.mdc) - Overall coding and testing standards

## Quick Reference

### Test Decorators

- `@pytest.mark.slow` - Tests taking >30 seconds
- `@pytest.mark.integration` - Tests requiring external services
- `@pytest.mark.usefixtures("api_key_name")` - Test requires a specific API key to be set in the environment

### Test Utilities

- `locate_example_config(ConfigClass)` - Find config files
- `run_workflow(config_file, question, expected_answer)` - Run and validate workflows
- `load_config(config_file)` - Load configuration objects

### Test LLM

- `_type: nat_test_llm` - Use in YAML configs to stub LLM responses
- `TestLLMConfig(response_seq=[...], delay_ms=0)` - Programmatic test LLM configuration