holmesgpt

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

HolmesGPT is an AI-powered troubleshooting agent that connects to observability platforms (Kubernetes, Prometheus, Grafana, etc.) to automatically diagnose and analyze infrastructure and application issues. It uses an agentic loop to investigate problems by calling tools to gather data from multiple sources.

## Development Commands

### Environment Setup
```bash
# Install dependencies with Poetry
poetry install
```

**Use Poetry 1.8.5 to regenerate `poetry.lock`.** The committed lockfile is
`lock-version = "2.0"`, which Poetry 1.8.x produces. Running `poetry lock` under
Poetry 2.x rewrites the whole file to `lock-version = "2.1"` and produces a huge
diff of unrelated churn. Always lock with 1.8.5, and prefer `--no-update` so only
your dependency change appears in the diff:

```bash
poetry lock --no-update   # with Poetry 1.8.5
poetry check --lock       # verify the lockfile matches pyproject.toml
```

If your system Poetry is 2.x, install 1.8.5 into a throwaway venv and use it just
for locking:

```bash
python3 -m venv /tmp/poetry185 && /tmp/poetry185/bin/pip install "poetry==1.8.5"
/tmp/poetry185/bin/poetry lock --no-update
```

### Testing

```bash
# Install test dependencies with Poetry
poetry install --with dev
```

```bash
# Run all non-LLM tests (unit and integration tests)
make test-without-llm
poetry run pytest tests -m "not llm"

# Run LLM evaluation tests (requires API keys)
make test-llm-ask-holmes          # Test single-question interactions
make test-llm-investigate         # Test AlertManager investigations
poetry run pytest tests/llm/ -n 6 -vv  # Run all LLM tests in parallel

# Run pre-commit checks (includes ruff, mypy, poetry validation)
# NOTE: Only run these when the user explicitly asks. They run in CI automatically.
make check
poetry run pre-commit run -a
```

### Code Quality (only run when explicitly asked)
```bash
# Format code with ruff
poetry run ruff format

# Check code with ruff (auto-fix issues)
poetry run ruff check --fix

# Type checking with mypy
poetry run mypy
```

## Architecture Overview

### Core Components

**CLI Entry Point** (`holmes/main.py`):
- Typer-based CLI with subcommands for `ask`, `investigate`, `toolset`
- Handles configuration loading, logging setup, and command routing

** Interactive mode for CLI** (`holmes/interactive.py`):
- Handles interactive mode for `ask` subcommand
- Implements slash commands

**Configuration System** (`holmes/config.py`):
- Loads settings from `~/.holmes/config.yaml` or via CLI options
- Manages API keys, model selection, and toolset configurations
- Factory methods for creating sources (AlertManager, Jira, PagerDuty, etc.)

**Core Investigation Engine** (`holmes/core/`):
- `tool_calling_llm.py`: Main LLM interaction with tool calling capabilities
- `investigation.py`: Orchestrates multi-step investigations with runbooks
- `toolset_manager.py`: Manages available tools and their configurations
- `tools.py`: Tool definitions and execution logic

**Plugin System** (`holmes/plugins/`):
- **Sources**: AlertManager, Jira, PagerDuty, OpsGenie integrations
- **Toolsets**: Kubernetes, Prometheus, Grafana, AWS, Docker, etc.
- **Prompts**: Jinja2 templates for different investigation scenarios
- **Destinations**: Slack integration for sending results

### Key Patterns

**Toolset Architecture**:
- Each toolset is a YAML file defining available tools and their parameters
- Tools can be Python functions or bash commands with safety validation
- Toolsets are loaded dynamically and can be customized via config files
- **Important**: All toolsets MUST return detailed error messages from underlying APIs to enable LLM self-correction
  - Include the exact query/command that was executed
  - Include time ranges, parameters, and filters used
  - Include the full API error response (status code and message)
  - For "no data" responses, specify what was searched and where

**Thin API Wrapper Pattern for Python Toolsets**:
- Reference implementation: `servicenow_tables/servicenow_tables.py`
- Use `requests` library for HTTP calls (not specialized client libraries like `opensearchpy`)
- Simple config class with Pydantic validation
- Health check in `prerequisites_callable()` method
- Each tool is a thin wrapper around a single API endpoint

**Server-Side Filtering is Critical**:
- **Never return unbounded data from APIs** - this causes token overflow
- Always include filter parameters on tools that query collections (e.g., `index` parameter for Elasticsearch _cat APIs)
- Example problem: `opensearch_list_shards` returned ALL shards → 25K+ tokens on large clusters
- Example fix: `elasticsearch_cat` tool requires `index` parameter for shards/segments endpoints
- When server-side filtering is not possible, use `JsonFilterMixin` (see `json_filter_mixin.py`) to add `max_depth` and `jq` parameters for client-side filtering

**Toolset Config Backwards Compatibility**:
When renaming config fields in a toolset, maintain backwards compatibility using Pydantic's `extra="allow"`:

```python
# ✅ DO: Use extra="allow" to accept deprecated fields without polluting schema
class MyToolsetConfig(BaseModel):
    model_config = ConfigDict(extra="allow")

    # Only define current field names in schema
    new_field_name: int = 10

    @model_validator(mode="after")
    def handle_deprecated_fields(self):
        extra = self.model_extra or {}
        deprecated = []

        # Map old names to new names
        if "old_field_name" in extra:
            self.new_field_name = extra["old_field_name"]
            deprecated.append("old_field_name -> new_field_name")

        if deprecated:
            logging.warning(f"Deprecated config names: {', '.join(deprecated)}")
        return self

# ❌ DON'T: Define deprecated fields in schema with Optional[None]
class BadConfig(BaseModel):
    new_field_name: int = 10
    old_field_name: Optional[int] = None  # Pollutes schema, shows in model_dump()
```

Benefits of `extra="allow"` approach:
- Schema only shows current field names
- `model_dump()` returns clean output without deprecated fields
- Old configs still work (backwards compatible)
- Deprecation warnings guide users to update

See `prometheus/prometheus.py` PrometheusConfig for a complete example.

**Class Hierarchy Placement**:
- When adding new config fields, methods, or behavior, always check the class hierarchy and place the change at the most general level that applies
- Don't scope a fix to a specific subclass just because the issue/request mentions it by name — check if sibling classes share the same need
- Example: `timeout_seconds` and `max_retries` belong on `GrafanaConfig`, not `GrafanaTempoConfig`, because all Grafana toolsets (Tempo, Loki, Dashboards) make HTTP requests

**LLM Integration**:
- Uses LiteLLM for multi-provider support (OpenAI, Anthropic, Azure, etc.)
- Structured tool calling with automatic retry and error handling
- Context-aware prompting with system instructions and examples

**Investigation Flow**:
1. Load user question/alert
2. Select relevant toolsets based on context
3. Execute LLM with available tools
4. LLM calls tools to gather data
5. LLM analyzes results and provides conclusions
6. Optionally write results back to source system

## Testing Framework

**Three-tier testing approach**:

1. **Unit Tests** (`tests/`): Standard pytest tests for individual components
2. **Integration Tests**: Test toolset integrations
3. **LLM Evaluation Tests** (`tests/llm/`): End-to-end tests using fixtures

**Running regular (non-LLM) tests**:
```bash
poetry run pytest tests -m "not llm"
make test-without-llm
```

**Running LLM eval tests**:
```bash
# Run specific eval - IMPORTANT: Use -k flag, NOT full test path with brackets
poetry run pytest -k "09_crashpod" --no-cov

# Run all evals in parallel
poetry run pytest tests/llm/ -n 6 --no-cov

# Regression evals
poetry run pytest -m 'llm and easy' --no-cov
```

For the complete eval CLI reference (flags, env vars, model comparison, debugging), see the `/create-eval` skill which contains full documentation in its reference files.

## Configuration

**Config File Location**: `~/.holmes/config.yaml`

**Key Configuration Sections**:
- `model`: LLM model to use (default: gpt-5.4)
- `api_key`: LLM API key (or use environment variables)
- `custom_toolsets`: Override or add toolsets
- `custom_runbooks`: Add investigation runbooks
- Platform-specific settings (alertmanager_url, jira_url, etc.)

**Environment Variables**:
- `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`: LLM API keys
- `OPENROUTER_API_KEY`: Alternative LLM provider via OpenRouter (domain: `api.openrouter.ai`). When using OpenRouter, you must also set `CLASSIFIER_MODEL` to an OpenRouter model (e.g., `CLASSIFIER_MODEL="openrouter/openai/gpt-4.1"`) because the default classifier model is not available via OpenRouter.
- `MODEL`: Override default model(s) - supports comma-separated list
- `CLASSIFIER_MODEL`: Override the classifier model used internally. Required when using OpenRouter (e.g., `openrouter/openai/gpt-4.1`)
- `RUN_LIVE`: Enable live execution of tools in tests (default: true)
- `BRAINTRUST_API_KEY`: For test result tracking and CI/CD report generation
- `BRAINTRUST_ORG`: Braintrust organization name (default: "robustadev")
- `ELASTICSEARCH_URL`, `ELASTICSEARCH_API_KEY`: For Elasticsearch/OpenSearch cloud testing

## Development Guidelines

**Code Quality**:
- Use Ruff for formatting and linting (configured in pyproject.toml)
- Type hints required (mypy configuration in pyproject.toml)
- Pre-commit hooks enforce quality checks in CI
- **ALWAYS place Python imports at the top of the file**, not inside functions or methods
- **Use `tenacity` for retries**, not hand-rolled retry loops — the `@retry(...)` decorator normally, or the `Retrying(...)` iterator form when the attempt budget/params must come from per-instance or dynamic values. Only hand-roll a retry when there's a real reason to, and document why.
- **NEVER run `pre-commit`, `ruff`, or `mypy` unless the user explicitly asks you to**. These tools are triggered by commit hooks which are not installed on all machines, and running them causes widespread formatting/type changes to files unrelated to your task. Only lint/format files you are actively editing, and only if asked.

**Documentation Examples**:
- **Primary examples should use the latest Anthropic Claude models**:
  - Recommended: `anthropic/claude-sonnet-4-5-20250929` or `anthropic/claude-opus-4-5-20251101`
  - Use the latest Claude 4.5 family models (Sonnet or Opus) as the default/primary examples
- You may include other providers (OpenAI, Gemini, etc.) where it would be useful for users, such as in model listing sections or provider-specific documentation
- Avoid using deprecated or older model versions like `claude-3.5-sonnet`, `gpt-4-vision-preview`

**Testing Requirements**:
- All new features require unit tests
- New toolsets require integration tests
- Complex investigations should have LLM evaluation tests
- Maintain 40% minimum test coverage
- **Live execution is now enabled by default** to ensure tests match real-world behavior
- **Use `responses` library for HTTP mocking**, not `@patch("requests.get")`. The `responses` library intercepts at the transport/adapter level, giving more realistic test behavior. Use `responses.RequestsMock()` with `rsps.add()` for mock responses.

**Pull Request Process**:
- PRs require maintainer approval
- Pre-commit hooks are checked in CI (do NOT run them locally unless asked)
- LLM evaluation tests run automatically in CI
- Keep PRs focused and include tests
- **ALWAYS use `git commit -s`** to sign off commits (required for DCO)
- **When committing, use `git commit -s --no-verify`** to skip local pre-commit hooks (they are not installed consistently and will cause unrelated changes)

**Git Workflow Guidelines**:
- ALWAYS create commits, NEVER amend
- ALWAYS merge, NEVER rebase
- ALWAYS push, NEVER force push
- Maintain a history of your work to allow the user to revert back to a previous iteration


**File Structure Conventions**:
- Toolsets: `holmes/plugins/toolsets/{name}.yaml` or `{name}/`
- Prompts: `holmes/plugins/prompts/{name}.jinja2`
- Tests: Match source structure under `tests/`

**Adding a New Integration (Toolset)**:
When adding a new toolset or integration, update all of the following pages to keep them in sync:

1. `README.md` — Data Sources table (add a row with logo, link, status, and description)
2. `docs/walkthrough/why-holmesgpt.md` — Categorized integration list under "Every Major Observability Platform"
3. `docs/data-sources/builtin-toolsets/index.md` — Grid cards listing on the toolsets index page
4. `docs/data-sources/builtin-toolsets/{name}.md` — Dedicated documentation page for the new toolset
   - Its deployment tabs follow the shape `tests/docs/test_deployment_tab_groups.py` checks
5. Add a logo image to `images/integration_logos/` if one is available

## Debugging CLI / Rich Live Display Issues

When troubleshooting terminal rendering bugs (ghost frames, flickering, misaligned output):

**Capturing terminal output through a PTY:**
```bash
# Use `script` to force a PTY and capture raw ANSI escape sequences
script -qec "poetry run python your_script.py" /dev/null > /tmp/raw_output.txt 2>&1
```
Without a PTY, Rich detects non-interactive mode and skips Live rendering entirely.

**Analyzing ANSI escape sequences:**
```python
# Key escape codes for Rich Live:
# \x1b[1A  = cursor up 1 line
# \x1b[2K  = erase entire line
# Rich erases previous frame with: (erase + cursor-up) × height, then prints new frame

# Count cursor-ups per frame transition to detect drift:
import re
erase_pattern = r"\x1b\[2K(?:\x1b\[1A\x1b\[2K)*"
for match in re.finditer(erase_pattern, raw_output):
    ups = match.group(0).count("\x1b[1A")
```

**Writing unit tests for Live display (no LLM required):**
```python
# Render to StringIO with force_terminal=True to get ANSI sequences
from io import StringIO
buf = StringIO()
console = Console(file=buf, force_terminal=True, width=120)
# ... render frames ...
raw = buf.getvalue()  # Contains full ANSI escape sequences
# Parse cursor-up counts vs frame heights to detect ghost frames
```

**Key patterns:**
- Ghost frames = cumulative drift where each frame leaves 1+ orphaned lines
- Verify by counting: cursor-ups per transition should equal rendered lines per frame
- Known Rich 13.9.4 bug: `Live.refresh()` calls `console.print(Control())` with default `end="\\n"`, adding a trailing newline not counted in `LiveRender._shape`. When the terminal has room below the display, each frame leaks 1 ghost line. When the display is at the bottom (common case), the `\\n` causes scrolling and `height-1` cursor-ups is correct.
- Workaround: subclass `Live` and override `refresh()` to pass `end=""`. Do NOT patch `position_cursor` — that over-erases when the display is at the terminal bottom (the common case).

## Investigating Eval Regressions / Holmes Behavior Changes

**Understand the behavior from trace data before designing a fix.** Braintrust (or the local evals_report.md) holds the rendered prompts, per-LLM-call metrics, and tool-call sequences for each iter. Pull traces for one baseline run and one current run for the same `(test, model)` before reading any source diffs — file-level diffs (prompts, code, config) routinely mislead about what actually changed at runtime (e.g. a jinja2 template can grow in source lines and shrink in rendered output). Look at individual runs first; aggregate statistics over n iters can hide deterministic per-iter differences under variance.

When reverting or fixing a suspect PR, read its full diff (`git show --stat <commit>`) before deciding what to change — a PR often has multiple effects, and reverting only the file you noticed leaves the others still acting on the model.

## Security Notes

- All tools have read-only access by design
- Bash toolset validates commands for safety
- No secrets should be committed to repository
- Use environment variables or config files for API keys
- RBAC permissions are respected for Kubernetes access

## Eval Tests (LLM Evaluations)

For creating, running, and debugging LLM eval tests, use the `/create-eval` skill. It contains the complete workflow, test_case.yaml field reference, anti-hallucination patterns, infrastructure setup guides, and CLI reference.

**Test Structure:**
- Use sequential test numbers: check existing tests for next available number
- Required files: `test_case.yaml`, infrastructure manifests, `toolsets.yaml` (if needed)
- Use dedicated namespace per test: `app-<testid>` (e.g., `app-177`)
- All resource names must be unique across tests to prevent conflicts

**Tags:**
- **CRITICAL**: Only use valid tags from `pyproject.toml` - invalid tags cause test collection failures
- Check existing tags before adding new ones, ask user permission for new tags

**Cloud Service Evals (No Kubernetes Required)**:
- Evals can test against cloud services (Elasticsearch, external APIs) directly via environment variables
- Faster setup (<30 seconds vs minutes for K8s infrastructure)
- `before_test` creates test data in the cloud service; `after_test` cleans up **only if safe** (see reentrancy below)
- Use `toolsets.yaml` to configure the toolset with env var references: `api_url: "{{ env.ELASTICSEARCH_URL }}"`
- **CI/CD secrets**: When adding evals for a new integration, you must add the required environment variables to `.github/workflows/eval-regression.yaml` in the "Run tests" step. Tell the user which secrets they need to add to their GitHub repository settings (e.g., `ELASTICSEARCH_URL`, `ELASTICSEARCH_API_KEY`).
- **HTTP request passthrough**: The root `conftest.py` has a `responses` fixture with `autouse=True` that mocks ALL HTTP requests by default. When adding a new cloud integration, you MUST add the service's URL pattern to the passthrough list in `conftest.py` (search for `rsps.add_passthru`). Use `re.compile()` for pattern matching (e.g., `rsps.add_passthru(re.compile(r"https://.*\.cloud\.es\.io"))`).
- **Cloud Service Eval Reentrancy**: The same eval can run on multiple PRs in parallel in CI. Cloud service evals that create resources with static names (e.g., Confluence spaces, Elasticsearch indices) must be **reentrant**:
  - `before_test` must be **idempotent**: create-or-reuse resources, never fail if they already exist
  - `after_test` must **NOT delete shared resources** that another parallel run may be using. Either omit `after_test` entirely, or limit cleanup to resources with a unique run-scoped identifier
  - Use test-ID-based resource names (e.g., `HLMS233` for space keys) to avoid collisions with other evals, but accept that the same eval may overlap with itself across parallel PR runs
  - Kubernetes evals don't have this problem because each PR gets its own KIND cluster, so namespaces are already isolated. Cloud service evals share a single account/instance across all PR runs

**User Prompts & Expected Outputs:**
- **Be specific**: Test exact values like `"The dashboard title is 'Home'"` not generic `"Holmes retrieves dashboard"`
- **Match prompt to test**: User prompt must explicitly request what you're testing
  - BAD: `"Get the dashboard"`
  - GOOD: `"Get the dashboard and tell me the title, panels, and time range"`
- **Anti-cheat prompts**: Don't use technical terms that give away solutions
  - BAD: `"Find node_exporter metrics"`
  - GOOD: `"Find CPU pressure monitoring queries"`
- **Test discovery, not recognition**: Holmes should search/analyze, not guess from context
- **Ruling out hallucinations is paramount**: When choosing between test approaches, prefer the one that rules out hallucinations:
  - **Best**: Check specific values that can only be discovered by querying (e.g., unique IDs, injected error codes, exact counts)
  - **Acceptable**: Use `include_tool_calls: true` to verify the tool was called when output values are too generic to rule out hallucinations
  - **Bad**: Check generic output patterns that an LLM could plausibly guess (e.g., "cluster status is green/yellow/red", "has N nodes")
- **expected_output is invisible to LLM**: The `expected_output` field is only used by the evaluator - the LLM never sees it. This means:
  - You can safely put secrets/verification codes in `expected_output` that the LLM must discover
  - `before_test` can inject a unique verification code into test data, and `expected_output` can check for it
  - This is a powerful pattern for cloud service tests: create data with a unique code in `before_test`, ask LLM to find it, verify with `expected_output`
  ```yaml
  # Example: before_test creates a page with verification code "HOLMES-EVAL-7x9k2m4p"
  # The LLM must discover this code by querying the service
  expected_output:
    - "Must report the verification code: HOLMES-EVAL-7x9k2m4p"
  ```
- **`include_tool_calls: true`**: Use when expected output is too generic to be hallucination-proof. Prefer specific answer checking when possible, but verifying tool calls is better than a test that can't rule out hallucinations.
  ```yaml
  # Use when values are generic (cluster health could be guessed)
  include_tool_calls: true
  expected_output:
    - "Must call elasticsearch_cluster_health tool"
    - "Must report cluster status"
  ```

**Infrastructure Setup:**
- **Don't just test pod readiness** - verify actual service functionality
- Poll real API endpoints and check for expected content (e.g., `"title":"Home"`, `"type":"welcome"`)
- **CRITICAL**: Use `exit 1` when setup verification fails to fail the test early
- **Never use `:latest` container tags** - use specific versions like `grafana/grafana:12.3.1`

### Running and Testing Evals

## 🚨 CRITICAL: Always Test Your Changes

**NEVER submit test changes without verification**:

### Required Testing Workflow:
1. **Setup Phase**: `poetry run pytest -k "test_name" --only-setup --no-cov`
2. **Full Test**: `poetry run pytest -k "test_name" --no-cov`
3. **Verify Results**: Ensure 100% pass rate and expected behavior

### When to Test:
- ✅ After creating new tests
- ✅ After modifying existing tests  
- ✅ After refactoring shared infrastructure
- ✅ After performance optimizations
- ✅ After adding/changing tags

### Red Flags - Never Skip Testing:
- ❌ "The changes look good" without running
- ❌ "It's just a small change"
- ❌ "I'll test it later"

**Testing is Part of Development**: Testing is not optional - it's an integral part of the development process. Untested code is broken code.

**Testing Methodology:**
- Phase 1: Test setup with `--only-setup` flag first
- Phase 2: Run full test after confirming setup works
- Use background execution for long tests: `nohup ... > logfile.log 2>&1 &`
- Handle port conflicts: clean up previous test port forwards before running

**Common Flags:**
- `--skip-cleanup`: Keep resources after test (useful for debugging setup)
- `--skip-setup`: Skip before_test commands (useful for iterative testing)

## Shared Infrastructure Pattern

**When to use shared infrastructure**:
- Multiple tests use the same service (Grafana, Loki, Prometheus)
- Service configuration is standardized across tests

**Implementation**:
```bash
# Create shared manifest in tests/llm/fixtures/shared/servicename.yaml
# Use in tests:
kubectl apply -f ../../shared/servicename.yaml -n app-<testid>
```

**Benefits**:
- Single place for version updates
- Consistent configuration across tests
- Reduced maintenance overhead
- Follows established pattern (Loki, Prometheus, Grafana)

## Setup Verification Best Practices

**Prefer kubectl exec over port forwarding for setup verification**:
```bash
# GOOD - kubectl exec pattern (no port conflicts)
kubectl exec -n namespace deployment/service -- wget -q -O- http://localhost:port/health

# AVOID - port forward for setup verification (causes conflicts)
kubectl port-forward svc/service port:port &
curl localhost:port/health
kill $PORTFWD_PID
```

**Performance optimization guidelines**:
- Use `sleep 1` instead of `sleep 5` for most retry loops
- Remove sleeps after straightforward operations (port forward start)
- Reduce timeout values: 60s for pod readiness, 30s for API verification
- Question every sleep - many are unnecessary

**Race Condition Handling:**
Never use bare `kubectl wait` immediately after resource creation. Use retry loops:
```bash
# WRONG - fails if pod not scheduled yet
kubectl apply -f deployment.yaml
kubectl wait --for=condition=ready pod -l app=myapp --timeout=300s

# CORRECT - retry loop handles race condition
kubectl apply -f deployment.yaml
POD_READY=false
for i in {1..60}; do
  if kubectl wait --for=condition=ready pod -l app=myapp --timeout=5s 2>/dev/null; then
    echo "✅ Pod is ready!"
    POD_READY=true
    break
  fi
  sleep 1
done
if [ "$POD_READY" = false ]; then
  echo "❌ Pod failed to become ready after 60 seconds"
  kubectl logs -l app=myapp --tail=20  # Diagnostic info
  exit 1  # CRITICAL: Fail the test early
fi
```

### Eval Best Practices

**Realism:**
- No fake/obvious logs like "Memory usage stabilized at 800MB"
- No hints in filenames like "disk_consumer.py" - use realistic names like "training_pipeline.py"
- No error messages that give away it's simulated like "Simulated processing error"
- Use real-world scenarios: ML pipelines with checkpoint issues, database connection pools
- Resource naming should be neutral, not hint at the problem (avoid "broken-pod", "crashloop-app")

**Architecture:**
- Implement full architecture even if complex (e.g., use Loki for log aggregation, not simplified alternatives)
- Proper separation of concerns (app → file → Promtail → Loki → Holmes)
- **ALWAYS use Secrets for scripts**, not inline manifests or ConfigMaps
- Use minimal resource footprints (reduce memory/CPU for test services)

**Anti-Cheat Testing Guidelines:**
- **Prevent Domain Knowledge Cheats**: Use neutral, application-specific names instead of obvious technical terms
  - Example: "E-Commerce Platform Monitoring" not "Node Exporter Full"
  - Example: "Payment Service Dashboard" not "MySQL Error Dashboard"
  - Add source comments: `# Uses Node Exporter dashboard but renamed to prevent cheats`
- **Resource Naming Rules**: Avoid hint-giving names
  - Use realistic business context: "checkout-api", "user-service", "inventory-db" 
  - Avoid obvious problem indicators: "broken-pod" → "payment-service-1"
  - Test discovery ability, not pattern recognition
- **Prompt Design**: Don't give away solutions in prompts
  - BAD: "Find the node_pressure_cpu_waiting_seconds_total query"
  - GOOD: "Find the Prometheus query that monitors CPU pressure waiting time"
  - Test Holmes's search/analysis skills, not domain knowledge shortcuts

**Configuration:**
- Custom runbooks: Add `runbooks` field in test_case.yaml (`runbooks: {}` for empty catalog)
- Custom toolsets: Create separate `toolsets.yaml` file (never put in test_case.yaml)
- Toolset config must go under `config` field:
```yaml
toolsets:
  grafana/dashboards:
    enabled: true
    config:  # All toolset-specific config under 'config'
      api_url: http://localhost:10177
```

**Always run evals before submitting when possible:**
1. `poetry run pytest -k "test_name" --only-setup --no-cov` — verify setup
2. `poetry run pytest -k "test_name" --no-cov` — run full test
3. Verify cleanup: `kubectl get namespace app-NNN` should return NotFound

## Running Kubernetes Evals in the Claude Code Sandbox

The sandbox does not ship with a running Kubernetes cluster, kubectl, helm, or a
running docker daemon. To run k8s-based evals (e.g.
`tests/llm/fixtures/test_ask_holmes/227_count_configmaps_per_namespace/test_case.yaml`,
which uses `kubectl create namespace` / `kubectl create configmap` in `before_test`),
run `./scripts/setup-sandbox-k8s.sh`. It brings up **k3s in a container** (KIND does
NOT work — the inner node's systemd cannot mount `/sys/fs/cgroup/systemd` under the
sandbox's cgroup v1 and restart-loops with "Failed to mount API filesystems"),
installs `kubectl` / `helm` / a static `jq`, and patches two sandbox-specific
issues that would otherwise prevent any pod from starting:

- The sandbox MITMs HTTPS to public registries with an internal CA
  (`egress-gateway-ca-*.crt`, `swp-ca-*.crt`). k3s's containerd doesn't
  trust those, so the inner cluster can't pull `rancher/mirrored-pause`
  and every pod sandbox fails. Fix: bind-mount the host CA bundle into
  k3s and write a `registries.yaml` pointing containerd at it. The
  script also mirrors `docker.io` through `mirror.gcr.io` to dodge
  Docker Hub's 100/6h anonymous pull limit (the sandbox shares an
  outbound IP across sessions).
- The sandbox strips `CAP_SYS_RESOURCE` from our user. K3s's pause
  container has `oomScoreAdj: -998`; setting a negative `oom_score_adj`
  requires CAP_SYS_RESOURCE. runc's `nsexec` fails with `failed to
  update /proc/self/oom_score_adj: Permission denied`, the child dies,
  the parent gets `can't get final child's PID from pipe: EOF`. Fix:
  wrap `/bin/runc` inside the k3s container with a shim that uses `jq`
  to strip `.process.oomScoreAdj` from the OCI config before
  `runc create` runs.

Verified end-to-end with the `llm and regression` test suite against a
freshly-bootstrapped cluster under opus-4.6: **10 of 11 regression evals pass**
in ~256s wall at -n 4. The eleventh (`176_network_policy_blocking_traffic_no_skills`)
cannot pass here — see the limitations section. Skip via `-m "llm and regression and not network"`.

**LLM provider env (OpenRouter — Anthropic / OpenAI keys are not in env):**

Two non-obvious requirements for the classifier scoring step:

- `api_base: https://openrouter.ai/api/v1` is **required** on every `model_list.yaml`
  entry. The autoevals/braintrust correctness classifier bypasses litellm and calls
  the raw OpenAI SDK — without `api_base` it hits `api.openai.com` and 401s with the
  OpenRouter key.
- Always use the `openai/<provider>/<model>` prefix in the `model:` field — including
  for Anthropic models routed through OpenRouter. Two problems get solved at once:
  - litellm's native Anthropic provider appends `/v1/messages` to `api_base`, which
    doubles the path to `/api/v1/v1/messages` and 404s. Forcing the `openai/` prefix
    routes through litellm's OpenAI-compatible path (`/chat/completions`) instead,
    which OpenRouter speaks fine for all models.
  - The autoevals classifier passes the post-prefix string straight to OpenRouter as
    a model ID, and `openrouter/openai/gpt-4.1` is rejected as "not a valid model ID";
    `openai/gpt-4.1` is accepted.

```bash
cat > /tmp/model_list.yaml << 'EOF'
opus-4.6:
  model: openai/anthropic/claude-opus-4.6
  api_key: "{{ env.OPENROUTER_API_KEY }}"
  api_base: https://openrouter.ai/api/v1
gpt-4.1:
  model: openai/gpt-4.1
  api_key: "{{ env.OPENROUTER_API_KEY }}"
  api_base: https://openrouter.ai/api/v1
gpt-4.1-mini:
  model: openai/gpt-4.1-mini
  api_key: "{{ env.OPENROUTER_API_KEY }}"
  api_base: https://openrouter.ai/api/v1
EOF

# BRAINTRUST_API_KEY / BRAINTRUST_SERVICE_TOKEN must be UNSET — the read-only
# service token in this env refuses to create experiments and crashes the test.
unset BRAINTRUST_API_KEY
unset BRAINTRUST_SERVICE_TOKEN

export MODEL=gpt-4.1-mini MODEL_LIST_FILE_LOCATION=/tmp/model_list.yaml
export CLASSIFIER_MODEL=gpt-4.1 RUN_LIVE=true OPENAI_API_KEY=dummy
```

Then run the regression suite using the invocation that `setup-sandbox-k8s.sh`
prints on success. Wall time is ~65s for a single k8s eval with gpt-4.1-mini
(~$0.03 OpenRouter spend); opus-4.6 is ~90s per run. The full 11-eval
`llm and regression` suite under opus-4.6 runs in ~256s wall at -n 4 (~$3.30).
Budget accordingly when looping.

**What does NOT work in the sandbox:**

- `kind create cluster` — inner container restart-loops on cgroup v1 (systemd mount fails)
- k3s without the `oomScoreAdj` runc wrapper — pod sandboxes never start; runc's
  nsexec fails with `failed to update /proc/self/oom_score_adj: Permission denied`
  because `CAP_SYS_RESOURCE` is stripped
- k3s without the host-CA bind-mount + `registries.yaml` — image pulls fail because
  the sandbox MITMs HTTPS with a CA the k3s container doesn't trust
- NetworkPolicy enforcement, period — both k3s's built-in NP controller
  (kube-router) and Calico's felix fail because `ipset` is restricted in this
  kernel (`ipset list -name` returns "Kernel error received: Invalid argument").
  Calico's eBPF dataplane also doesn't help: `/sys/fs` isn't a shared mount,
  so the `mount-bpffs` init container can't run. The
  `176_network_policy_blocking_traffic_no_skills` eval is the only regression
  test affected — it asserts a NetworkPolicy actually blocks traffic, which we
  cannot enforce here. Skip it via `-m "llm and regression and not network"`
  or `--deselect`
- `BRAINTRUST_API_KEY` / `BRAINTRUST_SERVICE_TOKEN` set — unset both or the test crashes
- `model_list.yaml` entries without `api_base` — classifier 401s against `api.openai.com`
- `openrouter/openai/<model>` prefix in `model:` — OpenRouter rejects as invalid model ID
- `anthropic/<model>` prefix in `model:` — litellm's Anthropic provider hits
  `/api/v1/v1/messages` (doubled path) and 404s; use `openai/anthropic/<model>` instead
- Assuming Anthropic / OpenAI keys are present — only `OPENROUTER_API_KEY` is available

**Caveats and gotchas (things that DO work but with strings attached):**

- **`oomScoreAdj` rewrite is a behavioural fudge.** The wrapper rewrites
  `oomScoreAdj: -998` (set by k3s on the pause container and a few system pods like
  coredns/local-path-provisioner) to `0`. User pods are unaffected — kubelet derives
  their `oomScoreAdj` from QoS class (0 for Guaranteed, ~999 Burstable, ~1000
  BestEffort), all non-negative so the `s/:-[0-9]+/0/` regex never matches them.
  In practice this means: under sustained memory pressure on the k3s container,
  the kernel OOM killer may pick a pause/coredns container before a user
  container that it would normally protect, which can tear the pod sandbox out
  from under a workload. Doesn't happen at the ~20-pod scale the regression
  suite hits here, but be aware if you write a memory-stress eval, or an eval
  that asserts directly on `/proc/<pid>/oom_score_adj` of a system pod.
- **The runc wrapper survives `docker restart k3s-server` but NOT
  `docker rm`.** The wrapper lives on the container's writable layer at
  `/bin/runc` (with the original at `/bin/runc.real`). A fresh container
  starts with stock runc and pods get stuck again — re-run
  `./scripts/setup-sandbox-k8s.sh`; it's idempotent and reinstalls the
  wrapper. To verify the wrapper is active: `docker exec k3s-server head -3
  /bin/runc` should show the shebang + comment, not a binary.
- **The sandbox itself is reclaimed after a period of inactivity.** dockerd,
  the k3s container, kubectl, and helm all disappear when the session container
  is reclaimed. Re-run `./scripts/setup-sandbox-k8s.sh`. Anything you didn't
  commit and push is also gone — including local kubeconfigs and `model_list.yaml`.
- **`CAP_SYS_RESOURCE` removal blocks more than just `oomScoreAdj`.** Anything
  that tries to raise rlimits (`RLIMIT_NOFILE`, `RLIMIT_NPROC`) also fails. The
  hard ulimit for open files is pinned at 4096 and not even root can raise it;
  `prlimit --pid $$ --nofile=65536:65536` returns "Operation not permitted".
  In practice this is fine for the regression suite (small workloads), but a
  test that spins up a process requiring tens of thousands of fds (a DB under
  load, an HTTP load-generator, etc.) will hit it. Do NOT pass `--ulimit
  nofile=65536:65536` to the k3s `docker run` line — it will fail the container
  start with `error setting rlimit type 7: operation not permitted`.
- **Parallel pytest workers cause image-pull contention.** With `-n 4` the first
  setup wave creates ~25 pods simultaneously and containerd pulls all the
  unique images in parallel through the sandbox proxy. The first parallel
  regression run takes ~80s of setup (mostly image pull) per worker; a second
  run on the same cluster reuses the image cache and is much faster. If you're
  iterating, **do NOT** delete the k3s container between runs.
- **Docker Hub anonymous pull rate limit can fail eval setup.** The sandbox
  shares an outbound IP across sessions, so anonymous Docker Hub pulls are
  subject to the 100-pulls-per-6h limit (`429 Too Many Requests: toomanyrequests:
  You have reached your unauthenticated pull rate limit`). Symptoms: eval
  setup times out with pods stuck `ImagePullBackOff`. The setup script works
  around this by configuring containerd to pull `docker.io` images through
  **`mirror.gcr.io`** (Google's public pull-through cache for Docker Hub
  `library/*` and the k8s ecosystem) before falling back to
  `registry-1.docker.io`. The mirror has no rate limit and needs no
  credentials. If a workload pulls an image the mirror doesn't have and
  the fallback still 429s, additional mitigations:
  - Pre-pull the image on the host (`docker pull python:3.9-slim`) and
    side-load into k3s with `docker save ... | docker exec -i k3s-server
    ctr -n k8s.io images import -` (the same image-import gotcha applies —
    do them one at a time).
  - Add a Docker Hub auth block to `/tmp/k3s-output/registries.yaml`:
    ```yaml
    configs:
      "registry-1.docker.io":
        auth:
          username: <hub-user>
          password: <hub-pat>
        tls:
          ca_file: /etc/ssl/certs/ca-certificates.crt
    ```
  - Re-run after the 6h window expires.
- **Sandbox network policy controls outbound access.** All recipe pulls
  (`get.helm.sh`, `dl.k8s.io`, `kind.sigs.k8s.io`, `openrouter.ai`,
  `registry-1.docker.io`, `ghcr.io` Calico images, etc.) succeed under the
  default policy that was active during verification, but if a future
  environment uses a stricter policy some installs may fail. The k3s
  `registries.yaml` only covers `docker.io`, `registry.k8s.io`, and `quay.io`;
  if a workload pulls from another registry (e.g. `ghcr.io`) extend
  `registries.yaml` with the same `ca_file:` block for it.
- **Eval `before_test` scripts can `kubectl exec` pods you just created.** If
  your test_case.yaml does that, the inner pod has to actually run — make
  sure the wrapper is in place before launching pytest, not after.
- **`docker save` / `ctr images import` for side-loading images is unreliable**
  in this combination. Some images import cleanly, others fail with
  `content digest ...: not found` (saw this with the multi-image tar bundle).
  If you need to pre-seed an image, save and import one at a time. Pulling
  through containerd with the registries.yaml fix is more reliable than
  side-loading.
- **kubeconfig regenerates if the k3s container is removed and recreated.**
  Refresh `~/.kube/config` from `/tmp/k3s-output/kubeconfig.yaml` (or via
  `docker cp k3s-server:/output/kubeconfig.yaml ~/.kube/config`) after any
  container recreation, or kubectl will fail with "x509: certificate signed
  by unknown authority".
- **NetworkPolicy enforcement isn't recoverable in this sandbox.** Both
  approaches I tried bottom out on `ipset`: k3s's kube-router skips its NP
  controller at startup, and Calico's felix logs the same kernel error
  continuously. Calico's eBPF dataplane is blocked separately by `/sys/fs`
  not being a shared mount, so the `mount-bpffs` init container can't run.
  Don't waste cycles installing another CNI for NP support — it won't work
  until the sandbox kernel allows ipset/netlink-NFNL_SUBSYS_IPSET operations.

## Reading CodeRabbit Review Comments

In the sandbox environment, `gh` CLI is not available and the GitHub REST API will quickly rate-limit unauthenticated requests. Use the following approach:

1. **Find the PR number** via the GitHub API (unauthenticated, one call):
   ```bash
   curl -s "https://api.github.com/repos/HolmesGPT/holmesgpt/pulls?head=HolmesGPT:BRANCH_NAME&state=open" \
     | python3 -c "import sys,json; [print(f'PR #{p[\"number\"]}') for p in json.load(sys.stdin)]"
   ```
2. **Fetch comments with WebFetch** (not rate-limited):
   Use the `WebFetch` tool on `https://github.com/HolmesGPT/holmesgpt/pull/<NUMBER>` and ask it to extract all CodeRabbit comments, including file/line references, full text, and code suggestions.

**What does NOT work:**

- `gh` CLI — not installed in the sandbox
- Multiple `curl` calls to `api.github.com` — hits unauthenticated rate limits (60/hour) fast
- The local git proxy (`127.0.0.1`) — only supports git protocol, not the GitHub REST API

## Documentation Lookup

When asked about content from the HolmesGPT documentation website (https://holmesgpt.dev/), look in the local `docs/` directory:
- Python SDK examples: `docs/installation/python-installation.md`
- CLI installation: `docs/installation/cli-installation.md`
- Kubernetes deployment: `docs/installation/kubernetes-installation.md`
- Toolset documentation: `docs/data-sources/builtin-toolsets/`
- API reference: `docs/reference/`

## MkDocs Navigation

The docs site uses the `awesome-nav` plugin. Navigation is controlled by `.nav.yml` files in each `docs/` subdirectory, **not** by the `nav:` section in `mkdocs.yml`. When adding a new docs page, you must add it to the `.nav.yml` file in the corresponding directory (e.g., `docs/reference/.nav.yml` for reference pages).

## Updating References When Docs Pages Change

When you rename or move a docs page, or change a section heading (which changes its anchor), the URL changes. Before committing, `grep -rn` across the entire repo for the old page path and old anchor slug and update every hit. References can appear in:

- Other `docs/*.md` files (relative links like `[text](../path/page.md#anchor)`)
- Python source (user-facing error messages and hints, e.g. `holmes/utils/memory_limit.py`)
- `README.md` and top-level markdown
- Code comments

This applies to both `https://holmesgpt.dev/...` absolute URLs and repo-relative `.md` links.

## MkDocs Formatting Notes

When writing documentation in the `docs/` directory:

- **Lists after headers**: Always add a blank line between a header/bold text and a list, otherwise MkDocs won't render the list properly
  ```markdown
  **Good:**

  - item 1
  - item 2

  **Bad:**
  - item 1
  - item 2
  ```

- **Headers inside tabs**: Use **bold text** for section headings inside tabs, not markdown headers (`##`, `###`, etc.)

  **Why:** MkDocs Material font sizes make H2 (~25px) and H3 (~20px) visually larger than tab titles (~14px). When a header inside a tab is bigger than the tab title itself, it looks like it belongs outside/above the tabs, breaking the visual hierarchy.

  ```markdown
  <!-- GOOD: Bold text for sections inside tabs -->
  === "Tab Name"

      **Create the policy:**

      Instructions here...

      **Create the role:**

      More instructions...

  <!-- BAD: Headers inside tabs look like they're outside -->
  === "Tab Name"

      ### Create the policy

      Instructions here...
  ```

- **Avoid excessive headers**: Don't create a header for every small section. Headers should be used sparingly for major sections. For minor sections like test steps or examples, use bold text or combine content into a single code block with comments instead of separate headers.

  ```markdown
  <!-- BAD: Header for every test step -->
  ## Testing
  ### Test 1: Check Status
  ### Test 2: Check Logs
  ### Test 3: Health Check

  <!-- GOOD: Single section with combined content -->
  ## Testing the Connection

  ```bash
  # Check pod status
  kubectl get pods -n YOUR_NAMESPACE

  # Check logs
  kubectl logs -n YOUR_NAMESPACE

  # Health check
  curl http://localhost:8000/health
  ```
  ```

- **Don't describe Holmes's behavior**: In "Common Use Cases" sections, show only the example prompts. Don't explain what Holmes will do or list steps like "Holmes will: 1. Query X, 2. Analyze Y, 3. Return Z". Users will see this when they run it.

- **Skip Capabilities sections**: Don't list what a toolset/integration can do. Users discover capabilities by using Holmes. Feature lists become stale quickly.

- **Skip Security Best Practices sections**: Assume users understand basics like rotating credentials, using least privilege, and deleting local secrets. These sections add little value.

- **Consolidate troubleshooting commands**: Instead of separate headers for each troubleshooting scenario, use a single code block with comments:
  ```bash
  # Authentication errors - check if secret is mounted
  kubectl exec ...

  # Permission denied - verify roles
  gcloud projects get-iam-policy ...
  ```

- **Common Use Cases format**: Just example prompts, one per code block, no sub-headers, no explanations.

## Robusta Platform/API URLs in Docs

Whenever you reference `api.robusta.dev`, `platform.robusta.dev`, or `sp.robusta.dev` (the Supabase host) in a docs page, use the `robusta-region` custom fence so readers can pick US/EU/AP. **Never hardcode a single region** — the Robusta platform is hosted in multiple regions and a bare `api.robusta.dev` link silently breaks for EU/AP users.

The fence (defined in `docs/custom_fences.py`, registered in `mkdocs.yml`) takes a US URL as input and emits a three-tab picker with the domain rewritten per region. Pick the input shape that matches your context:

- **Plain URL or text** (renders as a code block per tab):

    ````markdown
    ```robusta-region
    https://api.robusta.dev/litellm/model_prices_and_context_window.json
    ```
    ````

- **Markdown link** (renders as a clickable link per tab — use for "go to platform.robusta.dev" style prose):

    ````markdown
    ```robusta-region
    [platform.robusta.dev](https://platform.robusta.dev/)
    ```
    ````

- **YAML or other code with `{lang=<name>}`** (applies the `language-<name>` class for syntax highlighting):

    ````markdown
    ```robusta-region {lang=yaml}
    holmes:
      additionalEnvVars:
        - name: ROBUSTA_API_ENDPOINT
          value: "https://api.robusta.dev"
    ```
    ````

Author the URL once using the US domain (`api.robusta.dev` / `platform.robusta.dev` / `sp.robusta.dev`); the fence handles the `.eu` / `.ap` rewrites. The set of rewritten hosts lives in `ROBUSTA_DOMAIN_RE` in `docs/custom_fences.py` — add a host there if you need another subdomain covered. If you add a new region or rename one, update `ROBUSTA_REGIONS` in the same file — that single change propagates to every page.

Existing usages (greppable starting points): `docs/ai-providers/robusta-ai.md`, `docs/installation/ui-installation.md`, `docs/reference/environment-variables.md`, `docs/reference/troubleshooting.md`, `docs/data-sources/builtin-toolsets/coralogix-logs.md`, `docs/data-sources/builtin-toolsets/kubernetes-mcp.md`.

More agent context in HolmesGPT/holmesgpt

3 other files this repository gives its agents.