dingo / rules

---
description: Guide for developing new evaluators (rules, LLMs, agents)
globs:
  - dingo/model/**/*.py
---

# Evaluator Development Guide

## Architecture

```
Model (registry)
├── Rule evaluators  → @Model.rule_register(metric_type, groups)
│   └── BaseRule.eval(input_data) → EvalDetail
├── LLM evaluators   → @Model.llm_register(name)
│   └── BaseOpenAI.eval(input_data) → EvalDetail
└── Agent evaluators → @Model.llm_register(name)
    └── BaseAgent.eval(input_data) → EvalDetail
```

## Key Files

| File | Purpose |
|------|---------|
| `dingo/model/model.py` | `Model` class with `rule_register`, `llm_register`, `load_model` |
| `dingo/model/rule/base.py` | `BaseRule` base class |
| `dingo/model/llm/base_openai.py` | `BaseOpenAI` base class for LLM evaluators |
| `dingo/model/llm/agent/base_agent.py` | `BaseAgent` base class for agent evaluators |
| `dingo/io/input/data.py` | `Data` model (input to evaluators) |
| `dingo/io/output/eval_detail.py` | `EvalDetail` model (output from evaluators) |

## Registration Groups

Rules belong to groups that determine when they run:

- `default` — runs in default evaluation
- `pretrain` — pre-training data quality
- `benchmark` — benchmark evaluation
- `sft` — supervised fine-tuning data
- `rag` — RAG system evaluation
- `hallucination` — hallucination detection

## EvalDetail Contract

Every evaluator must return `EvalDetail` with:

| Field | Type | Description |
|-------|------|-------------|
| `metric` | str | Evaluator class name (`cls.__name__`) |
| `status` | bool | `True` = issue found, `False` = no issue |
| `label` | List[str] | Quality labels (e.g., `['QUALITY_GOOD']` or `['QUALITY_BAD_COMPLETENESS.RuleName']`) |
| `reason` | List[str] | Human-readable explanation of the finding |
| `score` | float | Optional numeric score (0.0–1.0) |
| `extra` | Dict | Optional extra metadata |

## Data Field Access

Since `Data` uses `extra = "allow"`, always access non-standard fields safely:

```python
# Safe access patterns
raw_data = getattr(input_data, 'raw_data', {})
context = getattr(input_data, 'context', None)
reference = getattr(input_data, 'reference', '')

# For RAG evaluators, common field access pattern
question = input_data.prompt or raw_data.get("question", "")
answer = input_data.content or raw_data.get("answer", "")
```

## Multi-Field Evaluation

Evaluators receive data after field mapping. The `fields` config maps source fields to standard Data fields:

```json
{
  "fields": {"content": "description", "prompt": "title"},
  "evals": [{"name": "RuleAbnormalChar"}]
}
```

The evaluator sees `input_data.content` = value of source field `description`.

## Tool Registration (for Agents)

```python
from dingo.model.llm.agent.tools.tool_registry import tool_register
from dingo.model.llm.agent.tools.base_tool import BaseTool

@tool_register
class MyTool(BaseTool):
    name = "my_tool"
    description = "What this tool does"

    def execute(self, **kwargs):
        # tool logic
        return result
```