Dingo — Agent Instructions

Project Overview

Dingo is a comprehensive AI data quality evaluation tool for ML practitioners, data engineers, and AI researchers. It systematically assesses training data, fine-tuning datasets, and production AI systems using rule-based, LLM-based, and agent-based evaluation methods.

Repository: https://github.com/MigoXLab/dingo PyPI: pip install dingo-python License: Apache 2.0

Tech Stack

Directory Structure

dingo/
├── AGENTS.md                ← this file (agent instructions)
├── setup.py                 ← package config (extras_require for optional deps)
├── mcp_server.py            ← MCP server entry point (legacy, use `dingo serve` instead)
├── requirements/
│   ├── runtime.txt          ← core dependencies (minimal)
│   ├── datasource.txt       ← optional datasource deps (S3, SQL, Parquet, etc.)
│   ├── optional.txt         ← heavy optional deps (torch, pyspark, etc.)
│   └── agent.txt            ← agent evaluation deps (langchain, tavily)
│
├── SKILL.md                 ← AI agent skill definition (symlink → clawhub/SKILL.md)
├── dingo/                   ← core Python package
│   ├── config/
│   │   └── input_args.py    ← InputArgs, EvalPiplineConfig, EvaluatorGroupConfig
│   ├── io/
│   │   ├── input/data.py    ← Data model (Pydantic, extra="allow")
│   │   └── output/          ← ResultInfo, EvalDetail, SummaryModel, BenchmarkReport
│   │       └── benchmark_report.py ← Multi-system/dataset comparison & aggregation
│   ├── data/
│   │   ├── datasource/      ← LocalDataSource, SQLDataSource, S3DataSource, HFDataSource
│   │   ├── dataset/         ← Dataset implementations per source
│   │   ├── searcher.py      ← Searcher protocol + SearchResult + registry (mock, ES)
│   │   └── converter/       ← Format converters (JSON, JSONL, CSV, Parquet, MinerU, etc.)
│   ├── model/
│   │   ├── model.py         ← Model registry (rule_register, llm_register)
│   │   ├── rule/            ← Rule-based evaluators (80+ built-in)
│   │   │   ├── base.py      ← BaseRule
│   │   │   ├── rule_common.py ← Common rules (text quality, format, PII, etc.)
│   │   │   ├── guobiao/
│   │   │   │   └── rule_tc609_quality.py ← TC609 quality metrics and placeholders
│   │   │   ├── rule_search_ranking.py ← IR ranking metrics (NDCG, MRR, Recall, Precision, MAP, HitRate)
│   │   │   └── utils/       ← Shared utilities (normalize, ngrams, etc.)
│   │   └── llm/             ← LLM-based evaluators
│   │       ├── base_openai.py ← BaseOpenAI (base class for all LLM evaluators)
│   │       ├── text_quality/  ← Text quality evaluators (V4, V5)
│   │       ├── rag/          ← RAG metrics (Faithfulness, Precision, Recall, etc.)
│   │       ├── llm_search_result_relevance.py ← Search result relevance (Exa-style pointwise)
│   │       ├── hhh/          ← 3H evaluators (Honest, Helpful, Harmless)
│   │       ├── compare/      ← Document comparison evaluators
│   │       └── agent/        ← Agent-based evaluators
│   │           ├── base_agent.py  ← BaseAgent
│   │           ├── tools/         ← Tool registry + implementations
│   │           ├── agent_fact_check.py
│   │           └── agent_hallucination.py
│   ├── exec/
│   │   ├── local.py         ← LocalExecutor (single machine, cross-layer conflict detection)
│   │   ├── spark.py         ← SparkExecutor (distributed)
│   │   └── retrieval.py     ← RetrievalExecutor (MTEB retrieval benchmarks)
│   ├── retrieval/            ← Retrieval evaluation module
│   │   ├── search_client.py ← SearchClient ABC + registry + PaperResult/SearchResponse
│   │   ├── backends/
│   │   │   └── agentic.py   ← AgenticSearchClient (local + public mode)
│   │   ├── mteb_adapter.py  ← SearchClientModel (MTEB SearchProtocol adapter)
│   │   └── eval_utils.py    ← normalize_title, resolve_hit, save_json
│   └── run/
│       └── cli.py           ← CLI entry point (subcommands: eval, eval-retrieval, info, serve)
│
├── examples/                ← Usage examples (SDK, CLI, various scenarios)
├── test/                    ← Test suite
│   ├── data/                ← Test data files
│   ├── env/                 ← Test environment configs
│   └── scripts/             ← Test scripts (pytest)
└── docs/                    ← Documentation

Core Concepts

Data Flow

Data Input → Interface (SDK/CLI/MCP) → Datasource → Dataset → Converter → Evaluator → Executor → Report

Registration System

All evaluators use decorator-based registration:

# Rule evaluator
@Model.rule_register('QUALITY_BAD_COMPLETENESS', ['default', 'pretrain'])
class MyRule(BaseRule):
    @classmethod
    def eval(cls, input_data: Data) -> EvalDetail: ...

# LLM evaluator
@Model.llm_register('MyLLMEvaluator')
class MyLLMEvaluator(BaseOpenAI):
    prompt = "..."
    @classmethod
    def build_messages(cls, input_data: Data) -> List: ...

# Agent evaluator
@Model.llm_register('MyAgent')
class MyAgent(BaseAgent):
    available_tools = ["tavily_search"]
    @classmethod
    def eval(cls, input_data: Data) -> EvalDetail: ...

Data Model

Data uses extra = "allow" — any field can be set dynamically:

class Data(BaseModel):
    class Config:
        extra = "allow"

Common fields: data_id, prompt, content, image, context, raw_data, reference, user_input, response, retrieved_contexts.

InputArgs Configuration

input_data = {
    "input_path": "data.jsonl",
    "dataset": {"source": "local", "format": "jsonl"},
    "executor": {"max_workers": 4, "batch_size": 10, "result_save": {"bad": True, "good": True}},
    "evaluator": [
        {
            "fields": {"content": "text_field", "prompt": "question_field"},
            "evals": [
                {"name": "RuleAbnormalChar"},
                {"name": "LLMTextQualityV5", "config": {"key": "...", "model": "gpt-4o", "api_url": "..."}}
            ]
        }
    ]
}

Supported Data Formats

MinerU formats automatically flatten blocks into Data objects, preserving all original fields (type, bbox, page_idx, etc.) via Pydantic extra="allow". Use dataset.mineru_config.include_types to filter by block type.

Execution Modes

Optional Dependencies (extras_require)

pip install dingo-python                # Core (includes all datasource support)
pip install "dingo-python[hhem]"        # + HHEM hallucination detection (transformers + torch)
pip install "dingo-python[agent]"       # + Agent-based evaluation (langchain)
pip install "dingo-python[retrieval]"   # + Retrieval benchmark evaluation (mteb, tqdm)
pip install "dingo-python[all]"         # + Everything

Development Conventions

General

• Python 3.10+ required
• Code comments in English or Chinese (both acceptable)
• All evaluators must return EvalDetail with metric, status, label, reason
• Use lazy imports for optional heavy dependencies (torch, transformers, pyarrow, etc.)
• Never hardcode API keys; use environment variables or config parameters

Adding a New Rule Evaluator

1. Create class in dingo/model/rule/rule_common.py (or a new file under dingo/model/rule/)
2. Inherit from BaseRule
3. Decorate with @Model.rule_register(metric_type, groups)
4. Implement eval(cls, input_data: Data) -> EvalDetail as classmethod
5. Set _required_fields to declare needed input fields

Adding a New LLM Evaluator

1. Create class in dingo/model/llm/ (appropriate subdirectory)
2. Inherit from BaseOpenAI
3. Decorate with @Model.llm_register('EvaluatorName')
4. Set prompt attribute or override build_messages()
5. Override parse_result() if custom response parsing needed

Adding a New Datasource

1. Create datasource class in dingo/data/datasource/
2. Create dataset class in dingo/data/dataset/
3. Register in the respective __init__.py
4. Use lazy imports if new dependencies required
5. Add dependency to requirements/runtime.txt (core) or setup.py extras (heavy/optional)

Testing

# Run all tests
pytest test/scripts --ignore=test/scripts/data

# Run specific test
pytest test/scripts/model/llm/test_rag.py -v

# Integration tests (CLI)
dingo eval --input .github/env/local_plaintext.json
dingo eval --input .github/env/local_json.json --json

CLI Reference

Dingo provides a dingo CLI command (installed via pip install dingo-python):

# Run evaluation (primary command)
dingo eval --input config.json            # Human-readable output
dingo eval --input config.json --json     # JSON output (for agents/automation)

# Run retrieval benchmark evaluation
dingo eval-retrieval --backend agentic --tasks SciFact \
  --api-url https://api.sciverse.space --api-token <token> --limit 100
dingo eval-retrieval --tasks SciFact LitSearch --max-queries 50 --max-workers 4 \
  --api-url http://localhost:8080

# List available evaluators, groups
dingo info                                # Show all (rules, LLM, groups)
dingo info --rules                        # Rule evaluators only
dingo info --llm                          # LLM evaluators only
dingo info --groups                       # Rule groups only
dingo info --json                         # JSON output
dingo info --count                        # Metric counts only (rules / llm / groups / total_metrics)
dingo info --json --count                  # JSON with top-level "counts" plus full lists

# Start MCP server (for AI agent integration)
dingo serve                               # SSE transport on 0.0.0.0:8000
dingo serve --port 9000                   # Custom port
dingo serve --transport stdio             # stdio transport (for local agent)

# Backward compatibility (no subcommand)
dingo --input config.json                 # Same as `dingo eval --input config.json`
python -m dingo.run.cli --input config.json

Version Conventions

• Version defined in setup.py (version="2.0.0")
• Follow semantic versioning (MAJOR.MINOR.PATCH)

MCP Server

Start via CLI: dingo serve (SSE default) or dingo serve --transport stdio Legacy entry point: mcp_server.py Framework: FastMCP

Available MCP Tools

MCP Environment Variables

Config Maintenance Rules

When these events occur, update the corresponding files:

After completing a feature, check if any of the above need updating.