Writing AI Eval Tests (test/ai_evals/)

Guardrails & Forbidden Actions

[!CAUTION]

1. CRITICAL RULE — SINGLE V2 TEST ONLY: Always write only one V2 evaluation test per eval (enabledFeatures: ['DevToolsAiV2Architecture']) using the canonical template below. NEVER create V1 tests or V1+V2 pairs.
2. CRITICAL RULE — NEVER SET OR PASS CONTEXT: NEVER call findAndSetContext (or any panel/UI context-selection helper) and NEVER pass context to evalTask.executePrompt(...), even when initial_context is present in task.textproto.
3. CRITICAL RULE — MINIMAL FILE READS & NO UNREQUESTED COMMANDS:
   • Only read task.textproto, agent_prompt.md, and BUILD.gn files. Do NOT read ai_eval-helpers.ts, ai-eval-state-provider.ts, base-apps/, front_end/, or legacy tests that use findAndSetContext.
   • Do NOT run git, autoninja, ./dt lint, or ./dt test unless the user explicitly asks to build or run the test.

Fast 2-Step Workflow (All Domains)

Step 1: Read Only Required Metadata & BUILD.gn in One Parallel Turn

Call view_file in parallel on:

1. task.textproto and agent_prompt.md for <eval-id>:
   • Check flat path test/ai_evals/eval_data/evals/<eval-id>/ (evalFolder = '') or nested path test/ai_evals/eval_data/evals/<eval-folder>/<eval-id>/ (evalFolder = '<eval-folder>', e.g. 'network').
   • From agent_prompt.md: extract the trimmed prompt string <agent_prompt>.
   • From task.textproto: infer <domain> in priority order:
     1. Explicit user instruction or <eval-folder> subfolder name.
     2. Domain tag in tags: "performance", "network", "elements", "sources", "console", "accessibility", "storage" (ignore generic tags like "cd4h" or "bugfix", and prefer panel domain tags over sub-topic tags like "cls").
     3. initial_context.panel fallback (for folder naming only — never set context in code):
        • NETWORK_REQUEST $\rightarrow$ network
        • PERFORMANCE $\rightarrow$ performance
        • DOM_NODE / ELEMENTS $\rightarrow$ elements
        • SOURCE_FILE / SOURCES $\rightarrow$ sources
        • CONSOLE $\rightarrow$ console
        • ACCESSIBILITY $\rightarrow$ accessibility
        • STORAGE / APPLICATION $\rightarrow$ storage
   • Note: setupAndRunEvalApp parses base_app and initial_url from task.textproto automatically at runtime—do not hardcode them.
2. test/ai_evals/BUILD.gn (and test/ai_evals/<domain>/BUILD.gn if <domain> is already known).

Step 2: Write Test and Update BUILD.gn in One Parallel Turn

In a single parallel tool-call turn:

1. Create test/ai_evals/<domain>/<eval-id>.test.ts (write_to_file) using this exact domain-agnostic V2 template (<Domain> is capitalized <domain>, e.g. Performance, Network, Elements, Sources, Console, Accessibility, Storage):

// Copyright 2026 The Chromium Authors
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import {assert} from 'chai';

import {
  setupAndRunEvalApp,
  stopEvalApp,
} from '../helpers/ai_eval-helpers.js';
import {describe, it, setup} from '../mocha-interface.js';

describe('<Domain> AI Eval: <eval-id>', function() {
  this.timeout(180_000);

  setup({
    dockingMode: 'undocked',
    enabledFeatures: ['DevToolsAiV2Architecture'],
    devToolsSettings: {
      'ai-assistance-enabled': true,
      'ai-assistance-v2-opt-in-change-dialog-seen': true,
    },
    evalConfig: {
      type: '<domain>',
      label: '<eval-id>',
      taskId: '<eval-id>',
    },
  });

  it('launches app, navigates to initial URL, and executes prompt', async ({inspectedPage, evalTask}) => {
    const app = await setupAndRunEvalApp('<evalFolder>', '<eval-id>');
    try {
      assert.isDefined(app.config.initialUrl, 'Expected initial_url to be defined in task.textproto');

      await inspectedPage.page.goto(app.config.initialUrl, {waitUntil: 'networkidle0'});

      await evalTask.executePrompt({
        prompts: ['<agent_prompt>'],
      });
    } finally {
      await stopEvalApp(app);
    }
  });
});

2. Update BUILD.gn:
   • If test/ai_evals/<domain>/BUILD.gn exists: Add "<eval-id>.test.ts" in alphabetical order to sources (replace_file_content).
   • If <domain> is a new domain folder: Create test/ai_evals/<domain>/BUILD.gn (write_to_file) and add "<domain>" to deps in both group("ai_evals") and generated_file("test_inputs") in test/ai_evals/BUILD.gn (replace_file_content):

     # Copyright 2026 The Chromium Authors
     # Use of this source code is governed by a BSD-style license that can be
     # found in the LICENSE file.
     
     import("../../../scripts/build/typescript/typescript.gni")
     
     node_ts_library("<domain>") {
       sources = [ "<eval-id>.test.ts" ]
     
       ts_deps = [
         "..:harness_library",
         "../../e2e/helpers",
         "../helpers",
       ]
     }