Skill: Run Automated Tests

Purpose

Work out how the target repository expects automated tests to be run (commands, frameworks, prerequisites, and scope), then run the best-matching test suite under a safety-first interaction policy.

Core Objective

Primary goal: produce test execution results through evidence-based command selection and safety guardrails.

Success criteria (all must hold):

1. ✅ Test plan discovered: the evidence source is identified (documentation, CI configuration, or build manifest)
2. ✅ Command selected: the appropriate test command is chosen for the mode (fast/ci/full) and the constraints
3. ✅ User confirmation obtained: approval received before installing dependencies, using the network, or starting services
4. ✅ Tests executed: the command was run with its output and exit code captured
5. ✅ Results summarized: a test plan summary with evidence, commands, execution status, and failures (if any)

Acceptance test: can a developer reproduce the test execution by following the test plan summary, with no extra context?

Scope Boundaries

This skill owns:

• Discovering the test command from repository evidence (documentation, CI, build manifests)
• Selecting the appropriate test command for the mode and the constraints
• Executing tests behind safety guardrails and user confirmation
• Summarizing test results with evidence and failure diagnostics

This skill does not own:

• Test quality assessment or coverage analysis (use review-testing)
• Fixing failing tests or debugging test failures (use orchestrate-repair-loop)
• Writing new tests or test infrastructure (use the development skills)
• Reviewing test code against best practices (use review-testing)

Handoff point: once the tests finish (pass or fail), hand off to orchestrate-repair-loop to fix the failures, or to review-testing for a quality assessment.

Use Cases

• You cloned a repository and want the right test command without guessing.
• The repository has several test layers (unit/integration/e2e) and you need a safe default run plan.
• CI failed and you want to reproduce it locally by running the same commands the workflow uses.

Behavior

1. Establish scope and constraints (ask when unclear)
   • With nothing specified by the user, default to a fast, local, non-destructive run:
     • Unit tests only, no external services, no Docker, no network-dependent setup.
   • Where it matters, have the user pick a mode:
     • fast: unit tests only, minimal setup.
     • ci: mirror the CI workflow commands as closely as possible.
     • full: include integration/e2e tests and service dependencies.
   • Ask whether Docker is allowed, whether network access is allowed, and whether dependency installation is allowed.
2. Discover the test plan (evidence-based)
   • If CLAUDE.md or .ai-cortex/config.yaml exists, prefer the test_command recorded there; otherwise discover it from the sources below. See docs/guides/project-config.md.
   • Read these sources in order; stop early once a clear, unambiguous test command turns up:
     • README.md, CONTRIBUTING.md, TESTING.md, docs/testing*, Makefile
     • CI configuration: .github/workflows/*.yml, .gitlab-ci.yml, azure-pipelines.yml, Jenkinsfile
     • Build manifests: package.json, pyproject.toml, setup.cfg, tox.ini, go.mod, pom.xml, build.gradle*, *.csproj, Cargo.toml
   • Identify:
     • The primary test entry point (npm test, pnpm test, yarn test, pytest, tox, go test, dotnet test, mvn test, gradle test, cargo test, and so on)
     • Test layers and markers (unit, integration, e2e)
     • Environment prerequisites (DB, Redis, Docker Compose, required environment variables, secrets)
     • How CI sets up its dependencies (services, caches, artifacts)
   • Prefer an explicit statement in the documentation or CI over heuristic inference.
3. Choose the execution plan
   • In ci mode: derive the run sequence from the repository's CI workflow steps (closest match).
   • In fast mode: pick the most direct unit test command, the one with the fewest prerequisites.
   • With several stacks present (backend + frontend, say), suggest running each stack separately in a fixed order.
   • If the plan needs a dependency install or a service start, ask for confirmation before continuing.
4. Execute with guardrails
   • Always print the exact command you are about to run, before running it.
   • Use a working directory rooted at the target repository (default .).
   • Read the execution log or framework report before deciding whether the selected scope passed; exit code zero alone is insufficient.
   • Report passed, failed, and skipped case counts when the framework supplies them. For check scripts without case counts, retain output identifying the checks actually performed and their outcomes; do not invent counts or require a new report format.
   • A selected test scope with zero executed cases or all cases skipped is not a pass. If the output does not establish what executed or whether it succeeded, report it as unverified and identify the missing evidence.
   • Distinguish a build or environment failure from a test assertion failure; both prevent claiming the selected scope passed.
   • Capture and summarize failures:
     • The first failing command and its exit code
     • The most relevant error excerpt
     • Next actions (missing toolchain, missing environment variable, service not running, and so on)
   • Avoid destructive operations:
     • Do not run rm -rf, git clean -fdx, docker system prune, or database drop/migration commands without the user's explicit approval.
   • If the repository needs secrets, do not ask the user to paste them into the chat. Use a .env file, a secret manager, or the documented local development flow.

Input & Output

Input

• Target repository path (default .).
• Mode: fast (default), ci, or full.
• Constraints: dependency install allowed (yes/no), network allowed (yes/no), Docker allowed (yes/no).

Output

• A short "test plan summary" containing:
  • Evidence: which files/paths informed the plan
  • The selected commands (in order)
  • Assumptions and prerequisites
  • What was executed and what was skipped (and why)
  • Execution evidence: available case counts or check outcomes, with the log/report location; explicitly identify zero execution, all-skipped, or unverified results
• Enough of the command log to debug a failure (do not dump extremely long logs unless asked).

Restrictions

Hard Boundaries

• Do not invent a test command when evidence exists (documentation/CI takes precedence).
• Do not install dependencies, run Docker, or start external services without confirmation.
• Do not modify repository files unless the user explicitly asks (exception: generating a report file when the user asks for an artifact).
• Do not leak secrets; do not request sensitive credentials in the chat.

Skill Boundaries (avoid overlap)

Do not do these (other skills handle them):

• Test quality assessment: judging test coverage, test design, or testing best practices → use review-testing
• Fixing test failures: debugging failing tests, repairing broken test code, or root-causing → use orchestrate-repair-loop
• Writing tests: creating new test cases, test infrastructure, or a test framework → use the development/implementation skills
• Code review: reviewing test code for quality, maintainability, or best practices → use review-testing
• Repository analysis: full codebase structure analysis or architecture review → use review-codebase

When to stop and hand off:

• Tests fail and the user asks "why?" or "how do I fix it?" → hand off to orchestrate-repair-loop for debugging and repair
• The user asks "are these tests any good?" or "what is our coverage?" → hand off to review-testing for a quality assessment
• The user asks "can you write tests for X?" → hand off to the development workflow for test implementation
• Tests pass and the user asks "what should we test next?" → hand the test strategy suggestion to review-testing

Self-Check

Core success criteria (all must hold)

• Test plan discovered: the evidence source is identified (documentation, CI configuration, or build manifest)
• Command selected: the appropriate test command is chosen for the mode (fast/ci/full) and the constraints
• User confirmation obtained: approval received before installing dependencies, using the network, or starting services
• Tests executed: the command was run with its output and exit code captured
• Results summarized: a test plan summary with evidence, commands, execution status, and failures (if any)

Process quality checks

• Evidence-based selection: did I identify at least one authoritative source for the test command (a doc file, a CI workflow, or a build manifest)?
• Safety guardrails applied: did I ask for confirmation before installing dependencies, using the network, starting Docker/services, or any state-changing action?
• Commands printed: did I print the exact command before running it?
• Failures diagnosed: when tests failed, did I give the first failing command, the exit code, and a likely root-cause category?
• Execution evidenced: the result uses the log/report, including available counts or check outcomes; zero execution, all-skipped, and unverified results were not presented as a pass
• No destructive operations: did I avoid running destructive commands (rm -rf, git clean, docker system prune, database drops) without explicit approval?
• No secret leakage: did I avoid asking for sensitive credentials in the conversation, using a .env file or the documented local development flow instead?

Acceptance test

Can a developer reproduce the test execution by following the test plan summary, with no extra context?

If no: the test plan summary is incomplete. Add the missing evidence, commands, or prerequisites.

If yes: the skill run is complete. Continue to the handoff if one is needed.

Examples

Example 1: a JavaScript repo with package.json

User: "run the tests for this repository."

Agent:

1. Check the package.json scripts and .github/workflows/*.
2. Settle on fast mode and propose:
   • npm test (or pnpm test / yarn test if the repository standardizes on one)
3. Ask: "install dependencies (npm ci) and allow network access?"
4. Run:
   • npm ci
   • npm test
5. Summarize the results and point at the failing test output, if any.

Example 2 (edge case): a monorepo that needs Docker for integration tests

User: "mirror CI locally."

Agent:

1. Parse .github/workflows/ci.yml and identify the separate jobs:
   • Backend unit tests
   • Frontend tests
   • Integration tests with docker compose
2. Ask for confirmation:
   • Docker allowed
   • Network allowed
   • Which jobs to run (all of them, or only the failing one)
3. Execute in a controlled order:
   • Install deps per job
   • Run unit tests first
   • Start the services for the integration tests
4. If the integration tests fail, summarize:
   • Service health/port conflicts
   • Missing environment variables
   • How the CI configuration differs from the local one

Example 3: successful command without a verified test pass

npm test exits zero but reports twelve skipped tests and none executed. Report the scope as skipped, retain the output, and investigate the selection or prerequisites before claiming a pass. If the log is incomplete and execution cannot be established, report unverified instead. A repository check script that identifies three completed checks and their successful outcomes can supply execution evidence without test-case counts.