agentleFS
mckinseymckinsey/agents-at-scale-ark/.claude/skills/chainsaw/SKILL.md

ark-chainsaw-testing

Skill439 starsChanged 7 months agoApache-2.0View on GitHub

Run and write Ark Chainsaw tests with mock-llm. Use for running tests, debugging failures, or creating new e2e tests.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 1 is distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run and write Chainsaw e2e tests for Ark resources.

    line 8
  2. Very specific:

    Never: use kubectl wait --for=condition=Established to check CRD readiness.

    line 46
  3. Very specific:

    Use a chainsaw assert with JMESPath instead — it polls gracefully and waits for the field to appear:

    line 52
  4. Very specific:

    Never: use assert to wait for a Query to complete.

    line 80
  5. Very specific:

    Use assert only after wait has confirmed the query is done — i.e., for post-completion validation of response fields.

    line 105
  6. Very specific:

    Avoid using shell scripts with kubectl get + jq to validate resource state when chainsaw assertions can express the check declaratively.

    line 135

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Ark Chainsaw Testing

Run and write Chainsaw e2e tests for Ark resources.

Running Tests

# Run all standard tests
(cd tests && chainsaw test --selector 'standard')

# Run specific test
chainsaw test ./tests/query-parameter-ref --fail-fast

# Debug mode - keep resources on failure
chainsaw test ./tests/query-parameter-ref --skip-delete --pause-on-failure

Writing Tests

Reference tests/CLAUDE.md for comprehensive patterns.

For a complete working example that shows the correct patterns for writing tests, see examples.md.

Test Structure

tests/my-test/
├── chainsaw-test.yaml      # Test definition
├── mock-llm-values.yaml    # Mock LLM config (if needed)
├── README.md               # Required documentation
└── manifests/
    ├── a03-model.yaml      # Model before Agent
    ├── a04-agent.yaml      # Agent before Query
    └── a05-query.yaml      # Query last

Antipatterns

kubectl wait for CRD conditions

Never use kubectl wait --for=condition=Established to check CRD readiness. This command crashes with a type error when .status.conditions is nil (a known kubectl bug: kubernetes/kubernetes#66439):

error: .status.conditions accessor error: <nil> is of the type <nil>, expected []interface{}

Use a chainsaw assert with JMESPath instead — it polls gracefully and waits for the field to appear:

# Bad - crashes if .status.conditions is nil
- script:
    content: |
      kubectl apply -f my-crd.yaml
      kubectl wait --for=condition=Established crd/my-crd.example.com --timeout=60s

# Good - polls until condition appears, no nil crash
- script:
    content: |
      kubectl apply -f my-crd.yaml
- assert:
    resource:
      apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      metadata:
        name: my-crd.example.com
      status:
        (conditions[?type == 'Established']):
          - status: "True"

This pattern converts a brittle timeout into an explicit condition check. The same applies to any resource whose .status.conditions may start as nil.

assert instead of wait for Query completion

Never use assert to wait for a Query to complete. assert polls the API server repeatedly; wait uses a Kubernetes watch (event-driven), which is faster and cheaper:

# Bad - polls every few seconds, hammers the API server
- assert:
    resource:
      apiVersion: ark.mckinsey.com/v1alpha1
      kind: Query
      metadata:
        name: test-query
      status:
        phase: done

# Good - uses a watch, fires once the condition is set
- wait:
    apiVersion: ark.mckinsey.com/v1alpha1
    kind: Query
    name: test-query
    timeout: 4m
    for:
      condition:
        name: Completed
        value: 'True'

Use assert only after wait has confirmed the query is done — i.e., for post-completion validation of response fields. Keep these as separate steps so timing is explicit:

- name: wait-for-query-completion
  try:
  - wait:
      apiVersion: ark.mckinsey.com/v1alpha1
      kind: Query
      name: test-query
      timeout: 4m
      for:
        condition:
          name: Completed
          value: 'True'

- name: validate-response
  try:
  - assert:
      resource:
        apiVersion: ark.mckinsey.com/v1alpha1
        kind: Query
        metadata:
          name: test-query
        status:
          (response != null): true
          phase: done

Shell scripts for condition checking

Avoid using shell scripts with kubectl get + jq to validate resource state when chainsaw assertions can express the check declaratively. Shell scripts fail immediately on unexpected nil values; assertions retry until the condition is met or the timeout expires.

# Bad - fails immediately if field is absent
- script:
    content: |
      val=$(kubectl get myresource foo -o jsonpath='{.status.phase}')
      [ "$val" = "Ready" ] || exit 1

# Good - retries until condition is true
- assert:
    resource:
      apiVersion: example.com/v1
      kind: MyResource
      metadata:
        name: foo
      status:
        phase: Ready

Environment Variables

For real LLM tests (not mock-llm):

export E2E_TEST_AZURE_OPENAI_KEY="your-key"
export E2E_TEST_AZURE_OPENAI_BASE_URL="your-endpoint"

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.