ark-chainsaw-testing
Skill439 starsChanged 7 months agoApache-2.0View on GitHub
Run and write Ark Chainsaw tests with mock-llm. Use for running tests, debugging failures, or creating new e2e tests.
In this repository29 other files it gives its agents.
- ark/CLAUDE.mdCLAUDE.md
- CLAUDE.mdCLAUDE.md
- tests/CLAUDE.mdCLAUDE.md
- a2a-protocolSkill
- Ark AnalysisSkill
- ark-architectureSkill
- ark-controller-developmentSkill
- ark-dependabot-managementSkill
- ark-sdk-developmentSkill
- ark-setup-clusterSkill
- Ark Dashboard and UI TestingSkill
- ark-documentationSkill
- issue-creationSkill
- ark-issuesSkill
- multi-branch-cluster-setupSkill
- openspec-apply-changeSkill
- openspec-archive-changeSkill
- openspec-bulk-archive-changeSkill
- openspec-continue-changeSkill
- openspec-exploreSkill
- openspec-ff-changeSkill
- openspec-new-changeSkill
- openspec-onboardSkill
- openspec-sync-specsSkill
- openspec-verify-changeSkill
- ark-pentest-issue-resolverSkill
- ark-researchSkill
- slack-announcementSkill
- ark-vulnerability-fixerSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 1 is distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Run and write Chainsaw e2e tests for Ark resources.
- Very specific: line 46
Never: use kubectl wait --for=condition=Established to check CRD readiness.
- Very specific: line 52
Use a chainsaw assert with JMESPath instead — it polls gracefully and waits for the field to appear:
- Very specific: line 80
Never: use assert to wait for a Query to complete.
- Very specific: line 105
Use assert only after wait has confirmed the query is done — i.e., for post-completion validation of response fields.
- Very specific: line 135
Avoid using shell scripts with kubectl get + jq to validate resource state when chainsaw assertions can express the check declaratively.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Ark Chainsaw Testing
Run and write Chainsaw e2e tests for Ark resources.
Running Tests
# Run all standard tests
(cd tests && chainsaw test --selector 'standard')
# Run specific test
chainsaw test ./tests/query-parameter-ref --fail-fast
# Debug mode - keep resources on failure
chainsaw test ./tests/query-parameter-ref --skip-delete --pause-on-failureWriting Tests
Reference tests/CLAUDE.md for comprehensive patterns.
For a complete working example that shows the correct patterns for writing tests, see examples.md.
Test Structure
tests/my-test/
├── chainsaw-test.yaml # Test definition
├── mock-llm-values.yaml # Mock LLM config (if needed)
├── README.md # Required documentation
└── manifests/
├── a03-model.yaml # Model before Agent
├── a04-agent.yaml # Agent before Query
└── a05-query.yaml # Query lastAntipatterns
kubectl wait for CRD conditions
Never use kubectl wait --for=condition=Established to check CRD readiness. This command crashes with a type error when .status.conditions is nil (a known kubectl bug: kubernetes/kubernetes#66439):
error: .status.conditions accessor error: <nil> is of the type <nil>, expected []interface{}Use a chainsaw assert with JMESPath instead — it polls gracefully and waits for the field to appear:
# Bad - crashes if .status.conditions is nil
- script:
content: |
kubectl apply -f my-crd.yaml
kubectl wait --for=condition=Established crd/my-crd.example.com --timeout=60s
# Good - polls until condition appears, no nil crash
- script:
content: |
kubectl apply -f my-crd.yaml
- assert:
resource:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: my-crd.example.com
status:
(conditions[?type == 'Established']):
- status: "True"This pattern converts a brittle timeout into an explicit condition check. The same applies to any resource whose .status.conditions may start as nil.
assert instead of wait for Query completion
Never use assert to wait for a Query to complete. assert polls the API server repeatedly; wait uses a Kubernetes watch (event-driven), which is faster and cheaper:
# Bad - polls every few seconds, hammers the API server
- assert:
resource:
apiVersion: ark.mckinsey.com/v1alpha1
kind: Query
metadata:
name: test-query
status:
phase: done
# Good - uses a watch, fires once the condition is set
- wait:
apiVersion: ark.mckinsey.com/v1alpha1
kind: Query
name: test-query
timeout: 4m
for:
condition:
name: Completed
value: 'True'Use assert only after wait has confirmed the query is done — i.e., for post-completion validation of response fields. Keep these as separate steps so timing is explicit:
- name: wait-for-query-completion
try:
- wait:
apiVersion: ark.mckinsey.com/v1alpha1
kind: Query
name: test-query
timeout: 4m
for:
condition:
name: Completed
value: 'True'
- name: validate-response
try:
- assert:
resource:
apiVersion: ark.mckinsey.com/v1alpha1
kind: Query
metadata:
name: test-query
status:
(response != null): true
phase: doneShell scripts for condition checking
Avoid using shell scripts with kubectl get + jq to validate resource state when chainsaw assertions can express the check declaratively. Shell scripts fail immediately on unexpected nil values; assertions retry until the condition is met or the timeout expires.
# Bad - fails immediately if field is absent
- script:
content: |
val=$(kubectl get myresource foo -o jsonpath='{.status.phase}')
[ "$val" = "Ready" ] || exit 1
# Good - retries until condition is true
- assert:
resource:
apiVersion: example.com/v1
kind: MyResource
metadata:
name: foo
status:
phase: ReadyEnvironment Variables
For real LLM tests (not mock-llm):
export E2E_TEST_AZURE_OPENAI_KEY="your-key"
export E2E_TEST_AZURE_OPENAI_BASE_URL="your-endpoint"Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

