agentleFS
maksutovicmaksutovic/joycraft/.github/skills/joycraft-implement-level5/SKILL.md

joycraft-implement-level5

Skill15 starsChanged 51 days agoMITView on GitHub

Invoked by tune's roadmap or the human opting into Level 5 — set up the autofix loop, holdout scenario testing, and scenario evolution from specs

  • Reads credentials
  • Commits and pushes

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 5 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Project must be initialized: Look for docs/.joycraft/manifest.json or docs/.joycraft/local/manifest.json.

    line 16
  2. Very specific:

    Project should be at Level 4: Check docs/joycraft-assessment.md if it exists.

    line 17
  3. Distinctive:

    If prerequisites aren't met, explain what's needed and stop.

    line 20
  4. Distinctive:

    Ask these questions one at a time:

    line 36
  5. Distinctive:

    Run the CLI command with the gathered configuration:

    line 66
  6. Very specific:

    Check scenario templates were copied: Verify the scenarios repo has example-scenario.test.ts (renamed from the .template starter), …

    line 116
  7. Distinctive:

    If the project's AGENTS.md doesn't already have an "External Validation" section, add one:

    line 121
  8. Distinctive:

    If the user wants to test the loop:

    line 136
  9. Very specific:

    Update docs/joycraft-assessment.md if it exists — set the Level 5 score to reflect the new setup.

    line 166

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

At skill entry, run node docs/.joycraft/check.mjs check --json --session <session-id> once, reusing JOYCRAFT_SESSION_ID if supplied or one ID chosen for this conversation. If the checker is missing or fails, continue the requested skill quietly without retrying setup. Offer updates only for display: true; current, postponed, off, and unknown stay quiet. If automaticUpdate is present, finish this skill, then run its exact command once from the project root: the candidate must verify every safety gate before applying. Other updates require approval. After an offer or automatic attempt, record it with node docs/.joycraft/check.mjs acknowledge <available-version> --session <session-id>. Apply updates only at this workflow boundary; after success, reinvoke the skill or restart the session to load changed instructions.

Implement Level 5 — Autonomous Development Loop

You are guiding the user through setting up Level 5: the autonomous feedback loop where specs go in, validated software comes out. This is a one-time setup that installs workflows, creates a scenarios repo, and configures the autofix loop.

Before You Begin

Check prerequisites:

  1. Project must be initialized. Look for docs/.joycraft/manifest.json or docs/.joycraft/local/manifest.json. Older installs may still have state at docs/.joycraft/state.json, .claude/.joycraft/state.json, or .joycraft-version at the repo root. If none exist, tell the user to run npx joycraft@latest init first.
  2. Project should be at Level 4. Check docs/joycraft-assessment.md if it exists. If the project hasn't been assessed yet, suggest running /joycraft-tune first. But don't block — the user may know they're ready.
  3. Git repo with GitHub remote. This setup requires GitHub Actions. Check for .git/ and a GitHub remote.

If prerequisites aren't met, explain what's needed and stop.

Step 1: Explain What Level 5 Means

Tell the user:

Level 5 is the autonomous loop. When you push specs, three things happen automatically:

  1. Scenario evolution — A separate AI agent reads your specs and writes holdout tests in a private scenarios repo. These tests are invisible to your coding agent.
  2. Autofix — When CI fails on a PR, Claude Code automatically attempts a fix (up to 3 times).
  3. Holdout validation — When CI passes, your scenarios repo runs behavioral tests against the PR. Results post as PR comments.

The key insight: your coding agent never sees the scenario tests. This prevents it from gaming the test suite — like a validation set in machine learning.

Step 2: Gather Configuration

Ask these questions one at a time:

Question 1: Scenarios repo name

What should we call your scenarios repo? It'll be a private repo that holds your holdout tests.

Default: {current-repo-name}-scenarios

Accept the default or the user's choice.

Question 2: GitHub App

Level 5 needs a GitHub App to provide a separate identity for autofix pushes (this avoids GitHub's anti-recursion protection). Creating one takes about 2 minutes:

  1. Go to https://github.com/settings/apps/new
  2. Give it a name (e.g., "My Project Autofix")
  3. Uncheck "Webhook > Active" (not needed)
  4. Under Repository permissions, set:
    • Contents: Read & Write
    • Pull requests: Read & Write
    • Actions: Read & Write
  5. Click Create GitHub App
  6. Note the App ID from the settings page
  7. Scroll to Private keys > click Generate a private key > save the .pem file
  8. Click Install App in the left sidebar > install it on your repo

What's your App ID?

Step 3: Run init-autofix

Run the CLI command with the gathered configuration:

npx joycraft@latest init-autofix --scenarios-repo {name} --app-id {id}

Review the output with the user. Confirm files were created.

Step 4: Walk Through Secret Configuration

Guide the user step by step:

4a: Add Secrets to Main Repo

You should already have the .pem file from when you created the app in Step 2.

Go to your repo's Settings > Secrets and variables > Actions, and add:

  • JOYCRAFT_APP_PRIVATE_KEY — paste the contents of your .pem file
  • ANTHROPIC_API_KEY — your Anthropic API key

4b: Create the Scenarios Repo

Create the private scenarios repo:

gh repo create {scenarios-repo-name} --private

Then copy the scenario templates into it. The starter ships as example-scenario.test.ts.template (the .template suffix keeps it out of the main project's test/lint/build globs); rename it to .test.ts once it's in the holdout repo so Vitest discovers it:

cp -r docs/templates/scenarios/* ../{scenarios-repo-name}/
cd ../{scenarios-repo-name}
mv example-scenario.test.ts.template example-scenario.test.ts
git add -A && git commit -m "init: scaffold scenarios repo from Joycraft"
git push

4c: Add Secrets to Scenarios Repo

The scenarios repo also needs the App private key:

  • JOYCRAFT_APP_PRIVATE_KEY — same .pem file as the main repo
  • ANTHROPIC_API_KEY — same key (needed for scenario generation)

Step 5: Verify Setup

Help the user verify everything is wired correctly:

  1. Check workflow files exist: ls .github/workflows/autofix.yml .github/workflows/scenarios-dispatch.yml .github/workflows/spec-dispatch.yml .github/workflows/scenarios-rerun.yml
  2. Check scenario templates were copied: Verify the scenarios repo has example-scenario.test.ts (renamed from the .template starter), workflows/run.yml, workflows/generate.yml, prompts/scenario-agent.md
  3. Check the App ID is correct in the workflow files (not still a placeholder)

Step 6: Update AGENTS.md

If the project's AGENTS.md doesn't already have an "External Validation" section, add one:

External Validation

This project uses holdout scenario tests in a separate private repo.

NEVER

  • Access, read, or reference the scenarios repo
  • Mention scenario test names or contents
  • Modify the scenarios dispatch workflow to leak test information

The scenarios repo is deliberately invisible to you. This is the holdout guarantee.

Step 7: First Test (Optional)

If the user wants to test the loop:

Want to do a quick test? Here's how:

  1. Write a simple spec in docs/features/<slug>/specs/ and push to main — this triggers scenario generation
  2. Create a PR with a small change — when CI passes, scenarios will run
  3. Watch for the scenario test results as a PR comment

Or deliberately break something in a PR to test the autofix loop.

Step 8: Summary

Print a summary of what was set up:

Level 5 is live. Here's what's running:

TriggerWhat Happens
Push specs to docs/features/<slug>/specs/Scenario agent writes holdout tests
PR fails CIClaude autofix attempts (up to 3x)
PR passes CIHoldout scenarios run against PR
Scenarios updateOpen PRs re-tested with latest scenarios

Your scenarios repo: {name} Your coding agent cannot see those tests. The holdout wall is intact.

Important: Tell the user:

Before you can test the loop, you need to merge this PR to main first. GitHub's workflow_run triggers only activate for workflows that exist on the default branch. Once merged, create a new PR with any small change — that's when you'll see Autofix, Scenarios Dispatch, and Spec Dispatch fire for the first time.

Update docs/joycraft-assessment.md if it exists — set the Level 5 score to reflect the new setup.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.