Evolve — Continuous Learning Pipeline

Automatic behavioral pattern extraction from Claude Code sessions. Observes tool usage, detects repeated patterns, and generates "instincts" (reusable behavioral rules) that improve future sessions.

Quick Start

1. Install the hooks

Add to your project-level settings file (~/.claude/projects/<project-hash>/settings.json):

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash|Edit|Write|Read|Glob|Grep|Agent",
        "hooks": [
          {
            "type": "command",
            "command": "node \"$CLAUDE_PROJECT_DIR/.claude/scripts/observe.js\" 2>/dev/null || true",
            "timeout": 5,
            "async": true
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Bash|Edit|Write|Read|Glob|Grep|Agent",
        "hooks": [
          {
            "type": "command",
            "command": "node \"$CLAUDE_PROJECT_DIR/.claude/scripts/observe.js\" 2>/dev/null || true",
            "timeout": 5,
            "async": true
          }
        ]
      }
    ],
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "node \"$CLAUDE_PROJECT_DIR/.claude/scripts/observer-analyze.js\" 2>/dev/null || true",
            "timeout": 10,
            "async": true
          }
        ]
      }
    ]
  }
}

2. Add to .gitignore

# Observations (local session data, not shareable)
.claude/homunculus/projects/*/observations-structural.jsonl
.claude/homunculus/projects/*/observations-content.jsonl
.claude/homunculus/projects/*/observations.archive/
.claude/homunculus/projects/*/.last-analysis
.claude/homunculus/projects/*/.analysis-prompt.md
.claude/homunculus/analysis-log.jsonl

# Generated instincts (local learning data)
homunculus/

3. Use it

Just work normally. Observations accumulate automatically. After 50+ new observations, the Stop hook triggers Opus analysis which writes instinct files. Run /evolve status to check progress.

Trigger

When the user says "evolve", "check instincts", "learning status", "prune instincts", or "promote patterns".

Commands

The CLI auto-detects the project root via git rev-parse --show-toplevel. Running from any subdir of the repo works:

node .claude/scripts/instinct-cli.js <command>

CLAUDE_PROJECT_DIR still takes precedence if set (used inside Claude Code sessions).

/evolve status

Show observation counts, instinct counts, last analysis time, evolved skills.

/evolve list

List all instincts with confidence scores, grouped by project and scope.

/evolve

Analyze instinct clusters and show evolution candidates (skills, commands, agents).

/evolve generate

Actually generate skill/command/agent files from qualified instinct clusters.

/evolve prune

Remove pending instincts older than 30 days (dry run by default). Use --no-dry-run to execute.

Architecture

Data Flow

Tool calls ──→ observe.js (PreToolUse/PostToolUse hook)
               │
               ├──→ observations-structural.jsonl (metadata only, injection-safe)
               └──→ observations-content.jsonl (full I/O, secrets scrubbed)
                                    │
                     observer-analyze.js (Stop hook)
                     checks: 50+ new obs AND 30min cooldown
                                    │
                     Spawns: claude --print --model opus
                     Reads observations, writes instinct .md files
                                    │
                     homunculus/instincts/<project-id>/*.md
                                    │
                     /wisdom (primary, human-driven, Fridays)
                       ──→ skill enhancements, new skills, CLAUDE.md edits
                     /evolve generate (fallback, mechanical)
                       ──→ SKILL.md scaffolds in .claude/homunculus/evolved/skills/

project-id is repo-stable: keyed off git remote get-url origin (normalized), not cwd. Worktrees, case-variant paths, and isolated subagents all share one project_id per repo.

Graduation path: /wisdom is the primary mechanism — it reads instincts + CLAUDE.md + all skill descriptions and proposes where each instinct should land. /evolve generate is a mechanical fallback that produces minimal SKILL.md scaffolds from clusters; its output dir (evolved/skills/) is expected to be empty in normal operation. See /observer-ops § Graduation path if available.

Directory Layout

<repo>/
├── .claude/
│   ├── scripts/
│   │   ├── observe.js              # Hook: logs tool calls to JSONL
│   │   ├── observer-analyze.js     # Hook: triggers analysis at threshold
│   │   └── instinct-cli.js         # CLI: list/evolve/prune/status
│   └── homunculus/                  # Observation data (gitignored)
│       ├── projects.json            # Project registry
│       ├── analysis-log.jsonl       # Analysis run history
│       └── projects/<hash>/
│           ├── observations-structural.jsonl
│           ├── observations-content.jsonl
│           ├── .last-analysis       # Timestamp marker
│           └── .analysis-prompt.md  # Last analysis prompt
└── homunculus/                      # Generated instincts (gitignored)
    └── instincts/<project-id>/
        ├── build-verify-after-edits.md
        └── prefer-edit-over-write.md

Why two directories?

Critical design constraint: Claude Code's Write tool blocks writes to .claude/ directories, even with --dangerously-skip-permissions. This is a hard security restriction.

• .claude/homunculus/ — Observations written by Node.js (fs.appendFileSync), no Claude CLI involvement. Works fine.
• homunculus/ (repo root) — Instincts written by the spawned claude --print analysis process using the Write tool. Must be outside .claude/.

Instinct Schema

---
id: kebab-case-unique-id
trigger: "when [specific condition]"
confidence: 0.7  # 0.3-0.9
domain: code-style|testing|git|debugging|workflow|file-patterns
source: session-observation
scope: project|global
project_id: 12-char-hash
project_name: project-name
created: YYYY-MM-DD
---

# Title

## Action
What to do when the trigger condition is met.

## Evidence
- Observed N instances of this pattern
- Specific examples from observations

Confidence Thresholds

Confidence is boosted +0.05 for confirming evidence and decayed -0.1 for contradicting evidence on subsequent analyses.

Scripts Reference

observe.js — Observation Logger

Runs as async PreToolUse/PostToolUse hook. Reads tool call JSON from stdin, writes two JSONL streams:

• Structural (metadata only): tool name, file paths, command previews, patterns, exit codes. Safe for automated analysis — no user content.
• Content (full I/O): complete tool input/output with secrets scrubbed. For manual review only.

Features:

• Secret scrubbing (API keys, tokens, passwords, connection strings, DB URLs)
• Truncation (3KB per field)
• Auto-rotation at 10MB per file
• Project identification via git remote hash (normalized: SSH vs HTTPS collapse to same ID)
• Recursion guard (ECC_SKIP_OBSERVE=1)

observer-analyze.js — Analysis Trigger

Runs as async Stop hook. Checks if enough new observations have accumulated:

1. Finds the most recently active project
2. Counts observations since last analysis
3. If 50+ new observations AND 30min cooldown elapsed:
   • Generates analysis prompt with recent observations + existing instincts
   • Spawns claude --print --model opus in background
   • The spawned process reads observations, identifies patterns, writes instinct files

Configuration constants:

instinct-cli.js — Management CLI

Commands: status, list, evolve, prune

Searches for instincts in both .claude/homunculus/ (legacy) and homunculus/instincts/ (current) directories. Displays sanitized output to prevent prompt injection from untrusted observation content.

Platform Notes

Windows

• spawn('claude') fails with ENOENT because Node.js can't resolve claude.cmd without a shell.
• observer-analyze.js prefers direct node cli.js invocation over claude.cmd to avoid console windows.
• Path separators are normalized to forward slashes in the analysis prompt.

macOS / Linux

• Should work without shell: true but it's kept for consistency.
• The claude binary is typically at ~/.npm-global/bin/claude or in /usr/local/bin/.

Settings File Location

Hooks MUST be in the project-level settings file:

~/.claude/projects/<project-hash>/settings.json

NOT in the repo's .claude/settings.json — that file is for permissions and project config, not hooks. Claude Code reads hooks from the user-level project settings file.

To find your project hash:

ls ~/.claude/projects/

Look for the directory matching your repo path (hyphens replace path separators).

Troubleshooting

No observations accumulating

• Check hooks are in ~/.claude/projects/<hash>/settings.json (not .claude/settings.json)
• Verify observe.js exists at .claude/scripts/observe.js
• Test manually: echo '{}' | CLAUDE_PROJECT_DIR="$(pwd)" node .claude/scripts/observe.js

Analyses running but no instincts generated

• Most likely: Claude CLI can't write to .claude/ — instincts must go to homunculus/ at repo root
• Check analysis log: tail .claude/homunculus/analysis-log.jsonl
• Check the analysis prompt: verify the "write to" path is outside .claude/

spawn claude ENOENT (Windows)

• The script prefers direct node path/to/cli.js invocation; falls back to shell invocation
• Ensure shell: true is set in the fallback spawn options

Too many project hashes

• Worktree agents create unique git remotes, generating separate project IDs
• This is expected — each worktree's observations are independent
• The main project (matching your repo name) accumulates the most observations

Workflow

1. Automatic: Observations accumulate from normal Claude Code usage (every tool call)
2. Automatic: When 50+ new observations accumulate and 30min cooldown passes, Opus analyzes and writes instincts
3. Manual: Run /evolve status to check observation counts and instinct generation
4. Manual: Run /evolve list to review generated instincts
5. Manual: Run /evolve to see evolution candidates (instinct clusters that could become skills)
6. Manual: Run /evolve generate to create skills from high-confidence clusters
7. Manual: Run /evolve prune to clean up stale pending instincts (30+ days old)