epic-harness

25 skills (9 pipeline + 16 quality) + self-evolving agent harness.

Structure

• skills/ — 25 skills + _dispatch router + _critic reviewer
• registry/ — Seeding resources (embedded in Rust binary at compile time)
  • presets/ — Cold-start skill templates
  • rules/ — Seeded host rule files
  • scripts/install.js — Plugin bootstrap (SessionStart)
• src/ — Rust crate; single binary (epic-harness)
  • src/hooks/ — Ring 0 automation (guard, polish, observe, resume, snapshot, reflect)
  • src/evolve/ — Ring 3 evolution loop
• docs/ — User-facing documentation and assets
  • architecture.md, quickstart.md, demo/, references/, specs/
• integrations/common/ — HARNESS.md embedded into the binary (include_str!), self-seeded to ~/.harness/HARNESS.md

Architecture: 4-Ring Model

• Ring 0 (Autopilot): Hooks auto-maintain quality, restore sessions, learn
• Ring 1 (Pipeline): 9 skills that orchestrate multi-step workflows (discover → spec → go → audit → eval → ship, orbit, evolve, team)
• Ring 2 (Quality Gates): 17 skills that auto-trigger on context signals (tdd, debug, secure, threat-model, vuln-scan, triage, verify, etc.)
• Ring 3 (Evolve): Observe → Analyze → Evolve → Gate → Reload self-improvement loop

/orbit — Autonomous Pipeline

Single-command spec-to-PR execution with three entry modes.

flowchart TD
    START["/orbit"] --> MODE{"requirement?"}
    MODE -->|"unclear"| WAIT["Interactive\nUser runs /discover → /spec"]
    MODE -->|"clear + complex"| COUNCIL["Council\n4-Voice Auto-spec"]
    MODE -->|"clear + simple"| DIRECT["Direct\nAuto-spec"]
    WAIT --> SPEC["Load Approved Spec"]
    COUNCIL --> SPEC
    DIRECT --> SPEC
    SPEC --> GO["Go Phase\nPlan → Execute → Integrate"]
    GO --> CHECK["Check Phase\nReview + Audit + Test"]
    CHECK -->|"PASS"| SHIP["Ship Phase\nIsolated Test → PR → CI"]
    CHECK -->|"WARN"| SHIP
    CHECK -->|"FAIL"| RETRY{"retry < 3?"}
    RETRY -->|Yes| FIX["Go Fix\nTarget fixes from action items"]
    FIX --> CHECK
    RETRY -->|No| PAUSE["Pause — User decides"]
    PAUSE -->|continue| FIX
    PAUSE -->|abort| ABORT["Abort"]
    SHIP --> EVOLVE["Evolve\nAuto-analyze session"]
    EVOLVE --> DONE["Orbit Complete\nConsolidated Report"]

State tracking: $HARNESS_DIR/orbit/PIPELINE-{timestamp}.json — updated after every phase transition, survives context compaction.

Human checkpoints: mode selection (interactive only when unclear), 3 failed audits (pause).

Evolve: runs automatically after PR created + CI green. Skipped on abort.

Eval System (Ring 3 Core)

Fuses A-Evolve benchmark patterns into Claude Code context.

Multi-Dimensional Scoring

Every tool call scored on 3 axes:

• tool_success (0/1): Did the tool succeed?
• output_quality (0.0-1.0): Output quality (per-tool criteria)
• execution_cost (0.0-1.0): Efficiency
• Composite: SCORE_WEIGHTS.success×tool_success + SCORE_WEIGHTS.quality×quality + SCORE_WEIGHTS.cost×cost (default 0.5/0.3/0.2)

All weights configurable via SCORE_WEIGHTS in common.rs.

Failure Classification (9 types)

type_error, syntax_error, test_fail, lint_fail, build_fail, permission_denied, timeout, not_found, runtime_error

Pattern Detection (4 types)

All thresholds defined as constants in common.rs for per-project tuning. Function-name-level context included (extracted from stack traces, error messages). Error message hash-based dedup for improved precision (hashString + normalizeError).

• repeated_same_error: Consecutive same error + same error hash (REPEATED_ERROR_MIN, default 3)
• fix_then_break: Edit success → Bash error cycle (FTB_LOOKAHEAD=3, FTB_MIN_CYCLES=2)
• long_debug_loop: Same file in consecutive operations (DEBUG_LOOP_MIN, default 5)
• thrashing: Edit↔Error alternating (THRASH_MIN_EDITS=3, THRASH_MIN_ERRORS=3)

Skill Seeding Thresholds

• Weak tool: success rate < WEAK_TOOL_RATE(0.6), min WEAK_TOOL_MIN_OBS(5) observations
• Weak file type: success rate < WEAK_EXT_RATE(0.5), min WEAK_EXT_MIN_OBS(3) observations
• High-frequency error: HIGH_FREQ_ERROR_MIN(5)+ occurrences

Stagnation Gating

• STAGNATION_LIMIT(3) sessions without improvement → auto-rollback evolved skills to best checkpoint
• IMPROVEMENT_THRESHOLD: 5%
• Trend tracking: improving / stable / declining

Evolved Skill Validation

Auto-validated by gate_skills() in reflect:

• Must have --- frontmatter delimiter
• Body (after frontmatter) must be ≥ 20 characters
• SKILL.md file must exist in skill directory
• Invalid skills silently removed; skill count capped at MAX_EVOLVED_SKILLS(10)

Evolved Skill Priority

Static skills (tdd, debug, secure, etc.) always take priority over evolved skills. Evolved skills supplement only.

Skill Structure

Core quality-gate skills (tdd, debug, secure, verify, ...) include 4 sections:

• Process: Step-by-step execution procedure
• Anti-Rationalization: Excuse | Rebuttal | What to do instead (table)
• Evidence Required: Checklist of proof needed for completion claims
• Red Flags: Anti-pattern warnings

Concurrent Session Safety

Obs files use session_{date}_{pid}_{random}.jsonl format for per-session isolation. Reflect merges all same-day session files for analysis.

Cold-Start Presets

On first session with no evolved skills, stack-appropriate preset skills auto-apply for detected stacks (Go, Java, Kotlin, Node.js, PHP, Python, Ruby, Rust).

Guard Rule Extension

Add custom block/warn rules via .harness/guard-rules.yaml in your project root:

blocked:
  - pattern: kubectl\s+delete  | msg: kubectl delete blocked
warned:
  - pattern: docker\s+system\s+prune | msg: Docker prune — check first

Project Overrides

Teams commit .epic/team.md (and gitignored .epic/team.local.md) beside the code; _dispatch loads them before every pipeline skill. See docs/project-overrides.md.

Cross-Project Learning

Opt-in by creating ~/.harness/projects/{slug}/.cross-project-enabled. On session end, patterns export to ~/.harness/global_patterns.jsonl. On next session start, weak patterns from other projects shown as hints.

Model Class Awareness

[model] in config.toml scales the harness to the model in use (class: auto detects from ANTHROPIC_MODEL/CLAUDE_MODEL/etc.):

• Thresholds: only when class = "frontier" is set explicitly — pattern thresholds (repeated_error_min, debug_loop_min, thrash_*) scale by frontier_threshold_scale (default ×2). The default auto resolution depends on which env vars reach the short-lived hook process, so scaling on it made pattern sensitivity machine-dependent; auto now always uses base thresholds.
• Injection: frontier-class sessions inject only synthesized skill bodies (template bodies are withheld and announced as awaiting accept-synth); light-class sessions inject templates too. inject_templates: true|false overrides per class.
• Budget: inject_per_skill_chars / inject_total_chars cap the SessionStart stdout injection.

Skill Attribution (Holdout A/B)

Per-evolved-skill effectiveness is measured with a genuine counterfactual:

• Each day, each under-evaluation skill is deterministically assigned to the active or holdout arm (hash(skill, date) % attribution_holdout_modulus == 0 → holdout). resume and reflect compute the same assignment independently.
• Active skills are injected into context at session start; holdout skills are withheld. avg_score_with averages active-arm sessions, avg_score_without averages holdout-arm sessions (sessions_holdout counts them).
• Skills seeded during a session are NOT credited with that session's score — attribution uses the pre-seed skill listing.
• A skill trailing its holdout arm (sessions_active >= 3, sessions_holdout >= 2, avg_score_with < avg_score_without - 0.02) is flagged in dashboards — it is not auto-evicted. Statistical deletion at these sample sizes destroyed healthy skills on easy-holdout-day noise.
• After attribution_eval_sessions (default 12) total samples the verdict is settled: survivors stay active every session.

The pre-holdout scheme (credit every skill on disk each session, derive "without" from pre-creation history) was confounded by regression to the mean and is gone.

Skill Synthesis (host-agnostic)

Seeded skills start from static templates. reflect then emits a pending-synthesis manifest ($HARNESS_DIR/projects/{slug}/pending_synth.jsonl) for each seeded skill — failure evidence (masked error snippets per category, counts, detected patterns) plus the template body. A host agent (claude/codex/agy, using its own subagent mechanism with no model specified) reads the manifest, synthesizes a better body, and applies it via:

epic-harness evolve accept-synth --skill <name> [--file <path> | --stdin]

The CLI validates the body and re-runs the Critic falsifiability gate before overwriting the template. Config ([evolution]):

• llm_synthesis (default true) — gates manifest emission.
• llm_synthesis_max_per_session (default 1) — caps manifests per session.

If no host ever runs accept-synth, the template body persists — synthesis can only improve a skill, never block seeding. The harness references no CLI binary and no model name; the previous synchronous claude -p subprocess (which hung under slow/remote hosts) is gone.

Evolved Skill Injection

epic resume (SessionStart) prints active evolved skills' bodies to stdout, which the host injects into the model's context — the Ring 3 loop closes deterministically instead of relying on the model scanning evolved/ by prompt instruction. Holdout-arm skills are withheld and only announced on stderr for transparency.

SkillOpt-Inspired Optimization

Three deep learning-inspired techniques adapted from SkillOpt applied to natural language skill evolution.

Negative Feedback Buffer

• Rejected proposals stored in rejected_buffer.json with TTL-based expiry (default: 10 sessions)
• curate_proposal() Rule 0: check buffer before generating proposals
• gate_skills() auto-registers invalid skills with rejection reasons
• Config: rejected_buffer_ttl in [evolution]

Minibatch Reflection

• Observations decomposed into fixed-size batches (default: 8) for structural pattern extraction
• Batches analyzed for dominant error, file clusters, and success rate
• reusable when: dominant error ≥60% + ≥2 distinct files + category ≠ "other"
• Reusable insights → skill proposals with origin "minibatch"
• Config: minibatch_size in [evolution]

Slow/Meta Update

• Epoch classification via linear regression over last 5 sessions: Improving / Regressing / PersistentFailure / StableSuccess
• meta.json per evolved skill tracks slow_updates array (capped at 20)
• Attribution verdicts are reported, not enforced: a skill whose avg_score_with trails its holdout avg_score_without (beyond sessions_active >= 3, sessions_holdout >= 2, delta 0.02) is flagged in dashboards but never auto-deleted — 3-vs-2 samples carry less evidence than session-difficulty variance

Security Pipeline (Ring 2)

Three security assessment skills ported from defending-code:

1. threat-model: Trust boundary enumeration, threat actor analysis, threat scenario generation → THREAT_MODEL.md
2. vuln-scan: 4-dimension systematic scanner (injection, auth, data exposure, dependencies) → VULN-FINDINGS.json
3. triage: Adversarial validation with severity adjustment, chaining analysis, root-cause grouping → TRIAGE.json

Pipeline flow: /threat-model → /vuln-scan → /triage

Audit --strict Mode

• Artifact-only delivery: audit modes receive only diff + spec, no builder context
• Cross-check independence: code/security/test modes run blind until synthesis
• Blind scoring: prevents anchoring bias between modes
• No self-review: builder session excluded from audit agent selection
• Activation: --strict flag or mode: strict in .harness/engagement.md

Engagement Context

• Optional .harness/engagement.md in project root for security assessment scoping
• Defines: Authorization, Scope (in/out), Constraints, Environment, Exclusions
• secure skill checks for engagement context and loads scope if present
• Reference template: docs/references/engagement.md

Polish → Observe Feedback

Polish hook (format/typecheck) results auto-record into observe pipeline. Format failure = lint_fail, typecheck failure = build_fail — feeds into pattern detection.

Dispatch Logging

Skill dispatches logged to ~/.harness/projects/{slug}/dispatch/dispatch_YYYYMMDD.jsonl. Analyze via /evolve history.

Unified Memory (harness-mem)

All agents share a single knowledge graph stored in ~/.harness/memory.db (SQLite + FTS5). Accessed via epic mem CLI commands (outputs JSON).

Smart Recall System

Memory retrieval uses composite scoring instead of simple latest-N:

• Scoring formula: recency(25%) + importance(35%) + access_freq(15%) + FTS_match(25%)
• Recency: Exponential decay with 30-day half-life
• Importance: Type-based defaults — decision(0.9), resolution(0.8), concept(0.7), project(0.7), pattern(0.5), error(0.4), session(0.2)
• Access frequency: Saturates at 20 accesses (access_count / 20)
• FTS match: 1.0 bonus when hint keyword matches via FTS5

CLI Commands (6)

Memory Lifecycle

• Access tracking: Every recall/search/context call increments access_count and updates accessed_at.
• Gradual decay: Nodes untouched for 30+ days lose 10% importance per cycle (floor=0.05). pinned tag prevents decay.
• Stale tagging: Nodes untouched for 180+ days tagged as stale and excluded from recall.
• Graph augmentation: epic mem recall follows 1-hop edges from top results, returning related nodes with connection counts.

Node Schema

id, type, title, tags, projects, agents, created, updated, body,
importance (REAL 0.0-1.0), access_count (INTEGER), accessed_at (TEXT)

Dispatch Integration

_dispatch skill runs epic mem recall with current task context before invoking any skill. Past decisions (importance=0.9) surface first, preventing contradictory choices across sessions.

Project Side Data

~/.harness/projects/{slug}/ directory accumulates per-project memory, observations, evolved skills:

• memory/ — Project patterns and rules
• sessions/ — Session snapshots
• obs/ — Tool usage observation logs (JSONL, 3-axis scores)
• evolved/ — Auto-evolved skills (pattern/tool/filetype/error based)
• evolved_backup/ — Best-state backup (for stagnation rollback)
• team/ — /team outputs
• orchestrator/ — Multi-agent orchestration state (run.json, control.json, agents/{id}/)
• dispatch/ — Skill dispatch logs (JSONL)
• orbit/ — /orbit pipeline state files (PIPELINE-*.json)
• eval/ — Eval config, baselines, and results (eval.yaml, baselines/.json, results/.json)
• metrics.json — Aggregate stats (score_history, trend, stagnation_count, skill_attribution)
• evolution.jsonl — Evolution history (SessionAnalysis + patterns)
• .cross-project-enabled — Cross-project learning opt-in marker (optional)

~/.harness/projects/{slug}/ auto-created on session start. Keep .harness/guard-rules.yaml in your project root to share safety rules with your team.

Version Bump Checklist

When creating a new release tag, update ALL of the following to the same version:

All nine must match before tagging.

Root plugin.json is agy's manifest, but grok reads it first and only falls back to .grok-plugin/plugin.json when the root is absent (measured on grok 1.0.13). Bumping only .grok-plugin leaves grok serving the old version with no error, so the two must never diverge.

After pushing, re-pin the hub catalog in epicsagas/plugins: the grok entry in .grok-plugin/marketplace.json (and .grok-plugin/plugin-index.json, if present) carries a 40-char source.sha plus version, and .hermes/epic/plugin.yaml carries a version. While that sha points at an old commit, grok plugin update keeps installing the pinned commit no matter how many times this repo is pushed. The claude and codex hub entries track remote HEAD and need no edit. forge.py publish --marketplace epicsagas/plugins does the grok and hermes updates. Update Cargo.lock with cargo update -p epic-harness --precise x.y.z after editing Cargo.toml — the cargo publish step of release.yml runs without --allow-dirty, so a stale lock fails the crates.io publish.

Dashboard rebuild (before tagging)

The web dashboard is a Svelte app under app/ that builds to assets/dashboard.html, which is embedded into the binary at compile time. After bumping versions, rebuild and commit it so the bundled dashboard matches:

make dashboard-build   # cd app && pnpm install --frozen-lockfile && pnpm run build && cp app/dist/index.html assets/dashboard.html
git add assets/dashboard.html

The running binary always stamps its own CARGO_PKG_VERSION into the served dashboard via <meta name="harness-version"> (see serve.rs), so the version shown is correct even if this step is missed — but rebuilding keeps the fallback (dev mode) and Settings page in sync.