agentleFS
Sign inStart free
DshamirDshamir/slm-forge/skills/forge-eval/SKILL.md

forge-eval

Skill1 starsChanged 6 months agoMITView on GitHub

Runs perplexity vs. baseline + 10 sample generations on the trained model, on the EC2 instance. Detects artifact patterns (repetition loops, GGUF tokenizer bugs, degenerate outputs) — fails the run when artifact_pct exceeds FORGE_EVAL_MAX_ARTIFACT_PCT (default 30%). The automated quality gate that replaces the v1 manual QUALITY_GATE.

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run perplexity on the held-out test split (vs.

    line 18
  2. Very specific:

    Write eval/{perplexity.json, samples.md, report.json} and sync to S3

    line 25
  3. Very specific:

    Hard-fail: if artifact_pct > FORGE_EVAL_MAX_ARTIFACT_PCT (default 30%)

    line 26
  4. Very specific:

    If manifest.artifacts.eval_reports_s3 is set, exits 0.

    line 43

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

forge-eval

When this fires

Phase position: EVAL — after forge-monitor returns completed. On the v2 path, forge-plan-fit already did corpus-level QA before spend; EVAL handles post-train output-quality validation.

What it does

  1. Verify manifest.artifacts.final_weights_s3 exists
  2. Upload eval.py to the instance via SSM
  3. Run perplexity on the held-out test split (vs. the base model for diff)
  4. Generate 10 sample completions on canonical prompts
  5. Artifact-pattern detection — scan generations for:

    • 30% repetition (n-gram self-similarity)

    • GGUF tokenizer artifacts (mojibake / control chars)
    • Degenerate loops (same token N+ times)
    • Refusal-only outputs (model collapsed to "I cannot help")
  6. Write eval/{perplexity.json, samples.md, report.json} and sync to S3
  7. Hard-fail if artifact_pct > FORGE_EVAL_MAX_ARTIFACT_PCT (default 30%)
  8. Advance to QUANTIZE (or QUALITY_GATE on legacy v1)

Inputs

  • $1 = forge-id
  • manifest.artifacts.final_weights_s3 (LoRA adapter or merged weights)
  • manifest.compute_target.instance_id
  • manifest.plan.base_model (for baseline perplexity)
  • FORGE_EVAL_MAX_ARTIFACT_PCT (env, default 30)

Outputs

  • /workspace/eval/{perplexity.json, samples.md, report.json} (on instance)
  • s3://.../eval/reports/
  • manifest.artifacts.eval_reports_s3
  • manifest.state.current_phase = QUANTIZE

Idempotency

If manifest.artifacts.eval_reports_s3 is set, exits 0.

Failure modes

ExitReason
0eval clean (artifact_pct under threshold)
1LoRA merge failed OR eval.py timeout OR artifact_pct > threshold (training collapsed — model unfit to ship)
64no forge-id provided

A failure here is terminal — dispatch-v2.sh will tear down the instance and skip QUANTIZE/REGISTER/PUBLISH. The repos stay private.

External resources

  • AWS EC2 (SSM RunCommand to execute eval.py)
  • AWS S3 (sync reports)
  • HuggingFace model cache (for the baseline diff)

Cost class

spends GPU time — eval runs on the same hourly EC2 meter.

Depends on

forge-monitor (must have synced final_weights_s3)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free