agentleFS
lucaswychanlucaswychan/quant-paper-agent/.claude/skills/paper-replicate/SKILL.md

paper-replicate

Skill0 starsChanged 3 months agoMITView on GitHub

Plan, request-authorize, implement, test, compare, and finalize a quantitative-paper portfolio backtest against a checksum-bound, metric-bearing replication target from extraction/reported.json. Use only when the finalized extraction declares at least one replication target with reported metric IDs; route theorem-only or formula-only papers to paper-theory-reconstruct. A direct user request to replicate authorizes uninterrupted execution without a second approval prompt. Supports exact, approximate, and illustrative metric-bearing backtests with hash-bound lineage, causality and accounting audits, metric-ID-native comparison, and immutable runs.

The file

We split this skill into the rules it gives an agent

12 rules. About 0 are common, 9 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈9

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Create one auditable run. Never mix plans, data, code, or results across run IDs.

    line 8
  2. Distinctive:

    Run the deterministic eligibility preflight before creating a run:

    line 32
  3. Very specific:

    If the finalized extraction has no metric-bearing target, stop and use paper-theory-reconstruct.

    line 39
  4. Very specific:

    Write plan.md and v3 config.json.

    line 42
  5. Very specific:

    When invoked by $quant-paper, pass --workflow-request <absolute-workflow.json> instead.

    line 56
  6. Distinctive:

    Run the standardized gate:

    line 70
  7. Distinctive:

    Only after the audit passes, run:

    line 79
  8. Distinctive:

    Never tune parameters, dates, universes, definitions, or variants after seeing results.

    line 108
  9. Distinctive:

    Never call proxy or survivorship-biased data exact.

    line 110
  10. Distinctive:

    Never hide costs, financing, missing-data treatment, or definition mismatches.

    line 111
  11. Distinctive:

    Never regenerate approval, audit results, comparisons, or finalized artifacts in place.

    line 112
  12. Distinctive:

    Treat surprising agreement as an audit trigger.

    line 113

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Paper replicate

Create one auditable run. Never mix plans, data, code, or results across run IDs.

Resolve paths and contracts

Resolve the absolute workspace root and this skill's directory. Read references/plan_template.md, references/run_schema.md, and references/config.schema.json before Phase A. Read references/benchmark_template.md before Phase C. Every machine-readable artifact is checked against its bundled current JSON Schema as well as phase-specific lineage and semantic invariants.

papers/<paper-id>/replications/<run-id>/
├── plan.md
├── config.json
├── approval.json
├── audit_report.json
├── data/{prices.csv,manifest.json}
├── code/
│   ├── signal.py
│   ├── portfolio.py
│   ├── backtest.py
│   └── tests/
├── outputs/
└── run_manifest.json

Phase A: plan and bind request authorization

  1. Run the deterministic eligibility preflight before creating a run:
python3 "$QPA_REPLICATE_SKILL_DIR/scripts/check_eligibility.py" \
  <paper-id> --root "$QPA_WORKSPACE_ROOT/papers"

If the finalized extraction has no metric-bearing target, stop and use paper-theory-reconstruct; never synthesize a target or metric.

  1. Select one declared target ID and classify the run as exact_replication, approximate_replication, or illustrative_reconstruction.
  2. Write plan.md and v3 config.json. Declare data vintage and bias policy; event timing and calendar; portfolio limits; costs and financing; metric-ID calculators, definition compatibility, and tolerances; audit tolerances and required paper-specific tests.
  3. Validate the plan:
QPA_REPLICATE_SKILL_DIR="/absolute/path/to/paper-replicate"
QPA_WORKSPACE_ROOT="/absolute/path/to/workspace"
python3 "$QPA_REPLICATE_SKILL_DIR/scripts/validate_run.py" \
  <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers" --phase plan \
  --show-approval-material
  1. Treat a direct user instruction to replicate a paper or declared target as authorization to execute the plan created for that request. Do not surface the hashes, halt, or ask for a second approval. A request only to review, diagnose, extract, or draft a plan is not execution authorization.
  2. Immediately record the request authorization with both --expected-*-sha256 values and --confirm-replication-request. The resulting approval.json preserves hash-bound lineage for the plan, config, extraction, source manifest, target, and classification without adding a conversational gate. Keep --confirm-user-approved for workflows where the user explicitly approves a separately presented plan.

When invoked by $quant-paper, pass --workflow-request <absolute-workflow.json> instead. The recorder verifies that the immutable workflow goal authorizes replication and binds approved_by to the workflow ID and request hash.

python3 "$QPA_REPLICATE_SKILL_DIR/scripts/record_approval.py" \
  <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers" \
  --expected-plan-sha256 <printed-plan-sha256> \
  --expected-config-sha256 <printed-config-sha256> \
  --confirm-replication-request

Phase B: implement, audit, and run

  1. Fetch or place approved data and write a v2 data manifest. fetch_data.py supports only approved yfinance configs, which cannot be exact replications.
  2. Implement small signal.py, portfolio.py, and backtest.py modules plus paper-specific unittest tests named in config.audit.required_custom_tests.
  3. Run the standardized gate:
python3 "$QPA_REPLICATE_SKILL_DIR/scripts/run_audit_tests.py" \
  <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers"

It requires backtest.run() to return both decision_weights and lagged applied weights, then enforces the exact configured shift, price/weight/gross-return reconciliation, future perturbation causality, exposure limits, turnover and cost accounting, missing-data policy, timestamp alignment, and required custom tests. Any subsequent code, data, plan, config, or extraction change invalidates the report.

  1. Only after the audit passes, run:
python3 "$QPA_REPLICATE_SKILL_DIR/scripts/run_backtest.py" <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers"
python3 "$QPA_REPLICATE_SKILL_DIR/scripts/plot_nav.py" <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers"

The harness computes costs and metrics. Replicated values are keyed by exact metric ID; outputs cannot be overwritten.

Phase C: compare, judge, and freeze

python3 "$QPA_REPLICATE_SKILL_DIR/scripts/compare_metrics.py" <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers"

Author outputs/benchmark.md from the comparison, input comparability, diagnostics, ambiguities, and audit evidence. Use exactly one verdict: replicated, partially replicated, not replicated, or inconclusive. Numerical closeness alone is insufficient.

Finalize only after plots and benchmark are complete:

python3 "$QPA_REPLICATE_SKILL_DIR/scripts/finalize_run.py" <paper-id> <run-id> --root "$QPA_WORKSPACE_ROOT/papers"

The resulting manifest freezes every replication input and output. Use a new run ID for any change.

Rules

  • Require a finalized extraction with at least one metric-bearing replication target.
  • Route formula, theorem, phase-boundary, and figure reconstructions without reported metrics to paper-theory-reconstruct.
  • Never tune parameters, dates, universes, definitions, or variants after seeing results.
  • Interpret the original direct replication request as authorization only for the bounded replication workflow; it does not authorize later scope expansion, external publication, or live trading.
  • Never call proxy or survivorship-biased data exact.
  • Never hide costs, financing, missing-data treatment, or definition mismatches.
  • Never regenerate approval, audit results, comparisons, or finalized artifacts in place.
  • Treat surprising agreement as an audit trigger.
  • Author only the current config.schema.json contract; reject noncurrent run configurations and create a new reviewed plan and run ID.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.