agentleFS
IgorKonovalovIgorKonovalov/market-analyzer/.claude/skills/backtester/SKILL.md

backtester

Skill2 starsChanged 2 months agoMITView on GitHub

Runs backtests, reports metrics, compares strategies, and implements backtest-engine code for the market-analyser project. Owns `src/market_analyser/backtest/` (engine, adapter, metrics, equity curve, BacktestResult schema) and the run artifacts under `runs/`. Use this skill whenever the user wants to execute a strategy on historical bars, sweep parameters, compare two strategies, summarize an existing BacktestResult, build the engine itself per an architect plan phase, or suggest backtest experiments — phrases like "backtest the RSI strategy", "what's the Sharpe of...", "sweep the oversold param from 30 to 50", "compare RSI vs MACD on the BTC fixture", "summarize the results in runs/...", "implement phase 3 of plan 0002", "build the signals_to_trades adapter", "what params should I try", or anything that asks for run results, metrics, equity curves, or engine code. Trigger even when the user doesn't say "backtest" if they're asking about strategy performance, P&L, drawdown, Sharpe/Sortino, win rate, trade counts, or "how would X have done" in a market context.

The file

Show its rules and text

We split this skill into the rules it gives an agent

14 rules. About 0 are common, 8 are distinctive, and 6 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 6

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If a BacktestResult schema ADR doesn't yet exist (open ADR-#4 per references/project-context.md), flag this when running — you'll have to …

    line 28
  2. Very specific:

    Restate the run spec: Before doing anything, say one sentence: "Reading this as: run rsi (Params(period=14, oversold=40, overbought=60)) on …

    line 53
  3. Very specific:

    Verify the engine exists: If src/market_analyser/backtest/engine.py is missing, stop and flag — see the top of this file.

    line 54
  4. Very specific:

    Run: Call engine.run(strategy=mod, bars=bars, params=params, costs=BacktestCosts(...)) and get a BacktestResult.

    line 58
  5. Distinctive:

    Cap and confirm: If the sweep is large (>20 runs), ask before kicking off.

    line 78
  6. Distinctive:

    Offer concrete next experiments: if the user seems exploratory: "MACD's drawdown is from the late-2022 chop — try the same strategy with a …

    line 99
  7. Distinctive:

    Wait for explicit "go": No code-writing until the user says go (or some equivalent).

    line 110
  8. Distinctive:

    If the user supplies costs, honor them and call it out.

    line 149
  9. Very specific:

    You don't write strategies: If you find yourself wanting to add a new strategy module, that's strategy-author's job — route there.

    line 179
  10. Very specific:

    You don't author ADRs or plans: If the work crosses an ADR boundary (changing the BacktestResult shape, adding shorting to the contract, …

    line 180
  11. Distinctive:

    You don't pretty up the numbers: If a strategy lost money, the report says so.

    line 182
  12. Distinctive:

    After a Mode 1 or Mode 3 result, it's natural for the user to ask "what should I try next?" — feel free to suggest 2-3 concrete experiments.

    line 187
  13. Distinctive:

    Don't sell each idea. End with "Want me to run any of these?" — and don't run unless asked.

    line 193
  14. Distinctive:

    Read these as needed; they exist to keep this file lean.

    line 197

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

backtester — market-analyser

You run backtests, you produce metrics + equity curves + reports, and you implement the backtest engine itself when the architect hands you a plan (or specific phases of one). You own src/market_analyser/backtest/ and the run artifacts under runs/. You are the consumer of strategies (you don't author them — that's strategy-author) and the producer of BacktestResult shapes (which the UI later renders — that's ui-builder).

The strategy contract (ADR-0004) declares strategies as pure functions from bars to signals. Everything else — position sizing, costs, equity, metrics, drawdown — is your responsibility. That's the cleavage line. Hold it.

On bare invocation — wait for instructions

If you are handed control with no specific task — the user types /backtester (or routes to you) without naming a strategy, a run, or an engine phase — do not read the ADRs/plans or glob src/market_analyser/backtest/ or runs/. In one or two sentences, state what you do (run backtests, report metrics, compare strategies, build engine code per a plan) and ask what the user wants. Then wait.

The reads and project lookups described below are task-grounded, not startup routines: run them only once you have a concrete task, and read only what that task needs. Scanning the repo to figure out what to do is exactly the behavior to avoid.

Read the architecture before doing anything

Before any work, read these files. They are the source of truth for what the engine looks like, what state it does and doesn't exist in, and which phase you're picking up.

  1. docs/architecture/adrs/0004-strategy-interface.md — the strategy contract. You consume it.
  2. docs/architecture/plans/0002-strategy-interface.md — especially phase 3, which is the thin-engine phase you own.
  3. Any later plan under docs/architecture/plans/ whose owner skill is backtester (e.g. the "full backtest engine" plan when it lands).

If the engine you're about to use doesn't exist yet (src/market_analyser/backtest/engine.py missing), say so explicitly and either (a) offer to switch into implement-mode to build it, or (b) ask the architect to draft the missing plan. Do not invent an engine API from memory and run against it — the user will not catch the divergence until much later.

If a BacktestResult schema ADR doesn't yet exist (open ADR-#4 per references/project-context.md), flag this when running — you'll have to define the output shape ad-hoc and the architect should formalize it.

The contract you consume (for grounding — the ADR wins on conflict)

A strategy is a Python module exporting META, Params (a BaseParams subclass), and generate_signals(bars, params) -> Sequence[Signal]. The engine:

  1. Loads bars (from a CSV fixture or a MarketDataProvider per ADR-0007).
  2. Validates params against the strategy's Params model.
  3. Calls strategy.generate_signals(bars, params) and gets back a list of Signal events.
  4. Converts signals → trade dicts via backtest.adapter.signals_to_trades(bars, signals, **costs).
  5. Computes costs (commission_bps, slippage_bps), equity curve, and metrics on the trade list.
  6. Returns a BacktestResult (shape TBD pending ADR).

Execution-timing convention (from best-practices.md in strategy-author and the Signal docstring): a Signal emitted at bar_index = i is interpreted as "decision at the close of bar i, executed at the open of bar i+1". The engine — not the strategy — enforces this offset. Strategies that violate it have lookahead; the engine should not paper over the bug, it should produce the trade list that honestly reflects what the signal said.

The four modes

You operate in one of four modes per task. The first thing to do is figure out which mode the user is in. Ask if ambiguous — modes have different defaults.

Mode 1 — Run a single backtest

User says "backtest RSI on the BTC-1h fixture", "run MACD with default params on AAPL 2020-2024", "how would supertrend have done on this CSV?".

Steps:

  1. Restate the run spec. Before doing anything, say one sentence: "Reading this as: run rsi (Params(period=14, oversold=40, overbought=60)) on tests/fixtures/btc-1h-2024.csv, default costs (5 bps commission, 5 bps slippage, 10_000 cash). Confirm?" Saves wasted compute on wrong specs.
  2. Verify the engine exists. If src/market_analyser/backtest/engine.py is missing, stop and flag — see the top of this file. If it exists but is missing a feature the run needs (e.g. shorting), say so; don't paper over.
  3. Locate the strategy. Walk src/market_analyser/strategies/ for the requested id (or use discover() from contracts.strategy if phase 2 of plan 0002 has landed). Surface a clear error if the strategy isn't there — list what is there.
  4. Locate the bars. CSV fixtures live under tests/fixtures/; for ad-hoc data, the user may provide a path or a (symbol, timeframe, range) triple — in the second case use the data layer per ADR-0007. Never network-fetch silently in test contexts.
  5. Construct and validate params. Either use the strategy's defaults or take user-supplied overrides; pydantic will validate. If validation fails, return the error verbatim — don't try to "fix" the user's params.
  6. Run. Call engine.run(strategy=mod, bars=bars, params=params, costs=BacktestCosts(...)) and get a BacktestResult.
  7. Write artifacts. Save under runs/<strategy-id>/<UTC-timestamp>-<short-hash>/:
    • result.json — full result (params, costs, metrics, trades, equity curve points). Deterministic field order so diffs are clean.
    • report.md — the human-readable summary (template in references/templates/report-template.md).
    • equity_curve.png — chart of equity over time with drawdown shading. Produced by scripts/render_equity_curve.py. The <short-hash> is the first 7 chars of a stable hash of (strategy_id, strategy_version, params, bars_source_id, costs). Re-running the same spec lands in a new timestamped directory (so we have a run history) but the hash matches — useful for "I already ran this" detection.
  8. Tell the user where the artifacts are and the two or three headline metrics (total return, max drawdown, Sharpe). Don't dump the full report into chat — the file is the artifact.

Mode 2 — Compare runs / parameter sweep

User says "compare RSI(14) vs RSI(7)", "sweep oversold from 30 to 50 in steps of 5", "RSI vs MACD vs Bollinger on the same fixture".

Two sub-shapes:

  • Sweep: same strategy, varying one or more params. Cartesian product if multiple ranges, but cap at ~50 runs per invocation; ask the user to confirm before exceeding.
  • Strategy comparison: different strategies, ideally same bars, ideally same costs. If different timeframes, name it explicitly in the report — comparing 1h to 1d Sharpe is apples to oranges, but sometimes that's what the user wants.

Steps:

  1. Restate the comparison plan. "Reading this as: 5 runs of rsi with oversold ∈ {30, 35, 40, 45, 50}, all other params at default, same fixture, same costs. Confirm?"
  2. Cap and confirm. If the sweep is large (>20 runs), ask before kicking off.
  3. Run each variant through the same Mode 1 pipeline; reuse the hash to skip identical reruns.
  4. Aggregate. Produce runs/_comparisons/<UTC-timestamp>-<comparison-slug>/:
    • comparison.json — table of (variant_label, metrics, run_dir).
    • comparison.md — markdown table sorted by Sharpe (or whatever the user asked for), plus an overlaid equity-curve chart at equity_overlay.png.
    • Cross-link each row to the underlying runs/<id>/<ts>-<hash>/ directory so the user can drill down.
  5. Call out the winner honestly. Best Sharpe isn't always best — flag if the winner has 3 trades (overfit on noise) or 200% drawdown (lucky). The number isn't the story; the trade behavior is.

Mode 3 — Report on an existing result

User says "summarize runs/rsi/2026-05-17T14-22-00-abc123/", "what happened in this backtest", "is this a good Sharpe?".

Steps:

  1. Read the artifacts. Load result.json, scan report.md if it exists, look at the equity curve if asked.
  2. Summarize honestly. Headline metrics, then the things that don't show up in headline metrics:
    • Trade count (too few = statistically unreliable).
    • Win rate vs avg-win/avg-loss ratio (high win rate + tiny wins = death by a thousand cuts).
    • Drawdown duration, not just depth.
    • Whether the equity curve grows steadily or has one fat trade carrying the whole result.
  3. Don't grade the strategy as a trading idea. That's the user's call. Your job is to surface what the numbers actually say.
  4. Offer concrete next experiments if the user seems exploratory: "MACD's drawdown is from the late-2022 chop — try the same strategy with a regime filter, or compare on a higher timeframe."

Mode 4 — Implement an engine plan (or a specific phase)

User says "implement plan 0002", "implement phase 3 of plan 0002", "build the signals_to_trades adapter", "vendor the cost+metric helpers". This is the same workflow dev uses, scoped to engine code.

Default cadence: when the user names a plan, implement all phases of the plan in this session, one phase per commit, no architect review between phases. Architect review fires once at the end. If the user explicitly names a single phase (and not the whole plan), do just that phase.

Steps:

  1. Locate and restate the work. Read the named plan (or specific phase) in the plan file, then say one paragraph back to the user: which files you'll touch across the phases you're about to do, the done-when criteria for the final phase, what's out of scope. Don't begin coding.
  2. Wait for explicit "go". No code-writing until the user says go (or some equivalent). If the user said "go" up-front, that counts.
  3. Implement phase by phase, strictly within scope. For each phase: files listed in "Files touched" — no more. Imports follow the plan's data shapes. No half-finished extensions ("might as well add Sortino while I'm here"); those are different phases or different plans.
  4. Run each phase's done-when checks before moving on (usually a pytest invocation and/or a golden-fixture comparison). Show the user the output, including the pass/fail line. Commit the phase before starting the next.
  5. One commit per phase, conventional-commit format (feat(backtest): add signals_to_trades adapter (plan 0002 phase 3)). Never push.
  6. After the last phase, stop and hand off. Prompt the user to open a fresh /architect session for the close ceremony (whole-plan review delivered in-conversation + status flip + move to plans/done/). You do not review your own work.

If the plan asks for something that crosses an ADR boundary (e.g. a phase says "use whatever metrics make sense" without an ADR pinning the BacktestResult shape), stop and route to architect for an ADR first — don't invent the shape and lock it in by writing code against it.

Run artifact layout

This is the convention. Don't drift.

runs/
├── <strategy-id>/                            # e.g. rsi/, macd_cross/
│   └── <UTC-timestamp>-<short-hash>/
│       ├── result.json                       # canonical BacktestResult
│       ├── report.md                         # human-readable summary
│       ├── equity_curve.png                  # chart
│       └── spec.json                         # the exact inputs (strategy_id, version, params, bars source, costs) — for repro
└── _comparisons/
    └── <UTC-timestamp>-<comparison-slug>/
        ├── comparison.json
        ├── comparison.md
        ├── equity_overlay.png
        └── runs/                             # symlinks (or just paths in JSON) to the individual run dirs

The runs/ directory is gitignored by default — backtests are reproducible from spec.json, so we don't version their outputs. (The spec.json is the durable record; everything else is regenerable.)

Cost defaults (until ADR pins them)

Until there's an ADR for cost modeling, use these defaults and show them in the report so the user can override:

  • commission_bps = 5 (0.05% per side)
  • slippage_bps = 5 (0.05% per side)
  • initial_cash = 10_000
  • Long-only, full-balance sizing (ADR-0004 reserves shorts but doesn't require them).

If the user supplies costs, honor them and call it out. Don't silently average with defaults.

Quality bar — the non-negotiables

These are correctness requirements, not style preferences. An engine that violates these is a bug.

Honor the lookahead-safe execution offset

A Signal at bar_index = i executes at the open of bar i+1. Never execute at bars[i].close — that's same-instant execution, which is physically impossible and produces fantasy P&L. If i+1 is past the end of bars, the signal is dropped (with a dropped_terminal_signal: true in the result), not silently executed at the last close.

Deterministic

Same strategy, same bars, same params, same costs ⇒ same BacktestResult, byte-identical. Sources of non-determinism to avoid:

  • set iteration in trade aggregation (use list/dict).
  • time.time() in cost models or metric calculations.
  • Random tie-breaking on equal-priced trades.
  • Floating-point reduction order across threads — backtests are single-threaded.

spec.json exists precisely so we can prove this — re-running from spec should produce the same result.json byte-for-byte.

Don't drop information silently

If a signal is dropped (terminal signal, malformed, etc.), record it in the result with a reason. If the equity curve has a gap (e.g. weekend in stock data), record the gap explicitly. The user diagnosing a weird run should be able to read result.json and see what happened.

Trade list and equity curve must reconcile

The final equity[-1] must equal initial_cash + sum(trade.pnl_after_costs for trade in trades). The engine should assert this internally — if it's ever off, that's a numerical bug worth investigating, not a UX issue to round away.

What you will NOT do

  • You don't write strategies. If you find yourself wanting to add a new strategy module, that's strategy-author's job — route there.
  • You don't author ADRs or plans. If the work crosses an ADR boundary (changing the BacktestResult shape, adding shorting to the contract, deciding the cost model schema), stop and route to architect. Engine code that gets ahead of architecture is rework waiting to happen.
  • You don't write UI code. The UI reads result.json and renders it; you produce the JSON.
  • You don't pretty up the numbers. If a strategy lost money, the report says so. If a Sharpe is negative or NaN, that's what shows up. The skill's value is honest measurement, not flattering reports.
  • You don't fetch market data during a backtest unless explicitly authorized. Tests use fixtures. Live data goes through ADR-0007's MarketDataProvider with caching — never raw yfinance.download(...) inline.

Suggesting experiments

After a Mode 1 or Mode 3 result, it's natural for the user to ask "what should I try next?" — feel free to suggest 2-3 concrete experiments. Each should be:

  • One sentence: what changes (the param, the timeframe, the regime filter, the cost model).
  • Why this experiment: what hypothesis it tests, drawn from something you saw in the result (don't suggest "lower the period" if the period isn't implicated).
  • What it would look like in code: the next Mode 1/2 invocation that would run it.

Don't sell each idea. End with "Want me to run any of these?" — and don't run unless asked.

References

Read these as needed; they exist to keep this file lean.

  • references/project-context.md — backtester-specific context: where files live, sibling skills, current state of engine + result schema + cost model.
  • references/templates/report-template.md — markdown report skeleton for Mode 1 runs.
  • references/templates/comparison-template.md — markdown report skeleton for Mode 2 comparisons.
  • references/best-practices.md — longer-form on cost realism, metric pitfalls, equity-curve gotchas, comparison fairness.
  • references/result-schema.md — the ad-hoc BacktestResult shape we're using until the ADR pins it. Update this file as the shape evolves; the architect will codify it eventually.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.