agentleFS
tatashbartatashbar/Economic-Modeling-Studio/.agents/skills/build-deep-forecaster/SKILL.md

build-deep-forecaster

Skill4 starsChanged 6 days agoMITView on GitHub

Builds a deep-learning or foundation-model forecaster with rolling-origin evaluation against statistical benchmarks.

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 6 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If a required input is ABSENT, that part is NOT ASSESSED — NO DATA: do not estimate or invent it.

    line 26
  2. Very specific:

    Collaboration: Follow .claude/docs/collaboration-protocol.md: ask questions first, present options, get approval before writing files, and …

    line 28
  3. Distinctive:

    Use data vintages where available.

    line 49
  4. Distinctive:

    Check whether the foundation model's training data may overlap the evaluation period

    line 53
  5. Distinctive:

    Document where the model wins, loses and fails.

    line 56
  6. Distinctive:

    After the protocol: confirm targets, horizons and the information set

    line 61
  7. Distinctive:

    After benchmarks: confirm the bar the model must clear

    line 62
  8. Distinctive:

    After evaluation: decide on ensemble inclusion

    line 63

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- Generated from .claude/skills/build-deep-forecaster/SKILL.md by tools/sync_agents.py. Edit the source, then run the script. -->

/build-deep-forecaster — Build Deep Forecaster

Category: Build (AI) · Primary agent: time-series-deep-learning-specialist

Builds a deep-learning or foundation-model forecaster with rolling-origin evaluation against statistical benchmarks.

Arguments: whatever the user wrote after the command name.

Purpose

Build a deep-learning or foundation-model forecaster and find out honestly whether it beats the statistical benchmarks that macro forecasters already use. The output is a forecaster evaluated in pseudo-real time with rolling origins, scored on point and density accuracy and calibration, with a documented account of where it wins, loses and fails.

Use it when there are many series, likely nonlinearities or mixed frequencies, or when a pretrained time-series model should be tested on a country's data.

Do not use it when the sample is short and a BVAR or DFM is the natural tool (/estimate-bvar, /nowcast), or when structural interpretation is required (/build-semi-structural).

Before you start

  1. Check required inputs. For each input below record FOUND or ABSENT (never 'assumed'):
    • Data sheet
    • Benchmark models (AR, BVAR or DFM) If a required input is ABSENT, that part is NOT ASSESSED — NO DATA: do not estimate or invent it; name what is missing and which skill produces it. If all required inputs are absent, stop and say so.
  2. Check confidentiality. Identify the data classification level of the inputs (.claude/docs/confidentiality-and-ai-use.md). Do not proceed with restricted or confidential data in a way the policy does not allow.
  3. Collaboration. Follow .claude/docs/collaboration-protocol.md: ask questions first, present options, get approval before writing files, and update production/session-state/active.md after each approved section.

Agents

  • time-series-deep-learning-specialist — primary agent; coordinates the work and integrates results
  • deep-learning-architect — Designs deep learning architectures for macro-financial data
  • forecast-evaluation-specialist — Evaluates forecasts
  • dfm-nowcasting-specialist — Builds dynamic factor models and nowcasts

Each agent is defined in .claude/agents/<name>.md. If your tool runs subagents, run these agents as subagents, in parallel where the steps are independent; otherwise read each agent's file and apply its instructions in turn. Subagents only write to the paths named in your instructions.

Key design decisions (present options, let the user choose)

DecisionTypical optionsWhat it changes
ArchitectureLSTM/GRU, TCN, transformer/TFT, N-BEATS, foundation modelCapacity and data needs
Global versus localone model across many series or one per seriesSample size effectively available
Outputspoint, quantiles, full densityRisk analysis capability
Training windowexpanding, rollingAdaptation to breaks
BenchmarksAR, BVAR, DFM, simple combinationsBar for success

Workflow

  1. Define target, horizons, information set and the rolling-origin protocol before training
    • Use data vintages where available; record release calendars
  2. Build benchmarks first (AR, BVAR, DFM) and record their scores
  3. Train candidate architectures including a pretrained foundation model; log every run
    • Tune only on validation windows; record seeds and hyperparameters
    • Check whether the foundation model's training data may overlap the evaluation period
  4. Evaluate point and density forecasts; test calibration
    • Equal-accuracy tests against benchmarks; PIT and coverage
  5. Document where the model wins, loses and fails; propose an ensemble if it adds value
    • Report by subperiod, including stress periods
  6. Register the model as a lego set (/lego-map): a baseplate plus bricks with declared studs (variables, units, timing) and stubs, so components can later be attached, detached or swapped

Checkpoints with the user

  • After the protocol: confirm targets, horizons and the information set
  • After benchmarks: confirm the bar the model must clear
  • After evaluation: decide on ensemble inclusion

Common pitfalls

  • Random splits or look-ahead in scaling and features
  • Weak benchmarks
  • Reporting the best seed
  • Averages that hide failures in stress periods

Output

models/ai/<name>/ and docs/models/<name>/experiment-log.md (+ model.lego.json)

The model folder holds code, benchmark scripts, training logs and evaluation results; the experiment log records every run with seeds, data vintages and scores; the lego manifest registers the forecaster as a satellite brick. Every run saves its results to output/<run-id>/ in the model folder, with a run log (.claude/rules/model-code.md).

Gate / verdict

  • Benchmarks were built first
  • Results use time-aware splits and pseudo-real-time data
  • Calibration reported
  • Lego manifest passes python tools/lego.py check
  • Performance is reported by subperiod and across seeds
  • Foundation-model data overlap is assessed

Suggested next steps

  • /validate-model
  • /ai-audit
  • /lego-map

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.