risk-aware-trading-agents / site

# Risk-Aware Trading Agents: the full text for language models

> A benchmark for LLM trading agents, with news replayed by the minute it was published: three frozen US stock markets (the 2020 crash, the 2022 bear market and 2026), a public leaderboard (rta-bench-1) that scores each model blind and with names, apart before and after its training cutoff, and every result a JSON file that replays to the same numbers. The results are based on 65,863 headlines and SEC filings replayed by publication time; the headline data has license restrictions and is not distributed with the package. Open source, MIT license. Nothing in it is investment advice.

This file gathers what the website says about the benchmark, its data, how to run and submit a model, how it compares with other benchmarks, the current results and common questions. Website: https://jackson-cmd.github.io/risk-aware-trading-agents/ (Chinese: https://jackson-cmd.github.io/risk-aware-trading-agents/zh/). Code: https://github.com/jackson-cmd/risk-aware-trading-agents. Package: `pip install risk-aware-trading-agents`. Data and entries on Hugging Face: https://huggingface.co/datasets/jacksonwangpro1/rta-bench-llm-trading-benchmark.

- Leaderboard models so far (last update 2026-11-14, 67 entries): claude-haiku-4.5 (Anthropic, training cutoff 2025-07), claude-haiku-5.5 (Anthropic, training cutoff 2026-06), claude-sonnet-4.6 (Anthropic, training cutoff 2026-01), gemini-3.5-flash (Google, training cutoff 2025-01), llama3.2 (Meta, training cutoff 2023-12), gpt-5.6-luna (OpenAI, training cutoff 2026-02-16).

## The benchmark: rta-bench-1

rta-bench-1 is a benchmark for LLM trading agents with news. It has six tasks: three frozen periods of the US stock market, each played blind and with the real names.

- Periods: 2026 (2 January to 11 November 2026, a rising market with a spring sell-off: SPY +12.1%, with a -8.9% drawdown in March); 2022 (the rate-hike bear market: SPY -18.2%, the Nasdaq 100 -32.6%); 2020 (the COVID crash and rebound: SPY -34% in 23 trading days, VIX closing at 82.7, then +18.3% for the year).
- Settings, the same for every model: the 56 stocks of the megacap list in 11 sectors; one decision after every fifth close, 44 to 51 decisions per task; at most 10 positions; up to three headlines per stock that were published before the close; orders fill at the next open with realistic costs (5 bps plus a tenth of the overnight gap plus square-root market impact, and no fill above a share of the average volume); long only; $100,000.
- Seeds: each task is run with seeds 0, 1 and 2, one entry per seed. A seed fixes the blind aliases and, where the API takes one, the sampling seed. The board shows the mean over a model's seeds and the range.
- Clean and may remember: a model may have read about a period that falls before its training cutoff (the training-data cutoff on the vendor's model page, not the release date). Scores from the days after a model's cutoff are clean and ranked by the Sharpe ratio over those days, next to SPY and the equal-weight portfolio of the 56 stocks over the same days. Scores from before the cutoff are listed apart as may remember.
- Blind mode: tickers, company names and the dates of the observation are replaced with aliases and prices are rebased to 100; since 13 November 2026 dates and dollar amounts inside headlines are masked too. A sector, a price path and an unusual headline together may still give a stock away.
- Memory probe: the model sees random moments of a period twice, with the same prices, indicators and headlines, once named and once blind with names, people, products and dates masked, and says each time how likely the stock is to beat SPY over the next 20 trading days. The score is how much better its answers rank the actual winners when it is told who and when (the AUC gap, named minus blind) on the moments before its training cutoff; the moments after the cutoff are the control.
- How each model makes money: each blind run is analyzed for its average exposure to stocks, its market beta, the part of its return that exposures to four factors explain (market, size, growth, momentum), alpha with a Newey-West t-statistic, the sectors that made and lost the most, the average holding time, how the stocks it bought did against SPY over the next five trading days, its hit rate, the rank IC of its decisions, and the share of decisions whose reason cites news.
- Checks: every run passes a look-ahead check (with the prices and news after a cut point replaced by random ones, everything the model was shown before that point must stay the same). An entry holds every reply of the model, and continuous integration replays the replies through the backtest; the numbers must match.

## The data

Each period has daily bars from two years before its test window for 64 large-cap US stocks and 25 ETFs (split-adjusted prices, dividends, splits and a total-return factor), the VIX term structure (VIX9D, VIX, VIX3M) and VVIX, and the macro calendar (FOMC, CPI, PPI, jobs reports). The benchmark replays the period's news with its publication time, a headline counting from the first close after it was published; the results are based on that news, and the headline data has license restrictions and is not distributed with the package.

- 2026: prices from 2024-01-02 (719 trading days, 69,738 rows); 16,945 news items: 16,249 headlines (titles only) and 696 SEC 8-K filings with SEC's acceptance time.
- 2022: prices from 2020-01-02 (756 trading days, 72,466 rows); 29,300 headlines.
- 2020: prices from 2018-01-02 (756 trading days, 71,490 rows); 19,618 headlines.

The price files are in the package (`src/riskaware/research/data/market_YYYY.csv.gz`) with a manifest of SHA-256 hashes per period; `bundled_market("2022")` loads a period in Python. The prices of the three periods are also on Hugging Face as Parquet: https://huggingface.co/datasets/jacksonwangpro1/rta-bench-llm-trading-benchmark. Prices come from Yahoo Finance daily bars and are adjusted for every split up to November 2026. The stock lists are today's large caps (survivorship bias), so compare with the equal-weight portfolio, not only with SPY.

## Run and submit a model

```
pip install risk-aware-trading-agents
riskaware research bench --llm openai:gpt-5.6-luna      # six tasks x seeds 0, 1, 2; --task, --seed for fewer
riskaware research submit RUN_ID --model gpt-5.6-luna --cutoff 2026-02 --provider OpenAI
riskaware research leaderboard                          # the table, from leaderboard/entries in the repository
```

Any model the package can call works: OpenAI-compatible APIs (`openai:`, or `compat:` with `--base-url`), Anthropic, local models through Ollama (`ollama:`, with a 16,384-token context), and the Claude Code, Codex and Gemini command-line tools. From Python, `bench.run_task(policy, "2026-blind")` takes any function from an observation to orders. `submit` refuses a run with other settings, other data or a failed look-ahead check. Then open a pull request that adds the new file in `leaderboard/entries/`. One task is 44 to 51 decisions of 11,000 to 15,000 input tokens each, so all six tasks are 3 to 4.5 million input tokens per seed.

## How it compares with other benchmarks

From each paper's arXiv abstract page, read on 13 November 2026 ("not described" means the abstract does not say; the full paper may):

- StockBench (arXiv 2510.02209): multi-month stock trading environments; news is among the daily signals, its timing not described; called contamination-free; released as open source.
- CLQT (arXiv 2606.29771): a year-long backtest plus a four-week live paper-trading track on post-cutoff data, with point-in-time data access through a "TimeGate"; news not described.
- LiveTradeBench (arXiv 2511.03628): live, US stocks and Polymarket, 21 LLMs over 50 days, with prices and news streamed live.
- AI-Trader (arXiv 2512.10971): live, US stocks, China A-shares and crypto; agents search for information themselves; code and evaluation data on GitHub.
- InvestorBench (arXiv 2412.18174): stocks, crypto and ETFs with 13 LLM backbones; open-source multi-modal datasets; news timing and periods not described.
- FinSABER (arXiv 2505.07078): backtests over two decades and more than 100 symbols, aimed at survivorship and data-snooping bias; news not described.
- BlindTrade (arXiv 2603.17692): historical backtests with tickers and company names anonymized and negative controls; out of sample in 2025 up to 1 August, extended over 2024-2025.
- Agent Market Arena (arXiv 2510.11695): live and continuous, crypto and stocks, with expert-checked news.

What rta-bench-1 puts together: news replayed by the minute it was published over three frozen periods, every model run both blind and named, scores split at each model's training cutoff, and each result kept as one file that replays. Live benchmarks answer a different question: how a model trades today, on news nobody has seen. Details: https://jackson-cmd.github.io/risk-aware-trading-agents/llm-trading-benchmarks.html

## Current results

Last update 2026-11-14: 67 entries from 6 models, blind runs, the mean over the seeds each model has so far. Clean: from the model's training cutoff to the end of the period, next to SPY and the equal-weight portfolio over the same days. May remember: the whole period, before the cutoff. Tables: https://jackson-cmd.github.io/risk-aware-trading-agents/leaderboard.html; data: https://jackson-cmd.github.io/risk-aware-trading-agents/assets/data/leaderboard.json

2026 (2026-01-02 to 2026-11-11):
- claude-haiku-4.5 (Anthropic, cutoff 2025-07, 1 seed), clean: +7.2% from 2026-01-02, Sharpe 0.42; SPY +12.1%, equal weight +11.4%
- claude-sonnet-4.6 (Anthropic, cutoff 2026-01, 1 seed), clean: +6.2% from 2026-01-30, Sharpe 0.34; SPY +10.7%, equal weight +10.6%
- gpt-5.6-luna (OpenAI, cutoff 2026-02-16, 3 seeds), clean: +4.5% from 2026-02-13, Sharpe 0.34; SPY +12.3%, equal weight +10.7%
- gemini-3.5-flash (Google, cutoff 2025-01, 3 seeds), clean: +6.5% from 2026-01-02, Sharpe 0.33; SPY +12.1%, equal weight +11.4%
- llama3.2 (Meta, cutoff 2023-12, 3 seeds), clean: -0.8% from 2026-01-02, Sharpe -0.32; SPY +12.1%, equal weight +11.4%
- claude-haiku-5.5 (Anthropic, cutoff 2026-06, 1 seed), clean: -0.6% from 2026-06-30, Sharpe -0.99; SPY +2.0%, equal weight +5.7%
- Whole period: SPY buy and hold +12.1%, equal weight of the 56 stocks +11.4%

2022 (2022-01-03 to 2022-12-30):
- claude-sonnet-4.6 (Anthropic, cutoff 2026-01, 1 seed), may remember: +0.4%, Sharpe -0.03, +12.7% against equal weight
- claude-haiku-5.5 (Anthropic, cutoff 2026-06, 1 seed), may remember: +1.4%, Sharpe -0.03, +13.7% against equal weight
- claude-haiku-4.5 (Anthropic, cutoff 2025-07, 1 seed), may remember: -5.3%, Sharpe -0.44, +7.0% against equal weight
- llama3.2 (Meta, cutoff 2023-12, 3 seeds), may remember: -1.6%, Sharpe -0.62, +10.8% against equal weight
- gpt-5.6-luna (OpenAI, cutoff 2026-02-16, 3 seeds), may remember: -5.0%, Sharpe -0.96, +7.4% against equal weight
- gemini-3.5-flash (Google, cutoff 2025-01, 3 seeds), may remember: -10.7%, Sharpe -0.97, +1.6% against equal weight
- Whole period: SPY buy and hold -18.6%, equal weight of the 56 stocks -12.4%

2020 (2020-01-02 to 2020-12-31):
- claude-sonnet-4.6 (Anthropic, cutoff 2026-01, 1 seed), may remember: +25.4%, Sharpe 1.38, -14.7% against equal weight
- claude-haiku-4.5 (Anthropic, cutoff 2025-07, 1 seed), may remember: +16.9%, Sharpe 0.89, -23.1% against equal weight
- claude-haiku-5.5 (Anthropic, cutoff 2026-06, 1 seed), may remember: +7.4%, Sharpe 0.77, -32.7% against equal weight
- gemini-3.5-flash (Google, cutoff 2025-01, 3 seeds), may remember: +16.4%, Sharpe 0.75, -23.7% against equal weight
- llama3.2 (Meta, cutoff 2023-12, 3 seeds), may remember: +10.6%, Sharpe 0.73, -29.5% against equal weight
- gpt-5.6-luna (OpenAI, cutoff 2026-02-16, 3 seeds), may remember: -0.5%, Sharpe 0.09, -40.6% against equal weight
- Whole period: SPY buy and hold +17.2%, equal weight of the 56 stocks +40.1%

## Questions and answers

What is rta-bench-1?
A benchmark for LLM trading agents with news. Each model trades the same 56 large-cap US stocks through three frozen periods: 2026 to date, the 2022 bear market and the 2020 crash. After every fifth close it sees prices, indicators and the headlines published before that close; its orders fill at the next open, with realistic trading costs. It plays each period blind and with the real names, with seeds 0, 1 and 2.

How does the benchmark avoid look-ahead bias in news?
Every headline and filing carries the minute it was published, and a day's observation holds only the news published before that day's close (16:00 New York time, 13:00 on early-close days); orders fill at the next open. Every run also passes a look-ahead check: with the prices and news after a cut point replaced by random ones, everything the model was shown before that point must stay the same.

Why does joining news by calendar date matter?
A headline published after the close, joined to that day's closing price, lets a backtest act on tomorrow's news today. In the site's example, the same headline rule makes 32 bp a trade with a calendar-date join and loses 6.7 bp a trade when news is replayed by publication time with fills at the next open.

What do "clean" and "may remember" mean?
A model may have read about a period that falls before its training cutoff. Scores from the days after a model's training-data cutoff are clean: they are ranked by the Sharpe ratio over those days, next to SPY and an equal-weight portfolio over the same days. Scores from before the cutoff are listed apart as may remember, next to a memory probe.

What is blind mode, and why is each period run twice?
Blind mode replaces tickers, company names and dates with aliases, rebases prices to 100, and masks dates and dollar amounts inside headlines. Running the same model blind and with names shows what the names are worth to it: after the training cutoff that is knowledge about the companies; before it, it may also be memory of what happened.

How do I submit a model?
Run `riskaware research bench --llm openai:gpt-5.6-luna` with any model the package can call (six tasks, seeds 0, 1 and 2), then `riskaware research submit RUN_ID --model gpt-5.6-luna --cutoff 2026-02`, and open a pull request that adds the new file in `leaderboard/entries/`. The entry holds every reply of the model; continuous integration replays them through the backtest and the numbers must match.

## Links

- Leaderboard: https://jackson-cmd.github.io/risk-aware-trading-agents/leaderboard.html
- Benchmarks compared: https://jackson-cmd.github.io/risk-aware-trading-agents/llm-trading-benchmarks.html
- News backtesting without look-ahead bias: https://jackson-cmd.github.io/risk-aware-trading-agents/news-backtesting.html
- Point-in-time market and news data: https://jackson-cmd.github.io/risk-aware-trading-agents/point-in-time-market-data.html
- News factors: https://jackson-cmd.github.io/risk-aware-trading-agents/news-factors.html
- Leaderboard documentation: https://github.com/jackson-cmd/risk-aware-trading-agents/blob/main/docs/leaderboard.md
- Results as JSON: https://jackson-cmd.github.io/risk-aware-trading-agents/assets/data/leaderboard.json