h5i-db

# h5i-db

> h5i-db is a high-performance, embedded, versioned analytical database for quantitative finance and time-series workloads. It runs full DataFusion SQL with native ASOF joins, OHLCV/VWAP rollups, time travel, and previewable mutations over immutable, time-sorted Parquet segments — driven from a CLI, Rust, or Python, and designed to be safe for AI agents. Written in Rust; Apache-2.0.

A database is a single directory on disk; there is no server. Every write is an atomic commit that produces a new immutable version, and any past version is readable in O(1). Storage is time-sorted and pruned by manifest statistics before I/O. Destructive changes (delete/replace ranges) can be staged as previewable plans and gated by a mutation policy. The CLI emits machine-readable output and structured errors with stable exit codes.

## Manual
- [Overview](https://db.h5i.dev/manual/): What h5i-db is, what it is for, and how the documentation is organized.
- [Installation](https://db.h5i.dev/manual/installation/): Install the h5i-db CLI and Python library, or build both from source.
- [Quickstart](https://db.h5i.dev/manual/quickstart/): From nothing to a queried, versioned market database in five commands, from the CLI or Python.
- [Migrating from DuckDB](https://db.h5i.dev/manual/migrating-from-duckdb/): Move tables from a .duckdb file into a versioned h5i-db store via Parquet, and the query differences to expect once they land.
- [Core concepts](https://db.h5i.dev/manual/concepts/): The mental model behind h5i-db: versions, segments, manifests, snapshots, previewable mutation plans, and the mutation policy.
- [CLI reference](https://db.h5i.dev/manual/cli/): Every command, flag, output format, and exit code of the h5i-db command-line tool.
- [Notebooks](https://db.h5i.dev/manual/notebooks/): In-terminal Jupyter notebooks whose kernel outlives the command, with %%sql cells that skip the interpreter entirely.
- [SQL reference](https://db.h5i.dev/manual/sql/): h5i-db's SQL beyond stock DataFusion: time travel with h5i(), ASOF joins, gapfill and resample, tail, time_bucket, vwap, ewma, and rolling sugar.
- [Agents & automation](https://db.h5i.dev/manual/agents/): The machine contract: structured output and errors, stable exit codes, resource limits as flags, and policy-gated review for AI agents and pipelines.
- [Quant workflows](https://db.h5i.dev/manual/quant/): Factor evaluation and performance tearsheets on pinned data: h5i_db.quant, its alphalens/empyrical parity, and the divergences that are deliberate.
- [Operations guide](https://db.h5i.dev/manual/operations/): Running h5i-db in production: backup and restore, vacuum and compaction cadence, disk-usage math, and the torn-HEAD recovery runbook.
- [Data on-ramp](https://db.h5i.dev/manual/data-onramp/): h5i_db.venues: turning vendor archives, bar files, trade dumps and live captures into the canonical tables a replay reads.
- [Backtesting](https://db.h5i.dev/manual/backtest/): Deterministic event-driven backtesting on versioned data: canonical tables, runs that live on forks, and settlement that refuses to book what a run never reached.

## Python API
- [Overview](https://db.h5i.dev/api/): The h5i_db Python library: install, the five-minute tour, data interchange, and error handling.
- [Database](https://db.h5i.dev/api/database/): h5i_db.Database reference: lifecycle, tables, writing, reading and SQL, time travel, forks, mutation plans, policy, and maintenance.
- [DataFrame builder](https://db.h5i.dev/api/dataframe/): Build queries as Python, not SQL strings: db.table(), lazy verbs, expressions, rolling and cross-sectional operators, and ASOF joins.
- [QueryResult & MutationPlan](https://db.h5i.dev/api/results-and-plans/): Converting query results to Arrow, pandas, and Polars; previewing and applying mutation plans.
- [Exceptions](https://db.h5i.dev/api/exceptions/): The typed error hierarchy: every h5i-db error carries a stable code, a hint you can act on, and a retryable flag.

## Cookbook
- [Cookbook](https://db.h5i.dev/cookbook/): Executed notebook tutorials for h5i-db: fundamentals, market data engineering, alpha research, risk & production, event-driven backtesting, prediction markets, performance analytics.

## Cookbook: Fundamentals
- [Quickstart: your first h5i-db market database](https://db.h5i.dev/cookbook/00_fundamentals/01_quickstart/): h5i-db is an embedded, versioned time-series database for quant workloads. There is no server to run.
- [Designing market data schemas](https://db.h5i.dev/cookbook/00_fundamentals/02_designing_market_data_schemas/): A table in h5i-db is an Arrow schema plus a time column, persisted as immutable, time-sorted Parquet segments under versioned manifests.
- [Ingestion patterns: five sources, one table](https://db.h5i.dev/cookbook/00_fundamentals/03_ingestion_patterns/): No desk gets its data from one place. The tick feed hands you Arrow batches, research notebooks live in pandas or polars, vendors drop Parquet, and some…
- [A SQL tour for quants](https://db.h5i.dev/cookbook/00_fundamentals/04_sql_tour_for_quants/): h5i-db's query layer is Apache DataFusion: full SQL with joins, CTEs and window functions.
- [Time travel and versioning: which version did my backtest see?](https://db.h5i.dev/cookbook/00_fundamentals/05_time_travel_and_versioning/): Every write to an h5i-db table is an atomic commit that produces a new immutable version. That covers append, write, delete and restore.
- [Previewable mutations: fix bad ticks without fearing the delete key](https://db.h5i.dev/cookbook/00_fundamentals/06_previewable_mutations/): Deleting or rewriting rows in a shared tick store is the scariest routine operation on a quant desk.
- [Streaming appends and tail(): a live feed on a versioned store](https://db.h5i.dev/cookbook/00_fundamentals/07_streaming_appends_and_tail/): An h5i-db table with an append-only history doubles as a message log. Every append is one commit, commits are strictly ordered, and a reader that…
- [Maintenance: verify, compact, vacuum](https://db.h5i.dev/cookbook/00_fundamentals/08_maintenance/): A versioned database makes an unusual bargain. It never overwrites data, so it accumulates manifests, segments and history. That is the feature.
- [The DataFrame builder: queries as Python objects](https://db.h5i.dev/cookbook/00_fundamentals/09_dataframe_builder/): db.table(...) starts a lazy query that you assemble with method calls instead of a SQL string. Nothing runs until a terminal call such as .collect().

## Cookbook: Market data engineering
- [OHLCV bars from tick data](https://db.h5i.dev/cookbook/01_market_data_engineering/01_ohlcv_bars/): Rolling ticks into bars is where most tick pipelines pick up their first bug: a bucket boundary off by one, a close taken in file order instead of event…
- [VWAP, TWAP and execution benchmarks](https://db.h5i.dev/cookbook/01_market_data_engineering/02_vwap_twap_execution/): Execution desks live and die by benchmark arithmetic: interval VWAP, TWAP, arrival price, slippage in basis points.
- [Regular grids for irregular markets: gapfill and resample](https://db.h5i.dev/cookbook/01_market_data_engineering/03_gapfill_resample/): Illiquid names do not trade every minute. Almost everything downstream wants a regular time grid anyway: covariance matrices, joins against liquid…
- [ASOF joins: trades vs quotes, signing and spreads](https://db.h5i.dev/cookbook/01_market_data_engineering/04_asof_join_trades_quotes/): Attaching the prevailing quote to every trade is the microstructure join. It powers trade signing, effective and realized spread measurement, TCA and…
- [Corporate actions: split adjustment without losing the tape](https://db.h5i.dev/cookbook/01_market_data_engineering/05_corporate_actions/): A stock split rewrites history. Every price before the effective date must be scaled, or every return computed across it is garbage.
- [Point-in-time fundamentals: killing lookahead bias with ASOF joins](https://db.h5i.dev/cookbook/01_market_data_engineering/06_point_in_time_fundamentals/): Fundamentals have two timestamps. periodend is the fiscal date the numbers describe.
- [Tick data cleaning: find it, preview the fix, keep the audit trail](https://db.h5i.dev/cookbook/01_market_data_engineering/07_tick_data_cleaning/): Raw vendor tick files arrive dirty. Fat-finger prints 10x off, zero prices from feed glitches, duplicated blocks from replayed packets, after-hours junk.
- [NBBO consolidation: best bid/offer across fragmented venues](https://db.h5i.dev/cookbook/01_market_data_engineering/08_nbbo_consolidation/): US equities trade on a dozen or more venues, each publishing its own top of book.

## Cookbook: Alpha research
- [Cross-sectional momentum: an honest monthly backtest](https://db.h5i.dev/cookbook/02_alpha_research/01_momentum_backtest/): The classic 12-1 momentum factor, end to end on real prices. ), and the signal table itself is versioned and snapshotted.
- [Pairs trading with a version-pinned data spine](https://db.h5i.dev/cookbook/02_alpha_research/02_pairs_trading/): A cointegration pair strategy on real prices. We scan candidate pairs with an Engle-Granger test, build a rolling hedge ratio, compute the spread z-score…
- [EWMA volatility and vol-targeted position sizing](https://db.h5i.dev/cookbook/02_alpha_research/03_ewma_vol_targeting/): RiskMetrics-style EWMA volatility is the workhorse conditional-vol estimate on every risk desk, and h5i-db ships it as a native SQL window function…
- [Realized volatility from ticks: signature plots, jumps, overnight risk](https://db.h5i.dev/cookbook/02_alpha_research/04_realized_volatility/): Realized variance, the sum of squared intraday returns, is the standard nonparametric vol estimate.
- [Building a point-in-time factor library](https://db.h5i.dev/cookbook/02_alpha_research/05_factor_construction/): Equity factors die by lookahead. A B/P ratio computed with a book value the market had not seen yet will backtest beautifully and trade terribly.
- [Event studies: CARs with ASOF-aligned announcement dates](https://db.h5i.dev/cookbook/02_alpha_research/06_event_study/): The classic event-study pipeline computes market-model abnormal returns and cumulative abnormal returns around announcements.
- [Order flow imbalance: does signed volume predict returns?](https://db.h5i.dev/cookbook/02_alpha_research/07_order_flow_imbalance/): Order flow imbalance is the excess of buyer-initiated over seller-initiated volume, and it is the workhorse microstructure signal.
- [Intraday seasonality: volume U-shape, volatility smile, spread decay](https://db.h5i.dev/cookbook/02_alpha_research/08_intraday_seasonality/): Almost every execution and alpha model conditions on time of day. Volume concentrates at the open and close, volatility peaks in the first hour, spreads…
- [Lead-lag discovery: cross-correlations on irregular ticks](https://db.h5i.dev/cookbook/02_alpha_research/09_lead_lag/): Who moves first? Lead-lag analysis pairs related instruments: index against futures, ADR against home listing, correlated FX crosses.
- [Portfolio rebalancing with versioned holdings](https://db.h5i.dev/cookbook/02_alpha_research/10_portfolio_rebalancing/): A portfolio book is the canonical versioned dataset. " become version queries rather than spreadsheet archaeology.
- [Retrieval-augmented forecasting: historical analogs as a knowledge base](https://db.h5i.dev/cookbook/02_alpha_research/11_retrieval_augmented_forecasting/): "What happened the last twenty times the tape looked like this?" Analog forecasting is one of the oldest ideas in the business. Retrieval-augmented…
- [Trend following: each asset against its own past](https://db.h5i.dev/cookbook/02_alpha_research/12_trend_following/): Recipe 02/01 ranks assets against each other and buys the winners. This one never compares two assets at all: each is measured against its own history…
- [Short-horizon mean reversion, and the edge it would have needed](https://db.h5i.dev/cookbook/02_alpha_research/13_mean_reversion/): Over a year, winners keep winning. Over a few days the textbook says they hand it back: a name that fell hard against its peers bounces, because part of…

## Cookbook: Risk & production
- [VaR and Expected Shortfall with an auditable risk table](https://db.h5i.dev/cookbook/03_risk_and_production/01_var_expected_shortfall/): A risk number nobody can reproduce is a liability.
- [Reproducible backtests: pin the data, not just the code](https://db.h5i.dev/cookbook/03_risk_and_production/02_reproducible_backtests/): Every quant team has lived this incident. A backtest from March cannot be reproduced in July.
- [EOD snapshots and the audit trail regulators actually ask for](https://db.h5i.dev/cookbook/03_risk_and_production/03_eod_snapshots_audit/): The question that arrives eighteen months later is never "what is the price now". ".
- [Data-quality gates: staging, policy, and previewable remediation](https://db.h5i.dev/cookbook/03_risk_and_production/04_data_quality_gates/): The worst place to discover a broken vendor file is in the P&L meeting.
- [A crash-safe paper-trading loop with full order attribution](https://db.h5i.dev/cookbook/03_risk_and_production/05_live_paper_trading_loop/): The hard part of a live loop is not the strategy. It is answering, a week later, "why did we send that order?".
- [Multi-writer coordination: optimistic locking, conflicts, and retries](https://db.h5i.dev/cookbook/03_risk_and_production/06_multi_writer_conflicts/): An h5i-db database is a directory, and nothing stops two processes from opening it at the same time: a feed handler and a corrections job, or two…
- [Options: implied-vol surfaces as versioned marks](https://db.h5i.dev/cookbook/03_risk_and_production/07_options_iv_surface/): A vol desk's surface is not one object. It is a sequence of marks: EOD snapshots, intraday re-marks, corrections.
- [FX and crypto: 24/7 data without an exchange session](https://db.h5i.dev/cookbook/03_risk_and_production/08_fx_crypto_24_7/): Equity tooling leans on the session. The exchange defines "the day", the open and the close.
- [Fixed income: versioned curve marks, restatements, carry & rolldown](https://db.h5i.dev/cookbook/03_risk_and_production/09_fixed_income_curves/): A rates desk's core dataset is small but unforgiving. One par curve per mark date, and every number on it feeds risk, P&L and client marks.
- [Performance tuning: pruning, projection, commit granularity, caches](https://db.h5i.dev/cookbook/03_risk_and_production/10_performance_tuning/): h5i-db stores each table as immutable, time-sorted Parquet segments under a versioned manifest.
- [arrival-delta: which of last night's backtests actually held up?](https://db.h5i.dev/cookbook/03_risk_and_production/11_arrival_delta/): A research agent runs forty backtests overnight. So does a parameter sweep, or a junior with a for-loop. In the morning there are forty Sharpes.

## Cookbook: Event-driven backtesting
- [Your first event-driven backtest](https://db.h5i.dev/cookbook/04_event_driven_backtesting/01_first_event_driven_run/): The backtests in section 02 are vectorized. Compute a signal for every date, multiply it by the return that followed, and sum.
- [Stress-test execution assumptions](https://db.h5i.dev/cookbook/04_event_driven_backtesting/02_execution_realism/): Every backtest contains execution assumptions, and most of them are never written down. Fills happen at the price you asked for.
- [Operate reproducible backtests](https://db.h5i.dev/cookbook/04_event_driven_backtesting/03_reproducible_backtest_operations/): A backtest is a claim about the past, and it is worth what the evidence behind it is worth.
- [A production data contract for Kaggle Polymarket L2](https://db.h5i.dev/cookbook/04_event_driven_backtesting/04_kaggle_polymarket_data_contract/): A public dataset is not a research input. It is a pile of files carrying whatever timestamps, units and duplicate rows the recorder happened to produce…
- [Causal signal replay on real Polymarket books](https://db.h5i.dev/cookbook/04_event_driven_backtesting/05_kaggle_polymarket_replay/): Synthetic data proves the plumbing works. It cannot tell you whether a strategy works, because what you find in it is the structure the generator put…
- [Order lifecycle and account risk](https://db.h5i.dev/cookbook/04_event_driven_backtesting/06_order_lifecycle_and_risk/): Most backtests model an order as a single event. It is sent and it fills. Real orders have a life.
- [Path-dependent Python strategies](https://db.h5i.dev/cookbook/04_event_driven_backtesting/07_python_strategy_callbacks/): Some strategies cannot be written as a table of order intent. A rule that enters only once the previous position is confirmed closed, or that waits…
- [From a vectorized equity backtest to an event-driven one](https://db.h5i.dev/cookbook/04_event_driven_backtesting/08_equity_bars_to_event_driven/): Section 02 backtests a monthly momentum rule by multiplying a signal by the return that followed. Section 04 has so far replayed prediction markets.
- [Market making: inventory, latency, and being run over](https://db.h5i.dev/cookbook/04_event_driven_backtesting/09_market_making_inventory/): Every other recipe in this section takes liquidity. A market maker supplies it, and the job is different in kind. There is no forecast.
- [Execution algorithms and the cost of not finishing](https://db.h5i.dev/cookbook/04_event_driven_backtesting/10_execution_algorithms/): Recipe 01/02 measures VWAP and TWAP as benchmarks. This one trades to them. The order is 20,000 shares to buy in a name that shows a few hundred at the…
- [Calibrating costs from your own fills](https://db.h5i.dev/cookbook/04_event_driven_backtesting/11_calibrating_costs_from_fills/): Recipe 04/02 varies fees, slippage and latency to see whether a conclusion survives them.
- [Searching a strategy space without fooling yourself](https://db.h5i.dev/cookbook/04_event_driven_backtesting/12_searching_a_strategy_space/): A backtest that has been run once is a measurement. A backtest that has been run four hundred times and reported once is a selection, and the number on…
- [Replaying an account's ledger](https://db.h5i.dev/cookbook/04_event_driven_backtesting/13_replaying_an_account_ledger/): Every other recipe in this section asks what a strategy would have done. This one asks the strictest question a backtester can be asked: here are the…

## Cookbook: Prediction markets
- [Binary parity and the fee curve](https://db.h5i.dev/cookbook/05_prediction_markets/01_binary_parity_and_fee_curves/): A binary event contract has an arithmetic identity: YES and NO settle to exactly 1.00 between them. When both offers sum to less than 1.00 you can buy…
- [Is the market's probability any good?](https://db.h5i.dev/cookbook/05_prediction_markets/02_probability_calibration/): A prediction market quotes a probability, so it can be scored the way any forecast is scored.
- [Trading the favorite-longshot bias](https://db.h5i.dev/cookbook/05_prediction_markets/03_favorite_longshot_bias/): The oldest documented anomaly in event contracts is that longshots are overpriced and favorites underpriced.
- [What your data can and cannot tell you about execution](https://db.h5i.dev/cookbook/05_prediction_markets/04_execution_fidelity_and_depth/): Prediction-market research usually runs on periodic book snapshots, because that is what the public APIs give you.
- [Two ways a prediction-market backtest lies to you](https://db.h5i.dev/cookbook/05_prediction_markets/05_settlement_and_selection_risk/): The first is settlement. A replay that stops before a market resolves still holds a position, and marking it to the eventual winner books money nobody…
- [From a vendor mirror to canonical tables](https://db.h5i.dev/cookbook/05_prediction_markets/06_vendor_data_onramp/): Most prediction-market research starts with a directory of vendor Parquet already on disk: hourly full-feed archives, or per-day channel files.
- [The whole loop, once](https://db.h5i.dev/cookbook/05_prediction_markets/07_end_to_end_workflow/): Vendor files to a decision, with nothing skipped: ingest, pin, build a strategy from the pack, search it walk-forward with the holdout spent on a…
- [The whole loop on real Polymarket books](https://db.h5i.dev/cookbook/05_prediction_markets/08_real_polymarket_end_to_end/): Recipe 05/07 runs the pipeline on a synthetic panel, which proves the plumbing and nothing about the world.

## Cookbook: Performance analytics
- [Tearsheets and performance statistics](https://db.h5i.dev/cookbook/06_performance_analytics/01_tearsheets_and_performance_stats/): Every recipe so far ends with a P&L number and a chart drawn by hand. That is fine for one strategy and hopeless for twenty, and it is how two people end…
- [Factor panels: IC, quantiles, turnover](https://db.h5i.dev/cookbook/06_performance_analytics/02_factor_panels_and_reports/): Recipe 02/05 builds a factor and scores it with pandas and scipy. That is the right way to learn what an information coefficient is, and the wrong way to…
- [Cross-validation that a time series survives](https://db.h5i.dev/cookbook/06_performance_analytics/03_cross_validation_without_leakage/): A backtest measures what happened. Cross-validation is supposed to answer a harder question: was the parameter that won chosen for a reason, or was it…
- [Sweeps, verification, and what a restatement did to your alpha](https://db.h5i.dev/cookbook/06_performance_analytics/04_sweeps_verification_restatements/): Three questions arrive in order at the end of a piece of research. What did the parameter grid say? Can somebody else reproduce the number?

## Optional
- [Full documentation in one file](https://db.h5i.dev/llms-full.txt): the entire manual and Python API reference as plain markdown.
- [GitHub repository](https://github.com/h5i-dev/h5i-db): source, issues, and README.
- [Design document](https://github.com/h5i-dev/h5i-db/blob/main/DESIGN.md): storage engine and query-layer internals.
- [Benchmark methodology](https://github.com/h5i-dev/h5i-db/blob/main/benchmarks/RESULTS.md): full benchmark setup and results.

## Related projects
- [h5i](https://h5i.dev/): the sibling project from the same authors. An auditable workspace layer for AI coding agents: isolated Git worktrees, sandbox policy, prompt and model provenance, and a neutral verifier, all stored under `refs/h5i/*`.
- [h5i docs index](https://h5i.dev/llms.txt): the structured, link-first index for the h5i manual, guides, and engineering blog.