AGENTS.md

Guidance for agents and contributors working on datui, a terminal UI for tabular data built on Polars and Ratatui. Read this before changing code. The user docs live in docs/ and are published at https://derekwisong.github.io/datui/; do not restate them here.

Build, test, lint

./scripts/dev/test.sh setup                     # once: .venv (with CI's ruff and typos), hooks, test fixtures (scripts/setup_dev.py)
cargo build                                     # debug binary at target/debug/datui
cargo run -- data.parquet
./scripts/dev/test.sh full                      # cargo test --workspace; `ci` runs CI's nextest + doctests
./scripts/dev/test.sh fmt                       # root, fuzz/ and crates/datui-pyo3
./scripts/dev/test.sh lint                      # fmt --check + clippy -D warnings in all three; ruff, shellcheck, typos
./scripts/dev/test.sh integration fuzz_corpus_test  # fast fuzz corpus replay; run if you touched a parser or matcher

A cargo build binary needs the building machine's glibc or newer, so one built on a rolling distro will not start on a current LTS or in a VM. To run a build elsewhere, link it as the release does: cargo zigbuild --release -p datui --target x86_64-unknown-linux-gnu.2.28 (see docs/for-developers/packaging.md).

CI rejects unformatted code and any clippy warning, and runs test.sh lint. The pre-commit hooks (setup installs them, but not from a linked worktree, which shares the main checkout's) run only fmt --check and root clippy, so run lint after touching fuzz/, crates/datui-pyo3/, scripts or docs. Never add #[allow(clippy::…)] to get past a lint; fix it.

Tests that need data read tests/sample-data/, generated by the setup script (Polars, NumPy, pyarrow, fastavro, openpyxl in .venv). The harness finds .venv/bin/python on its own and regenerates them, under a lock, when the generator or its pins change. DATUI_CACHE_DIR redirects the cache; tests set it so they never touch ~/.cache/datui. Never write into tests/sample-data: other test processes may have its files mapped. Write to common::fixture_dir() or a tempfile directory.

Development iteration

Select a package and test target before a name filter. Do not run the full suite after every edit: a bare cargo test FILTER can still build unrelated test executables. scripts/dev/test.sh --help lists the scoped commands; --print shows the Cargo commands without running them.

During iteration, run the regression for the behavior changed and the relevant module/target. Broaden once the change is ready: cross-cutting App/event-loop, LazyFrame, loading/schema, shared config, dependency/feature, or test-harness changes require the full workspace suite. A test-target reorganization also requires full-suite coverage/discovery verification. For isolated changes, run related targets and required lint checks and let CI cover the workspace; state exactly what ran and what remains for CI. Documentation-only changes need documentation checks, not Rust tests. Run fuzz replay for parser/matcher changes. Repeat broad checks only after new changes or failures justify it.

Heavy test.sh runs (unit, integration, lint, clippy, msrv, ci, python, features, full, anything with --release) wait on a machine-wide lock that admits two at once (DATUI_TEST_HEAVY_SLOTS; 0 turns it off). When several agents share a machine, run full suites, workspace clippy and release builds through test.sh rather than cargo directly, so they queue instead of exhausting memory together.

Prepare fixtures once; do not clear target/, change compiler flags/features, or regenerate all fixtures to diagnose an ordinary failure. Live/cloud tests are opt-in. Use small in-memory data for logic tests and preserve regression coverage.

Root tests are mostly five executables by domain, tests/{app,home,data,config,repo}/, each a main.rs declaring modules; a filter selects a module (integration app loading::). Every test executable links the app (about 400 MiB, a second or more of link after each library edit), so add a test to the module that fits, never a new top-level file. A test that changes the process for everyone in it (an environment variable others read, fd 2, Polars' configuration) keeps its own target, as quality_spill_test and the cloud credential targets do; prefer giving the code the setting directly instead.

Tests that drive an App wait on the work with the helpers in tests/common/ (pump_open_until_loaded, drain_events, next_event, handle_until, work_pending, and wait_for_event in a loop that draws frames), or wait for the frame's content, never on a sleep or a quiet channel. The helpers run the frame's work (App::frame_work) once the events on hand are handled, as the run loop does. A sleep stays only where a real timer is the subject, with a comment saying so. Size statistical and large-data tests to the smallest input that still makes the assertion; they set their executable's run time. Measured costs and the layout's reasons: tests/ORGANIZATION.md; commands and helpers: docs/for-developers/tests.md.

Layout

crates/datui-lib/src/, by directory:

Root modules are infra the rest share: cache.rs, glyphs.rs, numfmt.rs, logging.rs, error_display.rs, notes.rs, find.rs, clipboard.rs, limits.rs.

How the app works

Everything is a LazyFrame until it is drawn. DataTableState keeps original_lf (as loaded), base_lf (after the active query, reshape or drill-down) and lf (after sidebar filters, sort and column order). Only the rows on screen plus a lookahead buffer are ever collected. When you change how a frame is built, go through install_query_result / install_base / reset_view_state rather than assigning fields by hand; those helpers keep the three frames, the schema and the view state consistent. Outside DataTableState the pipeline fields are private: read them through accessors, run steps with deferred (nothing collects; the App then calls spawn_async_collect), and plan a view's steps with try_transition, whose checkpoint roll_back restores if its rows fail.

One event loop, background work by generation. Keys (read by the one reader in app/terminal_input.rs), worker results and continuations all arrive as AppEvents on one mpsc channel; EventPump::run sleeps on it until something arrives or a deadline passes (spinner, flash), with no polling tick, and App::event handles each. A key is offered once the results behind it are handled; keys that arrive together and act at once share a frame, and a key that queues a follow-up gets its frame first (EventPump::burst_goes_on). A frame that changes nothing writes nothing. Anything polled rather than sent must wake the loop (AppEvent::Wake). Anything that touches data runs off the UI thread as a job of jobs::Jobs (App::spawn_job), which keeps one record per job: what it is (Job, carrying whatever the app needs of it), whether the user waits on it (its keys and status line), and whether its answer is still wanted. A worker returns its Answer or Err with the message for the user; that, or a panic, is its one outcome, announced by AppEvent::JobEnded. App::job_ended takes the outcome and the record together (each answered arm names the job and its answer; an answer under another job is logged and dropped), so a job holds the generation and the keys until its answer is handled. Advancing the generation, or Jobs::supersede, makes answers stale; App keeps no flags of its own for a job. Counts keep their own markers, and home-screen workers, keyed by place rather than generation, run inside an OwedAnswer that sends their in-flight marker an answer if they panic. Async cloud calls go through wait_on_runtime on the shared Tokio runtime. Never collect inside a render function.

An open has one owner. loading::Loader holds the open in flight, from the request to its first rows: its LoadId, where it was asked from, its Phase, what the loading screen says, and what it holds (its footer counter, whose cancel flag stops it; its download; the hold while a download is asked about). App passes it requests and worker answers (open, answered, failed, confirmed) and carries out the Step it returns, starting the next phase's job in the same handler, so the generation never reads free between phases. Load jobs carry the LoadId; an answer for a load that was replaced or put down, or for a phase it has left, is dropped with what it carries. The schema read builds the dataset with everything the open found, its download included, through DataTableState::with_open(OpenFacts); there are no load-time setters. install_dataset hands the counter to the dataset and records the recent; going home or another open retires the load. State kept per dataset is reset by its owner's reset_for_dataset, which install_dataset calls: add a field to that owner's reset, not to install_dataset. Add a phase to Phase and Step, not a flag or a continuation event to App. A worker that writes a temp file creates it through the load's unfinished::Writer::create, which claims it; the claim lives as long as the file, and ExitSweep, dropped after the app in run, removes what is still claimed. SIGTERM and SIGHUP end the binary's session as a quit does.

Keys typed while busy are queued. While a job the user waits on runs, an errand is between phases, or an open is on its way to its dataset, App::is_busy is true and the footer shows a spinner. EventPump (app/event_pump.rs, owned by run()) holds the keys typed meanwhile and replays them in order, one per loop iteration, once the app is idle. Ctrl-Q, Ctrl-C outside a text field, Ctrl-O and confirmation-modal keys act at once; so do q, the column cursor and help at the plain table view when nothing is held. Held keys are dropped when the screen they were typed at goes away. Change this through classify and its tests, not by gating keys in App::key.

Overlays. What is open over the table (a dialog, sidebar or screen such as Export, Info or Analysis) is one value, App::overlay (app/overlay.rs); keys go to it and it is drawn over the table. Open one with open_overlay, or open_over for one that goes back to what it was opened over (Export over Value Counts); leave through close_overlay, which drops the state it held for that opening and goes back. step_back leaves it without dropping its state, behind a confirmation or while it runs. A new dialog is a state struct in a *_modal.rs (app/modals/, or its feature's directory; it owns its focus), an Overlay variant with its arms (keys, keys_context, close_overlay) and its drawing in widgets/ or render/; it has no active flag. What it asks of the app when answered is a variant of Applied (app/applied.rs), sent as AppEvent::Applied so the frame that closes it is drawn first and carried out by App::apply; not an AppEvent of its own. The error and confirmation modals and the help stack over any overlay and keep their own.

Config is TOML, in layers: defaults in Default impls, imported files, ~/.config/datui/config.toml, -c KEY=VALUE, then flags. Each file is a partial ConfigLayer merged by presence. Every key is one entry in the option registry (crates/datui-cli/src/settings.rs); -c, datui config init and keys, the settings reference and flag help come from it. Follow docs/for-developers/adding-configuration-options.md. The S3 endpoint, keys and region come from the environment alone (effective_cloud); blank values count as unset.

Text input is one editor (widgets/textarea/) wrapped by widgets/text_input/ for theming, focus and history. Read the value with value(), write with set_value(). Editing keys follow readline.

Visual language

The full canon — component patterns, shapes, text rules, the keybind compatibility contract, and the acceptance checklist for UI changes — is the ui-style skill (.claude/skills/ui-style/SKILL.md). Load it before any UI work. The short version:

• One accent (accent, accent_bright) for keys, focused titles and the selection rail. gradient_start/gradient_end color the wordmark only.
• Three chrome tiers a few shades apart: controls_bg, table_header_bg, table_alternate_row.
• The current row is tinted (table_selected) and carries a ▎ rail; use the theme's highlight_style helper, never Modifier::REVERSED directly.
• The column cursor tints its cells (table_column_cursor), its header and the current cell (table_cell_cursor), through the theme's column_cursor_style and cell_cursor_style helpers.
• The status footer (render/footer.rs): thin rule, no fill, status left in pipeline order, position and the active mode's keys right, ? keys at rest. Keys in dialog footers are chips: key on the accent, label beside it. Section titles sit on a rule with a flat count chip.
• Column names take their type's color; a second header row names the type. Nulls are ∅ in dimmed.
• Every glyph has an ASCII twin in glyphs.rs; never use Nerd Font characters.
• Defaults are hex and degrade to 256-color or ANSI automatically. Every color is a slot in ColorConfig; no hardcoded Color:: in widgets.
• Labels are short and plain. No filler text in the UI.

Conventions

• Rust edition 2024, MSRV 1.95 (rust-version, checked in CI). cargo fmt, zero clippy warnings, no lint suppressions.
• Comments are brief and explain why, especially around Polars operations. Do not comment what the code already says.
• Result<T> for anything fallible; user-facing errors go to the error modal, non-critical ones (cache, history) log and continue.
• Do not downgrade a dependency without being asked. Do not add one for something the tree already has.
• American English everywhere: code, UI strings, docs, notes.
• One word per concept: docs/reference/glossary.md. tests/repo/wording.rs fails on a retired word in UI strings, the key registry, docs or the manpage.
• Docs: lead with the command or the key, then a table. No essays. Verify every claim against the code. Never link plans/ or other unpublished paths.
• Commit and PR text is terse. One idea per commit where practical.

When you add or change a feature

1. Update the screen's keys in the key registry (crates/datui-cli/src/keys.rs): a one-line line for the help, more for the man page and docs. Anything longer than a key line goes in the docs page.
2. Update the page in docs/user-guide/ or docs/formats/.
3. Add a test. Integration tests in tests/ build an App and pump events; look at tests/common/ and the existing tests for the pattern.
4. Run cargo run -p datui-cli --bin gen_docs -- write: it rewrites the generated references (options, settings, environment, keys, formats).
5. Every code block you add is runnable, and runs under scripts/docs/doc_examples.py, or is labeled ,template with <PLACEHOLDER>s. See docs/for-developers/documentation.md. Files an example needs are titled file blocks, never heredocs; sample-data generators are readable scripts.
6. If it is worth telling users, add a line to release-notes/v<next>.md, most important first, one plain sentence each.

Releasing

scripts/bump_version.py owns versions across the crates, the Python package and the lockfiles. The flow is documented at the top of that script and in docs/for-developers/packaging.md: release --commit, push, wait for CI, release --tag-only, then patch --commit (or minor) for the next cycle. The tag triggers publishing to GitHub releases, crates.io, PyPI, winget, the AUR and the apt repo, so it is the maintainer's call.