AGENTS.md

This file provides guidance to AI agents such as Claude Code when working with code in this repository.

Project Overview

syntaqlite is a suite of tools for tokenizing, parsing, and formatting SQLite SQL queries. Key principles:

• Fast: High-performance for large/autogenerated files
• Accurate: Uses SQLite's own grammar (Lemon parser generator) for 100% compatibility
• Extensible: Designed to support SQLite dialects (databases that use SQLite as a base and add extensions)

The project uses autogenerated code from SQLite's grammar with selective human-written AST folding decisions to maintain resilience against upstream SQLite changes.

Architecture Notes

The codebase is designed to minimize maintenance by:

1. Autogenerating parser code from SQLite's parse.y grammar
2. Human-written CST→AST folding decisions in isolation
3. Tools that detect when upstream SQLite grammar changes require attention

When SQLite updates their grammar, the extraction tools regenerate tokenizer and parser code, and only AST folding decisions may need manual updates.

Dialect Extensibility

A key design goal is supporting SQLite-based dialects (e.g., databases that extend SQLite with additional keywords, functions, or syntax). Tools should:

• Accept extension grammar files (.y format) that get merged with SQLite's base grammar
• Support configurable prefixes for generated symbols and filenames
• Keep dialect-specific code isolated from the core SQLite extraction logic

This allows projects like libSQL, rqlite, or custom embedded databases to use syntaqlite with their extended syntax.

Key APIs

• Dialect (syntaqlite_parser::Dialect): re-export of the C ffi::Dialect struct. Used in FFI declarations: fn my_dialect() -> *const Dialect.
• DialectEnv<'d> (syntaqlite_parser::dialect): the primary dialect handle with safe accessors: is_list, field_meta, node_name, fmt_dispatch, fmt_string, fmt_enum_display_val, token/keyword helpers. Copy.
• TypedDialectEnv<'d, N: NodeFamily>: wraps DialectEnv<'d> and adds typed node access; obtained via TypedDialectEnv::from_raw_dialect(env).
• Formatter (in syntaqlite::fmt): high-level formatter:
  • Formatter::new(): default SQLite dialect
  • Formatter::with_config(dialect, &FormatConfig): custom dialect/config
  • .format(&mut self, source: &str) and .format_node(&mut self, node: NodeRef<'_>)
• Interpreter<'a, 'b>: bytecode interpreter, takes DialectEnv<'a> + ops byte slice

Coding Conventions

• Feature flags at module boundary: #[cfg(feature = "...")] goes on mod and pub use declarations, NOT on individual imports/functions/statements inside modules
• FFI types in dialect/ffi.rs and parser/ffi.rs: no Syntaqlite prefix in Rust, always accessed via ffi:: module path
• Compile-time layout checks (const _: () = { assert!(...) }) beside every #[repr(C)] struct
• DialectEnv<'d> parameterized on lifetime, with no 'static requirement, enables stack-allocated test dialects
• Unsafe minimized: DialectEnv safe accessors wrap all raw pointer access; Interpreter has zero unsafe
• AST dump is in C (syntaqlite_dump_node), not Rust, so metadata has a single source of truth
• Named diagnostic suppression: the analyzer never silently ignores a resolution failure (no bare => {} arms). Every intentional accept names an AcceptedRef variant (syntaqlite/src/analysis/engine/statement_visitor.rs): either modeled SQLite behavior (permanent) or an unmodeled construct (tracked debt: cite the construct, delete the variant once the walker models it in scope)

Git Workflow

All changes go through pull requests; never push directly to main.

1. Branch: Create a feature branch from main with a descriptive kebab-case name (e.g., add-cte-column-validation, fix-fmt-trailing-comma)
2. Commit: Use the synq: prefix (lowercase). Describe the "why", not the "what"
3. Push & PR: Push the branch and open a PR via gh pr create. CI runs automatically on all PRs
4. Merge: PRs merge to main after CI passes

Use /cp to commit, run pre-push checks, push, and create a PR in one step. Use /cpfast for the same without pre-push checks (CI will catch issues).

Key Directories

• syntaqlite-common/ - Shared primitives with no generated-file dependencies (e.g. fmt bytecode types). Safe for the bootstrap tool to depend on.
• syntaqlite/ - Unified crate: grammar-agnostic engine + SQLite dialect, feature-gated
  • src/parser/ - Grammar-agnostic parser FFI, node types
  • src/dialect/ - Dialect handle, config types, cflags
  • src/fmt/ - Formatter (bytecode interpreter), feature="fmt"
  • src/lsp/ - LSP analysis host, feature="lsp"
  • src/sqlite/ - SQLite dialect (AST, tokens, wrappers), feature="sqlite"
  • src/validation/ - Semantic validation (schema checks, fuzzy suggestions), feature="validation"
  • Features: default=["sqlite","fmt","validation"], lsp, validation, pin-version, pin-cflags
• syntaqlite-capi/ - C API, built as libsyntaqlite. build.rs exports every function in exports.txt; include/syntaqlite.h is an umbrella header over the per-module headers here and in syntaqlite-syntax/include. Release archives ship a single generated header instead.
• syntaqlite-buildtools/ - Bootstrap and codegen tool (library + binary):
  • util/ - Shared utilities (parsers, C/Rust writers, case conversion, tool_run)
  • dialect_codegen/ - .synq → C/Rust codegen (AST model, node/meta/dialect/fmt codegen)
  • parser_tools/ - Parser generation (lemon, grammar, tokenizer/keyword assembly, base files, amalgamation)
  • sqlite/ - SQLite-specific output routing (output_manifest)
  • extract/ - SQLite source extraction
  • version_analysis/ - Grammar version diffing
  • Binary syntaqlite-buildtools: internal bootstrap commands (codegen-sqlite, codegen-sqlite-parser, sqlite-extract, etc.)
• syntaqlite-cli/ - User-facing CLI tool (ast, fmt, validate, lsp, dialect commands)
• python/tools/ - Developer utilities (diff upstream grammar, format sources, etc.)
• third_party/src/sqlite/ - SQLite source (v3.51.2)
• docs/ - Design documentation (ethos.md, plan.md)

Autogenerated Files

Several files are autogenerated and must never be edited directly. They are regenerated by tools/run-codegen from .synq definitions and SQLite grammar. The canonical list of generated files lives in tools/generated-files.txt: always consult it, not this section, if you need the authoritative enumeration.

Generated files live, roughly, under:

• syntaqlite-syntax/csrc/sqlite/**: all C sources and internal headers for the SQLite dialect
• syntaqlite-syntax/include/syntaqlite_sqlite/**: all C public headers for the SQLite dialect
• syntaqlite-syntax/csrc/tokens.h: token IDs used by the runtime
• syntaqlite-syntax/include/syntaqlite/cflags.h: SQLite cflag index constants
• syntaqlite-syntax/src/sqlite/{ast,cflags,dialect,ffi,tokens}.rs: Rust AST and FFI for the syntax crate
• syntaqlite/src/sqlite/{cflags,functions_catalog}.rs: Rust outputs consumed by the semantic crate
• python/syntaqlite/{enums,nodes}.py: Python bindings

If changes are needed, modify the .synq files in syntaqlite-buildtools/parser-nodes/ or the SQLite grammar, then run tools/run-codegen to regenerate.

.synq Format

The .synq DSL defines the AST node types, enums, flags, and formatter instructions for a dialect.

• Location: syntaqlite-buildtools/parser-nodes/*.synq
• Defines: enums, flags, nodes (with inline/index fields), lists, and fmt blocks
• Codegen produces: C headers + Rust node/token types + fmt bytecode (see tools/generated-files.txt for exact output paths)
• After editing: run tools/run-codegen, then verify with cargo check && cargo clippy

Development Tools

Code Intelligence

Prefer LSP over Grep/Glob/Read for code navigation:

• goToDefinition / goToImplementation to jump to source
• findReferences to see all usages across the codebase
• workspaceSymbol to find where something is defined
• documentSymbol to list all symbols in a file
• hover for type info without reading the file
• incomingCalls / outgoingCalls for call hierarchy

Before renaming or changing a function signature, use findReferences to find all call sites first.

Use Grep/Glob only for text/pattern searches (comments, strings, config values) where LSP doesn't help.

After writing or editing code, check LSP diagnostics before moving on. Fix any type errors or missing imports immediately.

User-Facing Documentation (web/docs/content/)

The docs follow the Diataxis framework. Every page has one purpose and falls cleanly into one bucket. If a page would span buckets, split it into multiple pages.

Buckets

Page anatomy

Every page uses TOML frontmatter:

+++
title = "Page Title"
description = "One-line summary (optional but preferred)"
weight = 5           # controls nav ordering within its section
+++

The body starts with an # H1 matching the title, then uses ## for sections.

Section index pages (_index.md) contain a sentence or two describing the bucket, plus sort_by = "weight".

Cross-linking

Treat the docs as a web and cross-link ruthlessly using Zola's @/ path syntax:

[validation guide](@/guides/validation.md)
[config file reference](@/reference/config-file.md)

Repeat content freely when it makes sense in a different context. A guide and a concept page can both explain the same mechanism from different angles; each page should stand on its own without requiring the reader to jump elsewhere for the core message.

Conventions

• Tabs for install methods or alternative approaches: <div class="tabs" data-tab-group="name">
• Blockquotes for callouts: > **Note:** ...
• Code blocks are language-tagged (bash, sql, toml, rust) and copy-pasteable
• Tutorials show input → output progression (command block, then output block)

Parser Debugging

• SyntaqliteParserConfig config = {.trace = 1} enables Lemon parser trace (C-level only)
• Lemon .out file shows all states and conflicts

Verification

When checks fail on formatting or clippy warnings, run tools/pre-push --fix first; it auto-applies cargo fmt, cargo clippy --fix, and tools/format-c before running the remaining checks. Only fix issues manually if --fix can't resolve them.

For a faster mid-task check (no diff tests):

cargo check --tests --all-features --all-targets && cargo clippy --tests --all-features --all-targets -- -D warnings

Testing

Unit tests

Both syntaqlite and syntaqlite-buildtools have Rust unit tests. Run them with:

tools/run-unit-tests

This prefers cargo nextest if installed, falling back to cargo test.

Integration tests

Behavioral correctness of the CLI is verified with Python-based diff test suites that compare CLI output against expected baselines. All suites are run via a single unified runner:

tools/run-integration-tests               # run all suites
tools/run-integration-tests --suite fmt    # run a single suite
tools/run-integration-tests --list         # list available suites

Available suites

Prerequisites

Test runners invoke the CLI binary directly. Build it first:

cargo build -p syntaqlite-cli

Key flags

Token preservation

The formatter prints from the AST, so any syntax the AST does not model is lost on the way out. The token-preservation suite makes that loss measurable: it formats a corpus harvested from the ast and fmt suites, compares the token stream before and after, and groups every difference by the construct that caused it.

tools/run-integration-tests --suite token-preservation
tools/run-integration-tests --suite token-preservation --rebaseline

Keyword casing and the statement terminator are FormatConfig choices rather than lost information, so they are normalised before comparing. Everything else compares exactly, including identifier case and quoting style.

Each run writes tests/token_preservation/triage.md (gitignored): the constructs that did not round-trip, ranked by how often they occur, with example SQL. That file is the work queue — model the construct in .synq so the formatter can print it back, in the style of PRs #356-#361.

tests/token_preservation/baseline.json is checked in and ratchets one count per category. A category that grows past its baseline fails the suite; lowering counts needs a --rebaseline to lock the improvement in.

For a much broader check, set SYNQ_TOKEN_PRESERVATION_UPSTREAM=1 to add every statement in the upstream-sqlite logs, around 78,000 rather than 937. Those logs are gitignored and only exist once that suite has been run, so this corpus differs between machines and the run reports without touching the ratchet.

Upstream SQLite tests

The upstream-sqlite suite runs 1,174 SQLite TCL test files through our parser and validator, comparing results against real SQLite.

# Full run (~75s): builds the TCL extension, runs all tests, writes JSONL logs
tools/run-integration-tests --suite upstream-sqlite --validate

# Fast re-analysis (~1s): loads existing logs from disk, skips running tests
tools/run-integration-tests --suite upstream-sqlite --analyze-only

Use --analyze-only when iterating on analysis/reporting code. A full run is only needed when the parser, validator, or TCL harness (tclsyntaqlite.c) changes.

Logs are persisted per test file at tests/upstream_baselines/logs/<testname>.jsonl (gitignored). Pass --validate to enable the semantic validator; without it only the parser is tested.

The baseline (tests/upstream_baselines/parse_acceptance.<platform>.json) ratchets false positives and gaps (syntaqlite accepts, SQLite rejects) per error category; any category increasing past its baseline count fails the suite. Fixing mismatches requires a --rebaseline to lock in the lower counts, as does editing the _normalize_* categorizers.

Every run also writes tests/upstream_baselines/triage.md (gitignored): all gaps and false positives grouped by category with example SQL, sorted by count. This is the validator-improvement worklist, sorted so that you can start at the top.