AGENTS.md — AI Agent Guide for ExtendDB

This document provides AI agents with essential context for working on the ExtendDB project.

Project Overview

ExtendDB is an open source DynamoDB-compatible runtime with pluggable storage, released under the Apache 2.0 License. ExtendDB is a clean-room implementation of the DynamoDB wire protocol, written in Rust and initially developed by AWS engineers. It is not a fork of DynamoDB and contains no DynamoDB source code. ExtendDB speaks the DynamoDB wire protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged.

• Language: Rust (edition 2024, MSRV 1.88+)
• Storage backends: PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag sqlite, also sqlite-memory for dev and CI), MongoDB 7.0+ (feature flag mongodb), Cassandra 4.1+ (experimental; feature flag cassandra). One backend per binary, selected by mutually exclusive Cargo features.
• Architecture: Async (tokio), trait-based storage abstraction
• Authentication: Mandatory SigV4 with built-in IAM (users, groups, roles, policies)
• TLS: Mandatory (self-signed cert generated by default)

Use Cases

• Local development without cloud dependency
• CI/CD integration tests
• Self-hosted/on-premises deployments
• Multi-cloud portability
• Air-gapped environments

Project Structure

extenddb/
├── crates/
│   ├── app/              # Shared CLI surface the thin bin dispatches to
│   ├── auth/             # SigV4 verification, IAM policy engine
│   ├── bin/              # CLI, config, daemon lifecycle (extenddb binary)
│   ├── cache/            # Auth and authz caches
│   ├── config/           # Config parsing shared by the CLI and the server
│   ├── core/             # Pure sync types, validation, expression parsing (no async)
│   ├── engine/           # DynamoDB operation handlers (PutItem, Query, etc.)
│   ├── server/           # HTTP server, management API, web console
│   ├── storage/          # Storage trait definitions (TableEngine trait)
│   ├── storage-cassandra/ # Cassandra implementation (experimental; feature flag)
│   ├── storage-mongodb/  # MongoDB implementation (feature flag)
│   ├── storage-postgres/ # PostgreSQL implementation (default/reference)
│   └── storage-sqlite/   # SQLite implementation (dev, CI, and embedded use)
├── tests/                # Python integration tests (pytest)
├── docs/
│   ├── design/           # Architecture and component design docs
│   ├── manuals/          # User-facing guides (PDF pipeline)
│   └── adr/              # Architecture decision records
├── scripts/              # Platform installers (Linux, macOS)
├── devtools/             # Development utilities (test runners, license checks)
└── samples/              # Example Python applications

Key Architectural Concepts

Storage Abstraction

The TableEngine trait in crates/storage/src/lib.rs defines the storage interface. All storage backends implement this trait:

• storage-postgres (PostgreSQL) — default backend
• storage-sqlite (SQLite) — feature flag sqlite, and sqlite-memory for dev and CI
• storage-mongodb (MongoDB) — feature flag mongodb
• storage-cassandra (Cassandra) — experimental; feature flag cassandra

The trait uses RPITIT (return-position impl Trait in traits) for async methods — no #[async_trait] macro.

All operations are account-scoped via account_id parameter for multi-tenancy isolation.

Crate Dependencies

extenddb (bin)
  └─> extenddb-server
       ├─> extenddb-engine
       │    ├─> extenddb-core (pure sync, no async)
       │    └─> extenddb-storage (trait definitions)
       ├─> extenddb-auth
       ├─> extenddb-storage-postgres  (feature: postgres)
       ├─> extenddb-storage-sqlite    (feature: sqlite / sqlite-memory)
       ├─> extenddb-storage-mongodb   (feature: mongodb)
       └─> extenddb-storage-cassandra (feature: cassandra)

• extenddb-core: Pure synchronous Rust. No async, no I/O. Types, validation, expression parsing.
• extenddb-storage: Trait definitions only. No implementation.
• extenddb-storage-postgres: Concrete PostgreSQL implementation.
• extenddb-storage-sqlite: Concrete SQLite implementation, the dev and CI backend. It serves the same wire surface, vector search included, and needs no server.
• extenddb-storage-mongodb: Concrete MongoDB implementation.
• extenddb-storage-cassandra: Concrete Cassandra implementation. Experimental.
• extenddb-engine: Operation handlers that call storage traits.
• extenddb-server: HTTP server, management API, web console.
• extenddb-auth: SigV4 signature verification, IAM policy evaluation.
• extenddb (bin): CLI, config parsing, daemon lifecycle.

Building and Running

Prerequisites

• Rust 1.88+ (rustup update)
• Storage backend (one of):
  • PostgreSQL 14+ running locally (see docs/local-postgres-setup.md)
  • SQLite (embedded; nothing to install)
  • MongoDB 7.0+ with replica set (see docs/local-mongodb-setup.md)
  • Cassandra 4.1+ (experimental; see docs/cassandra-setup.md)
• Python 3.10+ for tests (python3 -m venv ~/venvs/extenddb-venv && source ~/venvs/extenddb-venv/bin/activate && pip install -r requirements.txt)

Build

# PostgreSQL backend (default)
cargo build --release

# MongoDB backend (backends are mutually exclusive — disable default postgres)
cargo build --release --no-default-features --features mongodb

# Cassandra backend (experimental)
cargo build --release --no-default-features --features cassandra

Backends are selected by mutually exclusive Cargo features (postgres, mongodb, sqlite, cassandra); a build enabling more than one is rejected at compile time. Each binary serves exactly one backend.

Binary: target/release/extenddb

Unit Test

cargo test --workspace

Lint

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings

Initialize (first time only)

# PostgreSQL (default)
./target/release/extenddb init --config extenddb.toml

# MongoDB
./target/release/extenddb init --backend mongodb --config extenddb.toml

# Cassandra (experimental)
./target/release/extenddb init --backend cassandra --config extenddb.toml

This creates:

• Databases (extenddb_catalog, extenddb_data)
• Admin user credentials (printed to stdout — save the password!)
• Self-signed TLS certificate at ~/.extenddb/tls/cert.pem
• Config file extenddb.toml

Start Server

./target/release/extenddb serve --config extenddb.toml

The server runs on https://127.0.0.1:18443 by default (TLS mandatory).

Stop Server

./target/release/extenddb stop --config extenddb.toml

Check Status

./target/release/extenddb status --config extenddb.toml

Verify Deployment

./target/release/extenddb verify --config extenddb.toml

Destroy (tear down all databases)

Caution: this is a destructive action.

./target/release/extenddb destroy --config extenddb.toml

Testing

Python Integration Tests

Tests live in tests/ and use pytest. They run against either ExtendDB or Amazon DynamoDB.

Against ExtendDB (recommended workflow):

# 1. Build and initialize (first time)
cargo build --release
./target/release/extenddb init --config extenddb.toml
# Save the admin password!

# 2. Start server
./target/release/extenddb serve --config extenddb.toml

# 3. Provision test credentials and run tests
export EXTENDDB_TEST_ENDPOINT=https://127.0.0.1:18443
export EXTENDDB_ADMIN_USER=admin
export EXTENDDB_ADMIN_PASSWORD=<password-from-init>
eval $(python3 devtools/provision-test-credentials)
pytest tests/ -v

Against Amazon DynamoDB:

# Ensure AWS credentials are configured and EXTENDDB_TEST_ENDPOINT is unset
unset EXTENDDB_TEST_ENDPOINT
pytest tests/ -v

Test Runner Script

# Run all test suites against local ExtendDB
devtools/run-tests --extenddb --all

# Run specific suites
devtools/run-tests --extenddb --rust
devtools/run-tests --extenddb --pytest
devtools/run-tests --extenddb --external
devtools/run-tests --extenddb --comprehensive

# Run against Amazon DynamoDB
devtools/run-tests --real-dynamodb --pytest

# Use release build
devtools/run-tests --extenddb --rust --release

# Filter tests
devtools/run-tests --extenddb --pytest --filter test_put_item

The script requires:

• EXTENDDB_TEST_ENDPOINT (e.g., https://127.0.0.1:18443)
• EXTENDDB_ADMIN_PASSWORD (from extenddb init)
• EXTENDDB_ADMIN_USER (optional, defaults to admin)

The script automatically provisions test credentials, configures runtime settings, and writes artifacts to discussions/test-*-<hash>.txt.

External Java Tests

Java tests live in tests/external/java/ and require Java 17+ and Maven 3.6+.

devtools/run-external-tests --endpoint https://127.0.0.1:18443

Coverage

devtools/run-coverage

License Checks

devtools/check-licenses

Configuration

extenddb init generates extenddb.toml. See extenddb.sample.toml for all options.

Environment variable overrides use EXTENDDB__ prefix:

export EXTENDDB__SERVER__PORT=9000
export EXTENDDB__AUTH__PROVIDER=builtin

Runtime settings (no restart required):

extenddb settings --config extenddb.toml set log_level debug
extenddb settings --config extenddb.toml list   # secrets are masked; `get --show-secret <key>` prints one

TLS and Self-Signed Certificates

TLS is mandatory. extenddb init generates a self-signed cert at ~/.extenddb/tls/cert.pem.

To use with AWS CLI/SDKs:

export AWS_CA_BUNDLE=~/.extenddb/tls/cert.pem
aws dynamodb list-tables --endpoint-url https://127.0.0.1:18443 --region us-east-1

Python tests automatically use verify=False when endpoint is HTTPS.

Authentication

ExtendDB uses built-in IAM by default (auth.provider = "builtin"). All requests must be SigV4-signed with valid credentials.

Management API

Create users, groups, roles, policies, and access keys via CLI:

extenddb manage --user admin --password <pw> create-account --account-name myaccount
extenddb manage --user admin --password <pw> create-user --account-id <id> --user-name alice
extenddb manage --user admin --password <pw> create-access-key --account-id <id> --user-name alice

Or use the web console at https://127.0.0.1:18443/console/.

Monitoring

# Health check
curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health

# JSON metrics (DynamoDB CloudWatch-style)
curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/metrics

# Syslog (Linux)
journalctl -t extenddb -f

# Syslog (macOS)
log stream --predicate 'processImagePath ENDSWITH "extenddb"' --level info

Supported DynamoDB Operations

Table Operations

CreateTable, DeleteTable, DescribeTable, ListTables, UpdateTable, DescribeEndpoints, DescribeLimits

Item Operations

PutItem, GetItem, DeleteItem, UpdateItem (SET, REMOVE, ADD, DELETE actions; condition expressions; all ReturnValues modes)

Query & Scan

Query, Scan (key conditions, filters, projections, pagination, index selection)

Batch & Transactions

BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops)

Vector Search

SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB and Cassandra provide no implementation and refuse every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: Vector indexes are not supported by this storage backend for CreateTable and UpdateTable, and SearchVectors is not supported by this storage backend for a search.

Streams

ListStreams, DescribeStream, GetShardIterator, GetRecords

Other

UpdateTimeToLive, DescribeTimeToLive, TagResource, UntagResource, ListTagsOfResource, ImportTable, ExportTableToPointInTime

Common Development Tasks

Adding a New DynamoDB Operation

1. Add types to crates/core/src/types.rs (input/output structs)
2. Add handler to crates/engine/src/ (e.g., put_item.rs)
3. Add storage trait method to crates/storage/src/lib.rs if needed
4. Implement in crates/storage-postgres/src/
5. Wire up HTTP route in crates/server/src/
6. Add integration test in tests/

Modifying Expression Parsing

Expression parsing lives in crates/core/src/expression/. This is pure sync Rust with no async or I/O.

• Key conditions: KeyCondition enum
• Filter expressions: Expr enum
• Update expressions: UpdateAction enum
• Condition expressions: Expr enum

Debugging

• Set RUST_LOG=debug for verbose logging
• Use extenddb settings set log_level debug for runtime log level changes
• Check syslog: journalctl -t extenddb -f (Linux) or log stream (macOS)
• Health-check endpoint: curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health

Documentation

Build PDFs:

python3 docs/build-docs.py

Code Style and Conventions

• Rust edition: 2024
• MSRV: 1.88
• Async: tokio, no #[async_trait] (use RPITIT)
• Error handling: thiserror for library errors, anyhow for application errors
• Serialization: serde + serde_json
• Database: sqlx with compile-time query checking (PostgreSQL)
• HTTP: axum + tower + hyper
• Logging: tracing + tracing-subscriber
• Metrics: in-memory MetricsCollector (crates/core/src/metrics/), exposed as JSON via /metrics with DynamoDB CloudWatch-style metric names and dimensions
• TLS: rustls + axum-server

Naming Conventions

• Crates: extenddb-<component> (e.g., extenddb-core, extenddb-storage-postgres)
• Modules: snake_case
• Types: PascalCase
• Functions: snake_case
• Constants: SCREAMING_SNAKE_CASE

Testing Conventions

• Integration tests in tests/ (Python pytest)
• Unit tests in src/ modules (#[cfg(test)])
• No target-specific branching in tests — if behavior differs, it's a bug
• All tests clean up resources, even on failure
• Auth tests use management_helpers.py for credential provisioning

Skills

This skill covers the full ExtendDB lifecycle: from a cold clone to a running server with working CRUD, sample app walkthroughs, and troubleshooting. It dispatches to domain-specific reference files based on user intent.

.agents/skills/extenddb/
├── SKILL.md              Dispatcher — routes by user intent
├── scripts/
│   └── detect-state.sh  Read-only environment state detection
└── references/
    ├── setup/            Build, init, serve, IAM user creation
    ├── postgres/         PostgreSQL readiness and installation
    ├── first-request/    AWS CLI/SDK configuration, first CRUD
    ├── samples/          sample_app.py and stream_consumer.py
    └── troubleshooting/  Symptom-to-fix lookup (17 indexed errors)

Activate when the user asks about installing, configuring, running, or debugging ExtendDB.

Important Files

Environment Variables

Common Pitfalls

1. TLS is mandatory — server refuses to start without it. Use AWS_CA_BUNDLE for self-signed certs.
2. Auth is mandatory — all requests must be SigV4-signed. Use extenddb manage or web console to create credentials.
3. Account isolation — all operations are scoped to account_id. Different accounts can have tables with the same name.
4. Storage backend must be running — extenddb init and extenddb serve require a running external backend (PostgreSQL, MongoDB, or Cassandra). SQLite is embedded and needs no server.
5. MongoDB requires a replica set — even single-node MongoDB must be configured with --replSet for transactions and streams.
6. Python venv — activate the venv before running tests: source ~/venvs/extenddb-venv/bin/activate
7. Test credentials — run devtools/provision-test-credentials before pytest to create test users and keys.

Getting Help

• Troubleshooting: docs/troubleshooting.md
• Design docs: docs/design/
• Manuals: docs/manuals/
• ADRs: docs/adr/

License

Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See LICENSE for the full text.

This software is provided "as is" without warranty of any kind.

Last Updated: 2026-06-23