datacontract-cli

# Data Contract CLI

## Development Environment Setup

```bash
# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate

# Install Requirements 
pip install --upgrade pip setuptools wheel
pip install -e '.[dev]'

# Setup pre-commit hooks
pre-commit install
```

Alternatively, using uv (recommended):

```bash
# Pin Python version
uv python pin 3.11
uv pip install -e '.[dev]'
```

## Common Commands

### Testing

```bash
# Run all tests
pytest

# Run tests in parallel
pytest -n 8

# Run a specific test file
pytest tests/test_specific.py

# Run a specific test function
pytest tests/test_specific.py::test_function_name
```

### Linting and Formatting

```bash
# Check code with ruff
ruff check .

# Fix linting issues automatically
ruff check --fix .

# Format code
ruff format .

# Run all pre-commit hooks
pre-commit run --all-files
```

### CLI Usage

```bash
# Initialize a new data contract
datacontract init

# Lint a data contract file
datacontract lint datacontract.yaml

# Test a data contract against actual data
datacontract test datacontract.yaml

# Export a data contract to a different format
datacontract export html datacontract.yaml --output datacontract.html

# Import from a different format
datacontract import sql --source my-ddl.sql --dialect postgres --output datacontract.yaml

# Show a changelog between two data contracts
datacontract changelog datacontract-v1.yaml datacontract-v2.yaml

```

### Maintenance Scripts

```bash
# Update the bundled Data Contract Editor (datacontract/editor_assets/, used by `datacontract edit`)
# to a specific version of the datacontract-editor npm package (latest if omitted)
python update_editor_assets.py 0.1.10

# Update the bundled ODCS Excel template (datacontract/templates/excel/, used by `datacontract export excel`)
# from the open-data-contract-standard-excel-template repository
python update_excel_template.py

# Validate every example data contract under examples/ (also runs in CI)
python lint_examples.py

# Regenerate the command + output examples on the docs export/import pages
# (runs the CLI against examples/ and updates the AUTOGENERATED blocks)
python update_export_examples.py
python update_import_examples.py

# Regenerate the logicalType -> native type tables on the docs Data Source Reference pages
# (derives them from datacontract/export/sql_type_converter.py, updates the AUTOGENERATED blocks)
python update_reference_types.py

# Regenerate the options listing on the docs Configuration page
# (derives it from datacontract/config/settings.py, updates the AUTOGENERATED block)
python update_config_options.py

# Regenerate the whole docs Commands section from the CLI --help output, including one
# sub-page per import/export/dbt subcommand. The docs build and the docs tests run this
# on their own, so run it by hand only to look at the output.
python update_command_docs.py
```

## Project Architecture

The Data Contract CLI is an open-source command-line tool for working with data contracts:

### Core Components

1. **CLI Interface (`datacontract/cli.py`)**: Entry point for the command-line interface using Typer.

2. **Data Contract Core (`datacontract/data_contract.py`)**: Central class for working with data contracts, handling operations like testing, validation, and export/import.

3. **Engines**: Modules for connecting to different data stores and executing tests:
   - `datacontract/engines/`: Contains implementations for testing against various data sources
   - Supports multiple backend types: S3, BigQuery, Postgres, Snowflake, Kafka, etc.

4. **Export/Import**: Modules for converting data contracts to/from different formats:
   - `datacontract/export/`: Converters for formats like Avro, SQL, dbt, HTML, etc.
   - `datacontract/imports/`: Importers from formats like SQL, Avro, JSON Schema, etc.

5. **Linting (`datacontract/lint/`)**: Tools for validating data contract files against schema and best practices.

6. **Changelog (`datacontract/changelog/`)**: Semantic comparison of ODCS data contracts.

### Extension Pattern

The project uses factory patterns for extensibility:
- `exporter_factory` and `importer_factory` allow registering custom exporters/importers
- You can create custom exporters for new output formats or importers for new input formats

### Testing Approach

- Tests are organized in the `tests/` directory
- Many tests use fixtures in `tests/fixtures/` which provide sample data contracts and test data
- Supports integration testing with various databases and data stores
- **Tests describe expected behavior, not actual behavior.** Write the test for what the code *should* do. If the test fails, fix the code under test, not the test (unless there is a justified reason for simplification).
- Don't add tests that duplicate existing coverage.

## Code Conventions

- Python 3.10+ syntax and features
- Uses Pydantic for data validation and schema definition
- Type hints throughout the codebase
- Follows PEP 8 style guidelines with some adjustments (120 character line length)
- Inline small functions that have only one caller. Prefer reusing existing functions, but don't create abstractions (helpers, wrappers, enums) just to share code — only abstract when there are 2+ real callers (YAGNI).
- Check for unused function parameters and drop them.
- Keep PR descriptions minimal: a short summary, no exhaustive change lists, no list of the performed unit tests.
- `CHANGELOG.md` entries should be one line each: what changed (user-facing), not how or why. Append the fixed issue, if exists, as `(#NNN)`. No details on mechanism, rationale, or edge cases.
- The docs list every importer and exporter three times (sidebar, card grid, and the subcommand table on the generated `commands/import/index.md`), all alphabetical, enforced by `tests/test_docs_ordering.py`. The sidebar is ordered by the label it **renders**, not the file name — `Import: AWS Glue` belongs under A, not G.
- `docs/docs/commands/` and `docs/docs/release-notes.md` are **generated** and **not committed** (see `.gitignore`) — the docs build regenerates them before every `npm start` and `npm run build`, so a changed help string or changelog entry can never leave a stale page behind for CI to trip over. The Commands section comes from the CLI `--help` output via `update_command_docs.py`; `docs/docs/commands/_category_.json`, which holds the section's position in the docs sidebar, is the only hand-written file there. `commands/index.md` is generated too, and is the only page documenting the global options (`--version`, `--system-truststore`). Prose guides belong in `docs/docs/imports`, `exports`, or `testing`; each guide links to its generated command page and each command page links back, which `tests/test_docs_commands.py` enforces (it generates the pages through the `command_docs` fixture in `tests/conftest.py`).

More agent context in datacontract/datacontract-cli

2 other files this repository gives its agents.