bruin

# AGENTS.md/CLAUDE.md - AI Agent Contribution Guide

This document gives AI agents the project-specific operating guidance needed to work safely in Bruin.

## Before You Finish: Checks

**You MUST run these commands before completing any task that modifies application code:**

1. **Format the code**: Run `make format` in the project root. Check `git diff` afterward — if there are formatting changes, stage and include them in your work.
2. **Run the tests**: Run `make test` in the project root. If any tests fail, fix the issues before finishing.

Do not consider an application-code task complete until both checks pass. Use the `/format-fix` and `/test` commands if needed.

For changes that only touch non-application files, such as documentation, GitHub Actions workflows, agent instructions, repository metadata, or other configuration that does not affect the Bruin binary/runtime behavior, do not run the full `make format` / `make test` suite by default. Instead, run the smallest relevant validation available for the changed files, such as YAML syntax checks, Markdown checks, workflow review, or `git diff` inspection, and clearly report what was and was not run.

## Table of Contents

1. [Project Overview](#project-overview)
2. [Architecture & Core Concepts](#architecture--core-concepts)
3. [Development Environment](#development-environment)
4. [Build System](#build-system)
5. [CLI Source of Truth](#cli-source-of-truth)
6. [Codebase Organization](#codebase-organization)
7. [Testing Strategy](#testing-strategy)
8. [Contributing Guidelines](#contributing-guidelines)
9. [Common Development Tasks](#common-development-tasks)

## Project Overview

Bruin is a CLI-first data framework for ingestion, SQL/Python/R transformations, data quality, materialization, lineage, and pipeline execution across many data platforms.

Most behavior is configured in version-controlled text files: `pipeline.yml`, asset files, templates, docs, and connection/config files. When changing behavior, prefer existing package patterns and keep docs/tests aligned with user-visible changes.

### Core Features
- **Data Ingestion**: Using `ingestr` and Python scripts
- **Transformations**: SQL, Python, and R on multiple platforms
- **Data Quality**: Built-in quality checks and validations
- **Materializations**: Table/view materializations and incremental tables
- **Python Isolation**: Isolated Python environments via `uv`
- **Templating**: Jinja templating for reusable code
- **Lineage**: Dependency visualization and tracking
- **Multi-platform Execution**: Runs locally, on EC2, or GitHub Actions
- **Secrets Management**: Environment variable injection
- **VS Code Extension**: Enhanced developer experience

## Architecture & Core Concepts

### Assets
Anything that carries value derived from data:
- Tables/views in databases
- Files in S3/GCS
- Machine learning models
- Documents (Excel, Google Sheets, Notion, etc.)

Assets consist of:
- **Definition**: Metadata enabling Bruin to understand the asset
- **Content**: The actual query/logic that creates the asset

### Pipelines
Groups of assets executed together in dependency order. Structure:
```text
my-pipeline/
├─ pipeline.yml
└─ assets/
   ├─ asset1.sql
   └─ asset2.py
```

### Pipeline Runs
Execution instances containing one or more asset instances with specific configuration and timing.

## Development Environment

### Prerequisites
- **Go**: Use the version declared in `go.mod`
- **Python**: For Python asset development and formatting
- **CGO**: Required for DuckDB support
- **Git**: For version control and repository detection

### Dependencies
The project uses extensive Go dependencies including:
- CLI framework: `urfave/cli`
- Database drivers: BigQuery, Snowflake, PostgreSQL, MySQL, DuckDB, etc.
- Cloud SDKs: AWS, GCP
- Templating: Jinja via Gonja
- Testing: Testify

## Build System

The Makefile provides comprehensive build and development targets:

### Core Targets

#### Build Targets
```bash
make build          # Build with DuckDB support (CGO_ENABLED=1)
make build-no-duckdb # Build without DuckDB (CGO_ENABLED=0)
```

#### Development Targets
```bash
make deps           # Install dependencies and tools
make clean          # Remove build artifacts
make format         # Format Go/Python and run fast changed-package lint
make lint           # Run fast lint on changed packages in every Go module
make lint-full      # Run all Go linters across the primary Go modules
make tools-update   # Update development tools
```

#### Testing Targets
```bash
make test                      # Run fast unit tests
make test-full                 # Run unit tests with race detection
make test-unit                 # Run unit tests specifically
make integration-test          # Full integration tests with ingestr
make integration-test-light    # Light integration tests without ingestr
make integration-test-cloud    # Cloud-specific integration tests
```

#### Development Utilities
```bash
make refresh-integration-expectations # Update integration test expectations
```

### Build Configuration
- **Build metadata**: The Makefile injects build metadata via linker flags
- **Telemetry**: Controlled via `TELEMETRY_KEY` and `TELEMETRY_OPTOUT` environment variables
- **Tags**: Uses `no_duckdb_arrow` for standard builds, `bruin_no_duckdb` for no-DuckDB builds

## CLI Source of Truth

Do not treat this guide as a current command inventory. If a task depends on commands, flags, hidden subcommands, or runtime help text, verify against the source of truth:

1. Check `main.go` for top-level command registration.
2. Check `cmd/*.go` and `cmd/mcp/*.go` for command definitions, flags, and action wiring.
3. After `make build`, use `./bin/bruin --help` and `./bin/bruin <command> --help` to confirm runtime behavior.
4. Check `docs/commands/` when changing user-facing command behavior, and update docs when behavior changes.

When adding or changing a command, update the command implementation, tests, and user-facing docs together.

## Codebase Organization

### Package Structure (`pkg/`)
The codebase is organized into focused packages:

#### Core Packages
- **`pipeline/`**: Pipeline parsing, execution, and management
- **`config/`**: Configuration file handling (.bruin.yml)
- **`connection/`**: Database connection management
- **`executor/`**: Asset execution engine
- **`lineage/`**: Dependency tracking and visualization
- **`query/`**: Query execution and management

#### Data Platform Packages
Each supported platform has its own package:
- **Database platforms**: `bigquery/`, `snowflake/`, `postgres/`, `mysql/`, `duckdb/`, `clickhouse/`, `athena/`, `mssql/`, `databricks/`, `oracle/`, `sqlite/`, `trino/`, `synapse/`, `hana/`, `spanner/`
- **Cloud storage**: `s3/`, `gcs/`
- **Ingestion sources**: 50+ packages for different data sources (e.g., `shopify/`, `hubspot/`, `salesforce/`, `stripe/`, etc.)

#### Utility Packages
- **`jinja/`**: Template processing
- **`python/`**: Python asset execution
- **`lint/`**: Code linting and validation
- **`diff/`**: Data comparison functionality
- **`path/`**: File system utilities
- **`git/`**: Git repository operations
- **`telemetry/`**: Usage analytics
- **`secrets/`**: Secret management
- **`logger/`**: Logging utilities

### Command Implementation (`cmd/`)
Each CLI command is implemented in its own file:
- Command structure definition
- Flag parsing and validation
- Business logic delegation to appropriate packages
- Error handling and output formatting

## Testing Strategy

### Test Types

#### Unit Tests
- **Location**: Throughout `pkg/` packages with `*_test.go` files
- **Execution**: `make test-unit`
- **Coverage**: Fast local tests by default; `make test-full` adds race detection
- **Scope**: Excludes cloud integration tests

Use narrow test loops while developing, then run the required full checks before finishing:
```bash
# Target one package or test while iterating
go test -tags="no_duckdb_arrow" ./pkg/foo -run TestName
go test -tags="no_duckdb_arrow" ./cmd/... ./pkg/...

# Required final unit-test command for application-code changes
make test
```

#### Integration Tests
- **Light Integration**: `make integration-test-light` (excludes ingestr)
- **Full Integration**: `make integration-test` (includes ingestr)
- **Cloud Integration**: `make integration-test-cloud` (cloud platforms)

Use `make integration-test-light` for changes that affect parsing, command workflows, local pipeline execution, or integration-test fixtures. Use full integration tests when touching ingestr behavior. Cloud integration tests require local cloud credentials/config and may skip tests when matching connections are absent.

#### Test Data
- **Location**: `integration-tests/test-pipelines/`
- **Coverage**: Parse tests, lineage tests, execution tests
- **Expectations**: JSON files with expected outputs
- **Refresh**: `make refresh-integration-expectations` updates expectations

### Test Patterns
- Mock databases with existing SQL mock helpers
- Mock PostgreSQL with existing pgx mock patterns
- Use existing concurrency helpers for parallel work
- Use the existing file system abstraction patterns in packages that already use them

## Contributing Guidelines

### Code Style & Formatting

#### Go Code
Tools automatically installed and run via `make format`:
- **`gci`**: Import organization
- **`gofumpt`**: Stricter Go formatting
- **`golangci-lint`**: Fast changed-package linting; `make lint-full` runs the comprehensive suite
- **`govet`**: Enabled through `golangci-lint`

### Secrets, Credentials, and Generated Files

- Do not commit local credentials, tokens, keys, or personal environment files.
- Treat `.bruin.yml`, `.bruin.cloud.yml`, cloud integration configs, and local connection files as sensitive unless they are clearly committed examples.
- If a task requires cloud integration config, use local untracked files and document what was needed.
- `make refresh-integration-expectations` rewrites JSON expectations. Inspect the diff carefully and include only intentional expectation changes.
- Do not commit build outputs, virtual environments, local caches, logs, or generated binaries unless the repository already tracks that exact artifact and the change is intentional.

### Development Workflow

1. **Setup**: `make deps` to install tools and dependencies
2. **Development**: Edit code with VS Code extension for enhanced experience  
3. **Formatting**: `make format` before committing
4. **Testing**: `make test` for unit tests, integration tests as appropriate
5. **Building**: `make build` to verify compilation

### Adding New Data Platforms

1. **Create package**: `pkg/newplatform/`
2. **Implement interfaces**: Connection, query execution, schema introspection
3. **Add CLI command**: Register in main command list
4. **Add tests**: Unit and integration tests
5. **Update documentation**: Add to supported platforms list

### Adding New CLI Commands

1. **Create command file**: `cmd/newcommand.go`
2. **Implement command structure**: Using `cli.Command` pattern
3. **Add business logic**: In appropriate `pkg/` package
4. **Register command**: In `main.go` commands slice
5. **Add tests**: Command and business logic tests

## Common Development Tasks

### Running Locally
```bash
# Basic build and run
make build
./bin/bruin --help

# Development mode with debug
make build
./bin/bruin --debug [command]
```

### Adding New Asset Types
1. Define asset type in `pkg/pipeline/asset.go`
2. Implement execution logic in `pkg/executor/`
3. Add parsing logic if needed
4. Update lineage detection if applicable
5. Add tests and integration tests

### Debugging Integration Tests
```bash
# Run specific test pipeline
(cd integration-tests && ../bin/bruin run test-pipelines/your-test)

# Refresh expectations after changes from the repository root
make refresh-integration-expectations
```

### Working with Templates
```bash
# Test template rendering
./bin/bruin render path/to/template.sql

# Test complete pipeline parsing
./bin/bruin internal parse-pipeline path/to/pipeline

# Regenerate the docs pages that mirror a template README
make sync-template-docs
```

Most templates have a docs page under `docs/getting-started/templates-docs/` that
is written by hand and diverges from the template's own README. For the templates
listed in `SYNCED` in `scripts/sync_template_docs.py`, the docs page is instead
generated from the README: edit `templates/<name>/README.md`, run
`make sync-template-docs`, and commit both. Those pages open with a
`<!-- Generated from … -->` comment; `make test` fails if one has drifted.

### Adding Docs Pages

Every page under `docs/` needs a sidebar entry in `docs/.vitepress/config.mjs`.
`make test` fails on a page without one (`docs/sidebar_test.go`). For a page that
is deliberately hidden, such as a redirect, add it to `unlistedPages` in that test.

### Checking Docs Links
```bash
npm run docs:build              # Fails on links to missing pages
scripts/check_docs_links.sh     # Internal pages and #anchors in the built site; CI blocks on this
make validate-links             # External URLs in docs, templates and root Markdown; runs weekly in CI
```

Checks need [lychee](https://github.com/lycheeverse/lychee#installation) (`brew install lychee`).

VitePress builds anchors differently from GitHub, so copy them from the rendered
page instead of guessing: `` `.bruin.yml` `` → `#bruin-yml`, `Usage & Billing` →
`#usage-billing`, `6. Add instructions` → `#_6-add-instructions`. When renaming a
heading, search `docs/` for its old anchor.

### Database Connection Testing
```bash
# List connections
./bin/bruin connections list

# Test connection
./bin/bruin connections test --name connection-name

# Add new connection
./bin/bruin connections add
```

---

This guide provides the foundational knowledge needed to contribute effectively to the Bruin project. For specific implementation details, refer to the extensive documentation in the `docs/` directory and examine existing patterns in the codebase.

More agent context in bruin-data/bruin

14 other files this repository gives its agents.