re_gent

# re_gent

> Version control for AI agents. A content-addressed, DAG-based version control system that captures *what* an agent did, *why* (which prompt), and lets you blame, log, and inspect steps across sessions.

This document is the project's source of truth for context, vocabulary, and architectural decisions. It is the first file Claude (or any contributor) should read when working in this repo.

---

## Mission

Today, AI coding agents have **no version control of their own**. Devs have built workarounds:

- `/compact` and pray
- copy-pasting code into a fresh chat
- screenshotting the "good" version before it changes
- *"that was working five minutes ago"*
- *"go back to before the refactor"*
- *"why did you change that file"*
- *"i told you this already"*

We gave agents write access to our codebases. We did not give ourselves git for it.

re_gent provides three primitives that should already exist for every agent-driven workflow:

- **blame** — which prompt produced this line of code?
- **log** — what did this session actually do, in causal order?
- **show** — inspect the full tool and conversation context for any step.

These three turn the agent's activity into something inspectable and shareable. They also unlock the next layer (branch, merge, rewind, multi-session orchestration) once the foundation is solid.

---

## Tool & CLI

- **Project name**: re_gent
- **CLI binary**: `rgt` (3 chars, ergonomic)
- **Storage directory**: `.regent/` at the project root (analogous to `.git/`)
- **License**: Apache 2.0 — see [`LICENSE`](./LICENSE)

---

## The conceptual model in 90 seconds

Four primitive object types, three immutable and content-addressed:

1. **Blob** — raw bytes (file content, message body, JSON payload). Identified by `blake3(content)`. Identical content stores once (free dedupe).
2. **Tree** — `{ path → blob_hash, blame_hash }` map describing the workspace at one moment. Itself a blob.
3. **Step** — analogous to a git commit. `{ parent, tree, transcript, causes, session_id, origin, turn_id, ... }`. Itself a blob. Auto-generated by hooks, never hand-authored.
4. **Ref** — the only mutable object. A named pointer (e.g. `refs/sessions/<session_id>`) holding a step hash.

Steps form a **DAG** through `parent` pointers. Each session has its own ref (its own branch tip). The DAG itself is shared across sessions: common ancestors dedupe and divergent work lives on parallel branches. Secondary-parent merge points are planned but not implemented yet.

The workspace on disk is **shared across sessions** by default. re_gent records each session's view of the workspace, but it does not currently isolate writes or resolve same-file conflicts between live agents. Use separate directories, git worktrees, or normal Git workflows when physical isolation matters.

---

## Architectural decisions (with rationale)

### Reimplement, don't fork git

We considered three options:

1. **Fork git's source.** Rejected. Git is a 25-year-old C codebase with a deeply opinionated commit format. Adding our extra fields (transcript, cause, session_id) means either jamming JSON into commit messages (ugly, slow, fragile) or modifying the object format (permanent fork, can't pull upstream changes). Almost every git fork has died.
2. **Use git as a library underneath** (libgit2, isomorphic-git). Rejected for v1. We'd still be expressing our model through git primitives, forcing awkward translations. Fine if the goal were to be a git client; we're building something different.
3. **Reimplement from scratch in a modern language.** **Chosen.** The git data model itself is small (~1–2k lines in any modern language). Our differences are fundamental enough — Step has more fields, Transcript object doesn't exist in git, sessions/agents are first-class — that having full design freedom is worth the cost. We get modern hashing (BLAKE3 vs SHA-1), single-binary distribution, and zero coupling to git internals.

### Go as the implementation language

Chosen for: gentle learning curve relative to Rust, single static binary distribution, excellent tooling, strong stdlib for I/O and concurrency. Reference projects in the same niche (Docker, Kubernetes, Hugo, GitHub CLI, Prometheus) all chose Go for the same reasons. If a hot path eventually needs Rust-level performance, individual modules can be swapped to Rust later via cgo or as separate processes.

### BLAKE3 for hashing

Faster than SHA-256, parallelizable, modern security margin. Pure-Go implementation available (`lukechampine.com/blake3`). 256-bit output, hex-encoded.

### Object store and index responsibilities

The object store and refs on disk are canonical for step, tree, blob, blame, and ref state. SQLite owns the current normalized conversation rows, message-to-step links, session metadata, and query indexes used by `log`, `show`, and `sessions`. Raw transcript snapshots are also stored as blobs when the host exposes a transcript path, but they are not available for every host/event.

Some hook recovery paths can repair missing index rows for recorded steps; a full `rgt reindex` command is still planned. This split means:

- Storage layer stays simple (just files in `objects/aa/aabbcc...`)
- Query layer can evolve independently for derived step/file views
- Backup/sync of recorded workspace state is `objects/` + `refs/`, while current conversation views also require `index.db` or archived raw transcripts

### Per-session branches, shared workspace

The DAG has multiple tips, one per active session. Refs at `refs/sessions/<origin>:<escaped-session-id>` are independent. Steps from different sessions live in the same object store; common ancestors dedupe naturally.

The disk is **not** isolated per session by default — multiple Claude Code or Codex sessions can write to the same project directory, and re_gent's hooks capture each session's view independently. This matches how people often run concurrent agents today, but file-level conflict detection and resolution are not implemented yet.

Worktrees-per-session remain a planned escape hatch for cases that need true physical isolation: long autonomous runs, sessions with different env/dependency setups, or genuinely contested files.

### Annotated-blob blame (compute at write time, not query time)

Two viable blame algorithms:

- **On-demand**: walk steps backward, diff each adjacent tree's version of the file, attribute lines as you go. O(history depth) per query. Simple but slow.
- **Annotated**: at write time, compute per-line provenance and store as a sidecar `BlameMap` blob next to each (step, file) entry. O(1) blame queries.

Chosen: **annotated**. Storage cost is bounded (per-line hash arrays compress well, can be RLE-encoded later) and blame becomes a single object lookup.

### Conversation capture

We store conversation metadata alongside the file state. Current Claude and Codex hooks write normalized message rows into the SQLite index, store tool payloads as content-addressed blobs, and keep raw agent transcript snapshots as blobs when the host exposes a transcript path. Older data may still use the chained `Transcript` blob: `{ prev: hash, new_messages: [hash...] }`.

This makes re_gent robust to host transcript churn: even if the live JSONL is rewritten or wiped, the SQLite index plus the content-addressed object store keep captured step context reconstructable.

### Hook-driven capture

re_gent is invoked by agent-host hooks. Claude Code uses `UserPromptSubmit`, `PostToolBatch`, and `Stop`; Codex uses `SessionStart`, `UserPromptSubmit`, `PostToolUse`, and `Stop`. Hooks are thin adapters into the shared capture service: normalize payloads, store messages/tool blobs, snapshot workspace on turn completion, write a step, and CAS the session ref forward.

Other agent tools can add small adapters that produce the same internal capture events. The shared engine handles storage, indexing, snapshots, blame, and transcript archives.

---

## Vocabulary

| Term | Meaning |
|---|---|
| **Step** | The unit of recorded agent action. Roughly equivalent to a git commit, but auto-generated per tool-using turn and carrying conversation + cause references. |
| **Cause** | One tool call in a step: `{ tool_use_id, tool_name, args_blob, result_blob }`. A step may have multiple causes. |
| **Tree** | A snapshot of the workspace at one step. `{ path → (blob_hash, blame_hash, mode) }`. |
| **BlameMap** | A parallel array to a file's lines, where each entry is the step hash that introduced or last modified that line. |
| **Transcript** | Legacy chained delta object: pointer to previous transcript + list of new message blob hashes. Current hooks primarily use SQLite `messages` plus optional raw transcript archive blobs. |
| **Effect** | An "uncompensated side effect" attached to a step — something that happened (HTTP call, db write) that a rewind cannot undo. Logged for transparency. |
| **Session** | A continuous stream of agent activity from one host (one Claude Code process, one SDK run). Has its own ref (branch). |
| **Agent** | An identity within a session. Sub-agents (e.g. spawned via Task tool) get their own agent_id but share session lineage. |
| **Hook** | The integration point with the agent host. Claude and Codex adapters normalize host payloads into shared capture events. |
| **Object store** | The content-addressed blob directory at `.regent/objects/`. Source of truth for recorded workspace state and payload blobs. |
| **Index** | The SQLite database at `.regent/index.db`. Stores query indexes, sessions, normalized conversation rows, and message-to-step links. |
| **Worktree** | A planned per-session physical working directory for cases needing true filesystem isolation between concurrent sessions. |

---

## Technical stack

- **Language**: Go 1.22+
- **Hashing**: BLAKE3 (`lukechampine.com/blake3`)
- **CLI framework**: Cobra (`github.com/spf13/cobra`)
- **Diff**: `github.com/sergi/go-diff` (Myers + line-mode)
- **Gitignore parsing**: `github.com/sabhiram/go-gitignore`
- **SQLite**: `modernc.org/sqlite` (pure Go, no CGO)
- **Config**: TOML (`github.com/pelletier/go-toml/v2`)
- **Build**: single static binary, no CGO required
- **Distribution (planned)**: `homebrew`, `cargo install`-style, GitHub Releases

---

## Reference projects

These are projects worth reading the source of when designing or implementing parts of re_gent. **Read, don't fork.**

- **Jujutsu (`jj`)** — <https://github.com/jj-vcs/jj>. From-scratch git-compatible VCS in Rust. Excellent reference for the *shape* of building a from-scratch object model. Their handling of operation logs (a parallel concept to our Step lineage) is particularly relevant.
- **isomorphic-git** — <https://github.com/isomorphic-git/isomorphic-git>. Pure TypeScript reimplementation of git's data model. The codebase is small enough to read in a weekend and demystifies how little code the core actually needs.
- **gitoxide** — <https://github.com/Byron/gitoxide>. Pure-Rust git implementation. Useful for understanding atomic operations, ref management, and pack file design (relevant when we eventually need GC).
- **libgit2** — <https://libgit2.org/>. C library for git operations. Useful as a reference for object encoding and edge cases.
- **Pijul** — <https://pijul.org/>. Patch-based VCS with a fundamentally different theoretical foundation. Worth reading their docs on patch theory if we ever consider a non-snapshot model.
- **Mercurial** — <https://www.mercurial-scm.org/>. Alternative to git with cleaner internals in places. Their handling of revsets is interesting for query design.

For the AI agent / observability side:

- **OpenTelemetry semantic conventions** — useful framing for thinking about Step + Cause + Effects as a kind of trace.
- **Claude Code and Codex hook documentation** — the host integration points we depend on. Host payloads should stay confined to adapter code.

---

## Status

**Active development.** Core storage, hook capture, log, blame, show, sessions, and init are implemented. Claude Code and Codex are the reference host integrations.

Current artifacts:

- [`README.md`](./README.md) — user-facing overview and install notes.
- [`TESTING.md`](./TESTING.md) — current manual and automated verification guide.
- [`CLAUDE.md`](./CLAUDE.md) — this architecture context.

---

## Next concrete steps

1. Keep Claude Code and Codex capture behavior aligned through `internal/capture`.
2. Add focused tests before extending hook payload handling or schema shape.
3. Implement missing advanced commands (`fork`, `rewind`, `reindex`) only when their storage semantics are explicit.
4. Improve snapshot performance for large repositories without weakening blame correctness.

---

## Open design questions

These are real design tensions we have not yet resolved. Flag them when relevant to the work being done:

1. **Bash side-effect attribution.** A single `Bash` tool call can produce arbitrary file changes. We snapshot after, so files are captured correctly, but blame becomes coarse: every line introduced by `find . -delete` or a multi-file `sed -i` gets attributed to a single step. Acceptable for v0; may want finer-grained tracking later (filesystem watcher during the bash exec).

2. **Sub-agent lineage.** Codex sub-agents and forked threads are captured as separate sessions today. The schema records session fork metadata, but secondary-parent step linking is not implemented.

3. **Conversation rewind into the live agent.** `rgt rewind` is not implemented. When it is, the file state and live agent transcript semantics must be designed together.

4. **Performance on monorepos.** Snapshotting every step works for small/medium repos but will hurt on 50k-file monorepos. Optimization candidates: incremental snapshots (track inode mtimes via the index), bloom filters for "did this directory change," or watchman-style integration. v0 ignores this; v1 must address.

5. **Garbage collection.** Non-destructive rewinds and abandoned exploration branches accumulate orphan steps. We need an analog of `git gc` and `git reflog` with grace periods. Out of scope for v0 but the storage layout should not preclude it (which it doesn't — content-addressed objects are trivially GCable by reachability).

6. **Cross-repo / monorepo subtree handling.** Some users have multiple agent-driven projects in nested directories. Whether `.regent/` should support subtree boundaries (like `git submodule`) is an open question.

7. **Multi-tool unification.** Claude Code and Codex now share the same capture engine. Future adapters should preserve the same `origin`, `session_id`, and `turn_id` invariants.

---

## Working notes for Claude

When working on this project:

- **Read current code before writing implementation code.** Older design docs may lag the implementation; `internal/capture`, `internal/store`, and `internal/index` are authoritative for current behavior.
- **Keep canonical boundaries explicit.** Steps, trees, refs, blame, and payload blobs go through the object store. Normalized conversation rows and message links live in SQLite, with raw transcript blobs archived when available.
- **Never break the user's agent.** Hooks run during agent turns. If `.regent/` doesn't exist, exit cleanly; if non-critical archival or blame work fails, log to `.regent/log/` without interrupting the agent.
- **Content addressing is the design discipline.** When in doubt about whether two objects should be the same, ask: do they have identical bytes? If yes, they should hash to the same value and store once.
- **CAS for refs, transactions for SQLite.** Never write a ref without compare-and-swap. Never write a multi-row index update outside a transaction.
- **Tests over commentary.** When implementing behavior, add focused tests next to the source. The tests are the spec made concrete.