Embody

# Embody + Envoy

## Project Overview

**Embody** is a TouchDesigner extension that automates externalization of COMP and DAT operators to version-control-friendly files (.tox, .py, .json, .xml, etc.). It solves the problem of TouchDesigner's binary .toe files being impossible to diff/merge in git.

**Envoy** is an MCP (Model Context Protocol) server embedded inside Embody that lets Claude Code create, modify, connect, and query TouchDesigner operators programmatically -- plus manage Embody externalizations.

## Critical Rules

1. **Prefer the externalized network file for reading TDXN-externalized COMPs** -- these are YAML on disk with complete network structure (operators, parameters, connections, positions, flags, DAT content, annotations). Reading them directly is faster than MCP round-trips. **Never glob for an extension:** Embody writes `.tdxn` as of v6.1.0 and keeps writing `.tdn` for any COMP externalized earlier, so a project legitimately holds a mix. Check `externalizations.tsv` (strategy column) or call `get_externalizations` to identify them -- **the strategy value is `tdxn`** (it read `tdn` before 6.2.30; both are accepted on read). To edit: modify the `.tdxn` file on disk, then **always** call `import_network` via MCP with the COMP path, the parsed network, and `clear_first=True` to reload it in TD. **Never leave a `.tdxn` edit unreloaded** -- the user must see updates immediately in TD. Use MCP when you need live runtime state (evaluated expressions, cook errors) or for non-TDXN operators. For **project-wide** questions -- who references this operator, what is the signal-flow topology, where is X used -- read the project-wide `.tdxn` snapshot with Grep/Read before making MCP calls; the Externalize Full Project pulse (`ext.Embody.externalizeProject()`) writes one (see the Embody parameters, or the setup wizard's externalize step). One grep over that file answers what would otherwise cost many round-trips.
2. **Use Envoy MCP tools for live TD state and non-TDXN operators** -- NEVER say "I can't edit that because it's in a .tox" or "these are binary files I can't access." For operators not externalized as TDXN, use MCP tools to inspect and modify them. The filesystem holds externalized files (`.py`, `.tox`, `.tdxn`, `.json`, `.xml`, etc.); MCP is for interacting with live operator state inside TD.
3. **NEVER create operators under `/local`** -- `/local` is volatile storage, not saved with the `.toe` file. Place new COMPs in the container that holds the `Embody` COMP (`op.Embody.parent().path`, the level the user chose) -- or a network the user has deliberately opened -- never `/local` and never the bare root `/`. See `/create-operator` step 1.
4. **Do NOT assume network paths** -- never guess `/project1`. Use `query_network` on `/` to discover the actual root structure.
5. **Default new COMPs to Embody's container** -- `execute_python` with `result = op.Embody.parent().path` returns the same home every run (the consistency anchor). Build in `ui.panes.current.owner.path` only when the user has deliberately navigated into a content network, and never treat bare `/` as that home. See `/create-operator` step 1.
6. **Verify unfamiliar TD behavior before asserting it** -- `describe_op_type` for an operator type's real parameter names, `get_docs` for the official page, a live probe for behavior. Never assume a TD feature, file type, or convention exists without a verified source.
7. **Binary files** (`.toe`, `.tox`) -- use MCP tools to inspect contents, not the filesystem.
8. **Always check for errors after creating operators** -- `get_op_errors` with `recurse=true` immediately after creating and connecting operators.
9. **Favor annotations over OP comments** -- use `create_annotation` for documenting operators and groups.
10. **When an MCP result looks wrong or a call fails, read the evidence** -- the `_logs`, `_effects` and `recovery_hints` on the response first, then `get_logs`, then `dev/logs/` (the ring buffer holds 200 entries; the files hold everything).
11. **Always update unit tests when modifying project code** -- check whether existing tests assert against changed behavior.
12. **Batch repetitive MCP operations** -- never make 3+ individual calls to the same tool. Use `batch_operations` to combine `set_op_position`, `connect_ops`, `set_parameter`, `set_op_flags`, etc. into a single request. For complex logic (conditionals, loops, computed values), use `execute_python` instead. Each MCP round-trip costs tokens and latency -- minimize them.
13. **Prefer the operator-creating MCP tools** (`create_op`, `copy_op`, `create_extension`) over raw `execute_python` -- they auto-position, lint layout, and (when the Envoy `Autoexternalize` preference is `DATs`/`COMPs`/`both`) auto-externalize new COMPs (TDN) and DATs (source) at their boundary -- additively, never inside an already-externalized ancestor. A `copy_op` gets a **fresh** externalization at its own path (inherited source tags/file-refs are cleared, so the copy never shares or overwrites the source's files); `create_extension` externalizes the host COMP it creates (its code DAT is captured inside). Batch via `batch_operations`. Reach for `execute_python`/`comp.create()`/`.copy()` only when you genuinely need computed/looped creation or connection-preserving `copyOPs`; those bypass auto-externalization (Envoy rides an `AUTO-EXTERNALIZE BYPASS` warning back on the response) and require manual layout + tagging.

## Approach Guidelines

- Before editing a file, verify it is the ACTUAL file responsible. Grep and trace the render path before making changes.
- Avoid over-engineering. Prefer minimal, targeted changes.
- When debugging, state your hypothesis, verify with evidence, then fix.
- Define success criteria before you start, then loop until you've verified them -- don't just run steps and declare done.
- Checkpoint after each significant step: what changed, what's verified, what's left. Don't continue from a state you can't describe.
- Fail loud -- "done" is wrong if anything was skipped silently, "tests pass" is wrong if any were skipped. Surface uncertainty; don't bury it.
- Surface conflicts, don't average them: when two patterns or rules contradict, pick one (more recent / more tested), say why, flag the other for cleanup -- never silently reconcile.
- Visual TOP work is output-first: create an Out TOP `out1` and turn its display flag on BEFORE building the chain, then keep the working chain wired into it -- the user watches live in the network backdrop.
- For visual or rendered output, success is a captured, assessed frame, not a clean network. Use `capture_top` to look at the result and judge it (`capture_op` for any non-TOP operator; load the `/visual-aesthetics` skill); never declare a visual task done on a black or empty frame.
- Guard TD's performance and stability: before and after any cook-heavy build, check `get_project_performance`; if FPS drops, frames drop, or GPU/CPU memory runs low, stop and diagnose instead of building further (see `rules/performance.md`). Never freeze or crash the user's TD.

## Project Structure

Discoverable with `ls`; the non-obvious locations:

- `dev/embody/externalizations.tsv` -- externalization tracking table (managed by Embody, never edit)
- `dev/embody/Embody/` -- main extension source (`EmbodyExt.py`, `EnvoyExt.py`, `TDXNExt.py`); `templates/` holds the generated-rule/skill templates shipped to user projects; `text_claude.md` is the user-project CLAUDE.md template
- `dev/Embody-6.toe` -- active development project (versioned siblings `dev/Embody-6.NNN.toe`); `dev/Backup/` -- versioned `.toe` backups
- `release/` -- latest release `.tox` + self-updater manifest

## Architecture

### Externalization Sync (.toe <-> externalized files)

Embody externalizes tagged operators to files under `dev/embody/` -- `.py` for DATs, `.tox` for COMPs (TOX strategy), `.tdxn` for COMPs (TDXN strategy). Edits to externalized files are read by TD on load/sync; changes inside TD are written out on save. Externalized files on disk are the source of truth.

**One carve-out: Embody's OWN COMP.** `_getTDXNStrategyComps` deliberately omits Embody, its ancestors and its descendants -- "reconstructing inside Embody is self-destruction" -- so nothing ever rebuilds `/embody/Embody` from `dev/embody/Embody.tdxn`. **Editing that file by hand is inert**: no reload reads it, the next save overwrites it from the live COMP, and `git status` goes clean, so the edit looks applied and never was. Change Embody's own network **live in TD** (MCP, or by hand in the UI) and let the save write the `.tdxn` as a receipt. The same holds for its descendants -- `tagger.tdxn`, `toolbar.tdxn`, `list.tdxn`, `manager.tdxn`. Externalized `.py` DATs are unaffected: those are file-synced and editing them on disk is the normal path.

### Automatic Restoration

On project open, Embody runs a three-phase startup:
- **Frame 30**: `_upgradeEnvoy()` -- extract Claude config if Envoy enabled but missing
- **Frame 45**: `ext.Embody.restoreTOXComps()` -- restore TOX-strategy COMPs from `.tox` files
- **Frame 60**: `ext.Embody.reconstructTDXNComps()` -- rebuild TDXN-strategy COMPs from `.tdxn` files

All externalized operators are fully recoverable from disk, regardless of `.toe` save state.

### Envoy MCP Architecture

Dual-thread design: worker thread runs MCP server (no TD imports), main thread executes TD operations via `_onRefresh()`. Communication via `threading.Event` + `Queue`. A main-thread handler that needs real frames (a non-TOP capture waiting for its OP Viewer TOP) returns a `{'_defer': {'frames', 'continue'}}` marker and is re-entered on later frames by `_scheduleDeferred`; the worker keeps waiting on its Event. Server auto-configures `.mcp.json` in the git root (or project folder if no git) on startup.

### TDXN Network Format

YAML-based on-disk format (v2.1; legacy JSON imports still read) for representing TD networks as diffable text. Non-default parameters only, expression shorthand (`=` prefix), type defaults, parameter templates. Full spec: `docs/tdxn/specification.md`

## Extension Referencing

Three tiers -- TD promotes every capitalized member, so the capital letter is the access modifier. See `.claude/rules/td-python.md` for the full model.

```python
# Tier 1, public API (UpperCamelCase) -- called directly on the COMP:
op.Embody.Update()
op.Embody.Refresh()
op.Embody.Save('/embody/some_op')   # opPath is REQUIRED -- Save() alone raises TypeError
op.Embody.InitEnvoy()               # Regenerate MCP + AI client config files
op.Embody.InitGit()                 # Init/reconnect git repo + .gitignore/.gitattributes
op.Embody.ExportPortableTox(target=some_comp, save_path='/path/to/output.tox')

# Tier 2, wiring (lowerCamelCase) -- through ext, called by the COMP's own callback DATs:
op.Embody.ext.Embody.getExternalizedOps(COMP)   # opFamily is REQUIRED

# Tier 3, private (_lowerCamelCase) -- inside the class only.
```

`.ext.<Name>` resolves by the **Extension Name parameter, not the class name**: the extensions are named `Embody`/`Envoy`/`TDXN`/`CatalogManager` against classes `EmbodyExt`/`EnvoyExt`/`TDXNExt`/`CatalogManagerExt`. `op.Embody.ext.Embody.x()` works; `op.Embody.ext.EmbodyExt.x()` raises.

**NEVER cache extension references in variables** -- always call inline.

## Key References

- **TD Wiki**: https://docs.derivative.ca/Main_Page
- **Skill prerequisites**: `rules/skill-prerequisites.md` (always loaded) is the authoritative load-this-skill-first table
- **Tests**: Use the `/run-tests` skill for running and writing tests
- **TDXN Spec**: See `docs/tdxn/specification.md` for the full format specification