agentleFS
dylanroscoverdylanroscover/Embody/.claude/skills/td-recovery/SKILL.md

td-recovery

Skill219 starsChanged 2 months agoMITView on GitHub

MUST READ when Envoy/TD connectivity is broken and has not self-healed: bridge internals, manual recovery, stale instances, broken venv.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 1 is distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If TD is not running: Call launch_td. The bridge launches this session's pinned instance's .toe (unpinned: the registry default's) and …

    line 40
  2. Very specific:

    Read .mcp.json -- find the command path for the envoy server.

    line 55
  3. Very specific:

    Test it: run <command> -c "print(1)" via Bash.

    line 56
  4. Distinctive:

    After the repair, reopen the AI client session so the bridge restarts on the repaired Python.

    line 61
  5. Very specific:

    Only if it is still broken after a restart: the USER closes TD, removes <project_dir>/.venv themselves, and reopens the project.

    line 62

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- Generated by Embody/Envoy - Do not remove this comment - sha:52ed745c06cd2f5b -->

TD Recovery

How the Bridge Works (v2)

The bridge runs a background reconciler thread that continuously manages connectivity:

  • Config polling (every 1s): Watches .embody/envoy.json for mtime changes and re-resolves this session's pinned instance (a new port after a restart, a new name after a save). If the pinned instance leaves the registry, the session has no target: calls fail with envoy.instance.unavailable and nothing is forwarded until it re-registers. It never moves to another instance on its own.
  • Heartbeat (every 10s, fixed -- HEARTBEAT_TICK_S): Pings the backend to detect connect/disconnect transitions, and reads how long TD's main thread has been away from Envoy's request loop.
  • Process discovery: Detects new and exited TD processes via find_all_td_pids(). Forces a config re-read when new TDs appear.
  • Tool cache: Persists the tool list to disk so new sessions start with full tools immediately, without waiting for a backend round-trip.
  • Single-attempt forwarding: Failed requests return an error immediately -- no per-request retry loop. The reconciler handles recovery in the background.
  • Streaming forwards (A-46): Tool responses stream incrementally -- server-pushed notifications reach the client per-frame, an idle window bounds a stalled stream, and an absolute cap bounds the whole forward. If a suspected STREAMING defect is wedging every forward (and only then), the env var EMBODY_BRIDGE_NO_STREAM=1 (set in .mcp.json's env block, then reopen the session) falls back to read-to-EOF with the same parser: pushed frames then arrive batched at the end, and the read is bounded ONLY by the per-recv socket timeout. It is a lever for a broken parser, not for a broken peer -- and note Envoy regenerates .mcp.json on config deploys, so re-add the var if it disappears.

The Envoy Liveness Watchdog (TD-side)

Connectivity self-heals at two independent layers, so a dropped connection almost never needs manual recovery:

  • The bridge reconciler (client-side, described above) reconnects the STDIO bridge to a live Envoy.
  • The Envoy liveness watchdog (TD-side, in EnvoyExt) revives the Envoy MCP server itself. It is a pure run()-loop that probes the server socket every ~4s and revives whenever Envoy is enabled-but-down -- a dead socket while TD keeps running, OR a project.save() / extension reinit that took the server down -- force-freeing port 9870 if it is still held and rebinding in seconds. No operator, no timer. It is armed from EnvoyExt's __init__ and tied to the instance lifetime (one loop per instance, dying only when a reinit replaces the instance, whose __init__ then arms a fresh one), so a save/reinit whose post-reinit auto-start never completes still leaves a live watchdog to recover Envoy.

This is verified, not aspirational: killing the live listener socket self-heals in ~6s, and two project.save() cycles each had the watchdog fire (running=False), force-free the still-held port, and rebind in ~1s -- all with no restart, after which the bridge returns to connected:true on its own.

Recovery -- Manual Intervention

Most connectivity issues self-heal (see the two layers above). Before any manual action, read get_td_status and distinguish the cause:

  • connected:false while td_process_alive:true (Envoy unreachable but TD still running) is the dropped-socket zombie. The TD-side watchdog self-heals it in ~6-8s. WAIT ~10s and re-check get_td_status (or probe the port directly: python3 -c "import socket; socket.create_connection(('127.0.0.1',9870),0.4)"). Do NOT restart_td, relaunch TD, or toggle Envoy for this -- it defeats the watchdog and is almost never necessary. Only escalate if it genuinely has not recovered after ~15s.
  • Editing an extension .py (EnvoyExt / EmbodyExt / TDXNExt) does NOT need a restart either -- the source DATs have syncfile=True and reinit on change, so edits go live on their own.
  • envoy_unresponsive:true (TD alive, port accepting, Envoy silent 60s+ since unresponsive_since) means TD looks frozen, or stuck in one very long call; main_thread_stalled:true (Envoy answers, but TD's main thread has not come back to Envoy's request loop for 60s+ since stalled_since) means a blocking dialog or a call that never returned. The TD-side watchdog cannot help either -- it runs on that thread. Call list_dialogs first; with no dialog up, report the timestamp to the user and let them decide. Never kill TD yourself. A call pinned on the frozen TD is answered once envoy_unresponsive has held another 120s (180s of silence), saying it was abandoned.

Manual recovery below is only for when TD is actually down or the bridge process itself is broken:

  1. Call get_td_status: This is always available (even when TD is down). It shows connection state, process liveness, instance registry, and any unregistered TD processes.
  2. If TD is not running: Call launch_td. The bridge launches this session's pinned instance's .toe (unpinned: the registry default's) and waits for Envoy to become reachable.
  3. If the wrong instance is targeted, or calls fail with envoy.instance.unavailable: Call switch_instance to list or switch instances, or launch_td to reopen the pinned one. Only switch_instance (or a peer's all_sessions=true switch) moves a pinned session.
  4. If the bridge process is stuck: Tell the user to reopen this session/conversation -- this is always the first recovery step. Only if that fails, suggest restarting the MCP server as a fallback.

Common failure: stale active instance

The most frequent cause of connectivity issues is .embody/envoy.json having active set to an instance whose TD process is no longer running. The bridge's reconciler detects this automatically via heartbeat failures and reports it in get_td_status. Use launch_td or switch_instance to recover.

Common failure: broken venv

Symptoms: Bridge process doesn't start at all, or starts and immediately exits. The envoy-bridge.log file inside Embody's logs directory (see the Logfolder parameter on the Embody COMP) is empty or shows a Python traceback about a missing interpreter. .mcp.json command points to a .venv/ Python that doesn't work.

Cause: The venv was created from a TD Python installation that has since been upgraded or removed. The home key in .venv/pyvenv.cfg points to a dead path (common on Windows with versioned TD directories like TouchDesigner.2025.32460/).

Diagnosis:

  1. Read .mcp.json -- find the command path for the envoy server.
  2. Test it: run <command> -c "print(1)" via Bash. If it fails with "No Python at ..." or similar, the venv is broken.
  3. Confirm by reading .venv/pyvenv.cfg -- check if the home path points to an existing directory.

Fix: Envoy repairs this itself -- never delete the venv. When the venv interpreter fails, it switches the bridge to the system Python, rewrites the venv's interpreter layer (pyvenv.cfg and the launchers) on a background thread, and points .mcp.json back at the repaired venv; installed packages are kept.

  1. Tell the user to restart TD if the Textport reports the repair failed (one repair is attempted per venv per session).
  2. After the repair, reopen the AI client session so the bridge restarts on the repaired Python.
  3. Only if it is still broken after a restart: the USER closes TD, removes <project_dir>/.venv themselves, and reopens the project. Envoy rebuilds it and re-installs the declared extras. Never run that delete for them.

Prevention: Envoy validates the venv Python on startup and logs a warning if broken. Check TD textport for "failed to execute" warnings after TD upgrades.

A "Frozen" TD May Be a Masked Crash

A worker-thread-rich project (MCP servers, download threads, any threading.Thread) that touched TD objects from a worker -- including worker-side run()/td.run(), which silently corrupts TD state (see rules/td-python.md) -- can die as a native access violation inside libTD.dll far from any worker thread. TD's crash handler can then hang inside its own CrashAutoSave pickling (/sys/pickleUtils), so the process looks FROZEN: no crash dump, no CrashAutoSave written, no exit (Derivative-confirmed failure signature, 2026-08-17).

  • Distinguish it from a real hang: py-spy dump --pid <td_pid> --native works against TD's embedded Python 3.11, including hung processes. A masked crash shows the main thread stuck in crash-handler / pickling frames rather than in a cook or a Python loop.
  • Do not kill the process yourself -- report the diagnosis and let the user decide.
  • Root-cause hunt: audit every worker thread in the project for TD-object touches, especially run() calls -- the corruption site is usually nowhere near the crash site.
  • Operational mitigation (UNVERIFIED against official docs): Derivative support has reported that TOUCH_QUICK_CRASH=1, set in the LAUNCHER environment (TD reads it at boot only), skips the slow crash handler so a supervisor can observe the death and relaunch. The variable does not appear in official documentation as of 2026-08-17 -- verify with Derivative before relying on it.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.