OpenAdapt

# OpenAdapt

> A person, or a computer-use agent once, shows a task in an application with no usable API. OpenAdapt compiles it into a program that makes the same entry every time with no model calls, confirms the right record before acting, reads the saved record back from the system of record, and stops instead of improvising or retrying.

This is the llms.txt for the OpenAdapt monorepo (github.com/OpenAdaptAI/OpenAdapt). Start with the capability map: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md. It lists every capability with its release (released, or on main for the next release), its evidence (real app or synthetic) and where it lives. Check it before concluding that OpenAdapt lacks something.

## OpenAdapt in ten capabilities

Each capability is implemented in the
[OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released"
means it ships in the current PyPI packages; "on main" means it ships in the
next release.

1. **Learns from any successful run, a person's or an agent's.** A person
   records a task, or a computer-use agent does it once and OpenAdapt compiles
   the verified run into a program (`openadapt_flow.jit`, released;
   prompt-to-program `compile-agent`, landing). Agents can drive the recording
   on web, native Windows, RDP and Citrix.
2. **Compiles to the fastest layer the app allows.** Page structure first (DOM,
   Windows UI Automation, macOS Accessibility), direct API calls where the app's
   own traffic allows them (`ApiActuator`, released; compiled from captured
   traffic in [#1261](https://github.com/OpenAdaptAI/OpenAdapt/pull/1261)),
   and pixels only
   where nothing else exists, such as RDP and Citrix. It is not a pixel-macro
   recorder.
3. **Builds branching programs, not one linear path.** One program from several
   demonstrations (`induce`, `disambiguate`), a branch learned from several
   halts fixed the same way (`teach-branch`), candidate branches mined from many
   real runs in the background (shadow mode,
   [#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)), and a router
   that sends each item to the right verified variant. Shadow mining and the
   router are on main; the rest is released.
4. **Works across surfaces in one workflow.** Web, native Windows and macOS,
   RDP and Citrix. Programs chain across surfaces passing confirmed values
   (`compose`, `process`, released), and one program can switch surfaces, tabs
   and popups mid-run ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)).
5. **Confirms the right record before every write and reads it back after.**
   The record check reads the system of record (API, database, FHIR with SMART
   Backend Services, or a separate signed-in session), never just the screen.
   `record --verify` proposes those checks from what the demonstration changed.
   Released.
6. **Stops instead of improvising, and never retries an uncertain save.** A run
   that can't prove a save ends `HALTED_BEFORE_EFFECT` or
   `RECONCILIATION_REQUIRED`, never "success". A ledger of what changed before
   a stop, with undo steps a person approves, is on main. The rest is
   released.
7. **Approvals are bound to exactly what will be saved.** Each save waits for a
   person by default and can be answered on a paired phone with one card per
   decision. The approval is bound to the exact screenshot and form state, and
   the form is read again before saving, so what was approved is what is saved.
   Person-signed grants let named agents answer. Released.
8. **Repairs itself, with a person approving.** Governed repair (propose,
   review, campaign, canary, rollback), agent-demonstrated fixes for a halted
   run, and fleet drift patches shared with no patient data are released.
   Self-repair, where an agent proposes a fix and Flow probes it up to the save
   without saving, is landing.
9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded
   save around any agent's write
   ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run`
   to judge any executor's write from the system of record, and Agent Skills and
   a CLI for coding agents. Released except where a PR is named.
10. **Runs locally and keeps data private.** Healthy runs make no model calls
    and send nothing out. Personal and health data are redacted from text,
    screenshots and structure (scrub-by-default for anything that leaves the
    machine is on main); a strict no-outbound mode, an unattended on-prem
    service, and signed, timestamped receipts are released.

## Often missed

| What OpenAdapt does | Where |
| --- | --- |
| Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md) |
| Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) |
| Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) |
| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. | `api`, [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md) |
| Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) |
| Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) |
| Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) |
| Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) |
| Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) |
| Makes no model calls on healthy runs. Model training (openadapt-ml) is separate research. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) |

The MCP server section follows; the other packages are listed at the end.

## MCP server: openadapt-agent (`uvx openadapt-agent serve --bundles <dir>`, MCP Registry `io.github.OpenAdaptAI/openadapt-agent`)

> Your AI agent decides what to enter. OpenAdapt enters it in the app and checks that it saved.

Use this when an AI agent has decided what to enter in an app with no usable API, such as an EMR, a payer portal, or a desktop or Citrix app. OpenAdapt enters the values in the same screens staff use, on a computer the customer controls, then reads the saved record back. When something doesn't match, it stops and a person decides. A normal run makes no AI model calls.

## The contract: three tools

- `list_workflows`: each workflow's `name`, `purpose`, `done_means`, and typed `inputs` (types, enums, formats).
- `run_workflow(workflow, inputs, request_id, wait_seconds?)`: one piece of work. `request_id` is the caller's own id and is required. The same `request_id` returns the latest result and never writes twice.
- `get_run(run_id, wait_seconds?)`: the result of a run that returned `outcome: running`.

Every result has `outcome`, `safe_to_retry`, `what_happened` (one plain sentence), `next_action`, `proof`, `run_id`, and `request_id`. Tools publish an MCP `outputSchema` and return `structuredContent`.

## Outcomes (the honesty rules)

- `done`: saved and checked by reading the record back. `proof` is `local` (checked on the customer's computer), `sealed` (signed receipt), or `simulated` (sandbox, no app opened). A verified local run is done; it is never reported as an error.
- `needs_review`: stopped before a save; a person decides. Nothing was written, unless `record_changed` is `unknown` (a run waiting for a person to approve a save may have made an earlier one). Don't start the same work again.
- `not_sure_if_saved`: a save may have gone through. Never retry. A person checks the record. Never tell the user nothing changed.
- `did_not_run`: this call wrote nothing. Fix the cause and retry with the same `request_id` when `safe_to_retry` is true. After `request_id_conflict`, an earlier request with the same `request_id` may have saved; call `get_run` with `first_run_id` to see it.

Only `done` is a success. Results come from openadapt-flow's `transaction_outcome`, never from the coarse halt label alone.

## Try it

`openadapt-agent serve` with no flags is a sandbox: one workflow, `add_triage_note`, on MockMed, a synthetic clinic app. The `sandbox_case` input picks the outcome: `normal` (done), `duplicate_record` (needs_review), `false_saved_banner` and `timeout_after_save` (not_sure_if_saved), `app_offline` (did_not_run). Install the `tutorial` extra to drive the synthetic app with the real engine; otherwise results are simulated and say so.

- From PyPI: `claude mcp add --scope user openadapt -- uvx --python 3.12 openadapt-agent serve`. This only registers the server; Claude Code starts it at the next `claude` session. Then ask it to run `add_triage_note` with a made-up note and a request id.
- Requires openadapt-flow 2.1.0 or later (2.2.0 is current on PyPI), which installs with openadapt-agent.

## Modes and older surfaces

- `openadapt-agent serve --mode production --bundles <dir>`: run an operator's compiled workflows through the governed `openadapt-flow run` CLI. `--bundles <dir>` alone stays read-only. `--mode attended` adds decisions on paused runs by a person at that computer. `--mode sandbox` is the default without `--bundles`.
- Older flags still work: `--allow-run` (same as `--mode production` with `--bundles`, or the sandbox without), `--tutorial` (the sandbox), `--allow-attended-actions` (part of `--mode attended`).
- Older tools stay registered with `--bundles`: `get_workflow`, `get_run_report`, `list_needs_attention`, `get_attention_item`, and deprecated per-workflow `run_workflow_<opaque-id>` tools (no request id; they now also return `outcome` and `safe_to_retry`).
- Attended actions: `continue_attention` (after a person completes the paused task; Flow rechecks and resumes without repeating it), `skip_attention` (declared skip only), `reject_attention` (ends the run; earlier steps may still have effects), `teach_attention`, `escalate_attention`. Each needs an exact capability digest, a stable idempotency key, a confirmation, and MCP form elicitation; otherwise use Flow's attended console.
- `openadapt-agent serve --authoring`: local first-demo tools `observe`, `start_record`, `click`, `halt`, and more over stdio. It doesn't enable runs and opens no HTTP listener. `openadapt-agent authoring connect <openadapt://runner or pack URL>`: outbound mailbox client for hosted ChatGPT.com and Claude.ai. Local stdio recording also drives native Windows apps through Flow's Windows agent server (`--windows-agent-url`; the agent may run on another computer, such as a Mac) and RDP or Citrix client windows from pixels (`--remote-window`, `--remote-window-title`, `--citrix`). Screens are off unless the operator passes `--authoring-screens scrubbed` or `unscrubbed`. Each step is marked `actor` agent or person, and a person accepts the recording. Guide: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RECORDING_ON_WINDOWS_RDP_CITRIX.md. Hosted chats can't click the user's GUI; send them to the job box at https://openadapt.ai/#job-box. The job page it makes has **Connect this computer**, which gives the runner link. Through Desktop they guide a person on native Windows, RDP, and Citrix. Hosted MCP is `https://openadapt.ai/mcp`.
- `openadapt-agent emit-skill <bundle> --out <dir>`: a PHI-safe Agent Skill per workflow (no recorded values, step text, or secrets; one description per workflow).

## Workflow cards

An operator can add `openadapt-card.json` next to a bundle's `workflow.json` with `name`, `purpose`, `done_means`, and input descriptions. Card text reaches the calling agent; it must not contain patient details.

## Product state

Standard and Regulated runs need a workflow admission; Demo and the synthetic sandbox may run without one. Target admissions: https://openadapt.ai/production-lifecycle.json (seven, remote-safe-synthetic). Workflow admissions: https://openadapt.ai/production-workflow-admissions.json (seven, 0.0.0-synthetic, not a customer job). A missing or expired admission means not actively admitted.

## Public capability vs private artifact

- The package and MCP server are the public OpenAdapt capability. Install from PyPI as `openadapt-agent`.
- A customer's compiled workflow bundle is their private artifact, supplied at server start with `--bundles`. It is never embedded in the package, `server.json`, or any registry listing. The sandbox generates its synthetic workflow on the fly.

## Key documents

- [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md): what it does, the contract, the sandbox, developer setup.
- [docs/DESIGN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/docs/DESIGN.md): architecture, the result contract, security model.
- [docs/DISTRIBUTION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/docs/DISTRIBUTION.md): registry artifacts and the public-vs-private distinction.
- [server.json](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/server.json): official MCP registry manifest.
- [manifest.json](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/manifest.json): portable MCPB launch config.

## Related

- [openadapt-flow](https://github.com/OpenAdaptAI/OpenAdapt/tree/main/packages/flow): the demonstration compiler and runtime this package connects to.
- [OpenAdapt](https://github.com/OpenAdaptAI/OpenAdapt): the monorepo that holds every package below.
- [docs.openadapt.ai](https://docs.openadapt.ai): OpenAdapt documentation.

## Other OpenAdapt packages

- [openadapt-flow](https://github.com/OpenAdaptAI/OpenAdapt/tree/main/packages/flow) (`pip install openadapt-flow`): records a task once, compiles it into a deterministic workflow, replays it and checks the result. The core package; `openadapt-flow[agent]` adds the MCP server code.
- [openadapt-types](https://github.com/OpenAdaptAI/OpenAdapt/tree/main/packages/types) (`pip install openadapt-types`): the shared request, result and receipt models and their JSON Schemas. Depends only on Pydantic; Flow re-exports it as `openadapt_flow.contracts`.
- [openadapt](https://github.com/OpenAdaptAI/OpenAdapt) (`pip install openadapt`): the launcher CLI.