E2E-Self-Heal
CLAUDE.md9 starsChanged 5 months agoMITView on GitHub
This document provides essential guidelines for AI agents working on this project. An engine that automatically repairs broken Playwright E2E tests. When a UI change breaks a test's selectors, the engine diagnoses the failure, generates a targeted patch, re-runs the test, and loops until the test passes or a retry cap is hit. Delivery model: a CLI core with a CI integration wrapper. - CLI core — the engine itself is a single command-line executable. It receives a target test…
In this repository4 other files it gives its agents.
- AGENTS.mdAGENTS.md
- code-reviewSkill
- docs-siteSkill
- python-proSkill
The file
Show its rules and text
We split this CLAUDE.md into the rules it gives an agent
25 rules. About 0 are common, 20 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈20
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 20
Keep this separation strict: repair logic lives in the CLI core so it stays runnable locally by a developer _and_ callable from CI without …
- Distinctive: line 137
Use structlog for all logging.
- Very specific: line 140
Use logger.exception() (not logger.error()) inside except blocks to keep tracebacks.
- Distinctive: line 145
Use the tenacity library for retrying flaky LLM / subprocess calls.
- Distinctive: line 151
Use the rich library for all CLI output (progress bars, tables, panels, diffs).
- Distinctive: line 158
Use type hints on every function signature.
- Very specific: line 159
Prefer Pydantic models / TypedDict over raw dictionaries for structured data.
- Distinctive: line 160
Use functional, declarative style.
- Distinctive: line 162
Use the RORO pattern (Receive an Object, Return an Object).
- Distinctive: line 167
Place the happy path last.
- Very specific: line 171
Use StateGraph to build the repair workflow.
- Very specific: line 173
Use Command / conditional edges to control flow between nodes.
- Distinctive: line 174
Keep node functions pure where possible: read from state, return a state update.
- Very specific: line 178
Run Playwright via subprocess (npx playwright test <path>).
- Distinctive: line 198
Use environment variables / a settings object for config (e.g.
- Distinctive: line 199
Never hardcode secrets or API keys.
- Distinctive: line 206
All LLM/subprocess retries use the tenacity library.
- Distinctive: line 207
All logs use structlog with lowercase_underscore event names and no f-strings.
- Distinctive: line 208
All CLI output uses rich formatting.
- Distinctive: line 209
All imports are at the top of the file.
- Distinctive: line 211
All function signatures have type hints.
- Distinctive: line 230
Read the existing implementation first.
- Distinctive: line 231
Check for related patterns in the codebase.
- Distinctive: line 232
Keep consistency with existing style.
- Distinctive: line 235
Add type hints and Pydantic/TypedDict models.
≈ “Common” is an estimate: at least 1 in 5 of the 27,641 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AI-Driven E2E Test Self-Healing Engine — Development Guide
This document provides essential guidelines for AI agents working on this project.
Project Overview
An engine that automatically repairs broken Playwright E2E tests. When a UI change breaks a test's selectors, the engine diagnoses the failure, generates a targeted patch, re-runs the test, and loops until the test passes or a retry cap is hit.
Delivery model: a CLI core with a CI integration wrapper.
- CLI core — the engine itself is a single command-line executable. It receives a
target test path plus the
git diffand failure log, runs the repair loop, and writes the patched test back. This is the source of truth; everything else calls into it. - CI wrapper — a thin pipeline layer (e.g. a GitHub Action / CI step) that detects a failing E2E run, invokes the CLI core, and surfaces the result (commit or PR with the patch). The wrapper orchestrates; it must contain no repair logic of its own.
Keep this separation strict: repair logic lives in the CLI core so it stays runnable locally by a developer and callable from CI without duplication.
The system is organized into four layers:
- CLI Interface — user-facing entry point that launches a repair run (also the invocation surface CI calls into).
- Data Preprocessor (AST analyzer) — abstracts raw inputs (error logs,
git diff) into compact, hallucination-resistant context for the LLM. - LangGraph Agent — stateful judgment + repair loop (Diagnoser → Patch Generator → Test Runner).
- Test Runner — validation tool that executes
npx playwright testvia subprocess.
Built with LangGraph (agent orchestration), OpenAI Structured Outputs (deterministic patches), and Playwright (test execution).
Architecture
Data Preprocessor
Input data is abstracted before it reaches the LLM to optimize context-window usage and prevent hallucination.
- Error Log Parser — from the full Playwright log, extract only the
Error:keyword, the failing line number, and the core failure reason from the top of the stack trace (e.g.locator.click: Timeout 5000ms waiting for selector...). Diff-JSX AST Analyzer — parse the JSX/TSX regions of a
git diffand convert the before/after DOM node tree into a lightweight JSON object:{ "file": "components/SubmitButton.tsx", "previous": { "tag": "button", "attributes": { "id": "old-id", "className": "btn" } }, "current": { "tag": "button", "attributes": { "id": "new-id", "className": "btn" } } }
LangGraph State
Shared state is immutable and traceable, defined with TypedDict:
from typing import TypedDict
class AgentState(TypedDict):
test_script_path: str # path to the test file under repair
original_code: str # the original test script
current_code: str # test script as modified in the current loop
error_log: str # latest Playwright error log
dom_diff_context: list[dict] # DOM changes from AST parsing
analysis_report: str # Diagnoser's failure-cause report
patch_instructions: dict # Patch Generator's fix guide (line, code)
loop_count: int # infinite-loop guard (max: 3)
is_success: bool # whether the test passedNodes & Conditional Edges
Three nodes and one conditional edge control the flow.
- Diagnoser — maps the failing selector in
error_logto the actual DOM change indom_diff_contextand infers the root cause. Updatesanalysis_report. - Patch Generator — uses OpenAI Structured Outputs to return the target line and
replacement code as strict JSON, preventing arbitrary rewrites. Updates
current_codeand recordspatch_instructions. - Test Runner — writes
current_codeto disk and runsnpx playwright test <path>via subprocess. On success setsis_success = True; on failure refresheserror_logand incrementsloop_count. - Router (conditional edge) — routes to
[End]whenis_successisTrueorloop_count >= 3; otherwise re-enters[Diagnoser].
Critical Rules
Guardrails (non-negotiable)
- Code Integrity — the Patch Generator may ONLY fix failing locators (selectors) and optimize wait conditions. It must never alter test business logic (assertions, flow). Enforce this at BOTH the prompt and the JSON schema level.
- Non-deterministic output control — wrap LLM calls in
try/except. On a JSON parse failure, feed the error back into the Patch Generator via an internal exception loop rather than crashing the graph. - Loop cap —
loop_countmust never exceed 3. The Router is the single source of truth for termination. - AST lock — after the per-line scope gates,
_applyrunscheck_ast_lockon the whole patched file, so both the Patch Generator and the memory lookup are covered. It is fail-closed: a parse failure or any change other than locator strings, timeouts, and wait names is aPatchGuardrailViolation. Retry semantics differ by producer. A Patch Generator violation is aPatchApplicationError, so it is handled by the existing internal exception loop (never raised out of the node), is fed back to the Patch Generator, and spends a loop; it can therefore never be retried past the cap, and a terminal one yieldsRefusalReason.GUARDRAIL_VIOLATION. A memory lookup (history candidate) violation is treated as a cache miss: it is recorded as a rejected candidate, does not incrementloop_count, and falls through to the LLM path. That is bounded because the memory lookup runs once at the start of the graph. Events:ast_lock_rejected(reason, node_kind, line) andast_lock_passed. - Learning never lowers the refusal bar — local healing history, and any future learned
rule or policy setting, may reorder candidates or save a model call. It may never skip,
relax or replace a check. A history candidate is held to everything a model candidate is:
the ambiguous-target refusal runs before
memory_lookup, so a remembered repair cannot pre-empt it, and an applied history candidate still goes through the_applyscope gates and AST lock, shadow replay, the live selector check and the test run. Only a verified pass is accepted, and only a passing LLM repair is written back to history. No configuration value may disable a refusal check or auto-apply an ambiguous change; the only way past the ambiguous-target refusal is a human--selector-hint. A PR that adds learning or a policy setting must say whether it can lower the refusal bar, and the answer must be no.
Imports
- All imports MUST be at the top of the file — never inside functions or classes.
Logging
- Use structlog for all logging.
- Event names must be lowercase_with_underscores (e.g.
"patch_generated"). - NO f-strings in structlog events — pass variables as kwargs.
- Use
logger.exception()(notlogger.error()) insideexceptblocks to keep tracebacks. - Example:
logger.info("test_run_finished", path=path, is_success=is_success, loop_count=count)
Retry
- Use the tenacity library for retrying flaky LLM / subprocess calls.
- Configure exponential backoff.
- Example:
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
Console Output
- Use the rich library for all CLI output (progress bars, tables, panels, diffs).
Code Style
Python
- Target Python 3.13; manage deps with uv.
- Use type hints on every function signature.
- Prefer Pydantic models /
TypedDictover raw dictionaries for structured data. - Use functional, declarative style; reserve classes for agents and services.
- File naming: lowercase with underscores (e.g.
error_log_parser.py). - Use the RORO pattern (Receive an Object, Return an Object).
Error Handling
- Handle errors at the start of functions with guard clauses / early returns.
- Place the happy path last.
LangGraph Patterns
- Use
StateGraphto build the repair workflow; compile toCompiledStateGraphfor use. - Define state with
AgentState(see above). - Use
Command/ conditional edges to control flow between nodes. - Keep node functions pure where possible: read from state, return a state update.
Subprocess (Test Runner)
- Run Playwright via
subprocess(npx playwright test <path>); never block the event loop. - Capture stdout/stderr and pass the raw log to the Error Log Parser, not to the LLM directly.
CLI Core & CI Wrapper
- Single entry point — the CLI core is the only place the repair loop runs. The CI wrapper (and any future surface) must shell out to it, never re-implement the graph.
- Clean I/O contract — the CLI takes explicit inputs (test path, diff source, config
via flags/env) and communicates outcome through exit codes:
0= test fixed, non-zero = still failing / gave up after the loop cap. CI branches on the exit code. - Machine-readable output — emit a structured summary (e.g. JSON to stdout or a file) describing what changed, so the CI wrapper can build a PR/commit message without parsing human text. Keep rich-formatted output on stderr / a separate stream.
- No side effects in the wrapper — the wrapper only detects failures, invokes the core, and reports results (open PR, post status). All diagnosis/patching stays in the core.
- Local == CI — the same command a developer runs locally is what CI runs; do not add CI-only code paths inside the core.
Configuration
- Use environment variables / a settings object for config (e.g. OpenAI API key, retry caps).
- Never hardcode secrets or API keys.
Commandments for This Project
- The Patch Generator only fixes selectors and wait conditions — never assertions/logic.
- Enforce the code-integrity guardrail at both prompt and schema level.
loop_countnever exceeds 3; the Router owns termination.- All LLM/subprocess retries use the tenacity library.
- All logs use structlog with lowercase_underscore event names and no f-strings.
- All CLI output uses rich formatting.
- All imports are at the top of the file.
- Preprocess (parse/abstract) inputs before sending anything to the LLM.
- All function signatures have type hints; structured data uses Pydantic/TypedDict.
- JSON parse failures feed back into the Patch Generator — never crash the graph.
- Repair logic lives in the CLI core only; the CI wrapper orchestrates, never re-implements.
- The CLI signals outcome via exit codes and machine-readable output for CI to branch on.
Common Pitfalls to Avoid
- ❌ Letting the Patch Generator rewrite assertions or test flow
- ❌ Sending raw Playwright logs / full
git diffto the LLM instead of abstracted context - ❌ Using f-strings in structlog events
- ❌ Adding imports inside functions
- ❌ Missing the
loop_countguard (risk of infinite loops) - ❌ Using
logger.error()instead oflogger.exception()for exceptions - ❌ Crashing the graph on a malformed LLM JSON response
- ❌ Hardcoding secrets or API keys
- ❌ Missing type hints on function signatures
When Making Changes
- Read the existing implementation first.
- Check for related patterns in the codebase.
- Keep consistency with existing style.
- Add structured logging.
- Include error handling with early returns.
- Add type hints and Pydantic/TypedDict models.
References
- LangGraph: https://langchain-ai.github.io/langgraph/
- OpenAI Structured Outputs: https://platform.openai.com/docs/guides/structured-outputs
- Playwright: https://playwright.dev/
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

