Skill: ExTester UI E2E Tests for VS Code Logic Apps Extension

Status: IN PROGRESS — Phase 4.1 (createWorkspace): 63 passing, 1 failing (pre-existing product bug). Phase 4.2 (designer actions): 2 passing, 0 failing — full lifecycle tests covering designer authoring → save → debug → overview → run trigger → verify succeeded. ~5 min runtime.

Last updated: 2026-03-03

1. What This Is

TRUE end-to-end tests using vscode-extension-tester (ExTester v8.21.0) that launch a real standalone VS Code instance, load the locally-built Logic Apps extension, open webviews, interact with form fields via Selenium WebDriver, and verify the wizard flow + designer functionality.

Not filesystem-only tests. Not @vscode/test-cli tests (those exist separately in src/test/e2e/).

1.5. Critical Rules — Read Before Writing Any New Test

These are the rules whose violation has caused entire test suites to be rewritten. Read them first.

Rule 1. Use Create Workspace + reopen — NEVER synthesize Logic App fixtures on disk

For any test that touches debug, the design-time API, the language server, the overview page, or anything that runs after a workspace opens:

1. Create the workspace via the real Create Workspace webview in one run-e2e.ts phase.
2. After it generates the .code-workspace, end that VS Code session.
3. Open a fresh phase with the generated .code-workspace as the resources argument.
4. Wait for design-time startup evidence (e.g., workflow-designtime/) before any debug or designer assertion.

Why this matters: Hand-written project folders skip everything the wizard does — local.settings.json shape, .vscode/launch.json, extension activation order, project-type detection, language-server registration, the prompts that the extension expects to have already been answered. A synthetic fixture passes locally and fails in CI for invisible reasons, or worse, passes everywhere while testing a non-existent code path.

Exceptions (and ONLY these):

• Pure non-Logic-App smoke tests (e.g., nonLogicAppStartup.test.ts).
• Standalone-runner unit tests that explicitly do not start VS Code.

Anti-precedents in this repo: createLegacyProjectFixture in run-e2e.ts exists because it predates this rule, and the former createCodefulProjectFixture was deleted after Phase 4.10 moved to Create Workspace + reopen. Do not use synthetic helpers as templates for new tests.

If the wizard helper does not yet support your project type, the correct fix is to extend createWorkspace.test.ts's helper to support the new appType, not to synthesize a fixture. The extension already supports codeful project creation via CreateLogicAppWorkspace.ts -> createCodefulWorkflowFile, and the E2E helper now exposes that flow for Phase 4.10.

Source: .squad/knowledge/vscode-e2e-testing.md — "Debug regressions need the real workspace and launch path".

Rule 2. One test per phase — fresh VS Code session

Tests sharing a Phase 4.x session fail due to lingering func.exe, language-server file locks, and webview iframes still in the DOM after a workspace switch. Every new test owns its own phase via prepareFreshSession().

Rule 3. Check focused-scenario prerequisites before running

Before running a focused E2E_MODE or LA_E2E_SCENARIO, inspect the scenario's workspaceSpec in run-e2e.ts:

1. workspaceSpec: { ... } and workspaceSpec: 'manifest-multi' require the Phase 4.1 fixture manifest at %TEMP%\la-e2e-test\created-workspaces.json.
2. Run p41a-fixtures or a grouped/full mode that runs Phase 4.1 first before scenarios such as p49-nugetdebugconversion / E2E_MODE=nugetdebugonly.
3. If the manifest exists but directories are missing or stale, rerun the fixture phase instead of debugging the focused scenario.
4. self-contained, self-creates, and plain-folder scenarios create or select their own resources and do not need the Phase 4.1 manifest.

Do not spend iteration cycles on a focused scenario until its prerequisite fixture setup is present.

Current manifest-backed scenarios:

No Phase 4.1 fixture manifest required: p40-nonlogicapp, p41a-fixtures, p41b-createworkspace-behavior, p48b-conversioncreate, and standalone p48d-conversionyes. The grouped conversiononly mode still needs the manifest because its p48a/p48c/p48e phases and workspace-dir resource selection are manifest-backed.

When adding a new scenario, update this map in the same PR as the run-e2e.ts scenario entry.

Rule 4. Use Selenium Actions API for clicks inside webview iframes

Direct element.click() does not dispatch native events that React's synthetic event system captures. Use driver.actions().move({ origin: element }).click().perform().

Rule 5. Find inputs by label, not by DOM index

findInputByLabel('Workspace name') survives re-renders; index-based lookups silently write into the wrong field when the wizard reorders inputs.

Rule 6. Gate Next/Create clicks on validation success, not on button visibility

Webview validators are async. Wait for the button to be enabled and for error decorators to clear, not just for it to appear.

Rule 7. Lint and rebuild after every test edit

Run npx biome check --write <files> and npx tsup --config tsup.e2e.test.config.ts before claiming the test passes. Biome failures break CI; an un-rebuilt test runs the stale compiled JS.

Rule 8. Wire new run-e2e.ts scenarios into CI

Adding a scenario to run-e2e.ts or a focused E2E_MODE is not enough. If the scenario is intended to protect PRs, add it to .github/workflows/vscode-e2e.yml in the grouped vscode-e2e matrix, or explicitly document it as local-only.

Why this matters: p49-nugetdebugconversion was available locally through E2E_MODE=nugetdebugonly, but until it was added to the workflow matrix, CI never ran the NuGet debug lifecycle regression.

When adding a scenario:

1. Add the scenario entry in run-e2e.ts.
2. Add the scenario to an appropriate .github/workflows/vscode-e2e.yml matrix shard with the correct use_workspace_fixtures and use_bundle_artifact flags.
3. Update this file's scenario map and mode table.
4. If the scenario is deliberately local-only, document that explicitly next to the scenario entry and in the local run instructions.

Rule 9. Opening designer requires popup hygiene

Before any test opens the designer, account for known blocking prompts:

1. Patch local.settings.json with WORKFLOWS_SUBSCRIPTION_ID: "" via ensureLocalSettingsForDesigner() so the "Enable connectors in Azure for Logic App" / "Use connectors from Azure" QuickPick does not appear.
2. Use shared helpers such as handleDesignerPrompts(), dismissNotifications(), command-palette retries, and modal/QuickInput cleanup around workspace open, designer launch, and webview waits.
3. If a new C# Dev Kit sign-in, auth, connector parameterization, notification, modal, or QuickPick blocker is found, fix it in shared helpers instead of adding a one-off sleep or test-local dismissal.

These popups are suite infrastructure blockers. Do not spend product-debug iteration cycles waiting on them unless the popup itself is the behavior under test.

Rule 10. Debug lifecycle regressions must run a real workflow, not just start the host

For debug lifecycle regressions, especially bundle → NuGet conversion, seed a deterministic built-in Request trigger + Response action workflow and verify callback URL, Run trigger, run history Succeeded, and action-level evidence such as Response:Succeeded. admin/host/status is not enough.

2. File Inventory

3. How to Build & Run

# From: apps/vs-code-designer/

# 1. Build the extension (REQUIRED — creates dist/)
cd ../../  # repo root
pnpm run build:extension
cd apps/vs-code-designer

# 2. Compile the test TypeScript (uses tsup → CJS for Mocha)
npx tsup --config tsup.e2e.test.config.ts

# 3. Run all tests (Phase 4.0+)
node out/test/run-e2e.js

# 4. Run only Phase 4.2 (designer tests) using existing workspaces
$env:E2E_MODE="designeronly"    # PowerShell
export E2E_MODE=designeronly    # bash
node out/test/run-e2e.js

# 5. Run only the plain-folder startup regression test
$env:E2E_MODE="nonlogicappstartup"  # PowerShell
export E2E_MODE=nonlogicappstartup  # bash
node out/test/run-e2e.js

# Or use the PowerShell helper (kills stuck processes first):
powershell -ExecutionPolicy Bypass -File src/test/ui/run-clean.ps1

You can also use the root VS Code Run and Debug menu:

• E2E: p49 NuGet debug conversion runs E2E_MODE=nugetdebugonly.
• E2E: Select mode (run-e2e) opens a picker for supported E2E_MODE values.
• Manifest-backed Debug menu modes (designeronly, newtestsonly, conversiononly, nugetdebugonly, and selected LA_E2E_SCENARIO runs) detect a missing/stale Phase 4.1 fixture manifest and run p41a-fixtures automatically before the focused scenario.

Build Scripts

pnpm run build:ui       # Compiles test TypeScript via tsup → CJS
pnpm run test:ui        # Runs node out/test/run-e2e.js

E2E_MODE Environment Variable

IMPORTANT: Any focused mode whose scenario uses a manifest-backed workspaceSpec requires that Phase 4.1 has been run previously in the same session and workspaces still exist on disk. This includes designeronly and nugetdebugonly. If the manifest is missing, stale, or a previous run's cleanup removed workspace directories, rerun the fixture/createWorkspace phase before the focused mode.

NOTE: On Windows, openWorkspaceFileInSession() may fail to switch workspaces through the simple dialog even when no exception is thrown. The helper falls back to VSBrowser.instance.openResources() on Windows and requires a positive title/Explorer postcondition before continuing.

NOTE: When running from a background terminal, run from apps/vs-code-designer:

cd apps/vs-code-designer
node out/test/run-e2e.js

4. Architecture

4.1 Session Learnings (2026-02-26 through 2026-03-03)

Phase 4.2 rewrite (2026-02-27 → 2026-03-03)

The original Phase 4.2 had 15+ scattered tests across designerOpen.test.ts and designerActions.test.ts. These were replaced with 2 focused end-to-end tests in designerActions.test.ts only:

• Before: 15+ tests, 23-minute runtime, ~20% pass rate
• After: 2 tests, 5-minute runtime, ~80% pass rate (failures are the intermittent CustomCode designer loading issue)

Detection-based polling replaces static sleeps

All sleep(WAIT) patterns were replaced with polling functions:

• waitForDesignerWebviewTab() — polls for webview iframe appearance
• switchToDesignerWebview() — 3-phase: iframe switch → spinner disappears → canvas + nodes render
• waitForDiscoveryPanel() — polls for panel element
• waitForSearchResults() — polls for result cards
• waitForNodeCountIncrease() — polls for node count change
• clickSaveButton() — polls for save completion
• waitForRuntimeReady() — polls for debug toolbar + terminal output
• waitForRunStatusInList() — polls overview list with periodic Refresh

Command palette retry (5 attempts)

executeOpenDesignerCommand() retries up to 5 times with 3s delays. The extension needs time to re-register commands after a workspace switch. Also checks the return value of waitForDesignerWebviewTab() — if the webview didn't appear after the command was selected, the command silently failed and needs to be retried.

Selenium Actions API for React clicks

Direct .click() calls on elements inside webview iframes don't trigger React event handlers. All clicks on React elements use driver.actions().move({ origin: element }).click().perform() instead.

Debug + Overview + Run verification flow (2026-03-02 → 2026-03-03)

Added complete lifecycle verification:

1. Debug: Start via command palette "Debug: Start Debugging". Wait for debug toolbar visible OR terminal output showing Functions runtime started.
2. Overview: Switch to Explorer view (Ctrl+Shift+E), right-click workflow.json, select "Overview" from context menu. Close all editors first to ensure correct webview frame.
3. Run trigger: Click "Run trigger" button in overview. Capture initial run status (Running or Succeeded).
4. Verify: Refresh overview list until status shows "Succeeded". Then click into the run to open details. Verify all individual action nodes show "Succeeded".

Auth dialog bypass for testing

The overview page calls getAuthorizationToken() which shows a "wants to sign in" dialog. Added silentAuth setting in product code (getAuthorizationToken.ts). When azureLogicAppsStandard.silentAuth: true in VS Code settings, uses { silent: true } for session retrieval, preventing the blocking dialog.

Compose inputs field (Lexical editor)

The Compose action parameter panel uses a Lexical contenteditable editor, not a standard <input>. Found via [contenteditable="true"].editor-input selector. Must click to focus then sendKeys() to type. Required for save to succeed (empty Compose inputs causes validation failure).

Strict interaction policy now used in designerActions.test.ts

• Add-trigger and add-action tests now fail if operation selection does not succeed.
• Verification is based on semantic node presence on the canvas:
  • Trigger flow asserts a request node is visible.
  • Action flow asserts a compose node is visible.
• The suite no longer treats “panel opened but operation not inserted” as pass-with-caveat for these core checks.

Discovery panel selection reliability improvements

• selectOperation() now uses broader candidate matching across multiple selectors/roles plus text-based fallbacks.
• Matching includes operation variants such as When an HTTP request is received for request.
• A guard blocks false positives such as selecting the All pseudo-result item.

Operational guidance from this session

• If Phase 4.2 reports missing workspace paths, re-run full run-e2e.ts (without E2E_MODE) to regenerate fresh Phase 4.1 outputs before retrying designer tests.
• If settings cleanup encounters file locks, clear stale VS Code test processes and retry with the existing cleanup fallback (settings/User deletion path).
• Use actions-only runs while iterating on discovery panel selectors, then promote to full Phase 4.2 once stable.

How ExTester Loads the Extension

ExTester cannot use --extensionDevelopmentPath reliably (it overrides EXTENSION_DEV_PATH to undefined unless coverage=true, and coverage=true adds a --coverage CLI flag that causes crashes). Instead:

1. Copy dist/ → test-extensions/<publisher>.<name>-<version>/ — makes it look like a marketplace-installed extension
2. Delete .obsolete file — VS Code creates this to mark extensions for removal; must be deleted or VS Code silently refuses to load the extension
3. Register in extensions.json — manually-copied extensions need an entry with identifier, version, location (URI), and relativeLocation
4. Pass --extensions-dir pointing to the test-extensions/ directory via ExTester's 3rd constructor arg

Key Invariant: No Auto-Update

VS Code will auto-update our old-versioned extension (5.110.0) from the marketplace (5.230.15+), creating duplicate commands in the command palette. This causes tests to select the wrong command.

Solution: The settings file (out/test/vscode-settings.json) disables auto-update:

{
  "extensions.auto Update": false,
  "extensions.autoCheckUpdates": false,
  "update.mode": "none"
}

And run-e2e.ts removes any stale auto-updated versions of our extension on startup.

Test-Extensions Directory Structure

dist/test-extensions/
├── extensions.json                              ← VS Code reads this
├── ms-azuretools.vscode-azurelogicapps-5.110.0/ ← OUR extension (copied from dist/)
├── azurite.azurite-3.35.0/                      ← extension pack member
├── ms-azuretools.vscode-azureresourcegroups-.../ ← dependency
├── ms-dotnettools.csharp-.../                   ← extension pack member
└── ms-dotnettools.csdevkit-.../                  ← extension pack member

5. Known Issues & Pitfalls (Debugging Guide)

Issue: .obsolete file blocks extension loading

Symptom: Extension host log shows no activation for our extension. Commands don't appear in palette. Cause: VS Code writes {"ms-azuretools.vscode-azurelogicapps-5.110.0":true} to .obsolete in the extensions dir. Fix: Delete .obsolete after copying extension. Already handled in run-e2e.ts.

Issue: Extension auto-updated from marketplace

Symptom: 5 command palette picks instead of expected 2-3. Errors reference 5.230.15/main.js instead of 5.110.0. Cause: VS Code downloads newer version from marketplace, creating duplicate extension. Fix: Set extensions.autoUpdate: false in VS Code settings. Remove stale versions in run-e2e.ts.

Issue: Wrong webview opened ("Create Workspace From Package")

Symptom: Webview shows "Package path" input. Tab title includes "From Package". Tests can't find expected fields. Cause: Two commands match "create workspace" search:

• azureLogicAppsStandard.createWorkspace → "Create new logic app workspace..." (CORRECT)
• azureLogicAppsStandard.cloudToLocal → "Create new logic app workspace from package..." (WRONG) Fix: Filter picks to exclude labels containing "package". The runCommandFromPalette() function does this.

Issue: EPERM on Windows during settings cleanup

Symptom: fs.removeSync fails with EPERM on settings/ directory. Cause: Previous VS Code test process left locked cache files. Fix: Pre-clean settings dir with fs.rmSync({recursive:true, force:true, maxRetries:3}).

Issue: Test file glob doesn't match on Windows

Symptom: 0 passing, 0 failing. Cause: path.resolve() produces backslashes on Windows; ExTester's glob matching expects forward slashes. Fix: .replace(/\\/g, '/') on the test glob path.

Issue: Command palette setText() clears the > prefix

Symptom: All command palette searches return "No matching results", even though the extension is loaded. Cause: VS Code's command palette prefixes the input with > to indicate command mode. ExTester's InputBox.setText() calls clear() first, which removes the >. Without >, VS Code treats input as a file search, so no commands appear. Fix: Always prefix search text with > when using setText() in the command palette:

await input.setText('> logic app workspace');  // ✅ stays in command mode
await input.setText('logic app workspace');    // ❌ switches to file search

Issue: Locked files from previous test run (EBUSY)

Symptom: Error: EBUSY: resource busy or locked, unlink '...\exthost.log' in "before all" hook. Cause: Previous test VS Code instances left child processes running (Roslyn Language Server, Azure Deployment Express Language Server, JSON Language Server, etc.) that hold file locks in test-resources/settings/logs/. The vscode.openFolder(uri, true) call (triggered by Create Workspace) spawns an entirely new VS Code window whose language servers persist after the test driver shuts down. Fix (multi-layered):

1. Kill ALL instances of known language server executables by name
2. WMIC search for processes with test-resources in command line → taskkill /F /T
3. Wait 5 seconds after killing for handles to release
4. 5-attempt retry loop with 3s delays for fs.rmSync on settings dir
5. Smart fallback: If full settings/ delete fails, delete just settings/User
6. Last resort: rename to settings-stale-<timestamp>

Issue: Tests pass but silently swallow errors

Symptom: All tests show ✔ but actual assertions fail (visible in console output as caught errors). Cause: Many tests wrap everything in try/catch and just log the error instead of failing. Status: KNOWN ISSUE. These need to be tightened — remove try/catch wrappers and let failures propagate.

Issue: Notification toasts block webview interaction

Symptom: ElementClickInterceptedError when trying to interact with webview. A notification toast covers the element. Cause: Extension activation (especially dependency extensions like C# DevKit) displays notifications. Fix: dismissNotifications(driver) helper clicks notification close buttons before interacting with the webview.

Issue: Azure Logic Apps ENOENT webview asset notification

Symptom: A notification from "Azure Logic Apps (Standard)" says ENOENT: no such file or directory, open ... vs-code-react/index.html, then webview-dependent steps retry until an unrelated timeout. Cause: The installed extension under test-resources/test-extensions is missing the built VS Code React webview bundle. This is a broken test/package setup, not a scenario-specific product behavior. Fix: Keep detection in shared helpers.ts via throwIfFatalLogicAppsNotification(). Common cleanup and wait helpers must rethrow FatalLogicAppsNotificationError so every E2E fails fast instead of dismissing the notification or waiting through retries.

Issue: Label ambiguity — "Function name" matches "Function namespace"

Symptom: Custom code fields filled incorrectly — namespace gets the function name value, function name stays empty. Cause: XPath contains(text(), 'Function name') matches BOTH "Function name" and "Function namespace". Fix: Use exclusive XPath matching: //label[contains(text(), 'Function name') and not(contains(text(), 'namespace'))]

Issue: Namespace validation mismatch (PRODUCT BUG)

Symptom: All custom code fields filled correctly, but Next button remains disabled. No visible validation error. Cause: In createWorkspace.tsx, canProceed() validates function namespace using nameValidation regex, which does NOT allow dots. But the field's own inline validation uses namespaceValidation which DOES allow dots. Workaround: Use dot-free namespaces in tests. Product fix needed: canProceed() should use namespaceValidation for the namespace field.

Issue: Function name validation allows hyphens (PRODUCT BUG — FIXED)

Symptom: Generated .cs files contain invalid C# identifiers like public class myfunc-abc. Cause: The webview wizard's nameValidation regex (/^[a-z][a-z0-9]*(?:[_-][a-z0-9]+)*$/i) was shared for ALL name fields including C# function names. Hyphens are invalid in C# identifiers. Root cause locations:

• apps/vs-code-react/src/app/createWorkspace/validation/helper.ts — shared regex for all names
• apps/vs-code-designer/src/app/commands/createNewCodeProject/CodeProjectBase/CreateFunctionAppFiles.ts — no sanitization before template substitution
• apps/vs-code-designer/src/app/commands/createProject/createCustomCodeProjectSteps/functionAppFilesStep.ts — same
• apps/vs-code-designer/src/app/commands/createCustomCodeFunction/createCustomCodeFunctionSteps/functionFileStep.ts — same Fix applied (2026-02-24):

1. Added functionNameValidation = /^[a-z][a-z\d_]*$/i in helper.ts — dedicated regex that rejects hyphens
2. Updated validateFunctionName() to use functionNameValidation instead of nameValidation
3. Added .replace(/-/g, '_') sanitization in all 3 template rendering locations (defense-in-depth)
4. Updated error message to say "letters, digits, and underscores" (removed hyphen mention)
5. Fixed E2E test uniqueName() to use underscores instead of hyphens
6. Added 47 unit tests in apps/vs-code-react/src/app/createWorkspace/validation/__test__/helper.test.ts Note: The old command-palette wizard (functionAppNameStep.ts) already correctly used /^[a-z][a-z\d_]*$/i and rejected hyphens. The bug was only in the new webview wizard.

Issue: Azure connector wizard blocks designer loading (FIXED)

Symptom: Designer never loads for CustomCode and RulesEngine workspaces. VS Code shows QuickPick prompts that require manual intervention. Cause: When WORKFLOWS_SUBSCRIPTION_ID is undefined in local.settings.json, the extension shows a blocking QuickPick prompt:

1. "Enable connectors in Azure for Logic App" → "Use connectors from Azure" / "Skip for now" Code path: getAzureConnectorDetailsForLocalProject() in apps/vs-code-designer/src/app/utils/codeless/common.ts triggers azureConnectorWizard.ts. Fix applied (2026-02-24): Two-layer fix in both designerOpen.test.ts and designerActions.test.ts:
2. ensureLocalSettingsForDesigner(appDir) — patches local.settings.json with WORKFLOWS_SUBSCRIPTION_ID: "" before opening the designer. Setting it to empty string (not undefined) prevents the wizard from launching.
3. handleDesignerPrompts(workbench, driver) — fallback safety net that polls for QuickPick dialogs and auto-selects "Skip for now" if they still appear. Standard workspaces already had WORKFLOWS_SUBSCRIPTION_ID: "" in their local.settings.json (set by the creation wizard), so they were unaffected.

Issue: Command palette fails to open in designer tests (FIXED)

Symptom: Waiting for element to be located By(css selector, .quick-input-widget) Wait timed out after 5xxxms — affects all tests that call executeOpenDesignerCommand(). Cause: workbench.openCommandPrompt() (ExTester API) intermittently fails to open the command palette after workspace switching. Fix applied (2026-02-27): executeOpenDesignerCommand() now retries up to 5 times with 3s delays. Before each attempt, it clears blocking UI (dialogs, notifications), dismisses JS dialogs, and focuses the editor. Additionally, after pick.select(), it checks whether waitForDesignerWebviewTab() actually found a webview — if not, it retries the command (the command may have silently failed).

Issue: Auth dialog blocks overview page loading (FIXED)

Symptom: Overview page shows sign-in dialog "Azure Logic Apps wants to sign in using Microsoft". Test hangs waiting for overview content. Cause: openOverview.ts calls getAuthorizationToken() which uses { createIfNone: true }, triggering an interactive auth dialog. Code path: getAuthorizationToken.ts → getSessionFromVSCode() → vscode.authentication.getSession() with createIfNone: true. Fix applied (2026-03-02): Added silentAuth setting support in getAuthorizationToken.ts. When azureLogicAppsStandard.silentAuth is true in VS Code settings, uses { silent: true } instead of { createIfNone: true }. This returns undefined if no session exists (instead of prompting), and the overview page handles the missing token gracefully.

Issue: Wrong webview frame entered for overview (FIXED)

Symptom: switchToOverviewWebview() enters the designer iframe instead of the overview iframe. Cause: If the designer webview tab is still open when the overview opens, ExTester's WebView.switchToFrame() targets the first matching iframe, which is the designer. Fix applied (2026-03-02): Added editorView.closeAllEditors() before opening the overview page. This ensures only the overview webview exists when switchToFrame() is called.

Issue: Debug view blocks Explorer right-click (FIXED)

Symptom: After starting debugging, right-click on workflow.json in Explorer fails because the Activity Bar shows the Debug view instead of Explorer. Fix applied (2026-03-02): Added Ctrl+Shift+E keyboard shortcut to switch back to Explorer view before attempting the right-click.

Issue: CustomCode designer content doesn't render (ACTIVE)

Symptom: Designer webview tab appears, switchToFrame() succeeds, but Phase 2: spinner still present or #root not found after 75s. React content never mounts inside the webview. Cause: Unknown. Happens ~30% of the time for CustomCode workspaces, particularly when they are the second workspace loaded in a test session (after Standard). May be resource contention from previous debug session's func.exe or language servers not fully releasing. Impact: Test 2 fails intermittently at the "Add-action button or trigger card should be visible" assertion because the designer canvas never renders. Potential fixes:

1. Add longer cleanup wait between test 1 and test 2
2. Explicitly kill func.exe between tests
3. Close all editors and wait for VS Code to fully settle after workspace switch
4. Increase Phase 2 timeout beyond 75s (though the real issue is the content never appearing)

Issue: CustomCode workspaces missing workflow.json

Symptom: workflow.json not found: .../ccapp-mm1b0gh2/ccwf-mm1b0gh2/workflow.json Cause: For CustomCode workspaces, the workflow directory structure differs — but the manifest's wfDir path is based on the naming convention from createWorkspace.test.ts which may not match the actual directory created by the wizard. Also: Previous test run's after() hook partially cleaned up workspaces, leaving some in a corrupted state. Impact: All CustomCode designer tests fail at the "verify workflow.json exists" step. Fix needed: Re-run Phase 4.1 to create fresh workspaces, and verify the manifest paths match actual disk layout.

Issue: Stale workspaces from previous test runs

Symptom: E2E_MODE=designeronly tests fail with "Missing workspace directories" for several workspace types. Cause: The after() hook in designerOpen.test.ts calls cleanupAllWorkspaces() which deletes workspace directories. If Phase 4.2 is re-run after cleanup, the workspaces no longer exist. Impact: Tests 1, 2, 6-9 fail in Phase 4.2 (workspace structure verification). Fix: Always run Phase 4.1 (createWorkspace) before Phase 4.2 to ensure fresh workspaces. The run-e2e.ts script does this automatically when E2E_MODE is not set.

Issue: openFileInEditor() via Quick Open is unreliable

Symptom: "Failed to open workflow.json" errors when using the old InputBox-based Quick Open approach. Cause: ExTester's workbench.openCommandPrompt() + setText(filePath) approach doesn't work reliably for opening files — Quick Open doesn't support absolute paths, and "Open File" triggers a native OS dialog that Selenium can't interact with. Fix applied (2026-02-24): Replaced with VSBrowser.instance.openResources(filePath) in both test files. This is the ExTester-supported way to open files and reliably makes them the active editor tab.

Issue: openWorkspaceFileInSession() insufficient wait time

Symptom: Extension not ready after workspace switch — commands fail, webview doesn't appear. Cause: Original wait was only 4s. After switching workspaces, VS Code needs time to: close the old workspace, load the new workspace file, discover logic app folders, activate the extension, register commands. Fix applied (2026-02-24): Increased to 8s wait + element check + 5s stabilization in both test files.

6. React Webview Form Structure

The createWorkspace command opens a webview with the React wizard. Understanding this structure is essential for writing correct selectors.

Two-Step Wizard

Step 0 Sub-Components (in order)

WorkspaceNameStep

LogicAppTypeStep

Radio options:

• Logic app (Standard) → value logicApp
• Logic app with custom code → value customCode
• Logic app with rules engine → value rulesEngine

DotNetFrameworkStep (conditional — only for customCode or rulesEngine)

WorkflowTypeStep

Dropdown options:

• Stateful (value: Stateful-Codeless)
• Stateless (value: Stateless-Codeless)
• Autonomous Agents (Preview) (value: Agentic-Codeless)
• Conversational Agents (value: Agent-Codeless)

Step 1 — ReviewCreateStep

Read-only summary table. Buttons: Back, Create workspace.

Navigation Buttons

• Step 0: Back (disabled), Next
• Step 1: Back, Create workspace

Panel Titles (Tab Names)

• createWorkspace command → Panel title: "Create Workspace"
• cloudToLocal command → Panel title: "Create Workspace From Package"

7. Selenium Selectors Reference

Finding inputs by label

Fluent UI renders <Label htmlFor={id}> + <Input id={id}>. Find label by text, read for attribute, then find input by id.

Finding radio buttons

Fluent UI v9 radios: input[type="radio"] exists but may be hidden. Click the <label> element instead.

Finding dropdowns

Fluent UI Dropdown renders a <button role="combobox"> as the trigger. Options are [role="option"] elements (may be portaled outside the immediate parent).

Webview iframe switching

Must call webview.switchToFrame(timeout) before any element interaction. Must call webview.switchBack() before interacting with VS Code chrome.

8. Extension Dependencies and Pack Members

These are installed from the marketplace into test-extensions/:

1. azurite.azurite (extensionPack)
2. ms-azuretools.vscode-azureresourcegroups (extensionDependencies)
3. ms-dotnettools.csharp (extensionPack)
4. ms-dotnettools.csdevkit (extensionPack)

9. Timing Constants

10. What Works (as of last run: 2026-02-24)

Phase 4.1 — Create Workspace (63 passing, 1 failing)

Runtime: ~10 minutes

• ✅ Extension loading via copy-to-test-extensions
• ✅ .obsolete file cleanup
• ✅ extensions.json registration
• ✅ ExTester downloads VS Code + ChromeDriver
• ✅ Extension dependencies installed from marketplace
• ✅ Command palette search finds picks (with retry logic)
• ✅ Correct webview detected (rejects "From Package")
• ✅ Webview opens and switchToFrame() succeeds (with notification dismissal + 3 retries)
• ✅ ALL standard form fields filled (path, workspace name, logic app name, radio, workflow name, workflow type)
• ✅ ALL custom code fields filled (.NET Version, folder name, namespace, function name)
• ✅ Review step verification (navigate forward, check summary, navigate back, values preserved)
• ✅ Workspace creation — clicks "Create workspace" button, waits for extension processing
• ✅ Disk verification — workspace dir, .code-workspace file, logic app dir, workflow dir
• ✅ Custom code disk verification — includes custom code folder
• ✅ EBUSY cleanup with multi-layered process killing
• ✅ afterEach recovery from vscode.openFolder driver state changes
• ✅ 12 workspace types: Standard Stateful/Stateless/Agent/Conversational × Codeless/CustomCode
• ✅ Workspace manifest (created-workspaces.json) persisted with workspace metadata for Phase 4.2

1 failing: Custom Code Stateful namespace validation (pre-existing product bug — canProceed() uses wrong regex)

Phase 4.2 — Designer Tests (2 passing, 0 failing)

Runtime: ~5 minutes

Phase 4.2 runs only designerActions.test.ts (the designerOpen.test.ts file was removed from Phase 4.2 runs). It contains 2 focused end-to-end tests that cover the complete workflow lifecycle:

Test 1: Standard Workflow (Stateful)

1. Reset workflow.json to empty state
2. Open designer via command palette (with 5-attempt retry)
3. Wait for 3-phase designer loading (iframe → spinner gone → canvas + nodes)
4. Add Request trigger via discovery panel search
5. Add Response action via add-action button
6. Save workflow and verify workflow.json on disk
7. Start debugging via "Debug: Start Debugging" command palette
8. Wait for Functions runtime to start (debug toolbar + terminal detection)
9. Open overview page via Explorer right-click on workflow.json
10. Click "Run trigger" button
11. Verify run shows in overview list (capture "Running" or "Succeeded" state)
12. Refresh until run shows "Succeeded" in overview list
13. Click into the succeeded run to open run details
14. Verify all action nodes show "Succeeded" status

Test 2: CustomCode Workflow (Stateful)

1. Open CustomCode workspace and workflow.json
2. Open designer (with command palette retry + webview tab detection)
3. Add Compose action via discovery panel search
4. Fill Compose inputs field (Lexical contenteditable editor)
5. Save workflow and verify workflow.json on disk
6. Start debugging and wait for runtime ready
7. Open overview page
8. Click "Run trigger"
9. Verify run shows "Running" → transitions to "Succeeded" in overview list
10. Open run details and verify all action nodes succeeded

Key helper functions in designerActions.test.ts (~2647 lines):

Product code change for testing:

• getAuthorizationToken.ts: Added silentAuth setting support. When azureLogicAppsStandard.silentAuth is true, uses { silent: true } instead of { createIfNone: true } for Azure session, preventing the "wants to sign in" dialog that blocks overview page loading in test environments.

Test Inventory (All Phases Combined: 87+ passing, 1 failing)

Phase 4.1 — createWorkspace.test.ts (63 pass, 1 fail)

Phase 4.2 — designerActions.test.ts + new tests (8+ pass, 0 fail)

Phase 4.3 — smoke/demo/standalone + Data Mapper (16+ pass, 0 fail)

11. What Needs Work (Prioritized)

P0: Fix intermittent CustomCode designer loading failure

Impact: Test 2 fails ~30% of runs Problem: After switching from Standard workspace to CustomCode workspace, the designer webview loads but React content doesn't render — Phase 2: spinner still present or #root not found after 75s. The webview tab appears (ExTester switchToFrame succeeds) but the React app inside never mounts. Root cause: Likely resource contention from the previous test's debug session (func.exe, language servers) not fully releasing before the second workspace loads. Current mitigation: stopDebugging() in Test 1 cleanup sends Shift+F5 twice and waits 2s. The executeOpenDesignerCommand() checks the webview tab appeared before returning — if the tab times out after 30s, it retries the command palette up to 5 times. Potential solutions:

1. Add longer stabilization wait between tests (5-10s after stopDebugging)
2. Kill func.exe explicitly between tests
3. Close all editors + wait after Test 1's debug session ends

P1: Fix namespace validation product bug

Impact: 1 test failure in Phase 4.1 Problem: In createWorkspace.tsx line ~271, canProceed() uses nameValidation regex for function namespace field. It should use namespaceValidation which allows dots. This causes dotted namespaces to pass field validation but silently disable the Next button. Workaround: Tests use dot-free namespaces.

P2: Improve run state transition visibility

Impact: Test 1's run completes so fast (~1s) that the initial status captured is always "Succeeded" instead of "Running" Problem: The HTTP Request trigger returns immediately, so the run transitions from Running → Succeeded before the first Refresh completes (~2s). Test 2 (CustomCode) does capture the Running state because the custom code function takes slightly longer. Current behavior: Both tests still verify the full lifecycle — they just can't always capture the intermediate "Running" screenshot for Test 1. Potential solutions:

1. Accept this as expected behavior (fast runs are good)
2. Add a deliberate delay to the Response action in the workflow definition
3. Reduce the initial Refresh delay to <500ms

P3: CI integration

• Tests require a display (Selenium drives a real VS Code window)
• For CI, need Xvfb (Linux) or similar virtual display
• Tests take ~5 minutes total for Phase 4.2
• Need process cleanup in CI (language servers and func.exe may persist between runs)
• The silentAuth product code change must be reviewed before merging

12. Separate Test Infrastructure

There are two E2E test systems in this extension:

They are independent. The CLI tests (src/test/e2e/) use @vscode/test-cli and run inside the extension host — they can call vscode.* APIs directly but cannot interact with webview UI. The ExTester tests (src/test/ui/) drive the VS Code GUI via ChromeDriver.

13. npm Scripts

// In apps/vs-code-designer/package.json:
"build:ui": "npx tsup --config tsup.e2e.test.config.ts",
"test:ui": "node out/test/run-e2e.js"

Note: The build was migrated from tsc to tsup for CJS output compatibility with ExTester/Mocha. The tsup.e2e.test.config.ts file compiles all src/test/ui/*.test.ts to out/test/*.test.js in CommonJS format.

14. Debugging Tips

See all console output

ExTester pipes VS Code's console to test stdout. Look for lines with INFO:CONSOLE for extension host messages.

Check which extension version loaded

Search output for Extension Host and ms-azuretools.vscode-azurelogicapps. If you see a different version number than expected, the auto-update issue is back.

Check what labels the webview actually renders

Add this diagnostic snippet in a test:

const labels = await webview.findWebElements(By.css('label'));
const labelTexts = await Promise.all(labels.map((l) => l.getText()));
console.log('All labels:', labelTexts);

If you see Package path* — you're in the wrong webview.

Kill stuck test processes (Windows)

# Kill test VS Code instances (careful not to kill your main one)
Get-Process | Where-Object { $_.Path -like "*test-resources*" } | Stop-Process -Force

# Kill language servers that hold file locks
Get-Process -Name "Microsoft.CodeAnalysis.LanguageServer" -ErrorAction SilentlyContinue | Stop-Process -Force
Get-Process -Name "Azure.Deployments.Express.LanguageServer" -ErrorAction SilentlyContinue | Stop-Process -Force

# Kill Functions runtime from debug sessions
Get-Process -Name "func" -ErrorAction SilentlyContinue | Stop-Process -Force

Full clean restart

# Remove all test artifacts
Remove-Item -Recurse -Force dist/test-extensions -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force "$env:TEMP\test-resources" -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force out/test -ErrorAction SilentlyContinue

Debug designer loading issues

1. Check local.settings.json in the workspace — WORKFLOWS_SUBSCRIPTION_ID must exist (can be empty string "") to prevent Azure connector wizard prompts
2. Check Extension Host logs for getAzureConnectorDetailsForLocalProject — if this triggers, the wizard prompts will block designer loading
3. Use ensureLocalSettingsForDesigner() helper before opening designer to patch settings

Debug overview / run trigger issues

1. Check that silentAuth: true is set in VS Code test settings — without this, the auth dialog blocks the overview page
2. Verify debug session is running — the "Run trigger" button is disabled until the Functions runtime starts
3. Check for stale func.exe processes — these can bind to the same port, causing the new debug session to fail
4. The overview page opens in a webview iframe — must close all editors first to avoid entering the wrong iframe
5. Run screenshots are saved to test-resources/screenshots/designerActions-explicit/<timestamp>/ — check step12-run-status-* and step15-all-nodes-succeeded for visual proof

Debug command palette failures

1. Check if VS Code is fully loaded: look for the status bar to have no loading spinners
2. Try increasing stabilization wait after workspace switch (currently 8s + 5s)
3. Check if notification toasts are covering UI elements — dismissNotifications() helper
4. In test output, look for quick-input-widget — if it never appears, the command palette keyboard shortcut may not be reaching VS Code

15. Lessons Learned (2026-03-03)

Key Insights from Test Development

1. Detection-based polling beats static sleeps. Replacing all sleep(WAIT) patterns with polling functions that detect DOM changes (spinner gone, canvas rendered, panel opened) reduced runtime from 23 min → 5 min while improving reliability from 20% → ~80%.
2. Command palette retry is the #1 reliability fix. The executeOpenDesignerCommand() function retries up to 5 times with 3s delays when "Open Designer" isn't found. The extension needs time to re-register commands after workspace switching. This single fix turned a 40% failure rate into near-zero for the command execution step.
3. Selenium Actions API is required for React-compatible clicks. Direct .click() calls don't trigger React event handlers inside webview iframes. Using driver.actions().move({ origin: element }).click().perform() works reliably because it dispatches native browser events that React's synthetic event system captures.
4. StaleElementReferenceError is expected with React Flow. After adding an action/trigger, React Flow re-renders all nodes, invalidating any cached WebElement references. A 3-attempt retry loop around the add-action button click handles this gracefully.
5. Auth dialogs block overview page silently. The extension's getAuthorizationToken() unconditionally shows a "wants to sign in" dialog when opening the overview page. Added silentAuth setting support in product code to use { silent: true } in test environments — this prevents the dialog without changing production behavior.
6. Close all editors before opening overview (CRITICAL). If the designer webview is still open when the overview page opens, WebView.switchToFrame() enters the designer iframe instead of the overview iframe. The symptom is that switchToOverviewWebview() times out waiting 60s for "Run trigger" button while sitting in the designer frame (you'll see designer content like action names in the body text diagnostic). The fix: always call result.webview.switchBack(), then driver.switchTo().defaultContent(), then new EditorView().closeAllEditors(), then sleep(2000) before starting the debug session. This ensures the designer webview iframe is removed from the DOM entirely before the overview page creates its own webview.
7. Debug view blocks Explorer right-click. After starting debugging, the Activity Bar switches to the Debug view. Must send Ctrl+Shift+E to switch back to Explorer before right-clicking workflow.json to open the overview page.
8. Lexical contenteditable editors require special handling. The Compose action's inputs field uses a Lexical editor with [contenteditable="true"].editor-input. Standard input selectors don't find it. Must click to focus, then use sendKeys() to type into the contenteditable element.
9. 3-phase run verification provides clear proof. The run verification flow captures three distinct phases: (a) run appears in overview list (Running/Succeeded status), (b) run transitions to Succeeded in overview list after Refresh, (c) opening run details shows all individual action nodes as Succeeded. This provides unambiguous visual evidence via screenshots at each phase.
10. Process cleanup must include func.exe. After debugging, func.exe (the Azure Functions runtime) keeps running and holds file locks. Must be explicitly killed alongside language servers to prevent subsequent test runs from failing.
11. Use findLastAddActionElement() when inserting a second action (CRITICAL). The canvas has multiple + buttons (one between each pair of nodes). findAddActionElement() returns the FIRST + button, which inserts the new action between the trigger and the first action — NOT at the end. When adding a second action (e.g., Response after Execute JavaScript Code), use findLastAddActionElement() which returns the LAST + button, ensuring the action is appended at the bottom of the flow in the correct topological order.
12. Initialize Variable parameter panel uses contenteditable editors, not <input> elements. The label-based lookup helpers (fillActionInput, selectDropdownInPanel) cannot find fields because the Initialize Variable action panel doesn't render standard HTML <label> + <input> pairs. Instead, it uses Lexical contenteditable editors with data-automation-id attributes and Fluent UI comboboxes. The working pattern: (a) click the node with openNodeSettingsPanel() to ensure the panel is open, (b) find contenteditable editors directly with [contenteditable="true"].editor-input selectors, (c) use index-based targeting (first editor = Name, combobox = Type, last editor = Value), (d) dump the panel's data-automation-id values for diagnostics when fields aren't found.
13. Each new test should run in its own phase (fresh VS Code session). Tests that share a Phase 4.x session with other tests fail because the previous test's debug session, workspace switch, and lingering processes leave VS Code in a degraded state (extension host unresponsive, func.exe holding locks, webview iframes still in DOM). Running each test in its own phase via prepareFreshSession() solves all of these issues. The runtime cost (~30s per extra session start) is worth the reliability gain.
14. Multiple webview iframes: ExTester's WebView targets the wrong frame (CRITICAL). When two designer tabs are open simultaneously, new WebView().switchToFrame() always enters the FIRST iframe in DOM order — not the active tab's iframe. The fix: use raw Selenium driver.switchTo().frame(element) with a custom switchToActiveDesignerFrame() function that (a) finds all iframe.webview elements, (b) checks which one is visible (.isDisplayed() + non-zero dimensions), (c) switches into it, (d) navigates the nested iframe structure (outer → #active-frame inner), and (e) polls until the designer canvas renders. This function must POLL (not check once) because VS Code takes time to create the new iframe when a second webview tab opens.
15. Designer tab names vs. file tab names — must match on "(Workspace)" (CRITICAL). The activateDesignerTab() function must distinguish designer tabs from workflow.json text file tabs. Both contain the workflow name, but designer tab titles follow the pattern {workspaceName} (Workspace)-{logicAppName}-{workflowName} and always contain "Workspace". Match on BOTH the workflow name AND "workspace" in the tab title. Without this, the function clicks the wrong tab (text editor instead of designer), leaving the webview invisible.
16. Right-clicking the correct workflow.json in the Explorer. When multiple workflows exist, the Explorer tree has multiple workflow.json rows. Use VSBrowser.instance.openResources(absolutePath) to open the specific file first — this reveals and selects it in the tree. Then find the row with selected CSS class and right-click it. Without this, the test right-clicks the first workflow.json it finds (wrong workflow).
17. Built-in operations (Request, Response) are always available; connectors (Compose) may not be. When the design-time API hasn't fully loaded the connector catalog, the search results may not include connector-level operations like Compose. Use only built-in operations (Request, Response) in tests that need reliability. The connector catalog loads asynchronously from the func host, which may not be running in all test scenarios.

18. Never trigger F5 / debug from the command palette in a test that is not already inside the designer (CRITICAL). > Debug: Start Debugging resolves launch.json from the active editor's workspace folder. In a multi-root .code-workspace opened headlessly on Linux CI, the active editor at F5 time is whatever VS Code decided to restore — a markdown preview, settings.json, an empty group — and none of those resolve the Logic App folder. runHelpers.startDebugging() does not throw in that case: it logs [debug] Selecting: "Debug: Start Debugging" and returns normally, so the test then waits out its full timeout on a product path that never executed. Anchoring the active editor first does not fix it — Quick Open depends on VS Code's file index being warm, which it is not right after a workspace open.

    Instead, install the codeful task recorder extension (§ 16) and trigger the debug session through it: set recorder: true on the scenario in run-e2e.ts (which installs the extension into the scenario's ext dir and sets LA_E2E_TASK_EVENTS_JSONL / CODEFUL_TASK_EVENTS_JSONL / LA_E2E_TRIGGER_DIR), then drop a start-debug marker into ${LA_E2E_TRIGGER_DIR} and wait for a debugInvoke line in the JSONL. The recorder waits for azureLogicAppsStandard.debugLogicApp to register, reads <folder>/.vscode/launch.json itself, and calls vscode.debug.startDebugging(folder, configName) with an explicit folder — zero dependence on editor focus, Quick Open, or the palette. It runs the identical resolution pipeline as F5, including ${command:azureLogicAppsStandard.pickProcess} substitution, so pickFuncProcessInternal -> preDebugValidate -> validateFuncCoreToolsInstalled still executes: only the trigger changes, not the product path under test.

    Failure signatures that mean you have hit this (all from real CI runs of p414-funcrepair):

    • [depValidation] VS Code title at start: "Preview WBD-hybrid announcement.md - testws_xxx (Workspace) - Visual Studio Code" — a restored markdown preview held focus (run 30549543233).
    • [openFileInEditor] Active tab is "settings.json", expected "workflow.json" — retrying ×3, then Warning: could not confirm active tab, and a window title stuck on a non-project file (run 30554041131). Deterministic, not flaky — it reproduced on both outer attempts and on the scenario retry.
    • Symptomatically: no repair AND no blocking modal, and grepping the whole job log for pickProcess / preDebugValidate / launch.json / No configuration returns zero hits from the extension side. Absence of both outcomes means the gate never ran.

    Two corollaries: (a) gate the recorder's debugInvoke event, not debugStarted — for a request: attach config startDebugging() only resolves after ${command:pickProcess} finishes, i.e. after the very repair you are waiting on, so debugStarted cannot be a fast-fail signal; (b) always add an explicit fast-fail that distinguishes "the product failed" from "the harness never triggered the product." A 300 s silent timeout costs a whole shard to learn "F5 did nothing."

19. The debug path prompts BEFORE the thing you are testing — poll for QuickPicks for the whole wait, don't pre-dismiss once (CRITICAL). pickFuncProcessInternal runs activateAzurite (pickFuncProcess.ts:87-106), then refreshConnectionKeys (:112-116), and only then preDebugValidate (:119). With no Azure account, refreshConnectionKeys raises GetSubscriptionDetailsStep.prompt (azureConnectorWizard.ts:50-68) — a showQuickPick with placeholder Enable connectors in Azure for Logic App <name> and rows Use connectors from Azure / Skip for now. It parks the entire debug flow until answered, so anything downstream of :116 never executes. Answering it with Skip for now (or Escape — azureConnectorWizard.ts:59-62 catches isUserCancelledError and returns the same { data: 'no' }) lets it through.

    This is not a one-off: prepareFreshSession() deletes the settings User dir every attempt and the choice is cached in ext.context.globalState (app/state/connectors.ts:13-22), so it appears on every run. It also arrives several seconds after the debug start, so a single pre-emptive dismissal races it — call dismissQuickPickIfVisible(driver) (helpers.ts:740) on every iteration of every polling wait that spans the debug start.

    Failure signature (p414-funcrepair, run 30557613214): the fast-fail liveness gate reported reached on the Azurite is setup to auto start line, then the repair wait timed out at 300 s with no repair and no blocking modal, the output channel frozen at that same Azurite line, and the screenshot showing the connectors QuickPick still open with the debug session live in the status bar.

    Two lessons beyond the dismissal itself:

    • A liveness marker from step 1 does not prove you reached step 3. Azurite is setup to auto start is written by pickFuncProcessInternal's first statement, so a gate keyed on it reports "the product reacted" while execution is parked two steps short of what you are testing. Track a separate set of markers that prove you got past the last step before your target — here Azure connectors are disabled. Skipping connection key refresh. / No connection keys found. (connectionKeys.ts:22/:29) — so a timeout can say stall-before-the-gate vs genuine product failure.
    • A prompt-dismisser must not be able to answer the prompt you are asserting on. showQuickPick renders .quick-input-widget; showWarningMessage(msg, { modal: true }, ...) renders .monaco-dialog-box. dismissQuickPickIfVisible only queries the former, so it cannot click a dialog button — but its Escape fallback fires on body and can cancel a modal. Read the widget text first and act only on a match for the specific prompt you mean to answer, refuse to run once the asserted-on prompt has been observed, and log both the dismissals and any QuickPick you deliberately left alone.
20. Watermarked log dumps must be pinned to the watermarked file path. A "find the newest *Azure Logic Apps*.log by mtime" helper is fine for taking a watermark but wrong for reading back against it: the newest file can change identity between the two calls, at which point you dump an unrelated file in full and call it "new output". In run 30557613214 this printed a node tarball listing on the failure path and hid a refreshConnectionKeys stall for an entire CI cycle. Resolve the path once, store it in the watermark, and read that file for every subsequent diff — treating a shorter file as rotated. Bound the dump (~8 KB) and, when the new region is empty, say so explicitly: "the extension wrote nothing after the trigger" is itself the diagnosis.

Architecture Decisions That Paid Off

• Phase separation (4.1 / 4.2): Allows re-running designer tests without re-creating workspaces
• tsup for test compilation: Reliable CJS output; tsc had inconsistent module format issues
• Workspace manifest: Single source of truth for workspace paths, prevents hardcoded path drift
• run-e2e.ts as orchestrator: Plain JS (no compile step needed), handles extension copying, dependency installation, process cleanup, and test execution in one script
• Multi-layered process cleanup: 6 fallback strategies for EBUSY errors; always succeeds eventually
• 2 focused tests instead of 15+ scattered tests: Reduced from 15+ tests (23 min, 20% pass rate) to 2 comprehensive end-to-end tests (5 min, ~80% pass rate). Each test covers the complete lifecycle.
• assert.ok() at every step: Each step has a clear assertion with a descriptive message. No more swallowed errors in try/catch blocks.
• Screenshots at every assertion point: 15+ screenshots per test run provide visual debugging evidence without needing to watch the test live.

What We'd Do Differently

• Start with detection-based polling instead of static sleeps. Static sleeps were the #1 cause of both flakiness and slow runtime.
• Start with 2 focused tests instead of 15+ tests that each test one small thing. The overhead of opening a workspace + designer is so high (~15s) that it's far more efficient to test the full lifecycle in fewer, longer tests.
• Pre-create local.settings.json in all workspace creation tests to prevent Azure connector prompts. Discovered late that this is needed for designer tests.
• Add silentAuth support from the start instead of discovering the auth dialog blocker late. Any test environment that opens the overview page needs this.
• Use VSBrowser.instance.openResources() from the start for file opening instead of building custom Quick Open logic. This ExTester API is more reliable.

VS Code Settings Required for Testing

The run-e2e.ts script generates these settings for the test VS Code instance:

{
  "extensions.autoUpdate": false,
  "extensions.autoCheckUpdates": false,
  "update.mode": "none",
  "git.enabled": false,
  "git.openRepositoryInParentFolders": "never",
  "azureLogicAppsStandard.autoRuntimeDependenciesPath": "~/.azurelogicapps/dependencies",
  "azureLogicAppsStandard.funcCoreToolsBinaryPath": "<path>/func.exe",
  "azureLogicAppsStandard.dotnetBinaryPath": "<path>/dotnet.exe",
  "azureLogicAppsStandard.nodeJsBinaryPath": "<path>/node.exe",
  "azureLogicAppsStandard.silentAuth": true
}

Key settings:

• Runtime dependency paths: Point to pre-installed binaries to avoid download delays
• silentAuth: true: Prevents the auth dialog when opening the overview page
• git.enabled: false: Prevents git operations that slow down workspace switching
• Auto-update disabled: Prevents marketplace extension replacement

16. Pattern: Deterministic Task-Event Capture via a Fake Listener Extension (Phase 4.10)

Use this pattern when you need to assert which VS Code tasks the product invokes during a workflow (debug, build, deploy, etc.) without depending on the tasks producing real artifacts.

Why this pattern exists

vscode-extension-tester runs Mocha tests inside the extension host, but the Mocha process and the extension host run in different V8 isolates. Test code cannot subscribe to vscode.tasks.onDidStartTask / onDidEndTaskProcess from outside the extension host — only an extension can. Polling the VS Code UI for task progress is also unreliable (the Output panel scrolls, terminal IDs are unstable, and dotnet/func binaries hang the test for minutes if their feeds are slow).

The pattern

Ship a tiny test-only "recorder" extension alongside the product extension and observe its JSONL output from the Mocha test.

Files for Phase 4.10 (regression guard for pickFuncProcess skipping publishCodefulProject when the modern codeful csproj already populates lib/codeful via Build hooks):

• src/test/ui/codefulTaskRecorderExtension/package.json — publisher: la-e2e, name: la-e2e-codeful-task-recorder, activationEvents: ["onStartupFinished"].
• src/test/ui/codefulTaskRecorderExtension/main.js — CommonJS extension that:
  1. Writes an activate JSONL entry to ${LA_E2E_TASK_EVENTS_JSONL} synchronously on activation (acts as the "recorder is ready" signal).
  2. Subscribes to vscode.tasks.onDidStartTask, onDidEndTask, onDidStartTaskProcess, onDidEndTaskProcess and appends { phase, taskName, scopeFsPath, processId, exitCode, timestamp } for each.
  3. Polls ${LA_E2E_TRIGGER_DIR} every 500 ms for marker files (start-debug, stop-debug, ping); consumes (unlinks) them and dispatches.
  4. For start-debug: waits for azureLogicAppsStandard.debugLogicApp to appear in vscode.commands.getCommands(true), reads launch.json from disk (JSONC-stripped), picks the first type: 'logicapp' config, calls vscode.debug.startDebugging(folder, configName), and writes debugStarted / debugStartFailed accordingly.
• src/test/ui/createWorkspace.test.ts — Phase 4.10A creates real codeful workspaces through the Create Workspace webview and records them in the shared manifest.
• src/test/ui/codefulDebugTasks.test.ts — Mocha test that:
  1. Reads the JSONL file once for the activate entry (waits up to 90 s).
  2. Runs one variant per fresh Phase 4.10B session, with the generated .code-workspace passed as the startup resource.
  3. Waits for design-time evidence (workflow-designtime/), truncates the JSONL, drops a start-debug marker, and polls the JSONL for the expected task chain.
  4. Summarizes counts of processStart per task name and asserts the modern variant has publishStart === 0 and the legacy variant has publishStart === 1.
• src/test/ui/run-e2e.js:
  • runCodefulDebugPhases() runs Phase 4.10A (Create Workspace) and Phase 4.10B (fresh reopen + recorder) using the D-001 create-and-reopen pattern.
  • patchLegacyCodefulCsproj(entry) minimally changes only the generated legacy control .csproj, replacing CopyToCodefulFolder AfterTargets="Build;Publish" with AfterTargets="Publish".
  • installCodefulTaskRecorderExtension(extDir) copies the recorder source dir to ${extDir}/la-e2e.la-e2e-codeful-task-recorder-0.0.1/ and refreshes extensions.json.
  • e2eMode === 'codefuldebugonly' branch runs Phase 4.10 standalone.
  • recorder: true on any scenarios[] entry opts that scenario into the same recorder: runScenarioPhases() installs it into the scenario's ext dir and calls configureCodefulRecorderEnvironment() after each prepareFreshSession(), so the test file can read LA_E2E_TASK_EVENTS_JSONL and write markers into LA_E2E_TRIGGER_DIR. Phase 4.14 (p414-funcrepair) uses this purely as a deterministic debug trigger — see rule 18.

Wire-up checklist

1. Package the recorder as a real extension. Real publisher, name, version, and activationEvents. Install it into <extDir>/<publisher>.<name>-<version>/ (note the dot between publisher and name in the folder name). Refresh extensions.json with rebuildExtensionsJson(extDir) so VS Code picks it up.
2. Use the activate JSONL entry as the readiness signal, NOT a command palette ping. The command palette is element not interactable right after a workspace switch. Drop the palette dependency entirely.
3. Use file-watcher triggers for test → recorder communication. Drop marker files via fs.writeFileSync and let the recorder poll. This bypasses ExTester's flaky workbench.executeCommand path.
4. Wait for azureLogicAppsStandard.debugLogicApp before calling vscode.debug.startDebugging. The LA extension's activate() registers the logicapp debug provider synchronously but only registers commands after several awaits (getResourceGroupsApi, startOnboarding, tasks.fetchTasks). Calling startDebugging before commands register fails with command 'azureLogicAppsStandard.debugLogicApp' not found.
5. Open the generated .code-workspace, NOT the project folder. Phase 4.10B passes the real .code-workspace created by Phase 4.10A as the runPhase() startup resource. Do not hand-write a wrapper or open only the project folder.
6. Use the generated tasks as-is for the modern control. The regression guard must exercise the real codeful launch/tasks shape produced by the Create Workspace flow.
7. Patch only the legacy control .csproj. To simulate the legacy template, change the generated CopyToCodefulFolder target from AfterTargets="Build;Publish" to AfterTargets="Publish"; do not synthesize project files or tasks.
8. Pre-set azureLogicAppsStandard.pickProcessTimeout: 15 in test settings. This bounds the debug process picker after the task chain has been dispatched.
9. Allow 10-15 minutes for the LA extension's cold-start activation per variant. vscode.tasks.fetchTasks() activates every task-type provider (grunt, gulp, jake, npm) and waits for all of them. Combined with the LA extension's own awaits, the first taskStart event can arrive 4-10 minutes after vscode.debug.startDebugging is called.
10. Sleep ≥ 30 s after stopDebug() before reading events. The task chain may finish writing processEnd entries after the wait deadline. The post-stop sleep gives the recorder time to flush remaining events to disk.
11. Assert on the JSONL summary, not the wait result. The wall-clock wait is an upper-bound optimization. The actual signal is what's in the JSONL after stopDebug + sleep + readEvents.
12. The recorder picks its own folder and config — do not assume folders[0]. pickDebugTarget() selects, in order: the workspace folder whose .vscode/launch.json contains a config with processId referencing azureLogicAppsStandard.pickProcess, else the first folder with any config, else folders[0]. Within that folder it prefers (a) the pickProcess config, (b) type === 'logicapp', (c) configs[0]. This matters because vscodeLaunch.ts generates the Standard codeless config as type: 'coreclr', request: 'attach', processId: '${command:azureLogicAppsStandard.pickProcess}' — not type: 'logicapp' — so without preference (a) it would only be found by the configs[0] fallback. Codeful/custom-code configs have no processId, so (a) never matches them and Phase 4.10's selection is unchanged.
13. Rules 9 and 10 are Phase-4.10-specific. They exist because Phase 4.10 asserts on task events, and vscode.tasks.fetchTasks() is what makes the first taskStart take minutes. A test that only needs a debug session started (e.g. Phase 4.14) should gate on debugInvoke and skip those long waits entirely.

Reusing this pattern

This pattern generalizes to any test that needs to observe extension-host-only events deterministically:

• Task events: as above (Phase 4.10).
• Debug session events: subscribe to vscode.debug.onDidStartDebugSession / onDidTerminateDebugSession.
• Workspace events: onDidChangeWorkspaceFolders, onDidChangeConfiguration.
• Command invocations (via commands.registerCommand shimming for assert-on-invocation): wrap a product command at registration and log invocations to JSONL.

Always:

• Put the recorder in src/test/ui/<feature>RecorderExtension/.
• Use a unique publisher (e.g., la-e2e) so it won't collide with the marketplace.
• Activate onStartupFinished.
• Write structured JSONL with a phase discriminator.
• Use file-watcher triggers for one-way test → recorder commands.

Anti-precedent flag

Synthetic codeful Phase 4.10 fixtures are prohibited by Rule 1 and D-001. Keep Phase 4.10 on the real Create Workspace + fresh reopen pattern; if the codeful wizard flow changes, update createWorkspace.test.ts helpers rather than writing project files directly to disk.

17. Pattern: Running One Scenario for Both App Kinds (Phase 4.13 APP KIND axis)

Phase 4.13 (Azurite auto-start readiness) runs the same scenario twice: once for a codeless (Standard) logic app and once for a codeful one. Use this pattern whenever a product code path is app-kind agnostic but the surrounding F5 flow is not.

When it is worth doing

Only when the product code under test is genuinely shared AND the second kind adds an assertion the first cannot make. For 4.13:

• pickFuncProcessInternal (app/commands/pickFuncProcess.ts) calls activateAzurite at line 85 unconditionally, before preDebugValidate (:100), before publishCodefulProject (:116) and before startFuncTask (:146). The debugConfig.isCodeless read is at :142.
• So a codeful F5 traverses the identical Azurite readiness code, and it lets 4.13B assert the codeful-only property: the bounded failure happens before the expensive publish/build.

If the second kind cannot add a new assertion, do not duplicate the run — you are just buying CI minutes.

The mechanism

1. One env var, read at module scope, in every file of the phase group. AZURITE_E2E_APP_KIND = codeless (default) | codeful. runPhase() drops the compiled test file from require.cache before each phase, so re-evaluating module scope is what makes the gate select a different layout. Same mechanism as AZURITE_E2E_STEP / LA_E2E_CODEFUL_CREATE_ONLY.
2. Throw on an unknown value at module scope. Only the launcher sets it, so a typo is a harness bug. Silently defaulting to codeless would run the covered kind twice and still report green.
3. Disjoint on-disk layout per kind — different workspace parent AND workspace/app names (azuritews/azuriteapp vs azuritecfws/azuritecfapp). Sharing a parent invites cross-kind contamination of .code-workspace, workflow-designtime and .vscode/launch.json.
4. Keep the default byte-for-byte identical to the pre-existing behavior, so an unset env var reproduces the historical run exactly.
5. Emit a module-scope layout banner. It prints during Mocha''s file load, so mocha --dry-run and every CI log record which layout each phase resolved.
6. Launcher loops the kinds and wraps EACH kind in withPhaseGroupRetries, not the whole sweep — a flake in the second kind must not re-run an already-green first kind. Do not double-wrap at the call site.

Kind-specific things you must not get wrong

• fillStandardFormFields takes the display label (Logic app (codeful)), not the short manifest kind (codeful).
• A codeful app has no workflow.json. Verify it with deepVerifyWorkspace(..., { appType: ''codeful'' }), which requires <wfName>.cs, <appName>.csproj, Program.cs, host.json, local.settings.json and fails if a workflow.json exists.
• The codeful launch config is isCodeless: false with no preLaunchTask (CreateLogicAppVSCodeContents.getDebugConfiguration). Mirror what the product generates; do not invent a shape.
• Assert the kind marker: local.settings.json Values.WORKFLOW_CODEFUL_ENABLED === ''true'' for codeful and not-''true'' for codeless. hasCodefulWorkflowSetting() is what routes F5 into the codeful branch, so without this a "passing" codeful run may have silently exercised the codeless path.
• Codeful does produce <app>/workflow-designtime: isLogicAppProject() accepts it via hasCodefulSdkReference(), so startAllDesignTimeApis() reaches ensureDesignTimeDirectory(). Phase 4.10B already hard-asserts this. Reuse the same freshness-checked design-time gate for both kinds; do not degrade it for codeful.

Asserting "it failed before the build"

Assert on build output that only a real build can produce, and clear it immediately before F5 so the check is not vacuous:

• <app>/lib/codeful is written only by the CopyToCodefulFolder MSBuild target (src/assets/CodefulProjectTemplate/CodefulProj, AfterTargets="Build;Publish").
• <app>.dll under <app>/bin or <app>/lib is emitted only by dotnet build/publish.
• Do not assert on obj/ — a C# Dev Kit design-time evaluation legitimately writes it.
• Treat a locked bin/lib at clear-time as a hard failure, not a skip: it means a build is already running, which invalidates the premise.

18. Rule: Start debug sessions from the recorder extension, never from the command palette

Referenced as "SKILL.md rule 18" by funcRepair.test.ts and by the recorder flag in run-e2e.ts's scenario table.

workbench.executeCommand('Debug: Start Debugging') resolves the launch configuration from whatever editor happens to be active, and a headless CI session does not settle focus reliably. Two CI runs proved it: run 30549543233 debugged an auto-opened markdown preview, and run 30554041131 debugged settings.json even after closing all editors and opening workflow.json via Quick Open three times. Both produced the same useless outcome — VS Code never resolved the Logic App folder's launch.json, pickProcess never ran, and the test timed out blaming the product.

Use the bundled task-recorder extension instead (see section 16). It calls vscode.debug.startDebugging(folder, configName) with an explicit folder, so the trigger has no dependency on focus at all.

Wire-up: set recorder: true on the scenario in run-e2e.ts. That installs the extension once per scenario and resets the events/trigger files per attempt via configureCodefulRecorderEnvironment(). Drop a marker file into LA_E2E_TRIGGER_DIR to start or stop the session, and always gate on the recorder reporting activate/ping first, so a wedged recorder is distinguishable from a product failure.

19. Porting an existing scenario to a Windows CI leg

Everything below was learned wiring p414-funcrepair (Phase 4.14) onto windows-latest in .github/workflows/vscode-e2e.yml. Read it before adding any further Windows job.

The workspace-fixtures-<sha> artifact is ubuntu-bound — do NOT consume it on Windows

created-workspaces.json records absolute paths (wsDir / wsFilePath / appDir, see workspaceManifest.ts), written under the ubuntu runner's RUNNER_TEMP (/home/runner/work/_temp/...). Extracting that tarball on windows-latest puts the tree under D:\a\_temp\la-e2e-test while every path inside the manifest still points at a directory that does not exist there. Nothing rebases them, so selectWorkspaceForSpec() falls through to resources: [] and the test opens a nonexistent .code-workspace. This is why no Windows job in the workflow consumes it.

A Windows leg that needs fixtures must create them on that runner by running p41a-fixtures first (which is Rule 1 compliant — real Create Workspace wizard). The logic-apps-extension-bundle-<sha> artifact is unnecessary too, because that run downloads and verifies the bundle itself. Net effect: a Windows leg consumes only the cross-platform JS extension-build artifact, exactly like vscode-e2e-codeful-windows.

A running .exe is unwritable on Windows — Linux cannot reproduce it

Any test that overwrites a runtime binary in place (funcRepair.test.ts corrupts func.exe) is a no-op risk on Linux and a hard failure on Windows. Three things must hold:

1. The scenario runs with autoStartDesignTime: false, so the session under test never spawns a design-time func host start.
2. prepareFreshSession() actually killed leftover func hosts from the PREVIOUS scenario. p41a-fixtures runs with autoStartDesignTime: true, so a Windows job that runs it before the corrupting scenario depends on that kill landing. Each PowerShell kill is now guarded individually and followed by a bounded poll (waitForWindowsFuncProcessesToExit) that confirms the processes are actually gone — Stop-Process only signals.
3. The runtime-dependency cache is WARM. A cold cache means activation performs a real install, and installFuncCoreToolsBinaries ends in startAllDesignTimeApis() (binaries.ts:213), which does not honour autoStartDesignTime. The resulting live func.exe makes the corruption fail with EPERM/EBUSY. Hence needs: setup-runtime-deps-windows plus the local p41a-fixtures step as a second line of defence.

A job that corrupts runtime dependencies must be cache RESTORE-ONLY

Never pair a corrupting scenario with actions/cache/save. afterEach restores the pristine bytes, but a crashed session can leave a corrupted or partly-restored tree, and saving that onto la-runtime-deps-Windows-v1 is unrecoverable: later runs restore it as an EXACT-key HIT and the hydrating jobs' cache-hit != 'true' guard suppresses the repair. The deps-check guard used elsewhere is not enough — it probes only the FIRST func it finds via find -maxdepth 2, so a still-corrupted in-proc6/func.exe slips past it.

Close the this.skip() hole with a manifest guard step

Tests that this.skip() when the manifest is missing report a passing job with zero coverage. On a brand-new platform leg that is the most likely silent failure. Add an explicit guard step between fixture creation and the scenario that asserts the manifest exists, has the entry the test looks for, and that the entry's wsFilePath exists on disk. Resolve the path with node + os.tmpdir() rather than $RUNNER_TEMP, so it checks the exact path the test will read.

shell: bash on windows-latest — env var gotchas

• RUNNER_TEMP contains backslashes (D:\a\_temp). Do not feed it to bash path tests; $HOME is POSIX (/c/Users/runneradmin) and is what the existing steps use.
• Set TEMP, TMP and TMPDIR — os.tmpdir() reads TEMP/TMP on Windows, and the screenshot/backup/manifest paths all derive from it.
• LA_E2E_STRICT_DEPENDENCY_VALIDATION is read by the product inside the VS Code process (strictDependencyValidation.ts), not just by the harness. Scope it to the step that needs it; leaking it into a corrupting scenario can force a reinstall during activation, which ends in startAllDesignTimeApis() and a live func.exe.

Sizing the timeout

Follow the policy documented on setup-extension-build: a backstop against a wedged runner, ~2x the worst HEALTHY observation, sized to absorb one LA_E2E_SCENARIO_RETRIES retry and a cold cache. Prefer borrowing a real observation from a job that already runs the same scenario on the same OS (setup-runtime-deps-windows runs p41a-fixtures on Windows and is the only existing datapoint for it) over copying a neighbour's number. Remember the in-test TEST_TIMEOUT already hard-bounds a single attempt, so the fast-failure signal lives there, not in timeout-minutes.

20. Pattern: Custom-code .NET version create/assert pair (Phase 4.15)

Phase 4.15 covers the CustomCode wizard's framework availability and .NET 10 workspace generation. The net10 target verifies the Logic App root local.settings.json, generated project template, and workflow shape. It intentionally does not build or run the project while Microsoft.Azure.Workflows.WebJobs.Sdk 1.4.0 is pending release. The net8 target verifies that .NET 8 does not appear in the custom-code framework picker. Existing .NET 8 project detection and compatibility behavior remains covered outside the new-project wizard.

• src/test/ui/customCodeDotNetVersionCreate.test.ts (4.15A) drives the real Create Workspace webview. For net10, it creates a Logic app with custom code + Stateful workspace. For net8, it opens the framework dropdown and asserts .NET 8 is absent.
• src/test/ui/customCodeDotNetVersionAssert.test.ts (4.15B) independently recomputes the generated net10 workspace layout and asserts the dotnet-version setting, net10.0 target framework, Workflows SDK 1.4.0 package reference, Request trigger, and generated InvokeFunction action. It does not build or start the workspace until Workflows SDK 1.4.0 is released.

No shared manifest. Unlike the declarative scenarios[] table entries, this pair (like Azurite and Codeful debug before it) uses fixed, deterministic temp paths (os.tmpdir()/la-e2e-test/customcode-dotnet-<target>-parent/cc<target>ws/...) computed independently by both test files from the same env var, instead of writing to or reading from created-workspaces.json. That manifest is already consumed by unrelated scenarios (p42-customcode/p43-customcode) keyed only on {appType: 'customCode', wfType: 'Stateful'} with no dotnet-version disambiguation — appending a second, differently-typed CustomCode entry would create an ambiguous match. Reusing the fixed-path idiom instead of inventing a new manifest field or WorkspaceSpec literal keeps the two scenario families independent.

The separate Phase 4.1 shared CustomCode fixture is still consumed by p42-customcode and p43-customcode for existing-project debug/run compatibility. createWorkspace.fixtures.test.ts creates it through the public .NET 10 wizard option, then run-e2e.ts normalizes only that shared fixture to the existing .NET 8 project template and setting before downstream scenarios consume it. Remove that normalization when Workflows SDK 1.4.0 is released and the runtime lifecycle moves to .NET 10.

Orchestration: runCustomCodeDotNetPhasesForTarget() / runCustomCodeDotNetPhases() in run-e2e.ts mirror runAzuriteReadinessPhasesForKind() / runAzuriteReadinessPhases() exactly — same writeTestSettings per phase (4.15A: {validateDependencies: true, autoStartDesignTime: true}, matching Phase 4.1a; 4.15B: {validateDependencies: false, autoStartDesignTime: false} because it validates files without building), same withPhaseGroupRetries wrapping per-target (not per-sweep), and the same disk-existence short-circuit before 4.15B if 4.15A did not produce the .code-workspace. The net8 negative picker test intentionally does not run 4.15B. E2E_MODE=customcodedotnetonly runs both targets by default; CUSTOMCODE_DOTNET_E2E_VERSIONS (comma-separated net8/net10, plural — mirrors AZURITE_E2E_APP_KINDS) shards it in CI, one target per matrix leg.

SDK selection: the net10 creation leg resolves the system dotnet installed by actions/setup-dotnet, verifies SDK 10 is available, and passes that exact binary through dotnetBinaryPathOverride for wizard validation. The net8 leg is a picker-visibility assertion and does not generate or build a net8 project.