Cloud E2E Tests

Run detached cloud E2E test attempts through the repository-owned launcher. The launcher uses pytest collection to discover every test module marked e2e. It then exits and hands Copilot every command, PID, log, result, and resubmission handle needed to babysit the runs.

Resolve the launcher from the active checkout:

REPO_ROOT="$(git rev-parse --show-toplevel)"
DRIVER="$REPO_ROOT/.github/skills/e2e-tests/scripts/run-e2e-tests.sh"

Operating Contract

Execute the workflow autonomously. Do not ask the user to run commands, monitor jobs, classify failures, or request routine retries.

1. Read Terraform outputs before any Azure, Kubernetes, OSMO, or pytest operation.
2. Stop if terraform output -json is unavailable, empty, stale against live resource IDs, or missing a required output. Do not infer deployed resource names from conventions, the current Kubernetes context, Azure resource enumeration, or remembered environments.
3. Validate Azure authentication, storage, and Azure ML compute before submitting tests. Validate OSMO authentication, the gateway, and AKS GPU pools when the selected subset includes OSMO tests.
4. Launch the requested test files concurrently, or every discovered default test when no subset is requested. Azure ML and OSMO use independent compute and do not contend with each other.
5. Capture E2E_HANDLE and take ownership of monitoring. The launcher never retries or diagnoses a completed attempt.
6. For every non-pass, inspect the exact command and complete output. Resubmit a classified transient failure only after evidence shows a supported remedy or changed cloud condition. Investigate other failures, correct clear in-scope defects, validate locally, then resubmit only the affected test.
7. Treat PASSED as success. Stop automatic submissions and report the unresolved result when the latest attempt repeats the same terminal error, cloud state, and evidence without a supported intervention or observable progress. Never alter Terraform or node-pool configuration to work around unavailable GPU capacity.
8. Report each test's commands, attempts, cloud job or workflow identifiers, durations, and final state. Run handle cleanup after all tests reach terminal states.

Each handle owns an XDG_CONFIG_HOME under its run directory. OSMO reads and rewrites login.yaml in that directory, so every launch, detached attempt, resubmission, and diagnostic command must load the handle's non-executable config.json.

The launcher accepts only a handle directory owned by the current user with mode 700; private configuration and process records must be owned by the current user with mode 600 and must not be symbolic links. Never use the global ~/.config/osmo/login.yaml while an E2E handle is active; concurrent deployments would redirect each other's CLI commands.

Local Dependency Bootstrap

Synchronize the frozen local environment before launching attempts. Use the package index configured for the current development environment:

cd "$REPO_ROOT"
uv sync --frozen --no-progress

Do not treat a direct files.pythonhosted.org connection failure as a cloud-test failure. Bootstrap through the configured index, then resubmit the affected terminal attempts.

Hugging Face Prerequisite

tests/e2e/test_e2e_aml_vla_pi0_training.py requires HF_TOKEN with access to the gated google/paligemma-3b-pt-224 model. Export the token before a default full run or before selecting that test explicitly. The requirement does not apply when an explicit subset omits tests marked requires_hf_token.

Never write the token to config.json, command.sh, logs, manifests, or chat output. Preserve it only in the process environment. A missing or empty token is a non-retryable client-side setup failure: the test must fail before Azure fixture resolution, dataset staging, or job submission.

Required Terraform Outputs

Run this first:

terraform -chdir=infrastructure/terraform output -json |
  jq '{
    resource_group,
    aks_cluster,
    azureml_workspace,
    storage_account,
    node_pools
  }'

This projection is safe for operator and Copilot output. Do not print or paste the unfiltered JSON because Terraform includes values marked sensitive, including credentials and connection details. The launcher stores the complete output only in the handle's owner-only terraform-output.json.

When the current feature worktree has no Terraform state or returns empty outputs, locate the main worktree with git worktree list --porcelain and use its infrastructure/terraform directory as the authoritative deployment source. Keep the feature worktree as the launcher repository root so pytest collects and runs the branch code:

"$DRIVER" \
  --repo-root <feature-worktree> \
  --terraform-dir <main-worktree>/infrastructure/terraform

Apply the same output and live-resource validation rules to the main worktree state. Stop if those outputs are unavailable, empty, stale, or incomplete.

Require these non-null outputs:

The committed node_pools output does not expose autoscaling bounds. Read live bounds with az aks nodepool list only after matching the AKS resource ID to Terraform.

Compute Validation

Validate the two independent backends before launching:

Scale-from-zero pools may have no registered GPU nodes. Do not interpret zero current nodes as a configuration failure when autoscaling can add capacity.

Launch

Run from the repository root:

"$DRIVER"

Select a subset by repeating --test:

"$DRIVER" \
  --test test_e2e_aml_il_pipeline \
  --test test_e2e_aml_il_lifecycle \
  --test test_e2e_osmo_il_lifecycle

The command exits after printing:

E2E_HANDLE=<absolute-run-directory>
E2E_STATUS_COMMAND=<absolute-script-path> --status <absolute-run-directory>
E2E_RESUBMIT_COMMAND=<absolute-script-path> --resubmit <absolute-run-directory> <test-name>
E2E_CLEANUP_COMMAND=<absolute-script-path> --cleanup <absolute-run-directory>

Save E2E_HANDLE. Each test owns one detached process. No coordinator or retry loop remains active after launch.

Handoff Data

The handle contains all data needed to investigate and reproduce an attempt:

Use command.sh as the authoritative record. Do not reconstruct commands from chat history or partial logs.

Tracking

Read aggregate and per-test state without attaching to the coordinator:

"$DRIVER" --status "$E2E_HANDLE"

Inspect a test's current attempt:

tail -n 100 "$E2E_HANDLE/<test-name>/attempt-<n>/output.log"

Use five-minute status intervals while jobs are provisioning or running. Use twenty-minute intervals when all pending jobs are blocked on documented GPU capacity. Do not poll continuously.

Resubmit one terminal attempt:

"$DRIVER" \
  --resubmit "$E2E_HANDLE" <test-name>

This creates attempt-N+1 with a new exact command.sh; it never overwrites earlier evidence.

After every test is terminal:

"$DRIVER" --cleanup "$E2E_HANDLE"

Cleanup verifies the exact recorded process identity for attempts left in STARTING or RUNNING. It refuses cleanup for a matching live attempt or any live PID whose identity cannot be verified. A dead stale attempt is marked INTERRUPTED, including its latest attempt result, before cleanup continues.

Test Discovery

The launcher runs pytest --collect-only over tests/e2e and derives the available E2E modules from every test marked e2e. The default selection includes the complete collected set, including tests with additional prerequisite markers such as requires_hf_token.

Retry Classification

The launcher labels these conditions transient but does not resubmit them:

• Watchdog exit code 124
• SkuNotAvailable
• node-disruption
• OSMO gateway HTTP 503 or 504
• upstream request timeout
• upstream connect error
• Timed out waiting for .* to start
• Timed out waiting for .* to complete
• did not .* complete within
• to start within

Before resubmitting, compare the latest attempt's command, exit status, cloud job or workflow state, and terminal error signature with the preceding attempt. Resubmit only when a supported intervention occurred or the evidence shows progress, such as refreshed authentication, recovered gateway health, changed capacity state, or a corrected in-scope defect. Each resubmission creates a billable workflow or job.

Stop automatic submissions when consecutive attempts repeat the same failure evidence without a supported remedy or changed condition. Report the command, attempts, cloud identifiers, and blocking evidence as an unresolved terminal handoff. Do not impose a universal retry count; evidence and progress determine whether another submission is justified. OSMO's _await_osmo_status_with_restarts already handles task-level spot disruptions inside one invocation; do not duplicate it.

Each E2E handle exports a private XDG_CONFIG_HOME and selects a free handle-local OSMO_GATEWAY_PORT before OSMO login. Preserve that profile and port for every attempt and resubmission so concurrent sessions targeting different deployments cannot rewrite each other's endpoint or contend for the same localhost listener. Every OSMO resubmission re-establishes the handle-local gateway login and selects the default pool inside that isolated profile.

After concurrent OSMO submissions return gateway 503/504 responses, serialize resubmissions. Wait for the active OSMO workflow to reach a terminal state before submitting the next test.

Failure Rules

• A pytest skip is FAILED_SETUP_SKIP, never a pass.
• A failure outside the retry classification is FAILED and requires investigation.
• Never resubmit a test while its state is STARTING or RUNNING. Use cleanup to recover a dead stale attempt before resubmitting it.
• Do not delete staged data manually; pytest finalizers own cleanup.
• Do not change GPU SKUs, regions, autoscaling limits, Terraform state, or node pools to make a test pass.
• Preserve every attempt log and XML result under the handle.
• If osmo-service returns repeated gateway 503/504 responses, inspect its container lastState before retrying. An OOMKilled termination at the 1 GiB limit requires increasing the service memory allocation and redeploying OSMO.
• Treat Azure CLI token-refresh TLS EOF and RemoteDisconnected failures during Blob staging or cleanup as transient. Refresh the storage token, then retry the test.
• An Azure ML parent pipeline in Running can still have a queued child; inspect child jobs before diagnosing progress.
• An OSMO pod in Pending is expected while AKS scales from zero. Confirm autoscaler configuration before attributing it to capacity.