Deploy RHDH

Deploy Red Hat Developer Hub to a cluster for E2E test execution using the existing local-run.sh workflow.

When to Use

Use this skill when you need a running RHDH instance to reproduce and fix a test failure.

Prerequisites

Before running the deployment, verify these tools are installed:

# Required tools (local-run.sh checks these automatically)
podman --version        # Container runtime
oc version              # OpenShift CLI
kubectl version --client # Kubernetes CLI
bw --version            # Bitwarden CLI (for local secrets)
node --version          # Runtime and standalone stream decoder
e2e-tests/node_modules/.bin/rhdh-e2e-secrets --help # Project-pinned secret profile CLI
jq --version            # JSON processor
curl --version          # HTTP client
rsync --version         # File sync
bc --version            # Calculator (for resource checks)

Install dependencies with yarn install in e2e-tests to provide the project-pinned CLI, then export an unlocked Bitwarden session with export BW_SESSION=$(bw unlock --raw). Only set RHDH_E2E_SECRETS_BIN when explicitly testing a different CLI build.

Podman Machine Requirements

The podman machine must be running with adequate resources:

podman machine inspect | jq '.Resources'
# Requires: >= 8GB RAM, >= 4 CPUs

If resources are insufficient:

podman machine stop
podman machine set --memory 8192 --cpus 4
podman machine start

Deployment Using local-run.sh

The primary deployment method uses e2e-tests/local-run.sh, which handles Bitwarden secret selection, cluster service account setup, RHDH deployment, and test execution.

Execution Rules

CRITICAL — deployment is a long-running operation:

1. Never run local-run.sh in the background. Operator installations can take 20-30 minutes. Use the Bash tool with timeout: 600000 (10 minutes) and if it times out, check the container log — do NOT assume failure.
2. Before starting a deployment, check for existing containers:

   podman ps --format "{{.Names}} {{.Status}}" | grep -i rhdh-e2e-runner

   If a deployment container is already running, wait for it to finish instead of starting a new one. Monitor via the container log:

   tail -f e2e-tests/.local-test/container.log

3. Never launch concurrent deployments. Two deployments to the same cluster will race and both fail. If a deployment appears stuck, check the container log and cluster state before deciding it failed. The runner rejects concurrent invocations in the same checkout and holds its lock until the wrapped runner stops, including on cancellation. After an uncatchable termination, confirm the previous runner has stopped before removing the stale lock as documented in e2e-tests/README.md.
4. How to detect actual failure vs slow progress: The operator install script outputs detailed debug logs. If the container log shows active progress (timestamps advancing), the deployment is still running. Only consider it failed if:
   • The podman container has exited (podman ps shows no running container)
   • AND the container log shows an error message (e.g., "Failed install RHDH Operator")

CLI Mode (Preferred)

CRITICAL: CLI mode requires -r and -t. Always supply -j with the full Prow job name to select the intended job; otherwise the runner defaults to the main OCP Helm PR job. Missing -r or -t enters interactive mode and can hang in automated contexts.

cd e2e-tests
./local-run.sh -j "$PROW_JOB_NAME" -r "$IMAGE_REPO" -t "$IMAGE_TAG"

Example — OCP job (deploy-only with -s):

cd e2e-tests
./local-run.sh -j periodic-ci-redhat-developer-rhdh-main-e2e-ocp-v4-20-helm-nightly -r rhdh-community/rhdh -t next -s

Example — K8s job (AKS/EKS/GKE) (full execution, no -s):

cd e2e-tests
./local-run.sh -j periodic-ci-redhat-developer-rhdh-main-e2e-eks-helm-nightly -r rhdh-community/rhdh -t next

Example — disconnected OCP nightlies (requires a real OpenShift cluster; connected is fine; no bastion):

cd e2e-tests
# Operator: prepare uses --to-registry OCP_INTERNAL; MIRROR_* bootstrapped in-container
./local-run.sh -j periodic-ci-redhat-developer-rhdh-main-e2e-ocp-disconnected-operator-nightly -r rhdh-community/rhdh -t next -s
# Helm: oc-mirror to the cluster image registry route
./local-run.sh -j periodic-ci-redhat-developer-rhdh-main-e2e-ocp-disconnected-helm-nightly -r rhdh-community/rhdh -t next -s

local-run.sh refuses disconnected jobs unless oc is logged into OpenShift. It passes DISCONNECTED=true and LOCAL_DISCONNECTED=1; the runner writes the mirror CA under ${SHARED_DIR}/disconnected-mirror/ and keeps the merged pull secret in a private container-only temporary directory. Prefer Operator first when validating OCP_INTERNAL, then Helm. Air-gap prepare details live in rhdh-operator — do not paste them here.

Parameters:

• -j / --job: The full Prow CI job name extracted from the Prow URL. The openshift-ci-tests.sh handler uses bash glob patterns (like *ocp*helm*nightly*) to match, so the full name works correctly. Example: periodic-ci-redhat-developer-rhdh-main-e2e-ocp-v4-20-helm-nightly
• -r / --repo: Image repository (required for CLI mode — without it the script enters interactive mode)
• -t / --tag: Image tag (e.g., 1.9, next)
• -s / --skip-tests: Deploy only, skip test execution. OCP jobs only — K8s jobs (AKS, EKS, GKE) do not support this flag and require the full execution pipeline

WARNING: Do NOT use shortened job names like nightly-ocp-helm for -j — these do not match the glob patterns in openshift-ci-tests.sh.

Image Selection

Refer to the e2e-fix-workflow rule for the release branch to image repo/tag mapping table.

Deploy-Only Mode (OCP Jobs Only)

For OCP jobs, deploy without running tests so you can run specific tests manually:

./local-run.sh -j "$PROW_JOB_NAME" -r "$IMAGE_REPO" -t "$IMAGE_TAG" -s

Note: K8s jobs (AKS, EKS, GKE) do not support deploy-only mode. They require the full execution pipeline — run without -s.

What local-run.sh Does

1. Validates prerequisites: Checks required tools, podman resources, the project-pinned secrets CLI, and the Bitwarden session
2. Verifies the image: Checks the image exists on quay.io via the Quay API
3. Pulls the runner image: quay.io/rhdh-community/rhdh-e2e-runner:main
4. Loads Bitwarden secrets: Retrieves and validates the rhdh-qe profile through rhdh-e2e-secrets before any cluster mutation
5. Sets up cluster access: Creates rhdh-local-tester service account with cluster-admin, generates an 8-hour token
6. Copies the repo: Syncs the local repo to .local-test/rhdh/ (excludes node_modules)
7. Runs a Podman container: Executes container-init.sh inside the runner image, which:
   • Receives the profile-selected FD 3 stream through Podman stdin and decodes it into private files in /tmp/secrets tmpfs before dependency installation, matching the original CI sourcing path
   • Logs into the cluster
   • Sets platform-specific environment variables
   • Runs .ci/pipelines/openshift-ci-tests.sh for deployment

Post-Deployment: Setting Up for Manual Testing

After local-run.sh completes (with -s for OCP jobs, or after full execution for K8s jobs), run Playwright on the host against the URL printed by the deployment:

export BW_SESSION=$(bw unlock --raw)
export BASE_URL=https://showcase.example.com
export NAME_SPACE=showcase
e2e-tests/local-test.sh -- --project=showcase --headed
# or
export BASE_URL=https://showcase-rbac.example.com
export NAME_SPACE_RBAC=showcase-rbac
e2e-tests/local-test.sh -- --project=showcase-rbac --headed

Use the actual deployed namespaces if they differ from these defaults. Set NAME_SPACE_RUNTIME for runtime tests. local-test.sh loads the Bitwarden profile for Playwright. Set K8S_CLUSTER_URL and K8S_CLUSTER_TOKEN explicitly for cluster-aware projects; the script does not generate them or read deployment configuration.

Verify RHDH is accessible:

curl -sSk "$BASE_URL" -o /dev/null -w "%{http_code}"
# Should return 200

Deployment Error Recovery

Common Deployment Failures

CrashLoopBackOff

Symptoms: Pod repeatedly crashes and restarts.

Investigation:

# Check pod status
oc get pods -n <namespace>
# Check pod logs
oc logs -n <namespace> <pod-name> --previous
# Check events
oc get events -n <namespace> --sort-by=.lastTimestamp

Common causes and fixes:

1. Missing ConfigMap: The app-config ConfigMap wasn't created → check .ci/pipelines/resources/config_map/ for the correct template
2. Bad plugin configuration: A dynamic plugin is misconfigured → check dynamic-plugins-config ConfigMap against .ci/pipelines/resources/config_map/dynamic-plugins-config.yaml
3. Missing secrets: Required secrets not mounted → verify secrets exist in the namespace
4. Node.js errors: Check for JavaScript errors in logs that indicate code issues

ImagePullBackOff

Investigation:

oc describe pod -n <namespace> <pod-name> | grep -A5 "Events"

Common causes:

1. Image doesn't exist: Verify on quay.io: curl -s 'https://quay.io/api/v1/repository/rhdh/rhdh-hub-rhel9/tag/?filter_tag_name=like:<tag>'
2. Pull secret missing: Check namespace::setup_image_pull_secret in .ci/pipelines/lib/namespace.sh
3. Registry auth: Ensure the pull secret has correct credentials

Helm Install Failure

Investigation:

helm list -n <namespace>
helm status <release-name> -n <namespace>

Common causes:

1. Values file error: Check merged values against .ci/pipelines/value_files/values_showcase.yaml
2. Chart version mismatch: Verify chart version with helm::get_chart_version from .ci/pipelines/lib/helm.sh

Operator Deployment Failure

Investigation:

oc get backstage -n <namespace>
oc describe backstage <name> -n <namespace>
oc get csv -n <namespace>  # Check operator subscription status

Common causes:

1. Backstage CR misconfigured: Compare against .ci/pipelines/resources/rhdh-operator/rhdh-start.yaml
2. Operator not installed: Check CatalogSource and Subscription
3. CRD not ready: Wait for CRD with k8s_wait::crd pattern from .ci/pipelines/lib/k8s-wait.sh

Cross-Repo Investigation

When deployment issues stem from the operator or chart, search the relevant repos using whichever tool is available. Try them in this order and use the first one that works:

1. Sourcebot (if available): search rhdh-operator and rhdh-chart repos for specific error patterns or configuration keys
2. Context7 (if available): query redhat-developer/rhdh-operator or redhat-developer/rhdh-chart for docs and code snippets
3. Fallback — gh search code: gh search code '<pattern>' --repo redhat-developer/rhdh-operator or redhat-developer/rhdh-chart
4. Fallback — local clone: clone the repo into a temp directory and grep for the pattern

Key areas to look for:

• rhdh-operator: Backstage CR configuration, CatalogSource setup, operator installation scripts
• rhdh-chart: Helm values schema, chart templates, default configurations

Reference Files

• Main deployment scripts: .ci/pipelines/openshift-ci-tests.sh, .ci/pipelines/utils.sh
• Library scripts: .ci/pipelines/lib/helm.sh, .ci/pipelines/lib/operators.sh, .ci/pipelines/lib/k8s-wait.sh, .ci/pipelines/lib/testing.sh
• Helm values: .ci/pipelines/value_files/
• ConfigMaps: .ci/pipelines/resources/config_map/
• Operator CRs: .ci/pipelines/resources/rhdh-operator/
• Environment variables: .ci/pipelines/env_variables.sh