Debug build failures

Investigate main and branches named N.N (for example, 1.26 or 1.25). If the workflow provides a branch input, investigate only that branch, after validating that it is main or matches ^[0-9]+\.[0-9]+$.

MCP tool naming

This skill calls the Buildkite MCP server. The server is named buildkite in both Claude Code (local) and the GitHub Agentic Workflow, so tool names are always mcp__buildkite__<tool_name> — e.g. mcp__buildkite__list_builds. No changes are needed when switching between local and GH AW execution.

Step 1 — Find recent builds and eligible branches

Use mcp__buildkite__list_builds (org_slug: elastic, pipeline_slug: golang-crossbuild, per_page: 100) to find builds from the last seven days on eligible branches. Follow pagination as needed. Unless a branch input restricts the analysis, include only main and branch names matching ^[0-9]+\.[0-9]+$; exclude all other branches. Inspect enough history on each eligible branch to determine whether each recent failure was subsequently recovered, including builds newer than that failure.

For each branch, a failed build is relevant only if there is no newer successful build on that branch in the available build history. If any later build succeeded, treat the earlier failure as recovered/intermittent and do not report it. Apply this check to the whole branch, not just adjacent builds or only the last failure. Failures within the seven-day window without a later successful build remain reportable. If none remain, report the branch as healthy.

Record in-window passing builds and only unrecovered failing builds for the report table. Exclude every failed build followed by a success on the same branch. Sort the combined table by build creation date descending (newest first). Use the numeric build number returned by Buildkite as a plain value in the table. Do not add Buildkite links or URLs in issue text.

Step 2 — Get the failure summary

Call mcp__buildkite__get_build_failure_summary for the most recent relevant failing build (org_slug: elastic, pipeline_slug: golang-crossbuild, log_tail: 100).

This single call returns:

• build.state and job_state_counts — overall picture
• jobs — the terminal problem jobs with bounded log tails
• Any annotations and failed test runs

Step 3 — Reproduce in Docker before classifying the failure

Before declaring any root cause, try to reproduce the failing behavior in a container that matches the affected image/arch. The goal is to distinguish a confirmed cause from a plausible guess based only on log lines.

For each failing job:

• Identify the exact Dockerfile template and target architecture.
• Re-run the smallest equivalent build or package install in Docker using the same Debian version and arch.
• Record whether the same error appears in the container output.
• If a reproduction does not work, keep the diagnosis tentative and continue investigating instead of asserting a root cause.

Minimal reproduction patterns:

Pattern A — apt dependency conflict (most common)

Reproduce the same package/install path in Docker before concluding it is an apt sources issue. For Debian 12 cross-arch failures such as go/armhf or go/armel, the relevant reproduction is the actual Dockerfile target, not a generic package install:

docker build --progress=plain --no-cache --platform linux/armhf \
  -f go/armhf/Dockerfile.tmpl \
  --build-arg VERSION=1.27.1 \
  .

If you need a focused probe for the exact package conflict, use the same multi-arch setup that the image performs:

docker run --rm --platform linux/amd64 debian:12 bash -lc '
  set -eux
  dpkg --add-architecture armel
  dpkg --add-architecture armhf
  apt-get -qq update
  for arch in amd64 armel armhf; do
    echo "--- $arch ---"
    apt-cache madison libpcre2-8-0${arch:+:$arch} 2>/dev/null | grep security || echo "(none)"
  done
  apt-get install -y --no-install-recommends crossbuild-essential-armhf linux-libc-dev-armhf-cross
'

This is the failure mode behind the Multi-Arch: same errors seen in Debian 12 armhf/armel images: the security repo has amd64 updates while the secondary ports have not yet published matching versions, so apt rejects the foreign-arch install. If the failure is tied to a repo/content mismatch, include the affected sources-debian*.list file in the reproduction check and verify whether the package version from the repo actually matches the image state.

Pattern B — Docker build failure (non-apt)

Replay the same Docker build using the exact target, no cached layers, and the affected arch:

docker build --progress=plain --no-cache --platform linux/amd64 -f go/main/Dockerfile.tmpl .

If the build is generated from templates, reproduce the precise image target from the failing Makefile job instead of guessing from the error snippet.

Pattern C — Go build / test failure

Re-run the same build or test target inside the image used by the job:

docker run --rm --platform linux/amd64 <image-tag> bash -lc 'make <target> || go test ./...'

Pattern D — Infrastructure / agent failure

A Docker reproduction is not the primary tool here; still check whether the job is transient and whether rerunning the same containerized step is stable.

Only after a Docker reproduction or equivalent targeted build confirms the issue should the workflow classify it as a root cause.

Step 3A — Classify the failure

Read the log tails and classify the root cause. Common failure patterns in this repo:

Pattern A — apt dependency conflict (most common)

Log contains lines like:

E: Unable to correct problems, you have held broken packages.
<pkg> : Depends: <dep> (= X) but Y is to be installed

Go to Step 4A.

Do not assume this means stale apt metadata. Exact-version mismatches can also come from different package versions selected for native and foreign architectures, stale packages already installed in the base image, or security repositories publishing updates for architectures at different times. Identify the image, architecture, source list, installed version, and candidate version before recommending a fix.

Pattern B — Docker build failure (non-apt)

Log contains ERROR: failed to build or failed to solve for a reason other than apt. Identify the failing Dockerfile:NN line and read the relevant Dockerfile template.

Pattern C — Go build / test failure

Log shows go build or go test errors. Read the relevant source.

Pattern D — Infrastructure / agent failure

Log shows agent disconnection, timeout, or cloud provider errors. Usually transient — check if a retry would fix it before digging deeper.

Step 4A — Diagnose apt dependency conflicts

4A-1. Identify which Makefile target failed

The job name tells you: e.g. Ubuntu X86_64 - Makefile.debian12 → target is Makefile.debian12.

Read the relevant Makefile (e.g. go/Makefile.debian12) to see which image directories are built and in what order. Build stops at the first failure.

4A-2. Identify the conflicting package and the Dockerfile

The error message names the package (e.g. libpcre2-8-0). The log also shows the Dockerfile line number (e.g. Dockerfile:23). Read the Dockerfile step immediately before the failure too: it may add a foreign architecture, upgrade packages, or alter apt sources. Build order matters; the first failing image can prevent later images from running, while a grouped log may contain multiple independent image builds.

Map the failing Docker image to its template:

• go/main → go/main/Dockerfile.tmpl
• go/base → go/base/Dockerfile.tmpl
• go/base-arm → go/base-arm/Dockerfile.tmpl
• go/armel → go/armel/Dockerfile.tmpl
• go/armhf → go/armhf/Dockerfile.tmpl
• etc.

Read that template. Note:

• Which RUN apt-get/apt install blocks exist
• Whether apt-get update is run before each install block (missing update = stale package list)
• Which Debian version is in scope (check sources-debian${DEBIAN_VERSION}.list)

4A-3. Trace installed versions, candidates, and sources per architecture

For the failing image and Debian version, read its effective sources-debian${VERSION}.list and its FROM base image. Debian 12+ images may also need to remove the upstream /etc/apt/sources.list.d/debian.sources; check the template's source replacement rather than assuming the checked-in list is the only active source. Check any [arch=...] filters: the native and foreign architecture may be receiving different indexes by design.

Classify the exact error:

• Depends: dep (= X) but Y is to be installed: an exact-version constraint. Find which repository and architecture provide both versions; do not infer that an apt-get update alone will make them compatible.
• Depends: dep (>= X) but it is not going to be installed or not installable: inspect all unsatisfied dependencies, pinning, architecture indexes, and the resolver's candidate versions. This can be a transitive dependency such as libkrb5-3:armel needing libssl3:armel.

When Docker is available, reproduce using the same platform, sources, and order as the failed Dockerfile. Add the foreign architecture before refreshing indexes, then inspect both native and target candidates. Replace the sample package with the conflicting runtime library from the log:

docker run --rm --platform linux/amd64 debian:12 bash -c '
  printf "deb http://deb.debian.org/debian bookworm main\n" > /etc/apt/sources.list
  rm -f /etc/apt/sources.list.d/debian.sources
  dpkg --add-architecture armel
  apt-get update
  for pkg in libpcre2-8-0 libssl3; do
    for arch in amd64 armel; do
      echo "--- $pkg:$arch ---"
      apt-cache policy "$pkg:$arch"
    done
  done
'

Use the actual image's source entries and package set in a real reproduction; the minimal source list above is only a diagnostic starting point. apt-cache policy shows installed, candidate, and repository versions. apt-cache madison <package>:<arch> can further compare versions across repositories. In a failing Docker layer, also inspect dpkg-query -W -f='${Package}:${Architecture} ${Version}\n' <package> for the installed version.

As of the investigated Debian 12 issue, go/base/sources-debian12.list permits security indexes for amd64, i386, and armhf, while excluding armel. go/base-arm/sources-debian12.list is scoped to arm64. Check the current files; this policy may change. Do not broadly enable or disable security repositories based only on a package name.

In this incident, a successful x86 build installed libpcre2-dev at 10.42-1 from Bookworm main and later installed librpm-dev:armhf successfully. A later build got past the amd64 RPM install but failed in the armel image because libkrb5-3:armel could not resolve libssl3:armel. Another attempt to pin versions in the shared base regressed go/main, where security libpcre2-dev required the security-updated exact version of libpcre2-8-0. These are distinct failures; compare the exact failed image and step rather than treating them as one repeated error.

For any proposed pin:

• Scope it to the affected cross-architecture image, before dpkg --add-architecture and before package installation; do not put it in a shared base consumed by native images unless those consumers have been verified.
• Confirm the pin causes the intended candidate selection and whether apt-get upgrade needs --allow-downgrades.
• Record the security trade-off: pinning a library to Bookworm main can defer its security update in that image. Add a concrete recheck condition (compare versions across relevant architectures) and treat the pin as temporary, not a permanent Debian policy.
• If package policy is being centralized for multiple Docker build contexts, account for each image's build context; Docker COPY cannot read a file outside the build context without explicitly staging it or changing the context.

4A-4. Missing apt-get update before install

Check for a missing or stale apt index, but distinguish it from a repository/version skew. Look for when sources changed, whether apt lists were removed by a previous layer, and whether the index update ran after adding a foreign architecture. A separate RUN apt install can legitimately use a fresh index retained from an earlier layer; adding apt-get update to every install is not automatically the fix. Reproduce with a clean build or explicit index refresh, then compare candidate versions.

Step 5 — Confirm the cause spans multiple builds

After identifying the root cause, confirm it's consistent by checking whether earlier relevant builds fail with the same error. Use mcp__buildkite__list_builds to list more builds and spot the first relevant failure. Do not use recovered failures (those followed by a successful build on the same branch) as evidence of an ongoing issue.

Step 6 — Summarise and propose a fix

Report:

1. Which builds are failing and since when
2. The exact error (package name, version conflict)
3. Whether it's an apt sources issue, missing apt-get update, or something else
4. The specific file(s) to change and what to change
5. Any risk or trade-off (e.g. security-update coverage, architecture filters, or temporary package pins)

Ask the user before implementing changes.

Output contract

When the investigation is complete, follow the separate issue-reporting instruction file at .apm/instructions/build-debug-issue.instructions.md.

That file defines the required GitHub issue title, body sections, and final format.

Notes

• Keep the scope to this repository only.
• Do not trigger, cancel, or modify any Buildkite builds.
• If there are no unrecovered failures in the lookback window, report the eligible branches as healthy and do not characterize recovered failures as active failures.