GitLab job triage

The answer is the first real error line with its cause, or for a plan the list of what it will destroy or replace. Never the last 20 lines of the log, never a retry.

1. Resolve the URL

glab api wants the path %2F-encoded, -R wants it plain. Logs and JSON go to files, never through echo (zsh mangles \n inside JSON).

S=$(mktemp -d)                      # or the session scratchpad
P="group%2Fsubgroup%2Fproject"      # project path, %2F-encoded
JOB=123456789
glab api "projects/$P/jobs/$JOB" > "$S/job.json"
jq -r '[.name, .stage, .status, .failure_reason, .ref, .pipeline.id, .commit.short_id, .web_url] | @tsv' "$S/job.json"
glab api "projects/$P/jobs/$JOB/trace" > "$S/job.log"

Pipeline URL: glab api "projects/$P/pipelines/<id>/jobs?per_page=100&include_retried=false", then triage each failed job, earliest stage first (a later failure is often the consequence).

2. Find the first real error

command rg -n -m 20 -i 'error|fatal|denied|forbidden|not found|timed out|no space|OOMKilled|exit code [1-9]' "$S/job.log"

Quote the error line with its unit prefix (Terragrunt run-all prefixes every line with the unit) and the file/line it names, then read that file in the local clone at the job's commit: git show <short_id>:<path>.

3. Read a plan critically (plan jobs)

command rg -n 'Plan: [0-9]+ to add|No changes\.|must be replaced|will be destroyed|forces replacement' "$S/job.log"

Report per unit:

Then flag, in this order:

1. Any destroy or replacement the MR description does not announce. A replaced cluster, node pool, database instance, bucket or DNS zone is a stop: name it first.
2. Units in the plan that the MR did not touch (dependency fan-out, or drift: the default-branch plan shows the same change).
3. In-place changes on fields the diff does not explain (provider upgrade, default changed upstream).

Drift check: the same unit planned on the default branch, glab ci trace <plan-job> -R <path> -b <default-branch> (or the latest scheduled pipeline). The same diff there means drift, not the MR.

4. Answer

Two or three lines: job, first error with its unit, cause, what to change (file:line). For a plan: the per-unit table with every destroy/replace named, then the flags, never a verdict from the summary line. Fix only when asked.