agentleFS
carverautocarverauto/serviceradar/.agents/skills/demo-local-rollout/SKILL.md

demo-local-rollout

Skill934 starsChanged 2 days agoApache-2.0View on GitHub

Roll unpublished sha-... test images to farm01 or the carverauto demo cluster. Use when the user asks to deploy, refresh, roll, patch, or test code in demo, farm01, or "the farm cluster" before a release. Pick the cluster first. Carverauto/demo images are built in-cluster by publish-oci.yml on a test/demo-test-NN branch, verified and signed with the OpenBao release key, then pinned by a PR into demo/prod-release. farm01 uses the same verified publication and a Helm digest-pin roll. Do not use for release cuts or Docker Compose.

  • Commits and pushes

The file

Show its rules and text

We split this skill into the rules it gives an agent

15 rules. About 0 are common, 8 are distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use this skill for unpublished sha-<git-sha> test tags.

    line 8
  2. Distinctive:

    Do this before Harbor auth, Bazel, signing, or any cluster write.

    line 12
  3. Distinctive:

    Keep publication in the verified workflow for both clusters.

    line 25
  4. Very specific:

    If only elixir/web-ng/ changed and the target is demo, consider $demo-web-ng-fastpath.

    line 27
  5. Very specific:

    Do not run bazel build --config=remote_base ...

    line 31
  6. Distinctive:

    Read the currently deployed tag on the chosen cluster before changing anything.

    line 42
  7. Very specific:

    Do not cut a release, edit VERSION, or create git tags.

    line 43
  8. Very specific:

    Never push to staging or demo/prod-release.

    line 45
  9. Distinctive:

    Never print, log, or paste a service-account JWT or a Vault token.

    line 49
  10. Distinctive:

    Keep each unchanged service at its existing digest.

    line 96
  11. Distinctive:

    Do not patch the Argo Application: its source file owns Helm parameters.

    line 112
  12. Very specific:

    Do not call demo finished until new pods are Running on the verified protected digests and Argo reports Synced|Healthy|Succeeded for that …

    line 132
  13. Very specific:

    After testing, $release-cut-and-demo-roll returns demo to the semver path.

    line 136
  14. Distinctive:

    Keep unchanged digest pins.

    line 151
  15. Distinctive:

    Verify rendered workload references and actual running image IDs, then report the publish run, protected digests, Helm revision, and any …

    line 155

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test Image Rollout

Use this skill for unpublished sha-<git-sha> test tags. Formal releases use semver and $release-cut-and-demo-roll.

Choose the cluster first

Do this before Harbor auth, Bazel, signing, or any cluster write.

carverauto demofarm01
WhenUser says demo, carverauto, or does not name a clusterUser says farm01, farm cluster, or is already on the farm01 kubeconfig
Kubeconfigkubectl --context carverautoexport KUBECONFIG="$HOME/.kube/farm01.yaml"
Namespacedemoserviceradar
Image buildIn-cluster: publish-oci.yml on a test/demo-test-NN branchPrefer the same in-cluster build; use the same verified publication boundary
How it rollsPR into demo/prod-release, then Argo auto-sync of serviceradar-demo-prodhelm upgrade --reuse-values
AdmissionKyverno: images must be signed with OpenBao cosign-releasePublication still signs and verifies before promotion; no manual signer access
RegistryHarbor registry.carverauto.dev/serviceradarSame Harbor project (public; no pull secret)
Live tag sourcehelm/serviceradar/.argocd-source-serviceradar-demo-prod.yaml on origin/demo/prod-release, plus the Deployments in demohelm get values serviceradar -n serviceradar (global.imageTag)

Keep publication in the verified workflow for both clusters; use the correct deployment path. Never helm-upgrade farm01 thinking it is demo.

If only elixir/web-ng/** changed and the target is demo, consider $demo-web-ng-fastpath. That fast path is demo-only.

Do not build demo images from a workstation

Do not run bazel build --config=remote_base ... --remote_download_outputs=all (or make push_all, or any other full image-set build) on a workstation to produce demo images. That build pulls every large CAS blob through cache-proxy.carverauto.dev, which is served by the shared Envoy gateway. During one such build the gateway was OOMKilled six times and went down for everything behind it. The in-cluster publish-oci.yml run below builds and pushes the same images without pulling them to a workstation.

Rules for every roll

  • Tag: sha-<full 40-char commit SHA> (what --stamp emits).
  • Read the currently deployed tag on the chosen cluster before changing anything.
  • Do not cut a release, edit VERSION, or create git tags. Do not roll demo-staging, production, or Docker Compose unless the user changes scope.
  • Never push to staging or demo/prod-release. Push only new branches, always with an explicit refspec: git push origin <local>:refs/heads/<branch>.
  • origin is GitHub (git@github.com:carverauto/serviceradar.git). There is no github remote.
  • Never print, log, or paste a service-account JWT or a Vault token. Keep them in files under a private temp directory and delete them when signing is done.

carverauto demo roll (in-cluster build, sign, pin PR)

1. Push a test branch at the commit

COMMIT="$(git rev-parse <commit-under-test>)"
git fetch origin
git ls-remote origin 'refs/heads/test/demo-test-*'   # pick the next unused NN
NN=<next number>
git push origin "${COMMIT}:refs/heads/test/demo-test-${NN}"

The commit must already contain the publish-oci.yml change that allows test/demo-test-<N> in the signing step (OPENBAO_SIGNING_ALLOWED_REFS_REGEX in "Publish images"). The workflow runs from the dispatched branch's own copy of the file, so a branch cut from an older commit still refuses to sign and ends red right after the pushes, with OpenBao signing is not allowed for ref 'refs/heads/test/demo-test-<N>'.

2. Dispatch the full publish and wait

gh workflow run publish-oci.yml --ref "test/demo-test-${NN}" -f publish_scope=full
gh run list --workflow publish-oci.yml --branch "test/demo-test-${NN}" --limit 1
gh run watch <run-id> --exit-status

The run builds on RBE from the in-cluster serviceradar-signing ARC runner, pushes every publishable image into the disposable build project, signs each built digest with hashivault://cosign-release, and verifies it before promotion into the protected project. If it fails, read the failing step with gh run view <run-id> --log-failed before retrying. A run that fails during signing or promotion has not produced a deployable receipt. Repair the failed stage and rerun the verified workflow before rolling.

3. Require verified promotion

The workflow builds in the private serviceradar-builds project, signs and verifies there, checks registry capacity, and promotes the complete artifact graph into serviceradar. Require a successful run and retain its promotion receipt. Use the receipt's published roots to select protected repository@sha256:... references. A build upload alone is not deployable. Never repair a failed promotion with a direct push, tag copy, or manual protected-project signature append.

Keep each unchanged service at its existing digest. Do not copy unchanged tags forward to create a common tag across a mixed deployment.

4. Pin verified digests with a PR into demo/prod-release

Inspect helm/serviceradar/.argocd-source-serviceradar-demo-prod.yaml on origin/demo/prod-release and the currently rendered images. Create a feature branch from that deployment branch. Set image.digests.<service> to each changed service's verified protected digest (webNg and core for the web stack). Preserve every unchanged service's digest and existing chart version. If an unchanged service still uses a tag, resolve its current protected digest before making the deployment bundle explicit.

Push only the feature branch with an explicit refspec and open a GitHub PR into demo/prod-release. Include the publish run, reviewed commit, and protected digest receipt. Do not remove digest pins or substitute a global mutable tag. Do not patch the Argo Application: its source file owns Helm parameters.

5. Let Argo roll it

serviceradar-demo-prod has automated sync on (prune and self-heal off), so after the PR merges Argo picks up the new demo/prod-release revision by itself. Do not sync manually and do not patch the Application. Confirm:

kubectl --context carverauto get application -n argocd serviceradar-demo-prod \
  -o jsonpath='{.status.sync.revision}{"|"}{.status.sync.status}{"|"}{.status.health.status}{"|"}{.status.operationState.phase}{"|"}{.status.operationState.operation.initiatedBy.automated}{"\n"}'
kubectl --context carverauto get deploy -n demo \
  serviceradar-web-ng serviceradar-core serviceradar-agent serviceradar-agent-gateway \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.template.spec.containers[*]}{.image}{" "}{end}{"\n"}{end}'

The sync revision must be the merge commit of the pin PR. Kyverno sets mutateDigest: true, so admitted pods show :<tag>@sha256:...; compare that digest with the one you verified.

Do not call demo finished until new pods are Running on the verified protected digests and Argo reports Synced|Healthy|Succeeded for that revision. Report the tag, test branch, publish run id, the protected digests in the workflow promotion receipt, the pin PR, and the Argo status.

After testing, $release-cut-and-demo-roll returns demo to the semver path. Do not push a v* release tag during the test window.


farm01 roll

Publish with the same in-cluster publish-oci.yml workflow above and require its verified promotion receipt. Raw Bazel push targets only upload disposable build artifacts; retired workstation launchers refuse publication.

Inspect helm get values serviceradar -n serviceradar and the current chart version. Keep the installed chart when the task changes only images. Use helm upgrade --reuse-values with --set image.digests.<service>=sha256:<digest> for each changed service, using only digests from the protected promotion receipt. Keep unchanged digest pins. Preserve cluster-specific values and wait for rollout completion. Record the same digest changes in the owning GitOps values file when that repository update is within the authorized task.

Verify rendered workload references and actual running image IDs, then report the publish run, protected digests, Helm revision, and any workload that is not Ready.

Operational notes (any namespace)

kubectl set image poisons later Helm upgrades

A hand-run kubectl set image takes server-side-apply ownership of .spec.template.spec.containers[].image under the kubectl-set field manager, and every later helm upgrade then fails with:

Apply failed with 1 conflict: conflict with "kubectl-set" using apps/v1

Resetting metadata.managedFields to [{}] does NOT fix it on its own -- the fields are re-attributed to a synthetic before-first-apply manager and the conflict count goes UP. The fix is Helm 4's --force-conflicts, which takes ownership in place (unlike --force-replace, which recreates the resource):

helm upgrade serviceradar ./helm/serviceradar -n <ns> --reuse-values --force-conflicts \
  --set image.digests.core=sha256:...

Forcing scheduled work instead of waiting

Oban-scheduled maintenance can trickle. Drive it directly over the release RPC rather than waiting for the next tick:

kubectl exec -n <ns> <core-pod> -- /app/bin/serviceradar_core_elx rpc \
  'ServiceRadar.Inventory.Identity.DuplicateSweep.reconcile_duplicates() |> inspect() |> IO.puts()'
kubectl exec -n <ns> <core-pod> -- /app/bin/serviceradar_core_elx rpc \
  'ServiceRadar.Observability.NetflowExporterCacheRefreshWorker.perform(%Oban.Job{args: %{}}) |> inspect() |> IO.puts()'

Measured difference: the scheduled duplicate sweep was merging ~1 device per run; the direct call merged all 11 outstanding in 1.5s.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.