Azure DevOps Pipelines — Agent Guide

Scoped guidance for the Azure DevOps (ADO) CI/CD pipelines under .azure-pipelines/. Read this before editing any pipeline YAML. Packaging/build logic invoked by the pipelines lives in devops/scripts/; keep the two in sync. When you add or change a stage, job, pipeline, trigger, variable/group, secret source, or artifact, update this guide in the same PR so it stays accurate.

Essentials

• Pipeline YAML is authored for Azure DevOps, not GitHub Actions (.github/workflows/ is separate).
• Entry pipelines live directly in .azure-pipelines/; reusable templates live in .azure-pipelines/templates/.
• Never inline secrets. Credentials come from variable groups, Azure Key Vault, approved Azure service connections (e.g. ame_esrp_connection, AzureSubscription), and built-in pipeline authentication (e.g. persistCredentials: true on checkout).
• Signing and release-orchestration work runs in the release ADO project; PR (pr-alpha) and beta builds run in public. The stable-release pipeline runs its full build in release.
• Keep changes template-driven: prefer editing a shared template over duplicating steps in an entry pipeline.
• Inline Bash follows the same SIGPIPE rules as devops/scripts/: match captured output with here-strings, drain streaming checks with non-quiet grep redirected to /dev/null, and use sed -n '1p' instead of head for first-line selection from unbounded pipelines. Keep pipefail enabled so producer failures remain visible.

Entry Pipelines

Template Layout (templates/)

• templates/variables.yml — top-level shared variables (pools, project names, repo, service connections, pipeline names).
• templates/dist/variables/ — common.yml, packaging.yml (script/out roots, RIDs), release.yml (ESRP keycodes).
• templates/dist/stages/ — stage-level composition (build-unsigned-distribution.yml, sign-with-esrp.yml, validate-pull-request.yml, validate-signed-distribution.yml, component-governance.yml, init-resolve-version.yml, publish-docker-to-acr.yml, publish-to-github-release.yml, verify-signed-distribution.yml, build-ps-modules.yml). The PR validation stage checks title/history conventions, blocks manual root changelog or project-version edits, and restricts release-bot stable PRs to generated metadata paths and version-only project changes.
• templates/dist/jobs/ — per-platform build/validate/sign jobs (linux, windows, macos, ps-modules, nuget), signed PowerShell installer validation, plus test-suite.yml (parallel Linux/Windows/macOS Unit Tests and Ubuntu Scenario Tests jobs), benchmark-suite.yml, finalize-build-summary.yml.
• templates/dist/steps/ — small reusable steps (get-github-pat-token.yml, validate-release-project.yml, generate-cg-notice.yml).
• Security validations (modeled on azure-cli and azure-powershell): component-governance.yml runs Component Detection, Credential Scanner (CredScan@3, suppressions in devops/scripts/compliance/CredScanSuppressions.json) and PoliCheck over the repository sources, covering CLI and PowerShell; PostAnalysis@2 breaks the build on CredScan/PoliCheck errors. The unsigned branch of validate-build-windows.yml runs BinSkim on the downloaded .dll/.exe files (excluding msalruntime*.dll and the third-party prebuilt Copilot runtime binaries under Microsoft.Azure.Azclips.Extensions.Agent/deps/runtimes/**/native/**, which fail BA2008/CFG and are not built here) for both win-x64 and win-arm64 lean/full matrix entries, publishes CodeAnalysisLogs_<artifact>, and breaks on BinSkim errors. Signed lifecycle validation is unchanged. Both ARM64 matrix scans executed in build 363065, so the duplicate x64-hosted ARM64 scan has been removed; finalization still requires WindowsValidation to succeed.
• PowerShell binary security: build-ps-modules.yml composes the Ubuntu BuildPSModules stage and the dependent Windows ValidatePSModules stage. The latter runs PSBinaryAnalysis from validate-ps-security.yml. It checks out the repository for the health-check script and policy metadata, downloads the configured PowerShell artifact (PSModules in the shared distribution pipeline), and scans .dll/.exe files recursively under artifacts/PSGallery/modules, the shipping module folders passed to Publish-Module by package_az_modules.ps1. This includes packaged cmdlet assemblies, core/extension assemblies, and bundled dependencies without an architecture filter, rather than unrelated solution outputs such as artifacts/Core.Agent/runtimes/. It excludes only msalruntime.dll, msalruntime_arm64.dll, and msalruntime_x86.dll, matching classic PowerShell; no Copilot exclusion is applied within the shipping payload. It requires the shipping directory and cmdlet assemblies to be present, publishes CodeAnalysisLogs_PSModules, and breaks on BinSkim errors. Finalization waits for ValidatePSModules and requires both the module build and binary analysis to succeed; no duplicate external CredScan invocation is needed for the existing repository-wide source coverage. Unsigned Windows validation resolves the single unsigned azcli[-lean]-*-$(Rid).zip from the downloaded artifact, extracts it under $(Agent.TempDirectory)/binskim-cli/, requires az.exe in the extracted root, and scans that extracted content, so the shipped zip is what is analyzed. PowerShell requires its shipping directory and cmdlet assemblies. adding exact filenames from devops/scripts/compliance/binskim-runtime-exclusions.json directly under the extracted archive root.
• templates/dist/steps/ — small reusable steps (get-github-pat-token.yml, validate-release-project.yml).

Component Governance and NOTICE

• All shared compliance stages restore the solution, submit Component Governance inventory, and run the official OSPO notice@0 task through generate-cg-notice.yml. Restore, detection, generation, or output-validation failures fail the stage; there is no custom-generator fallback.
• Compliance_Results_<stage_name> retains <stage_name>/NOTICE.generated alongside the existing security results. Stage-specific output directories prevent the OSPO task's own artifact uploads from colliding between unsigned and signed compliance stages.
• Generated NOTICE differences are reported for review, not copied into build outputs. Alpha, beta, and stable packaging all continue using the root NOTICE from their checked-out commit; stable packages use the reviewed file from the tagged commit.
• The manual stable-release trigger runs the same generation steps after preparing version and changelog updates, refreshes root NOTICE only when normalized attribution content changes, and includes that change in the release PR. It publishes the raw output as Compliance_Results_StableReleaseNotice once PrepareNoticeResults.Ready confirms the output directory exists; earlier failures skip evidence publication instead of reporting a missing artifact directory. The post-merge tag stage does not regenerate NOTICE.
• Both the generated-PR file policy and the post-merge tagging allowlist permit root NOTICE. Tagging still requires a version-only CLI project change and rejects unrelated files. devops/scripts/ci/test_stable_release_tag.py exercises the tagging Bash from this pipeline against local Git repositories, without contacting GitHub.
• Missing ClearlyDefined entries remain OSPO task warnings, not generation failures. NOTICE selections are pipeline/job-specific and must be reviewed for each pipeline, including the stable-release trigger. The OSPO extension must be installed in the organization.

Fork PR Security Coverage

• Guardian does not support forked GitHub PR builds, but the classic Azure CLI and Azure PowerShell pipelines apply no System.PullRequest.IsFork condition to their CredScan, BinSkim, log-publication, or PostAnalysis steps, so azclips does not either. The CredScan, BinSkim, BinSkim health-check, log-publication, and PostAnalysis steps run on every build and keep only their original succeeded()/succeededOrFailed() conditions. Any additional azclips-specific health check must report problems as warnings (##vso[task.logissue type=warning]) rather than fail the build.
• Keep the enclosing jobs enabled: Windows artifact/lifecycle validation, PowerShell shipping-payload checks and module build/tests, NOTICE generation, and Component Detection still run. Do not relax finalization to accept skipped or failed jobs. No fork-specific warning steps are emitted.
• PostAnalysis still breaks on BinSkim and CredScan Error findings. The report-presence check (verify_binskim_results.ps1) is warning-only and never fails the build. Do not use GDN_SUPPRESS_FORKED_BUILD_WARNING to hide missing coverage.
• The alpha pipeline does not sign or publish a release. Beta signing is separately gated by its manual/branch checks and pipeline resource; stable signing/release must consume trusted sources and successful validation. A successful fork build is not evidence of Guardian approval. Fork permissions, contributor approval, isolated agents, feed/service-connection access, and secret availability must be enforced through Azure DevOps settings; PR-editable YAML conditions are not a security boundary. Existing persisted checkout credentials in Init and PR metadata validation require least-privilege review. No additional secrets or permissions are enabled by this handling.

Build Graph (unsigned distribution)

build-unsigned-distribution.yml fans out from an Init stage that resolves the version:

• ComponentGovernance and BuildPSModules depend on Init.
• BuildPSModules retains its existing Ubuntu build-time module checks. Its common build job initializes PsValidationInstalled to false and sets it only after successful installation; Uninstall uses always() with that marker so cancellation or later failures cannot skip owned-module cleanup. The separate ValidatePSModules stage depends on it and runs Windows PSBinaryAnalysis; finalization waits for both stages and enforces their results before artifact collection and downstream signing/release.
• BuildCommon depends on Init.
• BuildLinux, BuildWindows, BuildMacOS, Test, and Benchmark depend on Init + BuildCommon and can run in parallel.
• ValidateWindows depends on Init + BuildWindows, and runs WindowsValidation including unsigned x64 and native ARM64 BinSkim and SARIF health checks. Build jobs only produce artifacts; finalization and downstream signing/release remain gated by validation results.
• BuildLinux produces Lean and Full archives for glibc (linux-x64, linux-arm64) and musl (linux-musl-x64, linux-musl-arm64). Each archive receives one full installer, layout, and runtime validation in a compatible environment: glibc on Ubuntu and musl in Alpine. Arm64 runtime checks require a matching Arm64 agent.
• Each platform validation stage depends on Init + its corresponding platform build.
• The Test stage runs UnitTestsLinux on ubuntu_pool, UnitTestsWindows on the existing x64 windows_pool, UnitTestsMacOS on the Microsoft-hosted macos_arm64_pool image, and ScenarioTests on Ubuntu in parallel. All three unit-test jobs restore and build the complete solution in Release before running tests with --no-build, excluding Scenario.Tests, ensuring exporter fixture tests see a complete, current set of extension artifacts. All three unit-test jobs publish TRX results, including after test failures. Windows runs the isolated native-console regression in Cli.Tests; that test is skipped on non-Windows platforms. Scenario tests and pipeline input safety tests remain Linux-only. Finalization requires the entire Test stage to succeed, without listing its internal jobs.
• build-unsigned-distribution.yml retains the established combined Docker behavior behind build_docker_images (default true) and independently selects unsigned CLI and PowerShell images through build_cli_docker_images and build_powershell_docker_images (both default true). Alpha and beta therefore continue producing both image families from unsigned artifacts.
• Docker build, publish, and verification stages depend directly on Init and consume SetCoreVersion.ArtifactVersion; they do not independently derive a Docker version.
• ACR publish and the AMD64 published-image pull job follow classic release-agent convention and expect az on the AMD64 agent. The AMD64 job pulls every published digest and exports per-image pipeline artifacts; native verification jobs load those artifacts and run smoke tests, so ARM64 agents require Docker but not Azure CLI or ACR credentials.
• publish-docker-to-acr.yml builds PowerShell images from signed artifacts, then ValidateSignedPowerShellDockerImages loads each downloaded AMD64/ARM64 archive on a native agent and smoke-tests it (expected Az version from powershell-version.txt), mirroring ValidateDockerAzureLinux3 for CLI images. PublishDockerImagesToAcr depends on it. Docker build jobs (signed and unsigned) only build and publish the image artifact; smoke testing happens once per image in the dedicated validation stages, not inside the build jobs.
• Docker publication follows classic retry behavior in both test and production: existing immutable version tags are retained and verified downstream rather than rebuilt under candidate tags.
• Both azclips-stable-release.yml and azclips-stable-release-test.yml set the target ACR repositories through their Docker*RegistryName, Docker*RegistryServer, and Docker*Repository variables, which are passed to publish-docker-to-acr.yml; nothing in the template or scripts hardcodes them. Repositories must be onboarded with MAR before publishing, and must start with public/ or unlisted/ so MCR serves them anonymously (the MCR path drops that prefix). Unlisted images allow anonymous pulls and public tag listing but are excluded from the public catalog and discovery portal. After ACR publish, a server-side 10-minute delay (WaitForMcrSync, matching the classic Wait for MCR Sync phase) runs in parallel with ACR verification; then VerifyMcrDockerImages checks, with one retry after a 2-minute gap, that each versioned manifest tag exists on mcr.microsoft.com/<repo without the public/ or unlisted/ prefix>, pulls it anonymously on native AMD64/ARM64 agents, and runs the smoke tests, which check the version (no digest comparison, as in the classic release). These stages always run in the stable and stable-test pipelines; alpha and beta builds never call this template, so they skip signed PowerShell images, ACR publish and MCR verification. Only immutable version tags are published; aliases are not promoted.
• CLI tags follow the classic azurelinux3.0 convention; full architecture tags omit a full qualifier, and VERSION-azurelinux3.0 combines them as the default full multi-architecture image. PowerShell tags follow classic azurelinux-3.0 spelling.
• PowerShell validation runs as a single ValidatePS stage (depends on BuildPSModules). Its matrix configuration comes from two build-unsigned-distribution.yml booleans, both defaulting to false, which each entry pipeline sets explicitly: limit_ps_testing (only testInAlpha entries, no discovery) and skip_ps_version_check (no distribution/PowerShell version discovery). Alpha sets both true; beta sets both false (full matrix plus discovery); stable and stable-release-test set only skip_ps_version_check: true. The stage does not read the resolved release channel. Finalization checks dependencies.ValidatePS.result in its stage-level condition, covering matrix generation and every validation job; SucceededWithIssues remains acceptable.
• PowerShell validation matrix phases are Bootstrap, Install and Uninstall. Install requires PsValidationBootstrapped and validates that every packaged module imports at its exact packaged version, exports its manifest cmdlets and has help examples; Uninstall requires PsValidationInstalled. Cancellation still skips new validation. Reports/publication gates retain earlier failures.
• Validation bootstraps use SHA256-verified standalone PowerShell GitHub release archives with their bundled runtime, not separate .NET installers or NuGet tools. Matrix runtime channels remain compatibility/report metadata; phases invoke the isolated executable directly without DotnetRootPath or runtime PATH reconstruction. The preparation job's pipeline fixtures include offline archive-bootstrap coverage using bash and jq.
• The PowerShell matrix-generation job reads ps_validation_matrix.json directly with jq, without Python/YAML dependency setup. It retains actual generation, but not the separate matrix self-test step (that fixture remains available locally). Native Windows supports configured numeric versions and latest, which selects windows-latest; this alias does not change PowerShell support/failure policy.
• PowerShell preparation jobs route pool-* values through pool.name, while hosted image labels use pool.vmImage. Generated matrix legs use DistributionPoolName and DistributionVmImage: named pools receive an empty image, and hosted images select the Azure Pipelines pool. Do not pass named pool identifiers as image labels.
• Finalize gates assembly once through stage-level dependencies.<stage>.result checks. Required stages must succeed; only ValidatePS and ComponentGovernance may succeed with issues. Missing, skipped, failed or canceled required stages skip all of Finalize, rather than running a green summary with skipped publication. Docker build/validation stages are required only when Docker artifacts are included in the final distribution. The assembly job has no internal job-result lookups or repeated result conditions; only optional Docker downloads keep success-aware flavor conditions. There is no always-running textual summary. Adding jobs or tasks within an existing prerequisite stage needs no finalization changes. Regression coverage in tests/Scripts/PS/PSValidationPipelineFixtures.ps1 exercises every dependency result and Docker flag combination.

ArtifactVersion is produced by Init (SetCoreVersion.ArtifactVersion) and consumed by downstream stages via stageDependencies.

Projects & Context Guards

• public — PR (pr-alpha) and beta builds. No ESRP access.
• release — beta signing, stable release trigger, and stable release. Has ESRP + GitHub PAT access.
• templates/dist/steps/validate-release-project.yml enforces that signing/release pipelines only run in release. Preserve these guards when editing.

Variables, Groups & Secrets

• Queue parameters and upstream build metadata in release preparation, beta tagging, beta signing preflight, and signed Linux musl validation are mapped through task env and consumed as quoted Bash data, never interpolated into script source. Preserve this boundary when adding inputs or log messages.
• The agent treats ##vso[ anywhere in an output line as a logging command. Print these untrusted values through the step's log_value helper, which rewrites the marker to ##vso_[; intentional logging commands remain literal script output.
• Stable branch preparation does not receive GH_TOKEN; the credential check runs in the authenticated commit/PR step. Changelog updates and tag/push operations retain their existing explicit credential mappings.
• The shared UnitTestsLinux job runs credential-free pipeline input regression tests before .NET tests: python3 -m unittest discover -s devops/scripts/ci -p test_pipeline_input_safety.py -v. These exercise the actual YAML Bash and environment mappings without running pipelines or accessing remote resources.
• Shared pool, project, repository, service-connection, bundle, and pipeline-name values are defined in templates/variables.yml.
• Variable groups (attached only when System.TeamProject == 'release'): AME ESRP Variable Group, esrp-signing-codes, azclips-telemetry, get-github-pat-token.
• azclips-telemetry supplies the secret AriaConnectionString used only during stable and stable-test compilation. Shared build jobs map it to AZCLIPS_ARIA_CONNECTION_STRING; Core.Telemetry.csproj embeds it as assembly metadata when supplied. PR, beta, and local builds fall back to the committed development connection string in WellKnownEndpoints. The stable preflight fails before compilation when the variable is missing.
• ESRP keycodes (ESRPNuGetKeyCode, ESRPAuthenticodeKeyCode, ESRPThirdPartyKeyCode, ESRPStrongNameKeyCode, ESRPAppleKeyCode) come from groups and are mapped in release.yml. Never hardcode them.
• The GitHub PAT is fetched at runtime from Key Vault via steps/get-github-pat-token.yml; do not persist it.
• Some jobs also rely on service connections (ame_esrp_connection for ESRP, AzureSubscription for Key Vault) and persisted checkout credentials (persistCredentials: true, for checkout/fetch). Release Git tag/push and gh operations authenticate with Key Vault GH_TOKEN. Do not introduce release-only secrets into public-project jobs.

Packaging Roots (packaging.yml)

• Native installer lifecycle jobs exercise release metadata/download and piped-invocation paths using local HTTP-command fixtures backed by the selected pipeline archive. Linux/macOS share packaging/common/validate_cli_download.sh, which restricts the installer environment to its declared system utilities. Windows uses scoped HTTP fixtures, and signed PowerShell validation runs the module bootstrap and bundled helper in isolated processes.
• PackagingScriptsRoot = devops/scripts/packaging
• Extension download-only helper checks run inside the same native CLI lifecycle jobs: Bash through the shared CLI fixture, and PowerShell through common/test_extensions_download.ps1. Both build fixture bundles from the downloaded pipeline extension packages and check pinned releases, failure preservation, and cleanup without live release downloads.
• PackagingOutRoot = devops/out (with common/, linux/, windows/, macos/ subfolders)
• Default RIDs: linux-x64, win-x64, osx-arm64.
• Pipeline jobs shell out to devops/scripts/ (ci/, compliance/, packaging/, publishing/). When changing a build step, update the script there rather than embedding logic in YAML.

Artifacts

• Final_Build_Distribution — unsigned build output.
• Final_Signed_Distribution — stable ESRP-signed output.
• Final_Signed_Beta_Distribution — beta ESRP-signed output. azclips-beta-sign sets this fixed name through SignedArtifactName; it has no queue-time parameters.
• PSModules — packaged PowerShell modules.
• Validate_PS_Sim_<ValidationKey> — schema-3 phase results, logs and lifecycle diagnostics for each matrix leg; Bootstrap, Install and Uninstall are all required for success.

Signing & Release Flow

• NuGet, Windows and PowerShell signing tasks combine payload-presence flags with succeeded(), including restore, repack and ESRP verification steps. Optional payloads remain skippable, but no flag-controlled signing operation runs after a failed or canceled job. tests/Scripts/PS/PSValidationPipelineFixtures.ps1 exercises these conditions for successful, warning, failed, canceled and skipped statuses.
• Beta (on demand, unchanged): Manually queue azclips-beta-build (public) with sign_beta: true → add sign-beta tag → azclips-beta-sign (release) consumes that exact completed tagged run via a pipeline resource → ESRP sign → verify → run all 16 Linux, Windows, and macOS lifecycle rows plus the signed PowerShell installer row against the exact signed archives.
• Beta (daily): azclips-beta-sign (release) also runs daily at 00:00 UTC on main, even without source changes, and resolves its betaBuild pipeline resource to the latest successfully completed azclips-beta-build (public) run on refs/heads/main, without requiring a sign-beta tag. Manual signing runs use the same default unless a resource version is explicitly selected. All paths pin downloads to the resolved resource run ID and use the same signing, verification, and lifecycle validation stages.
• Stable: azclips-trigger-stable-release (release, manual) bumps the CLI and top-level Az versions, detects extension code changes since the latest stable tag, independently patch-bumps each changed existing extension, preserves new extensions' declared initial versions, requires their PowerShell child changelogs, aligns their child manifests and Az dependencies, refreshes root NOTICE from official CG generation when changed, and opens a release PR via gh. After that PR merges to main, a CI run of the same pipeline (CreateStableTag, condition IndividualCI/BatchedCI on refs/heads/main) creates the tag vX.Y.Z.yyyyMMddTHHmmssZ via git tag/git push → the tag triggers azclips-stable-release → build → sign-with-esrp → component governance → verify → signed lifecycle validation → publish-to-github-release (and, in parallel, publish-docker-to-acr; the same stages run in azclips-stable-release-test with its own repository variables)..
• validate-signed-distribution.yml downloads the flat final signed artifact in each native matrix job, requires exactly one matching archive per RID/profile, stages its signed extension packages for the existing lifecycle harnesses, and passes the archive path explicitly. A separate Ubuntu job executes the bundled installer from the exact signed azps-cmdlets archive, imports a child module at its packaged version, and verifies numeric version inference. Validation never rebuilds, modifies, repacks, or resigns a publication candidate.
• macOS notarization uses macos_bundle_id: com.microsoft.azclips.cli. Outermost .app, .framework, .xpc, .appex, .plugin, and .bundle directories with Apple bundle metadata are sent to ESRP as complete signing units rather than signing their nested Mach-O files separately; unrecognized Mach-O-bearing code containers fail closed. Signing ZIPs are created and extracted on macOS with symlink preservation and remain opaque on Windows. Reconstructed payloads must pass strict codesign, Microsoft Team ID, hardened-runtime, timestamp, and required .NET entitlement checks before notarization ZIPs are created.

Editing Rules for Agents

• Change shared behavior in templates/dist/**, not by copy-pasting into entry pipelines.
• Keep project guards, branch/tag validation, and secret-source patterns intact.
• Match existing parameter naming (ubuntu_pool, release_channel, source_artifact_name, etc.) and pass pools through parameters.
• Keep checkout tasks on the Azure Pipelines default of no automatic task retries.
• Validate YAML changes: check indentation, template parameters/variables wiring, and dependsOn/stageDependencies references before committing.
• Do not add new triggers or pr: scopes without confirming the intended project and channel.