SKILL: Azure Kusto Spark Connector — Release Process

Identity

You are a release automation agent for the Azure Kusto Spark Connector. You execute the complete release lifecycle: cherry-picking changes between branches, bumping versions, updating the changelog, creating tags, and triggering the release pipeline. You operate by running git and shell commands in the repository.

How to Invoke This Skill

This file is not auto-loaded by AI agents. You must explicitly reference it when prompting. The file must be present in your working directory (i.e., merged to the branch you're working from).

Copilot CLI (Terminal)

Use the @ prefix to include this file in your prompt:

Development Workflow:

@SKILL.md I have a branch with changes ready. Take over and get it merged to both branches.

@SKILL.md My PR #485 was just merged to master. Cherry-pick to Spark 3.

Release Workflow:

@SKILL.md All features are merged. Do a release.

@SKILL.md Bump the version and release. Both branches already have the latest changes merged.

@SKILL.md Just tag and release version 7.0.6 on both branches.

VS Code — GitHub Copilot Chat

Use #file:SKILL.md to reference this file, or @workspace to let Copilot find it:

Development Workflow:

#file:SKILL.md My PR was just merged to master (commit abc1234). Cherry-pick to release/spark3.

@workspace Follow SKILL.md. I have changes on branch asaharn/feat/my-feature ready to push.

Release Workflow:

#file:SKILL.md Bump version to 7.0.6 and update the changelog on both branches.

JetBrains — AI Assistant

Reference the file in AI Assistant chat:

Look at SKILL.md in the project root. My PR #485 was merged to master. Cherry-pick it to release/spark3.

Follow the instructions in SKILL.md to do a release — bump version, update changelog, and tag both branches.

GitHub.com — Copilot in PR / Issues

When using Copilot on GitHub.com (e.g., in a PR comment or issue), reference the file path:

Follow the development workflow in SKILL.md at the repo root. This PR is now merged — describe the next steps to cherry-pick to release/spark3.

Note: GitHub.com Copilot cannot execute git commands directly. It will provide the commands for you to run, or guide you through the GitHub UI steps.

Prompt Tips

• Be specific about where you are: "My PR is merged", "I have a branch ready", "Both branches are updated"
• Provide commit SHAs or PR numbers when starting from Phase 2 (cherry-pick)
• Specify a version if you don't want auto-detection (e.g., "release version 7.0.6")
• The agent will ask for missing info — you don't need to provide everything upfront

How to Use This Skill

This skill has two independent workflows that can be invoked separately:

1. Development Workflow (Phases 1–2) — Repeat for each feature/fix. Gets changes onto both branches.
2. Release Workflow (Phases 3–5) — Run once when ready to release. Bundles all changes since the last tag.

You can invoke either workflow at any point. Tell the agent where you are and it will pick up from there.

Example Prompts — Development Workflow

"I have a branch with changes ready — take over from here"

The agent will detect your current branch, push it if needed, help create a PR to master, and then cherry-pick to release/spark3.

"My PR was just merged to master. Cherry-pick it to Spark 3"

Provide the merged PR number or the squash merge commit SHA. The agent will create a branch from release/spark3, cherry-pick, push, and create a PR.

Example Prompts — Release Workflow

"All features are merged. Do a release"

The agent will run Phases 3–5: bump the POM version on both branches, update the changelog with all changes since the last tag, create tags, and trigger the release pipeline.

"Bump the version and update changelog for release"

The agent will start at Phase 3 — bump the POM version, update the changelog, then ask for confirmation before tagging.

"Just tag and release version X.Y.Z"

The agent will start at Phase 5 — verify the version in pom.xml, create tags on both branches, and trigger the release.

What the Agent May Ask You

If information is missing, the agent should ask for:

Repository Context

Prerequisites

Before starting any phase, verify:

# 1. Working tree is clean
git status --porcelain
# Expected: empty output

# 2. Remote is up to date
git fetch origin

# 3. You are on the correct branch
git branch --show-current

State Detection

When the user asks you to take over, run these commands to determine the current state and decide which phase to start from.

Step 1 — Detect Current Branch and Changes

# What branch are we on?
CURRENT_BRANCH=$(git branch --show-current)
echo "Current branch: $CURRENT_BRANCH"

# Are there uncommitted changes?
DIRTY=$(git status --porcelain)
echo "Uncommitted changes: ${DIRTY:-none}"

# What is the base branch? (master or release/spark3)
BASE_BRANCH=""
if git merge-base --is-ancestor origin/master HEAD 2>/dev/null; then
  BASE_BRANCH="master"
elif git merge-base --is-ancestor origin/release/spark3 HEAD 2>/dev/null; then
  BASE_BRANCH="release/spark3"
fi
echo "Base branch: ${BASE_BRANCH:-unknown}"

# Are there unpushed commits?
UNPUSHED=$(git --no-pager log origin/${CURRENT_BRANCH}..HEAD --oneline 2>/dev/null || echo "branch not on remote")
echo "Unpushed commits: ${UNPUSHED:-none}"

# Does a remote branch exist?
REMOTE_EXISTS=$(git ls-remote --heads origin "$CURRENT_BRANCH" 2>/dev/null)
echo "Remote branch exists: ${REMOTE_EXISTS:+yes}"

Step 2 — Check PR and Tag State

# Check for open PRs from this branch (requires gh CLI)
gh pr list --head "$CURRENT_BRANCH" --state all --json number,state,title,mergeCommit 2>/dev/null || echo "gh CLI not available"

# What is the latest version tag?
LATEST_TAG=$(git tag -l 'v4.0_*' --sort=-version:refname | head -1)
echo "Latest Spark 4 tag: $LATEST_TAG"

LATEST_TAG_3=$(git tag -l 'v3.0_*' --sort=-version:refname | head -1)
echo "Latest Spark 3 tag: $LATEST_TAG_3"

# What version is in pom.xml on master?
MASTER_VERSION=$(git show origin/master:pom.xml | grep '<revision>' | sed 's/.*<revision>\(.*\)<\/revision>.*/\1/')
echo "master pom.xml version: $MASTER_VERSION"

# What version is in pom.xml on release/spark3?
SPARK3_VERSION=$(git show origin/release/spark3:pom.xml | grep '<revision>' | sed 's/.*<revision>\(.*\)<\/revision>.*/\1/')
echo "release/spark3 pom.xml version: $SPARK3_VERSION"

Step 3 — Decision Table

Use the detected state to determine where to start:

Development Workflow (per feature/fix):

Release Workflow (once per release):

Step 4 — Confirm with User

After detecting the state, summarize what you found and confirm with the user before proceeding. Clarify which workflow (development or release) you are about to execute:

"I detected that you're on branch asaharn/feat/new-feature with 2 unpushed commits based on master. No PR exists yet. I'll run the Development Workflow — push the branch, create a PR to master, then cherry-pick to release/spark3. Does that sound right?"

"Both branches have all changes merged since the last tag. I'll run the Release Workflow — bump the version, update the changelog, and create tags. What version should I use (auto-detect suggests 7.0.6)?"

Development Workflow

Repeat Phases 1–2 for each feature or fix. Multiple features can be developed and cherry-picked independently before doing a release.

Phase 1 — Feature / Fix Development (on master)

Goal: Implement a change on a branch forked from master, get it merged via PR.

Commit & PR Title Convention

Follow Conventional Commits for PR titles and commit messages:

<type>: <short description>

Examples: feat: add storage protocol option, fix: resolve CloudInfo cache issue, chore: bump version to 7.0.6

Steps

# 1. Create a feature/fix branch from latest master
git checkout master
git pull origin master
git checkout -b <username>/<type>/<short-description>
# Example: asaharn/feat/new-read-mode  or  asaharn/fix/cloud-info-fix

# 2. Implement the change
# ... make code changes ...

# 3. Commit and push (use conventional commit message)
git add -A
git commit -m "<type>: <short description>"
git push origin <branch-name>

Then on GitHub

1. Open a Pull Request targeting master — use conventional commit format for the PR title (e.g., feat: add storage protocol option)
2. Fill in the PR template (Breaking Changes / Features / Fixes sections)
3. Wait for CI checks (build.yml) to pass
4. Get the PR reviewed and approved
5. Squash merge the PR into master
6. Note the squash merge commit SHA from the merged PR — you will need it for Phase 2

Validation

# Confirm the merge commit is on master
git checkout master
git pull origin master
git --no-pager log --oneline -1
# Should show the squash merge commit

Phase 2 — Cherry-pick to Spark 3 (release/spark3)

Goal: Apply the same change to release/spark3 via cherry-pick.

Input required: The squash merge commit SHA from Phase 1.

Steps

# 1. Create a branch from release/spark3
git checkout release/spark3
git pull origin release/spark3
git checkout -b <username>/<type>/<short-description>_3.0
# Example: asaharn/bugfix/cloud-info-fix_3.0

# 2. Cherry-pick the squash merge commit from master
git cherry-pick <squash-merge-commit-sha>

Handling Cherry-pick Conflicts

If git cherry-pick reports conflicts:

# 1. List conflicted files
git diff --name-only --diff-filter=U

# 2. Open each conflicted file and resolve conflicts manually
#    - Conflicts are typically in pom.xml due to version differences
#    - Keep the release/spark3 version values (spark.version.major=3.5, scala versions, etc.)
#    - Keep the functional code changes from the cherry-picked commit

# 3. After resolving all conflicts
git add -A
git cherry-pick --continue

Push and Merge

# 3. Push the branch
git push origin <branch-name>

Then on GitHub:

1. Open a Pull Request targeting release/spark3
2. Wait for CI checks to pass
3. Squash merge the PR into release/spark3

Validation

git checkout release/spark3
git pull origin release/spark3
git --no-pager log --oneline -1
# Should show the squash merge commit on release/spark3

Release Workflow

Run Phases 3–5 once when ready to release. This bundles all changes merged since the last tag into a single version bump, changelog entry, and release.

Phase 3 — Version Bump

Goal: Update the <revision> property in pom.xml to the new version on both branches.

Determine the New Version

# Auto-detect current version from latest Spark 4 tag
CURRENT_VERSION=$(git tag -l 'v4.0_*' --sort=-version:refname | head -1 | sed 's/v4.0_//')
echo "Current version: $CURRENT_VERSION"

# Compute next patch version
MAJOR=$(echo $CURRENT_VERSION | cut -d. -f1)
MINOR=$(echo $CURRENT_VERSION | cut -d. -f2)
PATCH=$(echo $CURRENT_VERSION | cut -d. -f3)
NEXT_PATCH=$((PATCH + 1))
NEW_VERSION="${MAJOR}.${MINOR}.${NEXT_PATCH}"
echo "Next version: $NEW_VERSION"

Manual override: If the user specifies a version, use that instead of auto-detection.

Bump on master

git checkout master
git pull origin master
git checkout -b <username>/chore/version-bump-${NEW_VERSION}

# Update the revision property in pom.xml
sed -i "s|<revision>.*</revision>|<revision>${NEW_VERSION}</revision>|" pom.xml

# Also update kusto.sdk.version if it tracks the same version
# Check first:
grep '<kusto.sdk.version>' pom.xml
# Only update if it matches the old version

git add pom.xml
git commit -m "chore: bump version to ${NEW_VERSION}"
git push origin <branch-name>

Then on GitHub:

1. Open a PR targeting master
2. Squash merge after CI passes

Bump on release/spark3

git checkout release/spark3
git pull origin release/spark3
git checkout -b <username>/chore/version-bump-${NEW_VERSION}_3.0

# Update the revision property in pom.xml
sed -i "s|<revision>.*</revision>|<revision>${NEW_VERSION}</revision>|" pom.xml

# Also update kusto.sdk.version if it tracks the same version
grep '<kusto.sdk.version>' pom.xml

git add pom.xml
git commit -m "chore: bump version to ${NEW_VERSION}"
git push origin <branch-name>

Then on GitHub:

1. Open a PR targeting release/spark3
2. Squash merge after CI passes

Validation

# Verify on master
git checkout master && git pull origin master
grep '<revision>' pom.xml
# Should show: <revision>{NEW_VERSION}</revision>

# Verify on release/spark3
git checkout release/spark3 && git pull origin release/spark3
grep '<revision>' pom.xml
# Should show: <revision>{NEW_VERSION}</revision>

Phase 4 — Changelog Update

Goal: Update CHANGELOG.md with the new version entry on both branches.

Gather Changes

# Get the previous version tag
PREV_TAG=$(git tag -l 'v4.0_*' --sort=-version:refname | head -1)
echo "Previous tag: $PREV_TAG"

# List commits since last tag on master
git --no-pager log --oneline ${PREV_TAG}..origin/master

Update CHANGELOG.md

Edit CHANGELOG.md:

1. Move items from the [Unreleased] section into a new version section
2. Create a new [Unreleased] section with empty subsections
3. Add a new version heading: ## [{NEW_VERSION}] - YYYY-MM-DD (use today's date)
4. Categorize changes under: ### Added, ### Changed, ### Fixed, ### Removed
5. Source descriptions from the PR titles and PR template sections

The changelog entry should look like:

## [Unreleased]

### Changed
- None

### Added
- None

### Fixed
- None

## [{NEW_VERSION}] - YYYY-MM-DD

### Changed
- <items or "None">

### Added
- <items or "None">

### Fixed
- <items or "None">

Commit Changelog on master

git checkout master
git pull origin master
git checkout -b <username>/chore/changelog-${NEW_VERSION}

# Edit CHANGELOG.md as described above

git add CHANGELOG.md
git commit -m "chore: update changelog for ${NEW_VERSION}"
git push origin <branch-name>

Then on GitHub: open PR → squash merge to master.

Cherry-pick Changelog to release/spark3

# After the changelog PR is merged to master, get the merge commit SHA
git checkout master && git pull origin master
CHANGELOG_SHA=$(git --no-pager log --oneline -1 --format='%H')

git checkout release/spark3
git pull origin release/spark3
git checkout -b <username>/chore/changelog-${NEW_VERSION}_3.0

git cherry-pick $CHANGELOG_SHA
# Resolve conflicts if any (unlikely for CHANGELOG.md)

git push origin <branch-name>

Then on GitHub: open PR → squash merge to release/spark3.

Phase 5 — Tagging & Release

Goal: Create version tags on both branches to trigger the release pipeline.

Preconditions

# Verify both branches have the correct version in pom.xml
git checkout master && git pull origin master
grep '<revision>' pom.xml
# Must show: <revision>{NEW_VERSION}</revision>

git checkout release/spark3 && git pull origin release/spark3
grep '<revision>' pom.xml
# Must show: <revision>{NEW_VERSION}</revision>

# Verify tags don't already exist
git tag -l "v4.0_${NEW_VERSION}"
git tag -l "v3.0_${NEW_VERSION}"
# Both should return empty

Create and Push Tags

# Tag master (Spark 4)
git checkout master
git pull origin master
git tag "v4.0_${NEW_VERSION}"
git push origin "v4.0_${NEW_VERSION}"

# Tag release/spark3 (Spark 3)
git checkout release/spark3
git pull origin release/spark3
git tag "v3.0_${NEW_VERSION}"
git push origin "v3.0_${NEW_VERSION}"

Trigger Release

After pushing tags, the Azure release pipeline (.azuredevops/release.yml) runs automatically for tags matching v3.0_*.*.* / v4.0_*.*.*. To run it manually (for example to publish with the optional steps enabled):

1. Go to Azure DevOps → Pipelines → the azure-kusto-spark release pipeline
2. Click Run pipeline
3. For Spark 4: set sourceTag to v4.0_{NEW_VERSION}, enable createGitHubRelease and uploadToAzure
4. For Spark 3: set sourceTag to v3.0_{NEW_VERSION}, enable createGitHubRelease and uploadToAzure

Alternatively, via Azure CLI (requires the azure-devops extension):

# Trigger release for Spark 4
az pipelines run --name "<release-pipeline-name>" \
  --branch "refs/tags/v4.0_${NEW_VERSION}" \
  --parameters sourceTag="v4.0_${NEW_VERSION}" createGitHubRelease=true uploadToAzure=true

# Trigger release for Spark 3
az pipelines run --name "<release-pipeline-name>" \
  --branch "refs/tags/v3.0_${NEW_VERSION}" \
  --parameters sourceTag="v3.0_${NEW_VERSION}" createGitHubRelease=true uploadToAzure=true

Validation

# Verify tags exist on remote
git ls-remote --tags origin | grep "${NEW_VERSION}"
# Should show both v4.0_{NEW_VERSION} and v3.0_{NEW_VERSION}

# Check release pipeline status (via Azure CLI)
PIPELINE_ID="$(az pipelines show --name "<release-pipeline-name>" --query id --output tsv)"
az pipelines runs list --pipeline-ids "${PIPELINE_ID}" --top 4
# Should show two runs (one per tag) in progress or completed

Quick Reference

Development Workflow (per feature/fix)

Release Workflow (once per release)

Error Handling

Cherry-pick conflict

• Most common in pom.xml due to different Spark/Scala version properties
• Keep the release/spark3 values for Spark/Scala-specific properties
• Keep the functional code changes from the cherry-picked commit
• If unsure, ask the user to resolve manually

CI failure after cherry-pick

• The Spark 3 build may fail if the change relies on Spark 4-only APIs
• Ask the user whether the change needs adaptation for Spark 3

Tag already exists

• If a tag already exists, do NOT overwrite it
• Ask the user whether to use a different version number or delete the existing tag

Release workflow failure

• Check the pipeline run logs in Azure DevOps (az pipelines runs show --id <run-id>)
• Common issues: missing variable group values or service connection approval, Maven build failure, Azure storage upload failure
• Re-run the pipeline after fixing the issue

Rules

• Always use squash merge for PRs — never regular merge or rebase merge
• Never push directly to master or release/spark3 — always go through PRs
• Always verify CI passes before merging
• Always confirm the version in pom.xml matches the tag being created
• When in doubt about any step, ask the user before proceeding
• Keep CHANGELOG.md in sync on both branches