agentleFS
PatrickGallucciPatrickGallucci/fabric-skills/skills/fabric-spark-perf-remediate/SKILL.md

fabric-spark-perf-remediate

Skill17 starsChanged 9 months agoMITView on GitHub

Diagnose and resolve Apache Spark performance issues in Microsoft Fabric. Use when asked to troubleshoot slow Spark notebooks, optimize Spark SQL queries, fix data skew or shuffle bottlenecks, tune spark.sql.shuffle.partitions or autoBroadcastJoinThreshold, configure resource profiles (writeHeavy, readHeavyForSpark, readHeavyForPBI), enable autotune, resolve HTTP 430 throttling errors, analyze Spark UI stages and executors, optimize Delta Lake writes with VOrder or Optimized Write, run table maintenance (bin-compaction, vacuum, Z-Order), fix small files problems, tune streaming throughput, or right-size Fabric Spark pools and capacity SKUs.

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 9 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈9

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Start here when a Spark job is slow:

    line 36
  2. Distinctive:

    Is the job queued or throttled?: Check Monitoring Hub for HTTP 430.

    line 38
  3. Distinctive:

    Did the Spark Advisor flag warnings?: Check notebook cell indicators.

    line 42
  4. Distinctive:

    Are executors underutilized?: Check Resources tab in monitoring detail.

    line 51
  5. Distinctive:

    All cores busy → See Partitioning Strategy

    line 53
  6. Distinctive:

    Is the issue write-related?: Check write duration in Spark UI.

    line 55
  7. Distinctive:

    Check autotune status: after a query:

    line 100
  8. Distinctive:

    Run the Fabric Spark diagnostics script to collect Spark application metrics via the Fabric REST API:

    line 128
  9. Distinctive:

    Use the performance analysis notebook template as a starting point for in-session diagnostics:

    line 136

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Microsoft Fabric Apache Spark Performance remediate

Systematic workflows for diagnosing, analyzing, and resolving Apache Spark performance problems in Microsoft Fabric Data Engineering and Data Science workloads.

When to Use This Skill

Activate when encountering any of the following scenarios:

  • Spark notebooks or jobs running slower than expected
  • Capacity throttling errors (HTTP 430 / TooManyRequestsForCapacity)
  • Data skew detected by Spark Advisor in notebook cells
  • Excessive shuffle read/write in Spark UI stages
  • Small files accumulation in Delta Lake tables
  • Streaming ingestion throughput degradation
  • Need to select or tune a Fabric Spark resource profile
  • VOrder vs. Optimized Write decision-making
  • Autotune configuration and validation
  • Right-sizing Spark pools, node counts, or Fabric capacity SKUs

Prerequisites

  • Access to a Microsoft Fabric workspace with Data Engineering enabled
  • Contributor or higher role on the workspace
  • Familiarity with PySpark or Spark SQL
  • PowerShell 7+ (for diagnostic scripts)
  • Fabric REST API access token (for API-based diagnostics)

Quick Diagnosis Decision Tree

Start here when a Spark job is slow:

  1. Is the job queued or throttled? Check Monitoring Hub for HTTP 430.
    • Yes → See Capacity and Concurrency Tuning
    • No → Continue
  2. Did the Spark Advisor flag warnings? Check notebook cell indicators.
    • Data Skew detected → See Data Skew Resolution
    • No warnings → Continue
  3. Is a single stage disproportionately slow? Open Spark UI → Stages tab.
    • Yes, shuffle stage → See Shuffle Optimization
    • Yes, scan stage → See File Scan Optimization
    • No → Continue
  4. Are executors underutilized? Check Resources tab in monitoring detail.
    • High idle cores → See Pool and Executor Sizing
    • All cores busy → See Partitioning Strategy
  5. Is the issue write-related? Check write duration in Spark UI.
    • Yes → See Delta Write Optimization
    • No → See General Spark SQL Tuning

Core Spark Configuration Quick Reference

These are the three settings Fabric Autotune manages automatically. If autotune is disabled, tune them manually:

SettingDefaultPurposeTuning Guidance
spark.sql.shuffle.partitions200Partition count during joins/aggregationsSet to 2-3x total executor cores for your pool
spark.sql.autoBroadcastJoinThreshold10 MBMax table size for broadcast joinsIncrease to 100-256 MB for star-schema joins
spark.sql.files.maxPartitionBytes128 MBMax bytes per file-read partitionIncrease for large sequential scans, decrease for high parallelism

Resource Profiles Quick Reference

Fabric provides predefined profiles that bundle optimized Spark settings:

ProfileBest ForVOrderKey Characteristics
writeHeavyETL, batch ingestion, streamingDisabledDefault for new workspaces; optimized write throughput
readHeavyForSparkInteractive Spark queries, analyticsEnabledOptimized read paths for Spark workloads
readHeavyForPBIPower BI dashboards, DW queriesEnabledOptimized for DirectLake and cross-engine reads

Apply a profile at the environment level or override per-session:

# Per-session override example
spark.conf.set("spark.fabric.resource.profile", "readHeavyForSpark")

Autotune Quick Start

Enable autotune to let Fabric automatically optimize shuffle partitions, broadcast thresholds, and partition bytes:

# Enable in a notebook session
spark.conf.set("spark.ms.autotune.enabled", "true")

# Or set in Environment > Spark Properties
# spark.ms.autotune.enabled = true

Requirements: Runtime 1.1 or 1.2 only. Not compatible with high concurrency mode or private endpoints. Needs 20-25 iterations to converge on optimal settings.

Check autotune status after a query:

# View autotune decisions in Spark UI SQL tab
# Status values: QUERY_TUNING_SUCCEED, QUERY_TUNING_DISABLED,
#                QUERY_PATTERN_NOT_MATCH, QUERY_DURATION_TOO_SHORT

Common Error Patterns

Error / SymptomRoot CauseQuick Fix
HTTP 430: TooManyRequestsForCapacityAll Spark VCores consumedCancel idle jobs in Monitoring Hub or upgrade SKU
Stage with 200 tasks, 1 task 100x slowerData skew on join/group keyAdd salting or use broadcast join
OOM on executorPartition too large or broadcast too bigIncrease partitions or lower broadcast threshold
Write takes >60% of total job timeSmall files or missing optimizationEnable Optimized Write or run table maintenance
Streaming micro-batch latency increasingCheckpoint overhead or partition mismatchTune trigger interval and Event Hub partitions

Step-by-Step Workflows

For detailed procedures, see the reference guides:

  • Spark Configuration Tuning — Shuffle, broadcast, skew, pool sizing, capacity planning
  • Monitoring and Diagnostics — Spark UI navigation, Monitoring Hub, APIs, log analysis
  • Delta Lake Optimization — VOrder, Optimized Write, table maintenance, partitioning, streaming

Available Scripts

Run the Fabric Spark diagnostics script to collect Spark application metrics via the Fabric REST API:

./scripts/Get-FabricSparkDiagnostics.ps1 -WorkspaceId "<guid>" -Token "<bearer-token>"

Available Templates

Use the performance analysis notebook template as a starting point for in-session diagnostics:

# Paste into a Fabric notebook to analyze current session performance

remediate

ProblemCheckResolution
Autotune not activatingRuntime version, HC mode, private endpointSwitch to Runtime 1.1/1.2, disable HC mode
Resource profile not applyingEnvironment publish statusRepublish environment after profile change
Pool autoscale not scaling upCapacity SKU limitsVerify VCore headroom in Capacity Metrics app
Table maintenance job stuckConcurrent maintenance on same tableWait for previous job or cancel via API
Notebook cell shows no Spark AdvisorRuntime < 3.4Upgrade to Spark 3.4+ runtime

References

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.