agentleFS
PatrickGallucciPatrickGallucci/fabric-skills/skills/fabric-data-factory-perf-remediate/SKILL.md

fabric-data-factory-perf-remediate

Skill17 starsChanged 9 months agoMITView on GitHub

Diagnose and resolve Microsoft Fabric Data Factory pipeline performance issues. Use when pipelines are slow, copy activities timeout, dataflows stall, activities are stuck, throughput is low, capacity is throttled, or jobs queue indefinitely. Covers copy activity tuning (parallelCopies, DIU, ITO, partitioning), pipeline monitoring via Monitoring Hub and workspace monitoring, Spark job queueing, capacity SKU limits, error code resolution, and dataflow optimization. Keywords include Fabric pipeline slow, copy activity performance, Data Factory throttling, pipeline timeout, activity stuck, TooManyRequestsForCapacity, HTTP 430, pipeline troubleshoot, dataflow performance, copy parallelism, intelligent throughput optimization.

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 11 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈11

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the diagnostic script to gather baseline metrics:

    line 44
  2. Distinctive:

    Check source/sink connectivity and credentials

    line 75
  3. Distinctive:

    Verify Fabric capacity is not in throttled state

    line 76
  4. Distinctive:

    Check for connection timeout or network interruption

    line 78
  5. Distinctive:

    Check current Spark concurrency against SKU limits

    line 82
  6. Distinctive:

    Consider upgrading to a larger capacity SKU

    line 84
  7. Distinctive:

    Use job queueing for non-interactive Spark workloads

    line 86
  8. Distinctive:

    Avoid unnecessary data type conversions

    line 91
  9. Distinctive:

    Use staging for large datasets

    line 93
  10. Distinctive:

    Check for connector-specific throttling

    line 94
  11. Distinctive:

    Add a Monitoring Eventhouse and enable Log workspace activity

    line 139

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Microsoft Fabric Data Factory Performance remediate

Systematic approach to diagnosing and resolving performance issues in Microsoft Fabric Data Factory pipelines, copy activities, and dataflows.

When to Use This Skill

  • Pipeline execution takes longer than expected
  • Copy activities are slow or appear stuck
  • Activities show "Not Started" status for extended periods
  • Capacity throttling errors (HTTP 430, TooManyRequestsForCapacity)
  • Throughput is lower than expected for copy operations
  • Dataflow Gen2 refresh is slow or timing out
  • Pipeline monitoring shows performance degradation over time
  • Need to optimize parallelism, DIU, or partitioning settings

Prerequisites

  • Access to Microsoft Fabric workspace with Contributor or higher role
  • Familiarity with the Fabric Monitoring Hub
  • Understanding of Fabric capacity SKUs and their limits
  • PowerShell 7+ for running diagnostic scripts

Diagnostic Workflow

Step 1: Identify the Bottleneck Category

Determine which category your issue falls into:

CategorySymptomsStart Here
Copy Activity SlowLow throughput, long transfer durationcopy-activity-tuning.md
Pipeline StuckActivity shows In Progress with no movementpipeline-stuck-resolution.md
Capacity ThrottlingHTTP 430 errors, jobs queuedcapacity-throttling-guide.md
Dataflow SlowDataflow Gen2 refresh takes too longdataflow-optimization.md
Spark Job QueueJobs stuck in "Not Started" statuscapacity-throttling-guide.md

Step 2: Collect Diagnostics

Run the diagnostic script to gather baseline metrics:

./scripts/Get-FabricPipelineDiagnostics.ps1 -WorkspaceId "<guid>" -PipelineName "MyPipeline"

Or manually collect from the Monitoring Hub:

  1. Open Fabric portal and navigate to Monitoring Hub
  2. Filter by pipeline name and time range
  3. Select the run details (glasses icon) for the slow run
  4. Capture the Duration Breakdown for copy activities
  5. Note the queue time, transfer time, and pre/post-copy script duration

Step 3: Apply Targeted Fixes

Based on the bottleneck category, apply the appropriate optimization from the reference guides.

Quick Fixes for Common Issues

Copy Activity Running Slowly

  1. Set Intelligent Throughput Optimization to Maximum (or custom 4-256)
  2. Configure Degree of Copy Parallelism based on source type
  3. Enable Partition Option for SQL sources (Dynamic Range or Physical)
  4. Pre-calculate partition upper/lower bounds to avoid overhead
  5. Enable Staging when sink is Fabric Warehouse

Pipeline Activity Stuck

  1. Cancel the stuck activity and retry
  2. Check source/sink connectivity and credentials
  3. Verify Fabric capacity is not in throttled state
  4. Review if payload exceeds 896 KB limit
  5. Check for connection timeout or network interruption

Capacity Throttling (HTTP 430)

  1. Check current Spark concurrency against SKU limits
  2. Cancel unnecessary active Spark jobs via Monitoring Hub
  3. Consider upgrading to a larger capacity SKU
  4. Distribute pipeline trigger times to avoid burst load
  5. Use job queueing for non-interactive Spark workloads

Dataflow Gen2 Performance

  1. Reduce data volume with query folding and filters
  2. Avoid unnecessary data type conversions
  3. Minimize the number of transformation steps
  4. Use staging for large datasets
  5. Check for connector-specific throttling

Capacity SKU Quick Reference

SKUMax Spark CoresQueue LimitEquivalent Power BI
F2Limited4-
F4Limited4-
F8Limited8-
F16Limited16-
F32Limited32-
F64Standard64P1
F128Standard128P2
F256Standard256P3
F512Standard512P4
F1024Large1024-
F2048Large2048-
TrialP1 equivN/A (no queue)P1

Copy Activity Performance Settings Reference

SettingPropertyRangeRecommendation
Intelligent Throughput OptimizationdataIntegrationUnitsAuto, Standard (64), Balanced (128), Maximum (256), Custom (4-256)Start with Auto, increase for large datasets
Degree of Copy ParallelismparallelCopies1-256Auto for most; limit to 32 for Fabric Warehouse sink
Partition OptionSource settingsNone, Physical, Dynamic RangeUse Dynamic Range for large SQL tables
Enable StagingenableStagingtrue/falseRequired for Fabric Warehouse sink
Source Retry CountsourceRetryCountIntegerSet 2-3 for transient failures
Fault ToleranceenableSkipIncompatibleRowtrue/falseEnable for non-critical loads

Error Code Quick Reference

ErrorMeaningAction
HTTP 430Capacity compute limit reachedReduce concurrent jobs or upgrade SKU
Payload too largeActivity config exceeds 896 KBReduce parameter sizes
TooManyRequestsForCapacitySpark compute or API rate limitCancel active jobs or wait
Connection timeoutSource/sink unreachableCheck network, credentials, firewall
Deflate64 unsupportedCompression format not supportedRe-compress with deflate algorithm

Monitoring Setup

Enable workspace monitoring for ongoing performance analysis:

  1. Go to Workspace Settings > Monitoring
  2. Add a Monitoring Eventhouse and enable Log workspace activity
  3. Query the ItemJobEventLogs table with KQL for pipeline-level insights

Example KQL query for failure trends:

ItemJobEventLogs
| where ItemKind == "Pipeline"
| summarize count() by JobStatus

See workspace-monitoring-setup.md for detailed configuration.

References

  • Copy Activity Tuning Guide
  • Pipeline Stuck Resolution
  • Capacity Throttling Guide
  • Dataflow Optimization
  • Workspace Monitoring Setup
  • remediate Runbook Template

External Resources

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.