fabric-udf-perf-remediate
Skill17 starsChanged 9 months agoMITView on GitHub
Diagnose and resolve performance issues with Microsoft Fabric User Data Functions. Use when functions are slow, timing out, returning errors, consuming excessive capacity units, or exhibiting cold start latency. Covers execution timeouts, response size limits, connection bottlenecks, logging analysis, capacity metrics monitoring, invocation diagnostics, Python optimization, and UDF service limit remediate.
In this repository24 other files it gives its agents.
- fabric-data-agent-perf-remediateSkill
- fabric-data-agent-remediateSkill
- fabric-data-agentSkill
- fabric-data-factory-perf-remediateSkill
- fabric-dataflows-perf-remediationSkill
- fabric-delta-spark-perfSkill
- fabric-iqSkill
- fabric-lakehouse-access-controlSkill
- fabric-lakehouse-perf-remediateSkill
- fabric-lakehouse-views-perf-remediateSkill
- fabric-network-remediateSkill
- fabric-notebook-perf-remediateSkill
- fabric-onelake-perf-remediateSkill
- fabric-pandas-perf-remediateSkill
- fabric-pbi-perf-remediateSkill
- fabric-pbi-security-remediateSkill
- fabric-performance-monitoringSkill
- fabric-pyspark-perf-remediateSkill
- fabric-rest-api-perf-remediateSkill
- fabric-rest-api-remediateSkill
- fabric-rti-perf-remediateSkill
- fabric-spark-compute-perf-remediateSkill
- fabric-spark-compute-remediateSkill
- fabric-spark-perf-remediateSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
18 rules. About 0 are common, 13 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈13
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 71
Keep PyPI dependency list minimal in definition.json
- Distinctive: line 103
Data source queries: Add WHERE clauses, limit columns, use parameterized queries
- Distinctive: line 106
External API calls: Add timeouts, implement retry with backoff
- Distinctive: line 114
Verify connections in Manage connections panel
- Distinctive: line 116
Check that connected data source artifacts still exist
- Distinctive: line 117
Test the data source independently (run a query directly in the Warehouse/Lakehouse)
- Distinctive: line 124
Check if CU utilization exceeds 100% during the failure window
- Very specific: line 125
Look for HTTP 430 errors in logs: TooManyRequestsForCapacity
- Distinctive: line 131
Consider moving heavy processing to a Notebook and using the UDF as a thin API layer
- Very specific: line 132
Use logging.warning() to flag operations exceeding thresholds
- Distinctive: line 159
Add TOP/LIMIT clauses to queries
- Distinctive: line 182
Avoid unnecessary publishes (each triggers storage write operations)
- Distinctive: line 184
Run the capacity-analysis.ps1 script to generate a capacity usage summary.
- Very specific: line 192
Use only libraries you actually need in definition.json
- Very specific: line 194
Prefer lightweight alternatives (e.g., httpx over requests if async needed)
- Distinctive: line 196
Use lazy imports for rarely-used heavy libraries
- Distinctive: line 210
Use structured logging to make performance data queryable in historical logs:
- Distinctive: line 246
Check current region availability at Fabric region availability.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Microsoft Fabric User Data Functions Performance remediate
Systematic guide for diagnosing and resolving performance issues with Fabric User Data Functions (UDFs). Covers cold starts, execution timeouts, capacity consumption, connection bottlenecks, and Python code optimization.
When to Use This Skill
- Function invocations are slow or intermittently timing out
- Capacity metrics show unexpected CU consumption from UDF operations
- Functions fail with timeout, response size, or connection errors
- Cold start latency is impacting downstream consumers (Pipelines, Notebooks, Power BI)
- Historical logs show increasing duration trends
- Need to optimize UDF code for better performance within service limits
Prerequisites
- Access to the Fabric portal with permissions on the User Data Functions item
- Microsoft Fabric Capacity Metrics app installed (for CU analysis)
- Python 3.11+ locally (for code profiling outside Fabric)
- PowerShell 7+ (for running diagnostic scripts)
Service Limits Quick Reference
| Limit | Value | Impact |
| Request payload | 4 MB | All input parameters combined |
| Execution timeout | 240 seconds | Maximum function runtime |
| Response size | 30 MB | Maximum return value size |
| Log retention | 30 days | Historical invocation log window |
| Private library max | 28.6 MB | Per .whl file upload |
| Test session timeout | 15 minutes | Idle timeout in Develop mode |
| Daily log ingestion | 250 MB | Logs may be sampled beyond this |
| Python version (Run) | 3.11 | Published functions runtime |
| Python version (Test) | 3.12 | Develop mode test runtime |
Step-by-Step remediate Workflow
Step 1: Identify the Symptom
Determine which category your issue falls into:
| Symptom | Likely Root Cause | Go To |
| First invocation slow, subsequent fast | Cold start / initialization | Step 2 |
| All invocations consistently slow | Code inefficiency or data volume | Step 3 |
| Intermittent timeouts | Connection issues or capacity throttling | Step 4 |
| Response too large error | Unbounded query results | Step 5 |
| High CU consumption in Metrics app | Excessive execution frequency or duration | Step 6 |
| Function fails with import errors | Library loading overhead | Step 7 |
Step 2: Diagnose Cold Start Latency
Fabric User Data Functions run in a serverless environment. The first invocation after a period of inactivity incurs initialization overhead.
Check historical logs for the pattern:
- Switch to Run only mode in the Functions portal
- Open View historical log for the target function
- Compare Duration(ms) of the first invocation vs. subsequent ones
- A 3-10x difference confirms cold start behavior
Mitigations:
- Implement a health-check or warm-up invocation on a schedule via Pipeline
- Minimize top-level imports; use lazy imports for heavy libraries
- Reduce private library count and size (each
.whladds init time) - Keep PyPI dependency list minimal in
definition.json
Step 3: Profile Slow Function Code
For consistently slow functions, instrument your code with timing:
import logging
import time
@udf.function()
def my_function(param: str) -> str:
start = time.perf_counter()
# Phase 1: Data retrieval
t1 = time.perf_counter()
data = fetch_data(param)
logging.info(f"Data retrieval: {time.perf_counter() - t1:.3f}s")
# Phase 2: Processing
t2 = time.perf_counter()
result = process(data)
logging.info(f"Processing: {time.perf_counter() - t2:.3f}s")
logging.info(f"Total execution: {time.perf_counter() - start:.3f}s")
return resultReview logs in the Invocation details pane to identify the slowest phase.
Common bottlenecks and fixes:
- Data source queries: Add WHERE clauses, limit columns, use parameterized queries
- DataFrame operations: Filter early, avoid iterrows(), use vectorized operations
- Serialization: Return only required fields, use compact formats
- External API calls: Add timeouts, implement retry with backoff
See performance-optimization.md for detailed code patterns.
Step 4: Investigate Connection and Timeout Issues
Connection errors to Fabric data sources:
- Verify connections in Manage connections panel
- Confirm credentials are valid and not expired
- Check that connected data source artifacts still exist
- Test the data source independently (run a query directly in the Warehouse/Lakehouse)
Capacity throttling indicators:
- Open the Microsoft Fabric Capacity Metrics app
- Navigate to the Compute page
- Filter to the workspace containing your UDF
- Check if CU utilization exceeds 100% during the failure window
- Look for HTTP 430 errors in logs:
TooManyRequestsForCapacity
Timeout approaching 240s:
- Break large operations into smaller chunks
- Implement pagination in data retrieval
- Consider moving heavy processing to a Notebook and using the UDF as a thin API layer
- Use
logging.warning()to flag operations exceeding thresholds
Step 5: Resolve Response Size Issues
The 30 MB response limit triggers when functions return large datasets unbounded.
Diagnostic approach:
import sys
import json
import logging
@udf.function()
def my_query_function() -> list:
results = execute_query()
size_bytes = sys.getsizeof(json.dumps(results))
logging.info(f"Response size estimate: {size_bytes / (1024*1024):.2f} MB")
if size_bytes > 25_000_000: # 25 MB warning threshold
logging.warning("Response approaching 30 MB limit")
return resultsMitigations:
- Add TOP/LIMIT clauses to queries
- Implement pagination with offset parameters
- Return summary/aggregated data instead of raw rows
- Compress or filter response fields
Step 6: Analyze Capacity Consumption
UDF operations reported in the Fabric Capacity Metrics app:
| Operation | Type | Trigger |
| User Data Functions Execution | Interactive | Function invoked by portal, Fabric item, or external app |
| User Data Functions Portal Test | Interactive | Testing in Develop mode (minimum 15-min session) |
| User Data Functions Static Storage | Background | Metadata stored in OneLake (always-on cost) |
| User Data Functions Static Storage Read | Background | Metadata read after inactivity period |
| User Data Functions Static Storage Write | Background | Every publish operation |
Cost reduction strategies:
- Reduce invocation frequency from calling items (Pipelines, Notebooks)
- Cache results in the caller when data doesn't change frequently
- Optimize function duration (execution time directly impacts CU consumption)
- Consolidate multiple small functions into fewer, more efficient ones
- Avoid unnecessary publishes (each triggers storage write operations)
Run the capacity-analysis.ps1 script to generate a capacity usage summary.
Step 7: Resolve Library Loading Issues
Heavy or numerous libraries increase initialization time and can cause import errors.
Best practices:
- Use only libraries you actually need in
definition.json - Pin specific versions to avoid unexpected updates
- Prefer lightweight alternatives (e.g.,
httpxoverrequestsif async needed) - Custom
.whlfiles must be under 28.6 MB each - Use lazy imports for rarely-used heavy libraries
# Instead of top-level import
# import heavy_library
@udf.function()
def my_function() -> str:
import heavy_library # Lazy import - only loads when function is called
return heavy_library.process()Logging Best Practices for Performance Monitoring
Use structured logging to make performance data queryable in historical logs:
import logging
# Log at key decision points
logging.info(f"PERF|function_name|phase|{duration_ms}ms|{record_count} rows")
# Use appropriate levels
logging.warning(f"PERF|slow_query|{duration_ms}ms exceeds 5000ms threshold")
logging.error(f"PERF|timeout_risk|{duration_ms}ms approaching 240s limit")See the logging template for a reusable instrumented function pattern.
remediate Decision Tree
Function is slow or failing
├── First call only? → Cold start (Step 2)
├── All calls slow?
│ ├── Data source query slow? → Optimize query (Step 3)
│ ├── Processing slow? → Profile code (Step 3)
│ └── Response too large? → Add pagination (Step 5)
├── Intermittent failures?
│ ├── HTTP 430 errors? → Capacity throttling (Step 4)
│ ├── Connection timeout? → Data source issues (Step 4)
│ └── Import errors? → Library problems (Step 7)
└── High CU bill?
└── Analyze metrics app (Step 6)Regional Limitations
User Data Functions are not available in all Fabric regions. The Test capability in Develop mode is additionally unavailable in Brazil South, Israel Central, and Mexico Central. If your tenant region is unsupported, create a Capacity in a supported region.
Check current region availability at Fabric region availability.
References
- Performance Optimization Patterns - Detailed code optimization techniques
- Capacity Analysis Script - PowerShell script to summarize UDF capacity usage
- Diagnostic Checklist Script - Interactive remediate walkthrough
- Performance Logging Template - Instrumented function boilerplate
- Microsoft Docs: View Function Logs
- Microsoft Docs: Service Limits
- Microsoft Docs: Fabric Operations
- Microsoft Docs: Capacity Metrics App
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

