fabric-spark
Skill1 starsChanged 34 days agoMITView on GitHub
Use for Spark in Microsoft Fabric: a notebook (a `.Notebook` folder), PySpark or Spark SQL, a lakehouse and its tables, a Spark job definition or environment; a Spark job that is slow, queued or failed, Livy sessions or the Spark History Server; error handling in notebook code; a materialized lake view (MLV); or AI Functions on a DataFrame.
- Deletes or force-pushes
In this repository38 other files it gives its agents.
- claude/CLAUDE.mdCLAUDE.md
- CLAUDE.mdCLAUDE.md
- author-skillSkill
- drift-auditSkill
- drift-handoffSkill
- drift-updateSkill
- test-skillSkill
- triageSkill
- fabric-iqSkill
- fabric-catalog-governanceSkill
- fabric-deploySkill
- fabric-rest-apiSkill
- fabric-securitySkill
- fabric-variable-librarySkill
- fabric-databaseSkill
- fabric-dataflowSkill
- fabric-data-pipelineSkill
- fabric-warehouse-monitoringSkill
- fabric-warehouseSkill
- fabric-activatorSkill
- fabric-eventhouseSkill
- fabric-event-schema-setSkill
- fabric-eventstreamSkill
- fabric-realtime-dashboardSkill
- learnSkill
- fabric-semantic-model-auditSkill
- fabric-tmdlSkill
- pbip-project-structureSkill
- pbir-report-workflowSkill
- linkedin-highlightsSkill
- msix-packagingSkill
- code-reviewSkill
- commitSkill
- find-sessionSkill
- landSkill
- prune-branchesSkill
- recreate-repoSkill
- run-briefsSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 4 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 24
Use abfss:// URI format for OneLake paths in Spark: abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{item}.Lakehouse/{path}
- Distinctive: line 25
Use workspace GUIDs (not names) in ABFS URIs — spaces are not allowed
- Distinctive: line 27
Use Delta Lake format for all Lakehouse tables
- Very specific: line 46
Shortcuts as definition payload: when authoring a Lakehouse via REST, shortcuts.metadata.json is an array of {name, path, target} objects.
- Very specific: line 50
Run OPTIMIZE regularly to compact small files (target 128 MB – 1 GB per file)
- Very specific: line 51
Run VACUUM to remove unreferenced files
- Distinctive: line 53
Avoid high-cardinality partition columns.
- Distinctive: line 70
Limitations: no support for multi-part checkpoint (a deprecated Delta feature — tables containing them fail to update).
- Very specific: line 74
When creating notebooks via REST API, every code cell must include "outputs": [] and "execution_count": null.
- Very specific: line 111
After 202 + Location header, poll GET {Location} until Succeeded, then call GET {Location}/result (note the /result suffix) to retrieve the …
- Very specific: line 156
Default format (omit format or set to null).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Spark / PySpark in Fabric
This file holds notebooks, lakehouses and the Spark item definitions. Each job below has a reference of its own: read it before answering one.
| The job | Read first |
| A Spark job queued, slow or failed; Livy sessions; the History Server | spark-monitoring.md |
| Error handling in a notebook's code | error-handling.md |
| A materialized lake view | mlv.md |
| AI Functions on a DataFrame | ai-functions.md |
Key Constraints
- Fabric Spark cannot access arbitrary external HTTP/HTTPS URLs — land data in lakehouse
Files/first (via pipeline Copy activity, OneLake API, or curl) - Use
abfss://URI format for OneLake paths in Spark:abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{item}.Lakehouse/{path} - Use workspace GUIDs (not names) in ABFS URIs — spaces are not allowed
mssparkutilsfor Fabric-specific notebook operations (credentials, secrets, file management)- Use Delta Lake format for all Lakehouse tables
Runtime Context vs Spark Session Config
Two different things that are often confused:
| Need | API |
| Workspace / item identity (workspace ID + name, notebook ID + name, default lakehouse ID + name, userId) | notebookutils.runtime.context["currentWorkspaceId"] (etc.) — a dict, documented public API, works in pure-Python notebooks |
| Spark session tuning (shuffle partitions, AQE, Delta settings, case sensitivity) | spark.conf.set(...) / spark.conf.get(...) |
spark.conf.get("trident.workspace.id") also returns the workspace ID but is internal Spark conf, not documented surface, and is unavailable in pure-Python notebooks. Prefer notebookutils.runtime.context for identity lookups; reserve spark.conf.* for session tuning.
Lakehouse Setup
enableSchemasis set at lakehouse creation time only — cannot be retrofitted. Without it the lakehouse only has the defaultdboschema and you must recreate to gain named schemas. Set viacreationPayload: { "enableSchemas": true }onPOST /workspaces/{ws}/items.- Schemas use lowercase names by convention (
bronze/silver/goldfor medallion).DROP SCHEMA <name> CASCADEremoves the schema with all its tables. - Cross-lakehouse Spark SQL uses 3-part names:
lakehouse.schema.tablefor same-workspace,workspace.lakehouse.schema.tablefor cross-workspace. Verify access permissions on each lakehouse. - Lakehouse delete cascades irreversibly: SQL Endpoint deleted, all OneLake data permanently removed, shortcuts pointing in become inaccessible, dependent notebooks fail at runtime.
- Shortcuts as definition payload: when authoring a Lakehouse via REST,
shortcuts.metadata.jsonis an array of{name, path, target}objects. Supportedtarget.typevalues:OneLake,AdlsGen2,AmazonS3,GoogleCloudStorage,S3Compatible,Dataverse. Each target type has its own connection properties.
Lakehouse Table Maintenance (impacts SQL Endpoint performance)
- Run
OPTIMIZEregularly to compact small files (target 128 MB – 1 GB per file) - Run
VACUUMto remove unreferenced files - V-Order write optimization is default — do not disable
- Avoid high-cardinality partition columns; aim for partitions ≥ 1 GB
- SQL Endpoint metadata sync lag is normally < 1 minute but increases with: many lakehouses per workspace, small-file fragmentation, large ETL volume, or SQLEP idle > 15 min (this describes the legacy background sync)
New SQL analytics endpoint metadata sync (PREVIEW, May 2026)
Opt-in faster sync that keeps data queryable within seconds of landing. Preview — and applies to NEW SQL analytics endpoints only: existing endpoints in the workspace stay on the legacy sync above.
- Enable: Workspace settings → Warehouse → New metadata sync (preview). Only endpoints created after enabling get the new sync.
- Architecture: external-tables-based Delta-log parsing with decoupled schema-vs-data change detection (schema changes and data changes refresh separately), plus a periodic background refresh and on-demand refresh when a read query hits stale data.
- New DMV
sys.dm_db_external_tables_log_status—last_update_time_utc,latest_log_version,latest_checkpoint_version,is_blocked(1= last update blocked,0= succeeded). Targeted manual refresh (new-sync endpoints only, for data-only changes):
EXEC sys.sp_dw_refresh_ext_table 'dbo.<table>';For schema changes (add/drop tables or columns, type changes) use the full-item Refresh SQL endpoint metadata REST API instead.
- Limitations: no support for multi-part checkpoint (a deprecated Delta feature — tables containing them fail to update); cannot be enabled when the workspace uses workspace private link.
Notebook REST API / UI Upload
When creating notebooks via REST API, every code cell must include "outputs": [] and "execution_count": null.
Cell source must be an array of strings, not a bare string. nbformat permits either, but Fabric's UI upload (createArtifact) and the REST definition APIs reject the bare-string form with a generic 400 exceptionCulprit:1 that gives no clue which field is wrong. Split on \n and append \n to every line except the last — the standard nbformat convention:
"source": ["line one\n", "line two\n", "last line"]Not:
"source": "line one\nline two\nlast line"Applies to every cell in the notebook, markdown and code. A single bare-string source anywhere in the file fails the whole upload.
Default lakehouse binding uses metadata.dependencies.lakehouse in the notebook content — auto-mounts at runtime so spark.read.table("schema.table") resolves without 3-part naming:
{
"metadata": {
"dependencies": {
"lakehouse": {
"default_lakehouse": "<lakehouse-id>",
"default_lakehouse_workspace_id": "<workspace-id>",
"default_lakehouse_name": "<lakehouse-name>"
}
}
}
}One default lakehouse per notebook. Additional lakehouses are reachable via 3-part names (see Lakehouse Setup).
Definition API gotchas
getDefinitionis a POST, not GET — empty body returns HTTP 411 Length Required. Always send'{}'as the body.- After 202 +
Locationheader, pollGET {Location}untilSucceeded, then callGET {Location}/result(note the/resultsuffix) to retrieve the actual content. Without/resultthe operation reportsSucceededbut returns no payload. updateDefinition?updateMetadata=truerequires a.platformpart indefinition.parts; the flag without.platformreturns 400. For content-only updates omit the flag entirely.- Conversely, omitting
?updateMetadata=truesilently ignores any.platformpart in your payload —displayName/descriptionwon't update.
Notebook execution via REST
POST /v1/workspaces/{ws}/items/{itemId}/jobs/instances?jobType=RunNotebook (see fabric-rest-api skill for the full jobType table).
defaultLakehouserequires bothidANDnamein the execution config. Supplying onlyidreturns 400 — common cause of "DefaultLakehouse: missing name" errors.- Pool selection via
executionData.configuration:useStarterPool: true(dev/shared),useWorkspacePool: true(prod), or a custom pool name (high-memory/GPU). Starter pool falls back when the workspace pool is at capacity. Custom live pools (GA, September 2026) are a fourth option: pre-warmed clusters with ~5-second starts once hydrated, kept warm on a required schedule with an attached environment. They serve notebooks only — interactive, scheduled and pipeline-triggered — never Spark job definitions; they need a paid capacity, not a trial; and they're configured in the portal only, not through the Environment APIs or CI/CD. When every warm cluster is busy, a session falls back to on-demand provisioning. Members can manage them once an admin turns on Customize compute configuration for items; What's New lists delegated management as preview. - Job states:
NotStarted → Running → Completed | Failed | Cancelled. PollGET {Location}from the 202 response, orGET .../jobs/instances/{jobInstanceId}if you captured the ID. - Never retry POST after a network/timeout error. Query
GET .../jobs/instancesfiltered to the last 5 minutes first; if a recent run exists, monitor that. Retrying creates duplicate runs and burns CUs. - Job stuck in
NotStartedlonger than ~2 minutes usually means pool warm-up or capacity SKU contention, not a notebook bug.
In-notebook auto-restart (%%configure retriableOptions)
For pipeline-driven notebook runs, the notebook itself can opt into automatic restart after system failures using %%configure (April 2026):
%%configure
{
"retriableOptions": {
"enabled": true,
"maxAttempt": 3
}
}enabled is the on/off switch; maxAttempt (singular) caps total attempts. Place it as the first cell, same as any other %%configure. This complements — does not replace — the pipeline activity-level retry; configure one layer or the other to avoid stacking retries that multiply CU consumption.
Distinct from the "never retry POST" rule above: that rule applies to external orchestrators submitting jobs via REST; retriableOptions is evaluated inside Fabric's notebook runtime. New feature — verify the schema against current Microsoft Learn before relying on it in production.
Other Spark item definitions
SparkJobDefinition
| Format | Required parts |
SparkJobDefinitionV1 | SparkJobDefinitionV1.json |
SparkJobDefinitionV2 | SparkJobDefinitionV1.json (yes — the file name still says V1 in V2 format), plus Main/<file> and optional Libs/<file> |
V2 only accepts .py and .R files in Main/ and Libs/ — JAR files are not supported in V2 parts. The JSON schema includes executableFile, defaultLakehouseArtifactId, mainClass, additionalLakehouseIds, commandLineArguments, additionalLibraryUris, language, environmentArtifactId.
Environment
Default format (omit format or set to null). Key parts:
| Part Path | Content |
Libraries/PublicLibraries/environment.yml | Conda / pip dependencies |
Setting/Sparkcompute.yml | Pool config: driver_cores, driver_memory, executor_cores, executor_memory, dynamic_executor_allocation (min_executors / max_executors), runtime_version |
Libraries/CustomLibraries/<name>.{jar|py|whl|tar.gz} | Custom user uploads — JAR + Python + wheel + R archive all supported here (unlike SparkJobDefinition V2 Main/Libs) |
A custom live pool attaches to an environment, but its configuration doesn't travel in these parts: Learn says live pools can't be managed via environment public APIs or CI/CD, so a deploy carries the environment and not its pool. Set the pool up in the portal in each workspace.
Reference
- Microsoft Learn: What is a lakehouse in Microsoft Fabric?
- Microsoft Learn: Apache Spark compute in Microsoft Fabric
- Microsoft Learn: NotebookUtils (formerly MSSparkUtils) for Fabric
- Comprehensive MS Learn link bundle (concept / notebooks / lakehouse / performance / SJD / environments / runtime / best practices): references/REFERENCE.md
See also
- fabric-rest-api skill — notebook definition upload API and LRO pattern
- error-handling.md — Tier 1/2 convention for notebook code
- fabric-warehouse-monitoring skill — Query Insights for SQLEP queries against lakehouse tables
- spark-monitoring.md — Livy session listings, Spark History Server mirror, notebook wall-clock attribution
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

