agentleFS
wardawgmalviciouswardawgmalvicious/agent-config/skills/fabric-engineering/fabric-spark/SKILL.md

fabric-spark

Skill1 starsChanged 34 days agoMITView on GitHub

Use for Spark in Microsoft Fabric: a notebook (a `.Notebook` folder), PySpark or Spark SQL, a lakehouse and its tables, a Spark job definition or environment; a Spark job that is slow, queued or failed, Livy sessions or the Spark History Server; error handling in notebook code; a materialized lake view (MLV); or AI Functions on a DataFrame.

  • Deletes or force-pushes

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 4 are distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use abfss:// URI format for OneLake paths in Spark: abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{item}.Lakehouse/{path}

    line 24
  2. Distinctive:

    Use workspace GUIDs (not names) in ABFS URIs — spaces are not allowed

    line 25
  3. Distinctive:

    Use Delta Lake format for all Lakehouse tables

    line 27
  4. Very specific:

    Shortcuts as definition payload: when authoring a Lakehouse via REST, shortcuts.metadata.json is an array of {name, path, target} objects.

    line 46
  5. Very specific:

    Run OPTIMIZE regularly to compact small files (target 128 MB – 1 GB per file)

    line 50
  6. Very specific:

    Run VACUUM to remove unreferenced files

    line 51
  7. Distinctive:

    Avoid high-cardinality partition columns.

    line 53
  8. Distinctive:

    Limitations: no support for multi-part checkpoint (a deprecated Delta feature — tables containing them fail to update).

    line 70
  9. Very specific:

    When creating notebooks via REST API, every code cell must include "outputs": [] and "execution_count": null.

    line 74
  10. Very specific:

    After 202 + Location header, poll GET {Location} until Succeeded, then call GET {Location}/result (note the /result suffix) to retrieve the …

    line 111
  11. Very specific:

    Default format (omit format or set to null).

    line 156

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Spark / PySpark in Fabric

This file holds notebooks, lakehouses and the Spark item definitions. Each job below has a reference of its own: read it before answering one.

The jobRead first
A Spark job queued, slow or failed; Livy sessions; the History Serverspark-monitoring.md
Error handling in a notebook's codeerror-handling.md
A materialized lake viewmlv.md
AI Functions on a DataFrameai-functions.md

Key Constraints

  • Fabric Spark cannot access arbitrary external HTTP/HTTPS URLs — land data in lakehouse Files/ first (via pipeline Copy activity, OneLake API, or curl)
  • Use abfss:// URI format for OneLake paths in Spark: abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{item}.Lakehouse/{path}
  • Use workspace GUIDs (not names) in ABFS URIs — spaces are not allowed
  • mssparkutils for Fabric-specific notebook operations (credentials, secrets, file management)
  • Use Delta Lake format for all Lakehouse tables

Runtime Context vs Spark Session Config

Two different things that are often confused:

NeedAPI
Workspace / item identity (workspace ID + name, notebook ID + name, default lakehouse ID + name, userId)notebookutils.runtime.context["currentWorkspaceId"] (etc.) — a dict, documented public API, works in pure-Python notebooks
Spark session tuning (shuffle partitions, AQE, Delta settings, case sensitivity)spark.conf.set(...) / spark.conf.get(...)

spark.conf.get("trident.workspace.id") also returns the workspace ID but is internal Spark conf, not documented surface, and is unavailable in pure-Python notebooks. Prefer notebookutils.runtime.context for identity lookups; reserve spark.conf.* for session tuning.

Lakehouse Setup

  • enableSchemas is set at lakehouse creation time only — cannot be retrofitted. Without it the lakehouse only has the default dbo schema and you must recreate to gain named schemas. Set via creationPayload: { "enableSchemas": true } on POST /workspaces/{ws}/items.
  • Schemas use lowercase names by convention (bronze/silver/gold for medallion). DROP SCHEMA <name> CASCADE removes the schema with all its tables.
  • Cross-lakehouse Spark SQL uses 3-part names: lakehouse.schema.table for same-workspace, workspace.lakehouse.schema.table for cross-workspace. Verify access permissions on each lakehouse.
  • Lakehouse delete cascades irreversibly: SQL Endpoint deleted, all OneLake data permanently removed, shortcuts pointing in become inaccessible, dependent notebooks fail at runtime.
  • Shortcuts as definition payload: when authoring a Lakehouse via REST, shortcuts.metadata.json is an array of {name, path, target} objects. Supported target.type values: OneLake, AdlsGen2, AmazonS3, GoogleCloudStorage, S3Compatible, Dataverse. Each target type has its own connection properties.

Lakehouse Table Maintenance (impacts SQL Endpoint performance)

  • Run OPTIMIZE regularly to compact small files (target 128 MB – 1 GB per file)
  • Run VACUUM to remove unreferenced files
  • V-Order write optimization is default — do not disable
  • Avoid high-cardinality partition columns; aim for partitions ≥ 1 GB
  • SQL Endpoint metadata sync lag is normally < 1 minute but increases with: many lakehouses per workspace, small-file fragmentation, large ETL volume, or SQLEP idle > 15 min (this describes the legacy background sync)

New SQL analytics endpoint metadata sync (PREVIEW, May 2026)

Opt-in faster sync that keeps data queryable within seconds of landing. Preview — and applies to NEW SQL analytics endpoints only: existing endpoints in the workspace stay on the legacy sync above.

  • Enable: Workspace settings → Warehouse → New metadata sync (preview). Only endpoints created after enabling get the new sync.
  • Architecture: external-tables-based Delta-log parsing with decoupled schema-vs-data change detection (schema changes and data changes refresh separately), plus a periodic background refresh and on-demand refresh when a read query hits stale data.
  • New DMV sys.dm_db_external_tables_log_status — last_update_time_utc, latest_log_version, latest_checkpoint_version, is_blocked (1 = last update blocked, 0 = succeeded).
  • Targeted manual refresh (new-sync endpoints only, for data-only changes):

    EXEC sys.sp_dw_refresh_ext_table 'dbo.<table>';

    For schema changes (add/drop tables or columns, type changes) use the full-item Refresh SQL endpoint metadata REST API instead.

  • Limitations: no support for multi-part checkpoint (a deprecated Delta feature — tables containing them fail to update); cannot be enabled when the workspace uses workspace private link.

Notebook REST API / UI Upload

When creating notebooks via REST API, every code cell must include "outputs": [] and "execution_count": null.

Cell source must be an array of strings, not a bare string. nbformat permits either, but Fabric's UI upload (createArtifact) and the REST definition APIs reject the bare-string form with a generic 400 exceptionCulprit:1 that gives no clue which field is wrong. Split on \n and append \n to every line except the last — the standard nbformat convention:

"source": ["line one\n", "line two\n", "last line"]

Not:

"source": "line one\nline two\nlast line"

Applies to every cell in the notebook, markdown and code. A single bare-string source anywhere in the file fails the whole upload.

Default lakehouse binding uses metadata.dependencies.lakehouse in the notebook content — auto-mounts at runtime so spark.read.table("schema.table") resolves without 3-part naming:

{
  "metadata": {
    "dependencies": {
      "lakehouse": {
        "default_lakehouse": "<lakehouse-id>",
        "default_lakehouse_workspace_id": "<workspace-id>",
        "default_lakehouse_name": "<lakehouse-name>"
      }
    }
  }
}

One default lakehouse per notebook. Additional lakehouses are reachable via 3-part names (see Lakehouse Setup).

Definition API gotchas

  • getDefinition is a POST, not GET — empty body returns HTTP 411 Length Required. Always send '{}' as the body.
  • After 202 + Location header, poll GET {Location} until Succeeded, then call GET {Location}/result (note the /result suffix) to retrieve the actual content. Without /result the operation reports Succeeded but returns no payload.
  • updateDefinition?updateMetadata=true requires a .platform part in definition.parts; the flag without .platform returns 400. For content-only updates omit the flag entirely.
  • Conversely, omitting ?updateMetadata=true silently ignores any .platform part in your payload — displayName/description won't update.

Notebook execution via REST

POST /v1/workspaces/{ws}/items/{itemId}/jobs/instances?jobType=RunNotebook (see fabric-rest-api skill for the full jobType table).

  • defaultLakehouse requires both id AND name in the execution config. Supplying only id returns 400 — common cause of "DefaultLakehouse: missing name" errors.
  • Pool selection via executionData.configuration: useStarterPool: true (dev/shared), useWorkspacePool: true (prod), or a custom pool name (high-memory/GPU). Starter pool falls back when the workspace pool is at capacity. Custom live pools (GA, September 2026) are a fourth option: pre-warmed clusters with ~5-second starts once hydrated, kept warm on a required schedule with an attached environment. They serve notebooks only — interactive, scheduled and pipeline-triggered — never Spark job definitions; they need a paid capacity, not a trial; and they're configured in the portal only, not through the Environment APIs or CI/CD. When every warm cluster is busy, a session falls back to on-demand provisioning. Members can manage them once an admin turns on Customize compute configuration for items; What's New lists delegated management as preview.
  • Job states: NotStarted → Running → Completed | Failed | Cancelled. Poll GET {Location} from the 202 response, or GET .../jobs/instances/{jobInstanceId} if you captured the ID.
  • Never retry POST after a network/timeout error. Query GET .../jobs/instances filtered to the last 5 minutes first; if a recent run exists, monitor that. Retrying creates duplicate runs and burns CUs.
  • Job stuck in NotStarted longer than ~2 minutes usually means pool warm-up or capacity SKU contention, not a notebook bug.

In-notebook auto-restart (%%configure retriableOptions)

For pipeline-driven notebook runs, the notebook itself can opt into automatic restart after system failures using %%configure (April 2026):

%%configure
{
  "retriableOptions": {
    "enabled": true,
    "maxAttempt": 3
  }
}

enabled is the on/off switch; maxAttempt (singular) caps total attempts. Place it as the first cell, same as any other %%configure. This complements — does not replace — the pipeline activity-level retry; configure one layer or the other to avoid stacking retries that multiply CU consumption.

Distinct from the "never retry POST" rule above: that rule applies to external orchestrators submitting jobs via REST; retriableOptions is evaluated inside Fabric's notebook runtime. New feature — verify the schema against current Microsoft Learn before relying on it in production.

Other Spark item definitions

SparkJobDefinition

FormatRequired parts
SparkJobDefinitionV1SparkJobDefinitionV1.json
SparkJobDefinitionV2SparkJobDefinitionV1.json (yes — the file name still says V1 in V2 format), plus Main/<file> and optional Libs/<file>

V2 only accepts .py and .R files in Main/ and Libs/ — JAR files are not supported in V2 parts. The JSON schema includes executableFile, defaultLakehouseArtifactId, mainClass, additionalLakehouseIds, commandLineArguments, additionalLibraryUris, language, environmentArtifactId.

Environment

Default format (omit format or set to null). Key parts:

Part PathContent
Libraries/PublicLibraries/environment.ymlConda / pip dependencies
Setting/Sparkcompute.ymlPool config: driver_cores, driver_memory, executor_cores, executor_memory, dynamic_executor_allocation (min_executors / max_executors), runtime_version
Libraries/CustomLibraries/<name>.{jar|py|whl|tar.gz}Custom user uploads — JAR + Python + wheel + R archive all supported here (unlike SparkJobDefinition V2 Main/Libs)

A custom live pool attaches to an environment, but its configuration doesn't travel in these parts: Learn says live pools can't be managed via environment public APIs or CI/CD, so a deploy carries the environment and not its pool. Set the pool up in the portal in each workspace.

Reference

See also

  • fabric-rest-api skill — notebook definition upload API and LRO pattern
  • error-handling.md — Tier 1/2 convention for notebook code
  • fabric-warehouse-monitoring skill — Query Insights for SQLEP queries against lakehouse tables
  • spark-monitoring.md — Livy session listings, Spark History Server mirror, notebook wall-clock attribution

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.