spark-cli
Skill1.4k starsChanged 4 days agoMITView on GitHub
Manages Fabric Spark work, including notebook cell code with %%configure, %%sql, PySpark and notebookutils, named notebook runs, Livy sessions, triage of failed or OOM notebook and pipeline Spark runs, and the Materialized Lake View (MLV) lifecycle. Load it before writing MLV SQL, since CREATE MATERIALIZED LAKE VIEW and its CONSTRAINT clause are Fabric-only. KQL materialized views belong to eventhouse-cli.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about microsoft/skills-for-fabric
- bmad-code-org/BMAD-METHOD“BMAD as a Data Engineer”GitHub · Jun 19, 2026 · 1 reactions
- sql-bi/docs“chore: bump skills-for-fabric marketplace footer to 0.3.21”GitHub · Nov 7, 2026
- sudosubin/agents.nix“claude-code-marketplaces.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21”GitHub · Nov 6, 2026
- sudosubin/agents.nix“agent-skills.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21”GitHub · Nov 6, 2026
- sudosubin/agents.nix“claude-code-plugins.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21”GitHub · Nov 5, 2026
The biggest 5 of 37 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 37
Talked about on GitHub (37 times).
On GitHub · 37
- Jun 19, 2026BMAD as a Data Engineer — bmad-code-org/BMAD-METHOD1 reaction · 2 comments
- Nov 7, 2026chore: bump skills-for-fabric marketplace footer to 0.3.21 — sql-bi/docs0 reactions
- Nov 6, 2026claude-code-marketplaces.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21 — sudosubin/agents.nix0 reactions
- Nov 6, 2026agent-skills.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21 — sudosubin/agents.nix0 reactions
- Nov 5, 2026claude-code-plugins.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21 — sudosubin/agents.nix0 reactions
32 more on GitHub
- Nov 5, 2026copilot-plugins.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21 — sudosubin/agents.nix0 reactions
- Nov 5, 2026copilot-marketplaces.github.microsoft.skills-for-fabric: 0.3.20 → 0.3.21 — sudosubin/agents.nix0 reactions
- Oct 31, 2026AI guides: skills-for-fabric marketplace is now 0.3.20 — sql-bi/docs0 reactions
- Oct 29, 2026copilot-marketplaces.github.microsoft.skills-for-fabric: 0.3.19 → 0.3.20 — sudosubin/agents.nix0 reactions
- Oct 29, 2026claude-code-marketplaces.github.microsoft.skills-for-fabric: 0.3.19 → 0.3.20 — sudosubin/agents.nix0 reactions
- Oct 29, 2026agent-skills.github.microsoft.skills-for-fabric: 0.3.19 → 0.3.20 — sudosubin/agents.nix0 reactions
- Oct 29, 2026copilot-plugins.github.microsoft.skills-for-fabric: 0.3.19 → 0.3.20 — sudosubin/agents.nix0 reactions
- Oct 29, 2026FS-03: Databricks import report names the Fabric migration gotchas — KratosDataGod/KratosForge0 reactions
- Oct 29, 2026claude-code-plugins.github.microsoft.skills-for-fabric: 0.3.19 → 0.3.20 — sudosubin/agents.nix0 reactions
- Oct 28, 2026claude-code-plugins.github.microsoft.skills-for-fabric: init at 0.3.19 — sudosubin/agents.nix0 reactions
- Oct 28, 2026claude-code-marketplaces.github.microsoft.skills-for-fabric: init at 0.3.19 — sudosubin/agents.nix0 reactions
- Oct 21, 2026agent-skills.github.microsoft.skills-for-fabric: 0.3.18 → 0.3.19 — sudosubin/agents.nix0 reactions
- Oct 21, 2026copilot-plugins.github.microsoft.skills-for-fabric: 0.3.18 → 0.3.19 — sudosubin/agents.nix0 reactions
- Oct 21, 2026copilot-marketplaces.github.microsoft.skills-for-fabric: 0.3.18 → 0.3.19 — sudosubin/agents.nix0 reactions
- Oct 18, 2026build: bump @microsoft/powerbi-report-authoring-cli from 0.4.0 to 0.5.0 in /apps/australian-accounting-power-bi — ryanduguid/accounting-review-pipeline0 reactions
- Oct 17, 2026AI tools guides: the plugin's own MCP server definition has no --accepteula — sql-bi/docs0 reactions
- Oct 17, 2026AI tools guides: the powerbi-authoring plugin now has two skills, not five — sql-bi/docs0 reactions
- Oct 16, 2026[Candidate] microsoft/skills-for-fabric - large collection (51 SKILL.md files) — ProSkillsMD/proskills0 reactions · 1 comment
- Oct 15, 2026Bump @microsoft/powerbi-report-authoring-cli from 0.4.0 to 0.5.0 in /mcp — Antnioo/DataArcus0 reactions · 1 comment
- Oct 9, 2026Bump @microsoft/powerbi-report-authoring-cli from 0.1.4 to 0.5.0 — quantameridian/powerbi-kpi-semantic-model0 reactions
- Oct 4, 2026copilot-plugins.github.microsoft.skills-for-fabric: init at 0.3.18 — sudosubin/agents.nix0 reactions
- Oct 4, 2026copilot-marketplaces.github.microsoft.skills-for-fabric: init at 0.3.18 — sudosubin/agents.nix0 reactions
- Oct 3, 2026agent-skills.github.microsoft.skills-for-fabric: 0.3.17 → 0.3.18 — sudosubin/agents.nix0 reactions
- Sep 30, 2026agent-skills.github.microsoft.skills-for-fabric: init at 0.3.17 — sudosubin/agents.nix0 reactions
- Sep 25, 2026Bump @microsoft/powerbi-report-authoring-cli from 0.1.4 to 0.4.0 — quantameridian/powerbi-kpi-semantic-model0 reactions · 1 comment
- Aug 20, 2026fix: remove deprecated fabric-authoring, fabric-consumption, fabric-operations — github/copilot-plugins0 reactions
- Aug 17, 2026[BUG] Marketplace catalog outdated: `fabric-authoring`, `fabric-consumption` and `fabric-operations` — github/copilot-plugins0 reactions
- Aug 16, 2026Add Skills for Fabric to the catalog — gronnerup/fabricstack0 reactions
- Jul 2, 2026library: skills-for-fabric — gronnerup/fabricstack0 reactions
- Jul 2, 2026feat(plugin): ship a plugin marketplace so skills install in one command (GitHub Copilot CLI + Claude Code) — sdebruyn/fabric-dw-mcp-cli0 reactions · 1 comment
- Jun 29, 2026frabic skills — lagimik/PartnerCrucible0 reactions
- Apr 12, 2026[QUESTION] How to install the Skills for fabric-cli and keep them updated? — microsoft/fabric-cli0 reactions · 5 comments
In this repository56 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- activator-cliSkill
- azmon-mirroredcatalogs-operations-cliSkill
- databasehub-cliSkill
- databricks-migrationSkill
- dataflows-cliSkill
- deployment-pipelines-authoring-cliSkill
- e2e-fabric-cost-estimationSkill
- e2e-medallion-architectureSkill
- eventhouse-cliSkill
- eventschemaset-cliSkill
- eventstream-cliSkill
- fabriciqSkill
- fabric-policies-cliSkill
- git-integration-operations-cliSkill
- hdinsight-migrationSkill
- onelake-catalog-cliSkill
- ontology-cliSkill
- pipeline-migrationSkill
- project-osmosSkill
- search-consumption-cliSkill
- semantic-model-authoringSkill
- spark-cliSkill
- sqldb-cliSkill
- sqldw-cliSkill
- synapse-migrationSkill
- variable-library-cliSkill
- powerbi-report-cliSkill
- semantic-model-authoringSkill
- activator-cliSkill
- azmon-mirroredcatalogs-operations-cliSkill
- databasehub-cliSkill
- databricks-migrationSkill
- dataflows-cliSkill
- deployment-pipelines-authoring-cliSkill
- e2e-fabric-cost-estimationSkill
- e2e-medallion-architectureSkill
- eventhouse-cliSkill
- eventschemaset-cliSkill
- eventstream-cliSkill
- fabriciqSkill
- fabric-policies-cliSkill
- git-integration-operations-cliSkill
- hdinsight-migrationSkill
- onelake-catalog-cliSkill
- ontology-cliSkill
- pipeline-migrationSkill
- powerbi-report-cliSkill
- project-osmosSkill
- search-consumption-cliSkill
- semantic-model-authoringSkill
- sqldb-cliSkill
- sqldw-cliSkill
- synapse-migrationSkill
- variable-library-cliSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 5 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 35
If a request genuinely spans modes, handle them one at a time and read each reference before you start that part.
- Distinctive: line 48
Before you report the task done, confirm the terminal call returned success and, where the reference documents a readback, read the …
- Distinctive: line 69
Treat vague requests such as "set up my data" as insufficiently scoped.
- Very specific: line 70
Read references/<mode>.md end to end, as your FIRST tool call, before the first command of that mode.
- Distinctive: line 73
Stop after routing any Eventhouse/KQL materialized-view request.
- Distinctive: line 75
Treat the reference as instructions, never as the deliverable.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Telemetry — MANDATORY. Every
api.fabric.microsoft.comcall must carryx-ms-fabric-skill: spark-cli(az rest:--headers "x-ms-fabric-skill=spark-cli"), including every LRO poll,fabric_lroand retry. Snippets omit it — add it anyway.
CRITICAL NOTES
- To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
- To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
- Skill disambiguation: use
spark-clifor notebook cell code (including%%sqlcells), named notebook runs, Livy sessions, Spark failure triage, and everything about a Materialized Lake View -- writing the definition, reviewing a query for incremental-refresh readiness, and scheduling, refreshing, monitoring or diagnosing an existing one. A KQL materialized view in an Eventhouse iseventhouse-cli; plain read-only T-SQL against a Warehouse or Lakehouse SQL endpoint issqldw-cli.- Hard routing boundary: never execute an Eventhouse/KQL materialized-view request from this skill. Route it to
eventhouse-cli; if that skill is unavailable, state that the request cannot be completed in the current skill context and stop without calling Fabric APIs or creating artifacts.
Fabric Spark and Materialized Lake Views -- CLI Skill
This one skill owns Fabric Spark: notebook cell authoring, notebook runs, Livy-session analysis, Spark failure diagnostics, and the whole Materialized Lake View lifecycle.
It is a mode dispatcher and contains NO procedures. Pick the mode that matches the request from the table below, then read the matching references/<mode>.md file end to end with your file-reading tool BEFORE issuing a single command. That file holds the endpoints, payload shapes, templates and gotchas; acting without it produces wrong payloads and wrong results.
Mode selection
| Mode | Use when the request ... | Example triggers | Read this first |
authoring | writes notebook cell code (PySpark, Scala, SparkR, %%sql, %%configure), runs a notebook by name and reports a NORMAL run status, or authors a Materialized Lake View definition or reviews an MLV query for incremental-refresh readiness | write notebook code, notebook cell code, %%sql cell, run notebook, execute notebook, notebookutils, create materialized lake view, is this MLV query incremental-refresh ready | references/authoring.md |
consumption | runs interactive ad-hoc PySpark in a Lakehouse Livy session -- never a notebook | create a Livy session, run calculation in Livy, PySpark, DataFrame analysis, join tables across lakehouses, Delta time-travel | references/consumption.md |
operations | diagnoses a FAILED, unhealthy, throttled or slow Spark notebook / pipeline / Livy run | failed notebook, Spark Livy health, Spark OOM, why is my notebook slow, job diagnostics, 430 throttling | references/operations.md |
mlv | discovers or operates EXISTING Materialized Lake Views: Spark SQL discovery, refresh schedules, on-demand refresh, run history, cancellation, and refresh-failure classification | discover MLVs, list materialized lake views, schedule MLV, MLV run history, cancel refresh, trigger MLV refresh, diagnose MLV refresh failure | references/mlv.md |
Mode boundary rule
Mode is decided by the artefact and the outcome, not by the language. A notebook cell is always authoring even when the cell is %%sql. A Livy session is always consumption. A Spark run that FAILED or is unhealthy is operations; a run that succeeded is reported by authoring. For a Materialized Lake View the VERB decides: writing or reviewing the definition is authoring, while discovering, scheduling, refreshing, monitoring or diagnosing existing MLVs is mlv. If discovery must be executed, switch to consumption for Livy or authoring for a notebook only after reading the discovery command from references/mlv.md.
If a request genuinely spans modes, handle them one at a time and read each reference before you start that part. If the mode is ambiguous after reading this table, ask one short clarifying question instead of guessing.
Terminal write -- the step you must not skip
Reading the reference and planning the change is NOT completing the task. Each mutating mode ends with one state-changing call. If you did not issue it, nothing was persisted -- say so explicitly rather than reporting success.
| Mode | Terminal write |
authoring | for an EXISTING notebook, POST .../notebooks/{id}/updateDefinition to save the cell; a NEW notebook needs POST /v1/workspaces/{ws}/items first; for a run request, trigger the job via the Jobs API. Printing cell code into the chat is not saving or running it. |
consumption | none -- this mode is read-only |
operations | none -- this mode is read-only |
mlv | POST /v1/workspaces/{ws}/lakehouses/{lakehouse}/jobs/refreshMaterializedLakeViews/instances for an on-demand refresh, or the schedule create/update/delete call for a scheduling request. Reporting what the schedule would be is not creating it. |
Before you report the task done, confirm the terminal call returned success and, where the reference documents a readback, read the artefact back to prove the change landed.
Shared essentials (all modes)
Resolve the workspace and item first; every mode depends on it.
| Task | Reference | Notes |
| Finding Workspaces and Items in Fabric | COMMON-CLI.md | Mandatory -- read before resolving any workspace or item id |
| Fabric Topology & Key Concepts | COMMON-CORE.md | Item types, workspaces, capacities |
| Environment URLs | COMMON-CORE.md | Sovereign / non-public cloud hosts |
| Authentication & Token Acquisition | COMMON-CORE.md | Wrong audience = 401; read before any auth issue |
| Authentication Recipes | COMMON-CLI.md | az login flows and token acquisition |
| Core Control-Plane REST APIs | COMMON-CORE.md | Pagination, LRO polling, rate limiting |
| Gotchas & Troubleshooting | COMMON-CLI.md | az rest audience, shell escaping, token expiry |
Rules
MUST
- Select exactly one mode from the table above before doing anything else.
- Treat vague requests such as "set up my data" as insufficiently scoped. Ask what artefact, source, destination, and outcome the user wants; never infer intent by finding and running an existing notebook or job.
- Read
references/<mode>.mdend to end, as your FIRST tool call, before the first command of that mode. Read it ONCE, in a single full read: do not re-open it, do not grep it again, and do not page through it. You already have it. - Apply the same read-once discipline to every other file you load, including nested
references/<mode>/*.mdsub-references and sharedcommon/*.mdfiles: open only the ones you actually need, read each ONCE end to end, and never grep or re-open one you already loaded. - Resolve workspace and item ids by listing and filtering, never by guessing a GUID.
- Stop after routing any Eventhouse/KQL materialized-view request. Unavailable sibling skills are a blocking boundary, not permission to implement their workload directly.
- Announce a mode switch explicitly when the request crosses a boundary.
- Treat the reference as instructions, never as the deliverable. After reading it, RUN the documented commands against the live workspace and report the real results. Quoting what the reference says instead of executing it does not answer the request.
PREFER
- The narrowest mode that satisfies the request.
- Reading exactly ONE mode reference. Load a second only when the request genuinely spans modes, and say so before you do.
- Reporting the mode you chose in your first response so the user can correct you.
AVOID
- Acting from this dispatcher alone -- it intentionally omits the operational detail.
- Answering with a summary of the reference instead of executing it.
- Re-reading or re-grepping a reference you already loaded; it costs turns and tokens.
- Mutating anything while in a read-only mode.
- Loading a different skill for work this family already owns (see CRITICAL NOTES 3).
- Attempting an out-of-scope request through raw REST calls when the owning skill is unavailable. Name the owning skill and stop; do not improvise by creating Fabric artefacts outside
spark-cli's scope.
Examples
| User request | Mode | Reference to read |
| "Write a notebook cell that reads the bronze table and writes a silver Delta table." | authoring | references/authoring.md |
| "Create a Livy session and count the distinct customers in the lakehouse." | consumption | references/consumption.md |
| "My notebook run failed with an OOM -- work out why." | operations | references/operations.md |
| "Is this materialized lake view query ready for incremental refresh?" | authoring | references/authoring.md |
| "Show me the Spark SQL command that lists the MLVs in schema silver." | mlv | references/mlv.md |
| "Schedule a nightly refresh for the lakehouse MLV lineage and show me its run history." | mlv | references/mlv.md |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

