/fabric-notebooks — PySpark Notebooks on Microsoft Fabric

Fabric notebooks are the primary compute surface for Spark-based ETL and analytics on OneLake. This skill lays out environment configuration, notebookutils patterns, lakehouse I/O, and how notebooks depend on and schedule each other cleanly.

When to invoke

• Atlas or Diesel is building a Fabric ETL pipeline in Python / PySpark
• A notebook needs to be productionised (scheduled, parameterised, monitored)
• Environment libraries (custom wheels, pinned versions) need standardising across notebooks
• Cross-notebook orchestration is required (run notebook B after notebook A)

Steps

1. Notebook environment setup. Create a Fabric Environment — pin the Spark runtime version, list required libraries (PyPI or custom wheels), set Spark pool size. Attach the environment to notebooks explicitly — never rely on the workspace default in production.
2. PySpark session config. Set adaptive query execution (spark.sql.adaptive.enabled = true), delta autotune, and Fabric-specific optimisations (V-Order enabled for gold writes). Log the Spark UI link on session start so failures are diagnosable.
3. notebookutils patterns. Use notebookutils.credentials.getSecret(<vault>, <name>) for Key Vault access — never inline. Use notebookutils.notebook.run(<path>, <timeout>, <params>) to chain notebooks. Use notebookutils.fs for OneLake file operations. Verify SDK method names against current docs at author time.
4. Lakehouse read/write. Read: spark.read.format("delta").load("abfss://...") or spark.table("<lakehouse>.<schema>.<table>"). Write: df.write.format("delta").mode("overwrite").saveAsTable("...") with overwriteSchema=true only when intentional. Prefer partitioned writes for large tables.
5. Cross-notebook dependencies. Orchestrate with Fabric Data Pipelines (activity per notebook) rather than deep notebook.run chains — pipelines give retries, monitoring, and DAG visibility. Pass parameters via pipeline activity, not by string interpolation in the notebook.
6. Schedule. Attach a schedule via Fabric Data Pipeline trigger or notebook schedule. Set alerting on failure (email or Teams webhook). Log run metadata (start, end, row counts) to a monitoring table in the lakehouse.

Deliverable format

• notebooks/*.ipynb — parameterised, environment-attached notebooks committed to Git
• environment.yaml — Fabric Environment definition with pinned Spark runtime and libraries
• Pipeline JSON export defining the DAG and schedule
• Monitoring table schema (monitoring.runs) for observability

Non-negotiables

• Secrets always via notebookutils.credentials + Key Vault — never inline, never in cell metadata
• Every production notebook has an attached Environment — no reliance on workspace defaults
• Long chains of notebook.run are refactored into Data Pipelines — DAG visibility is not optional
• Every scheduled notebook writes a monitoring row — silent success is as bad as silent failure

References

• Fabric — notebooks overview
• Fabric — environments
• Fabric — notebookutils reference
• Fabric — Spark runtime
• Delta Lake — Spark integration
• Superteam ADR — MS Fabric skills pack