fabric-data-pipeline
Skill1 starsChanged 34 days agoMITView on GitHub
Use for Microsoft Fabric Data Pipeline item definitions in Git — `pipeline-content.json`, `.platform`, `.schedules`. Covers the envelope (`properties.activities`, plus the `parameters`, `variables` and `libraryVariables` the REST schema omits), `dependsOn` conditions, the activity-type enum and Fabric-specific `typeProperties` (`TridentNotebook`, `PBISemanticModelRefresh`, `SqlServerStoredProcedure`, `ExecutePipeline`), `InvokePipeline` deprecated for `ExecutePipeline` and the GUID-vs-`referenceName` rebinding that migration forces, activity `policy` (the portal's 12-hour timeout vs the 7-day default when absent; preview `retryConditions`, whose interval elapses before the condition is tested), deactivation via `state: Inactive` + `onInactiveMarkAs`, the 120-activity cap, and `.schedules` — `jobType: Execute`, Cron/Daily/Weekly/Monthly where Cron is an interval in minutes, the mandatory end date that silently expires a schedule, and scheduler auto-disable after consecutive failures.
In this repository38 other files it gives its agents.
- claude/CLAUDE.mdCLAUDE.md
- CLAUDE.mdCLAUDE.md
- author-skillSkill
- drift-auditSkill
- drift-handoffSkill
- drift-updateSkill
- test-skillSkill
- triageSkill
- fabric-iqSkill
- fabric-catalog-governanceSkill
- fabric-deploySkill
- fabric-rest-apiSkill
- fabric-securitySkill
- fabric-variable-librarySkill
- fabric-databaseSkill
- fabric-dataflowSkill
- fabric-sparkSkill
- fabric-warehouse-monitoringSkill
- fabric-warehouseSkill
- fabric-activatorSkill
- fabric-eventhouseSkill
- fabric-event-schema-setSkill
- fabric-eventstreamSkill
- fabric-realtime-dashboardSkill
- learnSkill
- fabric-semantic-model-auditSkill
- fabric-tmdlSkill
- pbip-project-structureSkill
- pbir-report-workflowSkill
- linkedin-highlightsSkill
- msix-packagingSkill
- code-reviewSkill
- commitSkill
- find-sessionSkill
- landSkill
- prune-branchesSkill
- recreate-repoSkill
- run-briefsSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 47
Every real pipeline also carries parameters, variables and libraryVariables at that level.
- Distinctive: line 234
First commit after enabling CI/CD is noisy: Every item that already had a schedule shows as uncommitted.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Fabric Data Pipeline definitions
Covers the Git-serialized form of a DataPipeline item — the JSON an agent edits when there is no portal to validate against.
Not here, deliberately: expression syntax (@concat, coalesce,
the ForEach variable race) belongs to the coding-expressions rule,
which globs the same file and always co-loads. Connection rebinding
across environments belongs to fabric-deploy. The libraryVariables
block's own mechanics belong to fabric-variable-library. Copy activity
source/sink/dataset internals are not covered at all.
1. What Git holds
| File | Required | What it is |
pipeline-content.json | yes | The pipeline: activities, parameters, variables |
.platform | no | Item metadata — displayName, logicalId |
.schedules | no | Job-scheduler triggers, one file per item |
Serialization mechanics (EOF newlines, CRLF/LF, .gitattributes) are in
the fabric-git-serialization rule, which globs this folder too.
2. The definition envelope
{ "properties": {
"activities": [ ... ],
"parameters": { "DryRun": { "type": "bool", "defaultValue": false } },
"variables": { "ChangedEntities": { "type": "Array", "defaultValue": [] } },
"libraryVariables": { "EnvSuffix": { "type": "String",
"variableName": "EnvironmentSuffix", "libraryName": "MyVarLib" } }
} }The REST schema page documents only activities and description.
Every real pipeline also carries parameters, variables and
libraryVariables at that level. The page is incomplete — do not
"correct" a file by deleting them (verified against live exports
2026-09-02).
parameters— set at trigger time, immutable during the run.variables— mutable, pipeline-scoped, not iteration-scoped.libraryVariables— Variable Library bindings; seefabric-variable-library. Read with@pipeline().libraryVariables.<alias>, and for the advanced types append.connectionId,.itemIdor.workspaceId.
3. Activity anatomy
{ "name": "IngestSource", "type": "TridentNotebook",
"description": "…", "state": "Inactive", "onInactiveMarkAs": "Succeeded",
"dependsOn": [ { "activity": "LookupConfig",
"dependencyConditions": [ "Succeeded" ] } ],
"policy": { "timeout": "0.02:00:00", "retry": 1 },
"typeProperties": { ... },
"externalReferences": { "connection": "<guid or expression>" } }name and type are required; typeProperties is required and its
shape is type-specific. dependencyConditions are Succeeded,
Failed, Skipped, Completed — an activity with several dependsOn
entries waits for all of them.
Run conditions (preview) are a pipeline-level layer above
dependsOn: they decide when the pipeline itself can start. How they
serialize to Git is undocumented (Learn, 2026-10-06).
externalReferences.connection is a single connection GUID. It is not
source-controlled meaningfully across workspaces — parameterize it
(fabric-deploy) or bind it to a Variable Library connection reference.
4. Activity types
The REST definition's DataPipelineActivityTypes enum has 36 entries;
the full table is in references/REFERENCE.md.
Learn's activity overview lists more activities than the enum carries.
Four newer activities have no documented type string: Lakehouse
maintenance (GA), and Refresh SQL analytics endpoint, Business actions
and Fabric actions (all three preview), per What's New of 2026-10-02.
None is in the enum and no Learn page shows its JSON, and a guessed
string breaks the definition silently. Lakehouse maintenance doesn't yet
support lakehouses with schemas enabled (Learn, 2026-10-06).
The Fabric-specific enum entries and their required typeProperties:
type | Required typeProperties | Notes |
TridentNotebook | notebookId, workspaceId | parameters, sessionTag optional; a sessionTag shares one Spark session across activities |
ExecutePipeline | pipeline (PipelineReference) | waitOnCompletion defaults true |
InvokePipeline | — | Deprecated. workspaceId + pipelineId; waitOnCompletion defaults false |
SqlServerStoredProcedure | storedProcedureName | database, storedProcedureParameters optional |
PBISemanticModelRefresh | — | groupId + datasetId; objects scopes to partitions; commitMode, maxParallelism |
InvokeCopyJob | — | Runs a Copy Job item |
RefreshDataFlow | — | See the CI/CD trap in §8 |
InvokePipeline is deprecated in favour of ExecutePipeline, and
migrating is a rebinding, not a rename. Three things change together:
InvokePipelinetargets a child by GUID pair (workspaceId+pipelineId), which is what makes it parameterizable from a Variable Library item reference.ExecutePipelinetargets bypipeline.referenceName— a name, with no workspace GUID.waitOnCompletionflips default: false onInvokePipeline, true onExecutePipeline. Set it explicitly either way; a fire-and-forget orchestrator silently becomes blocking, or the reverse.- Cross-workspace invocation is expressible in the GUID form and not in the name form.
So do not bulk-rewrite InvokePipeline activities. Set
waitOnCompletion explicitly and confirm the child resolves.
5. policy
"policy": { "timeout": "0.02:00:00", "retry": 1, "retryIntervalInSeconds": 60,
"secureInput": false, "secureOutput": false }timeout is D.HH:MM:SS. Two different defaults apply and both are
correct. The portal stamps an explicit "0.12:00:00" (12 hours) on
every activity it creates. Omit policy.timeout entirely — as
hand-written or generated JSON does — and the schema default is 7
days. Omitting the field is not "taking the default"; it is a 14x
increase over what the portal writes. Set it explicitly.
retry— portal default 1, schema default 0, range 1–1000.retryIntervalInSeconds— default 30, min 30, max 86400.secureInput/secureOutput— keep values out of monitoring logs. Set on any activity handling secrets or connection strings.
retryConditions is preview and has a trap. It gates retries on
error message, failure type or error code, so a retry is not spent on an
error that will never resolve:
"retryConditions": { "type": "Expression",
"value": "@equals(activity('Ingest').Error.FailureType, 'SystemError')" }- Supported on Copy, Notebook, Dataflow and Stored procedure activities only. On any other type it is silently inert.
- The retry interval elapses before the condition is evaluated. A 1-hour interval with a condition that does not match still costs the full hour before the run moves on. Keep intervals short where a condition is in play.
- With no
retryConditions, the activity retries on every failure.
6. Deactivate rather than delete
Fabric has no separate debug mode — deactivation is the debugging mechanism, and Microsoft recommends it over deletion for preserving structure.
"state": "Inactive", "onInactiveMarkAs": "Succeeded"onInactiveMarkAs | Branch that runs |
Succeeded (default) | UponSuccess, UponCompletion |
Failed | UponFailure, UponCompletion |
Skipped | UponSkip |
The dependency edge is honoured — the value above chooses which
branch fires. What does not exist is the activity's output: an inactive
activity never runs, so it has no Error field and no output
fields, and any downstream expression reading
activity('Skipped').output… fails. Deactivating one activity can break
an expression two steps away.
An inactive activity is also excluded from validation, which is what makes it usable as a placeholder before its connections exist.
Spelling: real files use "state": "Inactive". The REST schema page
writes InActive in its ActivityState table; that is a doc typo —
Inactive is what the portal emits and reads (checked against live
exports, 2026-09-02).
7. .schedules
Creating a schedule in the portal writes this file into the item definition, so schedules deploy with the pipeline.
{ "$schema": "https://developer.microsoft.com/json-schemas/fabric/gitIntegration/schedules/1.0.0/schema.json",
"schedules": [ { "enabled": true, "jobType": "Execute",
"configuration": { "type": "Weekly",
"startDateTime": "2026-08-17T00:00:00",
"endDateTime": "9999-12-31T23:59:00",
"localTimeZoneId": "Eastern Standard Time",
"times": [ "06:00" ],
"weekdays": [ "Monday", "Tuesday" ] } } ] }jobType is "Execute" for a pipeline. This is a different enum from
the run-on-demand REST API, where a pipeline is jobType: Pipeline
(see fabric-rest-api). Do not copy one into the other.
configuration.type is Cron, Daily, Weekly or Monthly:
| Type | Fields |
Cron | interval — minutes, not a cron expression |
Daily | times[] (hh:mm, ≤ 100) |
Weekly | times[] + weekdays[] |
Monthly | times[], recurrence, occurrence{occurrenceType, dayOfMonth | weekIndex + WeekDay} |
Four traps:
endDateTimeis mandatory and there is no open-ended schedule. A schedule silently stops firing at its end date with no error and no alert. Use a far-future sentinel —9999-12-31T23:59:00. Treat any end date within a few years as a bug.Cronmeans a fixed interval in minutes. It accepts no crontab expression. Anything calendar-shaped needsDaily/Weekly/Monthly.- The scheduler auto-disables after consecutive failures (threshold varies by item, typically 10) and must be restarted by hand. A pipeline that "stopped running" after a bad week is this, not a schedule that was deleted.
- First commit after enabling CI/CD is noisy. Every item that already had a schedule shows as uncommitted. Review rather than bulk-committing.
Also: max 20 schedules per pipeline; enabled: false is a real
state that survives deployment, so a disabled schedule promotes as
disabled.
8. CI/CD limitations specific to pipelines
Parameterisation and parameter.yml are fabric-deploy's. These three
are pipeline-shaped and bite after promotion:
- A promoted pipeline that invokes a dataflow still points at the source workspace's dataflow. The reference is not rebound: deployment pipelines list Dataflow Gen2 as supported but do not autobind it. Learn's pipeline CI/CD page still gives the older reason, that dataflows are unsupported (2026-10-09). Verify after every promotion.
- Teams and Outlook activities need a manual sign-in per activity in each target workspace — OAuth connections do not deploy.
- Workspace variables are unsupported by CI/CD.
9. Limits
| Limit | Value |
| Activities per pipeline | 120, inner container activities included |
| Schedules per pipeline | 20 |
retry | 1–1000 |
retryIntervalInSeconds | 30–86400 |
timeout | max 7 days |
times[] per schedule | 100 |
Reference
- DataPipeline definition — the authoritative schema
- Activity overview — general settings, deactivation, the 120 cap
- Activity retries — retry conditions and interval types
- Full activity-type enum, per-activity
typePropertiestables, the complete.schedulesschema and worked examples of all four configuration shapes: references/REFERENCE.md
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

