eventstream-authoring-cli
Skill3 starsChanged 4 months agoMITView on GitHub
Create, wire, and publish Fabric Eventstream real-time streaming topologies via the Items REST API. Build definitions with 25 source types (Event Hubs, IoT Hub, CDC, Kafka, SampleData), 8 operators (Filter, Aggregate, GroupBy, Join, ManageFields, Union, Expand, SQL), 4 destinations (Lakehouse, Eventhouse, Activator, Custom Endpoint), DefaultStream/DerivedStream routing. Use to: (1) author Eventstream topology, (2) add Event Hub source, (3) add filter operator, (4) add CDC source with Debezium flattening, (5) wire destinations, (6) modify/delete Eventstream definitions. Triggers: "create eventstream", "deploy eventstream", "eventstream topology", "add source to eventstream", "add event hub source", "add filter operator", "eventstream filter", "eventstream destination", "CDC source", "eventstream operator", "eventstream definition", "update eventstream", "wire eventstream", "real-time ingestion pipeline", "eventstream topology deployment".
1 other project uses a version of this, 1 of them edited.
Adopted by
Most recently taken up by
- kimtth/ms-fabric-skills-dev-starteredited, first seen May 18, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 1
- kimtth/ms-fabric-skills-dev-starter★ 288% same
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository36 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- activator-authoring-cliSkill
- activator-consumption-cliSkill
- check-updatesSkill
- databricks-migrationSkill
- dataflows-authoring-cliSkill
- dataflows-consumption-cliSkill
- dataflows-save-as-authoring-cliSkill
- e2e-medallion-architectureSkill
- eventhouse-authoring-cliSkill
- eventhouse-consumption-cliSkill
- eventstream-consumption-cliSkill
- fabriciq-ontology-authoring-cliSkill
- fabriciq-ontology-consumption-cliSkill
- fabriciqSkill
- hdinsight-migrationSkill
- mlv-operations-cliSkill
- pipeline-migrationSkill
- powerbi-report-authoringSkill
- powerbi-report-designSkill
- powerbi-report-managementSkill
- powerbi-report-planningSkill
- search-consumption-cliSkill
- semantic-model-authoringSkill
- semantic-model-consumptionSkill
- spark-authoring-cliSkill
- spark-consumption-cliSkill
- spark-operations-cliSkill
- sqldb-authoring-cliSkill
- sqldb-consumption-cliSkill
- sqldb-operations-cliSkill
- sqldw-authoring-cliSkill
- sqldw-consumption-cliSkill
- sqldw-operations-cliSkill
- synapse-migrationSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 8 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 65
Create an empty Eventstream item, then configure it with sources, destinations, and operators via the definition API.
- Distinctive: line 83
Prefer building the JSON programmatically to avoid serialization errors.
- Very specific: line 217
Use SampleData source type for testing and prototyping
- Distinctive: line 220
Use DerivedStreams to make operator output available in Real-Time Hub
- Distinctive: line 224
Do NOT use raw JSON in the definition payload — it must be base64-encoded
- Distinctive: line 225
Do NOT use underscores or dots in Eventstream display names (breaks SQL operator)
- Distinctive: line 226
Do NOT exceed 11 combined CustomEndpoint sources and CustomEndpoint/Eventhouse-direct-ingestion destinations
- Distinctive: line 227
Do NOT confuse Eventstream with Eventhouse — they are separate Fabric workloads
- Distinctive: line 228
Do NOT hardcode workspace or item IDs — always discover them via the API
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.
- GitHub Copilot CLI / VS Code: invoke the
check-updatesskill.- Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
- Skip if the check was already performed earlier in this session.
CRITICAL NOTES
- To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
- To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
- Eventstream ≠ Eventhouse. Eventstream is a real-time event ingestion and routing pipeline. For KQL database operations, use
eventhouse-authoring-clioreventhouse-consumption-cli.
Eventstream Authoring — CLI Skill
Table of Contents
| Task | Reference | Notes |
| Finding Workspaces and Items in Fabric | COMMON-CLI.md § Finding Workspaces and Items in Fabric | Mandatory — READ link first [needed for finding workspace id by its name or item id by its name, item type, and workspace id] |
| Fabric Topology & Key Concepts | COMMON-CORE.md § Fabric Topology & Key Concepts | |
| Environment URLs | COMMON-CORE.md § Environment URLs | |
| Authentication & Token Acquisition | COMMON-CORE.md § Authentication & Token Acquisition | Wrong audience = 401; read before any auth issue |
| Core Control-Plane REST APIs | COMMON-CORE.md § Core Control-Plane REST APIs | Includes pagination, LRO polling, and rate-limiting patterns |
| Gotchas, Best Practices & Troubleshooting | COMMON-CORE.md § Gotchas, Best Practices & Troubleshooting | |
| Tool Selection Rationale | COMMON-CLI.md § Tool Selection Rationale | |
| Authentication Recipes | COMMON-CLI.md § Authentication Recipes | az login flows and token acquisition |
Fabric Control-Plane API via az rest | COMMON-CLI.md § Fabric Control-Plane API via az rest | Always pass --resource; includes pagination and LRO helpers |
| Gotchas & Troubleshooting (CLI-Specific) | COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific) | az rest audience, shell escaping, token expiry |
| Quick Reference | COMMON-CLI.md § Quick Reference | az rest template + token audience/tool matrix |
| Eventstream Resource Model | EVENTSTREAM-AUTHORING-CORE.md § Eventstream Resource Model | Read first — graph-based topology with sources, operators, streams, destinations |
| Source Configuration | EVENTSTREAM-AUTHORING-CORE.md § Source Configuration | 25 API-supported source types with per-source properties |
| Transformation Operators | EVENTSTREAM-AUTHORING-CORE.md § Transformation Operators | 8 operator types: Filter, Aggregate, GroupBy, Join, ManageFields, Union, Expand, SQL |
| Destination Configuration | EVENTSTREAM-AUTHORING-CORE.md § Destination Configuration | 4 API-supported destination types with node schema |
| Stream Types | EVENTSTREAM-AUTHORING-CORE.md § Stream Types | DefaultStream (auto) and DerivedStream (from operators) |
| Eventstream Lifecycle (REST API) | EVENTSTREAM-AUTHORING-CORE.md § Eventstream Lifecycle (REST API) | CRUD + Definition endpoints |
| Item Definitions and Deployment | EVENTSTREAM-AUTHORING-CORE.md § Item Definitions and Deployment | Base64 encoding pattern for eventstream.json |
| Gotchas and Limitations | EVENTSTREAM-AUTHORING-CORE.md § Gotchas and Limitations | Max 11 custom endpoints, base64 encoding, naming constraints |
| Create an Eventstream | SKILL.md § Create an Eventstream | |
| Deploy Full Topology | SKILL.md § Deploy Full Topology | End-to-end: build topology JSON → base64 encode → submit definition |
| Update Eventstream Topology | SKILL.md § Update Eventstream Topology | |
| Delete an Eventstream | SKILL.md § Delete an Eventstream | |
| Gotchas, Rules, Troubleshooting | SKILL.md § Gotchas, Rules, Troubleshooting | MUST DO / AVOID / PREFER checklists |
Create an Eventstream
Create an empty Eventstream item, then configure it with sources, destinations, and operators via the definition API.
Step 1: Create the Item
az rest --method POST \
--url "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/eventstreams" \
--resource "https://api.fabric.microsoft.com" \
--headers "Content-Type=application/json" \
--body '{"displayName": "my-eventstream", "description": "IoT sensor pipeline"}'Save the returned id as EVENTSTREAM_ID.
Step 2: Build the Topology
Construct the eventstream.json topology with sources, streams, operators, and destinations. Each node references its upstream via inputNodes.
Prefer building the JSON programmatically to avoid serialization errors. Key rules:
- The topology must have exactly one DefaultStream — all sources feed into it via
inputNodes - Operators reference their input via
inputNodes[].name - DerivedStreams require
inputSerializationin properties - Destinations reference their input stream or operator
Step 3: Deploy the Definition
Base64-encode the topology JSON and submit via the definition API. See Item Definitions and Deployment for the full payload structure.
Deploy Full Topology
For deploying a complete Eventstream with topology in a single API call, use the Create Item with Definition endpoint:
# 1. Build eventstream.json content (topology)
TOPOLOGY_JSON='{"compatibilityLevel":"1.1","sources":[...],"streams":[...],"operators":[...],"destinations":[...]}'
# 2. Build eventstreamProperties.json (optional — controls retention and throughput)
PROPERTIES_JSON='{"retentionTimeInDays":1,"eventThroughputLevel":"Low"}'
# 3. Base64-encode both (no line wraps)
TOPOLOGY_B64=$(echo -n "$TOPOLOGY_JSON" | base64 -w 0)
PROPERTIES_B64=$(echo -n "$PROPERTIES_JSON" | base64 -w 0)
# 4. Submit via Items API
az rest --method POST \
--url "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/items" \
--resource "https://api.fabric.microsoft.com" \
--headers "Content-Type=application/json" \
--body "{
\"displayName\": \"my-eventstream\",
\"type\": \"Eventstream\",
\"definition\": {
\"parts\": [
{
\"path\": \"eventstream.json\",
\"payload\": \"${TOPOLOGY_B64}\",
\"payloadType\": \"InlineBase64\"
},
{
\"path\": \"eventstreamProperties.json\",
\"payload\": \"${PROPERTIES_B64}\",
\"payloadType\": \"InlineBase64\"
}
]
}
}"Note: If
eventstreamProperties.jsonis omitted, the API applies defaults:retentionTimeInDays: 1,eventThroughputLevel: "Low". Include it explicitly to control retention (1–90 days) and throughput.
On Windows (PowerShell), use
[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($json))for base64 encoding.
Update Eventstream Topology
- Get current definition:
POST /v1/workspaces/{wsId}/eventstreams/{esId}/getDefinition - Decode the
eventstream.jsonpayload from base64 - Modify the topology (add/remove/update nodes)
- Re-encode to base64
- Submit:
POST /v1/workspaces/{wsId}/eventstreams/{esId}/updateDefinition
API Note: The Eventstream Definition APIs use
POSTwith action verbs (getDefinition,updateDefinition), notGET/PUTon a/definitionresource. This follows the Fabric Items Definition pattern. See official docs.
The Update Definition API returns 202 Accepted for long-running operations. Poll the Location header URL until completion.
Adding a Filter Operator
⚠️ CRITICAL: Filter operator conditions use nested objects for
columnandvalue— NOT bare strings. Using"column": "temperature"instead of the object form below will cause a silent API rejection.
{
"name": "FilterHighTemp",
"type": "Filter",
"inputNodes": [{"name": "my-stream"}],
"properties": {
"conditions": [{
"column": {
"node": null,
"columnName": "temperature",
"columnPath": null,
"expressionType": "ColumnReference"
},
"operatorType": "GreaterThan",
"value": {
"dataType": "Float",
"value": "30.0",
"expressionType": "Literal"
}
}]
}
}Required structure for ALL operator condition fields:
column→ object with{node, columnName, columnPath, expressionType: "ColumnReference"}value→ object with{dataType, value, expressionType: "Literal"}operatorType→ string:GreaterThan,LessThan,Equals,NotEquals,GreaterThanOrEqual,LessThanOrEqualdataType→Float,Int,Long,String,DateTime
This same nested-object pattern applies to all operators that reference columns (Filter, Aggregate, GroupBy, Join, ManageFields).
Delete an Eventstream
az rest --method DELETE \
--url "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/eventstreams/${EVENTSTREAM_ID}" \
--resource "https://api.fabric.microsoft.com"Returns 200 OK on success.
Gotchas, Rules, Troubleshooting
MUST DO
- Always base64-encode the
eventstream.jsonpayload before submitting definitions - Always pass
--resource https://api.fabric.microsoft.comwithaz restcalls - Always use JMESPath filtering to resolve workspace name → ID and item name → ID
- Always use nested objects for operator column/value references —
"column": {"columnName": "x", "expressionType": "ColumnReference", ...}, never"column": "x"(API rejects bare strings silently) - Exactly one DefaultStream per topology — all sources connect to it (the API rejects multiple DefaultStreams)
- Poll LRO responses — Update Definition returns
202 Acceptedwith aLocationheader
PREFER
- Build topology JSON programmatically rather than manual string construction
- Use
SampleDatasource type for testing and prototyping - Set
retentionTimeInDaysexplicitly rather than relying on defaults - Validate cloud connections before referencing them in source configurations
- Use DerivedStreams to make operator output available in Real-Time Hub
AVOID
- Do NOT use raw JSON in the definition payload — it must be base64-encoded
- Do NOT use underscores or dots in Eventstream display names (breaks SQL operator)
- Do NOT exceed 11 combined CustomEndpoint sources and CustomEndpoint/Eventhouse-direct-ingestion destinations
- Do NOT confuse Eventstream with Eventhouse — they are separate Fabric workloads
- Do NOT hardcode workspace or item IDs — always discover them via the API
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

