agentleFS
alifadl009alifadl009/microsoft-fabric-claude-skill/skills/eventstream-authoring-cli/SKILL.md

eventstream-authoring-cli

Skill3 starsChanged 4 months agoMITView on GitHub

Create, wire, and publish Fabric Eventstream real-time streaming topologies via the Items REST API. Build definitions with 25 source types (Event Hubs, IoT Hub, CDC, Kafka, SampleData), 8 operators (Filter, Aggregate, GroupBy, Join, ManageFields, Union, Expand, SQL), 4 destinations (Lakehouse, Eventhouse, Activator, Custom Endpoint), DefaultStream/DerivedStream routing. Use to: (1) author Eventstream topology, (2) add Event Hub source, (3) add filter operator, (4) add CDC source with Debezium flattening, (5) wire destinations, (6) modify/delete Eventstream definitions. Triggers: "create eventstream", "deploy eventstream", "eventstream topology", "add source to eventstream", "add event hub source", "add filter operator", "eventstream filter", "eventstream destination", "CDC source", "eventstream operator", "eventstream definition", "update eventstream", "wire eventstream", "real-time ingestion pipeline", "eventstream topology deployment".

1 other project uses a version of this, 1 of them edited.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 8 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Create an empty Eventstream item, then configure it with sources, destinations, and operators via the definition API.

    line 65
  2. Distinctive:

    Prefer building the JSON programmatically to avoid serialization errors.

    line 83
  3. Very specific:

    Use SampleData source type for testing and prototyping

    line 217
  4. Distinctive:

    Use DerivedStreams to make operator output available in Real-Time Hub

    line 220
  5. Distinctive:

    Do NOT use raw JSON in the definition payload — it must be base64-encoded

    line 224
  6. Distinctive:

    Do NOT use underscores or dots in Eventstream display names (breaks SQL operator)

    line 225
  7. Distinctive:

    Do NOT exceed 11 combined CustomEndpoint sources and CustomEndpoint/Eventhouse-direct-ingestion destinations

    line 226
  8. Distinctive:

    Do NOT confuse Eventstream with Eventhouse — they are separate Fabric workloads

    line 227
  9. Distinctive:

    Do NOT hardcode workspace or item IDs — always discover them via the API

    line 228

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.

  • GitHub Copilot CLI / VS Code: invoke the check-updates skill.
  • Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
  • Skip if the check was already performed earlier in this session.

CRITICAL NOTES

  1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
  2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
  3. Eventstream ≠ Eventhouse. Eventstream is a real-time event ingestion and routing pipeline. For KQL database operations, use eventhouse-authoring-cli or eventhouse-consumption-cli.

Eventstream Authoring — CLI Skill

Table of Contents

TaskReferenceNotes
Finding Workspaces and Items in FabricCOMMON-CLI.md § Finding Workspaces and Items in FabricMandatory — READ link first [needed for finding workspace id by its name or item id by its name, item type, and workspace id]
Fabric Topology & Key ConceptsCOMMON-CORE.md § Fabric Topology & Key Concepts
Environment URLsCOMMON-CORE.md § Environment URLs
Authentication & Token AcquisitionCOMMON-CORE.md § Authentication & Token AcquisitionWrong audience = 401; read before any auth issue
Core Control-Plane REST APIsCOMMON-CORE.md § Core Control-Plane REST APIsIncludes pagination, LRO polling, and rate-limiting patterns
Gotchas, Best Practices & TroubleshootingCOMMON-CORE.md § Gotchas, Best Practices & Troubleshooting
Tool Selection RationaleCOMMON-CLI.md § Tool Selection Rationale
Authentication RecipesCOMMON-CLI.md § Authentication Recipesaz login flows and token acquisition
Fabric Control-Plane API via az restCOMMON-CLI.md § Fabric Control-Plane API via az restAlways pass --resource; includes pagination and LRO helpers
Gotchas & Troubleshooting (CLI-Specific)COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific)az rest audience, shell escaping, token expiry
Quick ReferenceCOMMON-CLI.md § Quick Referenceaz rest template + token audience/tool matrix
Eventstream Resource ModelEVENTSTREAM-AUTHORING-CORE.md § Eventstream Resource ModelRead first — graph-based topology with sources, operators, streams, destinations
Source ConfigurationEVENTSTREAM-AUTHORING-CORE.md § Source Configuration25 API-supported source types with per-source properties
Transformation OperatorsEVENTSTREAM-AUTHORING-CORE.md § Transformation Operators8 operator types: Filter, Aggregate, GroupBy, Join, ManageFields, Union, Expand, SQL
Destination ConfigurationEVENTSTREAM-AUTHORING-CORE.md § Destination Configuration4 API-supported destination types with node schema
Stream TypesEVENTSTREAM-AUTHORING-CORE.md § Stream TypesDefaultStream (auto) and DerivedStream (from operators)
Eventstream Lifecycle (REST API)EVENTSTREAM-AUTHORING-CORE.md § Eventstream Lifecycle (REST API)CRUD + Definition endpoints
Item Definitions and DeploymentEVENTSTREAM-AUTHORING-CORE.md § Item Definitions and DeploymentBase64 encoding pattern for eventstream.json
Gotchas and LimitationsEVENTSTREAM-AUTHORING-CORE.md § Gotchas and LimitationsMax 11 custom endpoints, base64 encoding, naming constraints
Create an EventstreamSKILL.md § Create an Eventstream
Deploy Full TopologySKILL.md § Deploy Full TopologyEnd-to-end: build topology JSON → base64 encode → submit definition
Update Eventstream TopologySKILL.md § Update Eventstream Topology
Delete an EventstreamSKILL.md § Delete an Eventstream
Gotchas, Rules, TroubleshootingSKILL.md § Gotchas, Rules, TroubleshootingMUST DO / AVOID / PREFER checklists

Create an Eventstream

Create an empty Eventstream item, then configure it with sources, destinations, and operators via the definition API.

Step 1: Create the Item

az rest --method POST \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/eventstreams" \
  --resource "https://api.fabric.microsoft.com" \
  --headers "Content-Type=application/json" \
  --body '{"displayName": "my-eventstream", "description": "IoT sensor pipeline"}'

Save the returned id as EVENTSTREAM_ID.

Step 2: Build the Topology

Construct the eventstream.json topology with sources, streams, operators, and destinations. Each node references its upstream via inputNodes.

Prefer building the JSON programmatically to avoid serialization errors. Key rules:

  • The topology must have exactly one DefaultStream — all sources feed into it via inputNodes
  • Operators reference their input via inputNodes[].name
  • DerivedStreams require inputSerialization in properties
  • Destinations reference their input stream or operator

Step 3: Deploy the Definition

Base64-encode the topology JSON and submit via the definition API. See Item Definitions and Deployment for the full payload structure.


Deploy Full Topology

For deploying a complete Eventstream with topology in a single API call, use the Create Item with Definition endpoint:

# 1. Build eventstream.json content (topology)
TOPOLOGY_JSON='{"compatibilityLevel":"1.1","sources":[...],"streams":[...],"operators":[...],"destinations":[...]}'

# 2. Build eventstreamProperties.json (optional — controls retention and throughput)
PROPERTIES_JSON='{"retentionTimeInDays":1,"eventThroughputLevel":"Low"}'

# 3. Base64-encode both (no line wraps)
TOPOLOGY_B64=$(echo -n "$TOPOLOGY_JSON" | base64 -w 0)
PROPERTIES_B64=$(echo -n "$PROPERTIES_JSON" | base64 -w 0)

# 4. Submit via Items API
az rest --method POST \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/items" \
  --resource "https://api.fabric.microsoft.com" \
  --headers "Content-Type=application/json" \
  --body "{
    \"displayName\": \"my-eventstream\",
    \"type\": \"Eventstream\",
    \"definition\": {
      \"parts\": [
        {
          \"path\": \"eventstream.json\",
          \"payload\": \"${TOPOLOGY_B64}\",
          \"payloadType\": \"InlineBase64\"
        },
        {
          \"path\": \"eventstreamProperties.json\",
          \"payload\": \"${PROPERTIES_B64}\",
          \"payloadType\": \"InlineBase64\"
        }
      ]
    }
  }"

Note: If eventstreamProperties.json is omitted, the API applies defaults: retentionTimeInDays: 1, eventThroughputLevel: "Low". Include it explicitly to control retention (1–90 days) and throughput.

On Windows (PowerShell), use [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($json)) for base64 encoding.


Update Eventstream Topology

  1. Get current definition: POST /v1/workspaces/{wsId}/eventstreams/{esId}/getDefinition
  2. Decode the eventstream.json payload from base64
  3. Modify the topology (add/remove/update nodes)
  4. Re-encode to base64
  5. Submit: POST /v1/workspaces/{wsId}/eventstreams/{esId}/updateDefinition

API Note: The Eventstream Definition APIs use POST with action verbs (getDefinition, updateDefinition), not GET/PUT on a /definition resource. This follows the Fabric Items Definition pattern. See official docs.

The Update Definition API returns 202 Accepted for long-running operations. Poll the Location header URL until completion.

Adding a Filter Operator

⚠️ CRITICAL: Filter operator conditions use nested objects for column and value — NOT bare strings. Using "column": "temperature" instead of the object form below will cause a silent API rejection.

{
  "name": "FilterHighTemp",
  "type": "Filter",
  "inputNodes": [{"name": "my-stream"}],
  "properties": {
    "conditions": [{
      "column": {
        "node": null,
        "columnName": "temperature",
        "columnPath": null,
        "expressionType": "ColumnReference"
      },
      "operatorType": "GreaterThan",
      "value": {
        "dataType": "Float",
        "value": "30.0",
        "expressionType": "Literal"
      }
    }]
  }
}

Required structure for ALL operator condition fields:

  • column → object with {node, columnName, columnPath, expressionType: "ColumnReference"}
  • value → object with {dataType, value, expressionType: "Literal"}
  • operatorType → string: GreaterThan, LessThan, Equals, NotEquals, GreaterThanOrEqual, LessThanOrEqual
  • dataType → Float, Int, Long, String, DateTime

This same nested-object pattern applies to all operators that reference columns (Filter, Aggregate, GroupBy, Join, ManageFields).


Delete an Eventstream

az rest --method DELETE \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/eventstreams/${EVENTSTREAM_ID}" \
  --resource "https://api.fabric.microsoft.com"

Returns 200 OK on success.


Gotchas, Rules, Troubleshooting

MUST DO

  • Always base64-encode the eventstream.json payload before submitting definitions
  • Always pass --resource https://api.fabric.microsoft.com with az rest calls
  • Always use JMESPath filtering to resolve workspace name → ID and item name → ID
  • Always use nested objects for operator column/value references — "column": {"columnName": "x", "expressionType": "ColumnReference", ...}, never "column": "x" (API rejects bare strings silently)
  • Exactly one DefaultStream per topology — all sources connect to it (the API rejects multiple DefaultStreams)
  • Poll LRO responses — Update Definition returns 202 Accepted with a Location header

PREFER

  • Build topology JSON programmatically rather than manual string construction
  • Use SampleData source type for testing and prototyping
  • Set retentionTimeInDays explicitly rather than relying on defaults
  • Validate cloud connections before referencing them in source configurations
  • Use DerivedStreams to make operator output available in Real-Time Hub

AVOID

  • Do NOT use raw JSON in the definition payload — it must be base64-encoded
  • Do NOT use underscores or dots in Eventstream display names (breaks SQL operator)
  • Do NOT exceed 11 combined CustomEndpoint sources and CustomEndpoint/Eventhouse-direct-ingestion destinations
  • Do NOT confuse Eventstream with Eventhouse — they are separate Fabric workloads
  • Do NOT hardcode workspace or item IDs — always discover them via the API

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.