agentleFS
wardawgmalviciouswardawgmalvicious/agent-config/skills/fabric-realtime/fabric-eventstream/SKILL.md

fabric-eventstream

Skill1 starsChanged 34 days agoMITView on GitHub

Use for Microsoft Fabric Eventstream — the streaming-ingestion item routing CDC / Event Hubs / Kafka / IoT / HTTP / MQTT events into Lakehouse, Eventhouse, Activator, or derived streams, and producing events to a schema-associated custom endpoint. Covers source connectors (Azure SQL / SQL MI / PostgreSQL / MySQL / MongoDB / Cosmos DB CDC, Mirrored DB Delta CDF preview, Event Hubs / IoT Hub / Kafka / MSK / Confluent / Kinesis / Service Bus / MQTT / HTTP / Solace), DeltaFlow analytics-ready CDC, Activator destination + `Set Alert` flow, workspace-monitoring KQL tables (`EventStream{NodeStatus,Metrics,ErrorMetrics,DiagnosticLogs}`), mTLS Key Vault on Kafka, Event Hubs workspace-identity auth, custom-endpoint CloudEvents producer format (binary mode, `dataschema` version routing), custom-endpoint connection anatomy (eseh* namespace, EntityPath, SAS policy), schema-registry URL anatomy, and gotchas (republish required, ~6h status lag, filter by ID not name, CloudEventPropertyMissingException).

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Check Learn's per-node table before picking a startType.

    line 133
  2. Distinctive:

    CI/CD does not carry pause state: After a Git sync or a deployment-pipeline deploy, every node in the target eventstream comes back active …

    line 136
  3. Distinctive:

    Only relevant when pushing events *into* a custom endpoint that has an associated schema.

    line 157

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Fabric Eventstream

Streaming-data ingestion item that pulls events from a wide source surface (CDC / Event Hubs / Kafka / IoT / HTTP / MQTT) and routes them into Fabric destinations (Lakehouse, Eventhouse, Activator, derived stream, custom endpoint). Authoring is graph-based: source nodes → optional transformations → destination nodes, edited then published to go live.

When to use vs not

Use Eventstream when the data is arriving as events and needs routing or transformation before it lands. Skip it when the data is bulk / batch (use a Data Pipeline Copy activity), already in the lake (use Spark / SQL directly), or when the only consumer is a Mirrored Database in append-only mode (mirroring lands data straight in OneLake without an Eventstream).

For real-time analytics on the resulting events, pair an Eventstream with fabric-eventhouse (KQL Database). For real-time rules, pair with an Activator destination (covered below).

Authoring model

  • Edit mode vs Live mode: changes only take effect after Publish. New nodes added in Edit mode produce no traffic until publish.
  • Sources = where events come from. Transformations = inline filter / aggregate / GroupBy / Manage Fields / SQL, plus a join against a reference data source (preview): Add source → Reference data sources, a Lakehouse Delta table that adds context to each event. Destinations = where events go. Each destination can have its own format (Delta / JSON / Avro) where applicable.
  • Permissions: workspace Contributor or higher to author; Viewer can read Data insights monitoring on a published stream.
  • Virtual-network injection (private-network sources): use Eventstream connector VNet injection for sources behind a firewall — see Microsoft Learn.

Sources

Grouped by what you have to get right, not by vendor. Full connector table with per-source notes: references/source-connectors.md.

FamilyConnectorsThe thing that bites
Database CDCAzure SQL, SQL MI, SQL Server on VM, PostgreSQL, MySQL, MongoDB, Cosmos DBAzure SQL needs sys.sp_cdc_enable_db; you cannot enable Mirroring and Eventstream CDC on the same database; MongoDB (GA September 2026) has no Git integration or deployment-pipeline support
Mirrored DB Delta CDFMirrored Database (preview, April 2026)Row-level CDC off a Mirrored DB's Delta Change Data Feed — opt in per database. What's New calls it GA (September 2026), but Learn's source list still says preview (2026-10-06), so it stays preview here
Native AzureEvent Hubs, IoT HubAuth is Shared Access Key; workspace-identity auth is preview (Aug 2026) and the UI leads the docs
Kafka protocolApache Kafka, Amazon MSK, Confluent CloudGA June 2026. Custom CA / mTLS GA July 2026, under TLS/mTLS settings — MQTT has the same block
Other cloud / brokerKinesis, Service Bus, Google Pub/Sub, Solace PubSub+, Cribl, SAP DatasphereService Bus GA June 2026; Solace PubSub+, Cribl and SAP Datasphere GA September 2026. Cribl has no Git integration or deployment-pipeline support
Protocol / pullMQTT, HTTP (preview), Azure Data Explorer, Real-time weatherMQTT is GA (August 2026) with its own TLS/mTLS settings, though Learn's source list still labels it preview (2026-10-06). HTTP ships predefined public feeds for testing
CustomCustom stream connector (preview)You upload a Kafka Connect source plugin; sources on private networks aren't supported

DeltaFlow (preview) is the schema-handling mode that turns raw Debezium CDC into a tabular shape mirroring the source table, plus an Op column, automatic destination-table creation, and schema evolution. Available on Azure SQL / SQL MI / SQL Server on VM / PostgreSQL CDC only; every other CDC source hands you raw Debezium envelopes to flatten yourself.

Sources behind a firewall need Eventstream connector VNet injection.

Destinations

DestinationUse when
LakehouseLand events as Delta files for batch analytics
Eventhouse / KQL DatabaseReal-time KQL queries; pair with fabric-eventhouse
ActivatorRule-based alerts and automation (see below)
Derived streamChain a downstream Eventstream — useful for fan-out and reusable transforms
Custom endpointPush to an external Event Hubs / Kafka / AMQP-compatible system

Activator destination

GA (August 2026). Add an Activator destination, then use the alert icon on it to view, add, edit, and stop/start rules without leaving Eventstream. Conditions fire on each event, on each event when (single-field condition), or on each event grouped by a field. Actions: Teams, email, webhook, Power Automate, custom. Republish after wiring the destination or no rule fires.

Pane walkthrough and condition detail: references/activator-destination.md.

Workspace monitoring (preview)

Workspace monitoring is managed through a monitoring item, created from Workspace settings → Monitoring; the old Log workspace activity toggle is now legacy. Eventstream logging is then opt-in per eventstream: turn on Log Eventstream activity on each one, because enabling monitoring for the workspace doesn't (Learn, 2026-10-06). Four tables land in the monitoring database. Republish any Eventstream that existed before monitoring was enabled — pre-existing streams emit nothing until republished.

TableCadenceWhat it tells you
EventStreamNodeStatus~6 hoursEach node's running / paused / failed state
EventStreamMetrics1 minuteIncoming / outgoing counts, bytes, watermark delay, backlog
EventStreamErrorMetrics1 minuteError counts by type (runtime, deserialization, conversion)
EventStreamDiagnosticLogsAs conditions occur; repeats throttled or aggregatedThe message behind an error: type, code, severity, whether it is fatal

Filter by the ID columns, never the name columns — names cache at emission time and go stale after a rename or move. Learn now names the eventstream's ID column ItemId (with ItemName and ItemKind), not ArtifactId; the samples in the reference predate that and are unmeasured against it (2026-10-06), so check the table's schema first. For live per-node numbers during authoring, the editor's Data insights tab needs no monitoring setup.

Worked KQL queries and the full shared-dimension list: references/monitoring.md.

Pause and resume

Pausing (Deactivate) stops data flowing through a supported node without touching its configuration; resuming (Activate) restarts it. In the portal that is Deactivate All / Activate All on the menu bar, or the per-node toggle. Over REST every call is a POST under https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/eventstreams/{eventstreamId}:

ScopePauseResume
Whole stream/pause/resume
One source/sources/{sourceId}/pause/sources/{sourceId}/resume
One destination/destinations/{destinationId}/pause/destinations/{destinationId}/resume

Pause takes no body. Resume requires one — omitting startType is the easy mistake:

{ "startType": "CustomTime", "customStartDateTime": "2026-09-11T08:00:00Z" }

startType is Now, WhenLastStopped, or CustomTime; the optional customStartDateTime is UTC, YYYY-MM-DDTHH:mm:ssZ. Learn marks the enum open — "additional start types may be added over time". Scope Eventstream.ReadWrite.All or Item.ReadWrite.All; user, service principal, and managed identity all work.

Not every node can pause, and resume options differ per node. The Custom endpoint (source and destination alike) has no toggle at all. An Eventhouse destination depends on its dataIngestionMode: DirectIngestion cannot pause, while ProcessedIngestion — event processing before ingestion — offers all three resume options. Most CDC and Kafka-family sources resume only from when streaming was last stopped. Check Learn's per-node table before picking a startType.

CI/CD does not carry pause state. After a Git sync or a deployment-pipeline deploy, every node in the target eventstream comes back active unless it fails on connection or configuration (Eventstream CI/CD). Re-pause after deploying if the stream should stay down.

Kafka custom CA / mTLS

Kafka, Amazon MSK, and Confluent Cloud sources can take a custom CA certificate and a client certificate from Azure Key Vault (GA July 2026), under the source connection's TLS/mTLS settings. SASL_SSL needs a CA cert only when the cluster uses a private CA; SSL (mTLS) needs both CA and client cert + key.

Certificates must be PEM with LF line endings, cert and private key concatenated, contentType: application/x-pem-file, with keySize matching the real key size — and whoever previews data needs Key Vault read access on them. Full setup, the trusted-CA list, and seven failure modes: references/kafka-mtls.md.

Producing to a schema-associated custom endpoint

Only relevant when pushing events into a custom endpoint that has an associated schema. Two facts decide whether it works at all:

  • CloudEvents binary mode is required. Attributes go in the Event Hub message's application properties as cloudEvents:*; the body is the payload JSON alone. Structured mode is silently dropped with CloudEventPropertyMissingException.
  • Schema support is a creation-time flag — it cannot be turned on for an existing eventstream, and schema-enabled eventstreams don't survive deployment pipelines with registries intact. Plan workspaces before you build.

The wire format, the dataschema URI anatomy, version-bump behaviour, table naming, and the custom-endpoint connection-string conventions are all in references/cloudevents-producer.md. Of these, only versioning is on Microsoft Learn, and only for a portal save, where an update mints a new version (read 2026-09-30); what a Git sync does is not. The rest was undocumented when verified.

Gotchas

IssueCauseFix
Existing Eventstream emits no monitoring dataStream was published before workspace monitoring was enabledOpen in editor and Republish — required once per pre-existing stream
Monitoring tables don't appear after enablingLog Eventstream activity is off: enabling workspace monitoring doesn't turn it onTurn it on per eventstream (§ Workspace monitoring); if it's on, allow for the database's refresh delay
ItemName / WorkspaceName show stale valuesName columns cached from emission timeFilter / join by ItemId / WorkspaceId only (ArtifactId in samples written before 2026-10-06)
EventStreamNodeStatus shows old status after a node failedStatus is emitted ~every 6 hoursFor real-time status, use the Eventstream editor's live view
CorrelationId maps to multiple nodesAdvanced processing (e.g. SQL operator with multiple destinations)Disambiguate using NodeDirection + NodeType together with CorrelationId
An error count with no message behind itEventStreamErrorMetrics holds counts onlyQuery EventStreamDiagnosticLogs: error type and code, severity, IsFatal and a message. Repeated events can be throttled or aggregated, so rows don't equal errors
Mirrored DB CDC source rejectedCan't enable Mirroring AND Eventstream CDC on same DBPick one — the docs explicitly call this out
New Activator rule doesn't fireEventstream wasn't republished after adding the destinationRepublish the Eventstream after wiring the destination
Connector behind firewall failsSource not publicly reachableUse Eventstream connector VNet injection
DeltaFlow not available on a CDC sourceCurrently scoped to Azure SQL / SQL MI / SQL Server VM / PostgreSQL CDCUse raw mode for other CDC sources and flatten Debezium yourself
Events pushed to a schema-associated custom endpoint are droppedWrong CloudEvents envelope, dataschema version, or per-environment registry hostSee references/cloudevents-producer.md — four distinct failure modes
Paused nodes are running again after a deployGit integration and deployment pipelines don't carry pause/resume state — after CI/CD every node in the target eventstream becomes activeRe-pause after the deploy (portal toggle or the REST /pause calls — see Pause and resume)
Creating or updating an Eventhouse destination on a schema-associated eventstream answers 422 "MessagingCatalogConfiguration is required …"A service-side fault, observed 2026-09-10 to 2026-09-14 in one tenant and not re-checked since: a fresh eventstream, KQL database and schema set failed alikeNone client-side. Published destinations keep routing, so don't republish or rebuild a working one. Re-test before relying on this row; reading the real state: references/cloudevents-producer.md
A new event type lands zero rows, .show ingestion failures is empty, and the older types still landThe destination registers types at publish, so one added to the schema set since is dropped before Kusto (observed 2026-09-14)Publish the destination after adding a type — see references/cloudevents-producer.md

Reference

Detail lives in references/ — each section above links the file that carries its full version.

See also

  • fabric-eventhouse — the natural KQL-Database pair for analytics on streamed events
  • fabric-rest-api — Eventstream item REST endpoints, LRO polling, jobType values; the token audience for Fabric REST against Eventstream items in its references/auth.md
  • fabric-warehouse-monitoring — Warehouse-side query monitoring; workspace-monitoring links in its references bundle

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.