fabric-eventstream
Skill1 starsChanged 34 days agoMITView on GitHub
Use for Microsoft Fabric Eventstream — the streaming-ingestion item routing CDC / Event Hubs / Kafka / IoT / HTTP / MQTT events into Lakehouse, Eventhouse, Activator, or derived streams, and producing events to a schema-associated custom endpoint. Covers source connectors (Azure SQL / SQL MI / PostgreSQL / MySQL / MongoDB / Cosmos DB CDC, Mirrored DB Delta CDF preview, Event Hubs / IoT Hub / Kafka / MSK / Confluent / Kinesis / Service Bus / MQTT / HTTP / Solace), DeltaFlow analytics-ready CDC, Activator destination + `Set Alert` flow, workspace-monitoring KQL tables (`EventStream{NodeStatus,Metrics,ErrorMetrics,DiagnosticLogs}`), mTLS Key Vault on Kafka, Event Hubs workspace-identity auth, custom-endpoint CloudEvents producer format (binary mode, `dataschema` version routing), custom-endpoint connection anatomy (eseh* namespace, EntityPath, SAS policy), schema-registry URL anatomy, and gotchas (republish required, ~6h status lag, filter by ID not name, CloudEventPropertyMissingException).
In this repository38 other files it gives its agents.
- claude/CLAUDE.mdCLAUDE.md
- CLAUDE.mdCLAUDE.md
- author-skillSkill
- drift-auditSkill
- drift-handoffSkill
- drift-updateSkill
- test-skillSkill
- triageSkill
- fabric-iqSkill
- fabric-catalog-governanceSkill
- fabric-deploySkill
- fabric-rest-apiSkill
- fabric-securitySkill
- fabric-variable-librarySkill
- fabric-databaseSkill
- fabric-dataflowSkill
- fabric-data-pipelineSkill
- fabric-sparkSkill
- fabric-warehouse-monitoringSkill
- fabric-warehouseSkill
- fabric-activatorSkill
- fabric-eventhouseSkill
- fabric-event-schema-setSkill
- fabric-realtime-dashboardSkill
- learnSkill
- fabric-semantic-model-auditSkill
- fabric-tmdlSkill
- pbip-project-structureSkill
- pbir-report-workflowSkill
- linkedin-highlightsSkill
- msix-packagingSkill
- code-reviewSkill
- commitSkill
- find-sessionSkill
- landSkill
- prune-branchesSkill
- recreate-repoSkill
- run-briefsSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 133
Check Learn's per-node table before picking a startType.
- Distinctive: line 136
CI/CD does not carry pause state: After a Git sync or a deployment-pipeline deploy, every node in the target eventstream comes back active …
- Distinctive: line 157
Only relevant when pushing events *into* a custom endpoint that has an associated schema.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Fabric Eventstream
Streaming-data ingestion item that pulls events from a wide source surface (CDC / Event Hubs / Kafka / IoT / HTTP / MQTT) and routes them into Fabric destinations (Lakehouse, Eventhouse, Activator, derived stream, custom endpoint). Authoring is graph-based: source nodes → optional transformations → destination nodes, edited then published to go live.
When to use vs not
Use Eventstream when the data is arriving as events and needs routing or transformation before it lands. Skip it when the data is bulk / batch (use a Data Pipeline Copy activity), already in the lake (use Spark / SQL directly), or when the only consumer is a Mirrored Database in append-only mode (mirroring lands data straight in OneLake without an Eventstream).
For real-time analytics on the resulting events, pair an Eventstream with fabric-eventhouse (KQL Database). For real-time rules, pair with an Activator destination (covered below).
Authoring model
- Edit mode vs Live mode: changes only take effect after Publish. New nodes added in Edit mode produce no traffic until publish.
- Sources = where events come from. Transformations = inline filter / aggregate / GroupBy / Manage Fields / SQL, plus a join against a reference data source (preview): Add source → Reference data sources, a Lakehouse Delta table that adds context to each event. Destinations = where events go. Each destination can have its own format (Delta / JSON / Avro) where applicable.
- Permissions: workspace Contributor or higher to author; Viewer can read Data insights monitoring on a published stream.
- Virtual-network injection (private-network sources): use Eventstream connector VNet injection for sources behind a firewall — see Microsoft Learn.
Sources
Grouped by what you have to get right, not by vendor. Full connector table with per-source notes: references/source-connectors.md.
| Family | Connectors | The thing that bites |
| Database CDC | Azure SQL, SQL MI, SQL Server on VM, PostgreSQL, MySQL, MongoDB, Cosmos DB | Azure SQL needs sys.sp_cdc_enable_db; you cannot enable Mirroring and Eventstream CDC on the same database; MongoDB (GA September 2026) has no Git integration or deployment-pipeline support |
| Mirrored DB Delta CDF | Mirrored Database (preview, April 2026) | Row-level CDC off a Mirrored DB's Delta Change Data Feed — opt in per database. What's New calls it GA (September 2026), but Learn's source list still says preview (2026-10-06), so it stays preview here |
| Native Azure | Event Hubs, IoT Hub | Auth is Shared Access Key; workspace-identity auth is preview (Aug 2026) and the UI leads the docs |
| Kafka protocol | Apache Kafka, Amazon MSK, Confluent Cloud | GA June 2026. Custom CA / mTLS GA July 2026, under TLS/mTLS settings — MQTT has the same block |
| Other cloud / broker | Kinesis, Service Bus, Google Pub/Sub, Solace PubSub+, Cribl, SAP Datasphere | Service Bus GA June 2026; Solace PubSub+, Cribl and SAP Datasphere GA September 2026. Cribl has no Git integration or deployment-pipeline support |
| Protocol / pull | MQTT, HTTP (preview), Azure Data Explorer, Real-time weather | MQTT is GA (August 2026) with its own TLS/mTLS settings, though Learn's source list still labels it preview (2026-10-06). HTTP ships predefined public feeds for testing |
| Custom | Custom stream connector (preview) | You upload a Kafka Connect source plugin; sources on private networks aren't supported |
DeltaFlow (preview) is the schema-handling mode that turns raw Debezium CDC
into a tabular shape mirroring the source table, plus an Op column, automatic
destination-table creation, and schema evolution. Available on Azure SQL / SQL
MI / SQL Server on VM / PostgreSQL CDC only; every other CDC source hands you
raw Debezium envelopes to flatten yourself.
Sources behind a firewall need Eventstream connector VNet injection.
Destinations
| Destination | Use when |
| Lakehouse | Land events as Delta files for batch analytics |
| Eventhouse / KQL Database | Real-time KQL queries; pair with fabric-eventhouse |
| Activator | Rule-based alerts and automation (see below) |
| Derived stream | Chain a downstream Eventstream — useful for fan-out and reusable transforms |
| Custom endpoint | Push to an external Event Hubs / Kafka / AMQP-compatible system |
Activator destination
GA (August 2026). Add an Activator destination, then use the alert icon on it to view, add, edit, and stop/start rules without leaving Eventstream. Conditions fire on each event, on each event when (single-field condition), or on each event grouped by a field. Actions: Teams, email, webhook, Power Automate, custom. Republish after wiring the destination or no rule fires.
Pane walkthrough and condition detail: references/activator-destination.md.
Workspace monitoring (preview)
Workspace monitoring is managed through a monitoring item, created from Workspace settings → Monitoring; the old Log workspace activity toggle is now legacy. Eventstream logging is then opt-in per eventstream: turn on Log Eventstream activity on each one, because enabling monitoring for the workspace doesn't (Learn, 2026-10-06). Four tables land in the monitoring database. Republish any Eventstream that existed before monitoring was enabled — pre-existing streams emit nothing until republished.
| Table | Cadence | What it tells you |
EventStreamNodeStatus | ~6 hours | Each node's running / paused / failed state |
EventStreamMetrics | 1 minute | Incoming / outgoing counts, bytes, watermark delay, backlog |
EventStreamErrorMetrics | 1 minute | Error counts by type (runtime, deserialization, conversion) |
EventStreamDiagnosticLogs | As conditions occur; repeats throttled or aggregated | The message behind an error: type, code, severity, whether it is fatal |
Filter by the ID columns, never the name columns — names cache at emission
time and go stale after a rename or move. Learn now names the eventstream's ID
column ItemId (with ItemName and ItemKind), not ArtifactId; the samples in
the reference predate that and are unmeasured against it (2026-10-06), so check
the table's schema first. For live per-node numbers during authoring, the
editor's Data insights tab needs no monitoring setup.
Worked KQL queries and the full shared-dimension list: references/monitoring.md.
Pause and resume
Pausing (Deactivate) stops data flowing through a supported node without
touching its configuration; resuming (Activate) restarts it. In the portal
that is Deactivate All / Activate All on the menu bar, or the per-node
toggle. Over REST every call is a POST under
https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/eventstreams/{eventstreamId}:
| Scope | Pause | Resume |
| Whole stream | /pause | /resume |
| One source | /sources/{sourceId}/pause | /sources/{sourceId}/resume |
| One destination | /destinations/{destinationId}/pause | /destinations/{destinationId}/resume |
Pause takes no body. Resume requires one — omitting startType is the
easy mistake:
{ "startType": "CustomTime", "customStartDateTime": "2026-09-11T08:00:00Z" }startType is Now, WhenLastStopped, or CustomTime; the optional
customStartDateTime is UTC, YYYY-MM-DDTHH:mm:ssZ. Learn marks the enum
open — "additional start types may be added over time". Scope
Eventstream.ReadWrite.All or Item.ReadWrite.All; user, service principal,
and managed identity all work.
Not every node can pause, and resume options differ per node. The Custom
endpoint (source and destination alike) has no toggle at all. An Eventhouse
destination depends on its dataIngestionMode: DirectIngestion cannot pause,
while ProcessedIngestion — event processing before ingestion — offers all
three resume options. Most
CDC and Kafka-family sources resume only from when streaming was last stopped.
Check Learn's per-node table
before picking a startType.
CI/CD does not carry pause state. After a Git sync or a deployment-pipeline deploy, every node in the target eventstream comes back active unless it fails on connection or configuration (Eventstream CI/CD). Re-pause after deploying if the stream should stay down.
Kafka custom CA / mTLS
Kafka, Amazon MSK, and Confluent Cloud sources can take a custom CA certificate
and a client certificate from Azure Key Vault (GA July 2026), under the
source connection's TLS/mTLS settings. SASL_SSL needs a CA cert only when
the cluster uses a private CA; SSL (mTLS) needs both CA and client cert + key.
Certificates must be PEM with LF line endings, cert and private key
concatenated, contentType: application/x-pem-file, with keySize matching the
real key size — and whoever previews data needs Key Vault read access on them.
Full setup, the trusted-CA list, and seven failure modes:
references/kafka-mtls.md.
Producing to a schema-associated custom endpoint
Only relevant when pushing events into a custom endpoint that has an associated schema. Two facts decide whether it works at all:
- CloudEvents binary mode is required. Attributes go in the Event Hub
message's application properties as
cloudEvents:*; the body is the payload JSON alone. Structured mode is silently dropped withCloudEventPropertyMissingException. - Schema support is a creation-time flag — it cannot be turned on for an existing eventstream, and schema-enabled eventstreams don't survive deployment pipelines with registries intact. Plan workspaces before you build.
The wire format, the dataschema URI anatomy, version-bump behaviour, table
naming, and the custom-endpoint connection-string conventions are all in
references/cloudevents-producer.md.
Of these, only versioning is on Microsoft Learn, and only for a portal
save, where
an update mints a new version
(read 2026-09-30); what a Git sync does is not. The rest was undocumented
when verified.
Gotchas
| Issue | Cause | Fix |
| Existing Eventstream emits no monitoring data | Stream was published before workspace monitoring was enabled | Open in editor and Republish — required once per pre-existing stream |
| Monitoring tables don't appear after enabling | Log Eventstream activity is off: enabling workspace monitoring doesn't turn it on | Turn it on per eventstream (§ Workspace monitoring); if it's on, allow for the database's refresh delay |
ItemName / WorkspaceName show stale values | Name columns cached from emission time | Filter / join by ItemId / WorkspaceId only (ArtifactId in samples written before 2026-10-06) |
EventStreamNodeStatus shows old status after a node failed | Status is emitted ~every 6 hours | For real-time status, use the Eventstream editor's live view |
CorrelationId maps to multiple nodes | Advanced processing (e.g. SQL operator with multiple destinations) | Disambiguate using NodeDirection + NodeType together with CorrelationId |
| An error count with no message behind it | EventStreamErrorMetrics holds counts only | Query EventStreamDiagnosticLogs: error type and code, severity, IsFatal and a message. Repeated events can be throttled or aggregated, so rows don't equal errors |
| Mirrored DB CDC source rejected | Can't enable Mirroring AND Eventstream CDC on same DB | Pick one — the docs explicitly call this out |
| New Activator rule doesn't fire | Eventstream wasn't republished after adding the destination | Republish the Eventstream after wiring the destination |
| Connector behind firewall fails | Source not publicly reachable | Use Eventstream connector VNet injection |
| DeltaFlow not available on a CDC source | Currently scoped to Azure SQL / SQL MI / SQL Server VM / PostgreSQL CDC | Use raw mode for other CDC sources and flatten Debezium yourself |
| Events pushed to a schema-associated custom endpoint are dropped | Wrong CloudEvents envelope, dataschema version, or per-environment registry host | See references/cloudevents-producer.md — four distinct failure modes |
| Paused nodes are running again after a deploy | Git integration and deployment pipelines don't carry pause/resume state — after CI/CD every node in the target eventstream becomes active | Re-pause after the deploy (portal toggle or the REST /pause calls — see Pause and resume) |
Creating or updating an Eventhouse destination on a schema-associated eventstream answers 422 "MessagingCatalogConfiguration is required …" | A service-side fault, observed 2026-09-10 to 2026-09-14 in one tenant and not re-checked since: a fresh eventstream, KQL database and schema set failed alike | None client-side. Published destinations keep routing, so don't republish or rebuild a working one. Re-test before relying on this row; reading the real state: references/cloudevents-producer.md |
A new event type lands zero rows, .show ingestion failures is empty, and the older types still land | The destination registers types at publish, so one added to the schema set since is dropped before Kusto (observed 2026-09-14) | Publish the destination after adding a type — see references/cloudevents-producer.md |
Reference
Detail lives in references/ — each section above links the file that carries its full version.
- Microsoft Learn: Add and manage an event source
- Microsoft Learn: Set alert on an Eventstream with Activator destination
See also
fabric-eventhouse— the natural KQL-Database pair for analytics on streamed eventsfabric-rest-api— Eventstream item REST endpoints, LRO polling, jobType values; the token audience for Fabric REST against Eventstream items in itsreferences/auth.mdfabric-warehouse-monitoring— Warehouse-side query monitoring; workspace-monitoring links in its references bundle
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

