setting-up-cloudwatch-observability

---
name: setting-up-cloudwatch-observability
description: >-
  Sets up CloudWatch Application Observability (also called Omni) for the first time. Covers creating an Omni
  Space or Domain; configuring and listing Omni access grants (who has access and at what level) and access
  profiles bounding what async alerts, integrations, or agents can do; instrumenting an application or AI
  agent with the ADOT SDK so traces reach CloudWatch Omni (Python/Node/Java/.NET on EC2/ECS/EKS/Lambda) —
  CloudWatch Omni only; for Application Signals (auto-instrumentation, monitored service, ServiceEvents,
  reporting telemetry) use aws-observability — including no-image-rebuild and .NET CoreCLR profiler env vars;
  ingesting Azure telemetry via the CloudWatch agent on an Azure VM or AKS cluster; connecting Slack to a
  Space (and whether GitHub can be connected); and registering a custom MCP tool server over HTTP or stdio and
  choosing its auth (API key, bearer, or OAuth2). For using a Space already set up — querying, dashboards,
  Omni alerts, or defining Omni resources as code — use aws-observability.
version: 2

---

# Setting Up CloudWatch Application Observability

> **Scope:** First-time setup of a CloudWatch Application Observability Space — from creation through first traces flowing. For using a Space that is already set up (queries, dashboards, alerts, evaluations), route to **aws-observability**.

**Works best with** the [AWS MCP server](https://docs.aws.amazon.com/aws-mcp/) — enables running AWS CLI commands directly. All guidance also works with standard AWS CLI access (`aws cloudwatchomni ...`).

## Concepts

| Term | What it is |
|---|---|
| **Domain** | The identity boundary. Carries the authorization provider (IAM or Identity Center) and owns the endpoint URL customers reach Omni through. One per account, or one shared across an AWS Organization. |
| **Space** | A workspace holding telemetry, in exactly one account and one Region. Created under a Domain. At most one per account per Region. Region rule: under an IAM-only Domain a Space may sit in a Region other than the Domain's; under an Identity Center Domain the Space must be in the Domain's own Region — Identity Center plus Spaces in several Regions needs the org-scoped Domain. |
| **Access grant** | Attaches a principal — person, group, IAM identity, or async workload — to one Space at a permission level. The only way anyone reaches data *through* a Space — it does not restrict the source CloudWatch log groups, which stay readable under their own IAM. |
| **Access Profile** | A named boundary for async workloads (alerts, integrations, agents) that act without a person in the loop. It is only a named container: `create-access-profile` takes a Space, a name, and a description and nothing else — no permission, action, or scope input. It does something only once two separate sets of grants exist (what the profile may do; which workloads may assume it) and a workload names it. |
| **Dataset** | What queries run against. Telemetry arrives through the CloudWatch OTLP endpoints, or by forwarding what is already in CloudWatch log groups. |

Setup order from nothing: **Domain → Space → grants → telemetry in → instrumentation.** "Telemetry in" (a collector exporting to CloudWatch's OTLP endpoints, plus dataset forwarding for what is already in CloudWatch) is its own step, separate from instrumenting the workloads. Access Profiles are **conditional**, not a step in the sequence — only when async workloads (alerts, integrations, agents) are involved. Whenever you give this sequence, also say that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see — a customer who checks Omni first reads a working, empty Space as a failure. For the concept relationships and the full arc, see `references/cloudwatch-omni/app-basics.md`.

This is a **routing skill**. Classify the user's setup request and delegate to the correct reference. References live under `references/cloudwatch-omni/` (CloudWatch Omni); this skill has no CloudWatch setup content, so it has no `references/cloudwatch/` folder — CloudWatch onboarding (Application Signals) lives in **aws-observability**.

| User intent | Reference |
|---|---|
| Understand **what Omni is**, its concepts, or **where to start** | `references/cloudwatch-omni/app-basics.md` |
| **Instrument an AI agent** (ADOT, OpenInference, framework detection, trace verification), or **deploy an agent to production** and get traces flowing to CloudWatch — env vars per platform (AgentCore, Lambda, or other platforms such as ECS/EC2/EKS), routing spans to a custom trace log group, IAM permissions needed, ADOT version requirements | `references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md` (§ Production deployment for the deploy case) |
| **Per-framework OpenInference guide** (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI) once the agent framework is known | `references/cloudwatch-omni/omni-agents-instrumentation/openinference-framework-guide.md`, then the matching `references/cloudwatch-omni/omni-agents-instrumentation/instrument-<framework>.md` |
| **Instrument an application** (ADOT SDK on EC2/ECS/EKS/Lambda — Python, Node.js, Java, .NET) | `references/cloudwatch-omni/instrumentation/instrumentation.md` |
| Emit a **custom application or agent metric** so it is queryable in Omni (why OTLP and not `PutMetricData`/EMF) | `references/cloudwatch-omni/instrumentation/instrumentation.md` (§ Custom metrics) |
| Create an **account-scoped Space or Domain** | `references/cloudwatch-omni/spaces-and-domains.md` |
| Whether a Space can be in a **different Region from its Domain** | `references/cloudwatch-omni/spaces-and-domains.md` (Prerequisites → Region rules) |
| The **AgentCore evaluation role** that `create-space` asks for — what it is, whether to create one | `references/cloudwatch-omni/spaces-and-domains.md` (Step 3 → Also resolve the AgentCore evaluation role) |
| A Space that **was created successfully but returns an authorization error** when used | `references/cloudwatch-omni/spaces-and-domains.md` (Troubleshooting → If the Space was created but cannot be used) — a space access role trust-policy problem, not a grant problem |
| Create a **Domain shared across an AWS Organization** | `references/cloudwatch-omni/org-domains.md` |
| Configure **access grants** for people or IAM identities or alerts | `references/cloudwatch-omni/access-grants.md` |
| Bound an **alert, integration, or agent** with an Access Profile | `references/cloudwatch-omni/access-profiles.md` |
| Deploy an **OTel Collector** so an instrumented app has somewhere to export to (EC2/ECS/EKS) — it exports to CloudWatch's own per-signal OTLP endpoints | `references/cloudwatch-omni/instrumentation/collector.md` |
| Enable **Transaction Search** so traces reach a Space (spans land in `aws/spans` only once it is on — per account, per Region) | `references/cloudwatch-omni/instrumentation/collector.md` (Step 1) |
| **Forward telemetry** already in CloudWatch into the Dataset | `references/cloudwatch-omni/data-forwarding-and-centralization.md` |
| **Send your application's own telemetry from Azure** (the logs/metrics/traces your service emits, via the CloudWatch agent on an Azure VM/AKS) | `references/cloudwatch-omni/azure-ingestion/azure-ingestion.md` (intent triage), then `references/cloudwatch-omni/azure-ingestion/custom-telemetry.md` (the CloudWatch-agent-on-VM/AKS procedure) |
| Connect **Slack** to a Space for the first time | `references/cloudwatch-omni/slack-integration.md` |
| Register a **custom MCP tool server** (HTTP or stdio) and choose its authentication | `references/cloudwatch-omni/custom-mcp-integration.md` |

## Routing Rules

1. If the user is asking **what Omni is**, what a Domain or Space or grant means, **where to start**, or **the end-to-end order of steps** from nothing — rather than asking to perform one setup step — route to `references/cloudwatch-omni/app-basics.md` and answer the sequence from its "Setup order" section (not from a single procedure file such as `collector.md`, which covers one step). It also carries the boundary against **aws-observability**. A question about one concept's rules — a Space's Region relative to its Domain, the roles `create-space` needs — belongs to the procedure file for that concept (`spaces-and-domains.md`), which the routing table names.
2. If the request is about **instrumenting an AI agent project** (adding OTel to a framework such as LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, or Vercel AI), route to `references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md`.
3. If the request is about **instrumenting a general application** (a service on EC2, ECS, EKS, or Lambda in Python, Node.js, Java, or .NET — not an AI-agent framework), route to `references/cloudwatch-omni/instrumentation/instrumentation.md`.
4. If an **instrumentation, ADOT, or collector** request names neither Omni or a Space, nor Application Signals, ServiceEvents, or the `amazon-cloudwatch-observability` add-on, probe the target Region before choosing: `aws cloudwatchomni list-spaces --region <region>`. A Space exists → this skill, `references/cloudwatch-omni/instrumentation/instrumentation.md`. No Space → the customer has not adopted Omni; route to **aws-observability**'s Application Signals onboarding reference. If the probe errors with an unknown service, that is the CLI model, not evidence Omni is absent — fall back to the customer's wording and ask only if still inconclusive.
5. If the request is about **Space/Domain creation or configuration**, route to the matching setup reference. An authorization error that appears the moment a *just-created* Space is used is part of this — it is a space access role trust-policy problem (`create-space` never verifies the role can be assumed), so route to `references/cloudwatch-omni/spaces-and-domains.md` → "If the Space was created but cannot be used", not to the access-grants reference.
6. If the user asks about getting telemetry **to** a destination — telemetry not yet reaching CloudWatch — **deploy an OTel Collector** on EC2/ECS/EKS so an instrumented workload has somewhere to send OTLP, and wire the app's `OTEL_EXPORTER_OTLP_ENDPOINT` to it: `references/cloudwatch-omni/instrumentation/collector.md`. The collector exports straight to CloudWatch's own per-signal OTLP endpoints. If instead the telemetry is **already in CloudWatch log groups** and needs forwarding into the Dataset, route to `references/cloudwatch-omni/data-forwarding-and-centralization.md`.
7. If the request is about **getting Azure telemetry into CloudWatch**, route to `references/cloudwatch-omni/azure-ingestion/azure-ingestion.md` and decide by intent. If the customer wants the telemetry their own application produces (the logs/metrics/traces from their code), that is supported via the CloudWatch agent on an Azure VM or AKS cluster — follow the reference. If they want telemetry their Azure resources emit on their own (the Azure equivalent of AWS VPC flow logs / Route 53 logs), that is not available — say so and do not attempt a setup.
8. If the user asks to **connect, enable, or authorize Slack** for a Space for the first time (including granting the operator permission to use it), route to `references/cloudwatch-omni/slack-integration.md`. Using Slack after it is connected (posting findings to a channel, mentioning the assistant, or searching Slack) happens in Slack and the console and is not covered by these skills; Slack as an **alert notification target** is covered by **aws-observability**'s Omni alerts reference.
9. If the user wants **Application Signals, ServiceEvents, or Dynamic Instrumentation**, STOP and route to **aws-observability** — those are CloudWatch features and are out of scope for Omni instrumentation.
10. If the user already has a working Space and is asking about **queries/dashboards/alerts**, STOP and route to **aws-observability**.
11. If the user asks to **register their own MCP tool server**, connect a custom MCP server, set up MCP authentication, or choose an auth type for an MCP server (including an OAuth-protected one), route to `references/cloudwatch-omni/custom-mcp-integration.md`. For an OAuth-protected server the choice is one of the two OAuth2 types — client credentials for machine-to-machine, authorization code when a user must consent.
12. If the user asks to **connect GitHub**, link a GitHub organization or repositories, or enrich the application map from source code: **CloudWatch Omni has no GitHub integration.** Say so plainly and stop — do not walk through a setup. Do not offer a substitute: the Application Signals GitHub Action and the CloudWatch Logs GitHub audit-log source are separate CloudWatch features, not a way to connect GitHub to Omni, and a custom MCP server is not a GitHub integration. A GitHub-related feature found in public documentation is not evidence of an Omni GitHub integration.
13. If the request names **neither product** — "add observability to my API", "how should I begin monitoring my workload", with no mention of Omni, a Space, a Domain, or of a CloudWatch feature — do not assume either one. Which product a customer wants to *set up* is their intent, not something the account's current state tells you, so ask, framed product versus product: **CloudWatch Omni** (Application/Agent Observability — Domains, Spaces, grants, a Dataset) or **CloudWatch** (log groups, alarms, Log Insights, Application Signals). In the same message, explain the boundary so they can choose — it is **setup versus use**: creating or configuring a Domain, Space, grant, or Access Profile, getting telemetry flowing for the first time, and instrumenting an application or agent for Omni are first-time Omni setup and belong here; querying telemetry, building dashboards, and configuring alerts are day-to-day use, and — together with anything about Application Signals, X-Ray, synthetics, or CloudWatch alarms — belong to **aws-observability**. Do not start walking through Domain or Space creation until they confirm Omni; having asked, stop and wait.