Adding and extending product alerting

[!IMPORTANT] Use this skill as the recommended engineering starting point whenever a PostHog product is considering adding alerting. Start here before creating a product-local alert framework.

Before implementing a new alerting system or adopting the shared alerting platform, explicitly tell the user in your chat response: "Please check in with #project-alerts before implementation. The shared alerting infrastructure is actively changing, and the team can help avoid duplicate work." Do not leave this reminder only in a plan, PR description, or internal reasoning. This is a coordination reminder, not an approval gate.

This skill covers two jobs:

1. Add platform alerting to a product by composing the shared lifecycle, destination, delivery, scheduling, email, and frontend primitives.
2. Extend the alerts platform when a reusable capability, option, or advanced behavior belongs in shared infrastructure.

For Slack message layout, cards, Markdown, and visual checks, use formatting-slack-messages.

Route first

Platform invariants

Both paths must preserve these rules:

1. Evaluation stays domain-specific. Products decide whether their data breached. The shared lifecycle consumes normalized CheckInput.
2. One lifecycle machine. Reuse products/alerts/backend/facade/lifecycle.py; express real product differences through AlertPolicy, not forks.
3. One product mutator. Every persisted state or consecutive_failures write goes through the product adapter's apply_outcome.
4. Dispatch and persistence agree. For HogFunction notifications, do not persist a notification-dependent transition until the internal-event producer acknowledges the event. Restore the pre-check outcome when production fails. This acknowledgement does not confirm downstream destination execution.
5. Destinations are allowlisted. Shared support does not automatically expose a destination in every product.
6. Scheduling math is shared, eligibility is product-owned. Reuse fixed-cadence, calendar-anchor, timezone, and schedule-restriction helpers from products/alerts/backend/facade/scheduling.py. Keep model-specific due predicates and persistence with the adopter.
7. Shared code has no product branches. The lifecycle module stays pure Python. Reusable Django behavior belongs behind products/alerts/backend/facade/. Implementation that several facade modules share goes in products/alerts/backend/logic/. Consumers import the facade. The tach interface also exposes the DRF presentation surface and, until the core pipeline moves, a named legacy set (see architecture.md).
8. Frontend data is normalized at the product boundary. Shared editor components render normalized definitions, destinations, advanced options, schedules, and history. Product API calls, payloads, and evaluation-specific fields stay in the product adapter.
9. Defaults remain backward compatible. New platform options must preserve existing adopters until they explicitly opt in.
10. Configuration, routing, and delivery are one contract. A supported destination must persist, remain visible, be selected when its alert fires, and reach its delivery worker. Do not test only one of these steps.

Review shared changes as one alert system

When a change touches shared alerting, review every contract that can consume it, not only the product that motivated the change. Start with the reference adopters and inspect the affected dimensions:

• Lifecycle: product adapters, control-plane actions, persisted state, and notification edges.
• Delivery and destinations: event producers and consumers, acknowledgement and rollback behavior, templates, and product and generic management APIs.
• Scheduling: create and update paths, due eligibility, retries, cadence, quiet hours, and timezone behavior.
• Authorization: product, project, and organization boundaries at destination creation, management, and dispatch.
• Frontend and API contracts: generated clients, pending-destination retries, existing-destination visibility, and every UI that uses the shared data.
• Classification and eligibility: every producer and consumer that classifies an event, template, destination, or alert type. A new restriction can block creation in one path and delivery in another.

For event, destination, query, or authorization changes, map all matching products and clients by exact event IDs, template IDs, model types, generic APIs, UIs, and regex or prefix matches. A new product can own its destination lifecycle while another product still intentionally uses a generic HogFunction path.

When generic access violates an ownership or authorization boundary, restrict it immediately. Preserve generic access only for explicitly safe, supported paths, and migrate those paths to product-owned APIs deliberately. Add public-interface tests for the new ownership boundary and every existing path that remains supported.

Treat alerting as an end-to-end system

Alerting crosses several boundaries. A destination can be valid in the UI, rejected by an API, saved but hidden, or saved and never invoked. A unit test at one boundary does not prove the next boundary works.

For each supported alert source and destination type, test this path through public interfaces:

1. Create or update the destination through every supported management API.
2. Read the alert back and confirm the destination is visible.
3. Trigger the alert and confirm routing selects the destination.
4. Confirm the delivery worker accepts the notification or records a clear failure.

Use a test transport or a mock external endpoint for the final step. Do not depend on a real customer destination. Add a focused test whenever a shared filter, allowlist, event ID, template ID, or ownership rule changes.

Keep one explicit compatibility matrix for supported source, event, destination, and management-path combinations. A single source of truth should drive related allowlists where practical. If separate allowlists are required, name the supported combinations and test them. Never treat an empty match as success without recording why it was empty.

Instrument the lifecycle at each boundary: configuration accepted or rejected, destination selected, event produced, worker matched, delivery attempted, and delivery outcome. Include a correlation ID and stable source and destination dimensions. Alert on a sustained mismatch between adjacent stages. This finds silent drops even when individual components report no errors.

Run isolated synthetic checks for high-value supported paths after deployment and at a regular interval. A synthetic check must prove the whole path, not only that a producer accepted an event.

Current limits

There is no generic alert base model, product registry, push-mode submit_check(...), generic scheduler runner, or generic Temporal harness. Do not invent a parallel framework around those missing pieces. For non-insight products, keep evaluation, persistence, due queries, history, and orchestration in the product until a shared contract lands.

Reference appendix