How do you configure Flux notifications and alerts to monitor reconciliation events?

Think of Flux notifications like the alert system in a hospital. The monitoring equipment (Flux controllers) continuously tracks patient vitals (reconciliation status), and the alert system (notification-controller) is configured to page specific doctors (Slack channels, PagerDuty) based on the severity of the event. A minor fluctuation triggers an informational note in the chart (info severity to a logging channel), while a critical vital sign breach triggers an immediate page to the on-call physician (error severity to PagerDuty). Different wards can have different notification rules, just as different Flux resources can route alerts to different teams based on the namespace or resource type.

The Flux notification-controller is one of the core controllers installed with Flux and handles both outbound notifications (alerts sent to external systems) and inbound webhooks (events received from external systems like GitHub or GitLab). For outbound notifications, you configure two resources: a Provider and an Alert. The Provider resource defines the external notification destination, including the type (Slack, Microsoft Teams, Discord, PagerDuty, Opsgenie, generic webhook, and others), the channel or endpoint URL, and optional authentication credentials via a secretRef. The Alert resource defines which Flux events trigger notifications by specifying event sources (a list of Flux resource kinds and names), event severity filtering (info or error), and a reference to the Provider where notifications should be sent. When a Flux controller emits an event — for example, when a Kustomization reconciliation fails or a GitRepository checkout succeeds — the notification-controller evaluates all Alert resources, matches the event against the configured sources, and forwards matching events to the associated Provider.

The event source configuration in an Alert is remarkably flexible. You can watch specific resource instances by specifying both the kind and name, or you can use a wildcard name of * to watch all resources of a given kind. You can also filter by namespace to scope alerts to a specific team's resources. The supported event source kinds include GitRepository, Kustomization, HelmRelease, HelmRepository, HelmChart, Bucket, OCIRepository, ImageRepository, ImagePolicy, and ImageUpdateAutomation. The severity field acts as a minimum threshold — setting it to info captures both informational and error events, while setting it to error filters out informational messages and only forwards failures. This granularity allows platform teams to configure a high-volume info channel for operational visibility while maintaining a separate error-only channel for incidents that require immediate attention.

In production environments managing services like payments-api and order-service, a common notification architecture involves multiple layers. The platform team configures a cluster-wide Alert that watches all Kustomization and HelmRelease resources with error severity, forwarding failures to a shared #platform-alerts Slack channel and PagerDuty for on-call escalation. Each application team then adds their own Alert resources scoped to their namespace, watching their specific GitRepository and Kustomization with info severity to track successful deployments in their team channel like #payments-deployments. For image automation workflows, a dedicated Alert watches ImageUpdateAutomation resources to notify teams when automated image updates are committed to Git. This layered approach ensures that critical failures always reach the on-call engineer while routine deployment notifications go to the appropriate team without creating noise for others.

A frequently encountered gotcha is misconfiguring the Provider secret. For Slack, the secret must contain an address field with the full webhook URL, not a token. For generic webhooks, the payload format is Flux-specific JSON and may not match the expected format of the receiving system without a middleware adapter. Another common mistake is setting severity to info on the PagerDuty Provider, which floods the on-call engineer with informational events every time a routine reconciliation succeeds. Teams should also be aware that the notification-controller has rate limiting built in — identical events within a short window are deduplicated to prevent notification storms during rapid reconciliation failures. Finally, for inbound webhooks, the Receiver CRD creates authenticated webhook endpoints that external systems like GitHub can call to trigger immediate reconciliation, bypassing the polling interval for faster deployments.