How do Flux webhook receivers work, and how do you configure them for secure, event-driven GitOps reconciliation in production?

Flux's default reconciliation model is poll-based: controllers check their sources at regular intervals defined by spec.interval and apply any changes found. While simple and reliable, this introduces latency between when a commit is pushed and when it is deployed. For production environments where deployment speed matters, Flux webhook receivers provide event-driven reconciliation by accepting HTTP webhook calls from external systems and immediately triggering the reconciliation of specified resources. This can reduce deployment latency from minutes to seconds, which is critical for incident response patches or time-sensitive releases.

The notification-controller manages Receiver resources, which define webhook endpoints that external systems can call. When a Receiver is created, the notification-controller generates a unique URL path with an embedded token, such as /hook/sha256-token-here. This URL is then configured as a webhook in GitHub, GitLab, Bitbucket, Docker Hub, Harbor, or any system that can send HTTP POST requests. When the webhook fires, the notification-controller validates the request, checks the HMAC signature or token against the stored secret, and annotates the referenced resources to trigger immediate reconciliation. The annotation forces the source-controller or kustomize-controller to reconcile regardless of the remaining time on their polling interval.

Security is paramount for webhook receivers because they accept inbound HTTP requests from external systems. Each Receiver resource references a Kubernetes secret containing the webhook token or HMAC secret used for signature verification. GitHub webhooks, for example, use HMAC-SHA256 to sign the payload, and the notification-controller verifies this signature before processing the event. The Receiver endpoint should be exposed through an ingress or load balancer with TLS termination, rate limiting, and IP allowlisting where possible. Network policies should restrict access to the notification-controller pod to only allow traffic from the ingress controller. Organizations with strict security requirements can place the webhook receiver behind an API gateway that performs additional authentication and logging before forwarding to the Flux endpoint.

Receiver filtering capabilities allow fine-grained control over which events trigger which resources. A single Receiver can reference multiple resources, and filters can be applied based on the event type, branch, or other payload fields. For example, a GitHub Receiver can be configured to only trigger on push events to the main branch, ignoring pull request events and pushes to feature branches. This prevents unnecessary reconciliations when developers push work-in-progress changes. Cross-referencing between receivers and resources is also possible: a commit to the infrastructure repository can trigger reconciliation of both the infrastructure Kustomization and the application Kustomizations that depend on it, ensuring changes propagate immediately through the dependency chain.

Operating webhook receivers in production requires monitoring and redundancy. The notification-controller should be deployed with multiple replicas behind a service for high availability, as a single pod failure would mean missed webhooks until the pod restarts. Prometheus metrics from the notification-controller track webhook reception rate, validation failures, and processing latency. Alert rules should fire when validation failure rates spike, indicating either a misconfigured webhook secret or an attack. Webhook delivery logs on the Git provider side should be monitored for repeated failures, which could indicate network issues between the provider and the cluster. Many teams also maintain the polling interval as a safety net even when webhooks are configured, using a longer interval like 30 minutes to catch any webhooks that were missed due to transient failures. This belt-and-suspenders approach ensures that the system eventually converges even if the event-driven path has gaps.