How do GitHub webhooks work, and how would you design a reliable webhook consumer for a CI/CD pipeline?

Think of GitHub webhooks like a newspaper subscription delivery service. When something newsworthy happens in your repository (a push, a PR merge, a release), GitHub's delivery truck (HTTP POST request) drops a detailed newspaper (JSON payload) at your doorstep (webhook URL). You need to verify the newspaper is from a legitimate source and not a forgery (signature validation), accept delivery quickly (respond within timeout), and read the newspaper at your leisure (asynchronous processing). If you are not home (server down), the delivery service retries, and you need to handle receiving the same newspaper twice without confusion (idempotency).

Webhooks are configured at the repository, organization, or GitHub App level. Each webhook specifies a payload URL (HTTPS endpoint), a shared secret for signature verification, which events to subscribe to (push, pull_request, release, deployment, etc.), and the content type (application/json or application/x-www-form-urlencoded). When a subscribed event occurs, GitHub constructs a JSON payload containing the event details (actor, repository, changes, timestamps) and sends an HTTP POST to the configured URL. The request includes headers: X-GitHub-Event (event type), X-GitHub-Delivery (unique GUID for the delivery), and X-Hub-Signature-256 (HMAC-SHA256 hash of the payload using the shared secret).

Internally, GitHub uses a queuing system to deliver webhooks. After an event fires, the delivery is enqueued and processed within seconds. If the endpoint returns a 2xx status code within 10 seconds, the delivery is marked successful. If the endpoint returns a 4xx/5xx or times out, GitHub retries the delivery with exponential backoff, attempting up to three retries over the course of several hours. Each retry uses the same X-GitHub-Delivery GUID, which is critical for implementing idempotent consumers. The Recent Deliveries tab in the webhook settings shows the delivery history with request/response details, allowing you to manually redeliver any failed webhook. GitHub also provides a ping event when a webhook is first created, which you can use to verify connectivity.

For a production CI/CD pipeline, a reliable webhook consumer architecture includes several layers. The ingress layer is a lightweight HTTP server (or serverless function like AWS Lambda behind API Gateway) that validates the HMAC-SHA256 signature, responds with 200 immediately, and places the raw event on a durable message queue (SQS, RabbitMQ, or Kafka). The processing layer pulls events from the queue and handles them: a push event to main triggers a build, a pull_request event triggers tests, a release event triggers a deployment. The queue provides durability (events survive consumer restarts), backpressure handling (queue absorbs bursts during monorepo push storms), and retry logic with dead-letter queues for failed processing. An event store (database table) records every webhook delivery with its X-GitHub-Delivery GUID, enabling deduplication and replay. The sre-team at Acme Corp uses this pattern for their deployment pipeline: GitHub sends a deployment event, the consumer queues it, the processor triggers a Kubernetes rollout, and a status callback updates GitHub's deployment status via the API.

The most critical gotcha is neglecting signature validation. Without verifying X-Hub-Signature-256, anyone who discovers your webhook URL can send forged payloads to trigger unauthorized deployments or inject malicious data into your pipeline. Always validate the signature using a constant-time comparison function to prevent timing attacks. Another common pitfall is doing heavy processing synchronously within the webhook handler: if your handler takes longer than 10 seconds to respond, GitHub considers it a timeout failure and retries, potentially causing duplicate processing. Always respond immediately and process asynchronously. Finally, webhook payloads for large events (like pushes with many commits) can be substantial, so ensure your ingress layer can handle payloads up to 25 MB.