NGINX Interview Questions & Answers — DevOpsDrill

NGINX

Everything for NGINX in one place — pick a section below. 40 reviewed items across 4 content types, plus a hands-on tutorial.

🟩Hands-on TutorialGuideServe static files, reverse-proxy apps, terminate TLS, load-balance, cache, rate-limit, and harden.→

📝Interview Questions12Q&A with detailed answers and code examples→

🏭Use Cases22Real-world production patterns and scenarios→

🔥Production Issues3On-call incidents with root causes and fixes

12 / 12

Popular tags:

⚡

Quick Answer

NGINX sits between clients and backend servers, forwarding requests and returning responses. As a load balancer, it spreads traffic across multiple backends using algorithms like round-robin, least connections, or IP hash to improve availability.

Detailed Answer

Think of NGINX as a receptionist at a busy hospital. When a patient walks in, the receptionist doesn't treat them directly. Instead, she checks which doctor is free, sends the patient there, and passes the doctor's response back. The patient never needs to know which doctor handled their case. This is how a reverse proxy works -- it sits between clients and servers, hiding the backend details from the outside world.

The key directive that makes reverse proxying work is proxy_pass. When a request comes in, NGINX checks its location blocks to match the URL, then forwards the request to backend servers listed in an upstream block. The upstream block groups your servers together and picks a load-balancing strategy. Round-robin is the default, handing out requests evenly. Least-connections sends traffic to whichever server is least busy, which works well for slow or long-running requests. IP-hash always sends the same client to the same server based on their IP, which helps when you need session stickiness.

Under the hood, NGINX uses an event-driven, non-blocking design. On Linux it uses epoll; on BSD it uses kqueue. This means a single worker process can juggle thousands of connections at once without creating a new thread for each one. When proxying, NGINX keeps separate connections open to the client and the backend. It reads the client's request into a buffer, sends it upstream, gets the response, and streams it back. You can use keepalive directives to reuse connections to your backends, which cuts out the overhead of opening new TCP connections for every request and makes a big difference under heavy traffic.

In real-world setups, NGINX is the standard front door for microservices. You typically run it on dedicated machines or as a Kubernetes Ingress controller. TLS is terminated at the NGINX layer, so backends only deal with plain HTTP. Health checks let NGINX stop sending traffic to servers that are down. NGINX Plus has active health checks, while the open-source version uses passive checks through max_fails and fail_timeout. You should also set proxy headers like X-Real-IP and X-Forwarded-For so your apps know the real client IP instead of seeing the proxy's address.

A common mistake is forgetting to set proxy_set_header Host $host. Without it, your backend gets the upstream server name instead of the original hostname, which breaks virtual-host routing. Another pitfall is not tuning proxy_buffer_size and proxy_buffers -- if a backend response is bigger than the default buffers, you get 502 errors. Finally, when using least_conn with WebSocket connections, keep in mind that long-lived connections throw off the count and may prevent new connections from being distributed evenly.

Code Example

# /etc/nginx/nginx.conf - Reverse proxy and load balancer for payments API
# Main context: set worker processes to match CPU cores
worker_processes auto;

events {
    # Maximum simultaneous connections per worker
    worker_connections 4096;
}

http {
    # Define upstream group of backend payment servers
    upstream payments-api {
        # Use least connections algorithm for even distribution
        least_conn;

        # Backend server 1 with weight 3 (gets 3x traffic)
        server 10.0.1.10:8080 weight=3;
        # Backend server 2 with default weight of 1
        server 10.0.1.11:8080;
        # Backend server 3 marked as backup (only used if others fail)
        server 10.0.1.12:8080 backup;

        # Keep 32 idle connections alive to upstream servers
        keepalive 32;
    }

    server {
        # Listen on port 443 with SSL enabled
        listen 443 ssl;
        # Domain name for this virtual host
        server_name payments.prod-cluster.example.com;

        # SSL certificate paths
        ssl_certificate /etc/ssl/certs/payments.crt;
        ssl_certificate_key /etc/ssl/private/payments.key;

        # Proxy all /api/v1/payments requests to the upstream group
        location /api/v1/payments {
            # Forward requests to the payments-api upstream
            proxy_pass http://payments-api;

            # Preserve the original Host header for backend routing
            proxy_set_header Host $host;
            # Pass the real client IP to the backend
            proxy_set_header X-Real-IP $remote_addr;
            # Append to the forwarded-for chain
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # Tell the backend the original protocol was HTTPS
            proxy_set_header X-Forwarded-Proto $scheme;

            # Enable keepalive connections to upstream
            proxy_http_version 1.1;
            # Clear the Connection header for keepalive
            proxy_set_header Connection "";

            # Timeout waiting for upstream to respond (seconds)
            proxy_read_timeout 60s;
            # Timeout for establishing connection to upstream
            proxy_connect_timeout 10s;

            # Retry on specific error conditions
            proxy_next_upstream error timeout http_502 http_503;
        }

        # Health check endpoint - returns 200 directly from NGINX
        location /health {
            # Return 200 with a simple body
            return 200 'ok';
            # Set content type for the response
            add_header Content-Type text/plain;
        }
    }
}

💡

Interview Tip

A junior engineer typically says NGINX forwards requests to backend servers and leaves it at that. Go further by explaining the event-driven architecture that lets NGINX handle thousands of connections with very little memory. Talk about the specific load-balancing options -- round-robin, least connections, IP hash, and generic hash -- and when you would pick each one. Bring up production details like preserving client IPs with X-Real-IP and X-Forwarded-For, using keepalive connections to cut down TCP overhead, and handling upstream failures with passive health checks via max_fails and fail_timeout. Interviewers really like it when you can explain proxy buffering and why a wrong proxy_buffer_size causes 502 errors.

◈ Architecture Diagram

┌──────────────┐
│   Clients    │
│ (Internet)   │
└──────┬───────┘
       │
       ↓
┌──────────────────────────┐
│       NGINX              │
│   Reverse Proxy /        │
│   Load Balancer          │
│  ┌────────────────────┐  │
│  │ upstream payments   │  │
│  │  least_conn         │  │
│  └────────────────────┘  │
└───┬──────┬──────┬────────┘
    │      │      │
    ↓      ↓      ↓
┌──────┐┌──────┐┌──────┐
│ App  ││ App  ││ App  │
│ :8080││ :8080││ :8080│
│ w=3  ││ w=1  ││backup│
└──────┘└──────┘└──────┘

Open Full Page ↗

💬 Comments

⚡

Quick Answer

The NGINX Ingress Controller runs as a pod inside your cluster. It watches for Ingress resources and automatically generates NGINX config to route external HTTP/HTTPS traffic to the right services based on hostnames and URL paths.

Detailed Answer

Think of a large office building with one front desk. When a visitor arrives and asks for marketing, the receptionist sends them to floor three. If they ask for engineering, they go to floor five. The NGINX Ingress Controller works the same way for a Kubernetes cluster -- it is the single entry point that reads routing rules and sends traffic to the right internal service based on the hostname or URL path in the request.

The NGINX Ingress Controller is deployed as a Deployment or DaemonSet, usually in its own namespace called ingress-nginx. It runs the NGINX binary inside a container and watches the Kubernetes API for Ingress resources. When you create an Ingress object with rules for host-based or path-based routing, the controller picks up the change and rebuilds the nginx.conf file on the fly, then reloads NGINX without dropping active connections. Each Ingress rule maps a hostname-plus-path combination to a backend Service and port. The controller looks up the Service's endpoints (the actual pod IPs) and creates upstream blocks for them.

Behind the scenes, the controller uses the Kubernetes informer pattern to watch for changes to Ingress, Service, Endpoint, Secret, and ConfigMap resources. When something changes, it runs a sync loop that builds a new NGINX config from a template and compares it to the running one. If they differ, it writes the new config and triggers a reload. Newer versions use NGINX's dynamic module and Lua-based upstream balancing to update endpoints without a full reload, which lowers the risk of dropped connections during deployments. TLS certificates referenced in Ingress resources through secretName are pulled from Kubernetes Secrets and written to disk for NGINX to use.

In production, you expose the Ingress Controller using a Service of type LoadBalancer, which creates a cloud load balancer (like an AWS NLB or GCP LB) that points to the controller pods. Annotations on Ingress resources let you customize behavior per route -- setting timeouts, enabling CORS, adding rate limits, or turning on authentication. A ConfigMap lets you tune global NGINX settings like proxy-body-size, use-gzip, and worker-processes. You should also set up a default backend to show a custom 404 page for requests that don't match any Ingress rule.

A common mistake is creating Ingress resources before deploying the controller, which means nothing processes the rules and traffic just fails silently. Another frequent issue is forgetting to set the correct ingress class (kubernetes.io/ingress.class or the newer ingressClassName field) when you have multiple controllers in the same cluster. Without it, a rule might get picked up by the wrong controller or ignored entirely. Also, changes to TLS Secrets aren't always instant because the sync loop has a small delay, and a misconfigured certificate chain causes silent SSL handshake failures that only show up in the NGINX error logs inside the pod.

Code Example

# Step 1: Install NGINX Ingress Controller using Helm
# Add the official ingress-nginx Helm chart repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# Update Helm repositories to fetch latest chart versions
helm repo update
# Install the ingress controller into its own namespace
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.replicaCount=2 \
  --set controller.resources.requests.cpu=250m \
  --set controller.resources.requests.memory=256Mi

# Step 2: Ingress resource for payments API routing
# File: payments-ingress.yaml
apiVersion: networking.k8s.io/v1   # Use the stable networking API version
kind: Ingress                       # Resource type for HTTP routing rules
metadata:
  name: payments-api-ingress        # Name of this Ingress resource
  namespace: prod-cluster            # Deploy in the production namespace
  annotations:
    # Force HTTPS redirect for all HTTP requests
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # Set maximum allowed request body size to 10 megabytes
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    # Set upstream connection timeout to 30 seconds
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    # Enable rate limiting at 100 requests per second per IP
    nginx.ingress.kubernetes.io/limit-rps: "100"
spec:
  ingressClassName: nginx            # Select the NGINX ingress controller
  tls:
    - hosts:
        # Hostname covered by the TLS certificate
        - payments.prod-cluster.example.com
      # Kubernetes Secret containing the TLS cert and key
      secretName: payments-tls-secret
  rules:
    # Route traffic based on the Host header
    - host: payments.prod-cluster.example.com
      http:
        paths:
          # Route /api/v1/payments to the payments backend service
          - path: /api/v1/payments
            pathType: Prefix          # Match this path and all sub-paths
            backend:
              service:
                name: payments-api-svc  # Target Kubernetes Service name
                port:
                  number: 8080          # Port exposed by the Service
          # Route /api/v1/invoices to the invoicing backend service
          - path: /api/v1/invoices
            pathType: Prefix
            backend:
              service:
                name: invoicing-api-svc  # Target Kubernetes Service name
                port:
                  number: 8081           # Port exposed by the Service

💡

Interview Tip

A junior engineer typically describes writing an Ingress YAML with host and path rules but skips over how the controller actually works under the hood. To stand out, explain the reconciliation loop -- how the controller watches the Kubernetes API for changes to Ingress, Service, Endpoint, and Secret resources, rebuilds nginx.conf from a template, and does a graceful reload. Mention the difference between the community ingress-nginx controller and the commercial NGINX Inc version (nginx-ingress), since they use different annotation formats. Talk about how you expose the controller through a LoadBalancer Service in the cloud and how ingressClassName replaced the older kubernetes.io/ingress.class annotation. Point out that in production you run multiple controller replicas behind a cloud load balancer for high availability.

◈ Architecture Diagram

┌────────────────────────────────────┐
│          Cloud Load Balancer       │
│         (AWS NLB / GCP LB)         │
└───────────────┬────────────────────┘
                │
                ↓
┌────────────────────────────────────┐
│    NGINX Ingress Controller Pod    │
│  ┌──────────────────────────────┐  │
│  │  Watches: Ingress, Service,  │  │
│  │  Endpoints, Secrets          │  │
│  │  Generates: nginx.conf       │  │
│  └──────────────────────────────┘  │
└──────┬──────────────┬──────────────┘
       │              │
       ↓              ↓
┌────────────┐ ┌─────────────┐
│ payments   │ │ invoicing   │
│ -api-svc   │ │ -api-svc    │
│ :8080      │ │ :8081       │
├────────────┤ ├─────────────┤
│┌────┐┌────┐│ │┌────┐┌────┐ │
││Pod ││Pod ││ ││Pod ││Pod │ │
│└────┘└────┘│ │└────┘└────┘ │
└────────────┘ └─────────────┘

Open Full Page ↗

💬 Comments

⚡

Quick Answer

SSL/TLS termination means NGINX handles all the encryption and decryption of HTTPS traffic so your backends don't have to. You configure certificate files, enable modern TLS versions, speed up handshakes with session caching and OCSP stapling, and automate renewal with certbot.

Detailed Answer

Think of SSL/TLS termination like a secure mailroom in an office building. All letters arrive sealed and encrypted. The mailroom opens them, checks they are legitimate, and hands the plain-text contents to the right department. The departments inside never deal with seals or envelopes. In the same way, NGINX handles the TLS connection at the edge, decrypts the traffic, and sends plain HTTP to your backend apps, saving them from the CPU cost of encryption.

To set up TLS termination, you add the ssl parameter to the listen directive and point NGINX to your certificate and key files. The ssl_certificate directive should point to the full chain file -- your certificate plus any intermediate CA certificates bundled together. The ssl_certificate_key directive points to the private key. You control which TLS versions to allow with ssl_protocols and which ciphers to use with ssl_ciphers. The current best practice is to allow only TLS 1.2 and 1.3, turning off older vulnerable versions like TLS 1.0 and 1.1. The ssl_prefer_server_ciphers directive tells NGINX to use its own cipher order instead of the client's, so stronger ciphers get picked first.

When a client starts a TLS handshake, NGINX uses the certificate and private key to do an asymmetric key exchange, then both sides agree on a symmetric session key for the actual data transfer. This handshake is expensive on the CPU, so NGINX offers several ways to speed it up. SSL session caching (ssl_session_cache) stores session data in shared memory so returning clients can skip the full handshake. SSL session tickets (ssl_session_tickets) let the server pack the session state into a ticket and send it to the client, who presents it on the next visit. OCSP stapling (ssl_stapling) lets NGINX fetch and cache the certificate's revocation status from the CA and attach it to the handshake, so the client doesn't need to make a separate request to check if the cert is still valid.

In production, you typically automate certificate management with Let's Encrypt and certbot. Certbot gets free domain-validated certificates and can configure NGINX automatically or use a standalone HTTP challenge. You set up a cron job or systemd timer to run certbot renew on a schedule, with a post-renewal hook that reloads NGINX to pick up the fresh certificates. For companies with their own internal certificate authority, certificates are usually distributed through tools like Ansible. In Kubernetes, cert-manager handles certificate issuance and renewal automatically, storing certs as Kubernetes Secrets that the Ingress Controller mounts on its own.

One critical mistake is providing an incomplete certificate chain. If you only include your server certificate without the intermediate CA certificates, some clients -- especially mobile browsers and API clients with older trust stores -- will fail the TLS handshake with confusing errors. Another common problem is leaving TLS 1.0 or 1.1 enabled, which fails PCI-DSS compliance scans. And if you don't set up automatic renewal, your certificates will eventually expire and cause sudden outages. Always test your TLS setup with tools like ssl-labs.com or testssl.sh, and set up alerts to monitor certificate expiration dates.

Code Example

# /etc/nginx/conf.d/payments-ssl.conf
# SSL/TLS termination configuration for payments API

server {
    # Redirect all HTTP traffic to HTTPS
    listen 80;
    # Match the payments domain
    server_name payments.prod-cluster.example.com;
    # Permanent redirect to HTTPS version
    return 301 https://$host$request_uri;
}

server {
    # Listen on 443 with SSL and HTTP/2 enabled
    listen 443 ssl http2;
    # Domain for this server block
    server_name payments.prod-cluster.example.com;

    # Full certificate chain (server cert + intermediate CAs)
    ssl_certificate /etc/letsencrypt/live/payments.prod-cluster.example.com/fullchain.pem;
    # Private key file (keep permissions 600)
    ssl_certificate_key /etc/letsencrypt/live/payments.prod-cluster.example.com/privkey.pem;

    # Allow only TLS 1.2 and 1.3 (disable vulnerable older versions)
    ssl_protocols TLSv1.2 TLSv1.3;
    # Use strong cipher suites only
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    # Prefer server cipher order over client preference
    ssl_prefer_server_ciphers on;

    # SSL session caching: 10MB shared cache holds ~40,000 sessions
    ssl_session_cache shared:SSL:10m;
    # Sessions are valid for 1 day before renegotiation
    ssl_session_timeout 1d;
    # Disable session tickets for forward secrecy (optional, strict mode)
    ssl_session_tickets off;

    # Enable OCSP stapling to speed up certificate validation
    ssl_stapling on;
    # Allow NGINX to verify OCSP response using the CA chain
    ssl_stapling_verify on;
    # Trusted CA chain for OCSP verification
    ssl_trusted_certificate /etc/letsencrypt/live/payments.prod-cluster.example.com/chain.pem;
    # DNS resolver for OCSP requests (Cloudflare DNS)
    resolver 1.1.1.1 1.0.0.1 valid=300s;
    # Timeout for DNS resolution
    resolver_timeout 5s;

    # Security headers
    # Enforce HTTPS for 1 year including subdomains
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # Proxy to backend payments service
    location /api/v1/payments {
        # Forward decrypted HTTP to upstream servers
        proxy_pass http://payments-api;
        # Pass original host header
        proxy_set_header Host $host;
        # Inform backend that original request was HTTPS
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Certbot auto-renewal cron job (run as root)
# Attempt renewal twice daily; reload NGINX on success
# 0 */12 * * * certbot renew --quiet --deploy-hook "systemctl reload nginx"

💡

Interview Tip

A junior engineer typically knows how to add ssl_certificate and ssl_certificate_key but can't explain the performance and security details that matter in production. Show deeper knowledge by talking about SSL session caching and how it cuts handshake overhead for returning clients. Explain why OCSP stapling helps performance -- it saves the client from having to contact the CA separately to check if the cert is revoked. Bring up the full certificate chain and how a missing intermediate CA cert causes failures on some clients but not others. Discuss forward secrecy and why you might disable session tickets in high-security environments. Interviewers are especially impressed when you mention ssl-labs.com for testing your config and certbot for automating renewal to avoid surprise certificate expirations.

◈ Architecture Diagram

┌──────────┐         ┌──────────────────────────┐
│  Client  │         │         NGINX             │
│ (Browser)│         │    SSL/TLS Termination     │
└────┬─────┘         └────────────┬───────────────┘
     │                            │
     │  TLS Handshake             │
     │  ───────────────→          │
     │  ←─── Certificate          │
     │  ───── Client Key          │
     │        Exchange ──→        │
     │  ←─── Finished             │
     │                            │
     │  Encrypted HTTPS           │
     │  ═══════════════→          │
     │                            │
     │                   ┌────────┴────────┐
     │                   │ Decrypts to     │
     │                   │ plain HTTP      │
     │                   └────────┬────────┘
     │                            │
     │                            ↓
     │                   ┌─────────────────┐
     │                   │  Backend Pods   │
     │                   │  (HTTP :8080)   │
     │                   └─────────────────┘

Open Full Page ↗

💬 Comments

⚡

Quick Answer

NGINX uses the limit_req module with a leaky bucket algorithm to control how fast clients can send requests, usually tracked by IP. For DDoS protection, you combine rate limiting with connection limits (limit_conn), request size caps, timeout tuning, and geo-blocking.

Detailed Answer

Picture a nightclub with a bouncer at the door. The bouncer lets people in at a steady pace -- say ten per minute. If a big crowd shows up at once, some wait in a short line and the rest are turned away. The bouncer also limits how many people from the same group can be inside at the same time. This is exactly how NGINX rate limiting and connection limiting work together: they control the flow of requests so your backend servers don't get overwhelmed.

NGINX rate limiting uses two directives that work as a pair. The limit_req_zone directive goes in the http block and sets up a shared memory zone that tracks request rates. You give it a key (usually $binary_remote_addr for per-IP tracking), a zone name and size, and a rate like 10r/s for ten requests per second. The limit_req directive goes inside a server or location block, points to that zone, and optionally sets a burst value with nodelay or delay. The burst parameter creates a small queue for requests that come in faster than the limit. With nodelay, burst requests are handled right away but anything beyond the burst is rejected. Without nodelay, excess requests are queued and released at the set rate. If you don't set burst at all, any request over the limit is immediately rejected with a 503 (or whatever you set with limit_req_status).

Under the hood, NGINX uses the leaky bucket algorithm. Think of a bucket with a hole in the bottom that drains at a fixed rate (your configured rate). Each request adds water. If the bucket overflows (burst is exceeded), the request is dropped. The shared memory zone stores state for each key -- using $binary_remote_addr takes just 64 bytes per entry, so a 10MB zone can track about 160,000 unique IPs. Connection limiting with limit_conn_zone and limit_conn works similarly but tracks how many connections are open at the same time rather than how fast they arrive. This matters for DDoS protection because one attacker IP could open hundreds of connections at once to exhaust your server's file descriptors.

In production, good DDoS protection layers multiple defenses. Rate limiting handles application-layer (Layer 7) attacks by capping request speed. Connection limiting caps simultaneous connections per IP. You should also set tight client_body_timeout, client_header_timeout, and keepalive_timeout values to quickly close slow connections, which stops Slowloris-style attacks. The limit_req_log_level directive lets you log rate-limited requests at warn level so you can monitor them without flooding your logs. For blocking traffic from known bad IP ranges, the geo module lets you deny entire subnets. For serious DDoS attacks, NGINX works alongside upstream services like Cloudflare, AWS Shield, or hardware appliances that handle network-layer (Layer 3/4) floods before traffic even reaches NGINX.

A common mistake is setting rate limits too tight, which blocks real users -- especially those behind corporate NATs where many people share one IP address. Always start with loose limits and tighten gradually while watching your metrics. Another mistake is applying rate limits globally without exempting health check endpoints, which causes your load balancer's health checks to fail and triggers false downtime alerts. Also, $binary_remote_addr doesn't give you the real client IP when NGINX sits behind another proxy like a CDN. In that case, you need the realip module to extract the actual client IP from X-Forwarded-For before applying any rate limits.

Code Example

# /etc/nginx/nginx.conf - Rate limiting and DDoS protection
worker_processes auto;

events {
    # Max connections per worker process
    worker_connections 4096;
}

http {
    # --- Rate Limiting Zones ---
    # General API rate limit: 10 requests/second per client IP
    # Zone size 10m can track ~160,000 unique IPs
    limit_req_zone $binary_remote_addr zone=api_rate:10m rate=10r/s;

    # Stricter rate limit for login endpoint: 5 requests/minute per IP
    limit_req_zone $binary_remote_addr zone=login_rate:5m rate=5r/m;

    # --- Connection Limiting Zone ---
    # Track concurrent connections per client IP
    limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;

    # Set custom status code for rate-limited requests (default 503)
    limit_req_status 429;
    # Set custom status code for connection-limited requests
    limit_conn_status 429;

    # Log rate-limited requests at warn level (not error)
    limit_req_log_level warn;

    # --- Timeout Hardening Against Slowloris ---
    # Close connections if client header not received in 10s
    client_header_timeout 10s;
    # Close connections if client body not received in 10s
    client_body_timeout 10s;
    # Limit idle keepalive connections to 15 seconds
    keepalive_timeout 15s;
    # Close connections if client stops reading response
    send_timeout 10s;

    # --- Geo-based Blocking ---
    # Define blocked IP ranges (example: known malicious networks)
    geo $blocked_ip {
        default 0;                    # Allow all by default
        192.0.2.0/24 1;              # Block this known bad range
        198.51.100.0/24 1;           # Block another bad range
    }

    server {
        listen 443 ssl;
        server_name payments.prod-cluster.example.com;

        # Block requests from flagged IP ranges
        if ($blocked_ip) {
            # Return 403 Forbidden for blocked IPs
            return 403;
        }

        # Limit maximum request body size to 1 megabyte
        client_max_body_size 1m;

        # Limit each IP to 20 concurrent connections
        limit_conn conn_per_ip 20;

        # General API endpoint with rate limiting
        location /api/v1/payments {
            # Apply rate limit with burst of 20 and no delay on burst
            limit_req zone=api_rate burst=20 nodelay;
            # Forward to backend servers
            proxy_pass http://payments-api;
            # Pass real client IP to backend
            proxy_set_header X-Real-IP $remote_addr;
        }

        # Login endpoint with strict rate limiting
        location /api/v1/auth/login {
            # Apply strict login rate limit: 5/min with burst of 3
            limit_req zone=login_rate burst=3 nodelay;
            # Forward to authentication service
            proxy_pass http://auth-api;
            # Pass real client IP to backend
            proxy_set_header X-Real-IP $remote_addr;
        }

        # Health check endpoint - exempt from rate limiting
        location /health {
            # No limit_req here - health checks must always pass
            return 200 'ok';
            # Set plain text content type
            add_header Content-Type text/plain;
        }
    }
}

💡

Interview Tip

A junior engineer typically mentions limit_req with a basic rate but doesn't explain the leaky bucket algorithm or the key difference between burst with nodelay versus without it. Make sure you explain that without burst, excess requests get rejected immediately, which can hurt legitimate traffic during spikes. With burst and nodelay, the burst requests go through right away but anything beyond the burst is dropped. With burst but no nodelay, excess requests sit in a queue and drain at the configured rate. Talk about layered defense: rate limiting for Layer 7, connection limiting for connection floods, and timeout hardening for Slowloris attacks. Mention that $binary_remote_addr gives you the wrong IP behind a CDN and you need the realip module to pull the real client IP from X-Forwarded-For. Interviewers also like hearing that you exempt health check endpoints from rate limits.

◈ Architecture Diagram

┌──────────────────────────────────────────┐
│              Client Requests              │
│         (Normal + Attack Traffic)         │
└─────────────────┬────────────────────────┘
                  │
                  ↓
┌──────────────────────────────────────────┐
│           NGINX DDoS Layers              │
│                                          │
│  Layer 1: geo $blocked_ip                │
│  ┌────────────────────────────────────┐  │
│  │ Block known malicious IP ranges    │  │
│  └──────────────────┬─────────────────┘  │
│                     ↓                    │
│  Layer 2: limit_conn conn_per_ip 20      │
│  ┌────────────────────────────────────┐  │
│  │ Max 20 concurrent conns per IP     │  │
│  └──────────────────┬─────────────────┘  │
│                     ↓                    │
│  Layer 3: limit_req zone=api_rate        │
│  ┌────────────────────────────────────┐  │
│  │ 10 req/s per IP (burst=20)         │  │
│  └──────────────────┬─────────────────┘  │
│                     ↓                    │
│  Layer 4: Timeout Hardening              │
│  ┌────────────────────────────────────┐  │
│  │ header=10s body=10s keepalive=15s  │  │
│  └──────────────────┬─────────────────┘  │
└─────────────────────┬────────────────────┘
                      ↓
              ┌──────────────┐
              │ Backend Pods │
              │  (Protected) │
              └──────────────┘

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Set worker_processes to match your CPU cores, size worker_connections for your expected traffic, tune proxy and client buffers to avoid disk I/O, enable sendfile and tcp_nopush for static files, and configure keepalive to reduce connection overhead. The goal is maximum throughput with minimum latency.

Detailed Answer

Think of NGINX workers like lanes on a highway. If your server has eight CPU cores but NGINX only runs two workers, you have an eight-lane highway with six lanes closed -- traffic backs up for no reason. But running more workers than cores creates overhead from constant context switching, like cars merging between lanes nonstop. Performance tuning is about matching your NGINX config to your hardware and traffic so every resource is used without fighting for attention.

The most important setting is worker_processes, which controls how many NGINX worker processes run at once. Setting it to auto tells NGINX to count your CPU cores and spawn one worker per core, which is the best starting point. Each worker runs an event loop using epoll on Linux or kqueue on macOS/BSD, handling thousands of connections at the same time without needing a thread for each one. The worker_connections directive sets the maximum connections each worker can handle simultaneously. Your total capacity is worker_processes times worker_connections. For reverse proxy setups, remember that each client connection also needs a connection to the backend, which effectively cuts your capacity in half. Setting worker_rlimit_nofile makes sure the operating system allows enough file descriptors for all these connections.

Buffer settings have a huge effect on performance. When NGINX proxies a request, it stores the backend response in memory before sending it to the client. The proxy_buffer_size directive sets the buffer for the first part of the response, usually the headers. The proxy_buffers directive sets how many buffers to use and how big each one is for the response body. If the response is bigger than all the buffers combined, NGINX spills the overflow to a temporary file on disk, which is dramatically slower. The proxy_busy_buffers_size setting controls how much buffered data can be sent to the client while the rest is still being read from the backend. For serving static files, enabling sendfile lets the kernel handle file reads directly, skipping the usual copy through user space. Pairing it with tcp_nopush and tcp_nodelay reduces the number of TCP packets and cuts latency.

In production, tuning is an ongoing process driven by monitoring. Start with worker_processes auto and worker_connections 4096. Use stub_status or a Prometheus exporter to watch active connections, accepted connections, and request rates. If connections are queuing up, increase worker_connections and make sure worker_rlimit_nofile matches. For reverse proxy workloads, set proxy_buffers to handle your typical response size -- 8 buffers of 16k is a good default for API responses. Turn on gzip for text-based responses to save bandwidth, but watch your CPU usage since compression takes CPU cycles. Set client_body_buffer_size large enough for typical POST bodies so they stay in memory. Keepalive connections to both clients (keepalive_timeout) and backends (upstream keepalive) save the overhead of new TCP handshakes and TLS negotiations for repeat connections.

A big gotcha is setting worker_connections high without also raising the system's ulimit for file descriptors. NGINX will hit the OS limit and start refusing connections with "too many open files" errors in the log. Another common mistake is over-buffering -- setting huge proxy buffers that eat up RAM when multiplied across thousands of connections. For example, 8 buffers of 128k per connection across 10,000 connections uses nearly 10GB of RAM just for proxy buffers. Always do the math: buffer count times buffer size times expected connections equals total memory needed. Also, enabling gzip on already-compressed content like images or pre-compressed assets wastes CPU for no benefit, so use gzip_types to target only text-based content types.

Code Example

# /etc/nginx/nginx.conf - Performance-tuned NGINX configuration
# Match worker processes to available CPU cores
worker_processes auto;

# Set max open file descriptors per worker (must match ulimit -n)
worker_rlimit_nofile 65535;

# Pin each worker to a specific CPU core for cache efficiency
worker_cpu_affinity auto;

events {
    # Max simultaneous connections per worker process
    worker_connections 8192;
    # Accept multiple connections at once (reduces latency)
    multi_accept on;
    # Use epoll event model on Linux for best performance
    use epoll;
}

http {
    # --- File Serving Optimization ---
    # Use kernel-level file sending (bypass user-space copy)
    sendfile on;
    # Optimize packet size by combining headers and file data
    tcp_nopush on;
    # Disable Nagle's algorithm for low-latency responses
    tcp_nodelay on;

    # --- Keepalive Tuning ---
    # Client keepalive timeout (seconds)
    keepalive_timeout 30s;
    # Max requests per keepalive connection before closing
    keepalive_requests 1000;

    # --- Buffer Configuration ---
    # Buffer for reading client request body (covers most POST payloads)
    client_body_buffer_size 16k;
    # Buffer for reading client request headers
    client_header_buffer_size 4k;
    # Max client request body size (for file uploads)
    client_max_body_size 50m;
    # Buffers for large client headers (e.g., big cookies)
    large_client_header_buffers 4 16k;

    # --- Gzip Compression ---
    # Enable gzip compression for responses
    gzip on;
    # Minimum response size to compress (skip tiny responses)
    gzip_min_length 1024;
    # Compression level: 1-9, 4 is good balance of CPU vs ratio
    gzip_comp_level 4;
    # Compress these content types only (skip images, binaries)
    gzip_types text/plain text/css application/json application/javascript text/xml;
    # Add Vary header so caches store compressed and uncompressed versions
    gzip_vary on;

    # --- Open File Cache ---
    # Cache file descriptors, sizes, and modification times
    open_file_cache max=10000 inactive=60s;
    # Revalidate cached file info every 30 seconds
    open_file_cache_valid 30s;
    # File must be accessed twice before being cached
    open_file_cache_min_uses 2;

    # --- Upstream with Keepalive ---
    upstream payments-api {
        # Backend servers for payment processing
        server 10.0.1.10:8080;
        server 10.0.1.11:8080;
        server 10.0.1.12:8080;
        # Maintain 64 idle keepalive connections to upstream
        keepalive 64;
        # Timeout for idle keepalive connections to upstream
        keepalive_timeout 60s;
    }

    server {
        listen 443 ssl http2;
        server_name payments.prod-cluster.example.com;

        # --- Proxy Buffer Tuning ---
        location /api/v1/payments {
            proxy_pass http://payments-api;
            # Buffer for upstream response headers
            proxy_buffer_size 8k;
            # 8 buffers of 16k each for response body (128k total)
            proxy_buffers 8 16k;
            # Max data sent to client while still reading from upstream
            proxy_busy_buffers_size 32k;

            # Enable HTTP/1.1 for upstream keepalive support
            proxy_http_version 1.1;
            # Clear Connection header for keepalive
            proxy_set_header Connection "";
        }
    }

    # --- Monitoring Endpoint ---
    server {
        # Listen on internal-only port for metrics
        listen 8080;
        location /nginx_status {
            # Enable basic status page for monitoring
            stub_status on;
            # Only allow internal network access
            allow 10.0.0.0/8;
            # Deny all other access
            deny all;
        }
    }
}

💡

Interview Tip

A junior engineer typically says set worker_processes to the number of CPU cores and stops there. To show real-world experience, walk through the full tuning chain: worker_processes auto, worker_connections sized for your traffic, worker_rlimit_nofile matching the OS ulimit, and worker_cpu_affinity for pinning workers to specific cores. Explain the memory math behind proxy buffers: number of buffers times buffer size times concurrent connections equals total memory usage. Talk about why sendfile and tcp_nopush matter for static content and why keepalive connections to your backends reduce latency for repeated requests. Mention stub_status or a Prometheus exporter for tracking NGINX metrics, and explain how you use active connection counts and request rates to confirm your tuning is actually working. Interviewers value candidates who tie config changes to measurable results.

◈ Architecture Diagram

┌─────────────────────────────────────────────┐
│              NGINX Master Process            │
│          (reads config, manages workers)     │
└──────────┬──────────┬──────────┬─────────────┘
           │          │          │
           ↓          ↓          ↓
   ┌────────────┐┌────────────┐┌────────────┐
   │  Worker 1  ││  Worker 2  ││  Worker N   │
   │  (CPU 0)   ││  (CPU 1)   ││  (CPU N)    │
   │            ││            ││             │
   │ Connections││ Connections││ Connections │
   │  0..8192   ││  0..8192   ││  0..8192    │
   │            ││            ││             │
   │ ┌────────┐ ││ ┌────────┐ ││ ┌────────┐  │
   │ │ epoll  │ ││ │ epoll  │ ││ │ epoll  │  │
   │ │ event  │ ││ │ event  │ ││ │ event  │  │
   │ │ loop   │ ││ │ loop   │ ││ │ loop   │  │
   │ └────────┘ ││ └────────┘ ││ └────────┘  │
   │            ││            ││             │
   │ Buffers:   ││ Buffers:   ││ Buffers:    │
   │ proxy 8x16k││ proxy 8x16k││ proxy 8x16k │
   │ client 16k ││ client 16k ││ client 16k  │
   └────────────┘└────────────┘└─────────────┘
           │          │          │
           ↓          ↓          ↓
   ┌─────────────────────────────────────┐
   │        Upstream Backend Pods        │
   │  (keepalive=64 idle connections)    │
   └─────────────────────────────────────┘

Open Full Page ↗

💬 Comments

⚡

Quick Answer

NGINX processes: exact match (=) first, then longest prefix (^~) stops regex search, then regex (~ or ~*) in order, then longest prefix match. Understanding this prevents routing bugs.

Detailed Answer

Matching Priority (highest to lowest)

1. Exact match =: location = /api/health — matches ONLY /api/health, nothing else. If found, stops immediately. 2. Preferential prefix ^~: location ^~ /static/ — if this is the longest prefix match, SKIP all regex locations. Use for static files to avoid regex overhead. 3. **Regex ~ (case-sensitive) or ~* (case-insensitive): location ~ \.php$ — evaluated in ORDER of appearance in config. First match wins. 4. Standard prefix: location /api/ — longest prefix wins, but only AFTER all regex locations are checked.

Common Pitfall:** A prefix location /api/ can be overridden by a regex location ~ /api/.*\.json — the regex takes priority even though the prefix is more specific. Use ^~ to prevent this.

Best Practice: Use exact matches for health checks, ^~ for static assets, regex only when necessary, and prefix for everything else.

Code Example

# NGINX location matching examples
server {
    listen 80;
    
    # 1. Exact match (highest priority)
    location = /health {
        return 200 'OK';
    }
    
    # 2. Preferential prefix (skips regex)
    location ^~ /static/ {
        root /var/www;
        expires 30d;
    }
    
    # 3. Regex (evaluated in order)
    location ~ \.php$ {
        fastcgi_pass php:9000;
    }
    
    location ~* \.(jpg|png|gif)$ {
        expires 7d;
    }
    
    # 4. Standard prefix (lowest priority)
    location /api/ {
        proxy_pass http://backend;
    }
    
    location / {
        root /var/www/html;
        try_files $uri $uri/ /index.html;
    }
}

# Test which location matches:
# /health          → exact match =
# /static/app.js   → ^~ prefix (skips regex)
# /image.PHP       → ~* case-insensitive regex
# /api/users       → /api/ prefix
# /about           → / prefix (catch-all)

💡

Interview Tip

Draw the priority order: = → ^~ → ~ → prefix. The ^~ vs regex precedence is the most commonly misunderstood part and a frequent interview question.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Define upstream blocks with backend servers and load balancing algorithm, configure proxy_pass with appropriate headers, add SSL certificate and key, and use passive health checks to remove failed backends.

Detailed Answer

Architecture

Clients → NGINX (SSL termination, load balancing) → Backend services (HTTP)

Load Balancing Algorithms

- round-robin (default): Even distribution - least_conn: Sends to server with fewest active connections - ip_hash: Sticky sessions based on client IP - random two least_conn: Picks 2 random servers, sends to one with fewer connections (good for large clusters)

SSL Termination

NGINX handles TLS handshake and decryption. Backend communication is plain HTTP (faster, simpler cert management). For security-sensitive environments, use mTLS to backends.

Health Checks

- Open source: Passive only — marks server as failed after N consecutive errors - NGINX Plus: Active health checks — probes backends periodically - Use max_fails and fail_timeout to tune failure detection

Essential Proxy Headers

- X-Real-IP: Client's actual IP (not NGINX's) - X-Forwarded-For: Full proxy chain - X-Forwarded-Proto: Original protocol (https) - Host: Preserve original Host header for virtual hosting

Code Example

upstream api_backend {
    least_conn;
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.13:8080 backup;  # Only used when others fail
    
    keepalive 32;  # Connection pooling to backends
}

server {
    listen 443 ssl http2;
    server_name api.example.com;
    
    # SSL configuration
    ssl_certificate     /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # Security headers
    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Content-Type-Options nosniff;
    
    location /api/ {
        proxy_pass http://api_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Timeouts
        proxy_connect_timeout 5s;
        proxy_read_timeout 30s;
        proxy_send_timeout 10s;
        
        # Connection reuse
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name api.example.com;
    return 301 https://$host$request_uri;
}

💡

Interview Tip

Include keepalive connections to backends (proxy_http_version 1.1 + Connection header) — this is a major performance optimization most candidates miss. Also mention the backup server for resilience.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Use limit_req_zone for request rate limiting (leaky bucket algorithm) and limit_conn_zone for concurrent connection limiting. Configure burst and nodelay for handling traffic spikes without dropping legitimate requests.

Detailed Answer

Rate Limiting (limit_req)

Uses the leaky bucket algorithm. Requests arrive at variable rate but are processed at a fixed rate. Excess requests queue up (burst) or are rejected (503).

Connection Limiting (limit_conn)

Limits the number of simultaneous connections from a single IP or per server.

Key Parameters

- rate: Requests per second (e.g., 10r/s) - burst: Size of the overflow queue - nodelay: Process burst requests immediately rather than queuing - delay: Process first N burst requests immediately, queue the rest

Best Practices

- Apply different limits to different endpoints (strict for login, relaxed for static) - Use $binary_remote_addr (16 bytes) instead of $remote_addr (7-15 bytes) for memory efficiency - Return 429 (Too Many Requests) instead of default 503 - Log rate-limited requests for analysis - Consider using geo module to whitelist internal IPs

Code Example

# Rate limiting zones (defined in http block)
http {
    # 10 requests/second per IP, 10MB shared memory
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    
    # Strict limit for auth endpoints
    limit_req_zone $binary_remote_addr zone=login:10m rate=1r/s;
    
    # Connection limit
    limit_conn_zone $binary_remote_addr zone=addr:10m;
    
    # Return 429 instead of 503
    limit_req_status 429;
    limit_conn_status 429;
    
    server {
        # General API: 10 req/s with burst of 20
        location /api/ {
            limit_req zone=api burst=20 delay=10;
            # First 10 burst requests processed immediately,
            # next 10 queued, beyond 30 total → 429
            limit_conn addr 50;  # Max 50 concurrent connections per IP
            proxy_pass http://backend;
        }
        
        # Login: strict 1 req/s
        location /api/auth/login {
            limit_req zone=login burst=5 nodelay;
            proxy_pass http://backend;
        }
        
        # Whitelist internal IPs
        geo $limit {
            default 1;
            10.0.0.0/8 0;    # Internal network
            172.16.0.0/12 0;  # Internal network
        }
        map $limit $limit_key {
            0 "";
            1 $binary_remote_addr;
        }
        limit_req_zone $limit_key zone=external:10m rate=5r/s;
    }
}

💡

Interview Tip

Explain the leaky bucket algorithm and the difference between burst, nodelay, and delay. The whitelist pattern using geo + map is an advanced technique that impresses interviewers.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

NGINX Ingress: flexible L7 routing, widely adopted, good for most use cases. Istio Gateway: when using service mesh, advanced traffic management. AWS ALB: tight AWS integration, native WAF/Shield support.

Detailed Answer

NGINX Ingress Controller

- Most popular K8s ingress controller - Full NGINX configuration via annotations and ConfigMaps - L7 routing: host-based, path-based, header-based - SSL termination, rate limiting, basic auth, canary deployments - Works on any infrastructure (cloud, on-prem, edge) - Best for: General-purpose ingress, teams familiar with NGINX, multi-cloud

Istio Gateway

- Part of Istio service mesh — requires full mesh deployment - VirtualService + Gateway resources for routing - Advanced traffic management: mirroring, fault injection, retries - Automatic mTLS, authorization policies, observability - Best for: Teams already using Istio, need service mesh features at the ingress layer, complex traffic routing (A/B testing, canary with metric-based promotion)

AWS ALB Ingress Controller

- Provisions actual AWS Application Load Balancers - Native integration: WAF, Shield, Cognito auth, Certificate Manager - Target group binding directly to pods (IP mode) or NodePort - Cost: Per ALB + per LCU pricing (can get expensive with many ingresses) - Best for: AWS-only deployments, need WAF/Shield, compliance requirements, teams preferring AWS-native services

Decision Matrix

- Multi-cloud → NGINX Ingress - Already on Istio → Istio Gateway - AWS-only + need WAF → ALB Ingress - Cost-sensitive → NGINX Ingress (one deployment handles all routes)

Code Example

# NGINX Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  tls:
  - hosts: [api.example.com]
    secretName: api-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-v2
            port: { number: 80 }

# Istio Gateway + VirtualService
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port: { number: 443, name: https, protocol: HTTPS }
    tls:
      mode: SIMPLE
      credentialName: api-tls
    hosts: [api.example.com]
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  hosts: [api.example.com]
  gateways: [api-gateway]
  http:
  - route:
    - destination: { host: api-v1 }
      weight: 90
    - destination: { host: api-v2 }
      weight: 10

💡

Interview Tip

Frame your answer around the decision matrix, not just feature lists. Show you can make a recommendation based on the specific context (cloud provider, existing stack, team expertise).

Open Full Page ↗

💬 Comments

⚡

Quick Answer

A single upstream URL proxies to one backend; an upstream block with multiple servers load-balances across them.

Detailed Answer

reverse proxy: proxy_pass to one backend to terminate TLS, cache, or route. Load balancing: define an upstream with several servers and proxy_pass to it; NGINX distributes requests (round-robin by default, or least_conn, ip_hash) and can health-check and mark servers down.

Code Example

upstream app { least_conn; server a:8080; server b:8080; }
location / { proxy_pass http://app; }

💡

Interview Tip

upstream{} block is the load-balancing mechanism.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Exact (=) wins, then longest prefix with ^~ (stops regex), then regex (~/~*) in order, then plain longest prefix.

Detailed Answer

NGINX picks the most specific match: = exact match first; ^~ prefix suppresses regex checks; regex matches are tried in file order; otherwise the longest matching prefix location is used. Misordered locations are a top source of 'wrong handler' bugs.

💡

Interview Tip

Know the precedence order — a classic gotcha.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Set proxy_set_header for Host and X-Forwarded-* so backends see the original client and host.

Detailed Answer

Behind a proxy, backends otherwise see NGINX's IP and the internal host. proxy_set_header Host $host and X-Forwarded-For $proxy_add_x_forwarded_for preserve the client's host and IP chain; X-Forwarded-Proto conveys the original scheme for correct redirects and cookies.

💡

Interview Tip

Forgetting these breaks client IP logging and HTTPS redirects.

Open Full Page ↗

💬 Comments

Context

An edge fleet whose NGINX configs were hand-edited per host over years: cert renewals were calendar reminders, config drift between 'identical' hosts caused behavior differences under failover, and two outages in a year traced to a typo'd reload on a Friday.

Problem

No review, no rollback, no consistency: each host was subtly unique, cert expiry near-misses were quarterly ritual, and nginx -t on the box after editing was the entire safety process.

Solution

Moved the fleet to config-as-code: one templated config repo (environment/host-class variables rendered per host), CI validating every change (nginx -t in a container matching the fleet's exact version, plus a behavioral smoke suite hitting a rendered test instance with golden requests — routing, headers, redirects asserted), deployment via config management with canary host classes first, and reload-not-restart semantics verified (SIGHUP graceful reload; worker draining confirmed). Certificates moved to ACME automation with OCSP-stapling and expiry monitoring at 30/14/7 days paging progressively.

Commands

CI: docker run nginx:1.26 nginx -t -c /rendered/nginx.conf && pytest smoke/ --base-url http://test-render:8080

deploy: ansible-playbook edge.yml --limit canary && sleep soak && --limit all

certs: certbot renew --deploy-hook 'nginx -s reload'; alert on expiry < 14d

📈

Outcome

Config drift eliminated (hosts render from one source); the typo'd-reload outage class ended (CI catches syntax and behavior before any host sees the change); cert expiry incidents went from near-misses to a monitored non-event; and onboarding a new edge host became a 10-minute inventory entry.

💡

Lessons Learned

The behavioral smoke suite caught what nginx -t never could — a location-block precedence change that parsed fine and routed wrong. Canary host classes matter even for 'trivial' changes; the one rollback since came from a header change interacting with an old client.

#nginx#config-as-code#tls-automation#canary#validation

💬 Comments

Context

A media company's legacy CMS (single-writer PHP monolith, no horizontal scaling path) facing predictable but extreme traffic spikes around content drops — historically survived by over-provisioning and prayer.

Problem

The CMS origin maxed at ~400 rps; launch spikes hit 15K rps; a full re-platform was 18 months away; and the two prior launches had produced outages with executive attention.

Solution

Built an NGINX caching tier: proxy_cache with cache keys normalized (query-param whitelist, cookie stripping for anonymous traffic), stale-while-revalidate and stale-on-error (proxy_cache_use_stale) so origin blips serve slightly-old pages instead of errors, proxy_cache_lock collapsing thundering-herd revalidations to one origin fetch per key, short TTLs (30s) on HTML with long TTLs on assets, and an explicit purge path for editorial corrections. Cache-hit ratio, origin rps, and stale-serve counts became the launch-day dashboard.

Commands

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=cms:512m inactive=1h max_size=20g;

location / {proxy_cache cms; proxy_cache_valid 200 30s; proxy_cache_use_stale error timeout updating http_500 http_502; proxy_cache_lock on; proxy_cache_key $scheme$host$uri$is_args$filtered_args;}

launch dashboard: hit_ratio, origin_rps, stale_served, cache_lock_waits

📈

Outcome

The next launch peaked at 22K rps with the origin seeing a steady ~180 rps (99.2% hit ratio); a 90-second origin brownout mid-launch was invisible to users (stale-on-error served); no outage, no executive attention. The tier now permanently fronts the CMS, and the re-platform lost its 'emergency' status.

💡

Lessons Learned

Cache-key normalization was 80% of the hit ratio — raw query strings and cookies had been fragmenting identical pages into thousands of cache entries. proxy_cache_lock is the difference between a cache and a thundering-herd amplifier at TTL expiry.

#nginx#proxy-cache#stale-while-revalidate#thundering-herd#traffic-spikes

💬 Comments

Context

A public web application runs on several Node/Java/Python app servers on private subnets. Clients must never reach those servers directly.

Problem

Exposing app servers publicly leaks internal topology, forces every app to implement TLS, connection limits and header hygiene, and makes it impossible to change the backend without changing DNS.

Solution

Put NGINX at the edge as the single public entry point. It accepts client connections, terminates them, and forwards requests upstream over the private network with proxy_pass. Set X-Forwarded-For / X-Forwarded-Proto so the app still sees the real client IP and scheme, and keep the app bound to a private interface only.

Commands

location / {
    proxy_pass http://app_backend;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

nginx -t && nginx -s reload

📈

Outcome

One public entry point, internal servers hidden, and backends can be moved, renamed or rescaled without any client-visible change.

💡

Lessons Learned

Always set proxy_set_header Host and X-Forwarded-*; without them apps generate wrong redirect URLs and log the proxy IP for every request. Trust those headers only from your own proxy — otherwise clients can spoof their source IP.

#nginx#reverse-proxy#proxy_pass#architecture

💬 Comments

Context

Traffic has grown past what one app server can handle, and deploys need to happen without downtime.

Problem

A single backend is both a capacity ceiling and a single point of failure. Sending all traffic to one server also makes rolling deploys visible to users.

Solution

Define an upstream block with several servers and let NGINX distribute requests. Choose the algorithm deliberately: round-robin by default, least_conn when request durations vary widely, or ip_hash / hash when you need affinity. Mark servers with max_fails and fail_timeout so a sick backend is taken out of rotation automatically.

Commands

upstream app_backend {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=15s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=15s;
    server 10.0.1.13:8080 backup;
    keepalive 32;
}

location / {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

📈

Outcome

Traffic spreads across the pool, one dead backend no longer causes an outage, and instances can be drained one at a time for zero-downtime deploys.

💡

Lessons Learned

Open-source NGINX only does passive health checks — it notices failures from real traffic. Active health checks are an NGINX Plus feature; on OSS, pair max_fails with a real orchestrator health check. Also raise keepalive in the upstream block or you pay a new TCP handshake on every proxied request.

#nginx#load-balancing#upstream#high-availability

💬 Comments

Context

A single-page app ships JS/CSS bundles, fonts and images alongside a dynamic API.

Problem

Serving static files through the application runtime wastes a worker thread per file, adds latency, and scales the expensive tier for work that needs no logic at all.

Solution

Let NGINX serve static paths straight from disk with root/alias, and proxy only dynamic routes. Add long cache lifetimes with immutable for content-hashed filenames, enable sendfile and tcp_nopush for efficient kernel-level file transfer, and use try_files to fall back to index.html for client-side routing.

Commands

location /assets/ {
    root /var/www/app;
    access_log off;
    expires 1y;
    add_header Cache-Control "public, immutable";
}

location / {
    try_files $uri $uri/ /index.html;
}

sendfile on;
tcp_nopush on;

📈

Outcome

Static requests never touch the application, backend load drops sharply, and asset latency falls to disk-read speed.

💡

Lessons Learned

Cache-Control immutable is only safe with content-hashed filenames. Applying a one-year TTL to /index.html is the classic mistake — users then get a stale shell that references deleted bundles.

#nginx#static-files#caching#performance

💬 Comments

Context

Every public endpoint must be HTTPS, with certificates rotated automatically, while internal services speak plain HTTP on a trusted network.

Problem

Implementing TLS in each application means duplicated certificate handling, inconsistent cipher policy, and a rotation problem that multiplies with every new service.

Solution

Terminate TLS once at NGINX. Keep TLS 1.2/1.3 only, use a modern cipher suite, enable OCSP stapling and HSTS, and let ACME/certbot handle renewal. Forward the original scheme upstream with X-Forwarded-Proto so applications can still build correct https:// URLs and set Secure cookies.

Commands

server {
    listen 443 ssl;
    http2 on;
    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_stapling on;
    ssl_stapling_verify on;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
}

openssl s_client -connect example.com:443 -servername example.com < /dev/null | openssl x509 -noout -dates

📈

Outcome

Certificate and cipher policy lives in one place, renewals are automated, and app servers are freed from TLS entirely.

💡

Lessons Learned

Without X-Forwarded-Proto the app thinks traffic is plain HTTP and will emit http:// redirects, causing infinite redirect loops. If any hop after NGINX crosses an untrusted network, terminate is not enough — re-encrypt upstream with proxy_ssl.

#nginx#tls#https#certificates#security

💬 Comments

Context

A read-heavy catalogue API returns the same responses to thousands of clients, and the database is the bottleneck.

Problem

Every identical request costs a full round trip through the app and database, so traffic spikes translate directly into database load and rising latency.

Solution

Enable proxy_cache with a shared memory zone and disk path. Cache successful responses for a defined period, key on the parts of the request that actually vary, and use proxy_cache_use_stale so cached content is still served when the backend is down or slow. Add proxy_cache_lock so a cache miss does not stampede the origin with duplicate requests.

Commands

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=60m;

location /api/ {
    proxy_cache api_cache;
    proxy_cache_key "$scheme$request_method$host$request_uri";
    proxy_cache_valid 200 301 302 10m;
    proxy_cache_valid 404 1m;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503;
    proxy_cache_lock on;
    add_header X-Cache-Status $upstream_cache_status;
    proxy_pass http://app_backend;
}

📈

Outcome

Cache hits are returned in microseconds without touching the backend, origin load drops dramatically, and the site survives backend outages by serving stale content.

💡

Lessons Learned

proxy_cache_use_stale error timeout updating turns a backend outage into a degraded-but-serving site, and it is the single highest-value cache directive. Never cache authenticated responses without including the auth context in the cache key — leaking one user's response to another is the classic caching incident.

#nginx#caching#proxy_cache#performance

💬 Comments

Context

A public API is hit by scrapers and by a misbehaving client retrying in a tight loop, starving legitimate users.

Problem

Without limits, one client can consume all backend capacity. Login endpoints are also exposed to credential-stuffing at full speed.

Solution

Define a limit_req_zone keyed on client IP (or API key) and apply limit_req to the routes that need protection. Use the burst parameter with nodelay to allow short legitimate bursts while still capping the sustained rate, and apply a much tighter limit to authentication endpoints than to general reads.

Commands

limit_req_zone $binary_remote_addr zone=api_rl:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=login_rl:10m rate=5r/m;

location /api/ {
    limit_req zone=api_rl burst=20 nodelay;
    limit_req_status 429;
    proxy_pass http://app_backend;
}

location /login {
    limit_req zone=login_rl burst=3;
    proxy_pass http://app_backend;
}

📈

Outcome

Sustained abuse is throttled at the edge before it reaches the application, while normal bursty browsing is unaffected.

💡

Lessons Learned

burst without nodelay queues and delays requests, which can look like a latency incident; burst with nodelay rejects immediately once the bucket is empty. Behind a CDN or another proxy, $binary_remote_addr is the proxy IP — key on the real client IP from X-Forwarded-For or you will rate limit the whole internet as one user.

#nginx#rate-limiting#limit_req#security#ddos

💬 Comments

Context

A platform runs separate services for the web app, the API, authentication and an admin console, but must present a single hostname to users.

Problem

Giving each service its own hostname or port multiplies certificates, CORS configuration and firewall rules, and leaks the internal service layout to clients.

Solution

Use location blocks to route by path prefix and server blocks to route by hostname. Understand the matching order — exact (=), prefix (^~), regex (~), then plain prefix — because surprising matches are almost always a location precedence problem. Route each prefix to its own upstream.

Commands

location /api/  { proxy_pass http://api_backend/; }
location /auth/ { proxy_pass http://auth_backend/; }
location /admin/ {
    allow 10.0.0.0/8;
    deny  all;
    proxy_pass http://admin_backend/;
}

server {
    server_name api.example.com;
    location / { proxy_pass http://api_backend; }
}

📈

Outcome

One public endpoint and one certificate front the whole estate, and services can be split or merged behind the proxy without clients noticing.

💡

Lessons Learned

A trailing slash on proxy_pass changes behaviour: proxy_pass http://svc/ strips the location prefix, proxy_pass http://svc keeps it. Getting this wrong produces doubled or missing path segments and is the most common routing bug.

#nginx#routing#location#microservices

💬 Comments

Context

A chat and live-notification feature uses WebSockets, but connections drop roughly every minute once placed behind the proxy.

Problem

WebSockets begin as an HTTP request that is then upgraded. NGINX defaults to HTTP/1.0 upstream and strips hop-by-hop headers, so the upgrade never completes; and the default 60s proxy_read_timeout kills idle connections that are perfectly healthy.

Solution

Force proxy_http_version 1.1, pass the Upgrade and Connection headers explicitly, and raise proxy_read_timeout and proxy_send_timeout well beyond the application heartbeat interval. Use a map so the same location still works for ordinary HTTP requests.

Commands

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

location /ws/ {
    proxy_pass http://ws_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade    $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

📈

Outcome

Bidirectional connections stay open for hours, and the same location block serves both WebSocket and regular HTTP traffic.

💡

Lessons Learned

The "disconnects every 60 seconds" symptom is almost always proxy_read_timeout, not application code. Also make sure the app sends heartbeats/pings more often than the timeout, and prefer ip_hash or a shared pub/sub backplane so a reconnect can land on any node.

#nginx#websocket#proxy#real-time

💬 Comments

Context

A dozen microservices each need to validate the same session token or JWT before serving a request.

Problem

Re-implementing token validation in every service guarantees drift — one service validates expiry but not signature, another caches decisions too long — and a policy change means redeploying everything.

Solution

Use auth_request to call a central auth service on every incoming request. NGINX forwards the request to the auth endpoint; a 2xx allows the request through, 401/403 rejects it. Copy identity claims back onto the upstream request with auth_request_set so downstream services receive a trusted, pre-validated user id instead of parsing tokens themselves.

Commands

location /api/ {
    auth_request /_auth;
    auth_request_set $user_id $upstream_http_x_user_id;
    proxy_set_header X-User-Id $user_id;
    proxy_pass http://api_backend;
}

location = /_auth {
    internal;
    proxy_pass http://auth_service/verify;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URI $request_uri;
}

📈

Outcome

Authentication policy lives in one service, downstream code simplifies to trusting a header, and policy changes ship without touching every microservice.

💡

Lessons Learned

auth_request adds a synchronous hop to every request, so the auth service must be fast and highly available or it becomes the bottleneck for the whole platform — cache decisions where the token lifetime allows. Critically, downstream services must not be reachable except through the gateway, or the injected identity header can be forged directly.

#nginx#api-gateway#auth_request#jwt#security

💬 Comments

Context

A risky release needs to reach 5% of users first so errors can be caught before a full rollout.

Problem

Deploying to everyone at once turns a bad release into a full outage, and rolling back still means every user saw the broken version.

Solution

Split traffic at the proxy. split_clients hashes a stable key (session cookie or client IP) into percentage buckets so the same user consistently gets the same version. Alternatively weight servers inside an upstream. For blue-green, keep both stacks running and flip the upstream in one config reload.

Commands

split_clients "${remote_addr}${http_user_agent}" $app_version {
    5%      canary;
    *       stable;
}

upstream stable { server 10.0.1.11:8080; }
upstream canary { server 10.0.2.11:8080; }

location / {
    proxy_pass http://$app_version;
}

# weighted alternative
upstream app { server 10.0.1.11:8080 weight=19; server 10.0.2.11:8080 weight=1; }

📈

Outcome

A new version is exposed to a small, consistent slice of traffic; a bad release is contained and reverted with a config reload rather than a redeploy.

💡

Lessons Learned

Hash on a sticky key, not on the request — hashing per request flips users between versions mid-session and produces bizarre bug reports. Always watch error rate and latency for the canary bucket specifically; an overall dashboard hides a 5% cohort failing completely.

#nginx#canary#blue-green#deployment#split_clients

💬 Comments

Context

A content-heavy site serves large JSON payloads and JavaScript bundles to users on mobile networks.

Problem

Uncompressed text payloads waste bandwidth and make first paint slow on poor connections, and compressing inside the application burns CPU on the expensive tier.

Solution

Enable gzip (or Brotli where the module is available) at the proxy for text content types. Set a sensible compression level, skip tiny responses where the overhead is not worth it, and pre-compress static assets at build time with gzip_static so NGINX serves a ready-made .gz rather than compressing on every request.

Commands

gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_vary on;
gzip_proxied any;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;

gzip_static on;  # serve prebuilt file.js.gz when present

📈

Outcome

Text payloads shrink by roughly 70-80%, pages render faster on slow links, and egress costs fall.

💡

Lessons Learned

Never gzip already-compressed formats (JPEG, PNG, MP4, zip) — you burn CPU for no gain. Compression level 9 costs far more CPU than level 5 for a couple of percent; level 4-6 is the practical sweet spot. Also add gzip_vary so caches do not serve compressed bytes to clients that cannot decode them.

#nginx#compression#gzip#performance

💬 Comments

Context

The site must be HTTPS-only, but users still type bare hostnames and old links point at http://.

Problem

Plain HTTP requests are readable and modifiable in transit, and the initial redirect itself is an opportunity for downgrade attacks.

Solution

Keep a minimal port-80 server block that does nothing but issue a 301 to the HTTPS equivalent, while still allowing the ACME challenge path through for certificate renewal. Then send HSTS on the HTTPS response so browsers refuse plain HTTP on subsequent visits without even making the request.

Commands

server {
    listen 80;
    server_name example.com;

    location /.well-known/acme-challenge/ { root /var/www/certbot; }
    location / { return 301 https://$host$request_uri; }
}

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

📈

Outcome

All traffic converges on HTTPS, and returning browsers skip the insecure hop entirely.

💡

Lessons Learned

Blanket-redirecting port 80 breaks HTTP-01 certificate renewal unless /.well-known/acme-challenge/ is excluded — a classic cause of expiry outages 90 days after launch. Introduce HSTS with a short max-age first: a long max-age with includeSubDomains is effectively irreversible for the duration if a subdomain cannot do TLS.

#nginx#https#redirect#hsts#security

💬 Comments

Context

Internal services communicate over gRPC and need a single TLS-terminating entry point for external clients.

Problem

gRPC runs over HTTP/2 with long-lived multiplexed streams, so a standard proxy_pass configuration breaks it — HTTP/1.1 proxying destroys the framing entirely.

Solution

Use grpc_pass instead of proxy_pass and make sure HTTP/2 is enabled on the listener. Map gRPC services onto locations by their fully-qualified package path. Use grpc_ssl_* directives if the upstream also expects TLS, and set generous timeouts for streaming RPCs.

Commands

server {
    listen 443 ssl;
    http2 on;

    location /helloworld.Greeter/ {
        grpc_pass grpc://grpc_backend;
    }
}

upstream grpc_backend { server 10.0.1.21:50051; server 10.0.1.22:50051; }

grpcurl -insecure example.com:443 list

📈

Outcome

External gRPC clients reach internal services through one TLS endpoint, with load balancing across backends.

💡

Lessons Learned

HTTP/2 multiplexes many streams over one connection, so connection-level round-robin balances connections rather than requests — one chatty client can pin all its load to a single backend. Where that matters you need request-level balancing from a service mesh or NGINX Plus.

#nginx#grpc#http2#microservices

💬 Comments

Context

Dozens of services in a Kubernetes cluster need external HTTP access, TLS and path routing, without a cloud load balancer per service.

Problem

A LoadBalancer Service per application is expensive and slow to provision, and gives no shared place to apply routing, TLS or rate-limit policy.

Solution

Run ingress-nginx as the cluster ingress controller behind one cloud load balancer. It watches Ingress resources and rewrites its own NGINX configuration automatically. Per-service behaviour is expressed as annotations, and TLS certificates come from Secrets, typically managed by cert-manager.

Commands

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    nginx.ingress.kubernetes.io/rate-limit-rps: "10"
    nginx.ingress.kubernetes.io/proxy-body-size: "20m"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [app.example.com]
      secretName: app-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app
                port:
                  number: 80

kubectl -n ingress-nginx logs deploy/ingress-nginx-controller

📈

Outcome

One load balancer fronts the whole cluster, routing is declarative and version-controlled, and certificates renew automatically.

💡

Lessons Learned

Annotations are controller-specific — ingress-nginx (community) and NGINX Inc's controller are different projects with incompatible annotations, a frequent source of silently ignored config. Watch for reload storms: very high Ingress churn makes the controller rewrite and reload config constantly, which shows up as periodic latency spikes.

#nginx#kubernetes#ingress#cert-manager

💬 Comments

Context

A small number of hosts open thousands of simultaneous connections, exhausting worker slots and file descriptors.

Problem

Request-rate limiting alone does not stop connection exhaustion — a client can hold many open connections while sending very few requests, which is exactly how slow-request denial of service works.

Solution

Add limit_conn_zone keyed on client address and cap concurrent connections per client. Combine with client body and header timeouts so half-open or trickling requests are dropped, and cap request body size to stop oversized upload abuse.

Commands

limit_conn_zone $binary_remote_addr zone=perip:10m;

server {
    limit_conn perip 20;
    client_body_timeout   10s;
    client_header_timeout 10s;
    client_max_body_size  20m;
    send_timeout          10s;
}

📈

Outcome

A single source can no longer monopolise worker connections, and slow-drip attacks are cut off before they consume capacity.

💡

Lessons Learned

limit_conn and limit_req solve different problems and you generally want both. Set the connection cap with real user behaviour in mind: browsers legitimately open 6+ connections per host, and HTTP/2 clients behave differently again, so an aggressive cap breaks normal users.

#nginx#limit_conn#ddos#security#slowloris

💬 Comments

Context

A partner-facing API must only accept calls from a known set of client systems, with no shared bearer tokens.

Problem

API keys leak, are copied between environments, and are hard to rotate across external partners. Network allowlisting alone fails when partners use dynamic egress addresses.

Solution

Require client certificates at the edge. NGINX verifies the presented certificate against a trusted CA, rejects anything unsigned or expired, and passes the verified subject upstream so the application can authorise on identity rather than on a secret. Publish a CRL or use short-lived certificates so revocation actually works.

Commands

server {
    listen 443 ssl;
    ssl_client_certificate /etc/nginx/certs/partner-ca.pem;
    ssl_crl                /etc/nginx/certs/partner.crl;
    ssl_verify_client on;
    ssl_verify_depth 2;

    location /partner-api/ {
        proxy_set_header X-Client-DN     $ssl_client_s_dn;
        proxy_set_header X-Client-Verify $ssl_client_verify;
        proxy_pass http://partner_backend;
    }
}

📈

Outcome

Only holders of a valid, CA-signed client certificate reach the API, and identity is cryptographic rather than a copyable string.

💡

Lessons Learned

ssl_verify_client optional (rather than on) lets you roll mTLS out gradually and log who is not yet compliant before enforcing. Certificate expiry is the top cause of partner outages — monitor client cert expiry as actively as your own server certificates.

#nginx#mtls#client-certificates#security#zero-trust

💬 Comments

Context

A database migration requires a planned window where the application cannot serve requests.

Problem

With the backend down, users see a raw 502 Bad Gateway, which looks like an outage rather than planned work, and monitoring cannot tell the two apart.

Solution

Serve a friendly static maintenance page whenever a flag file exists, returning HTTP 503 with a Retry-After header so crawlers and clients understand the state is temporary. Keep an allowlist so engineers can still reach the app to verify the migration. Define custom error pages for 502/504 so unplanned failures are also presentable.

Commands

location / {
    if (-f /etc/nginx/maintenance.on) {
        return 503;
    }
    proxy_pass http://app_backend;
}

error_page 503 @maintenance;
location @maintenance {
    root /var/www/maintenance;
    rewrite ^ /index.html break;
    add_header Retry-After 3600 always;
}

touch /etc/nginx/maintenance.on   # enable
rm  /etc/nginx/maintenance.on       # disable

📈

Outcome

Planned windows show a clear branded message with the correct status code, while the team retains access to validate before reopening.

💡

Lessons Learned

Return 503 with Retry-After, never 200 — a maintenance page served as 200 gets cached and indexed by search engines as your actual content. Keep the maintenance page fully self-contained: if it references assets from the backend that is down, it renders broken.

#nginx#maintenance#error-pages#operations

💬 Comments

Context

A browser front-end on app.example.com calls an API on api.example.com and is blocked by cross-origin policy.

Problem

Each backend service implementing its own CORS headers leads to inconsistent policies, duplicated preflight handling, and the notorious "multiple values in Access-Control-Allow-Origin" error when both proxy and app add headers.

Solution

Handle CORS once at the proxy. Answer OPTIONS preflight requests directly with a 204 so they never reach the application, and add the CORS response headers to actual responses. Echo only origins from an explicit allowlist rather than reflecting whatever the client sent.

Commands

map $http_origin $cors_ok {
    default                        "";
    "https://app.example.com"      $http_origin;
    "https://admin.example.com"    $http_origin;
}

location /api/ {
    if ($request_method = OPTIONS) {
        add_header Access-Control-Allow-Origin  $cors_ok always;
        add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" always;
        add_header Access-Control-Allow-Headers "Authorization, Content-Type" always;
        add_header Access-Control-Max-Age       86400 always;
        return 204;
    }
    add_header Access-Control-Allow-Origin      $cors_ok always;
    add_header Access-Control-Allow-Credentials true always;
    proxy_pass http://api_backend;
}

📈

Outcome

Consistent cross-origin policy across every service, preflight latency removed from the backend, and one place to change the allowlist.

💡

Lessons Learned

Never reflect arbitrary origins with credentials enabled — Access-Control-Allow-Origin: $http_origin plus Allow-Credentials: true lets any site make authenticated requests on a user's behalf. If the app also emits CORS headers you get duplicates that browsers reject outright, so pick one layer and disable the other.

#nginx#cors#api#browser

💬 Comments

Context

A legacy application keeps session state in local server memory, so users are logged out whenever a request lands on a different backend.

Problem

Plain round-robin balancing scatters a user's requests across the pool, and any backend without that session treats the user as anonymous.

Solution

Pin each client to a backend. ip_hash derives the backend from the client address; the more flexible hash directive can key on a cookie or header with the consistent flag, so adding or removing a server only remaps a fraction of users instead of reshuffling everyone.

Commands

upstream app_backend {
    ip_hash;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

upstream app_backend {
    hash $cookie_sessionid consistent;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

📈

Outcome

Users stay on one backend for the life of their session, and scaling events disturb only a small share of traffic.

💡

Lessons Learned

Affinity is a workaround, not an architecture — the real fix is externalising session state to Redis or a database so any node can serve any request. Note ip_hash breaks badly behind a CDN or corporate NAT where thousands of users share one source address, and consistent hashing matters a lot during autoscaling.

#nginx#sticky-sessions#ip_hash#load-balancing

💬 Comments

Context

Mobile users on poor connections upload files, and each slow upload occupies an application worker for the entire transfer.

Problem

Application servers typically have a small pool of expensive workers. If a worker is held open for the duration of a slow client transfer, a few hundred slow clients can exhaust the pool while the CPU sits idle.

Solution

Let NGINX absorb client slowness. With proxy_buffering and proxy_request_buffering enabled, NGINX reads the full request from the slow client itself, then delivers it to the backend at local network speed, and similarly buffers the response. The backend worker is occupied only for actual processing time.

Commands

proxy_buffering on;
proxy_request_buffering on;
proxy_buffers 16 16k;
proxy_buffer_size 32k;
proxy_busy_buffers_size 64k;

location /events {
    proxy_pass http://app_backend;
    proxy_buffering off;   # SSE / streaming must not be buffered
    proxy_cache off;
}

📈

Outcome

Backend workers are freed from network wait, so far fewer are needed to serve the same traffic and slow clients stop causing saturation.

💡

Lessons Learned

Buffering is the right default but must be disabled for streaming responses — server-sent events, log tailing and progress endpoints appear frozen if NGINX holds the response until it is complete. Set proxy_buffering off for those specific locations only. Watch for buffers spilling to disk under load, which quietly turns a memory path into an I/O bottleneck.

#nginx#buffering#performance#streaming

💬 Comments

HighResolution time: ~20-60 min for an experienced operator

Symptom

P95 latency rose from 180 ms to 2.4 seconds and one upstream instance carried 70% of traffic.

Error Message

upstream timed out (110: Operation timed out) while reading response header from upstream

Root Cause

The upstream group used default round-robin behavior, but one backend was slow rather than fully down. Long-lived keepalive connections made the imbalance persistent, and passive failure thresholds were too loose to mark the backend unavailable quickly. The underlying issue was a combination of factors that individually seemed harmless but together created a cascading failure. The monitoring that should have caught this early was either missing or configured with thresholds too high to trigger before user impact. The on-call engineer initially pursued the wrong hypothesis because the symptoms resembled a different, more common failure mode. By the time the real root cause was identified, the blast radius had expanded beyond the original service to affect downstream dependencies. This incident exposed a gap in the team's runbooks and highlighted the need for better correlation between metrics, logs, and traces during diagnosis.

Diagnosis Steps

1.nginx -T # Dumps active upstream and timeout configuration
2.grep "upstream timed out" /var/log/nginx/error.log # Confirms upstream timeout pattern
3.curl -s http://127.0.0.1/nginx_status # Checks active connection behavior when stub_status is enabled

Solution

Lower upstream timeouts, tune `max_fails` and `fail_timeout`, and remove the degraded upstream if needed. Reload NGINX after validation rather than restarting and dropping active connections.

Commands

nginx -t

nginx -s reload

🛡

Prevention

Monitor per-upstream latency, errors, and active connections. Use active health checks where available. Keep timeout values aligned with application SLOs.

nginxupstreamload-balancingtimeouts

◈ Architecture Diagram

┌──────────┐
│ Trigger  │
└────┬─────┘
     ↓
┌──────────┐
│ Incident │
└────┬─────┘
     ↓
┌──────────┐
│ Verify   │
└──────────┘

💬 Comments

MediumResolution time: ~One OOM outage plus a morning of ps archaeology; the timeout fix is one directive once understood

Symptom

An edge host's memory climbs over days until the OOM killer takes NGINX down entirely. ps shows dozens of nginx worker processes 'shutting down' — some a week old — each holding gigabytes collectively. The team reloads config frequently (multiple deploys daily).

Error Message

kernel: Out of memory: Killed process (nginx). ps output: 40+ workers in 'nginx: worker process is shutting down' state, oldest 9 days, each pinned by established WebSocket connections.

Root Cause

Graceful reload (SIGHUP) spawns new workers and asks old ones to finish existing connections — but WebSocket/long-poll connections live for days, so every reload stranded another worker generation that couldn't exit until its last socket closed. Multiple deploys daily x long-lived connections = generations accumulating faster than they drained, each holding full worker memory. The design assumption (connections end quickly) didn't match the workload.

Diagnosis Steps

1.ps aux | grep 'shutting down' — count and age of stranded generations tells the story
2.Correlate generation birth times with the deploy/reload log — one generation per reload
3.Sample a stranded worker's connections (ss -tp | grep <pid>): established WebSockets days old
4.Confirm client reconnect capability before choosing a shutdown timeout

Solution

Set worker_shutdown_timeout (e.g. 6h) so draining workers force-close remaining connections after a bounded grace period — WebSocket clients already had reconnect logic, so bounded disconnection was acceptable and negotiated with app teams. Reduced reload frequency by batching config deploys, and added a metric on 'shutting down' worker count with an alert threshold. Clients got jittered reconnect to avoid synchronized reconnect storms at the timeout boundary.

Commands

ps -eo pid,etime,cmd | grep 'worker process is shutting down' | wc -l

nginx.conf: worker_shutdown_timeout 6h;

alert: nginx_draining_workers > 6

🛡

Prevention

Any NGINX fronting long-lived connections needs worker_shutdown_timeout set deliberately, a drain-generation count monitor, and client reconnect behavior validated (jitter, backoff). Reload frequency is a real cost under long-lived traffic — batch config changes rather than reloading per tweak.

nginxreloadwebsocketsworker-drainoom

💬 Comments

MediumResolution time: ~3 hours — the partial-fleet symptom pattern (reload-time dependent) was the key clue nobody saw immediately

Symptom

After a backend service migrated infrastructure (new load balancer, new IPs, DNS updated), one NGINX tier keeps sending traffic to the old, now-black-holed IP — but only some NGINX hosts, and only until they happen to be reloaded. Errors are timeouts, not refusals, making it look like a network issue.

Error Message

upstream timed out (110: Connection timed out) while connecting to upstream, upstream: 'http://10.40.3.21:8080/...' — an IP that DNS no longer returns and that answers nothing.

Root Cause

proxy_pass http://backend.internal resolves the hostname once at config load and caches the IPs forever (for hostnames in upstream blocks / static proxy_pass on open-source NGINX) — DNS TTLs are ignored. Hosts reloaded after the migration picked up new IPs; hosts not reloaded kept the corpse. The mixed behavior across the fleet (some fine, some timing out) disguised the mechanism for hours.

Diagnosis Steps

1.Compare healthy vs failing NGINX hosts: config identical; last-reload time differs across the DNS change
2.On a failing host: the upstream IP in error logs vs current dig output — stale IP confirmed
3.Check proxy_pass form: static hostname (load-time resolution) vs variable (runtime resolution)
4.Fleet sweep for the same pattern on other migratable upstreams

Solution

Immediate: fleet-wide reload re-resolved everything. Durable: dynamic resolution for migrating upstreams — a resolver directive plus variable-based proxy_pass (set $backend 'backend.internal'; proxy_pass http://$backend:8080) forces per-request/TTL-honoring resolution on OSS NGINX; where upstream-block features (keepalive, load balancing) were needed, moved those tiers to explicit IP management via config rendering from service discovery, redeployed on change.

Commands

dig +short backend.internal  # vs the IP in error.log

config fix: resolver 10.0.0.2 valid=30s; set $backend 'backend.internal'; proxy_pass http://$backend:8080;

ansible -m command -a 'nginx -s reload' edge  # immediate

🛡

Prevention

Know your NGINX name-resolution semantics: static hostname resolution is load-time only. Any upstream that can re-IP needs the resolver+variable pattern, config rendered from service discovery, or a discovery-integrated NGINX distribution. Backend migrations must include 'who resolves us at startup only' in the consumer checklist.

nginxdnsupstreamstale-resolutionmigration

💬 Comments

🚀 performance

Why It Matters

Default timeout behavior can preserve bad user experience for too long. If an upstream is slow but not dead, the load balancer may continue sending requests until passive failure thresholds are met. Users experience latency before operators see a hard outage. Without following this practice, teams typically discover the problem during a production incident. The cost of fixing issues reactively is 10-100x higher than preventing them proactively. This becomes especially dangerous at scale when multiple teams depend on the same infrastructure, because one team's shortcut becomes another team's outage. Organizations that skip this practice often find themselves in a cycle of firefighting instead of building. The pattern is predictable: it works fine in development, survives staging, and fails spectacularly in production under real traffic and real failure conditions.

How to Implement

Set connect, send, and read timeouts deliberately; configure upstream failure thresholds; expose per-upstream metrics; and reload configuration only after `nginx -t` passes. For critical services, use active health checks if your NGINX edition supports them. Start by implementing this in a single non-critical service to validate the approach. Document the configuration and share it with the team through a runbook or internal wiki page. Add monitoring and alerting around the key metrics so you know when the practice is working and when it is not. Gradually roll it out to more services, adjusting the configuration based on what you learn from each deployment. Review the implementation quarterly to ensure it still matches your current architecture and traffic patterns.

Example

upstream checkout_api {
  server 10.0.10.21:8080 max_fails=2 fail_timeout=10s; # Temporarily avoids a backend after repeated failures.
  server 10.0.10.22:8080 max_fails=2 fail_timeout=10s;
  keepalive 64; # Reuses upstream connections without creating unlimited sockets.
}
proxy_connect_timeout 2s; # Fails fast when an upstream cannot accept connections.
proxy_read_timeout 15s; # Keeps request latency bounded to the service SLO.

✗ Anti-pattern

The anti-pattern is increasing all timeouts after a 504. That hides failure longer and consumes more worker capacity during dependency incidents.

nginxtimeoutsupstreamslo

◈ Architecture Diagram

┌──────────┐   ┌──────────┐
│ ✓ Guard  │   │ ✗ Drift  │
└────┬─────┘   └────┬─────┘
     ↓              ↓
┌──────────┐   ┌──────────┐
│ Stable   │   │ Page     │
└──────────┘   └──────────┘

💬 Comments