An Application Load Balancer is returning 502s for a subset of requests even though EC2 target health checks show healthy — how do you diagnose the mismatch?

Imagine a restaurant host who checks every 30 seconds whether a specific table is occupied or free by glancing at it, a very cheap, quick check. That host can honestly report table 12 is free while completely missing that the kitchen is actually so backed up that any customer seated there right now would wait an hour for food, or that table 12's chair just broke in a way the host's quick glance never caught. The health check and the actual dining experience are checking fundamentally different things, and one being fine says nothing reliable about the other.

AWS's Application Load Balancer health checks are deliberately lightweight and narrow by design — they hit a configured path, like /health, at a configured interval and expect a specific status code, existing purely to detect whether a target is completely down or unresponsive so the ALB can stop routing to a dead instance. They were never designed to guarantee every real request will succeed, because a full simulation of production traffic patterns on every health check interval would itself add load and complexity that defeats the purpose of a cheap, frequent liveness signal in the first place.

The actual 502 usually stems from a layer the health check doesn't touch: TCP and HTTP connection handling between the ALB and the target. ALBs keep connections to backend targets alive and reuse them for multiple requests for efficiency, governed by an idle timeout on both the ALB side, default 60 seconds, configurable, and the backend web server's own keep-alive timeout — a Node.js or nginx backend serving payments-api might have a shorter keep-alive timeout than the ALB expects. If the backend closes a connection the ALB still considers open, and then the ALB tries to reuse it for a new request, the backend responds with a connection reset, and the ALB surfaces that as a 502 to the client, a failure mode entirely invisible to a simple periodic health check hitting a fresh connection each time it runs.

In production, the ALB's access logs, delivered to S3 if enabled, include a target_status_code field and error_reason. A 502 where target_status_code shows -1 means the ALB never received a valid HTTP response from the target at all, a connection-level failure, versus a genuine 502 the application itself returned deliberately. Cross-referencing the timing of these 502s against deploy events is also critical, since a common cause is a rolling deployment where a target gets deregistered and stops accepting new connections, but in-flight requests routed to it right before deregistration get abruptly cut off if the deregistration delay, the connection draining timeout, is set too short for how long real requests to something like checkout-worker actually take to complete.

The non-obvious gotcha: the fix for the keep-alive mismatch is counterintuitive to many engineers — you must set the backend server's keep-alive or idle timeout to be LONGER than the ALB's idle timeout, not shorter, so the backend never closes a connection the ALB still considers usable. Getting this backwards, a very common misconfiguration since it feels natural to want the backend to be the more aggressive timeout, is one of the single most common causes of intermittent, hard-to-reproduce 502s that show up only under real concurrent traffic and never during simple manual health-check-style testing.