How do health checks work in Consul and what types of health checks are available?
Quick Answer
Consul health checks are mechanisms that continuously monitor the health of services and nodes, automatically marking them as passing, warning, or critical. Consul supports HTTP, TCP, gRPC, script, TTL, Docker, and alias health checks, with the local client agent executing checks and propagating results through the cluster to ensure only healthy instances receive traffic.
Detailed Answer
Health checks are the foundation of reliable service discovery in Consul. Without accurate health information, service discovery becomes a liability rather than an asset because it could route traffic to broken instances. Consul health checks run on the local client agent where the service is registered, which means the health check executes close to the service, providing accurate and timely detection of failures. The check results directly influence whether a service instance appears in DNS queries and API responses, making health checks the gatekeeper for all service-to-service traffic routing.
Consul supports several health check types, each designed for different service architectures. HTTP checks are the most common type, where the Consul agent makes an HTTP GET request to a specified URL at a configured interval. If the response status code is 2xx, the check passes; 429 (Too Many Requests) is treated as a warning; and any other status code marks the check as critical. TCP checks verify that a TCP connection can be established to a specified address and port, useful for services that do not expose HTTP endpoints such as database proxies or message brokers. gRPC checks use the gRPC health checking protocol to verify service health, ideal for gRPC-based microservices. Script checks execute an arbitrary command and use the exit code to determine status: exit code 0 is passing, exit code 1 is warning, and any other exit code is critical.
TTL (Time-to-Live) checks invert the responsibility by requiring the service itself to periodically report its health to Consul. The service must send a PUT request to the local agent within the configured TTL period, or the check automatically transitions to critical. This pattern is useful when the service has internal knowledge about its health that cannot be assessed externally, such as whether it has a valid database connection pool or whether its internal queue depth is within acceptable limits. Docker checks execute a command inside a running Docker container, and alias checks mirror the health status of another service or node, creating health check dependencies.
Each health check can be configured with several parameters that control its behavior. The interval determines how frequently the check runs, typically between 5 and 30 seconds. The timeout specifies how long to wait for the check to respond before considering it failed, and should be shorter than the interval. The deregister_critical_service_after parameter automatically removes a service registration if it remains in a critical state for longer than the specified duration, which is essential for cleaning up stale registrations from terminated instances. Multiple health checks can be associated with a single service, and the overall service health is determined by the worst check status across all associated checks.
In production, health check design requires careful consideration of what constitutes a healthy service. A shallow health check that simply returns HTTP 200 from a /health endpoint might miss scenarios where the service is running but cannot reach its database or downstream dependencies. Deep health checks that verify database connectivity and downstream service availability provide more accurate health signals but risk cascading failures where a database outage causes all services to be marked unhealthy simultaneously. The recommended practice is to use a combination: a shallow liveness check with a short interval for rapid failure detection, and a deeper readiness check that validates critical dependencies. Teams frequently make the mistake of setting check intervals too aggressively, such as every second, which creates unnecessary load on both the Consul agent and the service being checked.