What is the difference between a Consul client agent and a Consul server agent?

Every node in a Consul cluster runs a Consul agent, but agents operate in one of two modes that serve fundamentally different purposes. Understanding the distinction between client agents and server agents is essential for designing and operating a reliable Consul deployment. The two types work together in a layered architecture where clients handle local responsibilities and servers maintain global cluster state.

Consul client agents are lightweight processes that run on every node in your infrastructure where services need to be registered or discovered. Their primary responsibilities include registering local services with the cluster, executing health checks against those services, and forwarding queries and writes to the server agents. Client agents participate in the gossip protocol (using Serf) for cluster membership and failure detection, but they do not store any replicated state. When payments-api registers itself with the local client agent, the client forwards that registration to a server agent via RPC. Clients cache recent query results locally to reduce server load and provide faster responses for repeated lookups. Because clients are stateless from a catalog perspective, losing a client node only affects the services running on that specific node.

Consul server agents carry the heavy lifting of maintaining the distributed, consistent service catalog. Servers participate in the Raft consensus protocol, which elects a single leader responsible for processing all write operations. When a service registers, the leader appends the registration to the Raft log and replicates it to follower servers. A write is only considered committed when a majority (quorum) of servers have acknowledged it, ensuring data durability even if individual servers fail. Server agents also handle cross-datacenter communication through WAN gossip, enabling multi-datacenter service discovery and federation. The recommended production deployment is 3 server agents for small to medium clusters and 5 server agents for large-scale or mission-critical deployments, as these numbers provide fault tolerance while keeping replication overhead manageable.

The gossip protocol ties both agent types together into a cohesive cluster. All agents, both clients and servers, participate in the LAN gossip pool within a datacenter. This gossip layer uses UDP-based protocol for efficient dissemination of membership changes and failure detection. When a client agent detects that a local service health check has failed, it propagates this information through gossip, and the server agents update the catalog accordingly. Server agents additionally participate in a separate WAN gossip pool for cross-datacenter communication, which operates at a lower frequency to accommodate higher-latency network links between datacenters.

In production environments, the ratio of clients to servers varies significantly based on workload. A typical Kubernetes deployment might have 3 Consul server pods and a Consul client DaemonSet that places one client agent on every worker node, resulting in dozens or hundreds of clients. The client agents on each worker node handle registrations for all services running on that node. Resource requirements differ substantially: server agents need fast disk I/O and adequate memory for the Raft log and service catalog, while client agents are much lighter, typically requiring minimal CPU and memory. A common architecture mistake is running too many server agents, such as 7 or more, which actually degrades performance because every write must be replicated to more nodes before being committed, increasing write latency without meaningful improvement in fault tolerance.