Envoy

Think of Envoy Proxy as a universal translator and traffic controller that sits between every microservice in your infrastructure. Instead of each service having to handle network concerns like retries, timeouts, load balancing, and observability on its own, Envoy takes over all of these responsibilities transparently, allowing developers to focus purely on business logic.

Envoy was originally created at Lyft in 2016 to solve the challenges of managing network communication across hundreds of microservices. Before Envoy, Lyft's services used a patchwork of language-specific libraries and ad-hoc proxy configurations to handle service-to-service communication. This led to inconsistent behavior, difficult debugging, and fragmented observability. Envoy was designed from the ground up as a self-contained process that runs alongside every service, intercepting all inbound and outbound network traffic. It was donated to the Cloud Native Computing Foundation (CNCF) in 2017 and graduated in 2018, becoming one of the most widely adopted infrastructure components in the cloud-native ecosystem.

At its core, Envoy is a L3/L4 network proxy with an L7 filter architecture. This means it can handle raw TCP connections at the network and transport layers while also understanding and manipulating application-layer protocols like HTTP/1.1, HTTP/2, and gRPC. Envoy is written in C++ for maximum performance, uses an event-driven, non-blocking architecture similar to Nginx, and supports hot restarts that allow configuration updates without dropping any active connections. The proxy is fully dynamic, meaning its entire configuration can be updated at runtime through a set of discovery service APIs called xDS, without requiring restarts or reloads. This dynamic nature is what makes Envoy the data plane of choice for service meshes like Istio, Consul Connect, and AWS App Mesh.

In production environments, Envoy is deployed in several patterns. The most common is the sidecar pattern, where an Envoy instance runs alongside each microservice in the same pod or VM. In this configuration, all traffic to and from the payments-api service passes through its local Envoy sidecar, which handles TLS termination, retries, circuit breaking, rate limiting, and telemetry collection. Envoy is also used as an edge proxy or API gateway, sitting at the ingress point of a cluster and routing external traffic to internal services. Companies like Airbnb, Pinterest, Stripe, and Salesforce use Envoy at massive scale, processing millions of requests per second. The proxy emits detailed statistics for every connection and request, integrates with distributed tracing systems like Zipkin and Jaeger, and provides an admin interface for runtime debugging.

A critical point that differentiates Envoy from traditional proxies is its API-driven configuration model. While Nginx and HAProxy rely on static configuration files that require reloads, Envoy can discover listeners, routes, clusters, and endpoints dynamically through its xDS APIs. This makes Envoy ideal for environments where services scale up, scale down, and change frequently. The combination of high performance, dynamic configuration, protocol awareness, and rich observability is why Envoy has become the foundational building block for modern service mesh architectures and cloud-native networking.

Think of Nginx and HAProxy as experienced highway toll operators who are extremely efficient at directing traffic along pre-planned routes. Envoy is more like a GPS navigation system that continuously recalculates routes based on real-time traffic conditions, road closures, and new destinations appearing on the map. Both approaches work, but they excel in different scenarios.

Nginx was originally built as a high-performance web server and reverse proxy, optimized for serving static content and handling large numbers of concurrent connections. HAProxy was purpose-built as a load balancer with exceptional performance for TCP and HTTP traffic. Both tools use static configuration files that require a reload or restart when changes are made. While Nginx Plus and newer HAProxy versions have added some dynamic capabilities, their core architecture is fundamentally file-driven. Envoy, by contrast, was designed from day one with dynamic configuration as a first-class feature. Its entire configuration, including listeners, routes, clusters, and endpoints, can be updated at runtime through the xDS API without any restarts or connection drops.

The protocol support is another major differentiator. Envoy natively supports HTTP/1.1, HTTP/2, and gRPC as first-class protocols, with the ability to translate between them. For example, Envoy can accept an HTTP/1.1 request from a legacy client and forward it as HTTP/2 to the upstream service, or proxy gRPC traffic with full understanding of the protocol semantics. Nginx added HTTP/2 support later, and HAProxy's gRPC support is more limited. Envoy also supports advanced L7 protocols through its extensible filter chain, including MongoDB, MySQL, Redis, and Thrift protocol awareness, allowing it to parse and route application-specific traffic.

In terms of observability, Envoy generates detailed statistics for every upstream cluster, downstream connection, and individual request. It natively integrates with distributed tracing systems by propagating trace context headers and generating spans. It exports metrics in formats compatible with Prometheus, StatsD, and other monitoring backends. Nginx and HAProxy provide access logs and basic metrics, but they require additional modules or external tools to achieve the level of observability Envoy provides out of the box. Envoy's admin interface exposes runtime configuration, cluster health, statistics, and debugging endpoints, making it significantly easier to troubleshoot in production.

Choose Envoy when you are building a microservices architecture with dynamic service discovery, need a service mesh data plane, require native gRPC support, or need rich observability without additional tooling. Choose Nginx when you need a high-performance web server for static content, a well-established reverse proxy with extensive module ecosystem, or when your team has deep Nginx expertise. Choose HAProxy when you need a battle-tested TCP/HTTP load balancer with deterministic performance characteristics and simpler operational requirements. In practice, many organizations use Envoy as a sidecar proxy alongside Nginx or HAProxy at the edge, combining the strengths of each tool in different layers of their architecture.

Think of Envoy like a large hotel. Listeners are the doors to the hotel, each accepting guests on different entrances. Filter chains are the check-in procedures that inspect and process each guest. Routes are the directory signs that tell guests which floor and room to go to based on their reservation. Clusters are the actual floors with available rooms, and endpoints are the individual rooms on each floor.

The listener is the first component in Envoy's request processing pipeline. A listener binds to a specific IP address and port, accepting incoming connections from downstream clients. Each Envoy instance can have multiple listeners, for example one on port 80 for HTTP traffic and another on port 443 for HTTPS. Listeners are defined with a name, a bind address, and one or more filter chains. When a connection arrives, the listener selects the appropriate filter chain based on connection properties like SNI (Server Name Indication) for TLS or the source IP address. This is the entry point where all traffic first touches Envoy.

Filter chains contain an ordered list of network filters that process the connection. The most important filter for HTTP traffic is the HTTP Connection Manager (HCM), which upgrades a raw TCP connection into HTTP protocol handling. The HCM parses HTTP requests, manages connection pooling, and applies HTTP-level filters like rate limiting, authentication, and CORS. Within the HCM, the route configuration defines how requests are matched and forwarded. Routes match on URL paths, headers, query parameters, and other HTTP attributes, then direct the request to a specific cluster. The route configuration can be defined statically in the config file or dynamically through the Route Discovery Service (RDS).

Clusters represent groups of logically similar upstream hosts that Envoy forwards traffic to. A cluster named payments-api might contain three backend instances running on different IP addresses. Each cluster defines its load balancing policy (round-robin, least-request, random, ring-hash), health checking configuration, circuit breaker thresholds, and connection pool settings. Endpoints within a cluster are the actual network addresses of upstream service instances. Envoy supports multiple service discovery mechanisms for endpoints: static (hardcoded IPs), strict DNS (DNS resolution), logical DNS, EDS (Endpoint Discovery Service for dynamic environments), and original destination (for transparent proxying).

In production, a typical Envoy configuration for a microservices environment defines one listener on port 15001 that intercepts all outbound traffic from the local application. The route table maps destination hostnames to clusters: requests to order-service.default.svc.cluster.local go to the order-service cluster, requests to inventory-service go to the inventory-service cluster, and so on. Each cluster is configured with appropriate timeouts, retry policies, and health checks. The admin listener on port 9901 provides debugging endpoints. Understanding this architecture is essential because every service mesh configuration ultimately translates into these Envoy primitives.

A common misconception is that routes and clusters are the same thing. Routes define the matching logic (which requests go where), while clusters define the upstream behavior (how to connect, which endpoints to use, what load balancing algorithm to apply). Separating these concerns allows you to route different URL paths to different clusters with different policies, or to route the same path to different clusters based on headers for canary deployments.

Imagine every employee in an office building having a personal assistant who handles all their mail, phone calls, and meeting scheduling. The employee focuses purely on their work while the assistant manages all communication logistics, screens calls, retries failed deliveries, and keeps detailed records of every interaction. The sidecar proxy pattern works exactly this way: Envoy acts as the personal assistant for each microservice.

The sidecar pattern is a deployment architecture where a proxy process runs alongside the main application process, sharing the same network namespace. In Kubernetes, this means Envoy runs as a second container within the same pod as the application container. Because they share a network namespace, Envoy can intercept all network traffic to and from the application using iptables rules or similar mechanisms. The application sends traffic to localhost, Envoy intercepts it, applies policies (TLS encryption, retries, circuit breaking), and forwards it to the destination. Similarly, all incoming traffic first hits Envoy, which applies inbound policies before forwarding to the application on localhost. The application is completely unaware that Envoy exists.

Envoy implements the sidecar pattern through its lightweight, self-contained architecture. A single Envoy binary handles all proxy functionality without external dependencies. In Kubernetes environments, sidecar injection is typically automated: a mutating admission webhook modifies pod specifications to add the Envoy container and an init container that configures iptables rules. The init container sets up traffic redirection so that all TCP traffic leaving the application container is captured by Envoy's outbound listener, and all TCP traffic arriving at the pod is captured by Envoy's inbound listener. Envoy then uses its route table to determine the correct upstream cluster for each request. The service mesh control plane (like Istio's istiod) pushes configuration to each Envoy sidecar through the xDS API, telling it about available services, their endpoints, routing rules, and security policies.

In production, the sidecar pattern provides several critical benefits. First, it achieves language-agnostic networking: whether your payments-api is written in Java, your order-service in Go, and your catalog-service in Python, they all get the same networking capabilities through Envoy without importing any language-specific libraries. Second, it enables consistent observability because every request passes through Envoy, which emits metrics, access logs, and trace spans uniformly. Third, it provides security through automatic mTLS encryption between all sidecars without application changes. Fourth, it enables traffic management features like canary deployments, traffic mirroring, and fault injection at the infrastructure level.

The sidecar pattern does have trade-offs that are important to understand. Each Envoy sidecar consumes approximately 50-100MB of memory and adds 1-3ms of latency per hop. For a cluster with 500 pods, that is 25-50GB of additional memory dedicated to proxies. The iptables-based traffic interception can cause debugging challenges because tools like tcpdump show traffic going to localhost instead of the actual destination. Some teams opt for Envoy's ambient mesh mode or per-node deployment to reduce the overhead, but this sacrifices per-service granularity. Understanding these trade-offs is essential for making informed deployment decisions.

Think of the Envoy admin interface as the dashboard of a car. While the engine (proxy) handles the actual work of moving traffic, the dashboard gives you real-time information about speed, fuel level, engine temperature, and warning lights. Without the dashboard, you would be driving blind, unable to diagnose problems until something breaks down completely.

The admin interface is an HTTP server built into every Envoy instance, typically bound to localhost on port 9901. It provides a comprehensive set of endpoints for inspecting Envoy's runtime state without affecting traffic processing. The most commonly used endpoints include /stats for viewing all counters, gauges, and histograms that Envoy tracks; /clusters for seeing the health status of every upstream endpoint; /config_dump for viewing the complete active configuration; /listeners for listing all active listeners; /logging for dynamically changing log levels; and /ready for health check readiness. The admin interface should never be exposed to untrusted networks because it provides sensitive information and can modify runtime behavior.

The /stats endpoint is the most powerful debugging tool. Envoy tracks thousands of statistics organized by category: cluster metrics show upstream connection counts, request totals, error rates, and latency histograms. Listener metrics show downstream connection counts and HTTP protocol errors. HTTP metrics show request counts by response code, retry counts, and timeout counts. You can filter stats by prefix to focus on a specific cluster or listener. For example, querying /stats?filter=cluster.payments-api shows only statistics related to the payments-api upstream cluster. The /stats/prometheus endpoint exports all statistics in Prometheus format, which can be scraped directly by a Prometheus server for long-term storage and alerting.

The /clusters endpoint shows the real-time health status of every endpoint in every cluster. For each endpoint, it displays the IP address, health status (healthy, unhealthy, or degraded), the number of active connections, outstanding requests, and the results of recent health checks. This is invaluable when debugging routing issues: if requests to the order-service are failing, the /clusters endpoint immediately shows whether the upstream endpoints are healthy, how many connections are active, and whether circuit breakers have tripped. The /config_dump endpoint outputs the entire active Envoy configuration as JSON, which is essential for verifying that dynamic configuration updates from the control plane have been applied correctly.

In production environments, operators use the admin interface as the first line of investigation when diagnosing issues. If a service is returning 503 errors, the typical debugging workflow is: check /clusters to see if upstream endpoints are healthy, check /stats for circuit breaker trip counts, verify /config_dump to ensure routes are configured correctly, and use /logging?level=debug to temporarily increase log verbosity for deeper investigation. Many teams build monitoring dashboards and alerts on top of the Prometheus-format stats. The admin interface also supports POST endpoints for runtime modifications like draining listeners for graceful shutdown and resetting statistics counters.

Think of Envoy as a river dam. Downstream is the direction water flows toward, meaning the clients that receive responses. Upstream is where the water comes from, meaning the backend services that generate the responses. Envoy sits in the middle, controlling the flow of water in both directions. This terminology is consistent throughout Envoy's documentation, configuration, and statistics.

In Envoy's architecture, a downstream connection is established when a client connects to one of Envoy's listeners. This client could be an end user's browser, another microservice, or any application making a network request. The downstream side is where Envoy accepts connections and receives requests. An upstream connection is what Envoy creates when it needs to forward a request to a backend service. The upstream side is where Envoy acts as a client, connecting to the endpoints defined in its cluster configuration. This bidirectional terminology is fundamental because Envoy processes traffic differently in each direction and tracks separate statistics for downstream and upstream activity.

The distinction becomes critical when reading Envoy statistics and logs. Downstream metrics include counters like downstream_cx_total (total downstream connections), downstream_rq_total (total downstream requests), and downstream_cx_active (currently active downstream connections). These tell you about the load Envoy is receiving from clients. Upstream metrics include upstream_cx_total, upstream_rq_total, upstream_rq_timeout, and upstream_rq_retry, which tell you about Envoy's communication with backend services. When debugging a 503 error, understanding this distinction is essential: a downstream 503 means Envoy sent a 503 to the client, but the root cause might be an upstream connection failure, an upstream timeout, or a circuit breaker trip on the upstream cluster.

In a service mesh with sidecar proxies, the upstream and downstream terminology applies at each hop. When the payments-api calls the order-service, the request passes through two Envoy proxies. The payments-api's Envoy sidecar treats the payments-api container as the downstream client and the order-service as the upstream. The order-service's Envoy sidecar treats the payments-api's Envoy as the downstream client and the order-service container as the upstream. Understanding this double-proxy hop is essential for interpreting metrics and access logs correctly in a service mesh environment.

A common source of confusion is that the upstream/downstream terminology in Envoy is opposite to how some people intuitively think about data flow. People sometimes assume upstream means the client because it is 'higher up' in the call chain. In networking and proxy terminology, upstream always refers to the server or backend that provides the content, and downstream always refers to the client or consumer that requests the content. This convention is consistent across Envoy, Nginx, HAProxy, and most proxy technologies. Getting this terminology right is essential for clear communication in incident response scenarios.

Think of Envoy's logging as two separate record-keeping systems. The application log is like a mechanic's diagnostic journal that records internal engine events, warning lights, and system errors. The access log is like a toll booth receipt printer that records every vehicle that passes through: the license plate, the lane used, the time of entry, the time of exit, and the toll charged. Both are essential but serve completely different purposes.

Envoy's application log controls the verbosity of internal operational messages. It uses a hierarchical logger system where different components (connection, http, router, upstream, admin) can be set to different log levels: trace, debug, info, warning, error, and critical. In development, you might set the log level to debug to see detailed information about connection handling, header processing, and routing decisions. In production, the log level is typically set to warning or error to avoid overwhelming log storage with verbose output. The log level can be changed at runtime through the admin interface endpoint /logging, which is invaluable for temporary debugging without restarting Envoy.

Access logs are the primary mechanism for recording information about every request that Envoy processes. They are configured within the HTTP Connection Manager filter and support multiple output sinks. The most common sink is a file-based access log that writes to a specified file path or stdout. Each log entry can be formatted using Envoy's command operator syntax, where placeholders like %REQ(:METHOD)%, %RESP(:STATUS)%, %DURATION%, %UPSTREAM_HOST%, and %RESPONSE_CODE% are replaced with actual values from the request and response. Envoy provides dozens of command operators covering request headers, response headers, connection properties, upstream metadata, and timing information. You can also use JSON format for access logs, which produces structured log entries that are much easier to parse with log aggregation tools like Elasticsearch, Splunk, or Loki.

In production environments, access log configuration is a balancing act between observability and performance. Writing a log entry for every request in a high-throughput service can generate massive log volumes and consume significant disk I/O and CPU. Teams typically configure access logs with a subset of the most useful fields: request method, path, response code, upstream host, total duration, upstream response time, and a few key headers. Some teams use conditional access logging to only log requests that result in errors (4xx or 5xx) or requests that exceed a latency threshold, dramatically reducing log volume while preserving diagnostic value. The gRPC access log sink sends log entries to an external service, enabling centralized log collection without local file management.

A critical aspect of Envoy access logs is understanding the response flags. Envoy adds response flags like UH (no healthy upstream), UO (upstream overflow due to circuit breaking), UT (upstream timeout), NR (no route configured), and DC (downstream connection termination). These flags appear in the access log and provide immediate insight into why a request failed. For example, seeing a 503 response with the UH flag tells you the upstream cluster had no healthy endpoints, while a 503 with UO tells you the circuit breaker tripped. Learning to read these response flags is one of the most valuable debugging skills for operating Envoy in production.

Think of running Envoy with Docker as setting up a pre-built security checkpoint at a building entrance. The Docker image provides the checkpoint equipment (Envoy binary), you provide the rules for who can enter and which floor they go to (envoy.yaml configuration), and you connect it to the building's entry points (port mappings). Docker ensures the checkpoint runs consistently regardless of the building's infrastructure.

The official Envoy Docker image is published on Docker Hub as envoyproxy/envoy and is available with various tags corresponding to Envoy versions. The image contains the Envoy binary, default configuration, and minimal system dependencies. To run Envoy with a custom configuration, you mount your envoy.yaml file into the container at the default configuration path /etc/envoy/envoy.yaml. Port mappings expose the listener ports and the admin interface port to the host machine. The most basic run command maps the primary listener port (e.g., 10000) and the admin port (9901), mounts the configuration file, and starts the container.

Envoy's Docker image supports several command-line flags that control runtime behavior. The --concurrency flag sets the number of worker threads, which should typically match the number of CPU cores allocated to the container. The --log-level flag sets the initial application log verbosity. The --component-log-level flag allows setting different log levels for different components, such as --component-log-level upstream:debug,connection:trace. The --config-path flag specifies the configuration file location, and --config-yaml allows passing inline YAML configuration for simple setups. The --service-cluster and --service-node flags set metadata that identifies this Envoy instance to the control plane.

In production Docker deployments, Envoy is typically run with resource constraints to prevent it from consuming excessive CPU or memory. Docker's --cpus and --memory flags limit resource usage. Health checks are configured to use the admin interface's /ready endpoint, ensuring the container orchestrator knows when Envoy is ready to accept traffic. The container should run as a non-root user for security, and the admin interface should only be bound to localhost or an internal network. When running Envoy alongside an application in Docker Compose, both containers share a Docker network, allowing Envoy to reach the application by container name.

A common beginner mistake is running Envoy without validating the configuration first. Envoy provides a --mode validate flag that checks the configuration file for syntax errors and semantic issues without actually starting the proxy. Always validate configuration changes before deploying them. Another gotcha is forgetting to expose the admin port for debugging. While the admin interface should not be publicly accessible, having it available internally is essential for troubleshooting. In Docker Compose environments, define the admin port mapping only for development profiles and exclude it from production configurations. File permission issues with mounted configuration files are another frequent problem: ensure the Envoy user inside the container has read access to the mounted envoy.yaml file.

Think of service discovery as the way Envoy builds its phone book of available backend services. Just like you can have a printed phone book (static), look up numbers through directory assistance (DNS), or use a constantly updated contact app that syncs across devices (EDS), Envoy provides multiple methods for discovering where upstream services are located.

The cluster type configuration in Envoy determines how it discovers and maintains the list of endpoints (IP addresses and ports) for each upstream service. The simplest type is STATIC, where you hardcode the endpoint addresses directly in the configuration file. This is straightforward and works well for infrastructure services with fixed addresses, like a database server or an external API endpoint. However, static configuration does not adapt to dynamic environments where services scale up and down.

STRICT_DNS resolves a DNS hostname and uses all returned IP addresses as endpoints. Envoy continuously re-resolves the DNS name at a configurable interval and updates the endpoint list when the DNS response changes. If the DNS query for order-service.default.svc.cluster.local returns three IP addresses, Envoy creates three endpoints and load-balances across all of them. When a pod is removed and the DNS response changes to two IP addresses, Envoy automatically removes the stale endpoint. LOGICAL_DNS is similar but only uses the first IP address from the DNS response, creating a single logical connection. This is useful for services behind a load balancer where you want DNS-based resolution but do not want Envoy to track individual backend instances.

EDS (Endpoint Discovery Service) is the most powerful and dynamic service discovery mechanism. Instead of relying on DNS, Envoy communicates with an external management server that provides endpoint information through a gRPC or REST API. The management server pushes endpoint updates to Envoy in real-time as services scale, deploy, or fail health checks. This is the mechanism used by service mesh control planes like Istio's istiod, Consul Connect, and AWS App Mesh. EDS provides richer information than DNS, including endpoint metadata, health status, load balancing weights, and locality information for zone-aware routing. ORIGINAL_DST is used for transparent proxying, where Envoy uses the original destination address from the connection metadata (set by iptables redirection) to determine the upstream endpoint.

In production Kubernetes environments, EDS is the standard discovery mechanism because it integrates with the Kubernetes API to track pod lifecycle events in real-time. When a new payments-api pod starts and passes its readiness check, the control plane pushes an EDS update to all Envoy sidecars, immediately making the new endpoint available for load balancing. When a pod is terminating, the endpoint is removed before the pod shuts down, preventing requests from being sent to a dying instance. This real-time synchronization is far more responsive than DNS-based discovery, which has inherent TTL delays.

A common mistake is choosing STRICT_DNS in Kubernetes and assuming it will work like EDS. While DNS-based discovery works, it has limitations: DNS TTLs introduce propagation delays, DNS responses may be truncated for large clusters, and DNS does not provide health status or weighting information. For any environment with more than a handful of services, EDS is the recommended approach because it provides real-time updates, richer metadata, and better integration with the overall service mesh architecture.