How does Envoy handle logging and access logs, and how do you configure them?

Think of Envoy's logging as two separate record-keeping systems. The application log is like a mechanic's diagnostic journal that records internal engine events, warning lights, and system errors. The access log is like a toll booth receipt printer that records every vehicle that passes through: the license plate, the lane used, the time of entry, the time of exit, and the toll charged. Both are essential but serve completely different purposes.

Envoy's application log controls the verbosity of internal operational messages. It uses a hierarchical logger system where different components (connection, http, router, upstream, admin) can be set to different log levels: trace, debug, info, warning, error, and critical. In development, you might set the log level to debug to see detailed information about connection handling, header processing, and routing decisions. In production, the log level is typically set to warning or error to avoid overwhelming log storage with verbose output. The log level can be changed at runtime through the admin interface endpoint /logging, which is invaluable for temporary debugging without restarting Envoy.

Access logs are the primary mechanism for recording information about every request that Envoy processes. They are configured within the HTTP Connection Manager filter and support multiple output sinks. The most common sink is a file-based access log that writes to a specified file path or stdout. Each log entry can be formatted using Envoy's command operator syntax, where placeholders like %REQ(:METHOD)%, %RESP(:STATUS)%, %DURATION%, %UPSTREAM_HOST%, and %RESPONSE_CODE% are replaced with actual values from the request and response. Envoy provides dozens of command operators covering request headers, response headers, connection properties, upstream metadata, and timing information. You can also use JSON format for access logs, which produces structured log entries that are much easier to parse with log aggregation tools like Elasticsearch, Splunk, or Loki.

In production environments, access log configuration is a balancing act between observability and performance. Writing a log entry for every request in a high-throughput service can generate massive log volumes and consume significant disk I/O and CPU. Teams typically configure access logs with a subset of the most useful fields: request method, path, response code, upstream host, total duration, upstream response time, and a few key headers. Some teams use conditional access logging to only log requests that result in errors (4xx or 5xx) or requests that exceed a latency threshold, dramatically reducing log volume while preserving diagnostic value. The gRPC access log sink sends log entries to an external service, enabling centralized log collection without local file management.

A critical aspect of Envoy access logs is understanding the response flags. Envoy adds response flags like UH (no healthy upstream), UO (upstream overflow due to circuit breaking), UT (upstream timeout), NR (no route configured), and DC (downstream connection termination). These flags appear in the access log and provide immediate insight into why a request failed. For example, seeing a 503 response with the UH flag tells you the upstream cluster had no healthy endpoints, while a 503 with UO tells you the circuit breaker tripped. Learning to read these response flags is one of the most valuable debugging skills for operating Envoy in production.