What is the difference between monitoring and observability, and why does centralized logging matter in Kubernetes?

Think of the difference between a car dashboard and a mechanic's diagnostic computer. The dashboard shows predefined gauges: speed, fuel, temperature (monitoring). You know something is wrong when the temperature gauge is high, but you cannot ask why. The diagnostic computer lets you query any sensor, trace any circuit, and correlate signals to find the root cause (observability). Both are necessary, but observability lets you investigate problems you never anticipated.

Monitoring is the practice of collecting, aggregating, and alerting on predefined metrics. In Kubernetes, this means tracking CPU usage, memory consumption, pod restart counts, request rates, error rates, and latency percentiles. Tools like Prometheus scrape metrics endpoints, store time-series data, and fire alerts when thresholds are breached. Monitoring answers known questions: is the service up? Is CPU above 80 percent? Are there more than five restarts in ten minutes? The limitation is that monitoring only answers questions you thought to ask in advance.

Observability is a property of a system that lets you understand its internal state from external outputs. The three pillars are metrics (numeric measurements over time), logs (discrete events with context), and traces (request paths across services). An observable system lets you ask new questions without deploying new instrumentation. When a user reports that checkout is slow for customers in Europe but not the US, observability tools let you slice latency by region, trace a specific request through fifteen microservices, and correlate with database query times — even if you never built a dashboard for that exact scenario.

Centralized logging is critical in Kubernetes because the platform treats pods as disposable. When a pod crashes, Kubernetes restarts it and the previous container's logs are only accessible through kubectl logs --previous until the pod is rescheduled or evicted, at which point they vanish entirely. During an incident at 3 AM, the pod that caused the problem may have already been replaced three times. Without centralized logging (Fluentd/Fluent Bit shipping to Elasticsearch, CloudWatch, or Loki), the evidence is gone. Centralized logging also enables searching across hundreds of pods simultaneously, correlating events across namespaces, and retaining logs for compliance.

In production, the common mistake is treating metrics and logs as separate concerns. A mature observability stack correlates them: a Prometheus alert for high error rate links to a Grafana dashboard, which links to Loki log queries filtered by the same timeframe, labels, and pod. Distributed tracing with OpenTelemetry connects a single user request across service boundaries, showing exactly where latency accumulates. The goal is reducing mean time to detection (MTTD) and mean time to resolution (MTTR) by making every failure diagnosable without SSH access to individual containers.

The non-obvious gotcha is log volume cost. A busy Kubernetes cluster can generate gigabytes of logs per hour. Without structured logging (JSON format with consistent fields), log search becomes grep across unstructured text. Without log-level filtering at the agent, debug logs flood the central store and inflate storage costs. Teams must balance verbosity against cost, typically shipping warn and error in production while keeping debug available for temporary escalation during incidents.