How do you design an observability stack with Prometheus, Grafana, and OpenTelemetry for a microservices platform?

Think of observability like a hospital's monitoring systems. Metrics are the vital signs monitors — heart rate, blood pressure, oxygen level — constantly streaming numbers that tell you the patient's overall condition at a glance. Logs are the patient's medical chart — detailed notes about every event, every medication administered, every nurse interaction. Traces are the surgical camera footage — they follow a single procedure from start to finish, showing every step, every hand-off between specialists, and exactly where delays occurred. No single system is sufficient; a doctor needs all three to diagnose a complex case.

The metrics layer starts with Prometheus, deployed via the kube-prometheus-stack Helm chart which includes Prometheus Operator, Grafana, Alertmanager, and a suite of default recording rules and dashboards. Prometheus scrapes /metrics endpoints from your applications and Kubernetes components at configurable intervals (typically 15-30 seconds). For a banking microservices platform, your applications should expose RED metrics (Rate, Errors, Duration) and USE metrics (Utilization, Saturation, Errors) using the Prometheus client library. OpenTelemetry can also emit metrics in Prometheus exposition format, letting you standardize on a single instrumentation SDK. For high-cardinality environments, consider Thanos or Mimir as a long-term storage backend that extends Prometheus with global querying across clusters and months of retention.

The logging layer uses Fluent Bit as a lightweight DaemonSet collector that reads container logs from /var/log/pods on each node. Fluent Bit parses, filters, and enriches logs with Kubernetes metadata (pod name, namespace, labels) before shipping them to a backend. For cost-effective log storage, Grafana Loki is the preferred choice — it indexes only labels (namespace, pod, container) rather than full-text, making it 10x cheaper than Elasticsearch for log volumes typical in microservices platforms. OpenTelemetry Collector can replace Fluent Bit as the log collector, providing a single agent for all three signals. In banking environments, logs must be retained for regulatory compliance — typically 7 years for transaction logs — so a tiered storage strategy (hot/warm/cold with S3-compatible object storage) is essential.

The tracing layer uses OpenTelemetry SDKs embedded in each microservice to generate distributed traces. Each incoming request gets a trace ID that propagates through all downstream service calls via HTTP headers (traceparent) or gRPC metadata. The OTel SDK records spans for each operation — database queries, HTTP calls, message queue operations — and exports them to the OTel Collector, which buffers, processes, and ships traces to Tempo or Jaeger. Tempo is preferred for Grafana-native stacks because it integrates with Loki and Prometheus for cross-signal correlation. For a payments flow, a single trace shows the complete journey: API gateway receives request (50ms) → payments-api validates input (10ms) → fraud-detector scores risk (80ms) → settlements-processor initiates transfer (200ms) → transaction-router sends to payment network (500ms).

The unifying layer is Grafana, which connects to Prometheus (metrics), Loki (logs), and Tempo (traces) as data sources and enables correlation between all three signals. A metric spike on the payments-api error rate dashboard links to exemplars — specific trace IDs that experienced the error. Clicking the trace ID opens the trace view in Tempo showing the exact failing span. The span metadata includes a link to Loki logs filtered by the same trace ID and time window. This metrics-to-traces-to-logs workflow reduces incident diagnosis time from hours to minutes.

The gotcha that undermines most observability implementations: collecting everything and alerting on nothing useful. Teams deploy the full stack, create 50 dashboards, set up 200 alerts, and then suffer alert fatigue so severe that engineers start ignoring PagerDuty. The fix is ruthless prioritization: dashboard the four golden signals (latency, traffic, errors, saturation) for each service, alert only on SLO violations using multi-window burn rate, and treat every other metric as diagnostic data you look at during incidents, not something that wakes people up.