Your service's p99 latency spiked from 50ms to 500ms but p50 remains at 20ms. The service has not been deployed recently. How do you investigate?
Quick Answer
A p99 spike with stable p50 indicates a small percentage of requests hitting a slow path — check for noisy neighbors, GC pauses, network partition to specific backends, or a single degraded node.
Detailed Answer
Investigation Framework
Step 1: Scope the problem - Is it affecting all endpoints or specific ones? - Is it correlated with time of day, specific users, or geographic regions? - When exactly did it start? Correlate with infrastructure events.
Step 2: Narrow to infrastructure layer - kubectl top nodes — check for a single hot node - Check if slow requests correlate with specific pods/nodes (add node labels to metrics) - Look at network latency between AZs (cross-AZ calls add 1-2ms each)
Step 3: Common root causes for p99 spike with stable p50 1. GC pauses: JVM/Go GC pauses affect ~1% of requests. Check GC logs, -XX:+PrintGCDetails 2. Noisy neighbor: A pod on a shared node is CPU-throttled. Check cpu.throttled_time in cgroup stats 3. Connection pool exhaustion: Database connection pool full, 1% of requests wait for a free connection 4. Single degraded backend: If service calls 10 backends, one slow backend affects ~10% of requests 5. Kernel CPU scheduling: CFS throttling with small CPU limits causes periodic latency spikes 6. Disk I/O on shared storage: EBS/NFS latency spike on underlying hardware
Step 4: Deep diagnosis - Distributed tracing (Jaeger/Tempo): Filter for traces > 200ms, look at span waterfall - Check Prometheus: histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m])) - Node-level: perf top, iostat -x 1, ss -s for socket stats
Production at Scale
Always have per-node and per-pod latency breakdowns in your dashboards. Use request hedging for latency-sensitive services. Set up burn-rate alerts on SLO violations, not just threshold alerts.
Code Example
# Check CPU throttling kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.stat # Look for nr_throttled and throttled_time # PromQL for p99 by pod histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, pod) ) # Check GC pauses in Java kubectl logs <pod> | grep 'GC pause' # Network latency between nodes kubectl exec <pod-on-node-a> -- ping -c 10 <pod-ip-on-node-b>
Interview Tip
This is the classic SRE debugging question. The key insight is that p99 spike with stable p50 means only 1% of requests are affected — so look for intermittent issues, not systemic ones. Always structure your answer: scope → narrow → diagnose → fix.