Mock scenario: TF Serving p99 latency doubles at peak traffic while p50 barely moves. What do you investigate?
Quick Answer
That signature points at queueing — most likely server-side request batching tuned wrong (batch_timeout too high or max_batch_size too big), thread pool saturation, or several models sharing one GPU. Check the batching config, monitoring metrics, and per-model isolation.
Detailed Answer
p50 flat + p99 exploding under load means most requests are fine but some wait in a queue. In TF Serving, work through: (1) Batching config — enable_batching groups requests to boost GPU throughput at the cost of waiting up to batch_timeout_micros; oversized max_batch_size or long timeouts trade tail latency for throughput. Tune max_batch_size down, timeout to ~1-5ms, and set num_batch_threads ~ CPU cores and a bounded max_enqueued_batches so overload fails fast instead of queueing forever. (2) Saturation — check the rest/grpc worker threads and whether TF intra/inter-op parallelism is fighting other processes on the box. (3) Multi-model contention — several models in one server share the GPU and batching queues; hot models deserve their own server/GPU. (4) Version reloads — model hot-swaps at peak can stall requests (there are documented memory/latency spikes on reload); schedule rollouts off-peak. Instrument via TF Serving's Prometheus endpoint (:tensorflow:serving:batching queuing latency histograms) before turning any knob.
Code Example
# batching_parameters.txt
max_batch_size { value: 16 }
batch_timeout_micros { value: 2000 }
num_batch_threads { value: 8 }
max_enqueued_batches { value: 100 }
tensorflow_model_server --enable_batching=true \
--batching_parameters_file=batching_parameters.txt \
--monitoring_config_file=prometheus.confInterview Tip
Recognize the p50-vs-p99 divergence as a queueing signature out loud — that one sentence of reasoning is worth more than listing every TF Serving flag.