Why do teams serve LLMs through TGI or vLLM instead of calling model.generate() behind FastAPI?
Quick Answer
model.generate() handles one request at a time with static batching and no KV-cache management — GPUs idle while requests queue. Dedicated engines add continuous batching, paged KV-cache (vLLM's PagedAttention), tensor parallelism, quantized kernels, and token streaming, commonly delivering 5-20x higher throughput on the same hardware.
Detailed Answer
Autoregressive generation makes naive serving pathological: each request holds the GPU for its full generation length, requests of different lengths can't share a static batch efficiently, and the KV cache for long contexts fragments memory. Continuous (in-flight) batching schedules at token granularity — new requests join the running batch as others finish — keeping the GPU saturated at high concurrency. vLLM's PagedAttention allocates KV cache in fixed-size blocks (virtual-memory style), nearly eliminating fragmentation and enabling prefix sharing across requests with a common prompt. TGI (Rust server + optimized kernels) brings the same class of features with tight HF Hub integration. Both give you: token streaming (SSE/gRPC), tensor parallelism across GPUs, quantized model support (AWQ/GPTQ/FP8), Prometheus metrics, and an OpenAI-compatible API. The DIY FastAPI path is legitimate only for small models, low QPS, or non-generative tasks (classification/embeddings) where a single forward pass per request batches trivially.
Code Example
# vLLM, OpenAI-compatible, TP across 2 GPUs python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 2 --max-model-len 8192 \ --gpu-memory-utilization 0.90 # TGI docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference \ --model-id mistralai/Mistral-7B-Instruct-v0.3 --quantize awq
Interview Tip
Continuous batching and PagedAttention are the two magic words; explain them in one sentence each. Also credit the DIY path's legitimate niche (embeddings/classification) to show judgment, not fashion.