Explain what torch.compile does and when you would NOT use it in production.
Quick Answer
torch.compile traces your model with TorchDynamo and JIT-compiles fused kernels via TorchInductor, typically 1.3-2x speedups. Skip it when inputs have highly variable shapes (recompilation storms), when custom ops cause graph breaks, or when another engine like TensorRT-LLM already owns optimization.
Detailed Answer
torch.compile captures the Python-level model code into an FX graph (TorchDynamo), then compiles fused GPU kernels (TorchInductor), eliminating Python overhead and kernel launch latency. As of PyTorch 2.6+ it is production-stable and widely used for LLM inference, often combined with CUDA graphs. But it has real costs: a 30-60 second warm-up compile for a 7B model, cache invalidation on shape changes, and recompilation whenever guard conditions fail. Do not use it when (1) request shapes vary wildly and you cannot bucket-pad — recompilation overhead can exceed the speedup; (2) the model uses custom ops Dynamo cannot trace, so graph breaks fall back to eager and you keep the complexity without the gains; (3) you already serve through TensorRT-LLM or vLLM, which manage their own compilation; (4) cold-start latency matters more than steady-state throughput (e.g., scale-to-zero serverless). Also note a documented gotcha: torch.compile can create a CUDA context even for CPU-only code paths, which surprises multi-process CPU deployments.
Code Example
model = torch.compile(model, mode='reduce-overhead') # CUDA graphs for small batches
# Bucket-pad to avoid recompilation storms
# lengths -> nearest of {128, 256, 512, 1024}
x = F.pad(x, (0, bucket_len - x.shape[-1]))Interview Tip
Everyone can say "it makes models faster." Differentiate by naming the failure modes: recompilation storms from dynamic shapes, graph breaks from custom ops, and the compile-time cold-start tax.