Mock scenario: you must serve a PyTorch model to a fleet of CPU-only servers with strict p99 latency. What deployment options do you weigh?
Quick Answer
Export to ONNX Runtime or torch.export/AOTInductor for a Python-free, optimized runtime; quantize to int8; pin thread counts; and batch carefully. Raw eager PyTorch behind FastAPI is the fallback, not the goal.
Detailed Answer
For CPU inference the wins come from (1) getting out of eager Python execution, (2) quantization, and (3) thread discipline. Options: ONNX Runtime — export once, get graph fusions and a mature int8 quantization toolchain; usually the best latency/effort ratio and language-agnostic serving. torch.export + AOTInductor (the modern TorchScript replacement) — ahead-of-time compiles to a shared library loadable from C++ without the Python interpreter. TorchScript still works but is de-emphasized. Dynamic int8 quantization typically gives 2-4x on linear-heavy models with small accuracy cost — validate on a golden set. Thread control matters more than people expect: the default 'use all cores' setting causes thrashing when multiple workers share a box — set torch.set_num_threads / OMP_NUM_THREADS so workers x threads = physical cores. Finally decide batching policy: micro-batching helps throughput but hurts p99; for strict p99, batch size 1 with more replicas is often correct.
Code Example
# ONNX export + int8 torch.onnx.export(model, sample, 'model.onnx', dynamo=True) # quantize with onnxruntime.quantization.quantize_dynamic # Thread discipline per worker OMP_NUM_THREADS=4 uvicorn app:api --workers 8 # 32 physical cores
Interview Tip
Interviewers want the decision framework, not one tool: escape eager mode, quantize, control threads, then choose batching by latency SLO. Mentioning that torch.export/AOTInductor superseded TorchScript is a current-knowledge marker.