Explain device_map='auto' and quantized loading — how do you fit a model that's bigger than one GPU?

from_pretrained with device_map='auto' computes a placement plan from each submodule's size and each device's free memory, then streams checkpoint shards to their target devices — avoiding the naive path of materializing the whole model in host RAM first. Layers that don't fit spill to CPU (and then disk offload), which works but serializes execution across the PCIe bus — fine for batch jobs, usually unacceptable for latency-sensitive serving. Quantization changes the arithmetic: BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4') stores weights in 4-bit with computation in bf16, cutting a 70B model from ~140GB to ~35GB. For inference at scale, pre-quantized AWQ/GPTQ checkpoints served through vLLM/TGI typically beat on-the-fly bnb quantization on both speed and simplicity. Two caveats worth voicing: quantization quality loss is task-dependent (always eval on your own benchmark), and device_map='auto' is a placement tool, not tensor parallelism — for real multi-GPU throughput you want an inference engine with TP, not naive layer spreading.