Hugging Face Transformers

The Auto classes are a factory pattern: AutoModelForSequenceClassification.from_pretrained('x') fetches x's config.json, reads its architecture/model_type field, and returns the matching concrete class (e.g., RobertaForSequenceClassification) with weights loaded. That indirection is what makes checkpoints interchangeable in code. pipeline() layers task logic on top: tokenization with sensible truncation, forward pass, and task-specific post-processing (label mapping, span extraction, generation loop). Use pipeline for internal tools and quick evaluation; drop to Auto classes when you need padding/batching strategy control for throughput, torch_dtype/device_map placement, custom generation loops, streaming token output, or export paths (ONNX/torch.compile). Also know the task-head distinction: AutoModel returns hidden states only; the ForXxx variants add task heads — using bare AutoModel then wondering where the logits went is a classic mistake.

PyTorch's original .bin checkpoints are Python pickles, and unpickling executes arbitrary embedded code — downloading community weights meant running untrusted code with your service's credentials. safetensors eliminates the class of attack by design: the format holds only metadata + tensor bytes, no code paths. Operationally it also loads faster: the file is memory-mapped, so tensors materialize lazily and can be read zero-copy straight to the target device, and sharded checkpoints (model-00001-of-00004.safetensors + index.json) let loaders place shards across devices without reading everything into host RAM first. It's now the Hub default and effectively mandatory in security-conscious pipelines; scanners flag pickle files on upload. In an interview, tie it to policy: enforce use_safetensors=True (or equivalent) in production loaders so a compromised or sloppy upstream repo can't regress you to pickle.

from_pretrained with device_map='auto' computes a placement plan from each submodule's size and each device's free memory, then streams checkpoint shards to their target devices — avoiding the naive path of materializing the whole model in host RAM first. Layers that don't fit spill to CPU (and then disk offload), which works but serializes execution across the PCIe bus — fine for batch jobs, usually unacceptable for latency-sensitive serving. Quantization changes the arithmetic: BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4') stores weights in 4-bit with computation in bf16, cutting a 70B model from ~140GB to ~35GB. For inference at scale, pre-quantized AWQ/GPTQ checkpoints served through vLLM/TGI typically beat on-the-fly bnb quantization on both speed and simplicity. Two caveats worth voicing: quantization quality loss is task-dependent (always eval on your own benchmark), and device_map='auto' is a placement tool, not tensor parallelism — for real multi-GPU throughput you want an inference engine with TP, not naive layer spreading.

LoRA reparameterizes weight updates as W + BA where B and A are low-rank (r=8-64). With PEFT: wrap the model in get_peft_model(model, LoraConfig(...)), train normally (Trainer/TRL), and save only the adapter. QLoRA goes further — base model in 4-bit, adapters in bf16 — bringing 70B fine-tunes into single-node reach. Why it won: (1) memory — no optimizer states for frozen weights, which is where most training memory goes; (2) artifact size — adapters are MBs, so per-customer/per-task variants are storable and auditable; (3) serving economics — engines like vLLM and LoRAX hot-load many adapters over one shared base model, making 50 fine-tunes cost roughly one deployment; (4) merge option — merge_and_unload() folds adapters into the base for zero inference overhead when you only need one variant. Know the knobs: rank r and lora_alpha scale capacity, target_modules choose which projections get adapters, and full fine-tuning still wins when you must shift the model's behavior broadly (new language/domain) rather than specialize a task.

Runtime downloads are a triple liability: slow autoscaling (every new pod pays the download), an availability dependency on huggingface.co (rate limits: HTTP 429s during mass scale-out), and unpinned drift if the repo moves its main branch. Fixes in preference order: (1) bake weights into the image at build time with huggingface-cli download --revision <sha> into HF_HOME — images get big (use layer caching so weights are a stable layer), but startup is pure disk mmap of safetensors; (2) shared read-only volume or cloud snapshot (EFS/GCS FUSE/PVC) pre-populated by a sync job — keeps images small, adds a mount dependency; (3) a warm cache layer like an internal Hub mirror for many models. In all cases set HF_HUB_OFFLINE=1 (hard-fail on network attempts) and pin revision= to a commit SHA in from_pretrained. Add a readiness probe that actually runs one inference so the pod joins the LB only after the model is mapped and warm.

Autoregressive generation makes naive serving pathological: each request holds the GPU for its full generation length, requests of different lengths can't share a static batch efficiently, and the KV cache for long contexts fragments memory. Continuous (in-flight) batching schedules at token granularity — new requests join the running batch as others finish — keeping the GPU saturated at high concurrency. vLLM's PagedAttention allocates KV cache in fixed-size blocks (virtual-memory style), nearly eliminating fragmentation and enabling prefix sharing across requests with a common prompt. TGI (Rust server + optimized kernels) brings the same class of features with tight HF Hub integration. Both give you: token streaming (SSE/gRPC), tensor parallelism across GPUs, quantized model support (AWQ/GPTQ/FP8), Prometheus metrics, and an OpenAI-compatible API. The DIY FastAPI path is legitimate only for small models, low QPS, or non-generative tasks (classification/embeddings) where a single forward pass per request batches trivially.

The headline: TensorFlow and JAX support was dropped — transformers is now PyTorch-first, betting on interop instead of multi-framework maintenance. Model definitions were simplified so the same modeling code powers training frameworks and serving engines (vLLM can load architectures directly from transformers), reducing 'works in training, differs in serving' drift. Quantization became first-class, improving bitsandbytes support for tensor-parallel and MoE models and easing integration of new quant methods. Ops checklist for the migration: (1) grep for TFAutoModel/FlaxAutoModel and tf.* touchpoints — those paths are gone; (2) review the v5 deprecation list against your code (Trainer args and tokenizer behaviors changed across the 4.x line); (3) re-pin the ecosystem matrix together — peft/trl/accelerate/vllm versions are coupled; (4) golden-output regression tests per model, since kernel and default changes can shift numerics slightly; (5) canary in staging under production traffic shapes before fleet-wide rollout. Version-pin discipline (exact ==, upgrade deliberately) matters more than usual in this ecosystem because minor releases move fast.