Mock scenario: your container takes 6 minutes to become ready because it downloads a 15GB model from the Hub at startup. Fix the cold start.
Quick Answer
Stop depending on the Hub at runtime: bake the model into the image (or a mounted volume/snapshot) during build, set HF_HUB_OFFLINE=1 in production, and pin an exact revision. Cold start becomes disk-load time, and Hub outages or rate limits can no longer take down scale-up.
Detailed Answer
Runtime downloads are a triple liability: slow autoscaling (every new pod pays the download), an availability dependency on huggingface.co (rate limits: HTTP 429s during mass scale-out), and unpinned drift if the repo moves its main branch. Fixes in preference order: (1) bake weights into the image at build time with huggingface-cli download --revision <sha> into HF_HOME — images get big (use layer caching so weights are a stable layer), but startup is pure disk mmap of safetensors; (2) shared read-only volume or cloud snapshot (EFS/GCS FUSE/PVC) pre-populated by a sync job — keeps images small, adds a mount dependency; (3) a warm cache layer like an internal Hub mirror for many models. In all cases set HF_HUB_OFFLINE=1 (hard-fail on network attempts) and pin revision= to a commit SHA in from_pretrained. Add a readiness probe that actually runs one inference so the pod joins the LB only after the model is mapped and warm.
Code Example
# Dockerfile
RUN huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
--revision 8c22764a7e3675c50d4c7c9a4edb474456022b16 \
--local-dir /opt/models/llama-3.1-8b
ENV HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1
# app
model = AutoModelForCausalLM.from_pretrained('/opt/models/llama-3.1-8b')Interview Tip
Say 'the Hub is a build-time dependency, not a runtime dependency' — that principle, plus pinning to a commit SHA rather than a branch, is exactly what the interviewer is fishing for.