How do you stop TensorFlow from grabbing all GPU memory at startup, and when is that default actually the right choice?
Quick Answer
Enable memory growth per GPU (tf.config.experimental.set_memory_growth) so TF allocates on demand, or set a hard per-process cap with set_logical_device_configuration. The grab-everything default is right when one training job owns the whole GPU — it avoids fragmentation and allocator overhead.
Detailed Answer
By default TF maps nearly all GPU memory at initialization to run its own sub-allocator (BFC) on one big arena — efficient and fragmentation-resistant for a single tenant. It becomes a problem when the GPU is shared: notebooks + a service, multiple inference processes, or TF alongside PyTorch. set_memory_growth(gpu, True) makes TF start small and extend the arena as needed (memory is not returned once grown). set_logical_device_configuration with a memory_limit gives a hard cap, letting you bin-pack N processes deterministically onto one card — usually better than growth for co-located inference services because a leak in one process can't starve the rest. Both must be set before any op initializes CUDA. On the ops side, this pairs with Kubernetes GPU sharing choices (time-slicing/MPS/MIG), where per-process caps keep tenants honest.
Code Example
gpus = tf.config.list_physical_devices('GPU')
for g in gpus:
tf.config.experimental.set_memory_growth(g, True)
# Hard cap: two 20GB tenants on one 48GB card
tf.config.set_logical_device_configuration(
gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=20480)])Interview Tip
Most candidates only know 'set memory growth'. Adding when the default is correct (single-tenant training, fragmentation-resistant) and the hard-cap alternative for co-located services shows real operational judgment.