Your training job on a V100 shows the GPU at 20% utilization. Walk me through diagnosing the input pipeline with tf.data.
Quick Answer
Low GPU utilization almost always means the tf.data pipeline can't feed the GPU. Profile with the TensorBoard profiler, then fix with parallel map (num_parallel_calls=AUTOTUNE), prefetch(AUTOTUNE), caching, and moving heavy Python work out of the hot path.
Detailed Answer
Diagnose first: the TensorBoard profiler's input-pipeline analyzer shows exactly how long the model waits on input each step. Common causes, in the order I check: (1) sequential map calls doing Python-level decode/augment — set num_parallel_calls=tf.data.AUTOTUNE and vectorize where possible; (2) missing prefetch(AUTOTUNE) at the end, so CPU prep and GPU compute alternate instead of overlap; (3) re-doing expensive work each epoch that .cache() (RAM or file) could amortize; (4) using py_function for augmentation, which is single-threaded through the GIL — replace with native TF ops or preprocess offline; (5) small files on network storage — convert to TFRecord shards with interleave for parallel reads; (6) shuffle buffer too large causing startup stalls, or too small hurting randomness. Order matters too: the canonical chain is interleave(read) -> shuffle -> map(parse, AUTOTUNE) -> batch -> map(augment on batch) -> prefetch.
Code Example
ds = (tf.data.TFRecordDataset(files, num_parallel_reads=tf.data.AUTOTUNE)
.shuffle(50_000)
.map(parse, num_parallel_calls=tf.data.AUTOTUNE)
.batch(256)
.prefetch(tf.data.AUTOTUNE))
# TensorBoard: Profile tab -> input-pipeline analyzerInterview Tip
Name the profiler first — candidates who jump straight to 'add prefetch' without a measurement step read as cargo-cult. Then the AUTOTUNE trio: parallel reads, parallel map, prefetch.