Compare MirroredStrategy and MultiWorkerMirroredStrategy — what has to change in your code and infrastructure to go multi-node?
Quick Answer
MirroredStrategy replicates the model across GPUs in one process with all-reduce sync; MultiWorkerMirroredStrategy extends it across machines using the TF_CONFIG environment variable for cluster discovery. Code changes are minimal (same strategy.scope() pattern); the work is in TF_CONFIG orchestration, data sharding, and fault-tolerant checkpointing.
Detailed Answer
Both are synchronous data parallelism. In-process MirroredStrategy needs: create variables under strategy.scope(), scale the batch size by replica count, and use Model.fit or a distributed custom loop. Multi-worker adds infrastructure semantics: every worker runs the same script; TF_CONFIG (JSON: cluster spec + this task's type/index) tells each process who's who; collective ops run over gRPC/NCCL between hosts. Operationally you must (1) shard data — tf.data auto-shards by file with AUTO policy, which requires enough input files, or shard manually per worker; (2) checkpoint only on the chief but to storage all workers can read, using BackupAndRestore for preemption recovery; (3) scale the global batch and LR together; (4) on Kubernetes, this maps cleanly to a Kubeflow TFJob or an indexed Job that renders TF_CONFIG per pod. Failure semantics are all-or-nothing per step (synchronous), so slow/flaky nodes stall the cluster — same class of problem as NCCL timeouts in PyTorch DDP.
Code Example
strategy = tf.distribute.MultiWorkerMirroredStrategy()
with strategy.scope():
model = build_and_compile()
model.fit(ds, epochs=10,
callbacks=[tf.keras.callbacks.BackupAndRestore('gs://ckpt/backup')])
# TF_CONFIG per worker (rendered by the operator):
# {"cluster": {"worker": ["w0:2222", "w1:2222"]}, "task": {"type": "worker", "index": 0}}Interview Tip
Interviewers listen for TF_CONFIG, chief-only checkpointing, and file-based auto-sharding needing enough files — the three things that actually bite when you first go multi-node.