Abstract: GPU failures are common in large-scale AI training clusters, but not every failure requires new hardware. Some interrupt large language model training yet leave the GPUs usable, so training could resume on the same hardware without a reset, repair, or replacement. Existing recovery systems nevertheless restart from scratch: they reload checkpoints, recompute lost progress, and rebuild process state, wasting substantial GPU-hours.

This talk presents Leto, a fault-tolerant training system that enables fast in-place recovery on surviving hardware. Our key insight is that the state needed to resume training can be retained or prepared outside the active training process while remaining on the same hardware. We devise two-tier erasure protection and chunk-level transactional updates to keep the retained model state recoverable and consistent. We also preinitialize process state in a shadow trainer, reclaiming its GPU memory when needed for active training. On 6- and 72-GPU NVIDIA A100 clusters, Leto recovers 3.6–6.5× faster than the best-performing checkpointing baselines and improves productive training time by up to 13.7 percentage points. Large-scale simulation shows over 95% productive training time on a 131,072-GPU cluster.

Bio: Geon-Woo Kim is a PhD candidate in Computer Science at The University of Texas at Austin, working with Daehyeok Kim and Aditya Akella. His research addresses the systems challenges of scaling large language model training. He has proposed a communication-efficient optimization algorithm for cross-region training and developed a system that enables training jobs to recover from failures using surviving hardware.