A Distributed View from a Single GPU: RankLens for Distributed Training
RankLens is a single-rank emulation system that enables analysis, debugging, and tuning of distributed training with reduced resource requirements.
RankLens introduces a novel single-rank emulation approach that enables analysis and debugging of distributed training on a single GPU, unlike prior work that requires full-scale executions.
Before reading this…
Applications
- →Debugging
- →Analysis
- →Tuning
To understand this paper, make sure you know these concepts first:
- Distributed trainingfind papers →
- Single-rank emulationfind papers →
Abstract
More Like ThisModern distributed training is costly to debug, tune, and analyze because faithful analysis often requires repeated target-scale executions. We observe that many analysis tasks depend on the original distributed execution path, but not on full all-rank execution. We present RankLens, a single-rank emulation system that runs one survivor rank on a single GPU while preserving the target job's distributed semantics. The survivor rank executes the original training program and performs real local computation. At communication boundaries, the system replaces real multi-rank communication with task-dependent semantics, selectively restoring recorded communication results and recorded or estimated completion timing while preserving the asynchronous dependencies required for analysis. This enables replay from real executions and supports analysis, debugging, and tuning with existing debuggers and performance analysis tools. We evaluate the system on DeepSpeed and Megatron-LM across 8-64-GPU target configurations. RankLens achieves 100% training-tensor communication-structure match, keeps emulation timing error below 5%, supports local anomaly reproduction and execution analysis, and reduces the resource requirement of these analyses from target-scale clusters to a single GPU.