PyTorch RPC: Distributed Deep Learning Built on Tensor-Optimized Remote Procedure Calls

Distributed training technologies have advanced rapidly in the past few years and have unlocked unprecedented scalability with increasingly complex solutions. These technologies have made distributed training much more efficient and accessible, though they impose specific constraints on the training paradigm or the model structure.