What is BytePS?
BytePS is a method used for training deep neural networks. It is a distributed approach that can be used with varying numbers of CPU machines. BytePS can handle traditional all-reduce and parameter server (PS) as two special cases within its framework.
How does BytePS work?
BytePS makes use of a Summation Service and splits a DNN optimizer into two parts: gradient summation and parameter update. For faster DNN training, the CPU-friendly part, gradient summation, is kept on CPUs. The more computation-heavy part, parameter update, is moved to GPUs.
Why use BytePS?
BytePS is especially useful for those who are training deep neural networks and need to do so over a distributed system. The method offers faster training, as well as the ability to handle varying numbers of CPU machines. It also offers improved efficiency, as the computation-heavy part is moved to GPUs.
For those interested in accelerating DNN training times, BytePS offers a promising solution. Its ability to handle different numbers of CPU machines and its use of a Summation Service make it an attractive option for those looking to distribute their training across multiple machines. Additionally, the method's splitting of the DNN optimizer into gradient summation and parameter update makes it possible to leverage the computational power of GPUs for faster training times.
