Data Parallelism

Data parallelism distributes machine learning work by placing a copy of the model on multiple processors and giving each copy a different subset of a batch. Every device computes gradients from its local examples, then the gradients are aggregated so all replicas receive a consistent parameter update. The method can shorten training time when the model fits on each device and enough data is available to keep the hardware busy. Performance depends on communication speed, batch size, synchronization, load balance, and optimizer behavior. Very large global batches may change convergence, while slow connections can erase the benefit. Data parallelism differs from model parallelism, which divides one model across devices.