Weight Initialization - Page 3

Weight initialization determines the starting values of a neural network’s parameters before training begins. If weights start too large, activations and gradients may explode; if they are too small or identical, signals can vanish or neurons may learn the same features. Methods such as Xavier and He initialization choose random scales based on the number of incoming or outgoing connections. A suitable method helps information and gradients travel through deep networks during early updates. Initialization does not replace good data or optimization, and its effect depends on activation functions, normalization, architecture, and random seed. Repeated runs help reveal whether results are unusually sensitive to the starting state.