Mixture of Experts (MoE) - Page 8

A mixture of experts, or MoE, is a model architecture that contains multiple specialized subnetworks and a routing system that selects which experts handle each input. Only a subset may be activated for a given token or example, allowing the model to contain many parameters without using all of them for every calculation. This can expand capacity while controlling inference cost. Training remains complex because routing must distribute work effectively, avoid overloading a few experts, and maintain communication across hardware. An MoE model is not automatically interpretable simply because its components are called experts; their learned specializations may be unstable or difficult to describe and must be evaluated empirically.