Mixture of Experts (MoE)

Mixture of Experts (MoE) is a model design that selects a few expert parts to process each token.