Mixture of Experts (MoE)

Technical Overview of Mixture of Experts (MoE)

Mixture of Experts (MoE) is a sparse transformer architecture that replaces dense feed-forward network (FFN) layers with multiple independent “expert” networks.

Dense models process every token using all parameters, which increases computational requirements. MoE addresses this by using a gating network (router) to dynamically assign each token to a subset of experts. This increases overall model capacity while keeping the computational cost per token relatively constant.

Key Architecture & Implementation

The MoE architecture consists of two primary components:

1. Sparse MoE Layers:
   • A set of independent feed-forward networks (e.g., 8 or 16 experts per layer).
2. Gating Router:
   • A small parameterized network that calculates a probability distribution over the experts for each token.
   • Top-K Routing: Most models activate only the top-1 or top-2 experts per token ($K=1$ or $K=2$), keeping other expert networks inactive.

Input Token ---> [ Gating Router ] ---\
                   | (Top-2)         |
                   v                 v
             [ Expert 1 ]      [ Expert 4 ]  (Others remain inactive)

Core Parameters

• Active Parameters: The number of parameters used to process a single token (e.g., 12.9B active parameters in a model with 47B total parameters).
• Load Balancing: The router must distribute tokens evenly across experts to prevent hardware bottlenecks.

Real-world Applications

• Used in models like Mixtral 8x7B, Mixtral 8x22B, and GPT-4 to achieve high performance with lower compute budgets.

Why It’s Become Central to Frontier Models

MoE architectures let a model have a very large total parameter count — and therefore a large capacity to store learned knowledge — while keeping the actual compute cost per token much lower, since only a small subset of “expert” sub-networks are activated for any given token, decided by a learned routing function. This decoupling of total parameters from per-token compute cost is why several of the most capable open-weight and rumored closed-weight frontier models use MoE designs: it’s possible to scale a model’s knowledge capacity substantially without proportionally scaling inference cost, which was previously the harder constraint. The tradeoff is architectural and operational complexity — training a router that distributes tokens sensibly across experts without collapsing onto only using a few of them is a genuinely difficult optimization problem, and serving an MoE model efficiently requires keeping all experts loaded in memory even though only a few process any given token, which can make MoE models more memory-hungry to deploy than a dense model with the same active-parameter count, even though they’re cheaper to run computationally.