Mixture of Experts (MoE)

In a dense transformer, every parameter participates in every token. A mixture-of-experts model replaces some of the feed-forward layers with a set of parallel "experts" plus a small router that picks which few experts handle each token. The model can have a very large total parameter count while the compute per token stays close to that of a much smaller model.

That is why MoE shows up in many recent large open-weight and proprietary models, for example the Mixtral and DeepSeek-V3 families. When a model page lists both "total" and "active" parameters, it is describing an MoE design.

For people choosing agent models it matters mostly for cost and speed: a large MoE can be cheaper to serve per token than a dense model of equal total size, though it still has to hold all experts in memory.

How it works

Each MoE layer has N expert networks and a learned router. For every token the router scores the experts and selects the top few (commonly one to a handful), and the layer output is the weighted sum of just those experts. Training adds balancing pressure so tokens spread across experts rather than collapsing onto a few. Despite the name, experts do not usually correspond to human-readable subjects like "law" or "code"; specialization is learned and tends to be fuzzy, often at the level of token patterns.

Example

A model card says "600B total parameters, 40B active". Every token is routed through the router to a few experts, so the compute per token resembles a 40B dense model, while the full 600B of weights must still be loaded into GPU memory to serve it.

How it differs

Common misconceptions

FAQ