Mixture-of-Experts: The Architecture Behind Almost Every Frontier Model Now

A practical explanation of mixture-of-experts (MoE) architecture: what experts and routers actually do, why sparse activation cuts compute but not memory, how Mixtral, DeepSeek-V3 and Llama 4 use it, and what it means if you are choosing or deploying models.

What mixture-of-experts actually means

Mixture-of-experts (MoE) is a transformer design where the feed-forward layer is split into many parallel sub-networks called experts, and a small router network picks only a few of them to run for each token. So a model can hold 671 billion parameters in total while using around 37 billion for any single token it processes. That is the whole trick: total capacity goes up, compute per token stays roughly flat.

In a normal dense transformer, every parameter fires for every token. The word "the" costs exactly as much compute as a line of Python. Scaling a dense model means paying for that everywhere, in training and again on every inference call, forever.

MoE breaks that link. You replace one big feed-forward block with, say, 128 smaller ones, add a router that scores which experts suit the current token, and activate the top 2 or top 8. The attention layers usually stay dense and shared. Only the feed-forward part gets sparse, and since feed-forward layers hold most of a transformer's parameters, that is where the savings live.

The idea is not new. Google's GShard paper landed in 2020 and Switch Transformer in 2021, which pushed to 1.6 trillion parameters with top-1 routing. What changed recently is that MoE stopped being a research curiosity and became the default for frontier-scale models.

placeholder