Mixture of Experts (MoE) in Depth
Mixture of Experts lets a model have many parameters without using all of them for every token.
Dense model:
text
every token -> same feed-forward network
MoE model:
text
each token -> router -> a few selected experts
Why MoE exists
Large dense models are expensive because every token activates the whole model. MoE increases total capacity while keeping active compute lower.
| Benefit | Explanation |
|---|---|
| more capacity | many experts store different patterns |
| lower active compute | only a few experts run per token |
| specialization | experts can learn different domains or token types |
| efficient scaling | more parameters without proportional inference cost |
Router
The router decides which expert gets each token.
Important router choices:
- top-1 or top-2 experts
- load balancing loss
- capacity limits per expert
- dropping or rerouting overflow tokens
- expert parallelism across GPUs
What can go wrong
MoE is powerful but tricky:
- one expert gets overloaded
- experts collapse and stop specializing
- routing adds communication cost
- serving needs expert placement strategy
- batching becomes harder
- debugging is less intuitive than dense models
When engineers should care
You need MoE knowledge when:
- choosing open-weight models
- estimating inference cost
- deploying expert-parallel systems
- reading modern model cards
- comparing dense vs sparse architectures
Knowledge check
Q1: What does sparse activation mean?
Only part of the model, usually selected experts, runs for each token.
Q2: Why does MoE need load balancing?
Without it, too many tokens may route to the same expert and waste capacity.