A sparse model activates only a few experts for each token. Why might it still need more memory than a dense model? Separate the weights that exist from the weights used in one token's computation.
Before you begin: You understand transformer feed-forward layers and softmax.
Do not infer architecture from a company name. The original Mistral 7B is a dense model. Mixtral 8x7B uses a sparse mixture of experts in its feed-forward layers. They are distinct designs.
What the router actually chooses
In a Mixtral-style block, a router reads each token's current representation and scores a set of expert networks. A top-k rule selects a small number of experts. Their outputs are combined using routing weights and passed back into the residual stream.
In the original Mixtral 8x7B description, each token selects two of eight experts per layer. The next token, or the same token at another layer, can select different experts. Attention and other shared parts still participate. The model is not eight independent complete 7B assistants voting on an answer.
# Runnable: Python 3, standard library.
import math
scores = [2.0, 1.0, -1.0, 0.0]
chosen = sorted(range(len(scores)),
key=lambda i: scores[i], reverse=True)[:2]
peak = max(scores[i] for i in chosen)
raw = [math.exp(scores[i] - peak) for i in chosen]
weights = [v / sum(raw) for v in raw]
assert chosen == [0, 1]
assert math.isclose(sum(weights), 1.0)
print(chosen, [round(w, 3) for w in weights])
This illustrates selecting and renormalizing two experts. Real implementations also compute expert outputs, dispatch tokens, and handle training and communication details.
Separate capacity, computation, and memory
Total parameters describe all stored weights. Active parameters describe the subset participating for a token under a stated counting convention. Neither number alone is a complete memory or latency estimate.
Usually all expert weights must remain accessible somewhere. Keeping only the active weights for one token would not help much when the next token chooses different experts. Distributing experts across devices can reduce per-device storage, but token dispatch adds communication and can create bottlenecks.
A useful measurement includes batch size, context length, hardware, cache usage, and the distribution of tokens across experts. Sparse arithmetic does not guarantee a proportionally faster endpoint.
Move the bottleneck by changing the batch
Consider four experts stored on four devices. In a toy batch, every token selects experts zero and one. Those two devices receive the work while the others wait. In another batch, selections are evenly spread. The total number of selected expert operations can be the same, yet completion time can differ because the busiest device controls progress.
What would you measure before claiming that a routing change improved serving efficiency?
Inspect work distribution and movement
Record tokens dispatched per expert, time spent communicating, and end-to-end latency at the same workload. Also test output quality: a balanced router that sends tokens to unhelpful experts may improve utilization while harming the task. The toy example illustrates a bottleneck; it is not a measured Mixtral benchmark.
This is why “two active experts” is an incomplete performance description. Placement, batch composition, routing, and communication all affect how the sparse computation reaches the hardware.
Why balancing matters
If nearly every token selects the same expert, that expert becomes busy while others contribute little. Training methods can encourage a healthier distribution. Some systems impose expert capacity limits or use different balancing strategies, so inspect the specific implementation before assuming whether overflow tokens are dropped, rerouted, or processed.
Also keep attention mechanisms separate from expert routing. The original Mistral paper describes sliding-window attention and grouped-query attention. Those features do not imply that every Mixtral release uses identical attention settings. An MoE layer changes feed-forward computation; it does not by itself shorten the attention window.
Make a deployment prediction
Exercise: an MoE checkpoint has fewer active parameters per token than a dense checkpoint but requires more total weight storage. Your device has limited memory. Which is easier to run?
Compare your reasoning
Active-parameter count alone cannot decide. Check total resident weights, quantization support, cache memory, and whether offloading or distribution is acceptable. The dense model may be easier to fit even if the MoE has attractive arithmetic costs.
Next, return to a foundational model paper and practice separating architectural evidence from the reputation of a model family.
Sources
Compare Mistral 7B with Mixtral of Experts. Treat their configurations and reported experiments as release-specific historical evidence.
Continue: Paper: LLaMA : Open and Efficient Foundation Models.