Two checkpoints both say “Llama,” but one runs out of memory on a long conversation. What would you inspect before blaming the serving library? Start with the exact configuration, then account for the attention cache separately from the weights.
Before you begin: You understand causal attention, residual connections, and normalization.
The original LLaMA paper describes a decoder-only transformer with several established design choices combined into a training recipe. Later releases change parts of that recipe. This lesson uses the original dense architecture as a reference, not as a claim that every later Llama-branded model has identical internals.
Walk through one block
Token IDs first select embedding vectors. A block normalizes its input, applies causal self-attention, and adds that result to the residual stream. A second normalized path passes through a feed-forward network and is added back again. Repeating blocks produces a representation that a final normalization and output projection turn into vocabulary scores.
RMSNorm scales features by their root mean square without subtracting their mean. Rotary position embeddings rotate query and key components to make attention sensitive to relative positions. The feed-forward network uses a gated activation, commonly described as SwiGLU.
For the gate, one learned projection goes through the SiLU activation and is multiplied element by element with another projection. A final projection returns to the model width. “Gate” means a learned modulation of numbers here; it is not an explicit if-statement deciding whether a fact is true.
Read the fields that affect deployment
| Configuration field | Question it answers |
|---|---|
| Hidden size and layers | How wide and deep is the network? |
| Query and key/value heads | How are attention and cache storage organized? |
| Vocabulary and tokenizer | Which learned vector belongs to each token ID? |
| Position settings | How is position represented and scaled? |
| Weight dtype | How many bytes does each stored value use? |
For ordinary equal-width heads, the head width is the hidden size divided by the number of query heads. Grouped-query attention can use fewer key/value heads than query heads. For example, the Llama 2 paper uses GQA for its 70B model; it does not justify saying every Llama 2 size uses the same head layout.
Estimate the cache separately from weights
Each generated sequence usually stores a key and a value for every cached token, layer, and key/value head. This simplified calculation excludes allocator overhead and assumes an unquantized cache of fixed-width values.
# Runnable: Python 3, standard library.
# Hypothetical configuration, not a named model.
layers, kv_heads, head_width = 24, 8, 64
tokens, batch, bytes_per_value = 4096, 1, 2
cache_bytes = (2 * layers * kv_heads * head_width
* tokens * batch * bytes_per_value)
print(round(cache_bytes / 1024**2), 'MiB')
assert cache_bytes == 201326592
A model whose weights fit can still run out of memory when context or concurrent requests grow. Weight quantization does not automatically quantize this cache.
Double the conversation, then double the traffic
The hypothetical cache calculation gives 192 MiB for one 4,096-token sequence. Predict the result for 8,192 tokens at batch one, then for four equally long sequences. Under the same assumptions, these become 384 MiB and 1,536 MiB.
Now suppose you halve weight storage through quantization while leaving cache precision unchanged. Does the cache estimate halve too?
Keep the two memory accounts separate
No. The cache stores runtime keys and values, not the checkpoint's weight tensors. Its storage changes only if the cache representation or workload changes. The total device budget also needs activations, temporary buffers, and runtime overhead, so the simplified cache estimate is one component rather than a fit guarantee.
Use the exact checkpoint's layer count, key/value heads, head width, and cache dtype to replace the fictional values. Record the revision with the estimate so a later configuration change cannot silently invalidate it.
Check an architectural assumption
Exercise: two checkpoints have equal hidden size, layer count, and weight precision. One has four times as many key/value heads. What changes in the estimate above, and what can you not conclude?
Compare your reasoning
At the same batch and context length, the simplified cache is four times larger. You cannot infer that the model is four times slower or four times better. Kernels, hardware, routing, and task quality require measurement.
When adapting a checkpoint, also preserve its tokenizer revision, chat template, special tokens, and license conditions. A successful weight load is only the first compatibility check.
Next, examine an architecture that increases total parameter capacity while activating only part of its feed-forward network for each token.
Sources
Read the architecture sections of LLaMA and Llama 2. Their differences are a reason to inspect exact releases instead of copying a universal “Llama configuration.”
Continue: Mistral and Mixture of Experts (MoE).