“The workshop moved to the library because the original room flooded.” When you reach “flooded,” how do you connect it to the room mentioned earlier? A language model needs some way to combine information across the sequence.
Before you begin: Understand token representations and the idea of a learned layer.
How an RNN reads a sequence
A recurrent neural network, or RNN, processes a sequence step by step. At each step it combines the current input with a hidden state, a numerical summary passed from the previous step. The updated state becomes the starting point for the next step.
Imagine reading a message while repeatedly updating a small note. The note can preserve useful information, but the model has to learn what to keep. Information from an early word must influence later states through a chain of updates. Training that influence across long sequences can be difficult.
This analogy has a limit: an RNN state is a vector, not a literal written note. Also, RNNs are not incapable of long-range relationships. Architectures such as LSTMs introduce gates that help control how information changes. The challenge is learning and computing those relationships efficiently.
How attention uses earlier words
Attention lets a model form a weighted combination of representations from other positions. The weights depend on what information the current position is seeking and what the other positions offer. A translation model can, for example, focus on different source words while producing different target words.
Suppose three source positions receive weights 0.1, 0.7, and 0.2. Their value vectors are combined using those proportions. The mechanism does not choose one word and discard everything else. Several positions can contribute, and multiple attention heads can compute different mixtures.
Those weights are useful for understanding the calculation. They are not automatically a complete explanation of why a model produced a particular answer. Other layers, heads, and representations also affect the result.
The path between two words
The fact near the beginning of a sentence has to affect a prediction near the end. A recurrent model passes information through successive states. Attention adds a different route: the current position can compare directly with earlier representations. This shortens the path between distant positions, while introducing its own work and memory costs.
Try the attention controls as a small picture of that direct route. A score changes how much a source contributes; a value changes what it contributes. Neither is a human sentence stored inside a box. The broader lesson is about access to information, not a claim that recurrence is always useless or that attention reads every source equally well.
A useful fact appears at position 4; the prediction happens at position 300.
Allow the later position to score earlier representations directly.
There is a direct attention connection, but computing many such connections still costs work.
What changed with transformers?
The 2017 transformer used attention and position-wise feed-forward layers without the recurrent sequence processing in the earlier translation architecture. Self-attention combines positions within a sequence. Cross-attention in the original decoder combines information from the source sequence with the target-side representation.
During training, the model can calculate many positions in parallel because it does not need to finish an RNN state update for each previous position first. Position information must be supplied separately so that the model can distinguish word order.
Parallel training does not mean an ordinary autoregressive chatbot writes its whole answer at once. In that common generation setup, it still produces tokens one after another because each new token becomes context for the next. Separating training from generation prevents a frequent misunderstanding of the transformer's advantage.
The cost of attention
Standard full attention compares many pairs of positions. If a sequence grows from 100 tokens to 200, the number of position pairs grows from 10,000 to 40,000. This illustrates quadratic growth; it is not a direct prediction of total runtime because other operations and implementation choices matter.
Later architectures and kernels change how attention is computed or restricted. Some use local windows, shared key/value heads, or efficient memory scheduling. The basic question remains: which information needs to interact, and at what cost?
Compare an RNN with a transformer
In “The code on the first page expires tomorrow,” an RNN must carry the useful information about “code” through intermediate state updates. An attention layer can give the current position a direct computational connection to a representation at the earlier position, subject to its attention mask.
That direct connection is an opportunity, not a promise. A trained model can still focus on unhelpful information, misunderstand the sentence, or use a shortcut from its training data.
Try tracing a sentence
A developer says, “Transformers see all words at once, so a text generator can always look at the answer before predicting it.” What is wrong with that claim?
Inspect the attention mask
A causal language model uses a mask that prevents a position from reading future tokens during training. Many training positions are processed together, but each has restricted information. During generation, future tokens do not exist yet. Some other architectures, such as bidirectional encoders, do use both sides of an input; their training tasks differ.
You now have the main architectural idea. The next lesson places models, assistants, and applications on the same map so that product names do not obscure what a system actually does.
Practice with feedback
Move the attention, then change the information
Two source positions have scores [first score, 0] and scalar values [10, second value]. The score slider is already scaled. Softmax chooses weights; those weights mix values.
Mixed value: 13.303
Read the two weight bars together: they always sum to one. Changing a value changes the output without changing either bar. Changing a score reallocates weight between both positions.
What this experiment assumes. One unmasked attention row with hand-chosen numbers. It is not an explanation of a trained model’s final answer. Notes and recorded results here last until you leave this page.
Explain what attention bought and what it cost
A sentence is 300 words long. The key fact appears at word 4, and the answer must be produced at word 300.
Contrast sequential state with direct look-back, including the quadratic price.
Check your understanding
Your task
Write the comparison you would give a teammate who asks why transformers replaced recurrent models.
These notes stay on this page. Download them before leaving.
What to include
- You describe path length in steps, not in vague words like "better"
- You say why recurrence blocks parallel training over positions
- You give the growth rate for each: linear versus quadratic in sequence length
- Your counter-example names a real constraint such as streaming input or a tiny device
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Path from word 4 to word 300 Recurrent: 296 sequential steps, each one rewriting a fixed-size state, so the signal has 296 chances to be squeezed out. Attention: one step. Position 300 computes a score against position 4 directly and weights it.
What can be computed in parallel during training Recurrent: nothing across positions. Step t needs the state from step t-1. Attention: all positions at once. The whole score matrix is one matrix multiply, which is what made large-scale training practical.
What gets more expensive as the sequence grows Recurrent: time grows linearly with length; memory stays roughly flat. Attention: the score matrix grows with the square of the length. Doubling from 2k to 4k tokens roughly quadruples attention cost.
One case where I would still not reach for attention: A sensor stream on a small device that must emit an output after every reading with a fixed memory budget. A recurrent state is constant-size and never revisits history; full attention would need the history kept around and rescored at every step.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Further reading
Attention Is All You Need describes the original transformer. Neural Machine Translation by Jointly Learning to Align and Translate shows an earlier attention-based translation approach.