The paper title says “Attention Is All You Need.” Does that mean a transformer contains only attention? Open the architecture carefully and you find several other operations, each solving a different problem.
Before you begin: Understand embeddings and the beginner comparison of recurrence and attention.
Follow the source through the encoder
The original 2017 model was designed for sequence-to-sequence tasks such as translation. Source tokens become embeddings, combined with positional information. Encoder layers then apply self-attention and a position-wise feed-forward network, with residual connections and normalization around sublayers.
Self-attention mixes information across positions. The feed-forward network transforms each position's representation using shared learned parameters. A residual connection adds a sublayer's input back to its output, helping information and gradients travel through the stack. Normalization controls the scale and distribution of intermediate values according to its formula.
These descriptions identify roles, not guarantees that one component alone causes a particular capability. Later architectures change details such as normalization placement and activation functions. Do not assume every current model matches the original block exactly.
Follow the target through the decoder
The decoder reads a shifted target sequence during training. Its self-attention uses a causal mask so a position cannot inspect future target tokens. Cross-attention then lets target-side queries use keys and values derived from the encoder's source representations.
For translating a sentence, the source can be fully available while the target is generated left to right. This is why the decoder has two different attention relationships: target-to-earlier-target and target-to-source. They use similar arithmetic but different inputs and masks.
The final representation is projected to vocabulary scores. A probability distribution over next tokens is used for the training objective or generation procedure.
What question does each attention block answer?
The original encoder-decoder transformer contains several attention operations with different jobs. Encoder self-attention relates positions in the input. Decoder self-attention relates allowed output-prefix positions. Cross-attention lets the decoder read the encoded input. Keeping these jobs separate makes the architecture easier to trace than memorizing a large diagram.
The small experiment below isolates the shared operation: score sources, turn scores into weights, then combine values. When returning to the full architecture, ask where each operation gets its queries, keys and values and which positions it may read. The same arithmetic can participate in different information flows.
The decoder is predicting the next translated token from a partial output.
Distinguish its masked prefix attention from its attention over the encoded source.
The model can read the source sentence while remaining unable to read future target tokens.
Read the dimensions before the equation
Let n be the number of positions and d_model the representation width. The input to a layer is an n × d_model array. Attention projects it into queries, keys, and values. For one self-attention head, comparing n queries with n keys produces an n × n score array before masking and normalization.
If n doubles, that pairwise array has four times as many entries. This explains one scaling concern, but it does not mean every implementation stores the full array or that all runtime grows by exactly four. Efficient kernels and the rest of the model matter.
Multiple heads produce several learned mixtures, which are combined and projected back into the model's representation width. A head is not assigned a linguistic role by the architecture; its behavior emerges from training and can vary across inputs.
Separate parallel training from generation
During training, many target positions can be evaluated together using the known shifted target and causal mask. This is often called teacher forcing in sequence generation. The model receives the correct previous target tokens, not its own sampled mistakes, for that training pass.
At ordinary autoregressive inference time, the next token depends on the tokens actually generated so far. The system still grows the answer sequentially. Caching past keys and values can avoid recomputing some work, but it does not make unknown future tokens available.
Inspect what the paper establishes
The paper compared a particular architecture and training setup on translation tasks. Its results support that setting and motivated much later work. The title should not be read as proof that recurrence is never useful, that feed-forward layers are optional, or that attention weights are a full explanation of model reasoning.
When reading a later model, ask which design choices changed: mask, positional representation, normalization, feed-forward structure, head sharing, or training objective. This creates a useful comparison instead of treating every transformer as identical.
Trace an information path
During translation training, may a decoder position attend to the end of the source sentence? May it attend to the next target token? Explain the difference.
Separate source access from target leakage
It can normally attend to the available source through cross-attention, subject to padding or task-specific masks. It cannot attend to a future target token through causal self-attention. Source information is input to the translation task; future target text would reveal the prediction target.
Practice with feedback
Move the attention, then change the information
Two source positions have scores [first score, 0] and scalar values [10, second value]. The score slider is already scaled. Softmax chooses weights; those weights mix values.
Mixed value: 13.303
Read the two weight bars together: they always sum to one. Changing a value changes the output without changing either bar. Changing a score reallocates weight between both positions.
What this experiment assumes. One unmasked attention row with hand-chosen numbers. It is not an explanation of a trained model’s final answer. Notes and recorded results here last until you leave this page.
Read the transformer as a set of decisions
An encoder-decoder transformer translates a 7-token sentence into a 9-token sentence.
Trace source and target paths and separate parallel training from sequential generation.
Make the decision before reading the feedback
Check your understanding
Now make something you can check
Trace how information reaches the third generated token, naming every path.
These notes stay on this page. Download them before leaving.
Check your reasoning against these points
- All three paths are named: embeddings plus position, masked self-attention over targets 1-2, cross-attention over the 7 source tokens
- The mask is described as an additive negative-infinity term before softmax
- Shapes are consistent with 8 heads of 64 dimensions
- The unsupported claim is genuinely outside the paper's evidence
Compare with a worked answer
Compare the decisions and the evidence. Your wording can be different.
Paths into the prediction of target token 3: 1. Its own input embedding plus positional encoding (the token generated at step 2, fed back in). 2. Masked decoder self-attention over target positions 1 and 2 only. 3. Cross-attention over all 7 encoder outputs, unmasked.
What it must not see: target tokens 4 and beyond. The mechanism is an additive mask of -inf on those score entries before the softmax, which sends their weights to exactly zero.
Shapes (d_model=512, 8 heads, head dim 64, batch 1): encoder output: (1, 7, 512) decoder self-attention scores: (1, 8, 3, 3), upper triangle masked cross-attention scores: (1, 8, 3, 7), unmasked
One claim the paper supports: on the WMT tasks reported, this architecture reached competitive translation quality with substantially less training time than the recurrent and convolutional baselines it compared against. One it does not: that attention weights are an explanation of the model's reasoning. The paper shows some heads attending to syntactically related words; that is an observation about weights, not evidence that the weights explain the output.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, we will calculate one attention operation with small numbers so the query, key, value, and mask are no longer abstract labels.
Source
Attention Is All You Need is the original architecture reference. The dimension walkthrough here is an explanatory example rather than a reproduction of the paper's training results.