“The workshop moved to the library because the original room flooded.” When you reach “flooded,” how do you connect it to the room mentioned earlier? A language model needs some way to combine information across the sequence.

Before you begin: Understand token representations and the idea of a learned layer.

Carry information forward

A recurrent neural network, or RNN, processes a sequence step by step. At each step it combines the current input with a hidden state, a numerical summary passed from the previous step. The updated state becomes the starting point for the next step.

Imagine reading a message while repeatedly updating a small note. The note can preserve useful information, but the model has to learn what to keep. Information from an early word must influence later states through a chain of updates. Training that influence across long sequences can be difficult.

This analogy has a limit: an RNN state is a vector, not a literal written note. Also, RNNs are not incapable of long-range relationships. Architectures such as LSTMs introduce gates that help control how information changes. The challenge is learning and computing those relationships efficiently.

Look back instead of relying on one summary

Attention lets a model form a weighted combination of representations from other positions. The weights depend on what information the current position is seeking and what the other positions offer. A translation model can, for example, focus on different source words while producing different target words.

Suppose three source positions receive weights 0.1, 0.7, and 0.2. Their value vectors are combined using those proportions. The mechanism does not choose one word and discard everything else. Several positions can contribute, and multiple attention heads can compute different mixtures.

Those weights are useful for understanding the calculation. They are not automatically a complete explanation of why a model produced a particular answer. Other layers, heads, and representations also affect the result.

What was different about the transformer?

The 2017 transformer used attention and position-wise feed-forward layers without the recurrent sequence processing in the earlier translation architecture. Self-attention combines positions within a sequence. Cross-attention in the original decoder combines information from the source sequence with the target-side representation.

During training, the model can calculate many positions in parallel because it does not need to finish an RNN state update for each previous position first. Position information must be supplied separately so that the model can distinguish word order.

Parallel training does not mean an ordinary autoregressive chatbot writes its whole answer at once. In that common generation setup, it still produces tokens one after another because each new token becomes context for the next. Separating training from generation prevents a frequent misunderstanding of the transformer's advantage.

There is still a cost

Standard full attention compares many pairs of positions. If a sequence grows from 100 tokens to 200, the number of position pairs grows from 10,000 to 40,000. This illustrates quadratic growth; it is not a direct prediction of total runtime because other operations and implementation choices matter.

Later architectures and kernels change how attention is computed or restricted. Some use local windows, shared key/value heads, or efficient memory scheduling. The basic question remains: which information needs to interact, and at what cost?

Follow one example through two designs

In “The code on the first page expires tomorrow,” an RNN must carry the useful information about “code” through intermediate state updates. An attention layer can give the current position a direct computational connection to a representation at the earlier position, subject to its attention mask.

That direct connection is an opportunity, not a promise. A trained model can still focus on unhelpful information, misunderstand the sentence, or use a shortcut from its training data.

Test your mental model

A developer says, “Transformers see all words at once, so a text generator can always look at the answer before predicting it.” What is wrong with that claim?

Inspect the attention mask

A causal language model uses a mask that prevents a position from reading future tokens during training. Many training positions are processed together, but each has restricted information. During generation, future tokens do not exist yet. Some other architectures, such as bidirectional encoders, do use both sides of an input; their training tasks differ.

You now have the main architectural idea. The next lesson places models, assistants, and applications on the same map so that product names do not obscure what a system actually does.

Why did language models adopt attention?