Decoder-Only Models (GPT, Claude, Llama)
Most modern chat LLMs are decoder-only transformers.
They generate one token at a time:
text
prompt tokens -> predict next token -> append token -> repeat
Causal masking
Decoder-only models use causal masking. A token can attend to earlier tokens, but not future tokens.
That matches generation: the model writes from left to right.
Why this architecture won
Decoder-only models are:
- simple to scale
- natural for text generation
- good at in-context learning
- compatible with chat templates
- easy to adapt to tool calls and structured outputs
- strong when trained on mixed text/code/instruction data
Chat is still text completion
A chat conversation becomes a formatted token sequence:
text
system: ...
user: ...
assistant: ...
The model still predicts the next token. The chat behavior comes from training and formatting.
Limitations
Decoder-only models can:
- hallucinate facts
- lose track in long context
- over-generate
- need retrieval for private/current data
- require output validation for production
Knowledge check
Q1: What does causal masking prevent?
Looking at future tokens during generation.
Q2: Why are decoder-only models good for chat?
They naturally generate continuations and can be trained on formatted conversations.