A user sends “Explain embeddings.” Before any answer appears, a decoder processes the prompt. After the first output token appears, the system repeatedly extends the sequence. Why do these two phases have different performance characteristics?
Before you begin: Understand attention and next-token prediction.
Read the causal information pattern
In a standard causal decoder, each position can use itself and earlier positions under the attention mask. It cannot inspect later target tokens. Layers combine attention with learned transformations, residual paths, normalization, and positional information according to the architecture.
The last relevant representation is projected into vocabulary scores, often called logits. A decoding procedure turns those scores into a token choice. The chosen token is appended to the context, and generation continues until a stop condition, output limit, or failure.
The output is not selected from a fixed library of complete answers. It is constructed through a sequence of token decisions. That flexibility also allows unsupported or inconsistent text.
Distinguish prefill and decode
Prefill processes the supplied prompt and establishes intermediate state used for generation. Decode produces later tokens incrementally. A long prompt can make prefill expensive; a long answer extends decode time.
Many serving systems cache the attention keys and values for previous positions, called the KV cache. Reusing them avoids recomputing those projections for the entire history at every step. New queries still interact with relevant cached information, and the cache occupies memory.
Caching does not train the model, create durable user memory, or guarantee that a fact will be used correctly. It is an inference optimization for a particular computation. Application-level conversation storage is a separate mechanism.
Work through a tiny generation loop
Suppose a fictional vocabulary contains three tokens. The model emits logits [2, 1, 0]. Softmax gives the first token the largest probability. Greedy decoding picks it; sampling can pick another token according to the distribution and any supported decoding controls.
After choosing a token, the next distribution is conditioned on that choice. A different early token can change the rest of the answer. Lower temperature, when supported, changes sampling but does not supply missing evidence or make outputs perfectly reproducible across every runtime.
Chat formatting is part of the input
Conversation messages are encoded using the model's expected template, including role markers and special tokens. The literal text “user:” in a plain string is not necessarily equivalent to the tokenizer's actual user-message format.
Use the checkpoint's documented chat template and generation-start convention. Incorrect formatting can make a capable model behave as a raw continuation engine or emit unintended role text. Keep the tokenizer and model revision compatible.
For some closed models, architecture or formatting details are hidden behind the API. Use the documented message schema rather than inventing internal specifics from the product name.
Decide what belongs outside generation
A decoder can generate JSON, code, or a proposed tool call. The application must still parse, validate, authorize, and execute those outputs. The model's statement “saved” does not confirm a database write. Its generated function arguments do not prove that the current user can perform the action.
Treat the decoder as one component in a system with explicit inputs and outcomes. This makes errors easier to diagnose than attributing everything to “the AI.”
Explore the cache boundary
You reuse a KV cache created from one prompt after changing an earlier instruction. Can you assume the cached state is still valid?
Respect the computation's dependency
No. Cached state depends on the exact relevant prefix, model, positional setup, and runtime conventions. Changing earlier input can invalidate later cached representations. Serving systems need correct cache keys and isolation; a cache is not an interchangeable summary of similar text.
Next, encoder-only models will show another way to turn complete inputs into labels, embeddings, or relevance scores.
Sources
GPT-3 is a causal language-model reference. The Hugging Face chat-template guide documents model-specific conversation formatting, and its cache guide explains inference caching.