What changes when every task becomes text in and text out?

A classifier usually returns a label ID, a translator returns a sentence, and an extractor may return positions in a document. T5 asked whether these tasks could share an interface: provide text describing the task and generate text as the answer.

Before you begin: Understand encoder-decoder models and supervised versus self-supervised objectives.

Unify the interface, not the meaning of success

In a text-to-text setup, sentiment classification can generate “positive,” translation can generate a target-language sentence, and summarization can generate a shorter passage. The output type is text, but each task still needs its own evaluation.

For classification, “mostly delighted” may be understandable to a person but invalid if the allowed labels are exactly “positive” and “negative.” A unified generative interface does not remove the need for output constraints and validation.

T5 is an encoder-decoder transformer. Its text-to-text framing should not be confused with the decoder-only architecture of many later chat systems. Similar interfaces can sit on different models.

Follow a span-corruption example

T5 pretraining uses a denoising objective that replaces selected spans with special sentinel tokens. Consider this simplified example, with token boundaries chosen for teaching:

Original: The small boat crossed the quiet lake.
Input:    The <extra_id_0> crossed the <extra_id_1>.
Target:   <extra_id_0> small boat <extra_id_1> quiet lake <extra_id_2>

The input contains gaps, and the target contains the missing spans associated with their sentinels. This is different from simply predicting every next token of the original text or masking only isolated words. Exact corruption rates and tokenization belong to the training configuration.

The encoder reads the corrupted input. The decoder generates the target span sequence with causal access to earlier target tokens and cross-attention to the input. Both architecture and objective matter to what the checkpoint has learned.

Make the downstream task visible

A task prefix such as “summarize:” or a translation instruction can signal the desired transformation. But a base checkpoint is not guaranteed to follow every conversational instruction you invent. Its response depends on training and fine-tuning.

Instruction-tuned relatives, such as models trained using FLAN-style mixtures, add different supervision and capabilities. Do not treat “T5,” “instruction-tuned T5,” and a current assistant product as interchangeable names.

For an application, inspect the checkpoint's model card, tokenizer, expected task format, and loading interface. A correctly installed library cannot compensate for using a checkpoint unsuited to the task.

Work through a label task

Suppose the input is “Classify the message as question or statement: When does the class start?” A generated “question” fits the specified label set. “It is asking about the class time” conveys the same idea but fails an exact-label contract.

You can normalize allowed variants carefully, constrain generation where supported, or use a task-specific classifier. The best choice depends on the failure cost and measured behavior. Silently mapping any response containing “question” to that label can misread “not a question.”

Read the paper as a controlled study

T5 compared architectures, objectives, datasets, and transfer methods under a common framework. Its contribution is broader than a prompt prefix. When reading its results, ask which comparison holds the other factors fixed and which combines multiple improvements.

The historical benchmark results establish what was measured in that study. They do not imply that a 2019 checkpoint is the current best choice for every text task. The reusable lesson is to separate interface convenience from model suitability and experimental evidence.

Reconstruct a missing span

For “The class is in Room 4 on Friday,” replace “Room 4” with one sentinel. Write the corrupted input and target. Then explain why the answer text does not need to repeat the entire original sentence.

Next, we will follow the generative pretrained model family and distinguish scaling, instruction tuning, and product changes.

Sources

The T5 paper describes the unified framework and span-corruption objective. Scaling Instruction-Finetuned Language Models studies FLAN-style instruction tuning.

Continue to the next lesson.