A short history of NLP

You need to distinguish “The film was good” from “The film was not good.” A word-count model sees nearly the same vocabulary. Instead of treating NLP history as a list of inventions, use this failure to ask what each representation makes easier or harder.

Before you begin: Understand features, embeddings, and held-out evaluation from the beginner track.

Count what is present

A bag-of-words representation counts tokens while discarding their order. It can work well for tasks with strong lexical signals, such as many document categories. Its weakness is not that counting is primitive; it is that the representation deliberately omits some information.

For the two film reviews, the presence of “not” may help a trained classifier. But a bag of isolated words has difficulty distinguishing “not good” from a sentence where “not” modifies something else. Adding n-grams, short sequences of adjacent tokens, lets a feature represent “not good” directly.

This increases coverage of local order at the cost of more sparse features. A phrase absent from training may still be difficult. The tradeoff is about representation and data, not a simple progression from useless to intelligent.

Share information between related words

Learned embeddings give words dense vectors that can capture useful relationships from training contexts. A model may transfer some information between “excellent” and “good” rather than learning every word independently.

A single static vector still conflates senses. “Bank” in a loan question and “bank” in a fishing report start with the same word representation in a static embedding table. Contextual models build representations that depend on the surrounding sequence, helping later decisions distinguish these uses.

Which information did the representation throw away?

Make the sequence matter

RNNs and LSTMs process sequences through recurrent states. They offered a learned way to incorporate order and context. Attention later gave models more direct access to relevant positions, while transformers made attention central and enabled highly parallel training over positions.

These architectures did not erase the need for good labels or representative data. If a review dataset leaks the rating in a filename, many architectures can exploit that shortcut. A stronger representation does not repair an invalid experiment.

Reuse learning across tasks

Transfer learning starts from a model trained on a broad task and adapts it to a narrower one. This reduces the need to learn every representation from a small labeled dataset. ULMFiT, contextual representation methods, BERT, and generative pretrained models explored different ways to reuse language learning.

The choice includes what to reuse: frozen features, a model whose weights are fine-tuned, or a model prompted with examples. These mechanisms differ in data requirements, computation, and how updates are deployed. Keep them separate when reading papers that all use the word “pretrained.”

Compare on one controlled problem

Build a small review set with negation, mixed opinions, and unfamiliar synonyms. Compare a keyword baseline, a count-based classifier if you can train one, and a pretrained model. Keep the test split and label definition fixed. Report which errors change, not only the final score.

For a paper-only exercise, inspect the representation each method receives. The sentence “Not only was the film good, it was beautifully paced” is a useful challenge to the rule “not means negative.” A model needs to interpret the construction rather than treat the token as a universal negative flag.

Choose a representation for the job

Your task is to route documents containing exact product codes. A large model handles prose well but occasionally changes a code. Would replacing exact matching with embeddings necessarily improve the system?

The next chapter reads the original transformer as a solution to sequence modeling, with attention to what its architecture actually contained.

Practice with feedback

from collections import Counter

a = 'the tutor helped the machine did not'.split()
b = 'the machine helped the tutor did not'.split()

vocab = sorted(set(a) | set(b))
vec = lambda toks: [Counter(toks)[w] for w in vocab]
print(vec(a) == vec(b))          # True: indistinguishable

bigrams = lambda t: Counter(zip(t, t[1:]))
print(bigrams(a) == bigrams(b))  # False: ('tutor','helped') vs ('machine','helped')

# Bigrams separate this pair because the distinguishing relationship spans
# two adjacent tokens. They still fail when the dependency is long, e.g.
# 'the tutor, who arrived late after the bus broke down, helped' - the pair
# (tutor, helped) is now 9 tokens apart and no fixed n-gram window sees it.
# That gap is what attention closes.

Sources

Word2vec, ULMFiT, and BERT provide primary examples of representation and transfer-learning approaches. Their historical results are not presented as current product rankings.