How transformers represent position

“The dog chased the cat” and “The cat chased the dog” contain the same main words but describe different events. If an attention layer only sees a collection of token vectors, where does the ordering information enter?

Before you begin: Understand attention as a weighted mixture over token representations.

Start with the symmetry

Without a positional signal or other order-dependent structure, self-attention over a set of vectors is permutation equivariant: rearranging the input positions rearranges the outputs in the corresponding way. It does not by itself assign a unique meaning to “first” or “three positions earlier.”

Causal masking introduces an order-dependent visibility pattern in a decoder, but practical models still use positional mechanisms to represent sequence relationships. Do not confuse a mask that restricts access with a complete encoding of distance.

Add an absolute position vector

One approach adds a learned or fixed vector to each token embedding based on its position index. The same token at position two and position ten then enters the network with different combined values.

The original transformer used sinusoidal position vectors. For an even representation width d, coordinate pair 2i and 2i+1 uses sine and cosine at a scale determined by 10000^(2i/d). Here i counts coordinate pairs and pos is the position index. The formulas are sin(pos / scale) and cos(pos / scale).

Different pairs vary at different rates. The model receives a pattern across frequencies rather than a single scalar position number. The construction also gives structured relationships between shifted positions, but successful use of longer sequences still depends on training and the rest of the architecture.

What makes the same words in a different order different inputs?

Compute a small position vector

This complete Python 3 program calculates the fixed encoding for a width of four. It is an illustration of the formula, not a trained transformer.

# Runnable: Python 3, standard library.
import math

def position_vector(pos, width=4):
    if width <= 0 or width % 2:
        raise ValueError('Use a positive even width')
    result = []
    for i in range(width // 2):
        scale = 10000 ** (2 * i / width)
        result.extend([math.sin(pos/scale), math.cos(pos/scale)])
    return result

assert position_vector(0) == [0.0, 1.0, 0.0, 1.0]
for pos in [0, 1, 2]:
    print(pos, [round(x, 4) for x in position_vector(pos)])

Predict which coordinate pair changes more slowly before running it. The second pair has a larger scale, so moving one position changes its angle less.

Represent relationships more directly

Relative-position methods incorporate the distance or relationship between positions. Some add a relative bias to attention scores. Rotary position embeddings, studied later in the advanced track, rotate query and key coordinate pairs so their dot products depend on relative offsets.

These methods are not interchangeable configuration labels. They change the computation, checkpoint compatibility, and sometimes how a context-length extension is performed. Use the mechanism associated with the actual model rather than attaching a different one after loading weights without a tested adaptation.

Distinguish a formula from generalization

A sinusoidal formula can be evaluated at position 100,000. That does not prove a model trained on short sequences will use that position reliably. Learned position tables may have an explicit size limit; other methods have different failure modes beyond trained lengths.

To evaluate an extension, test retrieval, order-sensitive tasks, long-distance dependencies, and short-context behavior. A model that accepts a longer request can still lose accuracy or become inefficient. Capacity and useful reasoning over the input are separate measurements.

Change the order deliberately

Create two notices: “Register before paying” and “Pay before registering.” Ask what action comes first. If a system embeds both notices near one another because they share a topic, does that make the order unimportant?

Practice with feedback

import numpy as np

def attention(X):
    d = X.shape[1]
    s = X @ X.T / np.sqrt(d)
    s = s - s.max(1, keepdims=True)
    w = np.exp(s); w /= w.sum(1, keepdims=True)
    return w @ X

def sinusoid(pos, d=4):
    i = np.arange(d)
    angle = pos / np.power(10000, (2 * (i // 2)) / d)
    enc = np.where(i % 2 == 0, np.sin(angle), np.cos(angle))
    return enc

tutor   = np.array([1.,0,0,0]); helped = np.array([0,1.,0,0]); machine = np.array([0,0,1.,0])
A = np.stack([tutor, helped, machine])
B = np.stack([machine, helped, tutor])

sort = lambda M: M[np.lexsort(M.T[::-1])].round(6)
print(np.allclose(sort(attention(A)), sort(attention(B))))   # True: order-blind

P = np.stack([sinusoid(p) for p in range(3)])
print(np.allclose(sort(attention(A + P)), sort(attention(B + P))))  # False

# With position added, 'tutor' at index 0 and 'tutor' at index 2 are different
# input vectors, so the two sentences produce genuinely different outputs.
# Without it, no amount of training could separate 'tutor helped machine'
# from 'machine helped tutor'.

Next, T5 will show how a single input-output format can support different tasks while preserving a distinct encoder-decoder architecture and training objective.

Sources

Attention Is All You Need gives the sinusoidal construction. RoFormer introduces rotary position embeddings, which the advanced course develops separately.