A Transformer is often introduced through a large architecture diagram. That diagram is accurate, but it can hide the simple idea underneath: every token repeatedly gathers useful information from other tokens, transforms what it has learned, and passes an updated representation forward.

The architecture is easier to understand when we follow the information rather than memorise the boxes.

Start with token representations

Text is first split into tokens. Each token becomes a vector: a list of numbers that the model can update. Because attention alone does not know order, positional information is added so that the model can distinguish the first token from the fiftieth.

The resulting vectors enter a stack of Transformer layers. Each layer refines them, but keeps the sequence length stable. A token that began as a generic representation gradually becomes contextual: its meaning reflects the sentence around it.

Attention is learned routing

For every token, the model creates three projections called a query, a key, and a value. A query describes what information a token is looking for. A key describes what information another token offers. A value contains the information that can be passed along.

scores = query * transpose(keys)
weights = softmax(scores / sqrt(d))
output = weights * values

The score between one query and every key determines where information should come from. Softmax turns those scores into positive weights that sum to one. Dividing by the square root of the vector dimension keeps values numerically controlled as the model gets wider.

The routing is computed from the current sequence. The same token can gather different information in different contexts.

Why multiple heads?

A single attention calculation produces one routing pattern. Multi-head attention runs several smaller attention operations in parallel. This gives the model room to represent different relationships at the same time:

  • Nearby syntactic structure.
  • Long-range references.
  • Delimiters and formatting.
  • Agreement between subjects and verbs.
  • Task-specific patterns that resist simple labels.

Some heads show interpretable behaviour, but the model is trained as a whole. The important point is capacity: several routing patterns can operate at once.

Communicate, then compute

After attention moves information between positions, a feed-forward network transforms each position independently. The common structure expands the vector into a wider hidden space, applies a non-linearity, and projects it back to the model width.

hidden = activation(input * W1)
output = hidden * W2

Attention communicates across tokens. The feed-forward network computes over the features available at one token. That alternating pattern is a durable systems-level model of a Transformer layer.

Residual streams preserve information

If every layer completely replaced its input, deep models would be difficult to train and useful information could disappear. Residual connections add each operation's output back to the existing representation:

state = state + attention(normalize(state))
state = state + feed_forward(normalize(state))

The residual stream acts like shared working memory. Each layer reads from it and writes an update into it. Layer normalisation keeps activation scales controlled so deep optimisation behaves predictably.

Causal masking creates a language model

During next-token prediction, a token must not inspect future tokens. Otherwise the answer would leak into the input.

A causal mask removes attention connections to future positions. Position five can attend to positions one through five, but not position six. The model then predicts a probability distribution for the next token, compares it with the actual token, and adjusts its parameters to reduce error.

Repeated across large datasets, this objective teaches statistical structure, factual associations, syntax, style, and reusable computational patterns.

The quadratic cost

Standard attention compares every token with every other token. If sequence length doubles, the attention score matrix becomes roughly four times larger. Long context is therefore a memory, latency, and systems-design problem, not only a model feature.

Modern systems manage this cost through techniques including efficient attention kernels, sparse or local attention, grouped-query attention, retrieval, and architectures that compress or reuse state.

Architecture is only part of the model

The Transformer defines how information can move and transform. It does not specify what a model knows. Behaviour emerges from the combination of training data, objectives, optimisation, scale, post-training, and inference-time context and tools.

Two models can share nearly identical architecture and behave very differently because the rest of the training system differs.

A durable mental model

When reading about a new Transformer variant, ask:

  1. How does information move between tokens?
  2. How is information transformed at each position?
  3. What is stored in the residual stream?
  4. How does the design control memory and compute?
  5. What training signal shapes the final behaviour?

These questions transfer across language, vision, audio, multimodal, and code models. They are more useful than memorising one architecture diagram.

Public references