September 4, 2026

How Transformers Actually Compute Attention β€” A Step-by-Step Breakdown

Every GPT call, every Claude response, every Gemini generation β€” it all runs through this exact math.

By Swaroop

5 min read

Every time you use an LLM, the model is running one core computation thousands of times per second: attention. Not as a metaphor β€” literally, on every forward pass, for every token, at every layer.

But most explanations stop at the intuition: "queries, keys, and values." That leaves the part that actually matters β€” the exact arithmetic β€” as a black box.

This article traces the full attention calculation from scratch: what the input matrix really is, where Q/K/V come from, a complete numerical worked example you can run yourself, and why the whole thing is called "self-attention." By the end you should be able to trace a single attention head by hand.

The Problem Attention Is Solving

Start with a sentence:

"The animal didn't cross the street because it was tired."

When the model processes the token "it", it needs to resolve the reference. Is it the animal? The street? The word "it" alone carries no answer β€” you need context from the surrounding sequence.

Attention solves this by letting every token assign a weight to every other token β€” a number expressing how much it should "look at" that position when computing its updated representation:

"it" β†’ "animal"  : 0.65
"it" β†’ "street"  : 0.20
"it" β†’ "tired"   : 0.10
"it" β†’ others    : 0.05

These aren't hand-coded. They emerge from a learned computation involving three vectors β€” Query, Key, and Value. The complete formula:

Attention(Q, K, V) = softmax( QKα΅€ / √dβ‚– ) Β· V

Everything below is an explanation of that one line.

Step 0: What Is X?

Before Q, K, or V can exist, we need the input matrix X.

For the sentence "The cat sat", each token maps to a learned embedding vector. With embedding dimension d_model = 4:

The β†’ [0.2, 0.5, 0.1, 0.7]
cat β†’ [0.8, 0.3, 0.6, 0.2]
sat β†’ [0.4, 0.9, 0.5, 0.1]

Stack them row by row: that's X, shape [n_tokens, d_model].

Key detail: X is not the same across layers. Each attention layer takes the previous layer's output as its X. So Xβ‚€ is raw token + position, while X₁₂ in GPT-3 (96 layers) encodes deep contextual meaning built up over 12 rounds of attention.

Step 1: Computing Q, K, V

Query, Key, and Value are not manually assigned. They are computed from X using three separate learned weight matrices:

Q = X Β· WQ
K = X Β· WK
V = X Β· WV

Where WQ, WK, WV are parameter matrices that stay fixed after training. The shapes:

X.shape  = [n, d_model]
WQ.shape = [d_model, d_k]   β†’ Q.shape = [n, d_k]
WK.shape = [d_model, d_k]   β†’ K.shape = [n, d_k]
WV.shape = [d_model, d_v]   β†’ V.shape = [n, d_v]

Symbol: WQ, WK, WV What it is: Learned projection matrices Changes with input?: No β€” frozen after training

Symbol: Q, K, V What it is: Computed activations Changes with input?: Yes β€” depend on current X

This is why "bank" in "river bank" and "bank" in "deposit at the bank" produce completely different attention patterns β€” their context drives a different X, which flows through the same weight matrices to produce different Q/K/V vectors.

The search engine mental model:

  • Q = what you're searching for
  • K = what each token claims to contain
  • V = what information actually gets retrieved when there's a match

Q and K determine where to look. V determines what flows out.

Step 2: Dot Product β€” QKα΅€

Multiply Q by the transpose of K. The result is an [n, n] score matrix where cell [i, j] is the raw attention score of token i attending to token j:

Keys
       cat   eats  fish
     β”Œβ”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”
Q catβ”‚  ?   β”‚  ?   β”‚  ?   β”‚
  eatβ”‚  ?   β”‚  ?   β”‚  ?   β”‚
 fishβ”‚  ?   β”‚  ?   β”‚  ?   β”‚
     β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”˜

A higher dot product = those tokens are more relevant to each other.

Step 3: Scaling β€” Divide by √dβ‚–

With large d_k, raw dot products explode. With d_k = 512, scores like [180, 210, 95] are common. Feed those into softmax and you get [0.00, 1.00, 0.00] β€” completely collapsed. Gradients vanish and training breaks.

Dividing by √d_k keeps scores in range. This is called scaled dot-product attention.

Step 4: Softmax β€” Turning Scores into Weights

Applied row by row. Each row becomes a probability distribution over all tokens.

Full walkthrough for scores [3.2, 4.0, 1.0, 6.2, 8.0]:

exp(3.2) =     24.53
exp(4.0) =     54.60
exp(1.0) =      2.72
exp(6.2) =    492.75
exp(8.0) =  2,980.96
         ──────────
Ξ£        =  3,555.56

Token 1:   24.53 / 3555.56 β†’  0.69%
Token 2:   54.60 / 3555.56 β†’  1.54%
Token 3:    2.72 / 3555.56 β†’  0.08%
Token 4:  492.75 / 3555.56 β†’ 13.86%
Token 5: 2980.96 / 3555.56 β†’ 83.83%  ← winner

Token 5 dominates with 83.83% despite being only 1.8 points above Token 4. This non-linear amplification is what makes the model commit decisively to the most relevant tokens.

Numerical stability tip: Always do np.exp(x - x.max()) before dividing. exp(800) overflows a 64-bit float. Subtracting the max keeps all exponents safe while giving mathematically identical probabilities.

Step 5: Weighted Sum β€” Γ— V

output("it") =
    0.65 Γ— V("animal")
  + 0.20 Γ— V("street")
  + 0.10 Γ— V("tired")
  + 0.05 Γ— V(rest)

The result is a new vector for "it" that contains mostly information from "animal", some from "street", and very little from the rest β€” the contextualized representation passed to the next layer.

Complete Worked Example in NumPy

Sentence: "cat eats fish", d_model = 2:

import numpy as np

X  = np.array([[1, 2], [3, 1], [2, 4]], dtype=float)
WQ = np.array([[1, 0], [0, 1]], dtype=float)  # identity
WK = np.array([[1, 0], [0, 1]], dtype=float)  # identity
WV = np.array([[0, 1], [1, 0]], dtype=float)  # swaps dimensions

Q, K, V = X @ WQ, X @ WK, X @ WV

# Raw scores for "fish" (Q=[2,4]) against each key:
# fish→cat:  [2,4]·[1,2] = 10
# fish→eats: [2,4]·[3,1] = 10
# fishβ†’fish: [2,4]Β·[2,4] = 20  ← self-attention dominates

scores = Q @ K.T / np.sqrt(2)
e = np.exp(scores - scores.max(-1, keepdims=True))
weights = e / e.sum(-1, keepdims=True)  # softmax row-wise

output = weights @ V
# output[fish] β‰ˆ [3.997, 2.001] β‰ˆ V(fish)
# fish self-attends at ~99.8% in this toy example

In a real trained model, WQ and WK break this symmetry to produce meaningful cross-token attention.

Why "Self-Attention"?

Because Q, K, and V all originate from the same source matrix X. Every token produces a query, a key, and a value from its own representation β€” then all tokens attend to each other within the same sequence.

This distinguishes self-attention (GPT, LLaMA) from cross-attention (T5, encoder-decoder), where Q comes from one sequence and K/V come from another.

Engineering Notes

KV Cache: Recomputing K and V for all previous tokens at each generation step would be O(nΒ²). The KV Cache stores all previous K and V vectors and reuses them β€” turning O(nΒ²) into O(n) per step, at the cost of memory proportional to sequence length.

Head dimension: Typical configs: d_k = d_model / num_heads. GPT-2: 768/12 = 64. GPT-3: 12288/96 = 128.

Grouped Query Attention (GQA): LLaMA-2, Mistral, and Gemma share a single KV head across multiple Query heads. This dramatically reduces KV Cache memory β€” a key reason these models handle long contexts on consumer hardware.

The One-Line Summary

Attention is not magic. It is a learned, differentiable, soft-weighted lookup table β€” recomputed freshly for every input, producing representations that encode the full relational structure of the text.

Want to Go Deeper?

The full version of this article β€” with Multi-Head Attention, causal masking, the complete production-grade NumPy implementation, and GQA architecture details β€” is on my technical blog:

πŸ‘‰ thelp.me/blogs/ai-engineering-use-case/ml-attention-calculation

Related deep-dives also on thelp.me:

This article was written by Swaroop. More LLM internals, RAG systems, and production ML at thelp.me.