How Transformer Self-Attention Works

Introduction

At the heart of the modern artificial intelligence revolution (from GPT-4 to Claude 3.5 and Gemini Pro) is a single, unified mathematical operations layer: Self-Attention.

First introduced in the landmark 2017 paper “Attention Is All You Need” by Vaswani et al., the self-attention mechanism allowed neural networks to compute contextual connections between all tokens in a sequence simultaneously. This replaced sequential Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) layers, bypassing their sequential bottlenecks and enabling parallel training at massive scales.

---

THE INTUITIVE ANALOGY: DATABASE LOOKUPS

Before exploring the linear algebra equations, let us use a database retrieval analogy:

• Query ($Q$): The target search term you are looking for (e.g. the active word being processed).
• Key ($K$): The search label indexes of all records in the database (e.g. every other word in the sentence).
• Value ($V$): The actual content payload returned by the database.

During self-attention, the model matches the Query vector against the Keys of all surrounding tokens to calculate a compatibility score (attention map weight). The output is a weighted combination of the corresponding Values.

---

THE CORE MATHEMATICS: SCALED DOT-PRODUCT ATTENTION

For an input token sequence matrix $X$, the attention module projects the tokens into Query, Key, and Value matrices using trained weight parameters $W_Q$, $W_K$, and $W_V$: $Q = XW_Q, \quad K = XW_K, \quad V = XW_V$

Once projected, the Scaled Dot-Product Attention is calculated using the following equation: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

graph TD
    Q[Query Q] --> Dot[Dot Product Q*K^T]
    K[Key K] --> Dot
    Dot --> Scale[Scale / sqrt d_k]
    Scale --> Softmax[Softmax Normalized Weights]
    Softmax --> ValDot[Weighted Values Sum * V]
    V[Value V] --> ValDot
    ValDot --> Output[Attention Vector Output]

Breaking Down the Operation:

1. The Dot Product ($QK^T$)

Multiplying the Query matrix by the transpose of the Key matrix yields the raw similarity score between every pair of tokens. A higher dot product represents higher semantic alignment.

2. The Scaling Factor ($\sqrt{d_k}$)

The dimension of the key vectors is represented by $d_k$. As $d_k$ grows large, the dot product values scale to very large magnitudes. This pushes the subsequent softmax function into regions with extremely small gradients (the vanishing gradient problem). Dividing by the square root of the dimension ($\sqrt{d_k}$) scales the variance back to 1.0, preserving gradient flow during backpropagation training.

3. Softmax Normalization

The softmax operation is applied across the rows to convert raw similarity numbers into normalized probability weights between 0.0 and 1.0: $\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$

4. Vector Aggregation

Multiplying the normalized softmax weights by the Value matrix ($V$) yields the final contextual token embedding. Words that the model determines are contextually linked (like the pronoun “it” and its target noun “street”) are blended together into a single, high-dimensional vector.