Some Thoughts on Attention Mechanisms and the KV-Cache
Some notes on attention mechanisms.
LLM / Notes
1 Fundamentals of the Attention Mechanism and KV-Cache
1.1 Self-Attention vs. Cross-Attention
- Self-Attention: The Query , Key , and Value matrices are all derived from the same input sequence. This structure enables each token to observe and integrate information from all other tokens within the sequence, effectively capturing internal semantic dependencies.
- Cross-Attention: The matrix originates from one sequence (typically the Decoder), while the and matrices originate from another (typically the Encoder). This mechanism allows a target sequence to “query” information from a source sequence. A classic example is machine translation: when generating a target word (acting as ), the model queries the source sentence (acting as and ) to identify the most relevant context.
1.2 The Roles of , , and
The interaction within the attention mechanism can be intuitively understood as an information retrieval process from an “archive”:
- Key and Value - The Archive: These represent the deterministic, historical context. Specifically, acts as the label/index (determining who is important), while contains the actual content (determining what information to extract).
- Query - The Search Request: encodes the current step’s intent. It matches against all available matrices to compute attention weights, subsequently retrieving the most relevant combination of matrices.
1.3 Autoregressive Generation and the KV-Cache
During inference, standard Transformers operate autoregressively. Generating the -th token requires computing its attention relationship with all preceding tokens. Without caching, the Attention computation is formulated as:
In this naive approach, to compute the output at position , the model must redundantly recompute the keys and values for all historical positions.
Since drives the active generation while and provide static historical information, models store historical keys and values in the KV-Cache. Consequently, the memory capacity allocated for the KV-Cache directly dictates the maximum context window the model can retain.
1.4 Semantic Compression via Hidden States
Throughout the sequential generation pipeline:
- The query used to predict the -th token is intrinsically derived from the output embedding of the -th token.
- As the representation processes through the Transformer layers, the final layer’s output, , serves as the semantic compression (or summarization) of all sequential information spanning from position to position .
2 An Example to Understand the Generation Process
Suppose we have a simplified one-layer Transformer. Our target is to make the model autoregressively generate the sequence: “To be or not to be.”
Let us define the key notations and their corresponding dimensions. Let denote the model’s hidden dimension, and denote the vocabulary size:
- Input Representation : The input vector at step , which is the sum of the current token embedding and the positional embedding.
- : Obtained by multiplying with three respective weight matrices (). Here, acts as the query, while and are the keys and values to be stored in the KV-Cache.
- Context Vector : The output of the Attention layer. It is a weighted sum of the historical values () and represents the features incorporating context information.
- Hidden State : The final output of the Transformer layer after passing through a FFN/MLP. It represents the accumulated semantic information up to step .
Now, let’s examine how the model generates this sequence step-by-step.
Step 1: Initialization (Input: [BOS] Predict: "To")
We feed a special token [BOS] (Begin of Sequence) to prompt the model to start generating.
- Input Layer: Compute the token embedding for
[BOS], add the positional encoding, and obtain . - Generate : Apply linear projections to to obtain .
- Update KV-Cache: Store and into the cache. The current cache size for both K and V is .
- Attention : Since there is no prior context, only attends to (the attention weight is trivially ). Thus, .
- Hidden State & Prediction: Pass through the MLP to obtain . Finally, is projected by the Language Modeling (LM) head (using a weight matrix ) to yield a logits vector , predicting the highest-probability next token:
"To".
Step 2: Incorporating Context (Input: "To" Predict: "be")
In the autoregressive setting, the token predicted in the previous step ("To") becomes the input for the current step.
- Input Layer: Compute the token embedding for
"To", add the positional encoding for position 2, and obtain . - Generate : Apply the linear projections to to generate . Here, encodes the semantic query: “Given the context so far, what should follow the word ‘To’?”
- Update KV-Cache: Append the newly computed row vectors to the existing KV-Cache. The cache now holds the accumulated history matrices .
- Attention : The query attends to all available keys in the cache via dot-product, resulting in attention scores . After applying the softmax function to these scores, the resulting attention weights are used to compute a weighted sum of . The output, , is no longer the isolated representation of
"To"; it is now a context-aware feature vector. - Hidden State & Prediction: The context vector is processed by the MLP to yield the hidden state . Finally, the LM head maps to the vocabulary distribution , predicting the highest-probability next token:
"be".
The subsequent steps follow the exact same autoregressive pattern, appending new and matrices to the cache and generating the next sequence token until a termination condition is met.
If you have any thoughts you'd like to discuss, feel free to reach out: xsfeng@gatech.edu