Attention Mechanism
AI · Advanced
Definition
Attention lets a model weigh every token against every other using queries, keys and values.
The core idea behind Transformers and LLMs.
Diagram
Flowchart
Example: Softmax weights
import math
s = [2.0, 1.0, 0.0]
e = [math.exp(v) for v in s]
print([round(v / sum(e), 3) for v in e])Output: [0.665, 0.245, 0.09]
Complexity
O(n²) in sequence length
Key points
- Multi-head attention learns different relations
- Scaling by √d keeps softmax stable
Interview questions
Self-attention vs cross-attention?
Self-attention uses one sequence for Q, K, V; cross-attention takes Q from one and K, V from another.