Today's Agenda

1. The Transformer Revolution: Why it changed everything
2. Architecture Overview: Encoder, Decoder, and components
3. Self-Attention: The core mechanism (Q, K, V)
4. Self-Attention Example: Understanding pronoun resolution
5. Multi-Head Attention: Learning diverse relationships
6. Three Types of Attention: Self, Masked, Cross

Goal: Understand the Transformer architecture and self-attention

The Transformer Revolution

"Attention Is All You Need"

Reference: Vaswani et al. (2017) - "Attention Is All You Need"

Why Get Rid of RNNs?

Limitations of Recurrent Architectures:

1Sentence: "The cat that I saw
2yesterday at the park sat down"
3
4Token 1 ("The") to token 11 ("sat"):
5- RNN: Info passes through 10 steps
6- Gradients shrink: 0.9^10 = 0.35
7- By token 11, "The" is almost gone!
8
9Transformer: Direct connection!
10- "sat" attends directly to "cat"
11- No information degradation

Transformer Solution: Every token can attend to every other token directly!

Transformer Architecture Overview

1**Encoder -> Feed Forward -> Multi-Head Attention -> Feed Forward -> Multi-Head Attention -> Input -> \textbf{Decoder -> Feed Forward

\end{center**

Key Components:

• Multi-Head Self-Attention: Relate all positions to each other
• Feed-Forward Networks: Transform representations
• Residual Connections & Layer Norm: Training stability (not shown)
• Positional Encoding: Inject position information

Self-Attention: The Core Mechanism

Key idea: Each word attends to all other words in the sequence

Three learned projections:

• Query (Q): What am I looking for?
• Key (K): What do I contain?
• Value (V): What information do I have?

Computation:

1Q = X @ W_Q    # Transform input to queries
2K = X @ W_K    # Transform input to keys
3V = X @ W_V    # Transform input to values
4
5Attention(Q, K, V) = softmax(Q @ K.T / sqrt(d)) @ V

Scaled Dot-Product Attention

Step-by-step computation with concrete example:

Input: 3 tokens, embedding dim = 4

1# Input embeddings (3 tokens x 4 dims)
2X = [[0.1, 0.2, 0.3, 0.4],   # "The"
3     [0.5, 0.6, 0.7, 0.8],   # "cat"
4     [0.2, 0.3, 0.4, 0.5]]   # "sat"
5
6# Step 1: Compute Q, K, V (using learned weights W_q, W_k, W_v)
7Q = X @ W_q  # [3 x 4]
8K = X @ W_k  # [3 x 4]
9V = X @ W_v  # [3 x 4]
10
11# Step 2: Compute attention scores
12scores = Q @ K.T  # [3 x 3] - each token vs each token
13
14# Step 3: Scale by sqrt(d_k) to prevent large values
15scores = scores / sqrt(4)  # divide by 2
16
17# Step 4: Softmax to get attention weights
18weights = softmax(scores)  # rows sum to 1
19
20# Step 5: Weighted sum of values

Scaled Dot-Product Attention

21output = weights @ V  # [3 x 4] - new contextual embeddings

Self-Attention Example: Pronoun Resolution

Sentence: "The animal didn't cross the street because it was too tired"

Question: What does "it" refer to?

1Attention weights when processing "it":
2
3           The  animal  didn't  cross  the  street  because  it   was  too  tired
4"it" →    0.02  [0.45]   0.03   0.05  0.02  0.08    0.05   0.15  0.05 0.02  0.08
5                  ↑
6          High attention to "animal" - model learns coreference!

Self-attention allows the model to:

• Resolve pronouns ("it" → "animal", not "street")
• Understand long-range dependencies (8 tokens apart!)
• Capture syntactic and semantic relationships
• Do this in parallel for all positions!

Visualizing the Attention Matrix

For sentence: "The cat sat on the mat"

Observations:

• Each row sums to 1.0 (probability distribution)
• Diagonal elements often high (self-attention)
• "cat" attends to itself and "The" (determiner-noun relationship)
• "mat" attends to "the" (determiner) and "on" (preposition)
• Captures syntactic and semantic structure automatically!

Three Types of Attention

1. Self-Attention (Encoder)
   • Each position attends to all positions in same sequence

• Bidirectional: can see past and future
• Used in: BERT, encoder-only models

2. Masked Self-Attention (Decoder)
   • Each position attends only to previous positions

• Prevents "looking into the future"
• Used in: GPT, decoder-only models

3. Cross-Attention (Encoder-Decoder)
   • Decoder attends to encoder outputs

• Queries from decoder, Keys/Values from encoder
• Used in: T5, BART, machine translation

Masked Self-Attention

Preventing the model from "cheating" during generation

Problem: During training, we have the full target sequence. Without masking, the model could "peek" at future tokens!

Solution: Mask out future positions by setting attention scores to -infinity before softmax.

1# Example: Generating "The cat sat"
2# When predicting "sat", model should only see "The cat"
3
4scores = [[0.5, 0.3, 0.2],    # "The" can see: The
5          [0.4, 0.5, 0.1],    # "cat" can see: The, cat
6          [0.2, 0.4, 0.4]]    # "sat" can see: The, cat, sat
7
8# Apply causal mask (upper triangle = -infinity)
9mask = [[ 0,  -inf, -inf],
10        [ 0,   0,   -inf],
11        [ 0,   0,    0  ]]
12
13masked_scores = scores + mask
14# After softmax: future positions get weight 0!

Result: Token at position t can only attend to positions <= t

• Maintains autoregressive property
• Enables parallel training while preserving causality

Cross-Attention

Connecting encoder and decoder in seq2seq models

1Encoder Outputs ->  (Keys & Values) -> Decoder State ->  (Queries) -> Cross-Attention -> Context-Aware Decoder

Key Properties:

• Q comes from decoder (what decoder needs)
• K, V come from encoder (what input provides)
• Allows decoder to "look at" relevant parts of input
• Similar to the original attention mechanism from Lecture 12!

Used in: Machine translation, summarization, any encoder-decoder task

Implementing Self-Attention in PyTorch

21        scores = scores / math.sqrt(self.embed_dim)  # Scale!
22
23        if mask is not None:  # For causal/decoder attention
24            scores = scores.masked_fill(mask == 0, -1e9)
25
26        attn_weights = F.softmax(scores, dim=-1)  # Normalize
27        output = torch.matmul(attn_weights, V)    # Weighted sum
28
29        return output, attn_weights
30
31# Usage example:
32attn = SelfAttention(embed_dim=64)
33x = torch.randn(1, 5, 64)  # 5 tokens, 64-dim embeddings
34out, weights = attn(x)
35# out: [1, 5, 64] - contextualized embeddings
36# weights: [1, 5, 5] - attention matrix

Computational Complexity

Understanding the cost of self-attention

where n = sequence length, d = embedding dimension

Typical context limits:

• BERT: 512 | GPT-2: 1024 | GPT-3: 2048 | GPT-4: 128k (with optimizations)

Discussion Questions

1. Self-Attention vs RNN Attention:
   • What's the key difference?

• Why is self-attention more powerful?
• When might RNNs still be useful?

2. Query, Key, Value Framework:
   • Why three separate projections instead of one?

• What if we used ?
• How does this relate to information retrieval?

3. Multi-Head Attention:
   • Why not just use one big attention head?

• How many heads is optimal?
• Can we interpret what each head learns?

4. Scalability:
   • is problematic for long documents. Solutions?

• Sparse attention? Local attention? Other ideas?

Looking Ahead

What's Next?

Today we learned:

• Why transformers replaced RNNs
• Self-attention mechanism (Q, K, V)
• Multi-head attention
• Three types of attention (self, masked, cross)

Next lecture (Lecture 14 - Training Transformers):

• : How to inject position information
• : The other key component
• : Training stability
• : Making transformers faster
• : Encoder, Decoder, Encoder-Decoder
• : Training and using transformers

We're building up to BERT and GPT!

Summary

Key Takeaways:

1. Transformer Revolution
   • Pure attention, no recurrence

• Parallel processing, faster training

2. Self-Attention Mechanism
   • Query, Key, Value framework

• Each token attends to all others
• Scaled dot-product:

3. Multi-Head Attention
   • Multiple heads learn diverse relationships

• Concatenate and project back
• Richer representations

4. Three Attention Types
   • Self (encoder), Masked (decoder), Cross (encoder-decoder)

• Different uses for different architectures

Self-attention is the foundation of modern NLP!

References

Essential Papers:

• Vaswani et al. (2017) - "Attention Is All You Need"

• The original Transformer paper

• Introduced self-attention, multi-head attention

• Foundation of modern NLP

  \item Bahdanau et al. (2015) - "Neural Machine Translation by Jointly Learning to Align and Translate"

  • Original attention mechanism (for comparison)

Tutorials and Resources:

• The Illustrated Transformer by Jay Alammar

• https://jalammar.github.io/illustrated-transformer/

• Visual step-by-step explanation

  \item Annotated Transformer by Harvard NLP

  • https://nlp.seas.harvard.edu/annotated-transformer/

• Line-by-line implementation

  \item HuggingFace Course - Chapter 1.4

  • How Transformers work

Questions?

Discussion Time

Topics for discussion:

• Self-attention mechanism
• Query, Key, Value intuition
• Multi-head attention
• Masked vs. unmasked attention
• Implementation questions

Thank you!

Next: Training Transformers!