Today's Agenda

1. Positional Encoding: Injecting sequence order
2. Feed-Forward Networks: The other key component
3. Layer Norm & Residuals: Training deep networks
4. FlashAttention: Making transformers faster
5. Three Architectures: Encoder, Decoder, Both
6. Practical Implementation: Using HuggingFace

Goal: Complete understanding of transformer training and implementation

Positional Encoding

Problem: Self-attention is permutation-invariant!

Without position information:

• "The cat sat" = "sat cat the" = "cat the sat"
• Order matters in language!

Solution: Add positional encodings to input embeddings

1# PE(pos, 2i) = sin(pos / 10000^(2i/d))
2# PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
3
4# Position 0, dim 0: sin(0/10000^0) = 0
5# Position 1, dim 0: sin(1/10000^0) = 0.84
6# Position 2, dim 0: sin(2/10000^0) = 0.91

1Token embeddings:
2"The" = [0.2, 0.5, 0.1, 0.8]
3"cat" = [0.9, 0.3, 0.7, 0.2]
4
5Positional encodings:
6pos_0 = [0.0, 1.0, 0.0, 1.0]
7pos_1 = [0.84, 0.54, 0.01, 1.0]
8
9Final input (add them):
10"The" = [0.2, 1.5, 0.1, 1.8]
11"cat" = [1.74, 0.84, 0.71, 1.2]

References: Vaswani et al. (2017), Su et al. (2021)

Why Sinusoidal Positional Encoding?

Advantages of sine/cosine functions:

1Dim 0 (high freq): ~~~~~ (fast oscillation)
2Dim 1 (mid freq):  ~~~   (medium)
3Dim 2 (low freq):  ~     (slow)
4
5Position 0: [0.0, 0.0, 0.0, ...]
6Position 1: [0.84, 0.01, 0.0001, ...]
7Position 2: [0.91, 0.02, 0.0002, ...]
8...
9Position 100: [0.51, 0.86, 0.01, ...]
10
11Each position has a unique "barcode"!

Visualizing Positional Encodings

Each position gets a unique pattern across dimensions

1Pos 0 -> Pos 5 -> Pos 9 -> Dim 0 -> Dim 4 -> Dim 7

Key Insight:

• Lower dimensions: Rapid oscillation (high frequency)
• Higher dimensions: Slow oscillation (low frequency)
• Creates a unique "barcode" for each position
• Model learns to use these patterns for positional awareness

Feed-Forward Networks

After attention, apply position-wise feed-forward network

Architecture: Two linear layers with ReLU/GELU activation

1class FeedForward(nn.Module):
2    def __init__(self, d_model=768, d_ff=3072):  # 4x expansion
3        super().__init__()
4        self.linear1 = nn.Linear(d_model, d_ff)    # 768 → 3072
5        self.linear2 = nn.Linear(d_ff, d_model)    # 3072 → 768
6        self.relu = nn.ReLU()
7
8    def forward(self, x):
9        # x: [batch, seq_len, 768]
10        x = self.linear1(x)   # [batch, seq_len, 3072]
11        x = self.relu(x)      # Non-linearity!
12        x = self.linear2(x)   # [batch, seq_len, 768]
13        return x
14
15# Applied to each position independently
16# Same weights for all positions

Purpose: Add non-linearity and increase model capacity

• Attention is mostly linear (weighted sums)
• FFN adds expressiveness through ReLU/GELU

Why Feed-Forward Networks?

Role in the Transformer:

1. Add Non-linearity
   • Attention is mostly linear operations (weighted sums)

• FFN introduces non-linear transformations
• ReLU/GELU activation adds expressiveness

2. Position-wise Processing
   • Attention mixes information across positions

• FFN processes each position independently
• Allows position-specific feature transformations

3. Increase Model Capacity
   • Expansion (4x) provides more parameters

• Can learn complex feature combinations
• Most parameters in transformer are in FFN layers!

4. Feature Refinement
   • Attention gathers context

• FFN refines and transforms the representation
• Two complementary operations

Layer Normalization & Residual Connections

Critical for training deep transformers!

1# output = sublayer(x) + x
2x = x + self.attention(x)
3x = x + self.feedforward(x)

1# Normalize across features (not batch)
2def layer_norm(x, gamma, beta):
3    mean = x.mean(dim=-1)
4    std = x.std(dim=-1)
5    return gamma * (x - mean) / std + beta
6
7# Example: x = [0.2, 0.8, 0.5]
8# mean=0.5, std=0.25
9# normalized = [-1.2, 1.2, 0.0]

Standard Pattern (Post-Norm):

1x = LayerNorm(x + Attention(x))
2x = LayerNorm(x + FeedForward(x))

Complete Transformer Block

Putting it all together:

1class TransformerBlock(nn.Module):
2    def __init__(self, d_model=768, n_heads=12, d_ff=3072):
3        super().__init__()
4        self.attention = MultiHeadAttention(d_model, n_heads)
5        self.ffn = FeedForward(d_model, d_ff)
6        self.norm1 = nn.LayerNorm(d_model)
7        self.norm2 = nn.LayerNorm(d_model)
8
9    def forward(self, x):
10        # Self-attention with residual
11        x = x + self.attention(self.norm1(x))
12        # Feed-forward with residual
13        x = x + self.ffn(self.norm2(x))
14        return x
15
16# Stack N blocks!
17encoder = nn.Sequential(*[TransformerBlock() for _ in range(12)])

Model sizes:

• BERT-base: 12 blocks, 110M params | BERT-large: 24 blocks, 340M params
• GPT-3: 96 blocks, 175B params!

Other Attention Optimizations

Addressing the problem:

1. Sparse Attention
   • Only attend to subset of positions

• Local windows + global tokens
• Used in: Longformer, BigBird

2. Linear Attention
   • Approximate attention with linear complexity

• Kernel trick to avoid materializing attention matrix
• Used in: Performer, Linear Transformer

3. Low-Rank Approximation
   • Factorize attention matrix

• Reduce memory footprint
• Used in: Linformer

4. Sliding Window
   • Fixed-size local attention window

• Constant memory usage
• Used in: Mistral 7B

Trade-off: Efficiency vs. expressiveness. Full attention often still best for quality.

Three Transformer Architectures

Visual Comparison: Encoder vs Decoder vs Both

1**Encoder-Only (BERT) -> Self-Attention ->  bidirectional -> \textbf{Decoder-Only (GPT) -> Masked Attn ->  causal

\end{center**

Choosing the Right Architecture:

• Need to understand full context? → Encoder (BERT)
• Need to generate text? → Decoder (GPT)
• Need both (translate, summarize)? → Encoder-Decoder (T5)

Using Transformers in Practice

HuggingFace makes it easy!

1from transformers import AutoModel, AutoTokenizer
2
3# Load pre-trained model (downloads ~440MB first time)
4model_name = "bert-base-uncased"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModel.from_pretrained(model_name)
7
8# Tokenize input
9text = "The animal didn't cross the street because it was too tired"
10inputs = tokenizer(text, return_tensors="pt")
11print(inputs['input_ids'])
12# tensor([[  101,  1996,  4111,  2134,  1005,  1056,  2892,  1996,
13#           2395,  2138,  2009,  2001,  2205,  5458,   102]])
14#         [CLS]  The  animal didn  '     t    cross  the ...
15
16# Get contextualized embeddings
17outputs = model(**inputs)
18hidden_states = outputs.last_hidden_state  # [1, 15, 768]
19
20# "it" is at position 10 - its embedding knows it refers to "animal"!

Using Transformers in Practice

21it_embedding = hidden_states[0, 10, :]  # 768-dim context-aware vector

Reference: HuggingFace Course - Chapter 1.4

Comparing Architectures: Code Examples

Different architectures for different tasks

1from transformers import AutoModelForSequenceClassification, \
2                         AutoModelForCausalLM, \
3                         AutoModelForSeq2SeqLM
4
5# 1. ENCODER-ONLY (BERT): Classification
6encoder_model = AutoModelForSequenceClassification.from_pretrained(
7    "bert-base-uncased", num_labels=2
8)
9# Use for: Sentiment analysis, NER, classification
10
11# 2. DECODER-ONLY (GPT): Text generation
12decoder_model = AutoModelForCausalLM.from_pretrained("gpt2")
13# Use for: Text completion, creative writing, few-shot learning
14
15# 3. ENCODER-DECODER (T5): Seq2Seq tasks
16seq2seq_model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
17# Use for: Translation, summarization, question answering
18
19# All use transformer architecture, different attention patterns!

Key Takeaway: Choose architecture based on your task!

Computational Efficiency Tips

1. Batch Size
   • Larger batches = better GPU utilization

• Use gradient accumulation if GPU memory limited
• Typical: effective batch size 256-2048 tokens

2. Sequence Length
   • Shorter sequences train faster (quadratic complexity!)

• Consider truncation or sliding windows
• Pack multiple examples to maximize GPU usage

3. Model Size
   • Start small, scale up if needed

• DistilBERT: 40% smaller, 60% faster, 97% performance
• Consider model distillation for deployment

4. Hardware
   • GPUs with high memory bandwidth (A100, H100)

• Multi-GPU training with data parallelism
• Use FlashAttention when available

Discussion Questions

1. Positional Encoding:
   • Why add instead of concatenate?

• What happens without positional encoding?
• Learned vs. fixed: which is better?

2. Architecture Choice:
   • When would you use encoder-only vs decoder-only?

• Can GPT do classification? Can BERT generate?
• Why has decoder-only become more popular recently?

3. Scaling:
   • Is bigger always better?

• What are the limits to scaling transformers?
• How do we make them more efficient?

4. Training Stability:
   • Why are residual connections so important?

• Pre-norm vs post-norm: trade-offs?

Looking Ahead to Week 6

This week (Week 5) we learned:

• Attention mechanisms (Lecture 12)
• Self-attention and transformer architecture (Lecture 13)
• Training transformers: all the components (Lecture 14)

Next week (Week 6):

• Masked Language Modeling

• Pre-training and fine-tuning

• Contextual embeddings in action

  \item

  • RoBERTa, ALBERT, DistilBERT

• Improvements and optimizations

  \item

  • Real-world BERT applications

• Cognitive neuroscience connections

• Understanding vs. pattern matching

Assignment 4: Building context-aware systems (check syllabus for details)

Summary

Key Takeaways:

1. Positional Encoding
   • Sine/cosine functions inject position information

• Enables model to understand order

2. Feed-Forward Networks
   • Position-wise transformations

• Add non-linearity and capacity

3. LayerNorm & Residuals
   • Critical for training deep networks

• Stabilize gradients, enable deeper models

4. Three Architectures
   • Encoder (BERT), Decoder (GPT), Both (T5)

• Choose based on task requirements

5. Practical Considerations
   • Use HuggingFace for easy implementation

• FlashAttention for efficiency
• Careful hyperparameter tuning

References

Essential Papers:

• Vaswani et al. (2017) - "Attention Is All You Need"

• The original Transformer paper

  \item Su et al. (2021) - "RoFormer: Enhanced Transformer with Rotary Position Embedding"

  • Modern positional encoding approach

  \item Dao et al. (2022) - "FlashAttention: Fast and Memory-Efficient Exact Attention"

  • Making transformers faster

Tutorials:

• HuggingFace Course: Chapters 1.4, 1.5
• The Illustrated Transformer: https://jalammar.github.io/illustrated-transformer/
• Annotated Transformer: https://nlp.seas.harvard.edu/annotated-transformer/

Questions?

Discussion Time

Topics for discussion:

• Positional encoding approaches
• Architecture choices
• Training tips and tricks
• Implementation questions
• Assignment 4 preparation

Thank you!

See you next week for BERT!