Today's Journey

Required Libraries

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4from torch.utils.data import Dataset, DataLoader
5import numpy as np
6import tiktoken  # OpenAI's BPE tokenizer
7
8# Check for GPU
9device = 'cuda' if torch.cuda.is_available() else 'cpu'
10print(f"Using device: {device}")
11
12# Hyperparameters
13config = {
14    'vocab_size': 50257,      # GPT-2 vocabulary size
15    'd_model': 384,            # Embedding dimension
16    'n_layers': 6,             # Number of transformer blocks
17    'n_heads': 6,              # Number of attention heads
18    'max_seq_len': 256,        # Maximum sequence length
19    'dropout': 0.1,
20    'batch_size': 32,

Required Libraries

21    'learning_rate': 3e-4,
22    'num_epochs': 10
23}

Install: pip install torch tiktoken

What is Tokenization?

<div class="callout info">

Converting text into a sequence of integer IDs that the model can process.

**Common strategies:**
1. **Word-level**: Split on spaces
    -  Huge vocabulary, can't handle unknown words

2. Character-level: Individual characters
   - Very long sequences, loses word structure
3. Subword-level (BPE): Best of both worlds!
   - Frequent words: one token

• Rare words: multiple subword tokens
• Can handle any word via character fallback

Byte-Pair Encoding (BPE)

**How BPE works:**



1. Start with characters as base vocabulary

2. Find most frequent pair of adjacent tokens

3. Merge this pair into a new token

4. Repeat until desired vocabulary size

    **Text:** "low low low lower lower newest newest"

    

    **Iterations:**
    1. Merge "l" + "o" → "lo"

2. Merge "lo" + "w" → "low"

3. Merge "low" + "e" + "r" → "lower"

4. Merge "n" + "e" + "w" + "e" + "s" + "t" → "newest"

    **Vocabulary:** [l, o, w, e, r, n, s, t, lo, low, lower, newest, ...]
   

BPE Tokenization in Code

1import tiktoken
2
3# Load GPT-2 tokenizer (uses BPE)
4tokenizer = tiktoken.get_encoding("gpt2")
5
6# Example text
7text = "Hello, how are you doing today?"
8
9# Encode: text -> token IDs
10tokens = tokenizer.encode(text)
11print("Tokens:", tokens)
12# Output: [15496, 11, 703, 389, 345, 1804, 1909, 30]
13
14# Decode: token IDs -> text
15decoded = tokenizer.decode(tokens)
16print("Decoded:", decoded)
17# Output: "Hello, how are you doing today?"
18
19# See individual token strings
20for token_id in tokens:

BPE Tokenization in Code

21    token_str = tokenizer.decode([token_id])
22    print(f"{token_id}: '{token_str}'")
23
24# Vocabulary size
25print(f"Vocab size: {tokenizer.n_vocab}")  # 50257

Creating a Text Dataset

21        x = torch.tensor(chunk[:-1], dtype=torch.long)
22        # Target: all but first token
23        y = torch.tensor(chunk[1:], dtype=torch.long)
24
25        return x, y
26
27# Usage
28dataset = TextDataset('shakespeare.txt', tokenizer, config['max_seq_len'])
29dataloader = DataLoader(dataset, batch_size=config['batch_size'], shuffle=True)

Token and Position Embeddings

1class Embeddings(nn.Module):
2    def __init__(self, vocab_size, d_model, max_seq_len, dropout):
3        super().__init__()
4        # Token embeddings: map token IDs to vectors
5        self.token_embed = nn.Embedding(vocab_size, d_model)
6
7        # Position embeddings: encode position information
8        self.pos_embed = nn.Embedding(max_seq_len, d_model)
9
10        self.dropout = nn.Dropout(dropout)
11        self.d_model = d_model
12
13    def forward(self, x):
14        # x shape: (batch_size, seq_len)
15        seq_len = x.size(1)
16
17        # Token embeddings
18        tok_emb = self.token_embed(x)  # (batch, seq_len, d_model)
19
20        # Position embeddings

Token and Position Embeddings

21        positions = torch.arange(0, seq_len, device=x.device)
22        pos_emb = self.pos_embed(positions)  # (seq_len, d_model)
23
24        # Combine (broadcasting handles batch dimension)
25        embeddings = tok_emb + pos_emb
26
27        return self.dropout(embeddings)

Masked Multi-Head Attention

21
22        # Linear projections and split into heads
23        Q = self.q_linear(x).view(batch_size, seq_len, self.n_heads, self.head_dim)
24        K = self.k_linear(x).view(batch_size, seq_len, self.n_heads, self.head_dim)
25        V = self.v_linear(x).view(batch_size, seq_len, self.n_heads, self.head_dim)
26
27        # Transpose for attention: (batch, n_heads, seq_len, head_dim)
28        Q = Q.transpose(1, 2)
29        K = K.transpose(1, 2)
30        V = V.transpose(1, 2)

Attention Computation (cont.)

1# Scaled dot-product attention
2        # Scores: (batch, n_heads, seq_len, seq_len)
3        scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.head_dim)
4
5        # Apply causal mask (prevent attending to future tokens)
6        if mask is not None:
7            scores = scores.masked_fill(mask == 0, float('-inf'))
8
9        # Softmax to get attention weights
10        attn_weights = F.softmax(scores, dim=-1)
11        attn_weights = self.dropout(attn_weights)
12
13        # Apply attention to values
14        # Output: (batch, n_heads, seq_len, head_dim)
15        attn_output = torch.matmul(attn_weights, V)
16
17        # Concatenate heads
18        attn_output = attn_output.transpose(1, 2).contiguous()
19        attn_output = attn_output.view(batch_size, seq_len, d_model)
20

Attention Computation (cont.)

21        # Final linear projection
22        output = self.out_linear(attn_output)
23
24        return output

Creating the Causal Mask

1def create_causal_mask(seq_len, device):
2    """
3    Create a causal (lower-triangular) mask for autoregressive generation.
4
5    Returns:
6        mask: (seq_len, seq_len) with 1s on and below diagonal, 0s above
7    """
8    mask = torch.tril(torch.ones(seq_len, seq_len, device=device))
9    return mask  # Shape: (seq_len, seq_len)
10
11# Example: 5x5 causal mask
12mask = create_causal_mask(5, 'cpu')
13print(mask)
14# tensor([[1., 0., 0., 0., 0.],
15#         [1., 1., 0., 0., 0.],
16#         [1., 1., 1., 0., 0.],
17#         [1., 1., 1., 1., 0.],
18#         [1., 1., 1., 1., 1.]])

Key insight: Each position can only attend to itself and previous positions!

Feed-Forward Network

1class FeedForward(nn.Module):
2    def __init__(self, d_model, dropout):
3        super().__init__()
4        # GPT uses 4 * d_model as the hidden dimension
5        self.net = nn.Sequential(
6            nn.Linear(d_model, 4 * d_model),
7            nn.GELU(),  # GPT uses GELU activation
8            nn.Linear(4 * d_model, d_model),
9            nn.Dropout(dropout)
10        )
11
12    def forward(self, x):
13        return self.net(x)

Why GELU (Gaussian Error Linear Unit)?

• Smooth, non-monotonic activation
• Used in BERT and GPT
• Slight improvement over ReLU for language models

Transformer Decoder Block

1class TransformerBlock(nn.Module):
2    def __init__(self, d_model, n_heads, dropout):
3        super().__init__()
4        self.attention = MultiHeadAttention(d_model, n_heads, dropout)
5        self.feed_forward = FeedForward(d_model, dropout)
6
7        # Layer normalization (applied before sub-layers in GPT)
8        self.ln1 = nn.LayerNorm(d_model)
9        self.ln2 = nn.LayerNorm(d_model)
10
11    def forward(self, x, mask):
12        # Pre-norm architecture (used in GPT)
13        # Attention with residual connection
14        x = x + self.attention(self.ln1(x), mask)
15
16        # Feed-forward with residual connection
17        x = x + self.feed_forward(self.ln2(x))
18
19        return x

Note: GPT uses pre-norm (LayerNorm before sub-layers), while original Transformer used post-norm.

Complete GPT Model

21        # Initialize weights
22        self.apply(self._init_weights)
23
24    def _init_weights(self, module):
25        if isinstance(module, nn.Linear):
26            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
27            if module.bias is not None:
28                torch.nn.init.zeros_(module.bias)
29        elif isinstance(module, nn.Embedding):
30            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)

GPT Forward Pass

21        # Compute loss if targets provided
22        loss = None
23        if targets is not None:
24            # Flatten for cross-entropy
25            loss = F.cross_entropy(
26                logits.view(-1, logits.size(-1)),
27                targets.view(-1)
28            )
29
30        return logits, loss

Training Loop

21for epoch in range(config['num_epochs']):
22    total_loss = 0
23    for batch_idx, (x, y) in enumerate(dataloader):
24        x, y = x.to(device), y.to(device)
25
26        # Forward pass
27        logits, loss = model(x, targets=y)
28
29        # Backward pass
30        optimizer.zero_grad()
31        loss.backward()
32        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
33        optimizer.step()
34
35        total_loss += loss.item()
36
37    avg_loss = total_loss / len(dataloader)
38    print(f"Epoch {epoch+1}/{config['num_epochs']}, Loss: {avg_loss:.4f}")

Training Tips

**Best practices for training GPT:**



1. **Gradient clipping**
    - Prevents exploding gradients

• Clip to max norm of 1.0

2. Learning rate schedule
   - Warmup for first few thousand steps

• Cosine decay afterwards

3. AdamW optimizer
   - Adam with decoupled weight decay

• Better than standard Adam for transformers

4. Batch size
   - Larger is better (up to memory limits)

• Use gradient accumulation if needed

5. Mixed precision training
   - Use float16 for speed

• 2-3x faster on modern GPUs

Greedy Decoding

21        x = torch.cat([x, next_token], dim=1)
22
23        # Stop if we generate end-of-sequence token
24        if next_token.item() == tokenizer.eot_token:
25            break
26
27    # Decode and return
28    generated_text = tokenizer.decode(x[0].tolist())
29    return generated_text
30
31# Example usage
32prompt = "Once upon a time"
33generated = generate_greedy(model, tokenizer, prompt, max_new_tokens=100)
34print(generated)

Sampling Strategies

**Different ways to sample next token:**



1. **Greedy Decoding**
    - Always pick most likely token

• Deterministic, fast
• Repetitive, boring

2. Temperature Sampling
   - Scale logits by temperature

• : More conservative (peaked distribution)
• : More random (flat distribution)

3. Top-k Sampling
   - Sample from top most likely tokens

• Typical:

4. Nucleus (Top-p) Sampling
   - Sample from smallest set with cumulative probability

• Typical: or
• Best for creative generation

Top-k and Nucleus Sampling

21    if top_p is not None:
22        sorted_logits, sorted_indices = torch.sort(logits, descending=True)
23        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
24
25        # Remove tokens with cumulative probability above threshold
26        sorted_indices_to_remove = cumulative_probs > top_p
27        sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
28        sorted_indices_to_remove[..., 0] = 0
29
30        indices_to_remove = sorted_indices[sorted_indices_to_remove]
31        logits[indices_to_remove] = float('-inf')
32
33    # Sample from distribution
34    probs = F.softmax(logits, dim=-1)
35    next_token = torch.multinomial(probs, num_samples=1)
36
37    return next_token

Complete Generation Function

21
22        x = torch.cat([x, next_token.unsqueeze(0)], dim=1)
23
24        if next_token.item() == tokenizer.eot_token:
25            break
26
27    return tokenizer.decode(x[0].tolist())
28
29# Creative generation
30text = generate(model, tokenizer, "The AI revolution",
31                temperature=0.8, top_p=0.9)
32print(text)

Model Size vs. Performance

1 10M ->  100M ->  1B ->  10B ->  100B+

**Practical model sizes for different use cases:**
- **10M-100M**: Learning/experimentation, simple tasks

• 100M-1B: Specialized domains, resource-constrained
• 1B-10B: General-purpose, good quality
• 10B-100B+: State-of-the-art performance

Computational Requirements

**Training a 125M parameter GPT:**




    | Training Time | 1-2 days (single GPU) |

| --- | --- |
| Training Data | $\sim$10-100 GB text |
| Total Compute | $\sim$100 GPU-hours |

**Scaling up to GPT-3 (175B):**
- 1000x more parameters

• ~10,000x more compute needed

• Requires distributed training across many GPUs

• Estimated $4.6M in compute costs

This is why pre-trained models are so valuable—you don't have to train from scratch!

Common Issues and Debugging

**Problems you might encounter:**



1. **Loss not decreasing**
    - Check learning rate (try 1e-4 to 3e-4)

• Verify data pipeline
• Check for NaN/Inf values

2. Out of memory
   - Reduce batch size

• Reduce sequence length
• Use gradient accumulation
• Enable mixed precision

3. Poor generation quality
   - Train longer

• Use larger model
• Improve data quality
• Tune sampling parameters

4. Repetitive text
   - Increase temperature

• Use nucleus sampling
• Add repetition penalty

Extensions and Improvements

**Ways to enhance your GPT implementation:**



1. **Architectural improvements**
    - Rotary Position Embeddings (RoPE)

• Flash Attention (faster attention)
• Grouped-Query Attention

2. Training techniques
   - Learning rate warmup and decay

• Gradient accumulation
• Mixed precision training (FP16/BF16)

3. Data improvements
   - Data deduplication

• Quality filtering
• Curriculum learning

4. Inference optimizations
   - KV-cache for faster generation

• Model quantization (int8)
• Speculative decoding

Resources for Further Learning

**Recommended resources:**



- **Andrej Karpathy's nanoGPT**

• Clean, minimal GPT implementation

• github.com/karpathy/nanoGPT

    \item **Andrej Karpathy's "Let's build GPT" video**
    - Excellent step-by-step tutorial
  

• YouTube

    \item **HuggingFace Transformers**
    - Production-ready implementations
  

• huggingface.co/transformers

    \item **PyTorch Documentation**
    - Official tutorials and guides
  

• pytorch.org/tutorials

Hands-On Exercise

<div class="callout warning">

Implement and train a small GPT model on your own text corpus!

**Steps:**
1. Choose a dataset (Shakespeare, Wikipedia, your own text)

2. Set up the data pipeline

3. Initialize the model (start small: 6 layers, 384 d_model)

4. Train for 10-20 epochs

5. Experiment with generation

6. Try different sampling strategies

   Starter code available:

   • Course GitHub repository

• Google Colab notebook
• Estimated time: 2-3 hours

Key Takeaways

1. **GPT is conceptually simple**
    - Stack of transformer decoder blocks

• Predict next token

2. Key components
   - BPE tokenization

• Token + position embeddings
• Masked multi-head attention
• Feed-forward networks

3. Training requires care
   - Good data, proper hyperparameters

• Gradient clipping, learning rate schedules

4. Generation is an art
   - Balance creativity and coherence

• Temperature, top-k, top-p sampling

5. Implementation teaches you how LLMs work
   - Understanding through building!

Next Week

**Week 8: No Classes - Instructor Away**



**Use this week to:**
- Complete the GPT implementation exercise

• Catch up on readings

• Work on assignments

• Experiment with different architectures

  Week 9: RAG & Mixture of Experts

  • Retrieval Augmented Generation

• Mixture of Experts architectures

• Ethics, Bias, and Safety in LLMs

    Questions? 

    

    Happy Coding!