Today's Journey

1. GPT-2 - 10x scale-up, zero-shot multitask learning
2. GPT-3 - 100x scale-up, few-shot learning emerges
3. Scaling Laws - Why bigger is (predictably) better
4. RLHF & ChatGPT - Aligning LLMs with human preferences
5. Modern Landscape - Open vs closed models
6. Hands-on Examples - Prompting techniques in practice

WebText: Concrete Examples

What kinds of documents were included:

1Reddit post (3+ karma): "Check out this great article about climate science"
2 -> Linked article scraped and included in training
3
4Reddit post (3+ karma): "Here's an amazing tutorial on machine learning"
5 -> Tutorial content included in training
6
7Reddit post (2 karma): "Random blog post"
8 -> EXCLUDED (below karma threshold)

Sample document types in WebText:

GPT-2 Model Sizes

Four model sizes released progressively:

Staged release strategy:

• Feb 2019: Released small model (117M)
• May 2019: Medium model (345M)
• Aug 2019: Large model (762M)
• Nov 2019: Full model (1.5B)
• Concerns about misuse led to gradual release

Zero-Shot Task Transfer

The surprising finding: GPT-2 can perform tasks without fine-tuning!

1English: I love machine learning
2French:

1Answer the question:
2Q: What is the capital of France?
3A:

1[Long article text here]
2
3TL;DR:

Zero-Shot: How It Works

GPT-2 saw similar patterns during pre-training:

1Example from training data (hypothetical):
2
3"...The meeting was held in Berlin. The German chancellor...
4Later that day in Tokyo, Japanese officials...
5
6Quick Summary: Leaders from Germany and Japan met to discuss..."

At inference time:

1User prompt: "[Article about climate conference]
2
3TL;DR:"
4
5GPT-2 thinks: "I've seen 'TL;DR:' followed by summaries thousands
6of times in my training. I should output a summary here."

Key insight: Zero-shot works because the model learned task formats implicitly from diverse web text!

GPT-2 Performance

Zero-shot results on various benchmarks:

Text Generation Quality

Observations:

• Coherent and fluent
• Maintains context and style
• Completely fabricated "facts"
• No grounding in reality

GPT-3: The 175B Parameter Model

Model size comparison:

Key insight: GPT-3 is so large that new capabilities emerge that weren't present in smaller models!

Reference: Brown et al. (2020) - "Language Models are Few-Shot Learners"

GPT-3 Model Specifications

GPT-3 Training Data

Training corpus composition:

Key differences from GPT-2:

• Much larger and more diverse
• Includes Common Crawl (with quality filtering)
• Multiple passes over high-quality data
• Carefully balanced mixture

Few-Shot Learning

GPT-3's key capability: In-context learning

1Translate to French: I love AI ->

1sea otter -> loutre de mer
2I love AI ->

1dog -> chien
2cat -> chat
3bird -> oiseau
4I love AI ->

No gradient updates! Just prompt engineering.

Few-Shot: Worked Example

Sentiment Classification with 3 examples:

1prompt = """
2Classify the sentiment of each review as Positive or Negative.
3
4Review: "This movie was amazing, I loved every minute!"
5Sentiment: Positive
6
7Review: "Terrible film, complete waste of time."
8Sentiment: Negative
9
10Review: "A masterpiece of modern cinema."
11Sentiment: Positive
12
13Review: "The acting was wooden and the plot made no sense."
14Sentiment:"""
15
16# GPT-3 output: "Negative"

Why this works:

• Model recognizes the pattern from examples
• Applies same pattern to new input
• No weight updates needed!

Emergent Abilities: Concrete Examples

Arithmetic (emerges around 10B parameters):

1Q: What is 47 + 58?
2A: 105

Code Generation:

1# Write a function to check if a number is prime
2def is_prime(n):
3    if n < 2:
4        return False
5    for i in range(2, int(n**0.5) + 1):
6        if n % i == 0:
7            return False
8    return True

Multi-step Reasoning:

1Q: If I have 3 apples and give 2 to my friend,
2   then buy 4 more, how many do I have?
3A: 3 - 2 + 4 = 5 apples

The Scaling Law Formula

Loss as a function of scale:

where:

• = Cross-entropy loss
• = Number of parameters
• = Scaling constant
• (empirically determined)

In practice, this means:

Scaling Laws: Visual Understanding

How loss decreases with scale:

1Loss
2  ^
3  |
43.5|  *
5   |    *
63.0|      *
7   |        *
82.5|          *
9   |            *
102.0|              *  *  *  *  * (diminishing returns)
11   +------------------------------------------>
12       10M   100M   1B    10B   100B   1T
13                   Parameters

Key observation: Returns diminish but never stop - every 10x increase helps!

Implications of Scaling Laws

Chinchilla Scaling Laws

Key insight:

• For a given compute budget, should balance model size and data
• Optimal ratio: ~20 tokens per parameter

Comparison:

Result: Chinchilla (70B) outperforms GPT-3 (175B)!

This influenced Llama 2, GPT-4, and other modern models

The Path to ChatGPT

Evolution from GPT-3 to ChatGPT:

Three key innovations:

1. Instruction tuning: Train to follow instructions
2. RLHF: Reinforcement Learning from Human Feedback
3. Safety guardrails: Reduce harmful outputs

Instruction Tuning

Training examples:

1Instruction: "Explain quantum computing to a 5-year-old"
2Response: "Imagine you have a magic coin that can be heads
3          AND tails at the same time until you look at it..."
4
5Instruction: "Write a Python function to sort a list"
6Response: "def sort_list(items):
7              return sorted(items)"
8
9Instruction: "Summarize this article in 3 sentences"
10Response: [concise 3-sentence summary]

Before vs After Instruction Tuning

1User: "Write a haiku about AI"
2
3GPT-3: "Write a haiku about AI
4is a common creative writing
5exercise that many people
6enjoy. Here are some tips
7for writing haikus..."

1User: "Write a haiku about AI"
2
3InstructGPT: "Silicon neurons
4Learning patterns in the void
5Dreams in binary"

RLHF: Reinforcement Learning from Human Feedback

The 3-step RLHF process:

1Step 1: Supervised Fine-tuning (SFT)
2- Train on human-written examples of good responses
3- Model learns basic instruction-following
4
5Step 2: Reward Model Training
6- Generate multiple responses to same prompt
7- Humans rank responses from best to worst
8- Train a model to predict human preferences
9
10Step 3: RL Optimization (PPO)
11- Generate responses with policy model
12- Score with reward model
13- Update policy to maximize reward

RLHF: Worked Example

Training the reward model:

1Prompt: "How do I make a cake?"
2
3Response A (Rating: 4/5):
4"Here's a simple recipe: Preheat oven to 350F.
5Mix 2 cups flour, 1.5 cups sugar..."
6
7Response B (Rating: 2/5):
8"Cake is a type of dessert that originated in
9ancient civilizations..."
10
11Response C (Rating: 1/5):
12"I cannot help with that request."
13
14Reward model learns:
15- Helpful, direct answers get high scores
16- Off-topic or unhelpful responses get low scores

ChatGPT's Impact

Launched: November 30, 2022

Growth:

• 1 million users in 5 days
• 100 million users in 2 months
• Fastest-growing consumer application ever

Why so successful?

• Easy to use (conversational interface)
• Broadly capable (many tasks)
• Accessible (free tier)
• Impressive demos went viral
• Timing (post-pandemic digital adoption)

The Modern LLM Landscape

Post-GPT-3 developments (2020-2024):

Key trends:

1. Multimodal capabilities (vision, audio)
2. Longer context windows (100K+ tokens)
3. Better reasoning and factuality
4. Open-source alternatives
5. Efficiency improvements

Open Source LLMs

Practical Prompting Techniques

Effective prompting strategies for modern LLMs:

1. Zero-shot with clear instructions:

1Classify the following text as spam or not spam.
2Only respond with "spam" or "not spam".
3
4Text: "Congratulations! You've won $1,000,000!"

2. Few-shot with examples:

1Text: "Meeting at 3pm tomorrow" -> not spam
2Text: "URGENT: Send money now!" -> spam
3Text: "Can you review this document?" -> not spam
4Text: "You've been selected for a prize!" ->

Chain-of-Thought Prompting

For complex reasoning tasks:

1Q: Roger has 5 tennis balls. He buys 2 more cans of
2   tennis balls. Each can has 3 balls. How many
3   tennis balls does he have now?
4
5Let's think step by step:
61. Roger starts with 5 tennis balls
72. He buys 2 cans of tennis balls
83. Each can has 3 balls, so 2 cans = 2 * 3 = 6 balls
94. Total = 5 + 6 = 11 tennis balls
10
11A: 11 tennis balls

Current Limitations

What LLMs still struggle with:

1. Factual accuracy

   • Hallucinations and confabulation
   • No citations or sources

2. Reasoning

   • Multi-step logic
   • Mathematical proofs

3. Knowledge grounding

   • Knowledge cutoff date
   • Can't access real-time info

4. Personalization

   • No persistent memory
   • Stateless conversations

5. Reliability

   • Inconsistent outputs
   • Prompt sensitivity

Key Takeaways

1. Scaling works

   • GPT -> GPT-2 -> GPT-3 showed clear improvements
   • Power law scaling continues to hold

2. Few-shot learning emerged at scale

   • No fine-tuning needed for many tasks
   • In-context learning is powerful

3. Scaling laws provide predictability

   • But diminishing returns and compute costs are real
   • Chinchilla scaling: balance model size and data

4. RLHF changed everything

   • ChatGPT = GPT-3.5 + instruction tuning + RLHF
   • Alignment is crucial for deployment

5. The field is rapidly evolving

   • Open vs closed debate continues
   • New capabilities emerging

Next Lecture Preview

Questions?