Today's Agenda

1. BERT Introduction: What makes it special?
2. Masked Language Modeling: The key training objective
3. BERT Architecture: Model sizes and specifications
4. Pre-training & Fine-tuning: The two-stage paradigm
5. Contextual Embeddings: Seeing polysemy in action
6. Using BERT: Practical code examples

Goal: Deep understanding of BERT and how it revolutionized NLP

BERT: Bidirectional Encoder Representations

BERT = Encoder-only Transformer

Key Innovation: Masked Language Modeling (MLM)

Reference: Devlin et al. (2019) - "BERT: Pre-training of Deep Bidirectional Transformers"

Why BERT Was Revolutionary

Before BERT (2018):

• Feature-based approaches (use Word2Vec/GloVe as features)
• Task-specific architectures
• Limited transfer learning
• Unidirectional or shallow bidirectional models

BERT's Contributions:

1. Deep Bidirectionality
   • True bidirectional context at every layer

• Not just concatenating left-to-right and right-to-left

2. Pre-train + Fine-tune Paradigm
   • Single pre-trained model for all tasks

• Fine-tune with minimal architecture changes
• Democratized NLP (no need to train from scratch!)

3. State-of-the-Art Results
   • Beat previous best on 11 NLP tasks

• Large performance gains (sometimes 10+ points!)
• Showed power of pre-training

Masked Language Modeling (MLM)

BERT's Pre-training Objective

Training Procedure:

1. Take a sentence
2. Randomly mask 15% of tokens
3. Of the masked tokens:
   • 80%: Replace with [MASK]

• 10%: Replace with random word
• 10%: Keep unchanged

4. Predict the original tokens

Why the 80/10/10 split?

• Prevents overfitting to [MASK] token
• Forces model to maintain representations for all tokens

MLM Example: Step by Step

Sentence: "The quick brown fox jumps over the lazy dog"

1tokens = ["The", "quick", "brown", "fox",
2          "jumps", "over", "the", "lazy", "dog"]
3# Randomly select: "quick" (idx 1), "over" (idx 5)

1"quick" → 80% → [MASK]
2"over"  → 10% → "under" (random)

1input:  "The [MASK] brown fox jumps
2         under the lazy dog"
3labels: [-1, "quick", -1, -1, -1,
4         "over", -1, -1, -1]
5# -1 = no loss computed

1P("quick" | context) → high (adjective slot)
2P("over" | context)  → high (preposition slot)

Key insight: Model must understand syntax AND semantics to predict masked words!

BERT Architecture Variants

Architecture Details (BERT-Base):

• 12 transformer encoder layers
• 768-dimensional hidden states
• 12 attention heads per layer (64 dims each)
• 3072-dimensional feed-forward intermediate size (4x expansion)
• Maximum sequence length: 512 tokens
• Vocabulary size: 30,000 WordPiece tokens

Special Tokens:

• [CLS]: Classification token (first token, used for sequence-level tasks)
• [SEP]: Separator token (between sentences)
• [MASK]: Mask token (for MLM)
• [PAD]: Padding token (for variable-length sequences)

BERT Input Representation

Three types of embeddings are summed:

1# Example: Sentence pair for NSP
2sentence_a = "My dog is cute"
3sentence_b = "He likes playing"
4
5# Tokenization
6tokens = ["[CLS]", "my", "dog", "is", "cute", "[SEP]", "he", "likes", "playing", "[SEP]"]
7
8# Three embedding types (each is a 768-dim vector):
9token_emb   = [E_CLS, E_my, E_dog, E_is, E_cute, E_SEP, E_he, E_likes, E_playing, E_SEP]
10segment_emb = [E_A,   E_A,  E_A,   E_A,  E_A,    E_A,   E_B,  E_B,     E_B,       E_B   ]
11position_emb= [E_0,   E_1,  E_2,   E_3,  E_4,    E_5,   E_6,  E_7,     E_8,       E_9   ]
12
13# Final input = token + segment + position (element-wise sum)
14input_embedding = token_emb + segment_emb + position_emb

Three embedding types:

1. Token Embeddings: WordPiece vocabulary (30K learned vectors)
2. Segment Embeddings: Which sentence (A or B)? (2 learned vectors)
3. Position Embeddings: Learned position 0-511 (512 learned vectors)

BERT Pre-training

Massive scale pre-training on unlabeled text

Pre-training Data:

• BooksCorpus: 800M words (novels, fiction)
• English Wikipedia: 2,500M words
• Total: 3.3 billion words
• Diverse, high-quality text

Training Details:

• Batch size: 256 sequences (128,000 tokens)
• Training steps: 1M steps
• Optimization: Adam (lr=1e-4, warmup=10k steps)
• Hardware: 4-16 Cloud TPUs
• Training time: 4 days (BERT-Base), 4+ days (BERT-Large)

Fine-tuning BERT

Two-stage process: Pre-train then Fine-tune

1# Expensive: weeks on TPUs
2# Data: 3.3B words (books + Wikipedia)
3# Task: MLM + NSP
4# Result: General language understanding
5
6model = pretrain_bert(
7    data=["BooksCorpus", "Wikipedia"],
8    steps=1_000_000,
9    hardware="16 TPUs"
10)

1# Cheap: hours on single GPU
2# Data: 1K-100K labeled examples
3# Task: Your specific task
4# Result: Task-specific model
5
6model = load_pretrained("bert-base")
7model.add_classifier(num_labels=2)
8model.train(
9    task_data,
10    epochs=3,
11    lr=2e-5  # Small learning rate!
12)

Benefits: Pre-training captures language; fine-tuning adapts to your task

Fine-tuning for Different Tasks

Minimal architecture changes needed!

1. Single Sentence Classification
   • Input: [CLS] sentence [SEP]

• Output: [CLS] representation → classifier
• Example: Sentiment analysis

2. Sentence Pair Classification
   • Input: [CLS] sentence A [SEP] sentence B [SEP]

• Output: [CLS] representation → classifier
• Example: Natural Language Inference

3. Question Answering
   • Input: [CLS] question [SEP] passage [SEP]

• Output: Token-level predictions for start/end positions
• Example: SQuAD

4. Token Classification
   • Input: [CLS] sentence [SEP]

• Output: Each token representation → classifier
• Example: Named Entity Recognition

Fine-tuning BERT: Code Example

Using HuggingFace Transformers

1from transformers import BertForSequenceClassification, Trainer, TrainingArguments
2
3# Load pre-trained BERT with classification head
4model = BertForSequenceClassification.from_pretrained(
5    'bert-base-uncased',
6    num_labels=2  # Binary classification
7)
8
9# Define training arguments
10training_args = TrainingArguments(
11    output_dir='./results',
12    num_train_epochs=3,
13    per_device_train_batch_size=16,
14    learning_rate=2e-5,
15    warmup_steps=500,
16)
17
18# Train
19trainer = Trainer(
20    model=model,

Fine-tuning BERT: Code Example

21    args=training_args,
22    train_dataset=train_dataset,
23    eval_dataset=eval_dataset,
24)
25
26trainer.train()

Reference: HuggingFace Course - Chapters 1.5, 7.3

Contextual Embeddings in Action

Remember "bank"? Let's see BERT handle it!

1from transformers import BertTokenizer, BertModel
2import torch
3
4tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
5model = BertModel.from_pretrained('bert-base-uncased')
6
7# Two different contexts for "bank"
8sent1 = "I deposited money at the bank"
9sent2 = "We sat by the river bank"
10
11# Get embeddings
12def get_embedding(sentence, target_word):
13    inputs = tokenizer(sentence, return_tensors='pt')
14    outputs = model(**inputs)
15    # Find position of target word
16    tokens = tokenizer.tokenize(sentence)
17    idx = tokens.index(target_word) + 1  # +1 for [CLS]
18    return outputs.last_hidden_state[0, idx, :]
19
20emb1 = get_embedding(sent1, "bank")  # Financial bank

Contextual Embeddings in Action

21emb2 = get_embedding(sent2, "bank")  # River bank
22
23# Compare similarity
24similarity = torch.cosine_similarity(emb1, emb2, dim=0)
25print(f"Similarity: {similarity:.3f}")  # Low! (~0.3-0.5)
26# Different contexts → Different embeddings!

BERT's Impressive Results

State-of-the-art on 11 NLP tasks when released (2018)

Key Observations:

• Largest gains on tasks requiring understanding (QA, NLI)
• Improvements even on well-studied benchmarks
• BERT-Large generally better than BERT-Base
• Fine-tuning is simple but very effective

What Does BERT Learn?

Probing BERT's internal representations

Research has shown BERT captures:

1. Syntactic Information
   • Part-of-speech tags

• Constituent structure
• Dependency relations
• Lower layers encode more syntax

2. Semantic Information
   • Word sense disambiguation

• Semantic roles
• Entity types
• Middle layers encode more semantics

3. Pragmatic Information
   • Coreference resolution

• Discourse relations
• Higher layers encode more pragmatics

4. World Knowledge
   • Factual knowledge (to some extent)

• Common sense reasoning (limited)

Reference: Tenney et al. (2019) - "BERT Rediscovers the Classical NLP Pipeline"

BERT Layer Analysis

Different layers capture different linguistic properties

1# Probing experiment: Train linear classifiers on each layer's representations
2from transformers import BertModel
3import numpy as np
4
5model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
6
7# Get hidden states for all 12 layers
8outputs = model(**inputs)
9hidden_states = outputs.hidden_states  # (13 layers: embedding + 12 transformer)
10
11# Results from probing studies (Tenney et al., 2019):
12layer_specialization = {
13    "Layers 0-2":  ["POS tagging", "Word boundaries"],     # Surface
14    "Layers 3-6":  ["Parse trees", "Dependencies"],        # Syntax
15    "Layers 7-9":  ["Semantic roles", "Coreference"],      # Semantics
16    "Layers 10-12": ["Task-specific representations"]       # Task
17}

Observations:

• Lower layers: surface features (word forms, POS)
• Middle layers: syntax (phrase structure, dependencies)
• Higher layers: semantics and task-specific features

Similar to CNNs: edges → shapes → objects

Discussion Questions

1. MLM vs. Autoregressive:
   • Why is MLM better for understanding tasks?

• Can BERT generate text like GPT?
• What are the trade-offs?

2. The 80/10/10 Masking Strategy:
   • Why not just use 100% [MASK]?

• What problem does the random replacement solve?
• Could we improve this strategy?

3. Pre-training Data:
   • Why use books and Wikipedia?

• Would social media text work as well?
• How does data quality affect pre-training?

4. Fine-tuning:
   • Why does fine-tuning work so well?

• When might fine-tuning fail?
• How much labeled data do we need?

Looking Ahead

Today we learned:

• BERT architecture and innovations
• Masked Language Modeling
• Pre-training and fine-tuning
• Contextual embeddings
• What BERT learns

Next lecture (Lecture 16 - BERT Variants):

• : Optimized BERT training
• : Parameter-efficient BERT
• : Smaller, faster BERT
• : ELECTRA, DeBERTa, and more
• Comparative analysis and when to use which

BERT started a revolution in NLP!

Summary

Key Takeaways:

1. BERT = Encoder-only Transformer
   • Bidirectional self-attention

• Trained with Masked Language Modeling

2. Pre-train + Fine-tune Paradigm
   • Expensive pre-training on unlabeled data (once)

• Cheap fine-tuning on task-specific data (per task)

3. Contextual Embeddings
   • Different representations based on context

• Solves polysemy problem

4. Hierarchical Learning
   • Lower layers: syntax

• Higher layers: semantics
• Learns linguistic structure automatically

5. Revolutionary Impact
   • Established pre-training as standard

• Democratized NLP research
• Foundation for modern LLMs

References

Essential Papers:

• Devlin et al. (2019) - "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

• The original BERT paper

• Introduced MLM and NSP

  \item Tenney et al. (2019) - "BERT Rediscovers the Classical NLP Pipeline"

  • Analysis of what BERT learns

• Layer-wise linguistic properties

  \item Clark et al. (2019) - "What Does BERT Look At? An Analysis of BERT's Attention"

  • Understanding BERT's attention patterns

Tutorials:

• HuggingFace Course: Chapter 1 (Transformer Models)
• Jay Alammar: "The Illustrated BERT, ELMo, and co."
• BertViz: Interactive attention visualization

Questions?

Discussion Time

Topics for discussion:

• Masked Language Modeling
• Pre-training vs fine-tuning
• Contextual embeddings
• BERT architecture details
• Implementation questions

Thank you!

Next: BERT Variants and Improvements!