Today's Lecture

1. From Static to Contextual
2. Language Models as Feature Extractors
3. ELMo: Embeddings from Language Models
4. Universal Sentence Encoder
5. BERT: Bidirectional Transformers
6. Comparison & Applications

Goal: Understand how context transforms word representation

The Polysemy Problem Revisited

Recall: Static embeddings assign ONE vector per word

1# Static embeddings (Word2Vec)
2sim(bank_sent1, bank_sent2) = 1.0  # Same!
3
4# Contextual embeddings (BERT)
5sim(bank_sent1, bank_sent2) = 0.45
6sim(bank_sent1, bank_sent3) = 0.32
7sim(bank_sent2, bank_sent3) = 0.28
8
9# "bank" (financial) is closer to
10# "bank" (river) than to "bank" (tilt)
11# because nouns are more similar!

Language Models as Feature Extractors

Key Insight: Train a language model, use its internal states as embeddings

Language Modeling Task:

Predict the next word given previous words:

Why This Works:

• To predict "mat", model must encode that "sat on" precedes surfaces
• Hidden states capture this contextual understanding
• We extract these rich hidden states as embeddings!

ELMo: Embeddings from Language Models

The first widely-adopted contextual embedding (2018)

1Forward:  The → cat → sat → ...
2Backward: ... ← sat ← cat ← The

Reference: Peters et al. (2018). "Deep contextualized word representations"

ELMo: How It Works

Training:

1. Pre-train on large corpus (1B Word Benchmark)
2. Each position gets representation from all layers

Usage (downstream tasks):

1. Freeze ELMo weights
2. For each token, extract representations from all layers
3. Learn task-specific weighted combination
4. Concatenate with task model

Impact: Improved state-of-the-art on 6 NLP tasks!

ELMo in Practice

21# embeddings['elmo_representations'] contains:
22# - List of 2 tensors (one per layer)
23# - Shape: [batch_size, seq_len, 1024]
24
25# Different vectors for "bank"!
26bank1 = embeddings['elmo_representations'][0][0, 5, :]  # first sentence
27bank2 = embeddings['elmo_representations'][0][1, 5, :]  # second sentence
28
29# Cosine similarity will be lower than for static embeddings

Universal Sentence Encoder in Practice

21# Compute similarity
22from sklearn.metrics.pairwise import cosine_similarity
23
24sim_matrix = cosine_similarity(embeddings)
25print(sim_matrix)
26
27# Sentences 1 and 2 should be very similar (paraphrases)
28# Sentence 3 somewhat similar (animals)
29# Sentence 4 dissimilar
30
31# Use for semantic search
32query = "cat on mat"
33query_embedding = embed([query])
34similarities = cosine_similarity(query_embedding, embeddings)[0]
35most_similar_idx = np.argmax(similarities)
36print(f"Most similar: {sentences[most_similar_idx]}")

BERT: Bidirectional Encoder Representations

The model that changed everything (2018)

Reference: Devlin et al. (2018). "BERT: Pre-training of Deep Bidirectional Transformers"

BERT Architecture

• Token Embedding: What word is this?
• Segment Embedding: Which sentence (A or B)?
• Position Embedding: Where in the sequence?

BERT in Practice

21# Last hidden state: [batch_size, seq_len, hidden_size]
22embeddings1 = output1.last_hidden_state
23embeddings2 = output2.last_hidden_state
24
25# Extract "bank" embedding (position varies)
26# tokens1: [CLS] i deposited money at the bank [SEP]
27bank1_embedding = embeddings1[0, 6, :]  # 768-dim vector
28
29# tokens2: [CLS] we sat by the river bank [SEP]
30bank2_embedding = embeddings2[0, 6, :]  # 768-dim vector
31
32# Different vectors for "bank"!
33from torch.nn.functional import cosine_similarity
34sim = cosine_similarity(bank1_embedding, bank2_embedding, dim=0)
35print(f"Similarity: {sim:.3f}")  # Lower than with static embeddings

Summary

What we learned today:

1. Contextual vs. Static: Different vectors per occurrence
2. ELMo (2018):
   • BiLSTM language models

• Character-based, handles OOV
• Task-specific weighting

3. Universal Sentence Encoder (2018):
   • Sentence-level embeddings

• Two variants: Transformer & DAN
• Optimized for semantic similarity

4. BERT (2018):
   • Masked language modeling

• Deep bidirectional transformers
• Pre-train + fine-tune paradigm
• Revolutionized NLP

5. Impact: Established modern transfer learning in NLP
6. Next: Dimensionality reduction techniques for visualization

Key References

Foundational Papers:

• Peters et al. (2018). "Deep contextualized word representations" (ELMo)
• Cer et al. (2018). "Universal Sentence Encoder"
• Devlin et al. (2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

BERT Variants:

• Liu et al. (2019). "RoBERTa: A Robustly Optimized BERT Pretraining Approach"
• Sanh et al. (2019). "DistilBERT, a distilled version of BERT"
• Lan et al. (2019). "ALBERT: A Lite BERT for Self-supervised Learning"

Critical Perspectives:

• Bender & Koller (2020). "Climbing towards NLU: On Meaning, Form, and Understanding"
• Bender et al. (2021). "On the Dangers of Stochastic Parrots"

Resources:

• HuggingFace Transformers: https://huggingface.co/transformers/
• BERT Paper: https://arxiv.org/abs/1810.04805

Questions?

Next Lecture:

Dimensionality Reduction: PCA, t-SNE, UMAP

Visualizing high-dimensional embeddings!