Today's Lecture

1. The 2013 Revolution: Word2Vec
2. Word2Vec Architectures: CBOW & Skip-gram
3. GloVe: Global Vectors
4. FastText: Subword Information
5. Comparison & Evaluation
6. Applications & Best Practices

Goal: Understand how neural methods learn dense word representations

From Count-Based to Prediction-Based

Word2Vec: The Revolution

2013: Everything changed

Reference: Mikolov et al. (2013a). "Efficient Estimation of Word Representations in Vector Space"

Word2Vec: Core Intuition

Words that appear in similar contexts should have similar representations

Key Idea:

• Train a model to predict target from context (or vice versa)
• The learned weights become our word vectors!
• Similar words will have similar weight patterns

We don't actually care about the prediction task—we want the embeddings!

Context Windows: Worked Example

Sentence: "The cat sat on the mat"

Window size = 2 (2 words on each side)

1Position 0: "The"  → Context: [cat, sat]
2Position 1: "cat"  → Context: [The, sat, on]
3Position 2: "sat"  → Context: [The, cat, on, the]
4Position 3: "on"   → Context: [cat, sat, the, mat]
5Position 4: "the"  → Context: [sat, on, mat]
6Position 5: "mat"  → Context: [on, the]

Skip-gram training pairs (target → context):

1(The, cat), (The, sat)
2(cat, The), (cat, sat), (cat, on)
3(sat, The), (sat, cat), (sat, on), (sat, the)
4...

Result: Words appearing in similar contexts get similar vectors!

CBOW: Continuous Bag of Words

Predict the center word from context words

1Context words:
2  "the" ──┐
3  "cat" ──┼──→ Average ──→ "sat"
4  "on"  ──┼──→ Hidden  ──→ (predict)
5  "the" ──┘

Objective:

Skip-gram: The Inverse Approach

Predict context words from the center word

1Center word:           Predict context:
2                   ┌──→ "the"
3                   ├──→ "cat"
4  "sat" ──→ Hidden ┼──→ "on"
5                   └──→ "the"

Objective:

Negative Sampling: The Speed Trick

Problem: Softmax over entire vocabulary is too expensive!

For 100k vocabulary: Need to compute 100k exponentials per training example!

Solution: Negative Sampling

Instead of predicting across all words, create a binary classification:

• Positive sample: Actual context word (label = 1)
• Negative samples: K random words (label = 0)

Typical K: 5-20 for large datasets, 2-5 for small datasets

Training becomes instead of per example!

Reference: Mikolov et al. (2013b). "Distributed Representations of Words and Phrases"

Negative Sampling: Concrete Example

Training pair: ("cat", "sat") - cat is center, sat is context

1# Positive sample: Does "sat" appear near "cat"? YES (label=1)
2positive_pair = ("cat", "sat", label=1)
3
4# Negative samples: Random words that DON'T appear near "cat"
5# Sample 5 random words from vocabulary
6negative_pairs = [
7    ("cat", "algorithm", label=0),
8    ("cat", "president", label=0),
9    ("cat", "quantum", label=0),
10    ("cat", "democracy", label=0),
11    ("cat", "software", label=0),
12]
13
14# Train binary classifier: Is this a real context pair?
15# Instead of 100k-way softmax → 6 binary predictions!

Sampling distribution:

The 0.75 power gives rare words slightly higher probability of being sampled as negatives.

Word2Vec in Practice

21# Train your own model
22sentences = [
23    ['the', 'cat', 'sat', 'on', 'the', 'mat'],
24    ['the', 'dog', 'ran', 'in', 'the', 'park'],
25    # ... more sentences
26]
27
28custom_model = Word2Vec(
29    sentences,
30    vector_size=100,    # embedding dimension
31    window=5,           # context window
32    min_count=1,        # ignore rare words
33    sg=1,               # 1=skip-gram, 0=CBOW
34    negative=5,         # negative sampling
35    workers=4           # parallel threads
36)

GloVe: Global Vectors for Word Representation

Combining the best of count-based and prediction-based methods

Reference: Pennington et al. (2014). "GloVe: Global Vectors for Word Representation"

GloVe: The Ratio Intuition

Why ratios matter more than raw probabilities:

1Given words: "ice" and "steam"
2Probe with: "solid", "gas", "water"
3
4P(solid | ice)  = 0.00019    P(solid | steam) = 0.000022
5P(gas | ice)    = 0.000066   P(gas | steam)   = 0.00078
6
7Ratio: P(solid|ice) / P(solid|steam) = 8.9   → "solid" relates to ice
8Ratio: P(gas|ice)   / P(gas|steam)   = 0.085 → "gas" relates to steam
9Ratio: P(water|ice) / P(water|steam) = 1.36  → "water" is neutral

The insight: These ratios distinguish relevant context words from irrelevant ones!

GloVe learns vectors where:

GloVe: The Objective Function

Goal: Learn vectors that capture co-occurrence statistics

Objective Function:

where:

• = number of times word appears in context of word
• = word vector for word
• = separate context vector for word
• = bias terms
• = weighting function

Weighting Function:

Purpose: Prevent very common co-occurrences from dominating (typically , )

GloVe in Practice

21# Vector arithmetic
22king = glove['king']
23man = glove['man']
24woman = glove['woman']
25result_vec = king - man + woman
26
27# Find closest word
28closest = glove.similar_by_vector(result_vec, topn=1)
29print(closest)  # Should be close to 'queen'
30
31# Compute similarity
32similarity = glove.similarity('cat', 'dog')
33print(f"Similarity: {similarity:.3f}")

FastText: Enriching with Subword Information

The Problem with Word2Vec and GloVe:

Example:

• Have: "running", "runner"
• Need: "runnable"
• But these words share morphology! ("run" + suffix)

FastText Solution: Represent words as

Reference: Bojanowski et al. (2017). "Enriching Word Vectors with Subword Information"

FastText: Character N-grams

Key Idea: Break words into character n-grams

FastText: Morphology in Action

How FastText handles related words:

1# Word: "unhappiness" broken into character 3-grams:
2# <un, unh, nha, hap, app, ppi, pin, ine, nes, ess, ss>
3
4# Shares n-grams with:
5# "unhappy"   → <un, unh, nha, hap, app, ppy
6# "happiness" → hap, app, ppi, pin, ine, nes, ess, ss>
7# "happy"     → hap, app, ppy
8
9# Therefore: vec("unhappiness") is close to:
10#   - vec("unhappy")     (prefix overlap)
11#   - vec("happiness")   (suffix overlap)
12#   - vec("sadness")     (similar suffix pattern)

This is why FastText excels at morphologically rich languages!

FastText in Practice

21    sentences,
22    vector_size=100,
23    window=5,
24    min_count=1,
25    min_n=3,      # minimum n-gram length
26    max_n=6,      # maximum n-gram length
27    word_ngrams=1 # use word + ngrams
28)
29
30# Check similar words
31similar = ft_model.wv.most_similar('cat', topn=5)
32
33# Morphological awareness
34# 'running', 'runner', 'runnable' will be close

Embedding Methods Comparison

Typical dimensions: 50-300 (100-300 most common)

Bias Detection: Code Example

1import gensim.downloader as api
2model = api.load('word2vec-google-news-300')
3
4# Measure gender bias: which words are closer to "man" vs "woman"?
5def gender_bias_score(word):
6    """Positive = closer to man, Negative = closer to woman"""
7    return model.similarity(word, 'man') - model.similarity(word, 'woman')
8
9occupations = ['doctor', 'nurse', 'engineer', 'teacher',
10               'programmer', 'secretary', 'scientist', 'receptionist']
11
12for job in occupations:
13    score = gender_bias_score(job)
14    direction = "→ man" if score > 0 else "→ woman"
15    print(f"{job:12}: {score:+.3f} {direction}")
16
17# Output:
18# doctor      : +0.089 → man
19# nurse       : -0.109 → woman
20# engineer    : +0.078 → man

Bias Detection: Code Example

21# teacher     : -0.046 → woman
22# programmer  : +0.091 → man
23# secretary   : -0.107 → woman

These biases reflect stereotypes in the training data (news articles)!

Loading Pre-trained Embeddings: Quick Reference

1import gensim.downloader as api
2
3# Word2Vec (Google News, 3B words, 300d)
4w2v = api.load('word2vec-google-news-300')
5
6# GloVe options
7glove_50d = api.load('glove-wiki-gigaword-50')      # Small, fast
8glove_100d = api.load('glove-wiki-gigaword-100')    # Balanced
9glove_300d = api.load('glove-wiki-gigaword-300')    # Best quality
10glove_twitter = api.load('glove-twitter-100')       # Social media
11
12# FastText (handles OOV!)
13fasttext = api.load('fasttext-wiki-news-subwords-300')
14
15# Basic usage (same for all)
16similar = model.most_similar('computer', topn=5)
17vector = model['cat']
18similarity = model.similarity('dog', 'cat')

Pro tip: Start with glove-wiki-gigaword-100 for a good balance of speed and quality!

Discussion Question

Why do word analogies work so well?

The famous example:

Think about:

• What does vector subtraction represent linguistically?
• Why does this capture semantic relationships?
• What are the limitations of this approach?
• Does this mean embeddings "understand" gender?

Reference: Boleda (2020). "Distributional Semantics and Linguistic Theory"

Summary

What we learned today:

1. Word2Vec (2013): Neural revolution in embeddings
   • CBOW: Predict center from context

• Skip-gram: Predict context from center
• Negative sampling for efficiency

2. GloVe (2014): Combining count + prediction
   • Global co-occurrence statistics

• Factorization objective
• Ratio of probabilities

3. FastText (2017): Subword information
   • Character n-grams

• Handles OOV words
• Captures morphology

4. Evaluation: Intrinsic vs. extrinsic
5. Limitations: Static, biased, no polysemy
6. Next: Contextual embeddings solve polysemy!

Key References

Foundational Papers:

• Mikolov et al. (2013a). "Efficient Estimation of Word Representations in Vector Space"
• Mikolov et al. (2013b). "Distributed Representations of Words and Phrases and their Compositionality"
• Pennington et al. (2014). "GloVe: Global Vectors for Word Representation"
• Bojanowski et al. (2017). "Enriching Word Vectors with Subword Information"

Bias & Ethics:

• Bolukbasi et al. (2016). "Man is to Computer Programmer as Woman is to Homemaker?"
• Caliskan et al. (2017). "Semantics derived automatically from language corpora contain human-like biases"

Theory:

• Boleda, G. (2020). "Distributional Semantics and Linguistic Theory"
• Levy & Goldberg (2014). "Neural Word Embedding as Implicit Matrix Factorization"

Tools:

• Gensim: https://radimrehurek.com/gensim/
• Pre-trained vectors: https://github.com/RaRe-Technologies/gensim-data

Questions?

Next Lecture:

Contextual Embeddings: ELMo, Universal Sentence Encoder, BERT

One word, multiple meanings!