From count-based to prediction-based embeddings

Word2Vec: words that appear in similar contexts should have similar embeddings

CBOW: Continuous Bag of Words

Skip-gram

Context windows: worked example

Sentence: "The cat sat on the mat" | Window size = 2

1Position 0: "The" → Context: [cat, sat]
2Position 1: "cat" → Context: [The, sat, on]
3Position 2: "sat" → Context: [The, cat, on, the]
4Position 3: "on"  → Context: [cat, sat, the, mat]
5Position 4: "the" → Context: [sat, on, mat]
6Position 5: "mat" → Context: [on, the]

Skip-gram training pairs (target → context):

1(The, cat), (The, sat), (cat, The), (cat, sat), (cat, on), ...

Negative sampling example

Training pair: ("cat", "sat") — cat is center, sat is context

1# Positive sample: Does "sat" appear near "cat"? YES (label=1)
2positive_pair = ("cat", "sat", label=1)
3
4# Negative samples: Random words that DON'T appear near "cat"
5negative_pairs = [
6    ("cat", "algorithm", label=0),
7    ("cat", "president", label=0),
8    ("cat", "quantum", label=0),
9]
10
11# Train binary classifier: Is this a real context pair?
12# Instead of 100k-way softmax → just 4 binary predictions!

The famous result: vector arithmetic

$$\overrightarrow{\text{King}} - \overrightarrow{\text{Man}} + \overrightarrow{\text{Woman}} \approx \overrightarrow{\text{Queen}}$$

Word2Vec in Python

1from gensim.models import Word2Vec
2import gensim.downloader as api
3
4# Load pre-trained model (3B words, 300 dimensions)
5model = api.load('word2vec-google-news-300')
6
7# Find similar words
8model.most_similar('computer', topn=5)
9# [('computers', 0.72), ('laptop', 0.69), ('PC', 0.68), ...]
10
11# Word analogies: king - man + woman = ?
12model.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
13# [('queen', 0.71)]
14
15# Train your own
16sentences = [['the', 'cat', 'sat'], ['the', 'dog', 'ran']]
17custom_model = Word2Vec(sentences, vector_size=100, window=5, min_count=1, sg=1)

How does GloVe work?

FastText: leveraging subword information

FastText: character n-grams

Estimating bias

1def gender_bias_score(word):
2    return model.similarity(word, 'man') - model.similarity(word, 'woman')
3
4# Results from Google News Word2Vec:
5# programmer: +0.091 (toward man)
6# nurse: -0.109 (toward woman)