Today's Lecture

1. The Distributional Hypothesis
2. Vector Space Models
3. Latent Semantic Analysis (LSA)
4. Latent Dirichlet Allocation (LDA)
5. Modern Topic Modeling: BERTopic
6. Evaluation & Applications

Goal: Understand how classic methods represent meaning through co-occurrence

The Fundamental Question

How do we represent the meaning of words in a way that computers can understand?

\pause

The Distributional Hypothesis

"You shall know a word by the company it keeps"

--- J.R. Firth (1957)

\pause

Key Idea:
Words that appear in similar contexts tend to have similar meanings.

Reference: Firth, J.R. (1957). "A synopsis of linguistic theory 1930-1955"

From Words to Vectors

Goal: Represent each word as a point in high-dimensional space

Problem: Real vocabularies have 10,000+ words, contexts are even more numerous!

We need dimensionality reduction

Building a Term-Document Matrix

Step 1: Count how often each word appears in each document

Issues:

• Very sparse (mostly zeros)
• High dimensional (vocab size × doc count)
• Doesn't capture semantic relationships

\pause

Apply TF-IDF weighting and dimensionality reduction!

TF-IDF Weighting

Term Frequency - Inverse Document Frequency

TF-IDF: Worked Example

Corpus: 3 documents, 1000 total documents in collection

Calculate TF-IDF for "learning" in Doc 1:

1Step 1: TF("learning", Doc1) = 1/4 = 0.25  (1 occurrence, 4 words)
2
3Step 2: IDF("learning") = log(1000/500) = log(2) = 0.301
4        (appears in 500 of 1000 docs)
5
6Step 3: TF-IDF = 0.25 × 0.301 = 0.075

Compare: "the" (appears in 950 docs):

1IDF("the") = log(1000/950) = 0.022  ← Much lower! Common words penalized

Latent Semantic Analysis (LSA)

The OG of semantic embeddings (1990)

1X        =   U    ×    Σ    ×    V^T
2(m×n)      (m×k)    (k×k)     (k×n)
3
4words×docs  words×  topic    topics×
5            topics  strength  docs

Reference: Deerwester et al. (1990) - "Indexing by Latent Semantic Analysis"

LSA: Step-by-Step Worked Example

Mini corpus (4 words × 3 documents):

1           Doc1    Doc2    Doc3
2cat         2       0       1
3dog         0       3       1
4pet         1       2       0
5animal      1       1       1

Step 1: Apply SVD

Step 2: Keep top 2 dimensions (k=2)

1from sklearn.decomposition import TruncatedSVD
2import numpy as np
3
4X = np.array([[2,0,1], [0,3,1], [1,2,0], [1,1,1]])
5svd = TruncatedSVD(n_components=2)
6word_embeddings = svd.fit_transform(X)
7
8print("Word vectors (2D):")
9print(f"cat:    [{word_embeddings[0,0]:.2f}, {word_embeddings[0,1]:.2f}]")
10print(f"dog:    [{word_embeddings[1,0]:.2f}, {word_embeddings[1,1]:.2f}]")
11print(f"pet:    [{word_embeddings[2,0]:.2f}, {word_embeddings[2,1]:.2f}]")
12print(f"animal: [{word_embeddings[3,0]:.2f}, {word_embeddings[3,1]:.2f}]")

Result: "pet" and "animal" end up close together in 2D space!

Latent Dirichlet Allocation (LDA)

A probabilistic approach to topic modeling

What if we model documents as probabilistic mixtures of topics?

1Topic 1 -> Topic 2 -> Topic 3 -> sports, game -> tech, code -> animal, pet

Reference: Blei et al. (2003) - "Latent Dirichlet Allocation"

LDA: Concrete Generative Example

Imagine generating a document about "tech pets":

1Step 1: Pick topic mixture for this document
2        θ_doc = [0.6 Tech, 0.3 Animals, 0.1 Sports]
3
4Step 2: For each word, sample a topic, then sample a word:
5
6Word 1: Sample topic → Tech (60% chance)
7        Sample word from Tech → "software"
8
9Word 2: Sample topic → Animals (30% chance)
10        Sample word from Animals → "cat"
11
12Word 3: Sample topic → Tech (60% chance)
13        Sample word from Tech → "computer"
14
15Word 4: Sample topic → Animals (30% chance)
16        Sample word from Animals → "pet"
17
18Result: "software cat computer pet"

LDA reverses this: Given documents, infer the topics!

LDA in Practice

21    max_iter=50
22)
23
24doc_topics = lda.fit_transform(doc_term_matrix)
25
26# Print top words per topic
27for idx, topic in enumerate(lda.components_):
28    top_words = [vocab[i] for i in topic.argsort()[-10:]]
29    print(f"Topic {idx}: {', '.join(top_words)}")

Modern Topic Modeling: BERTopic

21# Get topic info
22topic_info = model.get_topic_info()
23
24# Visualize
25fig = model.visualize_topics()

Reference: Grootendorst (2022) - "BERTopic: Neural topic modeling with a class-based TF-IDF procedure"

Coherence Evaluation: Code Example

1from gensim.models import LdaModel
2from gensim.models.coherencemodel import CoherenceModel
3from gensim.corpora import Dictionary
4
5# Prepare corpus
6texts = [doc.split() for doc in documents]
7dictionary = Dictionary(texts)
8corpus = [dictionary.doc2bow(text) for text in texts]
9
10# Train LDA with different topic numbers
11coherence_scores = []
12for num_topics in [5, 10, 15, 20, 25]:
13    lda = LdaModel(corpus, num_topics=num_topics,
14                   id2word=dictionary, passes=10)
15
16    # Calculate coherence (C_V is recommended)
17    coherence = CoherenceModel(model=lda, texts=texts,
18                               dictionary=dictionary,
19                               coherence='c_v')
20    coherence_scores.append((num_topics, coherence.get_coherence()))

Coherence Evaluation: Code Example

21
22# Results: [(5, 0.42), (10, 0.51), (15, 0.48), (20, 0.45), (25, 0.41)]
23# Best: 10 topics with coherence 0.51

Rule of thumb: Higher coherence = more interpretable topics

Applications of Classic Embeddings

1. Information Retrieval
   • Semantic search
   • Document similarity
   • Query expansion
2. Document Organization
   • Clustering
   • Topic discovery
   • Trend analysis
3. Text Mining
   • Opinion mining
   • Literature review
   • Knowledge discovery
4. Preprocessing
   • Feature reduction for ML
   • Noise reduction
   • Transfer learning

Real-world examples:

• Academic paper recommendations (LSA)
• News article categorization (LDA)
• Social media trend detection (BERTopic)

Application Example: Semantic Search with LSA

1from sklearn.decomposition import TruncatedSVD
2from sklearn.metrics.pairwise import cosine_similarity
3
4# Documents already vectorized with TF-IDF
5# tfidf_matrix shape: (1000 docs, 5000 words)
6
7# Apply LSA to reduce dimensions
8lsa = TruncatedSVD(n_components=100)
9doc_embeddings = lsa.fit_transform(tfidf_matrix)
10
11# User query: "machine learning algorithms"
12query_tfidf = vectorizer.transform(["machine learning algorithms"])
13query_embedding = lsa.transform(query_tfidf)
14
15# Find most similar documents
16similarities = cosine_similarity(query_embedding, doc_embeddings)[0]
17top_5_docs = similarities.argsort()[-5:][::-1]
18
19print("Most relevant documents:")
20for idx in top_5_docs:

Application Example: Semantic Search with LSA

21    print(f"  Doc {idx}: similarity = {similarities[idx]:.3f}")

Key insight: LSA finds documents about "neural networks" and "deep learning" even though those exact words weren't in the query!

Discussion Question

Can distributional semantics truly capture meaning?

Reference: Harnad, S. (1990). "The symbol grounding problem"

Summary

What we learned today:

1. Distributional Hypothesis: Words in similar contexts have similar meanings
2. LSA (1990):
   • SVD on term-document matrix

• Linear dimensionality reduction
• Fast but limited interpretability

3. LDA (2003):
   • Probabilistic topic modeling

• Documents as mixtures of topics
• Highly interpretable

4. BERTopic (2022):
   • Neural embeddings + clustering

• State-of-the-art coherence
• Automatic topic detection

5. Evaluation: Coherence perplexity
6. Limitation: Static representations, no grounding

Next: Neural word embeddings (Word2Vec, GloVe, FastText)!

Key References

Classic Papers:

• Firth, J.R. (1957). "A synopsis of linguistic theory 1930-1955"
• Deerwester et al. (1990). "Indexing by Latent Semantic Analysis"
• Blei et al. (2003). "Latent Dirichlet Allocation"
• Harnad, S. (1990). "The symbol grounding problem"

Modern Extensions:

• Boleda, G. (2020). "Distributional Semantics and Linguistic Theory"
• Grootendorst, M. (2022). "BERTopic: Neural topic modeling with a class-based TF-IDF procedure"
• Grand et al. (2022). "Semantic projection recovers rich human knowledge of multiple object features"

Tools & Libraries:

• Scikit-learn: sklearn.decomposition.TruncatedSVD, LatentDirichletAllocation
• Gensim: gensim.models.LsiModel, LdaModel
• BERTopic: https://maartengr.github.io/BERTopic/

Questions?

Next Lecture:

Word Embeddings: Word2Vec, GloVe, FastText

From count-based to prediction-based methods!