How do we represent the meaning of words computationally?

The distributional hypothesis

Latent Semantic Analysis (LSA): the OG distributional word representation

Latent Semantic Analysis (LSA): the OG distributional word representation

• Building the term-document matrix gives us a high-dimensional representation of each word (one dimension per document)
• It also gives us a high-dimensional representation of each document (one dimension per word)
• But: real corpora have thousands of documents and tens of thousands of words!
  • Some dimensions (documents, words) will be noisy; we should try to remove them
  • Some dimensions might be highly correlated with each other; we should try to combine them to improve reliability and efficiency
  • We can use dimensionality reduction to find a lower-dimensional representation that captures the important structure in the data

Formal definition of LSA

LSA in Python

1from sklearn.decomposition import TruncatedSVD
2from sklearn.feature_extraction.text import CountVectorizer
3
4documents = [
5    "The cat sat on the mat",
6    "The dog ran in the park",
7    "Machine learning is amazing",
8]
9
10# Build term-document matrix (bag-of-words)
11vectorizer = CountVectorizer(max_features=5000, stop_words='english')
12doc_term_matrix = vectorizer.fit_transform(documents)
13
14# Apply LSA - reduce to 100 dimensions
15# (Note: for small data, use smaller n_components)
16lsa = TruncatedSVD(n_components=100, random_state=42)
17doc_embeddings = lsa.fit_transform(doc_term_matrix)

Finding similar words with LSA

1from sklearn.metrics.pairwise import cosine_similarity
2
3# Get word embeddings
4vocab = vectorizer.get_feature_names_out()
5word_embeddings = lsa.components_.T
6
7def find_similar(word, top_n=5):
8    idx = list(vocab).index(word)
9    word_vec = word_embeddings[idx].reshape(1, -1)
10    sims = cosine_similarity(word_vec, word_embeddings)[0]
11    top_indices = sims.argsort()[-top_n-1:-1][::-1]
12    return [(vocab[i], f"{sims[i]:.3f}") for i in top_indices]
13
14print(find_similar("computer"))
15# [('software', 0.82), ('program', 0.79), ('system', 0.71), ...]

Latent Dirichlet Allocation (LDA)

LDA in Python

1from sklearn.decomposition import LatentDirichletAllocation
2from sklearn.feature_extraction.text import CountVectorizer
3
4# Create bag-of-words
5vectorizer = CountVectorizer(max_features=1000, stop_words='english')
6doc_term_matrix = vectorizer.fit_transform(documents)
7vocab = vectorizer.get_feature_names_out()
8
9# Fit LDA
10lda = LatentDirichletAllocation(n_components=5, random_state=42)
11doc_topics = lda.fit_transform(doc_term_matrix)
12
13# Print top words per topic
14for idx, topic in enumerate(lda.components_):
15    top_words = [vocab[i] for i in topic.argsort()[-7:]]
16    print(f"Topic {idx}: {', '.join(top_words)}")

Key ideas from today

1. Distributional Hypothesis: words in similar contexts have similar meanings
2. LSA: SVD on term-document matrix— dense embeddings that capture global structure
3. LDA: probabilistic topic modeling— interpretable topics and word distributions