Lecture 7: Text classification workshop

PSYC 51.17: Models of language and communication

Jeremy R. Manning
Dartmouth College
Winter 2026

Learning Objectives

Always explore your data before building models

Convert text to numbers for machine learning

Bag of Words: count how many times each word appears in each document

1from sklearn.feature_extraction.text import CountVectorizer
2
3bow_vectorizer = CountVectorizer(
4    max_features=5000,   # Keep only top 5000 words
5    min_df=2,            # Word must appear in at least 2 docs
6    max_df=0.8,          # Word must appear in <80% of docs
7    stop_words='english' # Remove common words
8)
9
10X_train_bow = bow_vectorizer.fit_transform(train_data.data)
11# Result: Sparse matrix of word counts

Bag of Words: observations, strengths, and weaknesses

• Simple and effective for many tasks
• Captures word presence and frequency
• Ignores word order and context
• Not all words are equally informative
• Usually high-dimensional and sparse

Term Frequency-Inverse Document Frequency

TF-IDF in practice

1from sklearn.feature_extraction.text import TfidfVectorizer
2
3tfidf_vectorizer = TfidfVectorizer(
4    max_features=5000,
5    min_df=2,
6    max_df=0.8,
7    stop_words='english',
8    use_idf=True,
9    sublinear_tf=True  # Use log scaling for term frequency
10)
11
12X_train_tfidf = tfidf_vectorizer.fit_transform(train_data.data)
13# Result: Sparse matrix of TF-IDF scores

Three classifier approaches to compare

Evaluating classifier performance

• Accuracy: overall correctness: $\frac{TP + TN}{TP + TN + FP + FN}$
• Precision: of predicted positives, how many are truly positive?
• Recall: of actual positives, how many did we catch?
• F1-Score: harmonic mean: $F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$ (Intuition: balances precision and recall)
• Confusion Matrix: detailed breakdown of predicted versus actual labels

Error analysis: critical but often skipped

1. Find misclassified examples
2. Look for patterns
3. Understand why the model failed
4. Use insights to improve

Key Takeaways

1. Good features matter more than complex models (for many tasks)
2. TF-IDF usually beats raw BoW for text classification
3. Linear models are competitive with neural networks on bag-of-words
4. Always examine errors to understand model behavior

Putting this week's lectures into context

Questions? Want to chat more?