Word Embeddings:

• Word2Vec: 300 dimensions
• GloVe: 300 dimensions
• FastText: 300 dimensions
• BERT: 768 dimensions
• GPT-3: 12,288 dimensions!

Challenges:

• Can't visualize 300D space
• Distances behave strangely
• Computation expensive
• Storage requirements
• Interpretation difficult

Concrete Example: Distance Concentration

1import numpy as np
2
3# Sample 1000 random points in different dims
4for dim in [2, 10, 100, 300]:
5    points = np.random.randn(1000, dim)
6    dists = np.linalg.norm(points, axis=1)
7
8    print(f"Dim={dim}: mean={dists.mean():.2f}, "
9          f"std={dists.std():.2f}")
10
11# Output:
12# Dim=2:   mean=1.26, std=0.52
13# Dim=10:  mean=3.08, std=0.49
14# Dim=100: mean=9.95, std=0.50
15# Dim=300: mean=17.3, std=0.50

All points cluster around the same distance from origin!

Visualization:

• 2D/3D plots
• Explore semantic structure
• Identify clusters
• Discover patterns
• Quality assurance

Computation:

• Faster algorithms
• Less memory
• Enable real-time systems
• Scalability

Machine Learning:

• Reduce overfitting
• Feature selection
• Improve generalization
• Remove noise
• Combat curse of dimensionality

Interpretation:

• Understand relationships
• Identify important dimensions
• Communicate results
• Debug models

Original Data (3D → 2D):

Step 1: Center data (subtract mean)

Step 2: Compute covariance matrix C

Step 3: Find eigenvectors/eigenvalues

Eigenvalues (variance captured):

• (85% variance)
• (13% variance)
• (2% variance)

Keep top 2 PCs → 98% variance retained!

Projected Data (2D):

Gender still separable, royalty still clusters!

Assumes Linearity:

• Only captures linear relationships
• Word embeddings often have non-linear structure
• May miss important patterns

1Non-linear manifold ->  PCA struggles here!

Other Issues:

• Variance ≠ Importance:

• Preserves variance, not semantic structure

• Noisy dimensions may have high variance

  \item Global focus:

  • May lose local structure

• Clusters can become blurred

  \item Interpretability:

  • PCs are linear combinations

• Hard to interpret semantically

The Crowding Problem Illustrated:

1High-D: 10 points can each have
2        9 equidistant neighbors
3
4      *   *   *
5        * * *
6      *   *   *
7
8Low-D (2D): Can't fit 9 equidistant
9            neighbors around 1 point!
10
11Solution: t-distribution has
12          heavier tails → allows
13          moderately-distant points
14          to spread further apart

Perplexity Effect (Same Data!):

1Perplexity = 5:
2  Tight, fragmented clusters
3  Good for fine structure
4
5Perplexity = 30:
6  Balanced view
7  Standard choice
8
9Perplexity = 100:
10  Merged clusters
11  More global view

Rule of thumb: perplexity ~ sqrt(n)

For 1000 words: try perplexity 30-50

1. Slow: complexity

1# Timing comparison
2n=1000:  ~10 seconds
3n=5000:  ~4 minutes
4n=10000: ~15 minutes

2. Non-deterministic:

1tsne1 = TSNE(random_state=42)
2tsne2 = TSNE(random_state=123)
3# Different layouts!

3. No out-of-sample:

1# Can't do this with t-SNE:
2new_point_2d = tsne.transform(new_point)
3# Error! Must refit entire dataset

4. Interpretation Pitfalls:

1WRONG interpretations:
2✗ "Cluster A is bigger than B"
3  (sizes are arbitrary)
4
5✗ "A and B are far apart"
6  (global distances not preserved)
7
8✗ "This dimension means X"
9  (axes have no meaning)
10
11CORRECT interpretations:
12✓ "Points in cluster A are similar"
13✓ "These words form a group"
14✓ "There appear to be N clusters"

Key Advantages over t-SNE:

• ( vs )
• Preserves local and global structure
• Can transform new points
• Theoretically grounded (topology)
• Scales to millions of points
• More robust hyperparameters

When to Use:

• Large datasets ($>$10k points)
• Need global structure
• Production systems
• Consistent results important
• Most NLP visualization tasks!

How it Works:

1. Construct fuzzy topological representation of high-D data
2. Find low-D representation with similar topology
3. Use Riemannian geometry
4. Optimize cross-entropy loss

Main Hyperparameters:

1. n_neighbors (15):

• Local vs. global balance
• Small: local structure
• Large: global structure

2. min_dist (0.1):

• How tightly to pack points
• Small: tight clusters
• Large: loose, spread out

1. Embedding Quality Check:

1# Quick sanity check
2words = ['dog', 'cat', 'fish',  # animals
3         'car', 'bus', 'train'] # vehicles
4
5# If good embeddings: two clusters!
6# If bad: random scatter

2. Finding Polysemous Words:

1"apple" appears in TWO clusters:
2- Near: orange, banana, fruit
3- Near: microsoft, google, tech
4
5→ Word has multiple senses!

3. Domain Vocabulary Analysis:

1Medical corpus visualization:
2Cluster 1: symptoms (fever, cough...)
3Cluster 2: treatments (aspirin, surgery...)
4Cluster 3: anatomy (heart, liver...)

4. Model Layer Comparison:

1# Extract embeddings from different layers
2layer_1 = get_bert_layer(1)   # Syntax
3layer_6 = get_bert_layer(6)   # Semantics
4layer_12 = get_bert_layer(12) # Task-specific
5
6# Visualize each: different structure!

5. Document Clustering:

1# Visualize document embeddings
2doc_embeddings = model.encode(documents)
3umap_2d = umap.UMAP().fit_transform(
4    doc_embeddings)
5
6# Color by topic → see topic separation
7plt.scatter(umap_2d[:, 0], umap_2d[:, 1],
8            c=topic_labels)

Approach:

1. Extract embeddings from each layer
2. Apply UMAP to each layer separately
3. Visualize and compare

Typical Findings:

• Layer 0-2: Syntactic (POS tags cluster)
• Layer 3-8: Semantic (meaning clusters)
• Layer 9-12: Task-specific

Insights:

• Lower layers: syntax
• Middle layers: semantics
• Upper layers: task adaptation
• Confirms linguistic hierarchy

1from transformers import BertModel, BertTokenizer
2
3model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
4tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
5
6# Get embeddings from all layers
7tokens = tokenizer(sentences, return_tensors='pt', padding=True)
8outputs = model(**tokens)
9
10# outputs.hidden_states: tuple of 13 tensors
11# (embedding layer + 12 transformer layers)
12
13for layer_idx in range(13):
14    layer_embeddings = outputs.hidden_states[layer_idx]
15
16    # Average over sequence length
17    avg_embeddings = layer_embeddings.mean(dim=1).numpy()
18
19    # Apply UMAP
20    reducer = umap.UMAP(n_components=2)

What we CAN conclude:

• Rough semantic groupings
• Relative proximities
• Cluster existence
• Outliers and anomalies
• Qualitative patterns

What we CANNOT conclude:

• Exact distances
• True high-D structure
• Cluster sizes (t-SNE)
• Between-cluster distances
• Quantitative relationships