BERT Applications

BERT excels at understanding tasks:

Case Study: Google Search

BERT revolutionized search in 2019

1# Why word order matters: BERT understands prepositions!
2query = "2019 brazil traveler to usa need a visa"
3
4# Before BERT (bag-of-words matching):
5keywords = ["brazil", "traveler", "usa", "visa"]
6# Matches both: "US traveler to Brazil" AND "Brazil traveler to US"
7
8# With BERT (contextual understanding):
9bert_understanding = {
10    "subject": "brazil traveler",      # WHO is traveling
11    "destination": "usa",               # WHERE they're going
12    "direction": "brazil → usa",        # The preposition "to" is key!
13    "intent": "visa requirements"
14}
15# BERT correctly ranks: "Brazil citizen visa requirements for USA"

Google reported BERT improved 1 in 10 searches in English

Named Entity Recognition

Token-level classification task

1from transformers import pipeline
2
3# Load NER pipeline
4ner_pipeline = pipeline("ner", model="dslim/bert-base-NER")
5
6# Extract entities
7text = "Apple Inc. is headquartered in Cupertino, California."
8entities = ner_pipeline(text)
9
10for entity in entities:
11    print(f"{entity['word']}: {entity['entity']} (score: {entity['score']:.2f})")
12
13# Output:
14# Apple: B-ORG (score: 0.99)
15# Inc: I-ORG (score: 0.99)
16# Cupertino: B-LOC (score: 0.99)
17# California: B-LOC (score: 0.99)

Sentiment Analysis

Sequence classification task

1from transformers import pipeline
2
3# Load sentiment analysis pipeline
4sentiment_pipeline = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")
5
6# Analyze sentiments
7texts = [
8    "This movie was absolutely amazing!",
9    "The product broke after one week.",
10    "The weather is cloudy today."
11]
12
13for text in texts:
14    result = sentiment_pipeline(text)[0]
15    print(f"{text}")
16    print(f"  → {result['label']} (confidence: {result['score']:.2f})\n")

Applications: Customer reviews, social media monitoring, brand sentiment

Semantic Similarity

Measuring sentence similarity with BERT embeddings

1from transformers import BertTokenizer, BertModel
2import torch
3import torch.nn.functional as F
4
5tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
6model = BertModel.from_pretrained('bert-base-uncased')
7
8def get_sentence_embedding(sentence):
9    inputs = tokenizer(sentence, return_tensors='pt', padding=True, truncation=True)
10    outputs = model(**inputs)
11    # Use [CLS] token embedding as sentence representation
12    return outputs.last_hidden_state[:, 0, :]
13
14# Compare sentences
15sent1 = "The cat is sleeping on the couch"
16sent2 = "A feline is resting on the sofa"
17sent3 = "The weather is nice today"
18
19emb1 = get_sentence_embedding(sent1)
20emb2 = get_sentence_embedding(sent2)

Cognitive Neuroscience Perspective

How do brains and models process language?

References: Kuperberg & Jaeger (2016), Willems et al. (2016), Hagoort & Indefrey (2014)

Prediction in Brains vs. Language Models

Parallels between neural and artificial systems

1# Concrete example: Surprise/N400 parallel
2sentence_a = "I take my coffee with cream and sugar"  # Expected
3sentence_b = "I take my coffee with cream and socks"  # Surprising
4
5# Brain: N400 amplitude higher for "socks"
6# Model: Higher loss for "socks"
7loss_a = model.compute_loss("sugar", context)  # Low loss
8loss_b = model.compute_loss("socks", context)  # High loss
9
10# Both systems encode "surprisal" = -log P(word | context)
11surprisal = -np.log(model.predict_prob("socks", context))
12# Correlates with N400 amplitude in EEG studies!

Question: Are these superficial analogies or deep connections?

Reference: Kuperberg & Jaeger (2016) - "What do we mean by prediction in language comprehension?"

Discussion: What Does the Model "Understand"?

Does BERT understand language?

Class Discussion: What do YOU think?

Adversarial Examples and Brittleness

BERT can be fooled easily

1from transformers import pipeline
2classifier = pipeline("sentiment-analysis")
3
4# Works correctly
5classifier("This movie was absolutely wonderful!")
6# → [{'label': 'POSITIVE', 'score': 0.9998}]
7
8# Adding irrelevant negative words flips prediction!
9classifier("This movie was absolutely wonderful! [SEP] bad bad bad bad")
10# → [{'label': 'NEGATIVE', 'score': 0.9234}]  # WRONG!
11
12# Synonym substitution can break it
13classifier("The food was good")   # → POSITIVE (0.99)
14classifier("The food was fine")   # → POSITIVE (0.72)  # Less confident
15classifier("The food was ok")     # → NEGATIVE (0.51)  # WRONG!
16
17# Typos cause problems
18classifier("This is amazign!")    # Might work
19classifier("Thsi si amzaign!")    # Likely wrong prediction

Key insight: Models learn statistical patterns, which can include spurious correlations

Bias in Language Models

Models reflect and can amplify societal biases

1from transformers import pipeline
2unmasker = pipeline("fill-mask", model="bert-base-uncased")
3
4# Gender bias in occupations
5unmasker("The doctor said [MASK] would be late.")
6# → [('he', 0.62), ('she', 0.18), ('it', 0.08), ...]
7
8unmasker("The nurse said [MASK] would be late.")
9# → [('she', 0.71), ('he', 0.15), ('it', 0.06), ...]
10
11# Racial bias (different sentiment for names)
12classifier = pipeline("sentiment-analysis")
13classifier("Emily is a brilliant scientist.")  # POSITIVE: 0.98
14classifier("Jamal is a brilliant scientist.")  # POSITIVE: 0.94  # Lower!
15
16# Where does bias come from?
17# Training data (books, Wikipedia) contains historical biases
18# Model learns and sometimes amplifies these patterns

Practical Tips for Working with Transformers

1. Start with Pre-trained Models
   • Don't train from scratch (too expensive!)

• Use HuggingFace Model Hub
• Choose appropriate model size

2. Fine-tuning Best Practices
   • Use small learning rate (1e-5 to 5e-5)

• Add warmup steps
• Monitor for overfitting
• Freeze early layers if data is limited

3. Computational Efficiency
   • Use mixed precision training (FP16)

• Gradient accumulation for larger batch sizes
• Consider DistilBERT for faster inference
• Use FlashAttention when available

4. Evaluation
   • Use task-specific metrics

• Test on out-of-distribution data
• Check for biases
• Visualize attention for interpretability

Deployment Considerations

Moving from research to production

1# Example: Optimizing BERT for production deployment
2from transformers import BertModel, BertTokenizer
3import torch
4import onnxruntime
5
6# Step 1: Load model
7model = BertModel.from_pretrained("bert-base-uncased")
8tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
9
10# Step 2: Quantize for speed (INT8 instead of FP32)
11quantized_model = torch.quantization.quantize_dynamic(
12    model, {torch.nn.Linear}, dtype=torch.qint8
13)
14# Result: 4x smaller, 2x faster on CPU
15
16# Step 3: Export to ONNX for production
17dummy_input = tokenizer("Hello world", return_tensors="pt")
18torch.onnx.export(model, dummy_input, "bert.onnx")
19
20# Step 4: Use ONNX Runtime for inference

Future Directions

Where is the field heading?

1. Longer Context
   • Efficient attention mechanisms (linear, sparse)

• Models with 100K+ token context
• Better long-document understanding

2. Multimodal Models
   • Vision + Language (CLIP, DALL-E)

• Audio + Language (Whisper)
• Grounded understanding

3. Better Pre-training
   • More efficient objectives

• Curriculum learning
• Continual learning

4. Smaller, More Efficient Models
   • Better compression techniques

• Lottery ticket hypothesis
• Edge deployment

5. Addressing Limitations
   • Debiasing and fairness

• Robustness and adversarial training
• Common sense reasoning
• Interpretability and explainability

Assignment 4: Context-Aware Models

Hands-on experience with transformers!

Tasks:

1. Implement Attention Mechanism
   • Build scaled dot-product attention from scratch

• Visualize attention weights

2. Fine-tune BERT
   • Load pre-trained BERT

• Fine-tune on sentiment analysis
• Compare to baseline models

3. Analyze Contextual Embeddings
   • Extract embeddings for polysemous words

• Visualize how context changes representations
• Compare BERT vs Word2Vec

4. Explore Different Architectures
   • Compare BERT (encoder) vs GPT (decoder)

• Test on different tasks
• Analyze strengths/weaknesses

5. Research Component
   • Read one paper from references

• Write brief summary & critical analysis

Due: Check course website for deadline

Summary: Weeks 5-6

What we learned:

1. Evolution of Context
   • Seq2Seq → Attention → Transformers

• From bottleneck to full parallelization

2. Transformer Architecture
   • Self-attention, multi-head attention

• Positional encoding, layer norm, residuals
• Encoder-only (BERT), Decoder-only (GPT), Both (T5)

3. BERT & Variants
   • Masked Language Modeling

• Pre-train then fine-tune paradigm
• RoBERTa, ALBERT, DistilBERT, ELECTRA

4. Applications
   • Classification, NER, QA, similarity

• Real-world impact (Google Search, etc.)

5. Broader Implications
   • Brain-model parallels

• Understanding vs. pattern matching
• Limitations and future directions