Language models as feature extractors

ELMo architecture

ELMo in Python

1from allennlp.modules.elmo import Elmo, batch_to_ids
2
3# Initialize ELMo (need to download these files first)
4options_file = "elmo_options.json"
5weight_file = "elmo_weights.hdf5"
6elmo = Elmo(options_file, weight_file, 2, dropout=0)
7
8# prepare inputs and get embeddings
9sentences = [ 
10    ['I', 'deposited', 'money', 'at', 'the', 'bank'],
11    ['We', 'sat', 'by', 'the', 'river', 'bank']
12]
13character_ids = batch_to_ids(sentences)
14embeddings = elmo(character_ids)
15
16# Different vectors for "bank"!
17bank1 = embeddings['elmo_representations'][0][0, 5, :]  # first sentence
18bank2 = embeddings['elmo_representations'][0][1, 5, :]  # second sentence

Universal Sentence Encoder (USE)

Universal Sentence Encoder in Python

1import tensorflow_hub as hub
2import numpy as np
3from sklearn.metrics.pairwise import cosine_similarity
4
5# Load model
6embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")
7
8sentences = [
9    "The cat sat on the mat.",
10    "A feline rested on the rug.",  # Paraphrase!
11    "The dog ran in the park.",
12    "I love machine learning."
13]
14
15embeddings = embed(sentences) # shape: [4, 512]
16
17# Compute similarity - sentences 1 and 2 should be very similar
18sim_matrix = cosine_similarity(embeddings)

BERT: Bidirectional Encoder Representations

MLM training procedure

Next Sentence Prediction (NSP)

BERT input representation

Final embedding = Token + Segment + Position

BERT in Python

1from transformers import BertTokenizer, BertModel
2import torch
3
4tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
5model = BertModel.from_pretrained('bert-base-uncased')
6
7sent1, sent2 = "I deposited money at the bank", "We sat by the river bank"
8
9# Tokenize and get embeddings
10tokens1 = tokenizer(sent1, return_tensors='pt')
11tokens2 = tokenizer(sent2, return_tensors='pt')
12
13with torch.no_grad():
14    output1 = model(**tokens1)
15    output2 = model(**tokens2)
16
17# Extract "bank" embedding - different vectors!
18bank1 = output1.last_hidden_state[0, 6, :]  # 768-dim
19bank2 = output2.last_hidden_state[0, 6, :]  # 768-dim

Fine-tuning BERT

1for param in bert_model.parameters():
2    param.requires_grad = False
3classifier = nn.Linear(768, num_classes)
4# Train only classifier

1bert_model = BertModel.from_pretrained('bert-base-uncased')
2classifier = nn.Linear(768, num_classes)
3optimizer = Adam([...], lr=2e-5)  # Small learning rate!
4# Train everything end-to-end

Real-world applications

Discussion: do contextual embeddings reflect "understanding"?

Summary

1. Contextual vs. static: Different vectors per occurrence solves polysemy
2. ELMo (2018): BiLSTM language models, character-based, task-specific weighting
3. USE (2018): Sentence-level embeddings optimized for similarity
4. BERT (2018): Masked language modeling, deep bidirectional transformers
5. Impact: Established modern transfer learning paradigm in NLP