Lecture 6: Tokenization

PSYC 51.07: Models of language and communication

Jeremy R. Manning
Dartmouth College
Winter 2026

Learning objectives

What is tokenization?

The tokenization spectrum

Word-level limitations illustrated

Most words are made of smaller meaningful pieces

Subword tokenization solves the OOV problem

1# Word-level vocabulary (only words seen in training)
2vocab = {"the", "cat", "sat", "on", "mat", "dog", "ran"}
3
4# Trying to tokenize a new sentence:
5sentence = "The supercalifragilisticexpialidocious cat meowed"
6# Word-level result: ["<UNK>", "<UNK>", "cat", "<UNK>"]
7# We lost almost everything!

The subword solution:

1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("gpt2")
3
4# Same difficult sentence:
5tokens = tokenizer.tokenize("supercalifragilisticexpialidocious")
6print(tokens)
7# ['super', 'cal', 'if', 'rag', 'il', 'istic', 'exp', 'ial', 'id', 'ocious']
8
9# Every word can be tokenized! No information loss.
10# Model can learn that 'super-' often means "very/above"

Visual: How "unhappiness" gets tokenized

BPE visualization

BPE in practice with HuggingFace

1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("gpt2")  # GPT-2 uses BPE
3
4text = "I'm learning about tokenization!"
5tokens = tokenizer.tokenize(text)
6print("Tokens:", tokens)
7# ['I', "'m", 'Ġlearning', 'Ġabout', 'Ġtoken', 'ization', '!']
8# Note: 'Ġ' represents space
9
10token_ids = tokenizer.encode(text)  # Get token IDs
11print("Token IDs:", token_ids)  # [40, 1101, 4673, 546, 11241, 1634, 0]
12
13decoded = tokenizer.decode(token_ids)  # Decode back
14print("Decoded:", decoded)  # "I'm learning about tokenization!"

WordPiece merges "surprisingly common" pairs

• Used by: BERT, DistilBERT, Electra
• Special tokens: ## prefix for continuation ("playing" → ["play", "##ing"])

WordPiece example with BERT

1from transformers import AutoTokenizer
2
3# BERT uses WordPiece
4tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
5
6text = "I'm learning about tokenization!"
7
8tokens = tokenizer.tokenize(text)
9print("Tokens:", tokens)
10# Output: ['i', "'", 'm', 'learning', 'about', 'token', '##ization', '!']
11# ^^^ Note the ##
12
13# Compare with longer word
14text2 = "unhappiness"
15tokens2 = tokenizer.tokenize(text2)
16print("Tokens:", tokens2)
17# Output: ['un', '##hap', '##pin', '##ess']
18# Breaks into morphological components!

SentencePiece works for languages without spaces

Used by: T5, ALBERT, XLNet, mT5 (multilingual models)

SentencePiece in action

1from transformers import AutoTokenizer
2
3# T5 uses SentencePiece
4tokenizer = AutoTokenizer.from_pretrained("t5-small")
5
6text = "I'm learning about tokenization!"
7
8tokens = tokenizer.tokenize(text)
9print("Tokens:", tokens)
10# Output: ['I', "'", 'm', 'learning', 'about', 'token', 'ization', '!']
11# ^^^ Note the for spaces
12
13# Works seamlessly with other languages!
14text_chinese = ""
15tokens_cn = tokenizer.tokenize(text_chinese)
16print("Chinese:", tokens_cn)
17# Breaks into characters/subwords without needing pre-tokenization

Tokenization methods comparison

Comparing tokenizers side-by-side

1from transformers import AutoTokenizer
2text = "The unhappiest researchers couldn't preprocess data!"
3
4models = {"GPT-2 (BPE)": "gpt2", "BERT (WordPiece)": "bert-base-uncased",
5          "T5 (SentencePiece)": "t5-small"}
6
7for name, model_name in models.items():
8    tokenizer = AutoTokenizer.from_pretrained(model_name)
9    tokens = tokenizer.tokenize(text)
10    print(f"{name}: Tokens ({len(tokens)}): {tokens}")
11
12# Observe: Different handling of "unhappiest", spaces, and token counts!

Tokenizers add special tokens for model-specific purposes

Vocabulary size trade-offs

Debugging tokenization

1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("gpt2")
3text = "The antidisestablishmentarianism debate continues!"
4
5tokens = tokenizer.tokenize(text)
6token_ids = tokenizer.encode(text)
7print(f"Tokens ({len(tokens)}): {tokens}\n")
8
9for i, (token, token_id) in enumerate(zip(tokens, token_ids)):
10    decoded = tokenizer.decode([token_id])
11    print(f"{i:2d}. ID {token_id:5d} | Token: {token:20s} | Decoded: {decoded}")
12
13print(f"\nVocabulary size: {tokenizer.vocab_size}")
14print(f"Special tokens: {tokenizer.all_special_tokens}")

Connection to human language learning

Statistical learning in action

Primary references

Key takeaways

Looking ahead

Questions? Want to chat more?