What is tokenization?

The tokenization spectrum

Splitting on spaces fails for three reasons

The OOV problem in action

1# Simple word-level vocabulary
2vocab = {"hello", "world", "the", "cat", "sat"}
3
4def tokenize_word_level(text, vocab):
5 tokens = text.lower().split()
6 result = []
7 for token in tokens:
8   if token in vocab:
9     result.append(token)
10   else:
11     result.append("<UNK>") # Unknown token
12 return result
13
14# Example
15text = "Hello! The cat jumped"
16tokens = tokenize_word_level(text, vocab)
17print(tokens)
18# Output: ['hello', 'the', 'cat', '<UNK>']
19# ^^^ "jumped" is unknown!

Subword tokenization to the rescue: most words are made of smaller meaningful pieces

Subword tokenization solves the OOV problem

1# Word-level vocabulary (only words seen in training)
2vocab = {"the", "cat", "sat", "on", "mat", "dog", "ran"}
3
4# Trying to tokenize a new sentence:
5sentence = "The supercalifragilisticexpialidocious cat meowed"
6# Word-level result loses almost everything: ["<UNK>", "<UNK>", "cat", "<UNK>"]
7
8# The subword solution
9from transformers import AutoTokenizer
10tokenizer = AutoTokenizer.from_pretrained("gpt2")
11
12# Same difficult sentence:
13tokens = tokenizer.tokenize("supercalifragilisticexpialidocious")
14print(tokens)
15# ['super', 'cal', 'if', 'rag', 'il', 'istic', 'exp', 'ial', 'id', 'ocious']
16
17# Every word can be tokenized! No information loss.
18# Model can learn that 'super-' often means "very/above"

Byte-Pair Encoding (BPE)

• Original use: Data compression (1994)
• Adapted for NLP: Sennrich et al. (2016)
• Used by: GPT, GPT-2, GPT-3, RoBERTa, BART, many others!

BPE example: Step by step

Training data: "low low low lower lower newest newest newest newest widest"

Initial tokens (characters): l, o, w, e, r, n, s, t, i, d

...continue until we hit the target vocabulary size (e.g., 30,000) or run out of merges.

BPE in practice with HuggingFace

1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("gpt2")  # GPT-2 uses BPE
3
4text = "I'm learning about tokenization!"
5tokens = tokenizer.tokenize(text)
6print("Tokens:", tokens)
7# ['I', "'m", 'Ġlearning', 'Ġabout', 'Ġtoken', 'ization', '!']
8# Note: 'Ġ' represents space
9
10token_ids = tokenizer.encode(text)  # Get token IDs
11print("Token IDs:", token_ids)  # [40, 1101, 4673, 546, 11241, 1634, 0]
12
13decoded = tokenizer.decode(token_ids)  # Decode back
14print("Decoded:", decoded)  # "I'm learning about tokenization!"

WordPiece example with BERT

1from transformers import AutoTokenizer
2
3# BERT uses WordPiece
4tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
5
6text = "I'm learning about tokenization!"
7
8tokens = tokenizer.tokenize(text)
9print("Tokens:", tokens)
10# Output: ['i', "'", 'm', 'learning', 'about', 'token', '##ization', '!']
11# ^^^ Note the ##. Also: BERT lowercases by default (unless using cased model)
12
13# Compare with longer word
14text2 = "unhappiness"
15tokens2 = tokenizer.tokenize(text2)
16print("Tokens:", tokens2)
17# Output: ['un', '##hap', '##pin', '##ess']
18# Breaks into morphological components!

SentencePiece works for languages without spaces

SentencePiece in action

1from transformers import AutoTokenizer
2
3# T5 uses SentencePiece
4tokenizer = AutoTokenizer.from_pretrained("t5-small")
5
6text = "I'm learning about tokenization!"
7
8tokens = tokenizer.tokenize(text)
9print("Tokens:", tokens)
10# Output: ['I', "'", 'm', 'learning', 'about', 'token', 'ization', '!']
11# ^^^ Note the for spaces
12
13# Works seamlessly with other languages!
14text_chinese = "这是中文"
15tokens_cn = tokenizer.tokenize(text_chinese)
16print("Chinese:", tokens_cn)
17# Output: ['这', '是', '中', '文']
18# Breaks into characters/subwords without needing pre-tokenization

Tokenization methods comparison

Tokenizers add special tokens for model-specific purposes

Working with special tokens

1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
3
4text = "Hello world"
5encoded = tokenizer(text, return_tensors="pt")
6print("Input IDs:", encoded['input_ids'])  # [101, 7592, 2088, 102]
7# ^^^ [CLS] and [SEP] added automatically!
8
9full_decode = tokenizer.decode(encoded['input_ids'][0])
10print("With special:", full_decode)  # "[CLS] hello world [SEP]"
11
12clean_decode = tokenizer.decode(encoded['input_ids'][0], skip_special_tokens=True)
13print("Without special:", clean_decode)  # "hello world"

Tokenization pitfalls and gotchas

1. Tokenizer-model mismatch: Always use the tokenizer that matches your model! GPT-2 tokenizer ≠ BERT tokenizer
2. Maximum sequence length: BERT: 512 tokens | GPT-2: 1024 | GPT-3: 2048 — Text gets truncated if too long!
3. Case sensitivity: bert-base-uncased lowercases everything; bert-base-cased preserves case
4. Rare words → many tokens: "antidisestablishmentarianism" → 10+ tokens; can hit sequence limit faster than expected!
5. Special characters: Emoji, Unicode, accents may be split unexpectedly

Connection to human language learning

Discussion questions

Key takeaways

1. Tokenization is fundamental: turns raw text into model-ready input
2. Word-level has major limitations: OOV problem, vocabulary explosion, language-dependent
3. Subword tokenization is the sweet spot: Balanced vocabulary size and sequence length
4. Different methods, similar principles: BPE: frequency-based | WordPiece: likelihood | SentencePiece: language-agnostic
5. Statistical learning connects humans and machines: both discover structure from distributional patterns
6. Always match tokenizer to model! critical for correct predictions