Each token produces a query $q$, key $k$, and value $v$. Attention scores tell each token how much to "listen to" every other token:

$$A = \text{softmax}\left(\frac{QK^T}{\sqrt{H}}\right)V$$

• A decoder generates text one token at a time, left to right. It uses the causal mask so each token can only attend to previous tokens. GPT is a decoder.
• An encoder processes the entire input at once. Every token can attend to every other token — no mask needed. BERT is an encoder.

BERT simply removes the causal mask — all positions attend to all positions:

$$A_{\text{BERT}} = \text{softmax}\left(\frac{QK^T}{\sqrt{H}}\right)V$$

That's it. Every token sees the full sentence, left and right, at every layer.

Pronoun resolution challenges that test real-world understanding. Originally proposed by Winograd (1972), formalized as a benchmark by Levesque et al. (2012).

Ambiguous pronouns:

• "The delivery workers couldn't fit the packages into the lockers because they were too [large/small]." (they = packages or lockers)
• "The teachers gave the students extra time because they were [struggling/generous]." (they = students or teachers)

Filling in blanks forces BERT to learn grammar ("The [MASK] barked" → dog), semantics ("She [MASK] the exam" → passed/failed), and world knowledge ("The capital of France is [MASK]" → Paris).

Clark et al. (2019) discovered that specific BERT attention heads specialize in linguistic relations:

• Head 8-10: Direct objects (verb → object)
• Head 7-6: Possessive pronouns (noun → its/his/her)
• Head 5-4: Coreference (pronoun → antecedent)

Tenney et al. (2019) showed BERT's layers form a processing pipeline:

1. Lower layers (1–4): Surface features — POS tags, word boundaries
2. Middle layers (5–8): Syntax — parse trees, dependency relations
3. Upper layers (9–12): Semantics — word sense disambiguation, coreference

BERT was trained on 3.3B words of BooksCorpus + Wikipedia:

• Primarily English, formal writing.
• Over-represents certain demographics and viewpoints.
• No social media, no conversations, no code.
• Cutoff: 2018 (no knowledge of recent events).