Maximize the likelihood of each token given all preceding tokens:

$$\mathcal{L}_{\text{pre-train}} = \sum_{i=1}^{N} \log P(t_i \mid t_1, t_2, \ldots, t_{i-1}; \Theta)$$

This is simply next token prediction over a large corpus. No labels needed. The causal attention mask (Lecture 15) ensures each token only attends to previous positions, enabling parallel training over all positions simultaneously.

After pre-training, GPT is adapted to specific tasks by:

1. Reformatting task inputs with special delimiter tokens
2. Adding a small classification head (linear layer) on top
3. Training with both task loss and a language modeling auxiliary loss

$$\mathcal{L}_{\text{fine-tune}} = \mathcal{L}_{\text{task}} + \lambda \cdot \mathcal{L}_{\text{LM}}$$

The auxiliary LM loss ($\lambda = 0.5$) improves generalization and speeds convergence.

• Zero-shot: No task-specific examples. The model must generalize purely from its pre-training.
• Few-shot: A small number of examples (1–10) provided as context in the prompt.
• Fine-tuning: Full supervised training on a task-specific dataset.

Modern GPT models share weights between the token embedding layer and the output (lm_head) layer (Press & Wolf, 2017):

1self.token_embed = nn.Embedding(vocab_size, d_model)
2self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
3self.lm_head.weight = self.token_embed.weight  # Tied!

Jamba (AI21, 2024) interleaves Transformer attention layers with Mamba state-space layers:

• Attention layers: Good at precise retrieval ("what was the third item?")
• Mamba layers: Good at long-range compression and fast inference ($O(n)$ vs $O(n^2)$)
• Hybrid: Gets the best of both — 256K context at 3× the throughput of pure Transformers