Training data is built from raw text in three steps:

1. Tokenize every document in the corpus independently
2. Concatenate all token sequences into one long stream (optionally separated by <EOS> tokens)
3. Chunk the stream into fixed-length blocks of block_size tokens (e.g., 1024 or 2048)

Each chunk becomes one training example. Leftover tokens shorter than block_size are dropped.

Within each chunk, the causal attention mask (a lower-triangular matrix) ensures position $t$ can only attend to positions $1, 2, \ldots, t$. This means every position simultaneously predicts its next token — one forward pass through a chunk of length $L$ produces $L$ training signals.

Labels are simply the input shifted right by one position. The model handles this internally during the forward pass.

Perplexity is the exponential of the average cross-entropy loss across a sequence:

$$\text{PPL} = e^{\mathcal{L}} = e^{-\frac{1}{N}\sum_{t=1}^{N} \log P(x_t \mid x_{<t})}$$

A perplexity of $k$ means the model is, on average, as uncertain as if it were choosing uniformly among $k$ options.

1. Forward pass: feed a batch of text through the model to get predictions
2. Compute loss: compare predictions to actual next tokens using cross-entropy
3. Backward pass: compute gradients $\nabla_\theta \mathcal{L}$ via backpropagation
4. Update parameters: adjust $\theta$ to reduce the loss

$$\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$$

where $\eta$ is the learning rate — how big a step we take.

• Stochastic gradient descent (SGD): Use random mini-batches instead of the entire dataset. Faster and noisier, but works well in practice.
• AdamW optimizer: Gives each parameter its own adaptive learning rate based on gradient history. Adds momentum (smooths updates) and weight decay (prevents overfitting). The standard choice for transformers.
• Learning rate scheduling: Start with a warmup phase (gradually increase LR), then decay with a cosine or linear schedule. Prevents instability early in training.
• Gradient clipping: Cap gradient magnitude to prevent "exploding gradients" from destroying training progress. Typical max norm = 1.0.

Language model performance follows predictable power laws: smooth, straight lines on log-log plots when you increase model size, dataset size, or compute budget.

• Kaplan et al. (2020): Discovered these relationships across seven orders of magnitude
• Hoffmann et al. (2022, "Chinchilla"): Showed optimal training balances model size and data — a smaller model trained on more data can match a larger undertrained model (Chinchilla 70B matched GPT-3 175B)

Key takeaway: you can predict model performance before training by running small-scale experiments first.

1. Pre-training $\rightarrow$ Foundation model: Train on a massive general text corpus (books, web, code). The model learns language patterns, world knowledge, and reasoning. Expensive ($millions), done once. Examples: GPT-3 base, LLaMA, Mistral.

2. Instruction tuning + RLHF $\rightarrow$ Instruct model: Fine-tune the foundation model on instruction-following data and human preference feedback. The model learns to be helpful, harmless, and honest. This is what makes ChatGPT different from raw GPT-4. Examples: ChatGPT, Claude, Gemini.

3. Task-specific fine-tuning $\rightarrow$ Specialized model: Further fine-tune on domain data for a specific application. Cheap, fast, and highly effective. Examples: medical diagnosis, legal analysis, code generation for a specific codebase.