Every model we've studied so far (rules-based models, embeddings, autoregressive models, and diffusion models) improved primarily by making training bigger: more data, more parameters, more compute during training. In late 2024, a new paradigm emerged: test-time compute scaling — spending more compute at inference to improve results.

• Traditional scaling: improving models by increasing training data, parameters, and compute (Kaplan et al., 2020)
• Test-time scaling: improving outputs by spending more compute at inference (Snell et al., 2024)
• Scaling law: a power-law relationship between compute invested and model performance
• "Both axes" (purple in chart): models like o3 that invest heavily in both training and inference compute

Chain-of-thought (CoT) prompting showed that LLMs perform dramatically better on reasoning tasks when prompted to generate intermediate steps before answering. This requires no model changes — just different prompting.

• Threshold circuit: a Boolean circuit whose gates are threshold functions — a gate outputs 1 if at least $k$ of its $n$ inputs are 1. These circuits can compute addition, multiplication, and sorting.
• Constant-depth ($\mathsf{TC}^0$): a threshold circuit with a fixed number of layers, no matter the input size. It can do a lot in parallel but has limited sequential depth.
• Why this matters: a transformer with $L$ layers is essentially a constant-depth threshold circuit — it always performs exactly $L$ sequential steps per token, regardless of problem difficulty.

The key innovation: instead of supervised learning (human writes the reasoning), use reinforcement learning where the model discovers its own reasoning strategies:

1. Sample: Model generates multiple complete solutions (reasoning + answer) for each problem
2. Verify: Check which answers are correct (math: exact match; code: passes unit tests)
3. Update: Reinforce reasoning patterns that led to correct answers; suppress those that didn't
4. Repeat: Over millions of problems, the model learns which reasoning strategies work

DeepSeek-R1 uses GRPO (Group Relative Policy Optimization) — a simpler alternative to PPO (Proximal Policy Optimization, the standard RL algorithm used for RLHF, which requires a separate "critic" model to estimate value). GRPO eliminates the critic by comparing solutions within each group:

1. For each problem, generate a group of $G$ candidate solutions (e.g., $G = 64$)
2. Score each: correct answer → reward $+1$, wrong → reward $0$
3. Compute relative advantage: how much better/worse than the group average?
4. Update the model to increase probability of above-average solutions

The American Invitational Mathematics Examination is a 15-question, 3-hour competition for top high school math students. Problems cover algebra, geometry, number theory, and combinatorics. Each answer is an integer 0–999. It is widely used as a benchmark for AI mathematical reasoning.

When a reasoning model processes a query, it generates two types of content:

1. Thinking tokens — internal reasoning (may be hidden or visible depending on the provider)
2. Output tokens — the final answer shown to the user

The s1 model showed that test-time scaling can be achieved with remarkably little effort:

1. Fine-tune Qwen2.5-32B on just 1,000 curated examples — the "s1K" dataset, a curated set of 1,000 challenging math, science, and coding problems with detailed reasoning traces
2. At inference, apply budget forcing: when the model tries to stop reasoning, append the token "Wait" to the model's own output, forcing it to continue thinking rather than giving a final answer
3. Result: Exceeds o1-preview (a much larger model, trained using human feedback) on AIME 2024 (the 15-question high school math competition from a few slides ago) by up to 27%!

Anthropic implemented reasoning as a toggle within a single model — same weights, different inference behavior. Unlike OpenAI, Claude's thinking tokens are visible to developers and can be controlled using their API.

In a standard (dense) transformer, every parameter activates for every token. In a Mixture of Experts (MoE) model (Shazeer et al., 2017), each layer contains $N$ expert sub-networks (FFNs), but only $k$ are activated per token:

1. A small router network scores each expert for the current token
2. The top-$k$ experts are selected (e.g., 2 of 8, or 8 of 256)
3. Only selected experts compute; the rest are skipped entirely
4. Outputs are combined as a weighted sum based on router scores