In Lecture 21, we saw how diffusion models generate text by iteratively unmasking tokens (discrete diffusion). Today we turn to the domain where diffusion was born — images — and ask: how do we generate high-resolution images from text prompts?

The U-Net (Ronneberger et al., 2015) is an encoder-decoder neural network with skip connections, originally designed for image segmentation. It became the default backbone for diffusion models — we'll cover it in detail later this lecture.

Latent diffusion separates image generation into two stages:

1. Compression: A pretrained variational autoencoder (VAE) encodes images into a compact latent space (typically 8× spatial compression)
2. Generation: The diffusion process operates entirely in the latent space before decoding back to pixels

The VAE is described on the next slide.

A VAE learns to compress images into a low-dimensional latent representation and reconstruct them:

1. Encoder: Maps image $\mathbf{x} \in \mathbb{R}^{512 \times 512 \times 3}$ to mean $\boldsymbol{\mu}$ and standard deviation $\boldsymbol{\sigma}$ of a latent distribution

• $\boldsymbol{\mu}$ represents the "average" latent representation for the image
• $\boldsymbol{\sigma}$ captures uncertainty — how much the latent can vary while still reconstructing the image

2. Sampling: Draw $\mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2)$ — a latent vector in $\mathbb{R}^{64 \times 64 \times 4}$
3. Decoder: Reconstructs the image from $\mathbf{z}$

A convolution slides a small filter (kernel) across an image, computing a weighted sum at each position. With stride 2, the filter moves 2 pixels at a time, halving the spatial dimensions:

• 8×8 input → 3×3 conv, stride 2 → 4×4 output (4× fewer values)
• 4×4 → 3×3 conv, stride 2 → 2×2 (4× fewer again)

Stacking convolution layers creates a hierarchy: pixels → edges → textures → objects.

The Kullback-Leibler (KL) divergence measures how one probability distribution $Q$ diverges from a reference distribution $P$:

$$D_{KL}(Q \| P) = \int Q(z) \log \frac{Q(z)}{P(z)} dz$$

Intuitively, it quantifies how much information is lost when $Q$ is used to approximate $P$. In VAEs, we want the learned latent distribution to be close to a simple prior (e.g., standard normal) to ensure smoothness and generalization.

CLIP learns a shared embedding space for text and images. It was trained on 400 million image-text pairs from the internet:

1. Image encoder (Vision Transformer): Maps an image to a vector
2. Text encoder (Transformer): Maps a caption to a vector
3. Contrastive training: Matching pairs are pulled close; non-matching pairs are pushed apart

Classifier-free guidance (CFG) improves alignment between text and generated images. During training, the text condition is randomly dropped some fraction of the time. At inference:

$$\tilde{\boldsymbol{\epsilon}} = \underbrace{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)}_{\text{unconditional}} + w \cdot \Big(\underbrace{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c)}_{\text{conditional}} - \underbrace{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)}_{\text{unconditional}}\Big)$$

• $\boldsymbol{\epsilon}_\theta$ — the denoising network (U-Net or DiT), parameterized by $\theta$
• $\mathbf{x}_t$ — the noisy latent at timestep $t$
• $c$ — the text condition (CLIP embedding of the prompt)
• $\varnothing$ — no text (empty prompt, i.e., unconditional)
• Unconditional $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)$ — what the model predicts without any text guidance
• Conditional $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c)$ — what it predicts guided by the prompt
• $w$ — the guidance scale (typically 7–15): how much to amplify the text signal

The U-Net is the default backbone for diffusion models from DDPM through Stable Diffusion 1–2. Its U-shaped design provides multi-scale reasoning:

1. Encoder (downsampling): Reduces spatial resolution while increasing channels — captures global context
2. Bottleneck: Lowest resolution, largest receptive field — reasons about overall composition
3. Decoder (upsampling): Restores spatial resolution — generates fine details
4. Skip connections: Concatenate encoder features directly to the matching decoder layer — without these, the decoder must reconstruct spatial detail from the bottleneck alone, losing fine texture and edges

The Diffusion Transformer (DiT) replaces the U-Net with a standard Vision Transformer:

1. Patchify: Divide the noisy latent into non-overlapping patches (e.g., 2×2)
2. Flatten: Treat patches as a sequence of tokens (just like ViT treats image patches — Lecture 15)
3. Process: Apply standard Transformer blocks with self-attention
4. Unpatchify: Reshape back to spatial dimensions to get the predicted noise

• Vision Transformer (ViT): A Transformer that treats an image as a sequence of patches (like tokens in text) — no convolutions needed
• Noisy latent: The VAE-compressed image with Gaussian noise added at timestep $t$ — this is the input the denoising network must "clean up"
• Predicted noise: The network's estimate of what noise was added — subtract it from the noisy latent to get a cleaner image

• Fréchet Inception Distance (FID) (Heusel et al., 2017) measures the distance between real and generated image distributions using features from a pretrained Inception network. Lower FID = more realistic.
• Inductive biases are assumptions built into the architecture — e.g., convolutions assume local spatial structure; Transformers make fewer such assumptions and let the model learn structure from data.