Image and video generation concept

Diffusion models explained

Reference

A diffusion model is a generative model that learns to make samples by reversing a process that gradually adds noise to data. For an image, generation typically starts with random noise and applies a series of denoising steps. Text or bounding boxes can guide some image-generation systems.

Reviewed 2026-10-09

The basic idea

A diffusion model learns to reverse a gradual noising process. During training, the model is shown data with different amounts of noise added and learns to predict the noise. During generation, it starts from random noise and repeatedly estimates what to remove, turning the noisy starting point into a sample.

That repeated cleanup is the core idea behind many image generators. Some systems also use a prompt or another input to guide the sample as it takes shape.

How the generation process works

First, training data is gradually corrupted by adding noise. The model learns the reverse direction: given a noisy example and its noise level, it predicts the noise or a related quantity that helps recover a less noisy version. The forward corruption process is set up in advance; the reverse process is learned from data.

For generation, the system starts with noise and repeats a denoising update until it reaches an image. In text-guided systems, the prompt can condition the denoising process. One latent diffusion design uses cross-attention layers to connect conditions such as text or bounding boxes to image generation.

What can go wrong

Denoising takes repeated model evaluations, so sampling can be expensive when those evaluations run sequentially. A study of GLIDE reported that its model sometimes failed on prompts describing highly unusual objects or scenarios. That is an example of a prompt-following limitation in one system, not a claim about every diffusion model.

Also, not every diffusion model works directly on image pixels. Latent diffusion models first use a pretrained autoencoder to represent images in a compressed latent space, then run diffusion there. So a simple description of the process can be useful, but it may leave out how a particular system represents its data or conditions its output.

Check your understanding

Try explaining the model in two directions: what happens to data during training, and what happens when the model generates a sample? A sound answer says that training adds noise and teaches a model to reverse that corruption, while generation starts from noise and applies the learned reverse steps.

Now ask yourself what guides a text-to-image result. The prompt is a condition used by some systems during generation; it is not the same thing as the denoising process itself. In one latent diffusion design, cross-attention layers connect conditions such as text or bounding boxes to image generation. If you can describe both the noise-removal loop and the optional prompt connection, you have the main idea.

Questions people ask

Why does an image generator start with noise?

The model learns a reverse process for a process that turns data into noise. Generation can start with random noise and use the learned reverse steps to produce a sample.

How can a text prompt guide an image?

Some systems use a condition such as text to guide image generation. One latent diffusion design uses cross-attention layers to connect conditions to the image-generation model.

Do diffusion models always work directly on pixels?

No. Latent diffusion models can encode an image into a compressed latent representation, run the diffusion process there, and decode the result.

Learn more

17.7 Denoising Diffusion Probabilistic Models
Explains the forward noising process, the learned reverse process, and how generation samples from noise.
Understanding pipelines, models and schedulers
Breaks down how a diffusion pipeline uses a model and scheduler in a repeated denoising loop.
High-Resolution Image Synthesis with Latent Diffusion Models
Introduces latent diffusion and explains conditioning through cross-attention layers.

What changed

Sources

Part of Image and video generation: How generation methods work Choose a creation or editing workflow Assess the output