A Survey of Diffusion Models: Methods and Applications

Abstract

Diffusion models have emerged as the state-of-the-art generative paradigm, surpassing GANs in synthesizing high-fidelity images, videos, and audio. However, their reliance on iterative denoising processes imposes substantial computational burdens and memory overheads, creating a significant barrier to their deployment on resource-constrained edge devices. Unlike existing surveys that broadly cover general methodologies, this paper provides a focused review with a specific emphasis on efficient and lightweight diffusion models. We systematically analyze the trade-offs between generation quality and computational cost, categorizing acceleration techniques into sampling optimization, architectural compression, and knowledge distillation. Furthermore, we explore the integration of diffusion models with emerging architectures (e.g., Mamba) and their evolution towards general-purpose world simulators. This survey aims to provide a roadmap for “Green AI,” bridging the gap between high-end academic research and practical, real-world applications.

1. Introduction

In recent years, generative models have entered the fast lane in vision and cross-modal tasks, prior to the rise of diffusion models, there are three predominant models: Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and flow-based models. GANs are trained through an adversarial game between a generator and a discriminator, capable of generating very sharp and realistic images [1]. However, their training process is extremely sensitive and unstable, often facing the problem of mode collapse, where the generator in the adversarial model can only produce a limited variety of samples and cannot cover the full data distribution. VAEs, grounded in variational inference, learn latent representations of data by maximizing the evidence lower bound (ELBO), Kingma and Welling [2]. Their training process is stable, but due to the trade-off between the fidelity term and the regularization term in the optimization objective, the generated samples, particularly images, often appear somewhat blurred. Flow-based models utilize a sequence of reversible transformations to map simple distributions, such as Gaussians, onto complex data distributions, enabling precise calculation of data likelihood [3]. However, their architecture design is constrained by reversibility, incurs high computational costs, and typically underperforms GANs when handling high-dimensional data such as high-resolution images. These models collectively face the challenge of balancing generative quality, diversity, and training stability. Consequently, researchers require novel models that simultaneously ensure generative quality, stable training, and comprehensive coverage of the data landscape. As illustrated in Figure 1, unlike GANs, VAEs, and flow-based models which rely on adversarial learning or explicit latent mapping, diffusion models introduce a paradigm shift by learning to reverse a gradual noise degradation process.

The concept of diffusion models was first proposed by Sohl-Dickstein et al. [5], drawing inspiration from non-equilibrium thermodynamics in physics. Its core concept is remarkably elegant: a complex generative process can be decomposed into a series of extremely simple, incremental denoising steps. This process comprises two stages: First, the forward process, a fixed, non-learnable procedure. It simulates a data degradation process, progressively injecting small amounts of Gaussian noise into the original data across multiple time steps until the data ultimately transform into pure standard normal distributed noise. Finally, the reverse process constitutes the core learning component of the model. It attempts to reverse the forward process, starting from a purely noisy input. Through a series of denoising steps, it progressively removes the noise, ultimately reconstructing a clear, authentic sample. Diffusion models possess the following advantages over other models due to this unique mechanism: Firstly, diffusion models exhibit excellent training stability. Their training objective is maximum likelihood estimation or its surrogate objective, which does not involve adversarial training, rendering the process highly stable. Secondly, diffusion models produce high-fidelity outputs. Through multi-step iterative denoising, the model meticulously captures data details, yielding samples with exceptional fidelity and granularity that outperform GANs across multiple benchmarks. Finally, diffusion models exhibit strong diversity. As their training objective is to fit the entire data distribution, they are less prone to pattern collapse and generate diversification samples.

Diffusion models have become mainstream due to stable training and high-fidelity generation. The research community formulates this approach as a reversible process of “forward noise-reverse denoising”. Ho et al. [6] give an operational paradigm in terms of discrete Markov chains. Song et al. [7] use SDE and probabilistic flow ODEs to provide continuity perspective and theoretical unity, while engineering practice brings these two lines together to form a clear training and sampling pipeline. In order to lower the computational threshold, Rombach et al. [8] migrated denoising into the latent space and used cross-attention access to the text, so that the actual system can operate stably at high resolution. Controllability quickly became a central requirement: users expected the model to understand text, but also to obey structural signals such as edges, depth, pose, layout, and even audio or reference images. To this end, the community proposes two types of control mechanisms. One class comes from guidance strength, classifier guidance by Dhariwal and Nichol [9] and no classifier guidance by Ho and Salimans [10], allowing for tuning between faithfulness and diversity. The other type comes from structured conditions. The ControlNet of Zhang and Agrawala [11] hangs the control branch to the backbone with zero convolution, and various conditions can be combined. Hertz [12] directly rewrites the cross-attention heatmap for local and fine-grained editing. Video and 3D tasks further increase the difficulty, and long-term consistency, identity stability and cost control constrain each other. Therefore, researchers use a cascade scheme of “key-frame-interpolation-super-resolution”, and use DPM-Solver and distillation to reduce the number of steps. In reproduction experiments, research teams often reduce sampling steps from 50 to 100 to 20–30, and deploy multiple styles and conditions with LoRA or adapters. These practices can significantly reduce latency, but also bring side effects such as loss of detail and appearance rigidity.

1.1. Relation to Existing Surveys and Contributions

While comprehensive surveys such as Yang et al. [13] and Croitoru et al. [14] provide excellent overviews of the general diffusion landscape, they primarily focus on generation quality and broad application scenarios. As diffusion models transition from academic research to industrial deployment, the trade-off between performance and resource consumption has become increasingly critical. This paper distinguishes itself by prioritizing the efficiency and lightweight deployment of diffusion models. Our main contributions are threefold:

• We provide a specialized taxonomy of acceleration techniques, categorizing them into algorithmic solvers, architectural compression, and system-level lightweight paradigms.
• We bridge the gap between high-end foundation models and edge-side applications, highlighting strategies for deploying diffusion models on mobile and embedded devices.
• We offer an updated perspective on emerging architectures (e.g., Mamba/SSMs) and the convergence with LLMs, moving beyond the traditional U-Net-centric view.

1.2. Organization of the Paper

The remainder of this survey is organized to guide the reader from theoretical foundations to practical implementation and future trends. Section 2 unifies the theoretical basis of diffusion models, integrating both probabilistic Markov chains and continuous differential equation perspectives. Building upon these underpinnings, Section 3 dissects key methodologies, including the evolution of backbone architectures and controllable generation mechanisms. Subsequently, Section 4 explores the expanding application domains, ranging from 2D imagery to complex 3D, video and audio content. A pivotal component of this review is Section 5, which provides an in-depth analysis of efficient and lightweight diffusion models. This section comprehensively details quantization, pruning, and distillation techniques designed to mitigate computational overhead. Following this technical analysis, Section 6 addresses the associated societal and ethical challenges, while Section 7 proposes future research directions. Finally, Section 8 provides concluding remarks.

2. Basic Principles of Diffusion Models

The overall probabilistic framework is illustrated in Figure 2, where the reverse process $p_{\theta }$ iteratively removes noise to generate realistic samples.

2.1. Forward Process

The forward process, also termed the diffusion process, constitutes a predefined Markov chain. Commencing from an initial real data sample $x_0\sim q\left(x\right)$, it progressively incorporates Gaussian noise over T discrete time steps. At any time step t, the state transition from $x_{t-1}$ to $x_t$ may be defined by Equation (1). Among these, ${\left\{{\beta }_t\right\}}_{t=1}^T$ represents a predefined noise schedule that increases incrementally with each time step. Typically, it grows linearly or cosinusoidally from a very small value, such as ${\beta }_t={10}^{-4}$, to a larger value, such as ${\beta }_T=0.02$. Let ${\alpha }_t=1-{\beta }_t$ and ${\overline{\alpha }}_t={\prod }_{i=1}^t{\alpha }_i$. This process possesses a significant property: we may directly sample the noise-augmented sample $x_t$ at any time t from $x_0$, without requiring iterative computation. This process is illustrated in Equation (2). This closed-form solution is crucial, as it enables us to randomly sample any time step t and its corresponding noisy sample $x_t$ during training, thereby significantly enhancing training efficiency. When T is sufficiently large, ${\overline{\alpha }}_T\approx 0$, at which point the distribution of $x_T$ will closely approximate the standard normal distribution $\mathcal{N}(0,\mathbf{I})$.

$$q\left(x_t\right|x_{t-1})=\mathcal{N}(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_t\mathbf{I})$$

(1)

$$q(x_t\mid x_0)=\mathcal{N}(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)\mathbf{I})$$

(2)

2.2. Reverse Process

The reverse process constitutes a generative procedure whose objective is to progressively sample backwards from pure noise $x_T\sim \mathcal{N}(0,\mathbf{I})$ to ultimately yield a clear sample $x_0$. This necessitates calculating the reverse transition probability $p\left(x_{t-1}\right|x_t)$. By Bayes’ theorem, the true posterior distribution $q\left(x_{t-1}\right|x_t,x_0)$ is solvable. However, it depends on the unknown original data $x_0$ and thus cannot be directly utilized during generation. The key breakthrough of diffusion models lies in the fact that when the noise increment ${\beta }_t$ at each step is sufficiently small, the backward transition probability $p\left(x_{t-1}\right|x_t)$ can also be approximated as a Gaussian distribution. Consequently, we can employ a neural network $p_{\theta }$ to learn this distribution, as shown in Equation (3), where the mean ${\mu }_{\theta }(x_t,t)$ and variance ${\Sigma }_{\theta }(x_t,t)$ are predicted by a neural network (typically a U-Net architecture) based on the current noisy image $x_t$ and time step t.

$$p_{\theta }\left(x_{t-1}\right|x_t)=\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t),{\Sigma }_{\theta }(x_t,t))$$

(3)

2.3. Training Objectives and Loss Functions

The training objective of diffusion models is to maximize the log-likelihood of the data, $log{p}_{\theta }\left(x_0\right)$. Through variational inference, we can optimize its variational lower bound (ELBO). Following a series of derivations, the loss function can be simplified and restructured. The DDPM proposed by Ho et al. [6] introduced a revolutionary simplification: they fixed the variance ${\Sigma }_{\theta }$ as a constant related to ${\beta }_t$, allowing the model to learn only the mean ${\mu }_{\theta }$. By reparametrizing the mean, the prediction objective can be transformed from the more complex task of denoising the image $x_{t-1}$ to predicting the noise $ϵ$ added at step t. Utilizing the relationship $x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ$, the model ${ϵ}_{\theta }(x_t,t)$ attempts to predict the original injected noise $ϵ$ from $x_t$. This renders the loss function remarkably concise, as shown in Equation (4). This straightforward mean squared error (MSE) loss function is both intuitive and effective: at each time step, the model learns how best to estimate the noise component within the image. The training process is remarkably stable, requiring only the repetition of the following steps: first, sample an initial $x_0$ from the dataset; second, randomly select a time step t; third, sample a noise $ϵ$ and compute $x_t$; finally, have the model predict the noise, calculate the loss using MSE, and perform gradient descent.

$$L_{simple}\left(\theta \right)={\mathbb{E}}_{t,x_0,ϵ}[\parallel ϵ-{ϵ}_{\theta }(\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,t){\parallel }^2]$$

(4)

3. Methodologies

3.1. Fundamental Frameworks and Architectures

3.1.1. DDPMs (Denoising Diffusion Probabilistic Models)

In 2020, the DDPM proposed by Ho et al. [6] marked a milestone in modern diffusion models. It not only proposes the aforementioned concise loss function, but also employs the U-Net architecture proposed by Ronneberger et al. [15] as the denoising network. The encoder–decoder structure and skip connections of U-Net rendered it exceptionally well-suited for processing both local and global information in images, enabling effective structural recovery from noisy images. The DDPM achieved remarkable results in image generation tasks, with its output quality surpassing that of the then-state-of-the-art GAN models on datasets such as CIFAR-10 for the first time, formally heralding the dawn of the diffusion model era. Early DDPMs predominantly relied on convolutional U-Net architectures. However, around 2022, the core architecture of diffusion models underwent a profound transformation from U-Net to Transformer. Taking the Diffusion Transformer (DiT) proposed by Peebles and Xie [16] as an example, models treat image, video, and other data as sequences of spacetime patches, leveraging the long-range dependency modeling capabilities of Transformers to learn complex spatio-temporal relationships.

3.1.2. From Discrete Markov Chains to Continuous SDEs and ODEs

A DDPM views the forward diffusion as a noisy chain. By setting a set of variance coefficients (the beta sequence), it gradually pushes the clean samples towards an isotropic Gaussian distribution; the reverse diffusion trains a denoising network, gradually pulling the noise back to the data distribution [6]. The loss function ostensibly comes from ELBO, but the common implementation simplifies it into a mean squared error for predicting noise at each time step. The advantage of this is intuitive: stable training, clear gradients, and easy parallelization across multiple GPUs. In practical engineering, three details often affect the outcome: Firstly, the selection of parameters. The network can predict noise ($ϵ$), clean samples ($x_0$), or a compromise variable (v). Multiple studies have shown that using v to predict is more stable and has fewer overflows in strong guidance or high-resolution settings [17,18]. Secondly, noise scheduling: linear beta is the classic choice, but using cosine or data distribution-adaptive scheduling can retain more gradient signals in low-noise regions [9,18]. Finally, variance learning: using only a fixed posterior variance can train, but learning an additional variance branch often improves perceptual quality on complex data, at the cost of slightly increasing instability [9]. A common practical problem with discrete paths is that when strong CFG, few sampling steps, and high resolution occur simultaneously, the parameterization and noise scheduling in training will amplify errors. Researchers’ experience is that combining v-parameterization, minimum SNR weighting, and cosine or EDM-style sigma sampling distribution can significantly alleviate this [18,19]. When writing the diffusion process as an SDE, there is an important corollary: as long as the scores at each noise scale are known, an inverse-time SDE can be written, and by simulating along it, one can return to the data distribution [7]. The same set of marginal distributions also corresponds to a deterministic probabilistic ODE. If using ODE, one can use larger step sizes and higher-order numerical integrators, reducing the number of function evaluations. EDM demonstrated clearly that using the Heun second-order method, and with appropriate sigma sampling distribution and loss weights, improved stability and speed [18]. This leads to a problem: which is better, the “predictor-corrector” of the SDE or the “higher-order solver” of the ODE? Practice shows that if the goal is perceptual quality and fewer steps for generation, the probabilistic ODE combined with DPM-Solver or Heun, etc., is more suitable, but if the goal is strict likelihood evaluation or the need to inject randomness, the SDE and PC sampling are more flexible, while the aforementioned discrete methods focus on practical implementation. A rigorous theoretical foundation was established by Song et al. [20], who proved that as the number of time steps approaches infinity ($T\to \infty$), the discrete diffusion process converges to a continuous-time Stochastic Differential Equation (SDE). The forward process can be modeled as an Itô process:

$$d\mathbf{x}=\mathbf{f}(\mathbf{x},t)dt+g\left(t\right)d\mathbf{w}$$

(5)

where $\mathbf{f}(x·,t)$ is the drift coefficient, $g\left(t\right)$ is the diffusion coefficient, and $\mathbf{w}$ is the standard Wiener process. Crucially, the reverse generation process is also an SDE [21]:

$$d\mathbf{x}=[\mathbf{f}(\mathbf{x},t)-g{\left(t\right)}^2{\nabla }_{\mathbf{x}}log{p}_t\left(\mathbf{x}\right)]dt+g\left(t\right)d\overline{\mathbf{w}}$$

(6)

This formulation highlights the central role of the score function ${\nabla }_{\mathbf{x}}log{p}_t\left(\mathbf{x}\right)$. Instead of utilizing ELBO indirectly, the model is trained via Score Matching [22] to estimate this gradient field. The noise prediction network ${ϵ}_{\theta }$ in a DDPM is effectively learning a scaled score function: $s_{\theta }(\mathbf{x},t)\approx -{ϵ}_{\theta }(\mathbf{x},t)/{\sigma }_t$. Furthermore, this framework reveals the existence of a deterministic Probability Flow ODE that shares the same marginal distributions as the SDE:

$$d\mathbf{x}=[\mathbf{f}(\mathbf{x},t)-{\displaystyle \frac{1}{2}}g{\left(t\right)}^2{\nabla }_{\mathbf{x}}log{p}_t\left(\mathbf{x}\right)]dt$$

(7)

As illustrated in Figure 3, this ODE creates a smooth, bijective mapping between the data distribution and the latent noise. This perspective not only unifies DDPMs and score-based models but also enables the use of advanced numerical solvers (e.g., DPM-Solver [23]) for rapid sampling, significantly reducing the number of inference steps required compared to traditional methods.

3.1.3. Main Branch and Conditional Injection

Currently, U-NET remains the dominant backbone in diffusion. It has two practical advantages: multi-scale features and skip connections preserve local details, as well as the receptive field and computational mode of convolution being friendly to high-resolution images. The diffusion version of U-NET typically includes temporal embeddings (sine positional encoding plus perceptron), normalization layers (group normalization is common), and cross-attention layers (receiving text or other modalities). Many projects also use the scale and bias of channels, similar to FiLM which injects conditions into each layer of normalization [8,15]. The Transformer backbone brings another balance. DiT uses image blockings along with position and time tokens to view denoising as sequence modeling; U-ViT and MDT have, respectively, demonstrated that long skip connections and masked training are helpful for stability [24,25]. In projects, Transformer can achieve better scalability and out-of-distribution generalization, but it comes at the cost of secondary complexity and memory overhead. Therefore, many teams use U-NET as the base for high-resolution tasks, and Transformer for balanced designs for multimodal alignment or global modeling, which is more common in video and 3D tasks. From a controllable perspective, there are three key points in the backbone that are most commonly used: cross-attention (accessing text or semantic sequences), normalization modulation (using the scale/bias generated by conditions to control channels), and side-path control branches (such as the zero convolution injection of ControlNet). The selection of these three key points depends on the form of the conditions, such as semantic classes being more suitable for cross-attention, structural classes (edges, depth, posture) being more suitable for spatial alignment and side-path features, and mixed classes being commonly used together [11].

3.2. Sampling Acceleration and Efficiency

To provide a structured overview of backbone advancements, we illustrate the architectural evolution in Figure 4, which delineates the transition from traditional CNN-based U-Nets and Transformer-based DiTs to the emerging SSM frameworks.

3.2.1. Breakthroughs in Speed and Efficiency

Slow sampling speed constituted a core limitation of early diffusion models such as DDPMs. The Denoising Diffusion Implicit Models (DDIMs) proposed by Song [7] represent one of the most influential works in this field. DDIMs demonstrate that the Markov chain assumption underpinning DDPMs is not essential, enabling faster sampling through a non-Markovian forward process. It introduces an additional parameter $\eta$, when $\eta =1$, the sampling process is equivalent to DDPMs. When $\eta =0$, it becomes a deterministic generation process termed the implicit model. This enables DDIMs to generate images of comparable or superior quality to DDPM’s thousand-step samples using merely tens of steps, achieving a 10 to 50 fold increase in generation speed and significantly enhancing the practicality of diffusion models. Although techniques like the DDIM achieved preliminary acceleration, iterations spanning dozens of steps remained too sluggish for real-time applications. A central focus of recent research has been pushing generation speed to its limits, enabling high-quality output through single-step or few-step processes [26]. Kim et al. [26] have proposed a training paradigm termed Consistency Models, designed to directly learn the endpoint of a consistent trajectory from any noisy point to a generated sample. Through a technique known as Consistency Distillation, a trained multi-step diffusion model can be distilled into a model capable of generating high-quality images in a single step. Similarly, the theoretical framework of Flow Matching reformulates diffusion processes as ordinary differential equations (ODEs). It focuses on learning a “straighter” path from the noise distribution to the data distribution, thereby circumventing the curved, inefficient sampling trajectories inherent in traditional diffusion models. By learning this optimized mapping, corrected flow models achieve high-quality generation in minimal steps while retaining theoretical elegance and interpretability [27]. The maturation of these techniques heralds diffusion models’ complete liberation from the “sluggish” label, expanding their applications from offline content creation to interactive design, real-time rendering, and beyond. In the same year, Wang et al. [28] proposed a new GAN framework called Diffusion-GAN. This approach employs a pre-trained diffusion model as a “teacher” utilizing adversarial training to distil the teacher model’s multi-step generative capabilities into a single-step generator (the “student” model). This method achieved outstanding single-step FID scores on benchmarks such as ImageNet, marking a new pinnacle in single-step generation quality.

3.2.2. Sampling and Acceleration

DDIMs represent the reverse process as a deterministic mapping, enabling decent results to be obtained within just a dozen steps [29]. Subsequently, PNDM, DEIS, EDM, and the DPM-Solver series introduced numerical methods for ordinary differential equations into diffusion sampling, reducing error accumulation and decreasing the number of steps [30,31]. These methods gradually became universal accelerators for video and high-resolution generation. When single-step generation is still slow, researchers use distillation methods, gradually combining two steps and cycling multiple rounds to reduce 50–100 steps to a single digit [10]. Although guided distillation can also work, it requires handling issues related to guidance strength and distribution drift [32]. In multimodal scenarios, one approach is to first use higher-order solvers to reduce the number of steps to 10–20, then perform distillation, which can significantly reduce quality loss. Some studies have proposed squeezing the iterative process into a consistent model with a single step or a few steps, and this direction has achieved feasible results in image tasks, but the optimal practices for large resolutions and videos are still evolving.

3.2.3. Training Objectives, Sampling Intervals and Weights

Many projects have changed the loss weights for all time steps at the same time to be sampled according to SNR or sigma distribution. This is because samples with extremely high or low noise are either too difficult or too easy, and the average weights would cause the network to waste learning capacity in irrelevant intervals. The minimum SNR weighting and the loss formula of EDM have both shown consistent improvements on different datasets [19]. The selection of step size and time steps also has a significant impact. In most practices, a step size allocation that is dense at the beginning and sparse at the end is more stable; in ODE solvers, dense tail segments can reduce the final artifacts; in PC sampling of SDEs, the number of correction steps should not be excessive, otherwise the time cost will rise rapidly [18]. These experiences may seem fragmented, but in multimodal controllability and video editing, they often determine whether the standard can be achieved within 10–20 steps. There are also two points that are often overlooked at the implementation level. The first point is that exponential moving average (EMA) weights are almost a necessary condition, especially in cases of few-step sampling and distillation. The second point is that mixed precision and gradient clipping can significantly reduce NaN and overflow, especially in settings with strong guidance and large batch sizes. These practices are not algorithmic contributions, but they can determine whether the reproduction is smooth.

3.3. Controllable Generation Mechanisms

In this section, we review different controllable generation mechanisms.

Table 1. Classification of controllable generation mechanisms.

Category	Mechanism	Training Required	Key Technique and Description
Guidance	Classifier Guidance [9] Classifier-Free Guidance [6]	No	Modifies the sampling score using gradients from a classifier or by mixing conditional/unconditional noise predictions. (CFG requires joint training but no extra classifier.)
Structural Injection	ControlNet [11] T2I-Adapter [33]	Yes	Injects spatial conditions (edges, depth, pose) via side-branches connected by zero-convolutions to the locked main backbone.
Attention Control	Prompt-to-Prompt [12] Cross-Attention Control	No	Directly manipulates the cross-attention maps during the reverse process to preserve structure while changing style or specific objects.
Personalized Fine-tuning	DreamBooth [34] Textual Inversion [35] LoRA [36]	Yes	Adapts the model to specific concepts/identities. LoRA uses low-rank matrices to update weights efficiently; Textual Inversion optimizes word embeddings only.

shows the classification of controllable generation mechanisms.

3.3.1. Conditional Generation and Guidance

To enable diffusion models to produce controllable outputs based on user instructions such as text or category labels, researchers have developed conditional generation mechanisms. The early conditional generation method was proposed by Dhariwal and Nichol [9] as a classifier-guided approach. The concept involves utilizing the gradient of a classifier $p_{\varphi }\left(y\right|x_t)$, pre-trained on noisy images, at each denoising step to guide the generation process towards the target category y. Specifically, it modifies the scoring function illustrated in Equation (8). This approach yields significant results but necessitates training an additional classifier, which must be capable of handling images with arbitrary noise levels. The Classifier-Free Guidance (CFG) approach proposed by Ho and Salimans [10] overcomes this challenge. The ingenuity of CFG lies in its practice during training of setting conditional information such as text embeddings to null with a certain probability, thereby enabling the model to learn both unconditional and conditional generation simultaneously. During the sampling phase, the model simultaneously predicts conditional noise ${ϵ}_{\theta }(x_t,t,c)$ and unconditional noise ${ϵ}_{\theta }(x_t,t,Ø)$. The final guidance vector is then constructed via a simple linear combination as shown in Equation (9), where w denotes the guidance scale. This formula intuitively operates as follows: first, compute the ‘direction vector’ from unconditional to conditional predictions; then amplify it by a factor of w and add it back to the unconditional prediction, thereby amplifying the influence of conditional information. CFG requires no additional classifiers, delivers superior performance, and has become the standard configuration for contemporary state-of-the-art text-to-image and text-to-video models.

$${\tilde{\nabla }}_{x_t}logp\left(x_t\right|y)={\nabla }_{x_t}logp\left(x_t\right)+\gamma {\nabla }_{x_t}log{p}_{\varphi }\left(y\right|x_t)$$

(8)

$${\widetilde{ϵ}}_{\theta }={ϵ}_{\theta }(x_t,t,Ø)+w({ϵ}_{\theta }(x_t,t,c)-{ϵ}_{\theta }(x_t,t,Ø))$$

(9)

3.3.2. Form and Objectives

Controllability is not simply adding another layer of rules outside the model; instead, it integrates constraints, preferences, and prior knowledge into the generation process itself. Semantics, space, identity, and style are four common high-frequency requirements. They correspond to different information forms and determine different control interfaces. First, there is semantic control. The model needs to understand the text or category and maintain semantic consistency in the generation [9,17]. Attribute mismatch and missing goals are common failure modes, especially in scenarios with multiple entities and long descriptions [37]. Second, there is spatial control, including layout, segmentation, edges, sketches, depth, and posture, which requires where and what to appear. These conditions are usually given in the form of images or grids [11]. Alignment accuracy and local details are the key points of spatial control [38]. Then, there is identity control, which requires binding to specific individuals or objects and maintaining appearance and geometry in new scenes. These methods often use a small number of image fine-tuning or learning new word embeddings [34,35]. Overfitting and language drift are the two major risks in identity control [39,40]. Finally, there is style control, where users hope to transfer materials, brushstrokes, or color tones without disrupting the layout and subject. Common implementations include energy guidance, latent space direction, and small parameter adaptation [41,42]. To determine whether something is controllable, one often needs to pay attention to three key points simultaneously: the alignment of the text or conditions, the stability of space and time, and the preferences and details of the human eye. A single indicator rarely can explain the entire issue, which is the practical reason why communities have gradually adopted the practice of combining indicator groups with human evaluation [43].

3.3.3. Control Mechanism

The researchers roughly classified the control mechanisms into three categories. The first category is guidance, the second is condition injection, and the third is explicit rewriting of attention and features. Additionally, the researchers also proposed personalized and low-cost fine-tuning. Firstly, it is guidance. Dhariwal and Nichol proposed classifier guidance [9], where the authors provided a noise-robust classifier to give gradients, and added this gradient to the scores to obtain stronger conditional alignment. Classifier-free guidance originated from Ho and Salimans’ approach, often referred to as classifier-free guidance. The authors only trained one network, and during training, they removed the conditions with a certain probability while linearly combining the conditional and unconditional outputs to balance fidelity and diversity. Secondly, there is condition injection and control branches. Zhang and Agrawala [11] proposed ControlNet, where the authors used the replicated convolution blocks of the main backbone as the control branches, and used zero convolution to inject the branch features into the main backbone to support various conditions such as edges, depth, normals, segmentation, and poses. This technique helps to make the training more stable without destroying the original capabilities. Textual conditions use cross-attention mechanisms. Radford et al. [44] proposed CLIP, where the authors embedded the text and image into the same space, and latent diffusion used the text features as Keys/Values. The intermediate features of U-Net served as Queries, and it aligned the semantics with it. Then, there is attention control and training-free editing. Hertz et al. [12] presented the concept of cross-attention control for image editing, where the authors directly replaced or rewrote the attention heatmap to achieve fine-grained editing without modifying the weights, which was used for structure preservation and style replacement. Meng et al. [45] proposed SDEdit, where the authors first added noise to the input to a moderate level and then denoised along the diffusion trajectory to obtain stable results in restoration and stylization. Finally, there is personalized and parameter-efficient fine-tuning. Ruiz et al. proposed DreamBooth [34], where the authors bound a new concept to a few images with the same word, retained the shape and texture of the subject, and used text to control the scene. Gal and Alaluf proposed Textual Inversion [35], where the authors only optimized the embedding of a new word to enable several images to learn the style or features of an object. Hu et al. [36] proposed LoRA, where the authors used low-rank decomposition for fine-tuning, saving memory and parameters, and it has been widely used in the adaptation and deployment of image, video, and audio diffusion. Tewel et al. [40] proposed Perfusion, where the authors used very small rank-1 updates to lock the concept and style to maintain the synthesis quality and reduce forgetting.

3.3.4. Evaluation and Benchmarks

In terms of general quality metrics, FID and KID are used to measure the closeness of the overall distribution, while IS is more focused on sample diversity [43]. In terms of alignment and controllability, CLIPScore and text retrieval accuracy can reflect semantic matching, while regional control can be evaluated using the similarity of CLIP within the mask, and the accuracy of object detection or segmentation. For pose and depth control, key point errors and depth correlation can be used for assessment. For videos, FVD and inter-frame consistency are needed, which can be evaluated using flicker rate and optical flow distortion error [46]. For 3D and multi-view consistency, Chamfer distance and IoU and new view consistency can be used for assessment [47]. In human evaluation, researchers tend to use both double-blind preference and multi-dimensional questionnaires to evaluate alignment, details, stability, and overall preference, which is particularly important in stylization and editing tasks [17,48].

4. Applications

In this section, we review some important applications of diffusion models.

Table 2. Summary of Application Domains and Representative Diffusion Models.

Domain	Sub-Task	Representative Models	Key Innovation
2D Image	Text-to-Image	Stable Diffusion [8], DALL-E 2 [49], Imagen [17], GLIDE [5]	Introduced Latent Diffusion Models (LDMs) for efficiency; Utilized Deep Language Understanding (T5/CLIP) for semantic alignment.
3D	Text-to-3D & View Synthesis	DreamFusion [50], Magic3D [51], Zero-1-to-3 [52], Fantasia3D [42]	Score Distillation Sampling (SDS) transfers 2D priors to 3D; Coarse-to-fine optimization; Disentangled geometry and appearance.
Audio	Text-to-Audio, Music & Speech	AudioLDM [53], MusicLM [52], DiffWave [54]	Latent diffusion on Mel-spectrograms (“Spectrogram-as-Image”); CLAP embedding conditioning; Hierarchical generation for long-form consistency.
Restoration	Inpainting, Deblurring & Super-Resolution	DDRM [18], Palette [17], SDEdit [45], GDP [55]	Unified frameworks for multiple degradation tasks; Solves linear inverse problems using pre-trained priors without re-training.

summarizes the application domains of representative diffusion models.

4.1. Image Restoration

If convolution blur, down sampling or mask absence occur during the degradation process, the consistency constraint can be incorporated into the sampling loop, allowing the observations and priors to jointly determine the trajectory. DDRM uses a pre-trained denoising network as the prior and provides unsupervised posterior sampling for the linear inverse problem, demonstrating the feasibility of data consistency and prior denoising [18]. DDNM takes a step further, decomposing the reconstruction into the range part determined by the observations and the null space part completed by the prior, without the need to train the model again. Zero samples can handle super-resolution and deblurring problems [56]. ILVR fixes the structural alignment with low-frequency replacement, allowing the high-frequency details to be filled by the prior [57]. If the task is more about editing rather than rigorous solving, SDEdit injects weak noise in the sampling middle and converges along the guidance prompt, enabling stable modifications without re-training the model [45]. For scenarios requiring unified multi-tasking, Palette combines colorization, restoration, and de-compression into a conditioned framework, demonstrating the possibility of a diffuser corresponding to multiple restoration tasks [17]. GDP models the generation prior explicitly as a sampleable posterior, adapting to linear, nonlinear, and blind degradation, covering a wide range of requirements from denoising to multi-degradation enhancement [55]. From practical observations, trained general image priors, such as weights based on LDMs, when transferred to data with distribution shifts such as remote sensing and medicine, may exhibit style bias or excessive detail imagination. In such cases, moving the consistency step forward and reducing the strength of the later guidance usually stabilizes the PSNR or SSIM results while retaining the visual advantage of LPIPS. In terms of evaluation metrics, the restoration task still relies on PSNR and SSIM, with LPIPS and FID demonstrating perceptual quality and distribution proximity [43,58].

4.2. Two-Dimensional Image Generation

Text-to-image generation stands as the most widely recognized application of diffusion models. Models such as DALL-E 2 developed by Ramesh et al. [49], Imagen proposed by Saharia et al. [17], and Stable Diffusion introduced by Rombach et al. [8], can generate detailed and creative images matching any textual description provided by the user. These models typically employ a pre-trained text encoder, such as CLIP’s text encoder, to convert input text into an embedding vector. This vector is then injected as a conditional input into a U-Net via cross-attention mechanisms, guiding the image generation process. Advancements in this technology have not only transformed the landscape of digital art and content creation but also provided powerful tools for the design, advertising, and entertainment industries. Firstly, there is the transformation from categories to images. The category-conditioned diffusion model is the foundation of general image generation. ADM and -G, through classifier guidance and architecture modification on CIFAR and ImageNet, have achieved FID and IS indices at the level comparable to GANs [9]. CDM uses cascaded generation to gradually increase the resolution, making details and structures more stable [10]. At the backbone level, LDM moves denoising to the VAE latent space, significantly reducing memory and computing power [8]. DiT and MDT demonstrate the potential of Transformers in long dependencies and scalability [16]. During actual training, the coupling between CFG weights, steps, and scheduling is strong. A high CFG tends to improve class alignment and clarity, but it compresses diversity, especially more pronounced in few-step inference [9]. Therefore, many experiments make CFG a curve over time and encrypt the solution accuracy of the last few steps to reduce tail overshoot artifacts and align with the usage of DPM-Solver [18]. Secondly, there is the transformation from text to image. Text-conditioned generation has formed a standard paradigm combining cross-attention, latent space, and guidance. GLIDE and Imagen demonstrate the help of deep language representations for semantic alignment [9,17]. VQ-Diffusion, Parti, and Muse present different efficiencies from the perspectives of discrete tokenization, self-attention expansion, and masked modeling [59,60,61]. Although text is important, spatial constraints are also crucial. GLIGEN integrates text bounding boxes on large models, allowing object placement to be controllable [62]. ControlNet uses trainable side branches and zero convolution to inject conditions such as edges, depth, and posture into each layer, achieving combined control of text and structure [11]. T2I-Adapter provides a lighter adaptation interface, facilitating multi-condition assembly [33]. These methods significantly reduce the probability of attribute mismatches and target omissions in complex scenarios, such as multiple objects and multiple relationships [63]. Safety and alignment are also one of the dimensions that experiments must consider. Safe Latent Diffusion and self-distillation safe training demonstrate ways to mitigate inappropriate content, but the triple combination of model–tokens–filtering remains the mainstream method [64]. In terms of evaluation, CLIPScore is semantically consistent with the monitoring of text retrieval accuracy. FID and KID reflect the quality of the distribution. Human evaluation is responsible for conducting subjective evaluations such as whether it seems credible, and the combination of these three can better cover real usage scenarios [43].

4.3. Three-Dimensional Model/Content Generation

Diffusion models are also employed to generate 3D models from text or 2D images. Poole et al. [50] proposed a technique termed Score Distillation Sampling (SDS). Its core concept involves utilizing a pre-trained 2D text-to-image diffusion model as a knowledge source. On 3D representations, such as rendered views from NeRFs, SDS computes a loss that distills the knowledge encoded in the 2D diffusion model into the 3D model. This guides the 3D model towards optimization consistent with textual descriptions. Consequently, high-quality 3D assets can be generated from text without requiring any 3D training data, offering novel avenues for game development, virtual reality (VR), and metaverse construction. 3D data is scarce, and directly applying diffusion on voxel, point cloud, and mesh results in extremely high costs. SDS and SJC provide solutions, namely, transferring 2D diffused noise residuals or Jacobian chain back to 3D expressions, such as NeRF or SDF, allowing the rendered multi-view to gradually approach the conditioned text [50]. Magic3D uses sparse grids and differentiable rendering in two stages, namely, coarse-to-fine, to improve speed and detail [51]. Fantasia3D decouples geometry and appearance and introduces normal and material modeling, making the material realism better [42]. Single-image to 3D is also important. Zero-1-to-3 fine-tunes 2D diffusion on camera conditions for new view synthesis and further drives 3D optimization [29]. In terms of multi-view consistency, SyncDreamer learns synchronous features across views, jointly training with 2D text pairs and 3D assets to enhance consistency [65]. LAS-Diffusion, centered on SDF, improves controllability and details through local attention and signed distance field representation [66]. 3D evaluation typically focuses on Chamfer distance, voxel IoU, and consistency in new viewpoints, and datasets such as GSO provide objective standards [47]. Optimization based on SDS still has cost and stability issues, such as inducing local optima or view artifacts due to different texts or styles. These issues can be gradually alleviated through better rendering and geometric regularization.

4.4. Video Generation and Editing

Building upon image generation, extending diffusion models to the temporal dimension enables video generation. Models such as Sora (OpenAI), Lumiere (Google), Pika, and Runway have demonstrated the capability to generate videos lasting up to one minute, featuring high resolution, coherent content, and dynamic motion. The core challenge lies in effectively modeling spatial-temporal consistency. A common approach involves incorporating temporal modules, such as 3D convolutions or temporal attention into U-Net architectures, enabling simultaneous processing of spatial and temporal information to ensure coherence between generated video frames. This technology heralds a paradigm shift in video production, film special effects, and virtual world construction. Li et al. [38] details a method termed “Animate anyone”, which employs diffusion models to generate highly faithful and temporally coherent character animation videos from static reference images and motion sequences, demonstrating diffusion models’ advanced capabilities in generating complex dynamic content. The difficulty in video generation lies in inter-frame consistency and motion explainability. Due to the scarcity of data, many systems choose methods that combine image priors with lightweight temporal layers. MCVD and FDM apply diffusion to conditional tasks such as prediction and interpolation, verifying their feasibility on medium and short sequences [67,68]. For text-to-video, Make-A-Video and Imagen Video both adopt a cascading method, generating low-resolution base sequences first and then performing spatial or temporal super-resolution, significantly improving temporal consistency [69]. Dreamix demonstrates the path of video editing, using target text and low-resolution references to fine-tune the video diffuser while maintaining the appearance and changing the semantics [70]. Latent Video Diffusion moves the latent space method of LDM to videos, saving memory and making the project easy to deploy [71]. Conditional control is also important in videos, and VideoComposer integrates multiple visual conditions, such as depth, sketches, and optical flow, into U-Net and latent space, enabling fine manipulation. ControlVideo extends the spatial control of ControlNet to the temporal domain without training, reducing jitter through cross-frame interaction [72]. Practice shows that sharing initial noise, cross-frame attention, and key frame—interpolation—refinement pipeline processes helps alleviate flickering. However, identity drift and remote dependence remain challenges that constantly emerge in video generation. Evaluation typically uses FVD and CLIP-SIM, observing whether the human eye perceives changes, as pure objective scores cannot fully capture the subjective feelings of flickering and stable contrast [46].

4.5. Audio Generation: From Speech to Music

While image and video generation focus on spatial structures, audio generation tackles the complexity of high-temporal-resolution 1D signals. Early diffusion-based audio models, such as DiffWave [54] and WaveGrad [73], focused on the role of a neural vocoder, generating raw waveforms from mel-spectrograms conditioned on text or linguistic features. These models outperformed traditional autoregressive models like WaveNet in terms of inference speed and synthesis quality. However, generating complex audio (such as music or environmental sounds) from scratch requires modeling long-term dependencies. Inspired by the success of Stable Diffusion, AudioLDM [53] introduced a latent diffusion approach to the audio domain. By learning a continuous latent representation of mel-spectrograms using a VAE and conditioning the diffusion process on contrastive language-audio pre-training (CLAP) embeddings, AudioLDM achieved state-of-the-art zero-shot text-to-audio generation. This “spectrogram-as-image” paradigm allows the model to leverage image inpainting techniques for audio restoration and style transfer. More recently, the focus has shifted towards long-form music generation and high-fidelity text-to-speech (TTS). MusicLM [52] treats audio generation as a hierarchical sequence-to-sequence modeling task, ensuring consistency over several minutes of music. Meanwhile, models like Stable Audio typically utilize autoencoders with downsampling factors designed for audio to handle the extremely high sampling rates, balancing local texture quality with global musical structure.

5. Efficient and Lightweight Diffusion Models

Despite the remarkable generation quality of diffusion models, their deployment in real-world scenarios is often hindered by two inherent bottlenecks: the high computational cost of the iterative denoising process and the substantial memory footprint of the heavy backbone networks. These limitations are particularly prohibitive for edge devices and real-time applications. Consequently, “Green Diffusion”—the pursuit of efficiency without sacrificing fidelity—has become a pivotal research frontier. In this section, we provide a comprehensive review of recent advances in lightweight diffusion models. We categorize these acceleration techniques into three primary strategic pillars:

• Sampling Acceleration: Focusing on advanced ODE solvers and scheduling strategies to reduce the number of inference steps from hundreds to mere dozens or single digits.
• Architectural Compression: Employing techniques such as network pruning, quantization, and structural search to minimize parameter count and memory usage.
• Knowledge Distillation: Leveraging teacher–student frameworks to condense the multi-step diffusion trajectory into fewer steps, enabling rapid inference.

To provide a structured understanding of these efficiency techniques, we present a taxonomy of lightweight strategies in Figure 5, which categorizes current approaches into sampling acceleration, architectural compression, and numerical optimization. The following subsections discuss how these methodologies summarized in

Table 3. Comparison of lightweight diffusion strategies and performance.

Method	Core Strategy	Architecture	Task	Performance Gains
MobileDiT [74]	Hybrid Architecture (ViT + Conv)	MobileViT block + adaLN-Zero	General Image Generation	FID: 2.15 on ImageNet; Beat StyleGAN-XL.
L-DiffCOD [75]	Structural Pruning	PVTv2-B1	Camouflaged Object Detection	FLOPs ↓ 47.45%; Params ↓ 75%; Real-time inference on edge devices.
Q-GESCO [76]	Quantization (PTQ)	8-bit PTQ	Semantic Communication	Memory ↓ 75%; Robust reconstruction under channel noise.
Finger Vein Net [77]	Generation-Augmented Lightweight	Dual-branch Network, E-MHSA	Finger Vein Recognition	Only 2.15 M Params; High accuracy via synthetic data augmentation.
LWTDM [78]	Efficient Encoder-Decoder	Cross-Attention-based Lightweight Embedding	Remote Sensing Super-Resolution	Inference steps compressed to <200; Reduced Params and MACs.

address the trade-off between computational efficiency and generative performance.

5.1. Structural Efficiency and Backbone Optimization

The most straightforward path to lightweighting is to restructure the bloated U-Net backbone network. Research has shown that by introducing hybrid architectures or specialized convolutions, it is possible to significantly reduce the computational load while maintaining the generation quality. Liu and Sun [74] proposed the MobileDiT architecture in the context of general image generation. Instead of using conventional convolution, they chose the MobileViT block as the core component. This design is very smart: using lightweight convolution to handle local details and using Transformers to capture global shapes. To prevent training from crashing, they also introduced the adaLN-Zero (adaptive layer normalization) module to stabilize the signal. The results were impressive: the model achieved an FID score of 2.15 on ImageNet, with extremely low parameters and even surpassed the larger parameter StyleGAN-XL in the generation effect. Targeted structural pruning is more aggressive in specific tasks. Cai et al. [75] developed L-DiffCOD for camouflaged object detection. They used PVTv2-B1 as the encoder backbone and replaced all standard convolutions with Depthwise Separable Convolution. This operation reduces floating-point operations (FLOPs) by 47.45 percent and the number of parameters by 75 percent, making it possible to identify masquerading targets in real time on embedded devices. This was also verified by Li et al. [79] in the field of network security. To synthesize malicious traffic data at edge nodes, they similarly employ depth-wise separable convolution techniques. The experimental data show that the number of Multiplication and Addition Operations (MACs) of the model is only 20.39 percent of the baseline model, but it can train a classifier with an accuracy of 92.34 percent, which proves that the lightweight model is sufficient to capture complex traffic characteristics. For remote sensing image super-resolution, An et al. [78] completely abandoned U-Net. The LWTDM model they designed employs an efficient encoder–decoder based on cross-attention. By introducing lightweight embedding modules, they avoid the stacking of deep networks. To further accelerate, they combined DDIM hop sampling to compress the number of inference steps to less than 200, which meets the timeliness requirement of satellite data processing. These structural optimizations effectively maintain the generation quality and task accuracy while greatly reducing the number of parameters and floating-point operations, which lays a foundation for the real-time deployment of diffusion models in resource-constrained environments such as mobile terminals, satellites, and embedded devices.

5.2. Quantization and Frequency Domain Learning

When the structure optimization reaches the bottleneck, reducing the data accuracy or transforming the processing domain is another effective path. In semantic communication, Grassucci et al. [76] face severe bandwidth constraints. They proposed the Q-GESCO framework, which employs post-training Quantization (PTQ) to compress model weights to 8-bit. To address the cumulative error caused by low bits, they innovatively used noise-aware training and time-step calibration. This results in a 75 percent reduction in memory footprint, and the reconstructed image at the receiver remains sharp even in the presence of channel noise. The FedDifftraj model proposed by Gao et al. [80] in the field of autonomous driving focuses on data transmission. In the federated learning framework, the model on the vehicle side is quantified before uploading to the server. This not only saves valuable IoV bandwidth, but also enhances the protection of user trajectory privacy through parameter fuzzification. For super-sized medical images, Wilms et al. [81] avoid the lack of computing power by changing the “battlefield”. They used Wavelet Packet Transform to process 3D brain MRI. This method losslessly reduces the spatial resolution of the image and transfers the computation to the frequency domain. This allows 3D conditional diffusion models, which would otherwise require a high-end cluster to train, to run on a single consumer-grade GPU and generate interpretable “counterfactual images” to assist doctors. In this section, we explore two key paths beyond architecture optimization: Leveraging post-training Quantization to reduce numerical accuracy, which can dramatically reduce video memory footprint and communication bandwidth, and enhances privacy protection in federated learning. At the same time, signal domain transformation methods such as wavelet packet transform are used to reduce the spatial resolution of high-dimensional data, which makes it possible to train large-scale 3D medical imaging models on consumer-grade hardware. These strategies prove that optimization at the numerical and frequency domain level is an effective means to break through the bottleneck of edge computing resources.

5.3. The Generation-Augmented Lightweight Paradigm

This is a system-level approach to lightweight: leveraging powerful diffusion models to create data in the cloud, empowering small models on the end. Tao et al. [77] demonstrated this idea in finger vein recognition. Finger vein data are usually difficult to obtain, resulting in recognition models that are prone to overfitting. They utilized diffusion models to generate massive photorealistic finger vein images for augmentation. Finally, a dual-branch lightweight network with only 2.15 M parameters was deployed in the terminal. The network integrates the E-MHSA (Improved Multi-head Self-Attention) module to achieve high-accuracy recognition with very low computing power. Li et al. [82] also adopted this strategy in battlefield intention recognition. They developed the Wasserstein distance based WDiffusion model specifically for generating multivariate time series data. These synthetic data are used to train a minimalist GRU (Gated recurrent Unit) network. This approach demonstrates that high-quality synthetic data can make up for the small capacity of lightweight models, allowing them to generalize significantly. Wang et al. [83] used lightweight models for image inpainting in the field of pathology. They built a lightweight Transformer based denoising network specifically designed to remove folding and bubble artifacts from histological images. Compared to traditional CNNS, Transformers can better understand the global arrangement of cells, which leads to higher quality image restoration with lower parameters. In this section, we illustrate system-level lightweight strategies that leverage diffusion models to generate high-quality synthetic data to address data scarcity, and then train and empower end-to-end minimalist models (e.g., GRUs or lightweight Transformers). Through the idea of “data conversion power”, this paradigm proves that massive realistic synthetic samples can make up for the shortcomings of the small capacity of lightweight networks, so that they can achieve excellent generalization ability and recognition accuracy with a very low number of parameters in tasks such as finger vein recognition and battlefield intention judgment.

6. Challenges and Limitations

While diffusion models have revolutionized generative AI, their transition from academic research to ubiquitous deployment reveals critical bottlenecks. These challenges are not merely engineering hurdles but involve complex intersections of computational efficiency, legal frameworks, and societal ethics. This section critically analyzes three primary dimensions of these limitations.

6.1. Computational Cost and Environmental Sustainability

The most significant technical drawback of diffusion models, compared to single-pass GANs or VAEs, lies in their inherently iterative inference process, which necessitates evaluating the neural network tens to hundreds of times to solve the probability flow ODEs or SDEs. Although recent advancements in fast solvers (e.g., DPM-Solver) and distillation techniques have mitigated this issue, high-resolution generation remains computationally expensive, severely hindering real-time applications such as video conferencing or interactive gaming where millisecond-level latency is mandatory. This computational burden extends beyond mere latency to create a profound “Green AI” dilemma; while the training of foundation models is known to emit tons of $C{O}_2$, the cumulative energy cost of inference is often overlooked yet more insidious. As these models are deployed to millions of users, the daily energy consumption for routine generation tasks vastly outstrips the initial training cost, posing a serious challenge to global sustainability goals as highlighted by Luccioni et al. [84]. Furthermore, the stringent hardware requirements for fine-tuning or even deploying state-of-the-art models create a significant “compute divide,” effectively concentrating research capabilities within well-funded industry laboratories while marginalizing academic and independent researchers who lack access to high-performance GPU infrastructure.

6.2. Intellectual Property, Copyright, and Data Provenance

The training paradigm of diffusion models, characterized by the indiscriminate scraping of billions of images from the open web, has precipitated a profound crisis regarding intellectual property (IP) law and creator rights, while proponents often invoke a “fair use” defense by arguing that these models learn abstract representations analogous to human inspiration, empirical evidence presents a conflicting reality; specifically, Carlini et al. [85] demonstrate that diffusion models are prone to memorizing and regurgitating training data. This phenomenon, often driven by overfitting to duplicated examples, results in the generation of near-identical replicas of protected works, thereby constituting potential copyright infringement. Beyond the legal complexities of exact replication, the technology creates significant economic peril through style mimicry, where models emulate the unique aesthetic of living artists with high fidelity, allowing users to generate unlimited derivative works without compensating the original creators. Consequently, the research community is actively developing technical countermeasures, ranging from adversarial defense tools like Glaze [86]—which applies imperceptible perturbations to disrupt style learning—to provenance standards such as C2PA and invisible watermarking; however, the robustness of these attribution methods against common transformations remains an unresolved challenge.

6.3. Bias, Safety, and Misuse

Since diffusion models fundamentally approximate the conditional distributions of their vast training corpora, they inevitably inherit and often amplify the biases and toxicity embedded within the uncurated internet. This phenomenon is particularly evident in the amplification of societal stereotypes; for instance, Bianchi et al. [87] demonstrate that text-to-image models frequently exhibit severe occupational gender bias (e.g., defaulting to male depictions for high-status professions) and racial homogenization, thereby perpetuating representational harms when deployed in media and advertising. Furthermore, the potential for harm extends beyond passive bias to active malicious exploitation. In the absence of robust safeguards, these generative tools can be weaponized to produce Not-Safe-For-Work (NSFW) imagery, hate speech, and hyper-realistic Deepfakes, which pose significant threats to political stability through disinformation campaigns and violate personal privacy via non-consensual content. Addressing these risks presents a complex safety–utility trade-off: while current mitigation strategies rely on training data filtration or post hoc output checking (e.g., CLIP-based safety filters), these measures remain brittle against adversarial attacks. Sophisticated “jailbreaking” prompts can often bypass restrictions, whereas overly aggressive filtering risks incurring an “alignment tax”—diminishing the model’s diversity and rendering it unresponsive to benign requests.

7. Future Research Directions

The rapid evolution of diffusion models suggests a shift from specialized image generators to fundamental building blocks of general-purpose AI. We identify three critical directions that will shape the next generation of generative models.

7.1. Scalable Architectures: From Transformers to State Space Models

While Diffusion Transformers (DiTs) have successfully replaced U-Nets in high-fidelity generation, they face a bottleneck: the quadratic computational complexity ($O\left(n^2\right)$) of the self-attention mechanism with respect to sequence length. This poses a significant challenge for generating long-form videos or high-resolution 3D volumes. Recently, State Space Models (SSMs), particularly Mamba [88], have emerged as a compelling alternative. Mamba introduces a selective scan mechanism that achieves linear scaling ($O\left(n\right)$) while maintaining the modeling capacity of Transformers. Integrating Mamba into diffusion backbones offers the potential to process extremely long contexts efficiently. Future research will likely focus on hybrid architectures that combine the global addressing capability of attention with the efficiency of SSMs, enabling the generation of minute-long videos or entire musical compositions in a single forward pass.

7.2. Convergence of Reasoning and Generation (LLM + Diffusion)

A major limitation of current diffusion models is their lack of semantic reasoning; they generate pixels based on statistical correlations rather than an understanding of the world. Conversely, Large Language Models (LLMs) excel at reasoning but lack native sensory output. Early efforts like Visual ChatGPT-3.5 [89] bridged this gap by using LLMs as controllers to invoke diffusion models as external tools. However, the field is moving towards native multimodal integration. Future models will likely operate on unified token spaces, where visual and audio data are quantized (via VQ-VAE or similar techniques) and interleaved with text. This would allow a single foundation model to understand complex multi-step instructions, perform spatial reasoning, and generate consistent multimodal outputs, effectively blurring the line between “understanding” and “creation.”

7.3. Towards World Simulators and Emergent Capabilities

Scaling laws have shown that increasing model size and data volume leads to emergent capabilities. In the context of video diffusion, models are beginning to function as “World Simulators.” By observing vast amounts of video data, these models implicitly learn the laws of physics, object permanence, and 3D geometry without explicit supervision. Future research will likely leverage this property for embodied AI and robotics. Instead of relying on hand-crafted simulation environments, agents could be trained inside a diffusion-generated world simulator that can predict future states and reactions to actions. This represents a paradigm shift from generating media for human consumption to generating data for machine learning and scientific simulation.

8. Conclusions

This paper has presented a comprehensive survey of diffusion models, tracing their evolution from theoretical novelties to the backbone of modern generative AI. We have synthesized the core principles that enable high-fidelity generation and categorized the diverse mechanisms for controllability and multimodal synthesis. A central theme of our analysis is the inevitable shift towards efficiency, while early research prioritized generation quality at any cost, the future of diffusion models lies in “Green AI”—making these powerful models accessible and sustainable. Through our detailed review of lightweight architectures, solver acceleration, and knowledge distillation, it is evident that deploying diffusion models on edge devices is not only feasible but rapidly advancing. Looking forward, the convergence of diffusion models with reasoning capabilities (LLMs) and efficient architectures (SSMs) suggests a trajectory towards building comprehensive “World Simulators.” However, realizing this vision requires not only technical breakthroughs in computational efficiency but also robust solutions to the ethical challenges of copyright, bias, and safety. We hope this survey serves as a valuable roadmap for researchers aiming to bridge the gap between high-performance generation and practical, resource-constrained deployment.