AWDP-FL: An Adaptive Differential Privacy Federated Learning Framework

Abstract

Data security and user privacy concerns are receiving increasing attention. Federated learning models based on differential privacy offer a distributed machine learning framework that protects data privacy. However, the noise introduced by the differential privacy mechanism may affect the model’s usability, especially when reasonable gradient clipping is absent. Fluctuations in the gradients can lead to issues like gradient explosion, compromising training stability and potentially leaking privacy. Therefore, gradient clipping has become a crucial method for protecting both model performance and data privacy. To balance privacy protection and model performance, we propose the Adaptive Weight-Based Differential Privacy Federated Learning (AWDP-FL) framework, which processes model gradient parameters at the neural network layer level. First, by designing and recording the change trends of two-layer historical gradient sequences, we analyze and predict gradient variations in the current iteration and calculate the corresponding weight values. Then, based on these weights, we perform adaptive gradient clipping for each data point in each training batch, which is followed by gradient momentum updates based on the third moment. Before uploading the parameters, Gaussian noise is added to protect privacy while maintaining model accuracy. Theoretical analysis and experimental results validate the effectiveness of this framework under strong privacy constraints.

1. Introduction

Federated learning, as a distributed machine learning technique, successfully addresses the issue of data silos, allowing institutions to collaborate across regions without the need to transfer data, improving the accuracy of predictive models [1,2,3]. Researchers have significantly enhanced data privacy protection by integrating various privacy-preserving techniques, including secure multi-party computation, homomorphic encryption, and differential privacy. Ma et al. proposed a federated learning method based on multi-key homomorphic encryption, which effectively protects privacy and reduces computational costs. Park et al. employed homomorphic encryption to directly encrypt model parameters, enabling the central server to compute on encrypted data without decrypting it. However, despite these techniques achieving privacy protection through “computable but not visible” methods, they also introduce significant computational and communication overhead [4,5,6,7,8,9,10,11,12]. In particular, secure multi-party computation relies on complex communication protocols, while homomorphic encryption requires extensive encryption operations.

In contrast, differential privacy, with its simplicity and strong privacy protection capabilities, has become an important research direction in federated learning. In particular, Local Differential Privacy (LDP), which adds noise locally without relying on the trustworthiness of a central server, offers higher security and can prevent privacy leaks on the server side. However, LDP also faces some challenges, especially in terms of gradient clipping. The role of gradient clipping is to control the magnitude of gradients, prevent abnormal gradients from affecting global model updates, and avoid excessive gradients from amplifying the noise. Although each client in LDP independently implements privacy protection measures and needs to clip gradients before transmission to control the noise effect, excessive clipping may lead to degraded model performance and affect the convergence of the global model. Therefore, balancing the strength of gradient clipping to ensure privacy protection while maintaining model performance is a key challenge in LDP.

In addition, federated learning faces the challenge of communication costs, particularly in Internet of Things environments, where communication between devices is costly [13,14,15]. Federated learning relies on frequent parameter exchanges between clients and servers, and this frequent communication is especially significant in resource-constrained devices. Although differential privacy technology enhances privacy protection, it also increases communication overhead and training time. Therefore, effectively reducing communication costs while preserving privacy remains one of the key challenges in federated learning.

In response to the challenges in the aforementioned FL models, the contributions of this paper are as follows:

• We propose an Adaptive Weight-Based Differential Privacy Federated Learning (AWDP-FL) framework aimed at reducing communication costs during the federated learning process while safeguarding privacy and maintaining model accuracy. The two core improvements of this framework are reflected in gradient clipping and gradient updating. First, we introduce an adaptive gradient clipping method, incorporating two-layer historical gradient sequences to record and analyze trends in historical gradient changes. This enables dynamic weight coefficient calculations and adaptive gradient clipping for each data point in each batch, better controlling gradient fluctuations and reducing unnecessary communication overhead. Second, we propose an adaptive momentum update strategy based on the third moment, further optimizing model training to enhance convergence speed while ensuring privacy protection. To further enhance privacy protection, dynamic Gaussian noise is added when uploading parameters, ensuring the security of the transmission.
• We conducted multi-scenario experiments on the MNIST, Fashion-MNIST, and CIFAR-10 datasets, analyzing the algorithm from multiple dimensions, including external comparisons with other advanced algorithms and internal comparisons by adjusting local parameters to explore the effectiveness of the algorithm. In addition, we provide the source code used in the experiments.

2. Related Work

•

Differential Privacy Protection in Federated Learning

Local Differential Privacy (LDP) prevents privacy leakage by adding noise locally. The LDPFL algorithm proposed by Chamikapa et al. [16] reduces the risk of information leakage by increasing the randomness of input data while maintaining high model accuracy. The LDP-Fed algorithm developed by Truexs et al. [17] allows users to customize privacy budgets based on local privacy needs. However, the introduction of noise inevitably reduces model accuracy, especially in high-dimensional data and complex model scenarios. Sun et al. [18] proposed an adaptive weight parameter setting and data perturbation strategy, which improves model performance by addressing weight differences at various levels in deep learning models. The FedSGD method designed by Zhao et al. [19] improves model accuracy by perturbing gradients, but this comes at the cost of increased communication overhead.

The main challenges in federated learning include the following: (1) gradient clipping is required after adding noise, and improper clipping strategies can negatively affect model performance and convergence; (2) adding noise also increases communication overhead, especially in bandwidth-limited environments. Therefore, finding a balance between privacy protection and efficiency remains a pressing issue. Adaptive clipping strategies have shown clear advantages by dynamically adjusting the clipping threshold, significantly improving model accuracy and performance. Liu et al. [20] proposed an adaptive gradient clipping algorithm that optimizes model performance by dynamically adjusting the clipping threshold. Although it has achieved some success, the method is sensitive to parameter settings; improper settings may lead to excessive clipping or insufficient noise injection. The PEDPFL algorithm by Shen et al. [21] introduced regularization techniques to enhance model robustness, but the algorithm’s complexity may increase the computational burden during practical deployment.

In terms of improving communication efficiency, Liu et al. [22] developed the APFL algorithm, which assesses data contribution to model output via correlation propagation and injects noise based on these contributions, reducing the negative impact of noise on performance. However, this algorithm may increase computational complexity and extend training time when handling high-dimensional data. Wu et al. [23] improved the efficiency of client-side local gradient descent, but when there are significant differences in data distribution across clients, the stability of model training may be affected. The adaptive clipping framework proposed by Wang Fangwei et al. [24] improves model performance by dynamically adjusting the gradient clipping threshold. However, this method requires the precise tuning of clipping parameters in practical applications, or it may negatively impact convergence speed. Zhao et al. [25] combined self-sampling with an adaptive perturbation mechanism to effectively implement local differential privacy protection, but this increased communication costs. Additionally, Hu et al. [26] proposed the Fed-SMP algorithm, which enhances communication efficiency through model sparsification techniques. However, its impact is less significant in models with lower sparsity. The hierarchical federated learning method proposed by Lian et al. [27] reduces communication overhead by analyzing model correlations, but its generalizability is limited, and its applicability is restricted. Baek et al. [28] designed a robust differential privacy mechanism to address user dropout issues, but exhaustion of the privacy budget remains a potential risk in cases of large-scale user dropout.

Although these methods have made some progress in enhancing privacy protection and performance in federated learning, many limitations remain. This paper proposes the AWDP-FL framework, which combines adaptive clipping strategies with federated learning communication challenges to better address privacy protection issues. Its superiority over other algorithms is accurately analyzed through experiments, as detailed in

Table 1. Comparative analysis of AWDP-FL and other differential privacy federated learning algorithms.

Comparison Dimension	Comparison Algorithms	Methods Used by Other Algorithms	AWDP-FL Method	Algorithm Superiority
Clipping Threshold Selection	CDP-FL [22]	Uses a static clipping threshold, leading to high communication overhead.	Utilizes adaptive clipping threshold processing.	AWDP-FL is better suited for complex scenarios, allowing dynamic adjustments based on training conditions, improving model performance and convergence speed under a wider range of conditions with lower communication overhead.
Adaptive Update Steps	ADP-FL [24]	Based on the entire model, the traditional batch mean adaptation method is used, which averages the gradients of batches.	Single-sample gradient and weight analysis is used, introducing a dual historical sequence to determine weight coefficients, and processing is performed from a model hierarchy perspective.	AWDP-FL provides more granular control, suitable for complex models or scenarios requiring high-precision optimization, utilizing gradient information and historical changes at each layer more effectively.
Gradient Update Method	FDP-FL [23]	Uses traditional first-order moment momentum updates.	Employs higher-order moment analysis to reduce the impact of the initial learning rate.	The AWDP-FL algorithm is more refined, introducing higher-order information to mitigate the effects of the initial learning rate, making it particularly suitable for complex loss functions and high-precision optimization scenarios. This algorithm accelerates convergence in the later stages of model training and improves performance.

.

3. Preliminary

In this section, we will introduce the background knowledge relevant to this study. First, Section 3.1 will provide a detailed introduction to the federated learning (FL) model, helping readers gain a comprehensive understanding of the entire FL process. Next, Section 3.2 will delve into the concept of differential privacy and its application in FL models, introducing some key definitions commonly used in practical applications.

3.1. Federated Learning

The development of artificial intelligence has become a major trend, and the demand for high-quality data is increasing. These data often hold significant value, leading organizations and companies to be cautious about data sharing, which exacerbates the phenomenon of data silos. To address this issue, federated learning technology has emerged, allowing multiple participants to collaboratively build models without directly sharing data. Based on the distribution of participants’ data, federated learning can be divided into three types: horizontal federated learning, vertical federated learning, and federated transfer learning [29]. Among these, horizontal federated learning is suitable for participants with different data samples but similar features, while vertical federated learning is applicable when the participants have similar data samples but contain different features. By sharing only model parameters and other information, and collaboratively building models with the assistance of a central server, this approach reduces the risk of data leakage while maintaining model performance comparable to centralized machine learning. Federated learning primarily consists of a central server and N clients, denoted as $H=H_1,H_2,\cdots ,H_n$, where each client $H_i$ holds its dataset $M_i$. The data samples from all clients are represented as M, satisfying the condition ${\sum }_{i=1}^N{M}_i=\left|M\right|$. The global model parameters are obtained by weighted aggregation across all clients.

$$\theta ={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\theta }_i$$

(1)

The optimization problem in federated learning is as follows: $\min L(M,\theta )$, where the goal is to solve for the loss function and the minimal model parameters. In this paper, the loss function used is the cross-entropy loss function, which provides a finer-grained performance evaluation.

$$\underset{\theta }{min}L(M,\theta )={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}L(M_i,{\theta }_i)$$

(2)

3.2. Differential Privacy

The fundamental principle of differential privacy [30,31] is to add specific noise perturbations to the original data, ensuring that changes in a single sample do not significantly affect the distribution of the output results, protecting the sensitivity of individual data. By adding noise to datasets containing personal sensitive information, differential privacy effectively reduces the sensitivity of the data, transforming it into a more generalized dataset. Even if an attacker obtains certain data information, they cannot accurately identify the individual to whom the information belongs from the query results, protecting personal privacy. After adopting differential privacy techniques, data remain usable while ensuring privacy protection, effectively balancing the need for data protection and application.

In the application of differential privacy in federated learning, correctly setting the gradient clipping threshold is key to balancing model performance and privacy protection. If the clipping threshold is set too high, more noise may need to be added, which could reduce the model’s accuracy; if the threshold is set too low, too much information may be clipped, affecting the effectiveness of the gradients. Gradient clipping can be considered from two main aspects: first is value-based clipping, where all gradient values exceeding a fixed threshold are clipped to that threshold, precisely controlling the maximum values of all gradients in the model to ensure they do not exceed the set range. However, this method does not account for each gradient component, and when gradients are generally large, it may lead to the loss of critical gradient information. The second method is norm-based clipping, which adjusts the proportion of the entire gradient vector to meet the norm constraints, preserving the direction of the original gradient vector and helping retain more information about the optimization direction. This method is currently more mainstream, but there is still room for optimization in practical applications. This paper will explore this area in depth, aiming to further optimize gradient clipping methods to better balance privacy protection and model performance.

Definition 1 

(Adjacent Datasets). Given two datasets, M and $M_1$, when the distance between M and $M_1$ is 1, they are referred to as adjacent datasets.

$$\parallel M-M_1{\parallel }_2=1$$

(3)

The distance is defined as the sum of the absolute differences for each pair of records between the two datasets. If a corresponding record is missing in one dataset during the calculation of differences, 0 is used in place of the missing value, indicating that the two datasets are very similar.

Definition 2 

($(\epsilon ,\delta )$-Differential Privacy). Given a random algorithm R, if for any output set S, the following inequality holds for the adjacent datasets M and $M_1$, then algorithm R is said to satisfy $(ϵ,\delta )$-differential privacy:

$$Pr[R\left(M\right)\in S]\le Pr[R\left(M_1\right)\in S]·e^{\epsilon }+\delta$$

(4)

Here, $\mathrm{Pr}\left[R\right(M)\in S]$ denotes the probability that algorithm R, when applied to dataset M, produces an output that falls within set S. The privacy budget $ϵ$ controls the ratio of output probabilities between adjacent datasets. The smaller the $ϵ$, the closer the output probabilities of the two datasets, which indicates stronger privacy protection, making it more difficult for an attacker to distinguish which dataset the data came from. The parameter $\delta$ defines the probability of violating $\left(ϵ\right)$-differential privacy, providing flexibility in privacy protection. When $\delta =0$, the algorithm satisfies $\left(ϵ\right)$-differential privacy, representing a stricter privacy guarantee standard.

Definition 3 

(Sensitivity). The sensitivity of a function describes the maximum possible change in the function’s output when a data point is added or removed from the dataset, measuring the function’s responsiveness to changes in a single data item. For two datasets, M and $M_1$, differing by only one element, the sensitivity is determined by calculating the maximum possible difference in the output of function R between these two datasets, specifically,

$$\Delta s=\underset{M,M_1}{max}{\parallel R\left(M\right)-R\left(M_1\right)\parallel }_2$$

(5)

In the Gaussian mechanism, if the $L_2$ sensitivity of function R is $\Delta s$ (measured using Euclidean distance), $(ϵ,\delta )$-differential privacy is achieved by adding Gaussian-distributed random noise with mean 0 and variance ${\sigma }^2$ to the result.

Definition 4 

(Composition Theorem). Suppose there is a set of algorithms $R_1\left(M\right),R_2\left(M\right),\cdots ,R_n\left(M\right)$, each satisfying $(ϵ,\delta )$-differential privacy. When combined into a new algorithm $R_{\forall }=(R_1\left(M\right),R_2\left(M\right),\cdots ,R_n\left(M\right))$, the entire composite algorithm $R_{\forall }$ will also satisfy $(ϵ,\delta )$-differential privacy. The privacy of individual data points in the composite algorithm remains protected with the combination possibly being serial, parallel, etc.

Definition 5 

(Post-processing Immunity [30]). If algorithm R satisfies $(ϵ,\delta )$-differential privacy for the same dataset M, then even if another random algorithm A (which may not follow differential privacy principles) forms a new algorithm $B=A\left(R\right(M\left)\right)$ by applying A to the output of R, the new algorithm B will still satisfy $(ϵ,\delta )$-differential privacy.

4. Weighted Adaptive Differential Privacy Federated Learning Framework

To protect the privacy of client data in federated learning, this paper combines an adaptive weight clipping threshold selection strategy to design a novel weighted adaptive differential privacy federated learning framework (Algorithm 1). Unlike traditional federated learning frameworks, this framework introduces improvements in the way model gradients are handled with all client operations being performed from the neural network layer perspective. During each local training iteration, clients first calculate the threshold for adaptive gradient clipping using weight coefficients and then apply adaptive gradient clipping to limit the magnitude of the gradients, which is followed by adaptive gradient updates to the model, and they finally add dynamic Gaussian noise when uploading the parameters. This federated learning framework emphasizes an optimal balance between data privacy protection and model performance during communication, offering new perspectives and methods for privacy protection in federated learning. The communication process between the client and server is illustrated in Figure 1.

Algorithm 1 Weighted Adaptive Differential Privacy Federated Learning Framework

Input: Initial model parameters ${\theta }_t$, learning rate $\alpha$, client set O, randomly selected clients for training H, communication rounds between client and server T, number of neural network layers L, batch size B, client data M, adaptive clipping history norm sequence P, number of local iterations E, momentum update parameters ${\tau }_1^t$, ${\tau }_2^t$, $\rho$. Output: Model parameters ${\tilde{\theta }}_t$   1: for  $t=1$  to  T  do   2:         # Client Side   3:         for $h=1$ to H do   4:               Init ${\theta }_t\leftarrow {\tilde{\theta }}_t$ // Broadcast global parameters to update the local model   5:               for $e=1$ to E do   6:                      for $B_{i,h}\in M_h$ do // i is an integer from 1 to $\lceil M/B\rceil$     7:                            $G_{t,l}\leftarrow g_t\left(x_t\right)$     8:                            ${\tilde{G}}_{t,l}\leftarrow \mathrm{WDP}\left(G_{t,l}\right)$     9:                            ${\theta }_t\leftarrow \mathrm{AGU}\left({\tilde{G}}_{t,l}\right)$ 10:                      end for 11:               end for 12:               ${\tilde{\theta }}_t^h\leftarrow {\theta }_t+N(0,{\sigma }^2)$ // Noise perturbation 13:         end for 14:         # Server Side 15:         ${\tilde{\theta }}_t\leftarrow {\displaystyle \frac{1}{\left|H\right|}}{\sum }_{h=1}^H{\tilde{\theta }}_t^h$ // Server-side parameter aggregation 16: end for

The relevant symbols and parameters involved in this paper are shown in

Table 2. Relevant symbols and parameters.

Symbol	Meaning
$\epsilon$	Privacy budget in the definition of local differential privacy
T	Total communication rounds in federated learning
q	Client sampling rate during federated learning
$\alpha$	Learning rate of the federated learning client
$\theta$	Federated learning model parameters
C	Local iterations of the federated learning client
B	Local batch size of the federated learning client
${\tau }_1^t$	Momentum parameter for client updates in federated learning
${\tau }_2^t$	Momentum parameter for client updates in federated learning
$\rho$	Momentum parameter for client updates in federated learning
M	Datasets used by clients participating in federated learning training
L	Neural network layers
P	Selected value from the adaptive clipping history norm sequence
O	Federated learning client set

.

4.1. Workflow of the Framework

The framework consists of four main steps:

• Local Model Training: Clients must perform multiple gradient descents locally to reduce communication with the server. First, in each local iteration, the adaptive gradient clipping WDP (Algorithm 2), based on model weights, is used for gradient clipping, which enhances the model’s robustness. Next, the adaptive gradient update AGU (Algorithm 3) is employed to determine the result of the model training in this iteration. Notably, this algorithm does not need to consider the training results of other clients globally; clients operate independently without affecting each other. Detailed steps will be presented in the next section.
• Parameter Upload: To protect client privacy, hierarchical noise perturbation must be added to the model parameters before uploading. The model parameters processed by the WDP and AGU algorithms can reduce the noise impact caused by individual samples. This stability helps to smooth the loss function descent, accelerating convergence while minimizing the effect of noise on model parameters. The noise addition steps are as follows (using a single client as an example):

  $$\begin{array}{cc}\hfill \left({\tilde{\theta }}_t^h\right)& =\{\left({\tilde{\theta }}_{t,l1}^h\right),\left({\tilde{\theta }}_{t,l2}^h\right),\dots ,\left({\tilde{\theta }}_{t,ln}^h\right)\}\ni \hfill \\ \hfill & {\theta }_{t,l}-\alpha ·{\displaystyle \frac{{\widehat{V}}_{t,l,d,e}^1}{\rho +\sqrt[3]{{\widehat{V}}_{t,l,d,e}^3}}}(d=|M|,e=|E\left|\right)+N(0,{\sigma }_l^{t^2})\hfill \\ \hfill & ={\theta }_{t,l}-\alpha ·\sum _{e=1}^E{\displaystyle \frac{{\widehat{V}}_{t,l,d,e}^1}{\rho +\sqrt[3]{{\widehat{V}}_{t,l,d,e}^3}}}(d\in M)\hfill \\ \hfill & ={\theta }_{t,l}-\alpha ·\sum _{e=1}^E\sum _{d=1}^{\lceil M/B\rceil }{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{t,l,d,e}^3}}}(d=B)\hfill \\ \hfill & ={\theta }_{t,l}-{\displaystyle \frac{\alpha }{\left|B\right|}}·\sum _{e=1}^E\sum _{d=1}^{\lceil M/B\rceil }\sum _{b=1}^B{\displaystyle \frac{{\widehat{V}}_{t,l,d,e}^1}{\rho +\sqrt[3]{{\widehat{V}}_{t,l,d,e}^3}}}\hfill \end{array}$$

  (6)
  Here, L represents the neural network layer level, which serves as the basis for all operations in this paper. $({\tilde{\theta }}_t^h)$ denotes the noise-perturbed model parameters, ${\theta }_{(t,l)}$ are the unperturbed model parameters, M represents the client data, B are the training data for each batch, t is the current global round, d represents the current training data, and e is the current batch’s local iteration. ${\widehat{V}}_{(t,l,d,e)}^1/\left(\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}\right)$ represents the model gradient of the l-th layer in the t-th round after the adaptive momentum update. $N{(0,{\sigma }^2)}_l^t$ is the noise perturbation added before uploading, which satisfies $N{(0,{(\Delta s\sigma )}^2)}_l^t$, where $\Delta s_l^t$ is the sensitivity determined by the adaptive clipping threshold.
• Global Parameter Aggregation: After the local training of sampled clients is completed, the server performs weighted aggregation of the noise-perturbed parameters uploaded by the clients to update the global model for the next round, specifically as follows:

  $${\tilde{\theta }}_t=\sum _{h=1}^H{\tilde{\theta }}_t^h$$

  (7)
• Global Parameter Distribution: In the federated learning framework, the server employs a selective parameter broadcasting strategy. Specifically, the server does not broadcast the latest model parameters to all clients every time; instead, it randomly selects a portion of clients for parameter updates. This approach avoids redundant broadcasting and reduces unnecessary communication overhead. During this process, the server does not need direct access to the clients’ local data. After receiving the global model from the server, clients adaptively update their local models based on these parameters, maintaining the efficiency of model updates.

Client noise perturbation depends on sensitivity, sensitivity depends on the determination of the clipping threshold, and the clipping threshold depends on adaptive weight selection. Gradient clipping ensures that the model norm remains within a certain range. Specifically,

$$L_{t,l}={\parallel g_t\left(x_t\right)\parallel }_2\le {\mathrm{Clip}}_{t,l}$$

(8)

Thus, the sensitivity calculation is as shown in Lemma 1:

Lemma 1. 

${\tilde{\theta }}_t^h(d=|M\left|\right)$ represents the model parameters of client h after local training on a dataset M of size $\left|M\right|$. Therefore, the sensitivity is

$$\Delta s_l^t=∥{\tilde{\theta }}_t^h(d=|M\left|\right)-{\tilde{\theta }}_t^h(d=|M^{\prime }\left|\right)∥\le 2\alpha {\mathrm{Clip}}_{t,l}{\displaystyle \frac{E}{B}}$$

(9)

Proof. 

${\tilde{\theta }}_t^h(d=|M\left|\right)$ represents the model parameters obtained after E local iterations from ${\theta }_t^h(d=|M\left|\right)$. $\left|M\right|$ and ${\left|M\right|}^{\prime }$ are adjacent datasets of client h with their only difference being in batch B from $\left|M\right|$ and batch $B^{\prime }$ from ${\left|M\right|}^{\prime }$. Therefore, the standard deviation of the hierarchical Gaussian noise to be added before uploading the model parameters can be calculated as follows:

$$\begin{array}{cc}\hfill \Delta s_l^t& ={∥{\tilde{\theta }}_t^h(d=|M\left|\right)-{\tilde{\theta }}_t^h(d=|M^{\prime }\left|\right)∥}_2\hfill \\ \hfill & ={∥{\theta }_{t,l}-{\displaystyle \frac{\alpha }{\left|B\right|}}·\sum _{e=1}^E\sum _{d=1}^{\lceil M/B\rceil }\sum _{b=1}^B{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}-{\theta }_{t,l}-{\displaystyle \frac{\alpha }{\left|B\right|}}·\sum _{e=1}^E\sum _{d=1}^{\lceil M^{\prime }/B\rceil }\sum _{b=1}^B{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}∥}_2\hfill \\ \hfill & ={\displaystyle \frac{\alpha }{\left|B\right|}}{∥\sum _{e=1}^E\left(\sum _{d=1}^{\lceil M/B\rceil }\sum _{b=1}^B{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}-\sum _{d=1}^{\lceil M^{\prime }/B\rceil }\sum _{b=1}^B{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}\right)∥}_2\hfill \\ \hfill & ={\displaystyle \frac{\alpha }{\left|B\right|}}\sum _{e=1}^E{∥{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}(d=|M\left|\right)-{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}(d=|M^{\prime }\left|\right)∥}_2\hfill \\ \hfill & \le {\displaystyle \frac{\alpha }{\left|B\right|}}\left(\sum _{e=1}^E{∥{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}(d=|M\left|\right)∥}_2+\sum _{e=1}^E{∥{\displaystyle \frac{{\widehat{V}}_{(t,l,d,e)}^1}{\rho +\sqrt[3]{{\widehat{V}}_{(t,l,d,e)}^3}}}(d=|M^{\prime }\left|\right)∥}_2\right)\hfill \\ \hfill & \le {\displaystyle \frac{2\alpha {\mathrm{Clip}}_{t,l}E}{B}}\hfill \end{array}$$

(10)

   □

Algorithm 2 Weight-Based Adaptive Gradient Clipping (WDP)

Input: Unupdated model parameters after training ${\theta }_t$, number of neural network layers L, communication rounds between client and server T, batch size B, client data M, adaptive clipping history norm sequence P. Output: Processed model parameters ${\tilde{\theta }}_t$.   1: $\overline{G_{t,l}}={\displaystyle \frac{1}{\left|B\right|}}{\sum }_{t=1}^B{G}_{t,l}$     2: $\overline{{\mathrm{GB}}_{t,l}}={\displaystyle \frac{1}{\left|m\right|}}{\sum }_{m=1}^n\overline{G_{B_m,l}}$(m ≥ 2)     3: # Compute layer-specific weight coefficients     4: ${\mathrm{GW}}_{t,l}=cos\left(\overline{G_{t,l}},\overline{{\mathrm{GB}}_{t,l}}\right)={\displaystyle \frac{\overline{G_{t,l}}·\overline{{\mathrm{GB}}_{t,l}}}{\parallel \overline{G_{t,l}}\parallel ·\parallel \overline{{\mathrm{GB}}_{t,l}}\parallel }}$     5: # Compute weighted norm     6: ${\mathrm{realL}}_{t,l}={\displaystyle \frac{\left({max}_{0\le t\le B,0\le l\le L}{L}_{t,l}-\frac{1}{\left|B\right|}{\sum }_{t=1}^B{L}_{t,l}\right)}{2·{\mathrm{GW}}_{t,l}}}+{\displaystyle \frac{1}{\left|B\right|}}{\sum }_{t=1}^B{L}_{t,l}$     7: # Compute clipping threshold:     8: ${\mathrm{Clip}}_{t,l}={\left\{LL\right\}}_P={\{{\mathrm{realL}}_{1,l},{\mathrm{realL}}_{(2,l)},{\mathrm{realL}}_{(3,l)},\dots ,{\mathrm{realL}}_{(M,l)}\}}_P$     9: # Layer-specific gradient clipping   10: ${\tilde{G}}_{t,l}={\displaystyle \frac{1}{\left|B\right|}}{\sum }_{t=1}^B{G}_{t,l}/max\left(1,{\displaystyle \frac{L_{t,l}}{{\mathrm{Clip}}_{t,l}}}\right)$

Algorithm 3 Adaptive Gradient Update (AGU)

Input: Adaptively clipped model parameters ${\tilde{G}}_{t,l}$, learning rate $\alpha$, communication rounds between client and server T, number of neural network layers L, momentum update parameters ${\tau }_1^t$, ${\tau }_2^t$, $\rho$. Output: Model parameters ${\tilde{\theta }}_t$. 1: # Gradient Update   2: $V_t^1\leftarrow {\tau }_1^t·V_{t-1}^1+(1-{\tau }_1^t)·{\tilde{G}}_{t,l}$// Moment Estimation   3: $V_t^3\leftarrow {\tau }_3^t·V_{t-1}^3+(1-{\tau }_3^t)·{\tilde{G}}_{t,l}^3$   4: $\widehat{V_t^1}\leftarrow {\displaystyle \frac{V_t^1}{1-{\tau }_1^t}}$// Corrected Moment Estimation   5: $\widehat{V_t^3}\leftarrow {\displaystyle \frac{V_t^3}{1-{\tau }_3^t}}$   6: ${\theta }_t\leftarrow {\theta }_{t-1}-\alpha ·{\displaystyle \frac{\widehat{V_t^1}}{\rho +\sqrt[3]{\widehat{V_t^3}}}}$// Adaptive Gradient Update

4.2. Weight-Based Adaptive Gradient Clipping and Update

In traditional methods, the gradient clipping threshold is usually determined based on the researchers’ experience and pretraining on public datasets. This fixed threshold approach aims to ensure model convergence and performance but fails to account for differences between various layers of the neural network, which can negatively impact model performance in certain cases. With technological advancements, a new approach has been developed that dynamically adjusts the clipping threshold using the $L_2$ norm percentile of historical gradients, allowing for more flexible responses to gradient changes. However, despite this method’s increased flexibility in threshold adjustment, it may not effectively address situations where model training performs poorly, leading to wasted computational resources during the update process and a lack of personalized settings for different network layers. To address this issue, this paper proposes a weight-based adaptive gradient clipping and update algorithm with the detailed implementation steps as follows:

A.

Layer-wise Adaptive Weight Coefficients

Calculate the gradient and $L_2$ norm for each neural network layer during each iteration, where $l\in \{1,2,3,\cdots ,L\}$, and L is the number of neural network layers.

$$G_{t,l}=g_t\left(x_t\right)$$

(11)

$$L_{t,l}={\parallel g_t\left(x_t\right)\parallel }_2$$

(12)

Compute the mean of all gradients for each layer during each iteration as the preprocessing weight parameter ${\tau }_1$.

$$\overline{G_{t,l}}={\displaystyle \frac{1}{\left|B\right|}}\sum _{t=1}^B{G}_{t,l}$$

(13)

Add this value layer-wise to a sequence, generating a new weight preprocessing historical gradient sequence $GB$.

$$GB=\{\overline{G_{B_1,l}},\overline{G_{B_2,l}},\overline{G_{B_3,l}},\dots ,\overline{G_{B_m,l}}\}$$

(14)

Calculate the mean of $GB$ layer-wise as the preprocessing weight parameter ${\tau }_2$.

$$\overline{G{B}_{t,l}}={\displaystyle \frac{1}{\left|m\right|}}\sum _{m=1}^n\overline{G_{B_m,l}}(m\ge 2)$$

(15)

Use the local weight parameter ${\tau }_1$ and the global weight parameter ${\tau }_2$ to calculate the weight coefficient $G{W}_{B,l}$ layer-wise according to the following formula.

$$G{W}_{t,l}=cos(\overline{G_{t,l}},\overline{G{B}_{t,l}})={\displaystyle \frac{\overline{G_{t,l}}·\overline{G{B}_{t,l}}}{\parallel \overline{G_{t,l}}\parallel \parallel \overline{G{B}_{t,l}}\parallel }}$$

(16)

B.

Adaptive Clipping Threshold Selection

In this framework, a high cosine similarity between two updates indicates that the direction of gradient updates remains consistent, suggesting a stable training process. Conversely, low or negative similarity may indicate issues such as gradient explosion or vanishing gradients. Here, B represents the batch size, and M is the number of training data points in the client.

Weight the norm of each iteration according to the layer-wise weight coefficient $G{W}_{B,l}$.

$$\mathrm{real}{L}_{t,l}=\left(\underset{0\le t\le B,0\le l\le L}{max}{L}_{t,l}-{\displaystyle \frac{1}{\left|B\right|}}\sum _{t=1}^B{L}_{t,l}\right)/2·G{W}_{t,l}+{\displaystyle \frac{1}{\left|B\right|}}\sum _{t=1}^B{L}_{t,l}$$

(17)

Add the weighted gradient norms of this batch to a sequence layer-wise, resulting in the threshold processing historical norm list $LL$.

$$LL=\{\mathrm{real}{L}_{1,l},\mathrm{real}{L}_{2,l},\mathrm{real}{L}_{3,l},\dots ,\mathrm{real}{L}_{M,l}\}$$

(18)

Calculate the percentile P of the $LL$ sequence layer-wise as the clipping threshold for this iteration.

$${\mathrm{Clip}}_{t,l}={\left\{LL\right\}}_P={\{\mathrm{real}{L}_{1,l},\mathrm{real}{L}_{2,l},\mathrm{real}{L}_{3,l},\dots ,\mathrm{real}{L}_{M,l}\}}_P$$

(19)

Perform layer-wise gradient clipping on each data point in this iteration.

$$\mathrm{real}{G}_{t,l}={\displaystyle \frac{G_{t,l}}{max(1,L_{t,l}/{\mathrm{Clip}}_{t,l})}}$$

(20)

Take the mean layer-wise as the gradient for this batch iteration, preparing for the gradient update.

$$\widetilde{G_{t,l}}={\displaystyle \frac{1}{\left|B\right|}}\sum _{t=1}^B\mathrm{real}{G}_{t,l}$$

(21)

C.

Adaptive Momentum Gradient Updates

Initialize the first-order moment $V_t^1$ and third-order moment $V_t^3$ based on historical gradients, and define momentum decay parameters ${\tau }_1^t$, ${\tau }_3^t$, and hyperparameter $\rho$. The third-order moment measures the symmetry of the data distribution; if the data distribution is symmetric, the skewness is zero. If the distribution is skewed to the left or right, the skewness is non-zero. Using the first-order and third-order moments based on historical gradients allows the parameter update to accumulate past gradient directions, helping to escape local minima or saddle points. Momentum aids the algorithm in accelerating toward the optimal solution and reduces oscillations in directions with smaller gradients, speeding up convergence.

Calculate the first-order moment estimate:

$$V_t^1={\tau }_1^t·V_{t-1}^1+(1-{\tau }_1^t)·{\tilde{G}}_{t,l}$$

(22)

Calculate the third-order moment estimate:

$$V_t^3={\tau }_3^t·V_{t-1}^3+(1-{\tau }_3^t)·{\left({\tilde{G}}_{t,l}\right)}^3$$

(23)

Based on the initial gradient estimates, which often have large variance and bias, the correction mechanism adjusts the momentum estimates to more accurately reflect the true gradient information.

Correct the first-order moment:

$${\widehat{V}}_t^1={\displaystyle \frac{V_t^1}{1-{\tau }_1^t}}$$

(24)

Correct the third-order moment:

$${\widehat{V}}_t^3={\displaystyle \frac{V_t^3}{1-{\tau }_3^t}}$$

(25)

Update the model parameters:

$${\theta }_t={\theta }_{t-1}-\alpha ·{\displaystyle \frac{{\widehat{V}}_t^1}{\rho +\sqrt[3]{{\widehat{V}}_t^3}}}$$

(26)

The third-order moment historical gradient processing provides higher-dimensional data analysis capabilities, dynamically refining gradient information. This enables a better utilization of model gradients after weighted adaptive gradient clipping, and adaptive momentum updates produce more complex model parameters, enhancing model performance. For parameter selection, we recommend setting ${\tau }_1^t$ to 0.9, as it controls the first-order momentum update, allowing a quick response to changes in gradient direction, making it suitable for capturing newer information. Set ${\tau }_3^t$ to 0.99, as the calculation involves dual historical gradient sequences with more dimensional information, requiring finer distinctions. We recommend selecting a hyperparameter $\rho$ no higher than 0.2, as it controls the adjustment of certain weights or update steps, which should be based on the complexity of the specific dataset.

4.3. Privacy and Security Analysis

• Privacy Analysis

Client Parameter Upload Phase: In the t-th round of local training, each participating client h adds Gaussian noise (with a mean of 0 and a standard deviation of $\sigma$) to the model parameters ${\theta }_t^h$ obtained by minimizing the loss function, where ${\theta }_t^h=argminL(M_h,{\theta }_t)$, and ${\theta }_t$ is the global model parameter in the t-th round. The standard deviation $\sigma$ is calculated according to Equation (10), $\Delta s_l^t\le 2\alpha {\mathrm{Clip}}_{t,l}{\displaystyle \frac{E}{B}}$, where $\Delta s_l^t$ represents the sensitivity of client i in the l-th layer during the t-th round. In terms of privacy analysis, the client uploads the noise-added model parameters $\left({\tilde{\omega }}_t^i\right)$ to the server, and the client satisfies $(ϵ,\delta )$-differential privacy.

Server Parameter Distribution Phase: In each round, the server performs weighted aggregation of the noise-added model parameters from the participating clients and updates the global model. According to Differential Privacy Definition 5 (Post-processing Immunity [30]), the server’s parameter distribution process in each round satisfies $(ϵ,\delta )$-differential privacy. After T rounds of communication between the client and server, the client’s dataset has been accessed T times in sequence, forming a parallel sequence combination. According to Differential Privacy Definition 4 (Composition Theorem), the privacy budget equals the maximum of all client privacy budgets, i.e., $Tϵ$. The algorithm satisfies $(Tϵ,T\delta )$-differential privacy.

• Security Analysis

Our algorithm can effectively resist Membership Inference Attacks (MIAs) and Deep Leakage from Gradients (DLG) attacks. This is because the AWDP-FL framework uses a batch-by-batch training method and clips the gradients of each sample. For poorly performing samples, the algorithm can promptly remove their influence on gradient updates, while the global model parameters still retain the information from well-performing samples.

The goal of a membership inference attack is to infer whether a sample was involved in model training. Since our method removes the influence of poorly performing samples during gradient clipping, if an attacker attempts to identify these samples through membership inference, the results are likely to be misleading because these samples contribute little or nothing to the model weights. In other words, unlike traditional batch averaging methods, our algorithm can exclude samples detrimental to training, reducing the accuracy of membership inference attacks. This sample selection and clipping mechanism not only enhances the robustness of the algorithm but also greatly reduces the success rate of MIA attacks, further improving privacy protection and security defenses.

DLG attacks attempt to recover the original training data by reverse engineering the gradient information uploaded by clients. Our algorithm clips and perturbs gradients during training, so the gradients uploaded to the server no longer directly reflect the original data, weakening the attacker’s ability to infer the original data through DLG attacks.

Regarding DLG attacks, we designed specific defense measures in the experimental Section 5.2.3 and experimentally validated the security of AWDP-FL in countering DLG attacks. Both theoretical analysis and experiments demonstrate the robustness and security of our algorithm against various attack methods.

4.4. Algorithm Complexity Analysis

This section will analyze the time complexity of the AWDP-FL framework. The framework consists of four processes with T communication rounds and $O·q$ clients participating in the training. During each client–server communication, the time complexity of the AWDP-FL client is $O(log(O·q\left)\right)$, so the total communication time complexity of AWDP-FL is $O(Tlog(O·q\left)\right)$.

It is worth noting that the framework recommends using the parameter settings from Section 5.2, such as the values for q and E (see Table 2 for symbol meanings). These settings can effectively reduce communication overhead while maintaining high model accuracy with fewer communication rounds. However, this configuration will significantly increase the client’s training time. As the depth of the neural network model increases, the memory and GPU memory requirements will also rise significantly.

5. Experiments

5.1. Experiment Settings

• Datasets: Three public datasets are used

-

MNIST Dataset: Based on a single-channel handwritten digit recognition dataset, it contains images of digits from 0 to 9, with each image being a 28 × 28 pixel grayscale image. The dataset includes 60,000 images for training and 10,000 images for testing.

-

Fashion-MNIST: Based on a single-channel fashion classification dataset, it contains 70,000 grayscale images of 10 different types of clothing with each image being 28 × 28 pixels. The classification for testing and validation is the same as the MNIST dataset but is more challenging.

-

CIFAR-10 Dataset: Based on three-channel color images of 10 classes, such as ships, airplanes, cars, etc., each image is 32 × 32 pixels. The dataset includes 50,000 images for training and 10,000 images for testing.

• Models

The network models for the MNIST and Fashion-MNIST datasets consist of one convolutional layer and one fully connected layer with 256 output neurons. A ReLU activation layer is also applied, utilizing the built-in activation function to enhance the model’s nonlinear expression capabilities. For the CIFAR-10 dataset, two network architectures were used. The first model includes two convolutional layers, which are each followed by a ReLU activation layer to enhance feature extraction and reduce data dimensionality. Two fully connected layers are then employed to transform the features, converting 384 input features into 192 output features, further extracting and combining useful information. The second model uses four convolutional layers with a dropout layer and ReLU activation function added after every two convolutional layers. A MaxPool layer is then applied to halve the spatial dimensions of the feature map, retaining the most significant features. Finally, the model completes the classification task through three fully connected layers.

• Experimental Setup

The experiments were conducted using the Pytorch framework on an NVIDIA GeForce RTX 4090 server with a CPU configuration of 16 vCPUs (Intel(R) Xeon(R) Gold 6430). The results were averaged over multiple tests.

5.2. Experimental Results and Analysis

The experimental analysis in this section is divided into two parts, each employing different comparison methods. Below is a detailed description of both parts.

Part One: We compare the AWDP-FL algorithm with the base federated learning algorithm without differential privacy (NoDP-FL) [29] and three advanced differential privacy federated learning algorithms across three different datasets: (1) Client-level Differential Privacy Federated Learning (CDP-FL) [22]; (2) Fadam Differential Privacy Federated Learning (FDP-FL), which is based on adaptive learning rates [23]; and (3) Adaptive Differential Privacy Federated Learning (ADP-FL), which uses adaptive clipping thresholds [24]. To fully reflect the robustness of each algorithm, the evaluation metrics for this part of the experiment include the following: Section 5.2.1, which assesses model accuracy and loss values across the three datasets; Section 5.2.2, which evaluates algorithm usability based on privacy budget consumption, i.e., the privacy budget required to achieve the same model accuracy; and Section 5.2.3, which assesses the privacy protection capabilities of different methods on the three datasets using the traditional DLG attack.

Part Two: We explore the optimal parameter configurations for the AWDP-FL algorithm in practical applications and compare its performance with the base federated learning algorithm without differential privacy (NoDP-FL). Section 5.2.4 analyzes the impact of local iteration counts on model accuracy and communication costs; Section 5.2.5 discusses the effect of sampling rate on model accuracy; and Section 5.2.6 focuses on the adaptive hierarchical clipping strategy within the AWDP-FL algorithm, particularly its effectiveness in deeper neural networks. Each section provides a detailed description of the relevant experimental parameter settings.

5.2.1. Analysis of Algorithm Model Accuracy and Loss Values

Model Accuracy Analysis: Model accuracy reflects the model’s classification ability on the test dataset, that is, the proportion of samples correctly classified by the model. Higher accuracy indicates good classification performance on the test set, providing an intuitive representation of the model’s overall performance. Regarding privacy budget selection, we did not use a single fixed value but chose an arithmetic sequence. For example, in the MNIST dataset, the privacy budget sequence was set to 0.4, 0.6, and 0.8. The purpose of this sequence is to clearly demonstrate the robustness of different algorithms. If the model accuracy fluctuates little within this privacy budget range, it indicates that the algorithm still performs stably under a lower privacy budget. Conversely, if there are large fluctuations, it suggests that the algorithm has reached the maximum privacy budget it can accommodate. By using a privacy budget sequence, we can more clearly demonstrate the performance of each algorithm under different conditions, allowing us to determine the critical privacy budget for the dataset. In this experiment, we selected 50 clients, and model accuracy is expressed as a percentage. The parameters for all algorithms were kept consistent: the number of local iterations E was set to 3, batch size B was 64, and the number of global communication rounds T was 50 for the MNIST dataset, while it was 100 for the Fashion-MNIST and CIFAR-10 datasets. Next, we will analyze the experimental results for each dataset in detail.

In the MNIST dataset, the number of communication rounds was set to 50, and fewer communication rounds help reduce the consumption of the privacy budget. The privacy budget was selected within the range of 0.4 to 0.8, where a smaller budget provides a higher level of privacy protection. As shown in

Table 3. Comparison of model accuracy for the MNIST dataset across different algorithms and privacy budgets.

Algorithm	$\mathit{\epsilon }=0.4$	$\mathit{\epsilon }=0.6$	$\mathit{\epsilon }=0.8$
NoDP-FL [29]	89.84
CDP-FL [22]	78.01	79.32	79.85
ADP-FL [24]	83.64	83.80	84.01
FDP-FL [23]	79.25	82.48	84.17
AWDP-FL	87.59	87.96	88.10

, the model accuracy of the NoDP-FL algorithm was $89.84\%$ for privacy budgets of 0.4 and 0.8, while the model accuracy of the AWDP-FL algorithm was $87.59\%$ and $88.10\%$, respectively. This indicates that although the reduction in the privacy budget increases the perturbation, the accuracy drop for AWDP-FL is minimal, with the error remaining within $1.74\%$. Under the same conditions, the highest accuracy of FDP-FL was $84.17\%$, showing that AWDP-FL outperformed other algorithms by approximately $4\%$, demonstrating a good balance between privacy protection and model performance.

Figure 2 shows the convergence trends of each algorithm on the MNIST dataset. It can be observed that the NoDP-FL algorithm converges rapidly within 10 to 20 rounds with its fast convergence attributed to the absence of noise perturbation, thus providing limited privacy protection. The AWDP-FL algorithm gradually converges after 20 rounds, showing a smooth curve, while other algorithms require more than 40 rounds to reach a convergence trend. In federated learning, the smoothness of the model accuracy curve over consecutive rounds indicates the stability and robustness of the model during the training process. This demonstrates that the AWDP-FL algorithm, while introducing privacy protection, is still able to maintain efficient model convergence with minimal accuracy fluctuations, reflecting its good convergence performance and robustness.

In the Fashion-MNIST dataset, the number of communication rounds was set to 100, and the privacy budgets were 0.4, 0.6, and 0.8, respectively. As shown in

Table 4. Comparison of model accuracy for the Fashion-MNIST dataset across different algorithms and privacy budgets.

Algorithm	$\mathit{\epsilon }=0.4$	$\mathit{\epsilon }=0.6$	$\mathit{\epsilon }=0.8$
NoDP-FL [29]	84.53
CDP-FL [22]	77.01	77.12	77.58
ADP-FL [24]	79.02	79.40	79.45
FDP-FL [23]	79.12	79.46	79.52
AWDP-FL	82.13	82.76	83.22

, the NoDP-FL algorithm quickly converged within fewer rounds with a model accuracy of $84.53\%$. As the privacy budget increases, AWDP-FL gradually approaches the accuracy of NoDP-FL. When the privacy budget is 0.8, AWDP-FL achieves an accuracy of $83.22\%$, which is only $1.31\%$ lower than NoDP-FL. In contrast, the highest accuracy for FDP-FL, ADP-FL, and CDP-FL is $79.52\%$, $79.45\%$, and $77.58\%$, respectively. AWDP-FL outperformed the other algorithms by approximately $5.64\%$ in terms of model accuracy, proving its ability to maintain excellent classification performance even under a high privacy budget.

Figure 3 shows the convergence curves of each algorithm on the Fashion-MNIST dataset. It can be observed that AWDP-FL’s convergence speed significantly outperforms other algorithms as the number of communication rounds increases, achieving convergence within 70 to 80 rounds. In contrast, the other algorithms only begin to stabilize after 90 rounds. The smoothness of the AWDP-FL curve indicates that as the communication rounds increase, the algorithm is able to quickly and stably converge while maintaining strong privacy protection, thus reducing communication time and resource consumption.

In the CIFAR-10 dataset, the number of communication rounds was also set to 100, and the privacy budgets were chosen as 4, 5, and 6. Since the CIFAR-10 dataset contains three-channel color images, the training process is more complex. As shown in

Table 5. Comparison of model accuracy for the CIFAR-10 dataset across different algorithms and privacy budgets.

Algorithm	$\mathit{\epsilon }=4$	$\mathit{\epsilon }=5$	$\mathit{\epsilon }=6$
NoDP-FL [29]	76.99
CDP-FL [22]	53.05	53.37	53.65
ADP-FL [24]	66.72	67.45	70.00
FDP-FL [23]	59.44	61.56	61.87
AWDP-FL	70.00	72.00	74.42

, the AWDP-FL algorithm achieved a model accuracy of $74.42\%$ when the privacy budget was set to 6, significantly outperforming other algorithms. In comparison, NoDP-FL achieved a model accuracy of $76.99\%$ under the same privacy budget, while FDP-FL’s highest accuracy at $ϵ$ = 6 was $61.87\%$. AWDP-FL not only maintained a high accuracy under privacy protection but also significantly outperformed other differential privacy algorithms, particularly surpassing FDP-FL by $12.55\%$. The performance of AWDP-FL surpassed other algorithms across all privacy budgets, demonstrating its ability to balance privacy budget and model performance effectively.

Figure 4 shows the convergence trends of each algorithm on the CIFAR-10 dataset with 100 communication rounds. At a privacy budget of 6, the AWDP-FL algorithm exhibits higher convergence during the first 20 training rounds, slightly outperforming NoDP-FL in accuracy between rounds 5 and 20. The rapid convergence of AWDP-FL is especially apparent in the early stages of training, indicating that the algorithm can quickly optimize model performance in complex tasks. As the number of communication rounds increases, the accuracy gap between AWDP-FL and NoDP-FL gradually narrows with a final difference of $2.57\%$. In contrast, other differential privacy algorithms such as CDP-FL, ADP-FL, and FDP-FL have noticeably slower convergence speeds and much lower final accuracy compared to AWDP-FL. The smooth curve and significant accuracy improvement of AWDP-FL indicate its strong robustness and fast convergence ability in federated learning. A stable convergence curve means that the algorithm can maintain performance stability even when faced with privacy protection requirements, reducing fluctuations during training. This further demonstrates the algorithm’s superiority on complex datasets like CIFAR-10.

Model Loss Value Analysis In many practical applications, accuracy is often the preferred performance metric. However, in datasets with class imbalance, accuracy can lead to misleading conclusions. For example, in a dataset where $90\%$ of the samples belong to class A and $10\%$ belong to class B, a model that always predicts class A would still have $90\%$ accuracy, but clearly, it is not a good model. Accuracy only considers whether the prediction is correct without accounting for the confidence of the prediction. A loss function is used to measure the difference between the model’s predictions and the actual values; the smaller the loss value, the closer the predictions are to the true values. The loss function used in this study is cross-entropy loss, which provides a more granular performance evaluation by not only focusing on whether the prediction is correct but also considering the degree of deviation in the prediction. Based on this, we compared the loss function values of various algorithms under different privacy budgets, using the same parameters and configurations as in the previous section. Next, we will analyze the experimental results for each dataset.

In the MNIST dataset, the number of communication rounds was set to 50. As shown in

Table 6. Comparison of model loss for the MNIST dataset across different algorithms and privacy budgets.

Algorithm	$\mathit{\epsilon }=0.4$	$\mathit{\epsilon }=0.6$	$\mathit{\epsilon }=0.8$
NoDP-FL [29]	0.3601
CDP-FL [22]	0.8341	0.8013	0.7994
ADP-FL [24]	0.6734	0.6709	0.6686
FDP-FL [23]	1.0365	0.6180	0.5644
AWDP-FL	0.4384	0.4389	0.4323

, the NoDP-FL algorithm, without noise perturbation, achieves a small loss value, ultimately converging to 0.3601. This is because NoDP-FL is not affected by privacy perturbation, allowing the optimizer to quickly find the optimal solution. The AWDP-FL algorithm, after applying adaptive clipping and noise perturbation, has a loss value of 0.4323, which is slightly higher than NoDP-FL but still significantly better than other differential privacy algorithms. The loss values for FDP-FL, ADP-FL, and CDP-FL are 0.5644, 0.6686, and 0.7994, respectively. Compared to FDP-FL, AWDP-FL shows a loss difference of 0.3671, demonstrating stronger convergence and robustness, especially maintaining good performance even under privacy protection.

Figure 5 shows the convergence trends of the loss values for each algorithm on the MNIST dataset. It can be observed that the NoDP-FL algorithm converges rapidly within 20 rounds, and it is unaffected by noise perturbation. The AWDP-FL algorithm converges quickly in the early stages, finally completing convergence between 40 and 50 rounds. Its curve is smooth, demonstrating strong robustness. In federated learning, the smoothness of the loss value curve indicates that the model remains stable through multiple iterations. This characteristic of AWDP-FL shows that it can maintain stability and efficiency in model training while preserving privacy.

In the Fashion-MNIST dataset, the loss values of the various algorithms are relatively close, and they all converge quickly. As shown in

Table 7. Comparison of model loss for the Fashion-MNIST dataset across different algorithms and privacy budgets.

Algorithm/Privacy Budget	$\mathit{\epsilon }=0.4$	$\mathit{\epsilon }=0.6$	$\mathit{\epsilon }=0.8$
NoDP-FL [29]	0.4395
CDP-FL [22]	0.6817	0.6634	0.6540
ADP-FL [24]	0.6125	0.5977	0.5976
FDP-FL [23]	0.6097	0.6062	0.5959
AWDP-FL	0.5701	0.5197	0.5013

, the loss value of the NoDP-FL algorithm is 0.4395, while the loss value of the AWDP-FL algorithm gradually decreases as the privacy budget increases, eventually reaching 0.5013 at $ϵ$ = 0.8. In comparison, the loss values for FDP-FL, ADP-FL, and CDP-FL are 0.5959, 0.5976, and 0.6540, respectively. AWDP-FL’s loss value is 0.1527 lower than that of FDP-FL, demonstrating that the AWDP-FL algorithm can maintain a lower loss value under a higher privacy budget while also exhibiting strong robustness and stability.

Figure 6 shows the convergence of each algorithm on the Fashion-MNIST dataset within 50 to 80 rounds. With a privacy budget of 0.8, AWDP-FL rapidly converges between 50 and 60 rounds, while other algorithms fail to converge within this range. The loss curve of AWDP-FL is relatively smooth, indicating that the algorithm can reach the optimum point more quickly under privacy protection with minimal fluctuations in loss value. This further validates the efficiency and robustness of the AWDP-FL algorithm.

Due to the higher complexity of the CIFAR-10 dataset, the distribution of loss function values is more sparse. As shown in

Table 8. Comparison of model loss for the CIFAR-10 dataset across different algorithms and privacy budgets.

Algorithm/Privacy Budget	$\mathit{\epsilon }=4$	$\mathit{\epsilon }=5$	$\mathit{\epsilon }=6$
NoDP-FL [29]	0.6754
CDP-FL [22]	1.4076	1.3810	1.3744
ADP-FL [24]	1.0303	1.0383	0.9643
FDP-FL [23]	1.2275	1.1925	1.1844
AWDP-FL	0.9204	0.8488	0.7916

, the loss value of NoDP-FL is 0.6754, while AWDP-FL’s loss value is 0.7916 at a privacy budget of 6. Although it is higher than NoDP-FL, it shows a lower loss value compared to other differential privacy algorithms. The loss values of FDP-FL and ADP-FL are 1.1844 and 0.9643, respectively. AWDP-FL consistently maintains a lower loss value, and as the privacy budget increases, its performance remains stable. This demonstrates that AWDP-FL has good robustness and adaptability on more complex datasets.

Figure 7 shows the changes in loss values for each algorithm over the communication rounds in the CIFAR-10 dataset. AWDP-FL demonstrates lower loss values during the first 20 rounds of training, highlighting its efficiency and stability in complex tasks. Although the loss value of AWDP-FL is slightly higher than that of NoDP-FL as noise perturbation increases, its curve remains smooth and lower than that of other differential privacy algorithms. The final loss value difference between AWDP-FL and NoDP-FL is 0.1162, while the gap with other algorithms reaches up to 0.6032. This indicates that AWDP-FL has better adaptability than other algorithms on complex datasets.

5.2.2. Usability Analysis of the Algorithm

In this experimental section, we conducted a comparative analysis of the performance of the AWDP-FL algorithm and other differential privacy federated learning algorithms on three datasets. Fifty clients were selected, and model accuracy is expressed as a percentage. The parameters used for all algorithms were kept consistent: the number of local iterations was set to 3, the batch size was to 64, and the global communication rounds for the three datasets were set to 50, as shown by the experimental results in

Table 9. Comparison of algorithm effectiveness (model accuracy).

Algorithm	MNIST	Fashion-MNIST	CIFAR-10
CDP-FL [22]	90.14 ($ϵ=9$)	82.66 ($ϵ=9$)	65.56 ($ϵ=15$)
ADP-FL [24]	90.12 ($ϵ=3$)	82.41 ($ϵ=2$)	67.80 ($ϵ=9$)
FDP-FL [23]	88.92 ($ϵ=4$)	82.72 ($ϵ=4$)	66.2 ($ϵ=12$)
AWDP-FL	90.75 ($ϵ=0.4$)	83.52 ($ϵ=0.4$)	72.89 ($ϵ=4$)

.

On the MNIST dataset, the AWDP-FL algorithm achieved a notable performance with an accuracy of $90.75\%$ and a privacy budget of only $ϵ$ = 0.4. Compared to other algorithms, CDP-FL and ADP-FL had accuracies of $90.14\%$ and $90.12\%$, respectively, but their privacy budgets were $ϵ$ = 9 and $ϵ$ = 3, which are much higher than that of AWDP-FL. Additionally, FDP-FL had an accuracy of $88.92\%$ with a privacy budget of $ϵ$ = 4, which was inferior to AWDP-FL in both accuracy and privacy protection. This indicates that AWDP-FL can provide higher model accuracy with a smaller privacy budget, demonstrating a better privacy–utility trade-off.

On the Fashion-MNIST dataset, AWDP-FL achieved an accuracy of $83.52\%$ with a privacy budget of only $ϵ$ = 0.4, still outperforming other algorithms. The accuracies of CDP-FL and ADP-FL were $82.66\%$ and $82.41\%$, respectively, but their privacy budgets were $ϵ$ = 9 and $ϵ$ = 2. FDP-FL achieved an accuracy of $82.72\%$ on this dataset with a privacy budget of $ϵ$ = 4. Similarly, AWDP-FL significantly reduced the privacy budget while ensuring model accuracy, demonstrating its robustness on the dataset.

On the CIFAR-10 dataset, AWDP-FL again showed its superiority with an accuracy of $72.89\%$ and a privacy budget of $ϵ$ = 4. In comparison, CDP-FL achieved an accuracy of $65.56\%$ with a privacy budget of $ϵ$ = 15, ADP-FL had an accuracy of $67.80\%$ with a privacy budget of $ϵ$ = 9, and FDP-FL’s accuracy was $66.2\%$ with a privacy budget of $ϵ$ = 12. AWDP-FL not only achieved higher accuracy but also demonstrated better privacy protection with a lower privacy budget.

In summary, the AWDP-FL algorithm exhibited higher model accuracy and lower privacy budget consumption across all three datasets, demonstrating a good balance between privacy protection and model performance.

5.2.3. Privacy Validation of Algorithms under DLG Attack

In this section, we conducted DLG attack experiments on the AWDP-FL algorithm and other differential privacy federated learning algorithms using the MNIST, Fashion-MNIST, and CIFAR-10 datasets. The experimental parameters were based on the conclusions of Section 5.2.2, where the attacks were performed under the privacy budget required to achieve the same model accuracy. The conclusions from that section will not be referenced again here. We believe that federated learning needs to strike a balance between performance and privacy protection, but in practical applications, enhancing privacy protection while ensuring performance remains a major challenge. Therefore, we used the following two metrics to evaluate the experimental results: (1) the Structural Similarity Index (SSIM), which assesses the difference between the reconstructed images from the attack and the original images; and (2) DLG attack loss values, which measure the gradient differences between the original and disguised samples.

Specifically, the SSIM values range from 0 to 1, with values closer to 1 indicating higher image similarity and 1 representing two identical images. The calculation of SSIM is based on three aspects: first, brightness comparison, which measures the average brightness difference; second, contrast comparison, which evaluates contrast differences; and third, structural comparison, which reflects the local structural similarity of the images. These indicators allow us to more comprehensively evaluate the performance of the attack and the effectiveness of privacy protection.

In the experiments, each algorithm randomly selects two samples from each dataset for DLG attacks with the number of attack iterations set to 1000 to ensure convergence of the attack curve. We select the optimal value for each indicator for comparative analysis. Next, we will provide a detailed discussion of the experimental results across the three datasets.

As seen in

Table 10. Effectiveness of different algorithms under DLG attacks on the MNIST dataset.

Indicators	NoDP-FL [29]	CDP-FL [22]	FDP-FL [23]	ADP-FL [24]	AWDP-FL
Loss Value of Sample 1	0.0001	0.0001	0.0005	0.0008	0.0121
SSIM Value of Sample 1	0.9988	0.9970	0.9752	0.9585	0.6971
Loss Value of Sample 2	0.0001	0.0001	0.0005	0.0008	0.0118
SSIM Value of Sample 2	0.9995	0.9938	0.9786	0.9604	0.7551

, for each randomly selected sample, the NoDP-FL algorithm exhibits a near-zero loss value in image reconstruction after the attack with SSIM values close to 1. This is because the algorithm does not apply any noise, and as a baseline algorithm in federated learning, it provides a stable comparison for the performance of other algorithms. For the CDP-FL algorithm, we observe that its indicators are nearly identical to those of NoDP-FL, as it uses a large privacy budget, which improves model performance but neglects the effectiveness of privacy protection. For FDP-FL, ADP-FL, and AWDP-FL, our algorithms show the best performance in both indicators. For instance, the loss value for Sample 1 is 0.0008 with an SSIM value of 0.6971. As clearly shown in Figure 8, the image reconstructed by the AWDP-FL algorithm barely retains any identifiable features of the original image, demonstrating its significant advantage in privacy protection.

The results for the MNIST dataset are shown below.

For the Fashion-MNIST dataset, which is a clothing dataset, we selected two different categories of samples: a top and a shoe. As shown in

Table 11. Effectiveness of different algorithms under DLG attacks on the Fashion-MNIST dataset.

Indicators	NoDP-FL [29]	CDP-FL [22]	FDP-FL [23]	ADP-FL [24]	AWDP-FL
Loss Value of Sample 1	0.0000	0.0001	0.0005	0.0018	0.0120
SSIM Value of Sample 1	0.9998	0.9971	0.9829	0.9515	0.8076
Loss Value of Sample 2	0.0000	0.0001	0.0005	0.0018	0.0116
SSIM Value of Sample 2	1.0000	0.9966	0.9836	0.9385	0.7753

, the values for Sample 2 outperform those of Sample 1 across all indicators. This is because Sample 2’s features are harder to capture compared to those of Sample 1, making it more difficult to reconstruct.

From Figure 9, it can be observed that the image reconstruction performance of the ADP-FL and AWDP-FL algorithms is superior to that of other algorithms. This is because these algorithms consume far less privacy budget than the others, achieving a balance between privacy and performance. However, even with this advantage, these two algorithms still extract a significant amount of image features after the attack. This suggests that despite the addition of differential privacy protection, whether to further sacrifice model performance to enhance privacy protection remains a question worth considering.

The results for the Fashion-MNIST dataset are shown below.

For the CIFAR-10 dataset, unlike the other datasets used in the experiments, this dataset consists of three-channel images, which poses a significant challenge for model training. As a result, each algorithm uses a relatively large privacy budget with the privacy budgets for CDP-FL and FDP-FL set at 12 and 15, respectively. As shown in

Table 12. Effectiveness of different algorithms under DLG attacks on the CIFAR-10 dataset.

Indicators	NoDP-FL [29]	CDP-FL [22]	FDP-FL [23]	ADP-FL [24]	AWDP-FL
Loss Value of Sample 1	0.0000	0.0037	0.0061	0.0102	0.0324
SSIM Value of Sample 1	0.9998	0.9700	0.9427	0.9043	0.8773
Loss Value of Sample 2	0.0000	0.0039	0.0054	0.0403	0.0330
SSIM Value of Sample 2	0.9981	0.9836	0.9382	0.7932	0.8007

, the excessive privacy budgets result in the two indicators of the model after the attack being nearly identical to those of the NoDP-FL model without noise processing, indicating that these algorithms offer poor privacy protection on this dataset. For the ADP-FL and AWDP-FL algorithms, the table shows that the loss value for Sample 2 in ADP-FL is greater than that in AWDP-FL with values of 0.0403 and 0.0330, respectively. Meanwhile, the SSIM values indicate that ADP-FL’s value is lower than AWDP-FL’s, suggesting that ADP-FL has stronger privacy protection capabilities. Figure 10 also shows a visual comparison of the images produced by the respective algorithms, further validating this conclusion.

The results for the CIFAR-10 dataset are shown below.

Finally, the experimental scenarios in this section were designed based on the conclusions drawn in other chapters of this paper. While further exploration from other perspectives may be possible, the above experimental results indirectly support the conclusions of other chapters and demonstrate the superiority of our algorithms. In this scenario, we believe that the introduction of noise processing will inevitably affect model performance, thus reducing the probability of attackers obtaining local data. However, the key challenge remains how to enhance privacy protection without compromising model performance. Therefore, the selection of the privacy budget plays a crucial role in balancing model performance and privacy protection.

5.2.4. The Impact of Local Iteration Counts

This section will explore the impact of local iteration counts on model performance, time overhead, and global communication overhead. In the experiment, 50 clients were selected, and model accuracy was expressed as a percentage. The parameters for all algorithms were kept consistent: the batch size B was 64, and the number of global communication rounds T was 10 for the MNIST dataset, 20 for the Fashion-MNIST dataset, and 50 for the CIFAR-10 dataset. This experimental setup achieved a smaller global communication overhead (compared to the 50, 100, and 100 rounds in the experimental design of Section 5.2.1). Based on this, we will explore the model performance with different local iteration counts across different datasets.

In the MNIST dataset, as shown in

Table 13. Analysis of model performance under different privacy budgets and local iteration counts on the MNIST dataset.

Algorithm	E = 4	E = 5	E = 7	E = 9
NoDP-FL [29]	89.34	89.56	90.14	90.35
AWDP-FL ($\epsilon =0.4$)	88.77	88.46	87.24	87.11
AWDP-FL ($\epsilon =0.6$)	88.96	89.38	89.83	89.14
AWDP-FL ($\epsilon =0.8$)	89.05	89.25	89.94	89.44

, with the increase in local iteration count E, the performance of the AWDP-FL algorithm varies under different privacy budgets. For a smaller privacy budget $ϵ$ = 0.4, as E increases, the model accuracy gradually decreases, particularly at E = 7 and E = 9, where the accuracy is $87.24\%$ and $87.11\%$, respectively. However, with larger privacy budgets $ϵ$ = 0.6 and $ϵ$ = 0.8, the performance of AWDP-FL improves, especially at E = 7, where model accuracy approaches 90%.

Figure 11 further illustrates the trend of the AWDP-FL algorithm as local iteration count E varies under different privacy budgets. Under a smaller privacy budget $ϵ$ = 0.4, model accuracy shows a downward trend as E increases. However, under higher privacy budgets $ϵ$ = 0.6 and $ϵ$ = 0.8, the performance of AWDP-FL is more stable with the best performance observed at E = 7. This suggests that while E = 7 is an appropriate choice under a larger privacy budget, the exact choice should still be adjusted according to the specific privacy budget.

For the Fashion-MNIST dataset, as shown in

Table 14. Analysis of model performance under different privacy budgets and local iteration counts on the Fashion-MNIST dataset.

Algorithm	E = 3	E = 4	E = 5	E = 7
NoDP-FL [29]	83.54	83.70	83.71	83.98
AWDP-FL ($\epsilon =0.4$)	82.73	82.69	82.50	81.39
AWDP-FL ($\epsilon =0.6$)	83.43	83.39	82.85	82.26
AWDP-FL ($\epsilon =0.8$)	83.08	83.55	83.50	83.28

, the performance of the AWDP-FL algorithm varies significantly with the local iteration count E under different privacy budgets. As E increases, especially under a smaller privacy budget $ϵ$ = 0.4, model accuracy gradually decreases, dropping to $81.39\%$ at E = 7. In contrast, while performance improves under higher privacy budgets $ϵ$ = 0.6 and $ϵ$ = 0.8, it is not as stable as with smaller iteration counts. Therefore, E = 3 is an ideal iteration count for the Fashion-MNIST dataset, achieving a good balance between privacy protection and model performance.

Figure 12 shows the trend of the AWDP-FL algorithm on the Fashion-MNIST dataset as the local iteration count E changes under different privacy budgets. As E increases, especially under the smaller privacy budget $ϵ$ = 0.4, model accuracy decreases significantly with the worst performance observed at E = 7. The performance improves slightly under the higher privacy budgets $ϵ$ = 0.6 and $ϵ$ = 0.8, but it is not as stable as with smaller iteration counts. Therefore, E = 3 is the most appropriate iteration count.

In the CIFAR-10 dataset, as shown in

Table 15. Analysis of model performance under different privacy budgets and local iteration counts on the CIFAR-10 dataset.

Algorithm	E = 5	E = 7	E = 8	E = 10
NoDP-FL [29]	77.23	78.91	79.17	78.84
AWDP-FL ($\epsilon =5$)	75.33	73.11	72.82	70.19

, the accuracy of the AWDP-FL algorithm gradually decreases with the increase in local iteration count E under different privacy budgets. The decline in model accuracy is particularly noticeable under the smaller privacy budget $ϵ$ = 5 as E increases. At E = 7, the model shows relatively balanced performance, with higher accuracy, especially under lower privacy budgets. Therefore, E = 5 is the most ideal iteration count for the CIFAR-10 dataset.

Figure 13 shows the trend of the AWDP-FL algorithm on the CIFAR-10 dataset as the local iteration count E changes under different privacy budgets. As E increases, model accuracy shows a downward trend across all privacy budgets with the worst performance observed at E = 10. This is because as the number of iterations increases, the noise added during each local iteration also increases, as indicated by the noise calculation shown in Formula (9).

A comprehensive analysis of the three datasets leads to the following conclusions. In the MNIST dataset, E = 7 is the optimal local iteration count, performing exceptionally well under higher privacy budgets. In the Fashion-MNIST dataset, E = 3 is the optimal iteration count, offering higher model performance under smaller privacy budgets. In the CIFAR-10 dataset, E = 5 is the most suitable iteration count, achieving the best balance between privacy protection and model performance. These results indicate that different datasets have varying requirements for local iteration counts, and the AWDP-FL algorithm demonstrates strong robustness by selecting appropriate iteration counts under different privacy budgets. Notably, as seen in Section 5.2.1 (E = 3), as local iteration counts increase, the local training time overhead proportionally increases but does not raise global communication overhead. Instead, it can reduce global communication overhead while maintaining model accuracy. The number of communication rounds for the MNIST dataset was reduced from 50 to 10, for the Fashion-MNIST dataset from 100 to 20, and for the CIFAR-10 dataset from 100 to 50, which is beneficial in reducing the overall privacy overhead.

5.2.5. The Impact of Sampling Rate on Model Accuracy

This section will explore the impact of client sampling rate (defined as the ratio of the number of clients participating in training to the total number of clients) on model performance and global communication. This ratio can also be understood as the proportion of data involved in training relative to the total data. In the experiment, the optimal local iteration count E was selected for each dataset based on the conclusions from Section 5.2.4. The experimental setup includes 50 clients with model accuracy expressed as a percentage. The batch size B was 64, and the number of global communication rounds T was 10 for the MNIST dataset, 20 for the Fashion-MNIST dataset, and 50 for the CIFAR-10 dataset. Next, we will conduct an in-depth analysis of model performance across different datasets under various sampling rates.

In

Table 16. Analysis of model performance under different sampling rates on the MNIST dataset.

Algorithm	q = 0.4	q = 0.6	q = 0.8	q = 1
NoDP-FL [29]	89.52	90.14	90.62	90.83
AWDP-FL ($\epsilon =0.6$)	88.81	89.83	86.31	90.16

, the impact of different sampling rates on model performance in the MNIST dataset is clearly evident. The NoDP-FL algorithm achieved an accuracy of $90.14\%$ at a sampling rate q = 0.6, while its optimal performance occurred at q = 1 with an accuracy of $90.83\%$. For the AWDP-FL algorithm, with a privacy budget $ϵ$ = 0.6, the accuracy reached $89.83\%$ at a sampling rate q = 0.6 and increased to a maximum of $90.16\%$ when the sampling rate was increased to q = 1. Nevertheless, AWDP-FL already exhibited performance close to q = 1 at q = 0.6, indicating that it can achieve excellent model performance with lower communication costs (T = 10) while maintaining strong privacy protection.

In Figure 14, the trend shows that the accuracy of the NoDP-FL algorithm increases steadily with the sampling rate, reaching its highest value at q = 1. This indicates that the performance of a model without added noise improves as more clients participate in training, leading to a larger dataset and stronger performance. The trend for the AWDP-FL algorithm is more fluctuating. Although it peaks at q = 0.6, it then drops to $86.31\%$ at q = 0.8 before rising again to $90.16\%$ at q = 1. This “U-shaped” trend indicates that AWDP-FL performs better at higher sampling rates, but a lower sampling rate of q = 0.6 can already achieve a good balance between performance and communication costs.

In

Table 17. Analysis of model performance under different sampling rates on the Fashion-MNIST dataset.

Algorithm	q = 0.4	q = 0.6	q = 0.8	q = 1
NoDP-FL [29]	83.35	83.54	83.43	83.41
AWDP-FL ($\epsilon =0.6$)	82.83	83.43	83.33	83.19

, the impact of different sampling rates on model performance in the Fashion-MNIST dataset can be observed. The NoDP-FL algorithm achieved an accuracy of $83.54\%$ at a sampling rate of q = 0.6, while AWDP-FL, with a privacy budget of $ϵ$ = 0.6, achieved an accuracy of $83.43\%$ at the same sampling rate. Compared to other sampling rates, q = 0.6 already exhibited high model accuracy. Further increasing the sampling rate to q = 1 did not result in a significant improvement in accuracy but rather showed a slight decline. This suggests that a lower sampling rate of q = 0.6 can achieve a balance between performance and communication costs, effectively reducing local training time while maintaining strong model performance.

According to Figure 15, the trend shows that as the sampling rate increases from q = 0.4 to q = 0.6, the model performance of both NoDP-FL and AWDP-FL improves with AWDP-FL reaching an accuracy of $83.43\%$ at q = 0.6. However, when the sampling rate increases to q = 0.8 and q = 1, the model’s accuracy experiences a slight decline, especially with AWDP-FL, which does not show an improvement with the higher sampling rates. This indicates that for the Fashion-MNIST dataset, an excessively high sampling rate does not significantly improve model performance and may even affect overall training efficiency. Therefore, q = 0.6 is the optimal sampling rate for maintaining efficient training and accuracy performance.

In

Table 18. Analysis of model performance under different sampling rates on the CIFAR-10 dataset.

Algorithm	q = 0.4	q = 0.6	q = 0.8	q = 1
NoDP-FL [29]	75.42	75.14	76.98	77.23
AWDP-FL ($\epsilon =0.6$)	73.10	73.69	75.19	75.33

, the impact of different sampling rates on model performance in the CIFAR-10 dataset is relatively clear. The NoDP algorithm achieved an accuracy of $77.23\%$ at a sampling rate of q = 1, while the AWDP-FL algorithm, with a privacy budget of $ϵ$ = 5, achieved an accuracy of $75.33\%$ at q = 1. Compared to the lower sampling rates of q = 0.4 and q = 0.6, the model accuracy showed improvement. Although the lower sampling rate q = 0.6 provided faster training, in the CIFAR-10 dataset, the sampling rate of q = 1 is the ideal choice for achieving the highest model accuracy.

According to Figure 16, the trend shows that as the sampling rate increases, the performance of both NoDP and AWDP-FL improves, especially between q = 0.8 and q = 1, where model accuracy significantly improves. For the NoDP algorithm, model performance changes little at lower sampling rates q = 0.4 and q = 0.6, but it reaches its highest value of $77.23\%$ at q = 1. The accuracy of the AWDP-FL algorithm is relatively lower at the lower sampling rates q = 0.4 and q = 0.6, but it reaches $75.33\%$ at q = 1. Therefore, on the CIFAR-10 dataset, although lower sampling rates help reduce training time, q = 1 remains the best choice for ensuring higher model performance.

A comprehensive analysis of the results from the three datasets shows that a lower client sampling rate q = 0.6 provided high cost-effectiveness in the MNIST and Fashion-MNIST datasets, as it speeds up local training and improves experimental efficiency without significantly affecting overall model accuracy. However, in the CIFAR-10 dataset, although lower sampling rates can speed up training, the sampling rate q = 1 provided higher model accuracy. Therefore, if model accuracy is prioritized, q = 1 is the best choice.

5.2.6. The Impact of Hierarchical Gradient Clipping Optimization

Gradient clipping optimization at the neural network layer level, as one of the core optimization algorithms in this paper, will be explored in this section to determine whether different neural network layers affect the optimization effectiveness of this algorithm. We will compare the original AWDP-FL algorithm with a modified algorithm (which calculates the clipping threshold based on the entire model for each iteration without considering individual layers separately). This modified algorithm is based on AWDP-FL to assess the impact of layer-level adaptiveness. Additionally, a new, deeper network structure (Model 2, see Section 5.1) was designed compared to the original network used for the CIFAR dataset. The experimental setup includes 50 clients with model accuracy expressed as a percentage. The batch size B is 64, and the dataset used is the more complex CIFAR-10 dataset, with 100 global communication rounds T = 100. Next, we will conduct an in-depth analysis of the experimental results.

In

Table 19. Comparison of model accuracy for neural network layer-wise gradient clipping optimization on the CIFAR-10 dataset.

Algorithm	$\mathit{\epsilon }=4$	$\mathit{\epsilon }=5$	$\mathit{\epsilon }=6$
NoDP-FL [29]	81.89
AWDP-FL ($\epsilon =5$, layered)	75.94	79.17	80.67
AWDP-FL ($\epsilon =5$, not layered)	70.01	74.04	76.02

, the comparison between hierarchical and non-hierarchical processing shows a significant difference in the performance of AWDP-FL under the differential privacy mechanism. Hierarchical AWDP-FL ($ϵ$ = 5) consistently achieves higher accuracy than non-hierarchical AWDP-FL under different privacy budgets. For example, at $ϵ$ = 4, the accuracy of hierarchical processing is $75.94\%$, while non-hierarchical processing achieves only $70.01\%$. When $ϵ$ = 6, hierarchical processing further improves to $80.67\%$, while non-hierarchical processing reaches $76.02\%$. This indicates that the hierarchical mechanism better preserves model performance while ensuring privacy protection. It can also be observed that as the privacy budget $ϵ$ increases, the accuracy of both methods rises, suggesting that increasing the privacy budget reduces the impact of differential privacy on model performance. However, the advantage of hierarchical processing lies in its ability to deliver higher accuracy even under the same privacy budget. As for NoDP-FL, without the introduction of differential privacy, it achieved an accuracy of $81.89\%$, which aligns with the expected performance of models without differential privacy.

6. Conclusions

This paper focuses on the application of differential privacy in federated learning, particularly on the selection of adaptive thresholds during gradient clipping and the adaptive update of model gradients. By combining adaptive weight parameters and a layer-wise approach, an Adaptive Differential Privacy Federated Learning framework (AWDP-FL) was designed, introducing dynamic noise perturbation before model parameter transmission. Experimental results demonstrate that on the MNIST, Fashion-MNIST, and CIFAR-10 public datasets, AWDP-FL not only provides strong privacy protection but also effectively enhances model performance and significantly improves the stability of the training process in various experimental scenarios. Furthermore, this paper provides an in-depth analysis of the optimal parameter configuration for AWDP-FL, further enhancing the algorithm’s transparency and practicality.

In this paper, while the equal distribution of privacy budgets reduces waste, it may still be too high for some datasets. Future work will focus on developing more accurate privacy loss measurement methods to better control the consumption of privacy budgets. Data heterogeneity in federated learning is a common challenge. Due to significant differences in data distribution across clients, traditional privacy protection strategies may lead to performance degradation in heterogeneous environments. Addressing the issue of data heterogeneity will help improve the adaptability of models across different clients and allow for a more precise implementation of privacy protection. Deep neural networks perform better when processing complex data, as they are capable of extracting richer features. However, deep networks impose higher demands on differential privacy, and future research will explore how to optimize privacy protection in complex networks to balance privacy and performance.