Communication-Efficient and Private Federated Learning with Adaptive Sparsity-Based Pruning on Edge Computing

Abstract

As data-driven deep learning (DL) has been applied in various scenarios, the privacy threats have become a widely recognized problem. To boost privacy protection in federated learning (FL), some methods adopt a one-shot differential privacy (DP) approach to obfuscate model updates, yet they do not take into account the dynamic balance between efficiency and privacy protection. To this end, we propose ASPFL—an efficient FL approach with adaptive sparsity-based pruning and differential privacy protection. We further propose the adaptive pruning mechanism by utilizing the Jensen-Shannon divergence as the metric to generate sparse matrices, which are then employed in the model updates. In addition, we introduce adaptive Gaussian noise by assessing the variation of sensitivity through post-pruning uploading. Extensive experiments validate that our proposed ASPFL boosts convergence speed by more than two times under non-IID data. Compared with existing DP-FL methods, ASPFL can maximally achieve over 82% accuracy on CIFAR-10, while the communication cost is greatly reduced by 40% under the same level of privacy protection.

1. Introduction

With the vigorous development of deep neural networks (DNNs) for various tasks such as computer vision (CV) [1,2] and natural language processing (NLP) [3], deep learning (DL) has evolved into a technology dependent on large amounts of training data. Recently, the demand for efficient and privacy protection DL solutions has received high attention. Federated learning (FL) has emerged as a promising paradigm of distributed machine learning [4,5]. In contrast to traditional deep learning, which uploads data to the cloud for training, federated learning allows multiple edge devices to locally train the model and upload the model updates.

Nonetheless, model updates dependent on local private samples and large-scale parameters among clients raise significant communication burdens and privacy concerns. As for communication overhead, recent studies have revealed that most deep neural networks (DNN) are over-parameterized. Denil et al. [6] proposed that using only a few weights is enough to achieve similar performance to the MLP and CNN. Michel et al. [7] demonstrated that a significant proportion of the heads in a Transformer can be pruned without substantially degrading the model’s performance. A feasible scheme is to reduce parameter transmission by model compression (pruning and distillation) [8], resulting in an appreciably smaller model and faster inference.

Research on the privacy and security issues of FL has revealed that sensitive personal parameters uploaded by clients may be leaked through model inversion attacks [9,10]. Therefore, combining differential privacy (DP) with FL is considered an effective privacy protection technique [11,12,13,14]. The mechanism of DP typically involves adding random noise to intermediate outputs in order to ensure that a change in a particular input element will not have a significant impact on the output distribution [12]. However, in DP-based FL, an inevitable trade-off exists between model utility and privacy protection level. Enhanced privacy preservation measures typically necessitate the introduction of artificial noise with higher variance, which inevitably leads to a degradation in the model’s predictive accuracy and response speed.

Accordingly, we urgently need a general and common framework to strengthen data privacy protection and reduce communication costs while quantitatively answering important questions:

Can model pruning and differential privacy be orthogonal? How does pruning affect the fine-tuning of the private noise?

Several studies have attempted to address similar inquiries from various perspectives [15,16,17,18,19], and numerous concepts of privacy and associated strategies have been proposed to tackle related problems. Mireshghallah et al. [15] proposed a method for evolving large models into compressed ones through a differential privacy iterative magnitude pruning (DPIMP) framework. Its main idea is to sort the weights by magnitude, then eliminate the parameters with small magnitudes based on a pruning ratio. This approach can achieve up to 50% parameter sparsity while maintaining model performance. However, the system is configured under central machine learning and not placed in the context of FL, while the pruning ratio is usually determined empirically. Lin et al. [18] proposed a proprietary model compression framework named RONA, which integrates knowledge distillation and differential privacy techniques to achieve efficient deployment of deep learning models on mobile devices. But we cannot guarantee the existence of a large, powerful, and complex teacher model based on sensitive data for multiple tasks. GFL-ALDPA aimed to achieve gradient compression and user privacy protection by jointly using dimensional reduction and local differential privacy (LDP) [19]. However, it has not fully explained and directly quantified the impact of compression on the injected private noise.

To address the challenges of capacity bottlenecks and privacy protection when deploying deep learning models on edge devices, we propose an enhanced private model compression framework based on adaptive DP and develop a learnable sparsity-based pruning module. Model compression and data privacy protection are achieved jointly in this framework. We introduce the compressed local model parameters space through the function of sparsity matrix to reduce the communication overhead. And following rigorous DP, we propose an adaptive differential privacy to not only provide strong privacy protection but also effectively defend against non-adversarial attacks. It can also ensure a high accuracy of model training.

The contributions of this paper are summarized as follows:

(1)

We proposed a communication-efficient federated learning framework that achieves a high-accuracy classification model with a privacy guarantee. This framework jointly considers the dynamic balance between the efficiency and privacy issues under non-IID distributions while achieving good model performance.

(2)

To enhance communication efficiency, we proposed an adaptive channel pruning mechanism using a Jensen-Shannon divergence feature $p_i$ to generate the sparsity matrix, reducing the rounds required for equivalent performance by more than 2 times.

(3)

To ensure the privacy guarantee, we estimate the sensitivity after pruning and introduce a Gaussian noise to the aggregated model. In addition, we give a lower bound of variance ${\sigma }^2$for adaptive collaborative training. Our method achieves a higher model accuracy (82.42%) and 40% improvement in communication rounds to the baseline methods.

The rest of the paper is organized as follows: Section 2 introduces the context of differential privacy for federated learning and model pruning. Section 3 describes our proposed adaptive and efficient Sparsity-pruning-based federated learning scheme. Section 4 experimentally illustrates the accuracy and performance of our proposed method on CIFAR-10. Section 5 presents our conclusions.

2. Related Works

Federated learning (FL) is a fundamental training paradigm in machine learning that was first proposed in 2016 [20]. It aims to build a collaborative machine learning model on distributed datasets. Recently, federated learning has faced growing challenges, including communication overhead, data heterogeneity, and privacy threats. This section gives the introduction of two topics, namely differential privacy for federated learning and model pruning.

2.1. Differential Privacy for Federated Learning

Differential privacy (DP) has been widely adopted to mitigate user privacy leakage in federated learning [21,22,23]. Differential Private Stochastic Gradient Descent (DP-SGD) [22] was proposed to provide a data-level privacy guarantee and can be easily achieved by adding Gaussian noise to the clipped gradients. However, FL not only focuses on the privacy protection of local data but also requires information security between clients. Existing works can be summarized into two classes: centralized DP (CDP) [12,21,24] and local DP (LDP) [25,26,27]. The central idea of LDP is randomized response (RR). In federated learning, LDP can enable parties to scramble their data and then publish the obfuscated data to an untrusted server.

2.2. Model Pruning

With the extensive deployment of DNNs, model pruning has become an important topic to reduce the computational resources. A popular way is magnitude-based pruning [15,28]. Jiang et al. [28] apply the lottery hypothesis, which posits that pruning a network based on the magnitude of its weights, in principle, yields an optimal substructure of the original network.

In general, pruning methods can be divided into two categories: structured pruning and unstructured pruning. Structured pruning reduces the model complexity by removing structural units within the network such as kernels, filters, or layers in different granularities [29,30,31]. It typically requires specific network structures, which is inconsistent with the “lottery ticket hypothesis”. Moreover, selection of the optimal pruning rate mostly depends on the empirical knowledge, and the original model requires retraining. Zhu et al. [30] proposed an effective model compression and acceleration framework, FedLP-Q, through hierarchical pruning and quantization. The pruning process in the frame takes place at each layer of the model, removing unimportant weights or entire layers to reduce the depth and width of the model, but this approach results in fewer feature maps in the intermediate representation.

In contrast, unstructured pruning mainly focuses on pruning at the level of individual weights. Without changing the original structure of the network, it achieves sparsity by setting unimportant weights to zero. Qian et al. [32] propose a dynamic adjustment strategy to reduce unnecessary transmission costs by gradually increasing the sparsity ratio and replacing aggregation weights with the inverse ratio of sparsity. But the authors did not take into account that the unevenness of the data distribution can affect aggregate weights.

3. Our Methodology: ASPFL Framework

In this section, we present a general framework, ASPFL, introduce the adaptive sparsity-based pruning module and central differential privacy mechanism, and perform an extensive privacy analysis of the model after pruning. The notations used in this paper are summarized in

Table 1. Descriptions of Main Notations.

Symbol	Descriptions
N	total number of clients
T	total number of global epochs
t	the index of the t-th global epoch
L	number of local epochs for clients in a round
l	the index of the l-th local epoch
$\mathit{g}$	gradient of the model
${\mathcal{D}}_i$	local dataset of the i-th client
B	batch size
$K$	set of participating clients
${\mathit{\theta }}_i$	local model parameters of the i-th client
${\mathit{W}}_i$	weight matrices of the i-th client
$\left|\ast \right|$	size of set
$\epsilon$	global privacy budget
$\sigma$	the noise level
$S_f$	the sensitivity of the query function
${∆}_i$	model update of the i-th client
$\mathit{R}$	the sparse matrix
γ	tailoring factor

.

3.1. Overview and Problem Statement

Workflow: The workflow of our proposed ASPFL is shown in Figure 1. It can be summarized into five stages: (1) The server first performs global model initialization and distributes it to the clients. (2) Sampled clients train their local models based on the current global model and their respective datasets. During local training, each client executes local model updates and gradient clipping based on L₂-norm to meet the requirements of DP. (3) After L rounds’ local training, each client performs channel pruning according to a learnable sparsity matrix and estimates the sensitivity of all local model unions. Then updates of the local model are uploaded back to the server. (4) The server collects all model updates and introduces an adaptive noise based on the estimated sensitivity to perform model aggregation. (5) Finally, a new global model is obtained. Below, we will explain the specific operation of each stage in detail.

System Model: Based on the client/server (C/S) framework, the basic FL system comprises N clients and a server aiming to collaboratively train an optimal model while achieving privacy protection. Assume that the server is honest but curious, client $c_i$ holds the local dataset ${\mathcal{D}}_i$ with $\left|{\mathcal{D}}_i\right|$ samples, where $i\in \{1,2,\dots ,N\}.$ The union of all local datasets is $\mathcal{D}:=\bigcup {\mathcal{D}}_i$. The objective of our ASPFL is to aggregate local models from N clients and obtain an optimal global model $\mathit{\theta }$ that minimizes the global empirical risk:

$$\underset{\mathit{\theta }}{\min }\mathcal{L}\left(\mathit{\theta }\right)={\sum }_{i=1}^N{p}_i\mathcal{L}\left({\mathit{\theta }}_i\right),$$

(1)

where $\mathcal{L}\left({\mathit{\theta }}_i\right)$ is a non-negative and convex loss function on each client, $\mathcal{L}\left(\mathit{\theta }\right)$ is the global loss function. In FedAvg [4], $p_i$ denotes the aggregation weight, which can be calculated by the dataset size ${\displaystyle \frac{\left|{\mathcal{D}}_i\right|}{\left|\mathcal{D}\right|}}.$

Threat model: Assume that the server is honest but curious; it adheres to federation rules while maintaining a curiosity towards the local data of each client. Although FL keeps individual datasets locally, sharing model parameters with the server may expose clients’ private information through model-inversion attacks. These attacks include attempts by the server to recover training datasets or infer private features based on uploaded parameters and deduce whether a sample belongs to a client’s training dataset from output differences. Furthermore, each client can also be considered honest but curious. They potentially perform similar privacy attacks on other clients’ data after receiving broadcast model parameters.

3.2. ASPFL Algorithm

Figure 1 shows the workflow of our proposed scheme, consisting of 5 steps. We show the specific procedures in Algorithm 1, then illustrate each step below.

(1)

Global model initialization. The server first initializes global model parameters ${\mathit{\theta }}_0$ (Line 1 in Algorithm 1) and distributes ${\mathit{\theta }}_0$ to each client.

(2)

Local models update. In the global round t, K clients are randomly selected to participate in local training. Local model parameters ${\mathit{\theta }}_{i,0}^t$ is first initialized as the global model parameters ${\mathit{\theta }}_{i,0}^t\leftarrow {\mathit{\theta }}^{t-1}$. Then L steps local training are performed to update local model parameters ${\mathit{\theta }}_{i,l}^t$. We designed the gradient clipping process of incorporating tailoring factors $\gamma$ into the stochastic gradient descent (SGD) optimizer to obtain a local model ${\mathit{\theta }}_{i,l}^t$. A tailoring factor ${\gamma }_{i,l}^t$ is applied to control the magnitude of parameter updates, which is represented as:

$${\gamma }_{i,l}^t=\min (1, {\displaystyle \frac{C}{{‖{\mathit{g}}_{i,l}^t‖}_2}}),$$

(2)

where ${\mathit{g}}_{i,l}^t$ denotes the gradient at step l of round t, and C is the upper bound of gradients to prevent instability in the training regimen. Specifically, for the client $i$, the local gradient clipping process is formulated as:

$${{\tilde{\mathit{g}}}_{i,l}^t={\gamma }_{i,l}^t\mathit{g}}_{i,l}^t=\mathrm{m}\mathrm{i}\mathrm{n}(1,{\displaystyle \frac{C}{{‖{\mathit{g}}_{i,l}^t‖}_2}})\nabla \mathcal{L}({\mathit{\theta }}_{i,l-1}^t;B_{i,l}^t),$$

(3)

$${\mathit{\theta }}_{i,l}^t={\mathit{\theta }}_{i,l-1}^t-\eta {\tilde{\mathit{g}}}_{i,l}^t,$$

(4)

where $B_{i,l}^t$ is a random batch of data samples and η denotes the learning rate.

(3)

Sparsity-based channel pruning. In the round t, the client $i$ finishes L steps local training and obtains a new local model ${\mathit{\theta }}_{i,L}^t$. To reduce the communication overhead required for uploading parameters, we employ an adaptive sparse matrix to discard the less significant parts of the model updates. A sparse matrix ${\mathit{R}}_i$ composed of elements “1” and “0” is allocated for the client $i$. By Hadamard product of model updates with this matrix, the less important weight parameters will be zeroed out. The client $i$ combines the accumulated gradients ${\mathit{\theta }}_{i,L}^t-{\mathit{\theta }}^{t-1}$ with ${\mathit{R}}_i$ and the uploaded part is formulated as:

$${∆\mathit{\theta }}_i={(\mathit{\theta }}_{i,L}^t-{\mathit{\theta }}^{t-1})⨀{\mathit{R}}_i=\eta {\sum }_{l=1}^L{\gamma }_l^t{\mathit{g}}_{i,l}^t⨀{\mathit{R}}_i.$$

(5)

The selection method for the mask matrix ${\mathit{R}}_i$ will be elaborated in the next subsection. After channel pruning, the sensitivity of uploading model parameters $S_f$ can be estimated as:

$$S_f=2\eta CL{p}_i$$

(6)

where $p_i$ is a personalized parameter for the client $i$.

(4)

Model aggregation and noise perturbation. Initially, the server receives all model updates from participating clients and computes the weighted average of model updates, yielding a preliminary version of the aggregated global model. To fortify privacy preservation via differential privacy, the server introduces adaptive Gaussian noise onto the preliminary global model. The variance ${\sigma }^2$ of this noise is judiciously determined based on the desired privacy budget (ε and δ) and the sensitivity $S_f$. Mathematically, the noise-injected global model is denoted as:

$${\mathit{\theta }}^t={\mathit{\theta }}^{t-1}+\left({\sum }_{i=1}^N{\displaystyle \frac{\left|{\mathcal{D}}_i\right|}{\left|\mathcal{D}\right|}}{∆\mathit{\theta }}_i\right)+\mathcal{N}\left(0,{\sigma }^2{\mathit{I}}_d\right).$$

(7)

(5)

Global model broadcasting. Upon completion of the global model update in round t, the server is responsible for broadcasting the newly generated global model ${\mathit{\theta }}^t$ to all participating clients, thereby initiating the subsequent round of local training processes as ${\mathit{\theta }}_{i,0}^{t+1}\leftarrow {\mathit{\theta }}^t$.

The procedures of our ASPFL algorithm have been concluded in Algorithm 1. The primary improvements of this algorithm from traditional FL include three aspects: (1) Scaling rule of accumulated gradients during local training. (2) Adaptive channel pruning based on sparse matrices for non-IID data distributions. (3) Post-pruning noise addition strategy at the server side.

Algorithm 1 ASPFL

Initialization: tailoring threshold $C$, local models, local datasets ${\mathcal{D}}_i$, privacy configurations ($\epsilon , \delta$), etc.

1  for global round t = 1, 2, …, T do

2    for participating client i in parallel do            ※client side

3      for each step l = 1, 2, …, L do

4        Clip local gradient: ${\tilde{\mathit{g}}}_{i,l}^t$;

5        Update ${\mathit{\theta }}_{i, l}^t\leftarrow Local\_Update({\mathit{\theta }}_{i, l-1}^t;{\mathcal{D}}_i);$

6      end

7      Quantize the JS distribution feature: $p_i$;

8      Generalize the mask matrix: ${\mathit{R}}_i=${$r_{jk}$}$, r_{jk}~Bernouli\left(p_i\right)$;

9      Conduct sparsity pruning: ${(\mathit{\theta }}_{i, L}^t-{\mathit{\theta }}^{t-1}) ⨀{\mathit{R}}_i$;

10      Upload model updates to the server;

11     end

12   Aggregate updates and calculate the sensitivity $S_f$;         ※server side

13   Introduce Gaussian noise to ${\mathit{\theta }}^t$;

14   Download the global model: ${\mathit{\theta }}_{i, 0}^{t+1}\leftarrow {\mathit{\theta }}^t$;                ※client side

15   end

Output: global model: ${\mathit{\theta }}^t$.

3.3. Adaptive Sparsity-Based Channel Pruning

To mitigate client dependence on network stability during parameter uploads, we propose a method of achieving model sparsity through the introduction of learnable gating parameters to regulate the extent of model compression. To attain model sparsity within the context of a federated learning architecture, we incorporate a normalization technique that entails the direct application of a mask matrix, denoted as ${\mathit{R}}_i$, onto the model parameter. The mask matrix ${\mathit{R}}_i$ is derived via the gated parameter $p_i$, which serves as a quantitative indicator of the proportion of “1”s contained within ${\mathit{R}}_i$. This operation effectively prunes the model by zeroing out coefficients based on the binary values of ${\mathit{R}}_i$, leading to a decrease in the bandwidth requirements but also diminishing potential vulnerabilities to model inversion or membership inference attacks.

In the layer-wise pruning-quantization scheme for efficient federated learning proposed in the literature [29,30], $p_i$ represents the probability that the gradient of each layer will be preserved, catering to the demands of communication efficiency. Specifically, Zhu et.al. [29,30] proposed that $p_i$ is typically set by the central server following three steps:

• First set a fixed value $p$;
• Then, specify the layer-preserving rate (LPR) that satisfies the Bernoulli(p) distribution;
• Finally, prune the cumulative gradients according to LPR.

In this strategy, the parameter $p$ is designed to be a constant applicable to all clients in model homogeneity cases. Essentially, $p$ is an empirical parameter that may require a lot of prior knowledge and many attempts to determine. However, federated learning faces a critical challenge due to non-independent and identically distributed (non-IID) data [33]. If the nature of non-IID data is not considered, potential biases in capturing the feature representations across various clients will occur, further diminishing the model’s accuracy and generalization ability.

Based on this intuition, we integrate the characteristic of data distribution into the sparse matrix selection and propose an adaptive channel pruning strategy. There exists a typical scenario of label skew in federated learning with non-IID data, which may lead to performance degradation of the model on some clients. The disparity in label distribution across clients can be quantified through the probability of each label within their respective datasets. To this end, the distribution of labels from a given client’s dataset is extracted and denoted by $P=\left\{P_i\right\}, i\in N$. Then, a uniform distribution $Q$ is established as a benchmark for comparison. For an aggregated dataset $\mathcal{D}$ encompassing $\mathcal{z}$ distinct label categories, a perfectly uniform distribution $Q$ implies that each category is endowed with an equal probability of occurrence, precisely $Q=\left\{{\displaystyle \frac{1}{z}}\right\}$.

We adopt the Jensen-Shannon (JS) divergence as a metric to quantify the difference between each client’s data label distribution $P_i$ and a standard uniform distribution $Q$. JS divergence is a well-established measure for assessing the similarity, which can be formulated as:

$$D_{KL}\left(P_i‖Q\right)={\sum }_x{P}_i\left(x\right)\log {\displaystyle \frac{P_i\left(x\right)}{Q\left(x\right)}}$$

(8)

$$D_{JS}\left(P_i‖Q\right)={\displaystyle \frac{1}{2}}{D}_{KL}\left(P_i‖{\displaystyle \frac{P_i+Q}{2}}\right)+{\displaystyle \frac{1}{2}}{D}_{KL}\left(Q‖{\displaystyle \frac{P_i+Q}{2}}\right)$$

(9)

where $D_{KL}$ denotes the Kullback-Leibler (KL) divergence. By calculating the JS divergence between $P_i$ and $Q$ in Equations (8) and (9), we can quantify how far each client’s label distribution deviates from a uniform distribution. For each client, we utilize each local dataset’s JS divergence as a distinctive feature $p_i$, thereby instituting the following procedures to construct adaptive local sparse matrices:

• Extraction of Label Distributions: Initially, the label distributions pertinent to each client’s local dataset are meticulously extracted and quantified.
• Computation of JS Divergence: the JS divergence between the client’s label distribution and a predefined uniform distribution is calculated to quantify the similarity. Since the range value of JS divergence is in [0, $ln2$], we make a linear transformation of the feature $p_i$ following:

  $$p_i={\displaystyle \frac{J{S}_{max}-JS}{J{S}_{max}-J{S}_{min}}}={\displaystyle \frac{ln2-JS}{ln2}}$$

  (10)
• Normalization and Weighting: The resulting $p_i$ should be normalized to ensure comparability across different clients.
• Matrix Construction: The sparse matrix ${\mathit{R}}_i$ is sampled from the Bernoulli ($p_i$) distribution for each client. This matrix reflects the unique statistical characteristics of the client’s data, particularly focusing on the sparsity patterns that emerge from the label distribution disparities.

This parameter $p_i$ essentially governs the sparsity level of the matrix, precisely quantifying the probability of retaining the element “1” among the matrix elements. In this scenario, mask matrix ${\mathit{R}}_i$ is a 3D tensor composed of multiple 2D mask matrices, each of which matches the shape of individual slices within ${\mathit{W}}_i$. The elements $r_{jk}\left({\equiv r}_{i,njk}\right)$ of each mask matrix $R_{i, n}$ are independently sampled from a Bernoulli $\left(p_i\right)$. Here, n indexes the tensor, and j and k index the rows and columns of the matrix. Specifically, each element $r_{jk}$ in $R_{i,n}$ can be formulated as:

$$r_{jk}=\left\{\begin{array}{c}1 \mathrm{with} \mathrm{probability} p_i,\\ 0 \mathrm{with} \mathrm{probability} 1-p_i.\end{array}\right.$$

(11)

For the n-th weight tensor, we can get ${\mathit{W}}_{i,n}={\mathit{W}}_{i,n} ⨀ {\mathit{R}}_{i,n}$ ($w_{jk}=w_{jk}$×$r_{jk})$. If $r_{jk}$= 1, then the corresponding weight $w_{jk}$ in ${\mathit{W}}_i$ will be retained. Conversely, if $r_{jk}$= 0, $w_{jk}$ will be set to zero or effectively ignored. The generation process of the mask tensor ${\mathit{R}}_i$ has been illustrated in Figure 2. The process of generating ${\mathit{R}}_i$ through $p_i$ ensures that the resultant matrix selectively preserves critical information while discarding less relevant data points. By adopting this approach, we can enhance the adaptability and effectiveness of FL algorithms in scenarios characterized by non-IID, thus improving overall model performance and convergence rates.

3.4. Privacy Guarantee after Pruning

To address the privacy and security concerns of participants, we implement the Gaussian mechanism to achieve DP. This approach is instrumental during the aggregation phase at the central server, where parameter updates from local models are synthesized, thereby effectively mitigating the risk of privacy leakage. Compared to traditional privacy preservation mechanisms, differential privacy offers the advantages of quantifiable privacy budgets and the concealment of sensitive behaviors at the user level. Therefore, we introduce the basic knowledge of DP, which is defined as follows:

Definition 1

(($\mathit{\epsilon },\mathit{\delta }$ )-DP [23]).  For two adjacent datasets $\mathcal{D}$ and ${\mathcal{D}}^{\prime }$ that differ by only one record, a randomized mechanism $\mathcal{M}$ can preserve ($\epsilon , \delta$)-DP if and only if any output $S\subset Range\left(\mathcal{M}\right)$ holds:

$$\mathit{Pr}\left[\mathcal{M}\left(\mathcal{D}\right)\in S\right]\le Pr\left[\mathcal{M}\left({\mathcal{D}}^{\prime }\right)\in S\right]\times e^{\epsilon }+\delta$$

(12)

where $\epsilon >0$ is the privacy budget and $\delta \in \left(0, 1\right)$ means the failure probability. Equation (12) is also termed the Gaussian mechanism in DP.

During each round of updating information from local models, noise should be introduced, obeying the global privacy budget $\epsilon$ and the sensitivity $S_f$. The Gaussian mechanism stipulates to add Gaussian noise $\mathcal{N}(0, {\sigma }^2)$ to each output of the query function, where ${\sigma }^2$ must satisfy the following equation:

$${\sigma }^2\ge {\displaystyle \frac{2\mathrm{l}\mathrm{n}\left(\frac{1.25}{\delta }\right)}{{\epsilon }^2}}{S}_f^2$$

(13)

A.

L₂-norm Sensitivity

Now we specifically analyze the influence of the channel pruning module on the sensitivity of upload model parameters so as to customize appropriate noise to achieve ($\epsilon ,\delta$)-DP. First, we discuss the L₂-norm Sensitivity evaluation of the local parameter updating. The introduction of $\gamma$ in Equation (2) is to restrict the range of accumulated gradients. If ${‖{\mathit{g}}_{i, l}^t‖}_2>C$, then $\gamma <1$, and $\tilde{\mathit{g}}$ = $\gamma {\mathit{g}}_{i,l}^t$ to prevent gradient explosion. If ${‖{\mathit{g}}_{i,l}^t‖}_2\le C$, then $\gamma =1$ and $\tilde{\mathit{g}}={\mathit{g}}_{i,l}^t$ remains unchanged. Thus, the L₂-norm of $\tilde{\mathit{g}}$ can be formulated as:

$${‖\tilde{\mathit{g}}‖}_2=\left\{\begin{array}{c}C, C<{‖{\mathit{g}}_{i,l}^t‖}_2\\ {‖{\mathit{g}}_{i,l}^t‖}_2, C\ge {‖{\mathit{g}}_{i,l}^t‖}_2\end{array}\right.$$

(14)

It is observed that the clipped gradient $\tilde{\mathit{g}}$ for each client is bounded by $C$, i.e., ${‖\tilde{\mathit{g}}‖}_2 \le C,$ for any i, t, and l.

On this basis, the client $i$ completes channel pruning with their respective ${\mathit{R}}_i$ after L steps local training. The accumulated gradients after pruning are then uploaded to the server. In the whole process, the sensitivity of the gradient uploading function is calculated as follows:

Lemma 1.

For the round t at client i, the L₂-norm sensitivity of uploading after pruning function is

$${ S}_f=\underset{\mathcal{D},{\mathcal{D}}^{\prime } }{\mathit{max}}{‖M_i{\mathit{G}}_{i }^t\left(\mathcal{D}\right)-M_i{\mathit{G}}_{i }^t\left({\mathcal{D}}^{\prime }\right)‖}_2=2\eta CL{p}_i$$

(15)

where $M$ represents the Hadamard product of sparse matrix ${\mathit{R}}_i$ and G is the accumulated gradients.

Proof. 

In the round t, client i performs L steps of local clipping SGD to achieve ${\mathit{\theta }}_{i }^t$ following:

$${\mathit{\theta }}_i^t={\mathit{\theta }}^{t-1}-{\sum }_{l=1}^L\eta {\gamma }_{i, l}^t{\mathit{g}}_{i,l}^t$$

(16)

Since $r_{jk}$ is independently sampled from Bernoulli ($p_i$), then $E\left[{\mathit{R}}_i\right]$ is $p_i$. Consequently, under certain conditions (e.g., large sample size or independent sampling), the actual sensitivity can be approximated by its expectation $S_f$ $\approx$ $E\left[S_f\right]$. The expectation of $S_f$ can be represented as:

$$E\left[S_f\right]=\underset{\mathcal{D},{\mathcal{D}}^{\prime } }{\max } E[{‖M_i{\mathit{G}}_{i }^t\left(\mathcal{D}\right)-M_i{\mathit{G}}_{i }^t\left({\mathcal{D}}^{\prime }\right)‖}_2]$$

$$=\underset{\mathcal{D},{\mathcal{D}}^{\prime } }{\max } {‖E\left[{\mathit{R}}_i\right]{\mathit{G}}_{i }^t\left(\mathcal{D}\right)-E[{\mathit{R}}_i{]\mathit{G}}_{i }^t\left({\mathcal{D}}^{\prime }\right)‖}_2$$

$$=p_i\underset{\mathcal{D},{\mathcal{D}}^{\prime } }{\max }{‖{\mathit{G}}_{i }^t\left(\mathcal{D}\right)-{\mathit{G}}_{i }^t\left({\mathcal{D}}^{\prime }\right)‖}_2$$

(17)

Since ${\gamma }_{i,l}^t$ $\in \left(0,1\right],$

$${‖{\mathit{G}}_{i }^t\left(\mathcal{D}\right)-{\mathit{G}}_{i }^t\left({\mathcal{D}}^{\prime }\right)‖}_2={‖{\sum }_{l=1}^L\eta {\gamma }_{i, l}^t{\mathit{g}}_{i,l}^t\left(\mathcal{D}\right)-{\sum }_{l=1}^L\eta {\gamma }_{i, l}^t{\mathit{g}}_{i,l}^t\left({\mathcal{D}}^{\prime }\right)‖}_2$$

$$\le \eta {‖{\sum }_{l=1}^L{\tilde{\mathit{g}}}_{i,l}^t\left(\mathcal{D}\right)-{\sum }_{l=1}^L{\tilde{\mathit{g}}}_{i,l}^t\left({\mathcal{D}}^{\prime }\right) ‖}_2$$

(18)

Then, the upper bound of the sensitivity is:

$$S_f\approx E\left[S_f\right] \le \eta p_i{\sum }_{l=1}^L{‖{\tilde{\mathit{g}}}_{i,l}^t\left(\mathcal{D}\right)-{\tilde{\mathit{g}}}_{i,l}^t\left({\mathcal{D}}^{\prime }\right)‖}_2 \le 2\eta CL{p}_i.$$

(19)

Given that $\mathcal{D}$ and ${\mathcal{D}}^{\prime }$ differ by only one sample, the difference in gradient estimates, ${‖{\tilde{\mathit{g}}}_{i,l}^t\left(\mathcal{D}\right)-{\tilde{\mathit{g}}}_{i,l}^t\left({\mathcal{D}}^{\prime }\right)‖}_2$, reflects the sensitivity of the gradients to individual samples. Specifically, this difference captures the sum of gradients produced by performing L local iterations on that single sample. Given the constant $C$, which characterizes the maximum change in the gradient estimates, the upper bound on the sensitivity can be conservatively estimated as 2η$C$L$p_i$.  □

B.

Privacy Guarantee in Each Round

Privacy Amplification Theorem [34] states that if a mechanism $\mathcal{M}$ is $\left(\epsilon ,\delta \right)$-DP over a dataset $\mathcal{D}$ with size m and a subsampling mechanism is used to draw a subset ${\mathcal{D}}_s$ with size n (n ≤ m) uniformly at random from $\mathcal{D}$, then applying $\mathcal{M}$ to the subset ${\mathcal{D}}_s$ guarantees (ε′, (n/m)$\delta$)-DP, where ε′ is defined as ε′ = $\log (1+(n/m\left)\right(e^{\mathsf{\epsilon }}-1\left)\right)$.

Next, we establish the lower bound of the noise variance ${\sigma }^2$ for each client to achieve $\left(\epsilon ,\delta \right)$-DP in each round. This provides a valuable suggestion for introducing the appropriate noise for DP-FL. Given that the local updates of clients are conducted on random batches, we can apply the Privacy Amplification Theorem [34] to mitigate the noise variance and achieve $\left(\epsilon ,\delta \right)$-DP.

Lemma 2.

For each round t at client i, let $q_i=(Lb/\left|{\mathcal{D}}_i\right|)$ ($q_i\le 1$), and each batch is sampled without replacement, the Gaussian noise with variance ${\sigma }^2$ satisfying the condition

$${{\sigma }^2\ge 32\left({\displaystyle \frac{{\eta CL{q}_{i}p}_i}{\epsilon }}\right)}^2\mathrm{l}\mathrm{n}\left({\displaystyle \frac{1.25}{\delta }}\right)$$

(20)

can achieve  $\left(\epsilon , \delta \right)$-DP.

Proof. 

In the round t, client i performs L steps of local clipping SGD, where each batch $b$ is sampled without replacement from its local dataset ${\mathcal{D}}_i$. Let $q_i=(Lb/\left|{\mathcal{D}}_i\right|)$($q_i\le 1$), the Gaussian noise with variance ${\sigma }^2={\displaystyle \frac{2\mathrm{l}\mathrm{n}\left(\frac{1.25}{\delta }\right)}{{\epsilon }^2}}{S}_f^2$ can achieve ($\log (1+q_i\left(e^{\mathsf{\epsilon }}-1\right), q_i\delta )$)-DP based on Equation (13) and Privacy Amplification Theorem [34]. Since ε ∈ (0, 1), we can apply the following inequality:

$$\ln \left(1+q_i\left(e^{\mathsf{\epsilon }}-1\right)\right)\le q_i\left(e^{\mathsf{\epsilon }}-1\right)\le 2q_i\mathsf{\epsilon }$$

(21)

Thus, we can deduce that, given $\epsilon$ and $\delta$, the Gaussian noise with variance ${\sigma }^2={\displaystyle \frac{2\mathrm{l}\mathrm{n}\left(\frac{1.25}{\delta }\right)}{{\epsilon }^2}}{S}_f^2$ guarantees at least (2$q_i\epsilon$, $q_i\delta$)-DP. To achieve $\left(\epsilon ,\delta \right)$-DP in each global round t, the low bound for ${\sigma }^2$ can be reduced to

$${\sigma }^2={\displaystyle \frac{8{q_i}^2\ln (1.25/\delta )}{{\epsilon }^2}}{S}_f^2.$$

(22)

Since $S_f\le 2\eta CL{p}_i$ in Equation (19), we can infer that the low bound for ${\sigma }^2$ is

$${{\sigma }^2=32\left({\displaystyle \frac{{\eta CL{q}_{i}p}_i}{\epsilon }}\right)}^2\mathrm{l}\mathrm{n}\left({\displaystyle \frac{1.25}{\delta }}\right)$$

(23)

If p = $\underset{i}{\mathrm{m}\mathrm{ax}}{p}_i$, we can derive that the lower bound ${\sigma }_{min}=4q_i{\displaystyle \frac{\eta CLp}{\epsilon }}\sqrt{2\mathrm{l}\mathrm{n}\left({\displaystyle \frac{1.25}{\delta }}\right)}$.  □

3.5. Computational Complexity Analysis

The primary computational cost arises from the local model updates and the sparsity-based pruning processes. Accordingly, this section focuses primarily on analyzing the computational cost introduced by these two critical stages.

In the stage of local model training, the tailoring factor ${\gamma }_{i,l}^t$is computed based on the ${‖{\mathit{g}}_{i,l}^t‖}_2$ which involves summing the squares of all elements and taking the square root. Its complexity is O($d_j$), where $d_j$ is the dimensionality of the gradient vector for layer j. The process of multiplying each gradient vector ${\mathit{g}}_{i,l}^t$by a scalar has a one-time computational cost, which can be negligible. Considering that there are L iterations and J layers, the overall computational complexity of computing the scaling factors and updating the gradients is O(L$\ast {\sum }_{j=1}^J{d}_j$).

In the stage of sparsity-based channel pruning. To compute the JS divergence for client i, we first need to determine the probability of each class. This step has a minimum complexity of is O(|${\mathcal{D}}_i$|) since we need to iterate over each sample and update the counts for each class. Optimization techniques such as using a hash table or dictionary and parallel processing can improve the efficiency of the process. Then, a vector $P$ is constructed to store the probability of each class with a dimension equal to the number of class n. According to Equation (8), we need to access each element in vectors $P$ and $Q$ to compute KL divergence, so the complexity is O(n). Since addition operations have O(1) complexity, the overall complexity of computing JS divergence in Equation (9) is O(n + 1), approximately O(n). Next, we perform linear transformation and normalization to get $p_i$ per client, which has O(1) complexity as it involves a fixed number of simple arithmetic operations. Consequently, we sample a matrix ${\mathit{R}}_i$ from the Bernoulli($p_i$) distribution. Acquiring a binary mask based on $p_i$ introduces a relatively low computation cost, as Bernoulli sampling can be performed efficiently. The element-wise multiplication of the mask matrix ${\mathit{R}}_i$ and the accumulated gradients is an intrinsic operation and introduces a negligible computational overhead.

Incorporating Gaussian noise introduces additional computations during the global model aggregation process, which is executed on the server side. The complexity of computing the lower bound ${\sigma }_{min}$ is O(n), according to Equation (20), as it needs to find the maximum value within the vector $p_i$. The generation of Gaussian noise and the noise addition typically have a constant time complexity of O(1) depending on the random number generator used. These operations do not significantly impact the total computational overhead.

In summary, the main computational complexity comes from the computation of gradient norms and the counting of sample labels. In Python 3.8.18, the use of specialized libraries such as NumPy 1.24.3, Pandas 2.0.3, and PyTorch 2.1.2 can significantly accelerate the counting of sample labels and the computation of norms, reducing the overall computational complexity of the algorithm.

4. Results and Discussion

In this section, some experiments on the CIFAR-10 and Fashion-MNIST datasets are conducted to evaluate the performance of the proposed ASPFL algorithm.

4.1. Experimental Settings

The experimental results mentioned below are obtained on a computer with an AMD Ryzen 5 5600 6-core processor and 16 GB of RAM. Additionally, the computer is equipped with an NVIDIA GeForce RTX 3070 with 16 GB of memory.

The ratio of the training set to the test set is 8:2. The performance of the model is measured under non-IID distributions. Specifically, we utilize the Dirichlet (α) to simulate the dataset distributions across different clients. We adjust the degree of non-IID through the parameter α (α$>0$). A larger α indicates that the data sets are closer to being IID. To deploy the model on edge devices, we chose ResNet18 [35] as the backbone model, which mainly functions by multiple convolution layers. In the FL system, the number of clients N = 10 with various participating ratios. The initial learning rate η is set to 0.01 and the epochs of local training L is fixed at 3. We select the threshold $C$ for local gradient clipping as 1. As for the differential privacy, the privacy budget $\epsilon$ is pre-determined to 2.0 and $\delta$ = ${10}^{-5}$. We specify the maximum number of global iterations T as 200.

4.2. Pruning Performance Comparison

We first evaluated the performance of the pruning module in ASPFL without introducing the DP noise, hereafter denoted as ASPFL-p. With Dirichlet non-IID data, α is set to 0.5, 0.7, 1, and 10. As depicted in Figure 3, when α = 0.5, the distribution of sub-datasets across clients has shown the extreme imbalance numbers and severe label skew for non-IID data. It can be observed that some clients may have significantly high numbers of samples for a particular category and no samples for some categories, resulting in a skewed distribution.

Figure 4 and Figure 5 show that the ASPFL-p scheme achieved comparable accuracy and convergence speed to the original FedAvg [4] and FedLP [29] schemes. For instance, in Figure 4a and Figure 5a, when the data distribution is highly imbalanced ($\alpha =0.5$), ASPFL-p exhibits a smoother accuracy curve and faster convergence compared to the other two methods, without significant fluctuations. This suggests that our algorithm maintains stable performance during training, even when faced with extremely imbalanced data distributions. This phenomenon may be attributed to the fact that our algorithm incorporates L₂-norm clipping of accumulated gradients, which helps reduce the volatility of model updates and improve convergence stability on non-IID datasets.

Table 2. Comparisons on accuracy, communication rounds, and parameter saving.

Schemes	$\mathit{\alpha }$	Max acc %	Rounds for 80% acc
FedAvg [4]	$0.5$	74.32	7
	$0.7$	77.32	5
	$10$	80.34	4
FedLP [29]	0.5	74.33	15
	0.7	75.4	14
	10	76.79	12
ASPFL-p (ours)	0.5	76.75	5
	0.7	78.01	4
	1.0	79.22	3
	10	81.32	2

illustrates the comparison results on accuracy and communication overhead. Under different non-IID data distributions characterized by $\alpha$, ASPFL-p outperforms both FedAvg [4] and FedLP [29] in terms of accuracy. This is because our approach introduces an adaptive strategy that extracts the JS divergence as a feature to ensure the utility of the model. Additionally, the value of $\alpha$ also affects model performance; a larger $\alpha$ leads to a data distribution closer to IID, with a more balanced distribution of samples across clients, thus improving the performance of the aggregated global model. As for communication overhead, ASPFL-p also achieves lower communication rounds and smaller transmission parameters compared to the other two methods. In standard FL settings, models are typically fully connected without specific pruning techniques aimed at reducing model size. ASPFL-p introduces an adaptive pruning mechanism that is related to the data distribution feature to decrease the number of channels within the model. This approach considers the non-IID nature of the data, which may lead to reduced efficacy in model learning and introduce potential biases into the training process. Certain features might be more pronounced or significant in specific scenarios, such as medical data, thereby necessitating an adaptive approach to matrix design that accounts for these distributional disparities.

4.3. Privacy Performance Comparison

We assess the privacy performance of ASPFL with IID and Dirichlet (α = 15) non-IID data. We compare the proposed ASPFL with other DP-FL methods, which are either typical CDP-based [12,13] or LDP-based [25] in federated learning. Experimental results show that our ASPFL method has better performance compared with other methods. We use flat clipping with a constant clipping threshold $C$ = 1.

Next, we report the evaluation results on two image datasets: CIFAR-10 and Fashion-MNIST and provide a detailed explanation of the ASPFL framework. For the CIFAR-10 dataset, our proposed scheme demonstrates competitive accuracy across IID and non-IID. We set the ratio of participated clients to be 20%. The average test accuracy over all clients is reported in

Table 3. Comparison of DP-FL methods on CIFAR-10 over two participated clients.

Schemes	Test Accuracy %		Communication Rounds
	IID	Non-IID (α = 15)
DP-FedAvg [12]	82.68	81.12	200
$f$-DP [25]	80.37	78.84	200
Wu et al. [13]	81.25	79.05	170
ASPFL (ours)	83.57	82.43	120

. The accuracy under the IID setting (83.57%) is notably higher than DP-FedAvg [24] (82.68%), $f$-DP [25] (80.37%), and Wu et al. [13] (81.25%). Under non-IID settings, our method achieves higher accuracy (82.43%) than DP-FedAvg [24] (81.12%), $f$-DP [25] (78.84%), and Wu et al. [13] (79.05%) as well.

$f$ To analyze the communication overhead of our proposed method, Table 3 also lists the number of communication rounds (CR) required for the model to reach convergence. Our method demonstrates a relatively low number of communication rounds (120), potentially offset by improved accuracy. Relative to DP-FedAvg [24] (200) and f-DP [25] (200), our proposed ASPFL reduces the number of CRs by 40% for local optimization. In summary, our method maintains relatively low communication overhead while achieving competitive accuracy. These enhancements collectively enable federated learning to improve efficiency and privacy protection while maintaining accuracy. Specifically, the introduction of adaptive Gaussian noise, along with the use of the Jensen-Shannon divergence as a metric for generating sparse matrices, contributes to more efficient model updates. This reduces the communication overhead, which is crucial in distributed learning settings where data is stored across multiple devices.

4.4. Impact on the Number of Participants

The following presents the results of the ASPFL-p system under different participant client numbers. We set the participated clients to be 2, 5, 8, and 10. Figure 6 depicts the accuracy curves corresponding to various ratios of participated clients (20%, 50%, 80%, and 100%) with non-IID Dirichlet ($\alpha$ = 0.5) on the CIFAR-10 and Fashion-MNIST datasets.

As shown in Figure 6, it appears that increasing the number of participating clients leads to faster convergence and better final performance. On one hand, it attributes to the diversity and large amount of data because more clients mean a better representation of the underlying data distribution. Federated learning aggregates gradients from multiple clients, which helps the model learn from a wider variety of data points. This diversity can lead to better generalization and thus improved performance. On the other hand, with more clients contributing to the gradient aggregation, the resulting gradients are more stable and less noisy. This stability helps the model converge more quickly and accurately.

5. Conclusions

We present a communication-efficient federated learning framework with a privacy guarantee in this paper. Before transmitting model updates, a sparsity-pruning-based mechanism is employed based on the JS divergence of each client’s data distribution. When introducing privacy protection in model updates, we propose an adaptive Gaussian noise strategy to achieve ($\epsilon ,\delta$)-DP. Based on experimental results in terms of accuracy and communication cost, we can conclude that our ASPFL scheme exhibits robust performance and competitive accuracy on the CIFAR-10 dataset. From another perspective, a multitude of experiments have demonstrated the portability of our proposed strategies to other domains. With an increase in data diversity and complex tasks, the methods proposed in this paper are expected to achieve an even better balance on accuracy and overhead in the future.