One-Shot Federated Learning with Label Differential Privacy

Abstract

Federated learning (FL) has emerged as an extremely effective strategy for dismantling data silos and has attracted significant interest from both industry and academia in recent years. However, existing iterative FL approaches often require a large number of communication rounds and struggle to perform well on unbalanced datasets. Furthermore, the increased complexity of networks makes the application of traditional differential privacy to protect client privacy expensive. In this context, the authors introduce FedGM: a method designed to reduce communication overhead and achieve outstanding results in non-IID scenarios. FedGM is capable of achieving considerable accuracy, even with a small privacy budget. Specifically, the authors devise a method to extract knowledge from each client’s data by creating a scaled-down but highly effective synthesized dataset that can perform similarly to the original data. Additionally, the authors propose an innovative approach to applying label differential privacy to protect the synthesized dataset. The authors demonstrate the superiority of the approach over traditional methods by requiring only one communication round and by testing using four classification datasets for evaluation. Furthermore, when comparing the model performance for clients using their method against traditional solutions, the authors find that the approach achieves significant accuracy and better privacy.

1. Introduction

Federated learning (FL) is a powerful approach designed to leverage decentralized computing and data resources to overcome data silos [1]. Despite its potential, FL encounters significant challenges when applied to modern neural networks, which often consist of hundreds of millions of parameters [2,3]. These challenges are compounded by the traditional FL approach, which requires numerous communication rounds, exacerbating the issues of scalability and communication load. This dual burden of excessive parameters and frequent communications significantly hinders the efficiency of FL methods and can impede the widespread deployment of FL models, especially in environments where communication resources are limited [4,5]. Additionally, the uneven distribution of data across different clients typically seen in real-world settings makes it difficult to optimize a global model [6,7,8]. This often results in model biases and suboptimal performance. These issues highlight the need for a new FL framework that not only reduces the communication overhead but also enhances the performance of the global model in scenarios with non-independent identically distributed (non-IID) data [9].

In this aspect, though FL offers the advantage of allowing participants to retain their raw data locally, which inherently addresses some aspects of data privacy [10], the privacy concerns in FL are far from being fully resolved, as both the model parameters exchanged during training and the outputs from the trained models can still be exploited for privacy attacks [10,11,12]. To counter these vulnerabilities, various strategies have been developed to enhance privacy in FL, including the use of trusted aggregators and complex cryptographic techniques such as homomorphic encryption [13,14]. Despite these efforts, these methods often do not allow for personalized privacy settings and may not effectively protect against breaches involving high-dimensional parameter vectors, which are particularly challenging to secure [10].

In response to these privacy limitations, the LDP-fed model has been introduced as an innovative FL framework that supports local differential privacy. This model allows each participant to apply their own privacy settings, thus offering tailored privacy protection based on individual preferences [15,16]. However, this approach is weak given the increasing complexity and dimensionality of models used in modern FL applications, such as those based on deep learning architectures like ResNet and BERT [2,3,17]. Traditional privacy protection methods struggle to scale with the size of these models and often perform inadequately under high privacy requirements, necessitating extensive computational resources [8,18]. Therefore, LDP-fed requires innovations in protection mechanisms or greater efficiency and adaptability.

In this paper, a novel federated framework is designed to address issues related to both privacy and communication volumes. Traditional approaches often require each client to maintain and train its own model and send gradients or weights to a central server for aggregation. Instead, this work proposes creating compact yet high-performance surrogate data for each client. These surrogate datasets, which encapsulate knowledge from the original data, are difficult to visually interpret, thus preserving privacy. Meanwhile, only a one-shot communication round is required, achieving great server performance.

These local surrogate datasets are sent to the server, which constructs a global surrogate function to facilitate updates by minimizing this function. The surrogate data are generated through a process inspired by data distillation techniques. Local surrogate functions are crafted by synthesizing datasets that approximate the objectives; this is achieved by aligning gradients of the original and surrogate datasets across different neural network initializations.

Moreover, the paper explores the challenge of maintaining privacy when surrogate datasets are used. Visual inspection alone does not reveal sensitive information from these datasets; however, using labels could potentially expose privacy-sensitive category information. To address this, an optimized label perturbation method tailored for federated learning (FL) scenarios is developed, striking a balance between data utility and privacy protection.

The framework, including the processes for creating and utilizing surrogate data, and the method for label perturbation are detailed in Figure 1. This approach significantly reduces communication costs and enhances privacy without compromising the usability of the data in federated learning environments. The contributions of this work are summarized as follows:

• A method called FedGM is introduced, which utilizes iterative gradient matching to learn a surrogate function. This technique involves transmitting synthesized data to the server rather than sending local model updates, significantly enhancing communication efficiency and effectiveness. Additionally, a novel strategy for selecting the original dataset reduces the number of training rounds required while improving the training effectiveness of the distilled dataset.
• Label differential privacy is employed to protect the privacy of approximate datasets for each client. This method is found to be highly capable even with a small privacy budget and outperforms other methods.
• Comprehensive experiments are conducted on three tasks and show that the proposed framework can achieve high performance with just one communication round in scenarios marked by pathological non-IID conditions.

The remainder of this paper is organized as follows: Section 2 reviews the related work of the paper. Section 3.1 presents the difference between the FedGM method and traditional FL. Section 3.2 provides a detailed introduction to the FedGM method, including its advantages and innovative aspects. Section 3.3 experimentally demonstrates that the surrogate dataset sent by each client to the central authority in the FedGM setup is visually private, and label protection is achieved using an innovative approach to label differential privacy. In Section 4, numerous experiments are conducted to demonstrate the superiority of the work in terms of handling non-independent and identically distributed (non-IID) data, reducing communication overhead, and by comparing it with traditional LDP-fed approaches.

2. Related Works

2.1. Federated Learning

FL was first introduced by [1] and has the goal of training models collaboratively. Some well-known FL methods, such as FedAvg [1], involve aggregating models received from individual client devices and updating the global model by averaging the model parameters with appropriate weights. However, due to its reliance on iterative model averaging, FedAvg encounters challenges related to heterogeneity in the FL system, particularly when dealing with non-IID data partitioning. This leads to a degradation in the overall performance of the global model and contributes to increased communication overhead. To mitigate this issue, several modifications and improvements to FedAvg have been introduced. For example, FedProx [9] adjusts the loss function, while FedNova [19] and SCAFFOLD [20] utilize additional information to address distribution shifts. In addition to enhancing learning algorithms for faster convergence, another approach to enhance efficiency is the explicit reduction in communication costs. However, the majority of current improved algorithms for non-IID scenarios in FL require multiple rounds of gradient and model exchange, resulting in a significant number of communication rounds. At the same time, the privacy of client models and their private data is also challenged by these approaches.

2.2. One-Shot Federated Learning

FL with one-shot communication has been explored in various projects. For instance, ref. [21] presents an algorithm for training a support vector machine in a one-shot manner and proposes a framework that not only uploads models but also includes information about the local dataset distribution: a task that is highly inappropriate when dealing with privacy-sensitive datasets. Meanwhile, ref. [22] employs knowledge transfer to distill models within each client, potentially altering the original model structure and incurring significant communication costs when sharing student models. Furthermore, ref. [23] attempts to alter the original optimization objective by sending distilled images to the central server, but it does not take privacy protection into account.

In this work, instead of distilling models, our framework allows each client to create an approximate dataset that mimics the performance of the original dataset and sends it to the central server.

2.3. Local Differential Privacy

Differential privacy [24] is a privacy-preserving technique designed to ensure that sensitive information of individuals is not disclosed when analyzing data. This method involves adding noise to the data in such a way that the analysis results do not depend on the data contributions of individual entities, thus preventing re-identification attacks against individuals. It provides a mathematical framework to balance the trade-off between privacy and data utility by controlling the introduction of random noise. Local differential privacy [25] is a variant of differential privacy that focuses on protecting the privacy of individual data rather than the results of the analysis. In local differential privacy (LDP), each individual client adds noise to its data locally before reporting it to the data processor to prevent the data processor from obtaining detailed information about individuals. This approach allows individuals to have autonomous control over the privacy of their data and reduces dependence on trusted centralized data processing. Due to the lack of a trusted central authority in real-world scenarios, recent research on privacy protection in federated learning has primarily focused on LDP. However, with the increasing complexity of models, traditional LDP privacy protection methods [15] in FL inevitably face increased computational requirements. Additionally, as privacy budgets change, users often experience a significant decrease in accuracy.

2.4. Dataset Distillation

Dataset distillation [26] has emerged as an essential paradigm in the context of machine learning, particularly when dealing with the challenges posed by the increasing size of training data. This approach focuses on generating a condensed version of a large dataset without significantly compromising the model’s performance [27,28], primarily in terms of test accuracy. Initially proposed by [29], dataset distillation has been refined through various methodologies such as meta learning [29,30], gradient matching [31,32,33], distribution matching [34,35], neural kernels [27,28], and generative priors [36,37]. These approaches show significant promise when applied to datasets like CIFAR10, yet they encounter difficulties when attempting to scale to larger datasets such as ImageNet.

2.5. Dataset Distillation in Federated Learning

Previous research [38] on dataset distillation has mentioned the benefits of dataset distillation in FL. FedD3 [23] has attempted to build a surrogate dataset in each client through KIP, but the authors did not use privacy protection technologies to safeguard the clients’ distilled datasets. FedDM [39] tried to apply distribution matching for each client to build a surrogate dataset, and they used differential privacy to protect the distilled dataset. However, in the experiment, the research found that the distilled data itself are not sensitive, and some researches [40,41] have proven that the distilled data can withstand membership inference attacks. Thus, the research believes that privacy protection for distilled datasets should be more focused on the labels of the distilled dataset. Furthermore, both methods need a great deal of time to train at the client. In this work, the research uses a clustering process to select representative original images. This gradient matching target has shifted from randomly sampled original datasets at the outset to intentionally selected cluster datasets after clustering. Through experiments, the research has observed that the approach requires fewer training epochs compared to traditional dataset distillation in FL and achieves better performance more rapidly.

3. Methodology

In Section 3.1, the paper demonstrates the differences between the authors’ framework and the traditional iterative minimization framework in FL.

In Section 3.2, the paper shows details of the authors’ implementation of FedGM. The authors also demonstrate a novel method that can reduce the training time.

In Section 3.3, the paper analyzes the visual privacy of distilled images. Then, the authors proceed to demonstrate how to protect the distilled data using label differential privacy.

3.1. Differences between FedGM and the Traditional Iterative Minimization Framework in FL

Traditional FL is a scenario in which a set of clients individually train their own models using their private data, and a central server receives the models from the clients; the authors consider there are data D distributed among K clients, and each of the clients has a local private training dataset $D_k=\{x_k^i,y_k^i\},1\le i\le n_k$, where $n_k$ represents the number of data points assigned to each client, and ${\sum }_{i=1}^k{D}_k$=D. Each client has its weight value $p^k$, and ${\sum }_{i=1}^k{p}^k$=1. In a multi-class classification task under FL, the minimization target is as follows:

$$\underset{w}{min}\{L(w,D)\stackrel{▵}{=}\sum _{k=1}^K{p}^k{L}^k(w,D_k)\}$$

(1)

where $L^k(·)$ is a local objective function that is optimized on the k-th client. The loss function $L^k(·)$ is formulated as follows:

$$L^k(w,D_k)\stackrel{▵}{=}\frac{1}{n^k}\sum _{i=1}^{n^k}l(w,x_i^k,y_i^k)$$

(2)

where $l(·)$ is loss function for client, and w is model parameters. Most FL algorithms need the clients to transmit the locally trained models to the server multiple times: the communication burden cannot be neglected. Though one-shot FL has been proposed to reduce the cost of communication, its performance is not good. Meanwhile, as each client can only see local data, which can result in the absence of certain categories of information during the training process for each client, local updating is always unable to capture global information. Meanwhile, since the client is trained with biased data, it becomes challenging for the server to improve the joint update direction by taking into account more complex interactions among different clients. Regarding these factors, the authors have successfully altered the local update strategy. Instead of transmitting the model itself, the client transmits surrogate data $S_k$ that exhibits similar training performance to the original data $D_k$; thus, the authors change the client local objective to

$$L^k\left(w\right)=\frac{1}{n_s^k}\sum _{i=1}^{n_s^k}l(w,\widetilde{x_i^k},\widetilde{y_i^k})\approx L^k(w,S_k)$$

(3)

Therefore, the server can train the local model using the surrogate data transmitted by each client. The authors can denote the objective as

$$\underset{w}{min}L(w,D)\approx \underset{w}{min}L(w,\sum _{k=1}^K{S}_k)$$

(4)

where $L(·)$ is a global objective function that is optimized on the server.

3.2. FL with Efficient Local Gradient Matching

The authors show how to build a surrogate dataset in each client in Section 3.2.1, and they introduce an improvement in the training efficiency for their method in Section 3.2.2.

3.2.1. Local Gradient Matching

Drawing inspiration from recent advancements in data distillation [31,32,33], it becomes feasible to train a collection of synthesized data for each client that effectively represents the original data according to the objective function. The methodology of local gradient matching revolves around the concept of synthesizing a compact dataset that captures the essence of the original data distributed across clients in a federated setting. This compact dataset, denoted as $S^{*}$, is obtained by optimizing the following objective:

$$S^{*}=arg\underset{S}{min}{C}^T\left({\theta }^S\right)\mathrm{subject}\mathrm{to}{\theta }^S=arg\underset{\theta }{min}{L}^S\left(\theta \right).$$

(5)

Here, $L^S\left(\theta \right)$ represents the loss function computed over the synthetic dataset S, and $C^T\left({\theta }^S\right)$ denotes the curriculum-based cost function over the course of T iterations. The process begins by initializing two models with identical architectures and parameters, which are then trained on the client’s local original data and the synthetic dataset, respectively.

The optimization is carried out by matching the gradients of the synthetic model parameters (${\theta }^S$) with the target model parameters (${\theta }^T$) at each iteration. This is described by the equation:

$$\underset{S}{min}{E}_{{\theta }_0\sim P_{{\theta }_0}}\left[\sum _{t=0}^{T-1}D({\theta }_t^S,{\theta }_t^T)\right]$$

(6)

$$\mathrm{subject}\mathrm{to}{\theta }_{t+1}^S=\mathrm{opt}\text{-}\mathrm{alg}({\theta }_t^S,{\varsigma }_S)\mathrm{and}{\theta }_{t+1}^T=\mathrm{opt}\text{-}\mathrm{alg}({\theta }_t^T,{\varsigma }_T)$$

(7)

where D is a distance measure between the model parameters at each step, opt−alg denotes the optimization algorithm, and $\varsigma$ indicates the number of optimization steps. The primary goal is to generate synthetic samples that can result in gradient trajectories similar to those obtained from the original dataset.

In practice, this method leverages an approximate form of the backpropagation algorithm, which allows for a single backpropagation pass per step without the need for multiple levels of nesting, thereby simplifying the optimization process:

$$\underset{S}{min}{E}_{{\theta }_0\sim P_{{\theta }_0}}\left[\sum _{t=0}^{T-1}D({\nabla }_{\theta }{L}^S\left({\theta }_t\right),{\nabla }_{\theta }{L}^T\left({\theta }_t\right))\right]$$

(8)

This approach ensures that the optimization is both memory-efficient and computationally feasible for deployment on edge devices within an FL framework.

3.2.2. Local Gradient Matching with Training Efficiency

The efficiency of training processes in dataset distillation is highly contingent on the choice of original images. Traditional methods rely on random sampling to select original images for matching, which often leads to a significant waste of training time. However, this framework enhances efficiency by employing a strategic sampling technique that not only reduces the training time of each client but also improves the effectiveness of the distilled images compared with random sampling.

The authors adopt a novel strategy that clusters samples of the same class from the client into different regions. A previous study [42] shows that samples closer to the class distribution center provide smaller backward gradients and more effective supervision, whereas samples near the boundary offer diverse conditions. This method allows us to extract surrogate datasets in each client that more accurately reflect the global data distribution, thereby improving the gradient matching process.

Gradient matching is then performed by aligning the synthetic images with the representative samples, which avoids the chaotic targeting that degrades performance in traditional methods. Furthermore, this approach mitigates the imbalance in mini-batch coverage of the original class sample distribution, addressing the instabilities in random sampling and optimization observed in previous approaches.

Inspired by [42], to get representative samples, the authors need to divide sub-clusters. So the authors utilize the k-means [43,44] algorithm. This clustering occurs within each class and generates sub-clusters that cover the entire sample space of the class: ensuring an even distribution and providing the algorithm with a diverse set of samples that accurately represent the sample density.

This clustering leads to a more balanced gradient distribution across different samples, aligning perfectly with the principle of maintaining diversity while ensuring an even distribution. The complete FL training pipeline is then followed, wherein each client distills the data locally and transmits the distilled dataset to the server. The server leverages the synthesized data to update the global model, significantly enhancing the training efficiency and effectiveness of the FL system.

The overall framework is shown in Algorithm 1. Each client distills surrogate data and transmits them to the server; then, the server updates the global model with the surrogate data from all the clients.

3.3. One-Shot FL with Label Differential Privacy

The previous study of FL with dataset distillation seldom mentioned data privacy protection, while some studies that mention privacy protection only focus on the distilled images themselves. In this part, the authors analyze the visual privacy of distilled images and find the distilled images themselves are not sensitive when the authors try to protect privacy. Thus, the authors consider protecting the labels with label differential privacy.

Algorithm 1 Federated learning with efficient local gradient matching

1: Input: Client training set $D_k$, set of synthetic samples from client $S_k$, deep neural network parameterized with w, probability distribution over parameters $P_w$, global learning rate ${\eta }_g$, classes of original data c, size of minibatch gradient matching m, number of steps for updating weights ${\zeta }_{\theta }$ and synthetic samples ${\zeta }_S$ in each inner-loop step, learning rates for updating weights ${\eta }_{\theta }$ and synthetic samples ${\eta }_S$ 2: Server executes: 3: for  $k=1,\cdots ,K$  do 4:     Transmit set of synthetic samples from client $S_k$ to the server 5: end for 6: Aggregate synthesized dataset from each client and build the surrogate function update weights to $w_{r+1}$ on N by SGD with the learning rate ${\eta }_g$ 7: Each client distills data: 8: for  $e=1,\cdots ,E$  do 9:     Initialize ${\theta }_0\sim P_{{\theta }_0}$ 10:     for $t=0,\dots ,T-1$ do 11:         for $c=0,\dots ,C-1$ do 12:            Get embeddings for each sample in $D_k$ using ${\theta }_0$ 13:            Compute L2 distance among samples’ embeddings and using k-means to cluster them into m subsets 14:            Sample a minibatch pair $B_c^{D_k}\sim D_k$ from the center of clusters 15:            Sample a minibatch pair $B_c^{S_k}\sim S_k$ 16:            $B_c^{T_k}$ and $B_c^{S_k}$ are of the same class c 17:            Compute $L_T^c=\frac{1}{|B_c^{D_k}|}{\sum }_{(x,y)\in B_c^{D_k}}\ell ({\varphi }_{{\theta }_t}\left(x\right),y)$ and $L_S^c=\frac{1}{|B_c^{S_k}|}{\sum }_{(s,y)\in B_c^{S_k}}\ell ({\varphi }_{{\theta }_t}\left(s\right),y)$ 18:            Update $S_c\leftarrow {opt-alg}_{S_k}(D({\nabla }_{\theta }{L}_{S_k}^c\left({\theta }_t\right),{\nabla }_{\theta }{L}_{D_k}^c\left({\theta }_t\right)),{\zeta }_S,{\eta }_{S_k})$ 19:            Update ${\theta }_{t+1}\leftarrow {opt-alg}_{\theta }(L_S\left({\theta }_t\right),{\zeta }_{\theta },{\eta }_{\theta })$ 20:         end for 21:     end for 22: end for 23: Output the set of synthetic samples from client $S_k$ 24: Transmit synthetic samples from client $S_k$ to the server

3.3.1. Visual Privacy of Distilled Images

Assume the authors have a malicious server. It can visualize synthetic data directly and make comparisons with the target sample to deduce membership. The authors use VGG [45] as the backbone to recognize the synthetic images and measure the similarity between the synthetic images and the original images using the L2 distance. Figure 2 shows examples of clients’ distilled images and the their (top-two) most similar real images: images with the lowest L2 relative to the synthetic images at the tops of the columns [40]. The authors observe that the real data share similar contour patterns with the synthetic images, but the authors cannot observe more fine-grained features from the synthetic images. Meanwhile, the authors use ResNet [17] as the backbone to classify the distilled images; the authors input the distilled images as the test set into the network. Figure 3 shows the test accuracy. The authors find that few of the distilled images can be classified correctly, which means the distilled images themselves are not sensitive. An experiment is also conducted using the MMAFEDB facial expression recognition dataset provided by Kaggle to compare a dataset initialized with real images, as shown in Figure 4, and a dataset synthesized through knowledge distillation performed locally on the client side before being uploaded to the central party, as shown in Figure 5. It is found that the synthesized distilled dataset makes it difficult to observe the characteristics of the original data, as facial expressions and contours are distorted; thereby, the original facial information is protected. Thus, the authors try to protect the labels of distilled images.

3.3.2. Implementation of Label Differential Privacy

To protect the labels of the distilled dataset, the authors apply label differential privacy. Before analyzing their method, the authors first review some fundamentals of differential privacy.

Definition 1

(Randomized Response [24]). Given a positive integer K, let $\left[K\right]=\{1,\dots ,K\}$. The randomized response $RR$ is defined as follows: for a parameter $\epsilon \ge 0$ and a true value $y\in \left[K\right]$ known to $RR$, when an observer queries the value of y, the $RR$ mechanism provides a random response $\tilde{y}$ according to the following probability distribution:

$$Pr(\tilde{y}=y)=\left\{\begin{array}{cc}\frac{e^{\epsilon }}{e^{\epsilon }+K-1}\hfill & if\tilde{y}=y,\hfill \\ \frac{1}{e^{\epsilon }+K-1}\hfill & if\tilde{y}\ne y.\hfill \end{array}\right.$$

(9)

Definition 2

(Differential Privacy [24]). Let $ϵ,\delta \in {\mathbb{R}}_{\ge 0}$. A randomized algorithm A taking as input a dataset is said to be $(ϵ,\delta )$-differentially private $(ϵ,\delta )$-DP) if for any two neighboring datasets D and $D^{\prime }$ and for any subset S of outputs of A, it is the case that $Pr[A\left(D\right)\in S]\le e^{ϵ}·Pr[A\left(D^{\prime }\right)\in S]+\delta$. If $\delta =0$, then A is said to be ϵ-differentially private (ϵ-DP).

Definition 3

(Label Differential Privacy [18]). Let $ϵ,\delta \in {\mathbb{R}}_{\ge 0}$. A randomized training algorithm A taking as input a dataset is said to be $(ϵ,\delta )$-label differentially private $(ϵ,\delta )$-Label DP) if for any two training datasets D and $D^{\prime }$ that differ in the label of a single example and for any subset S of outputs of A, it is the case that $Pr[A\left(D\right)\in S]\le e^{ϵ}·Pr[A\left(D^{\prime }\right)\in S]+\delta$. If $\delta =0$, then A is said to be ϵ-label differentially private (ϵ-Label DP).

Traditional label differential privacy employs a randomized response algorithm on the label of each data point equally. However, this approach can significantly impact the training effectiveness. Motivated by [18], the authors employ RRTop-k and RRwithPrior, which is shown in Algorithms 2 and 3 and is a method that, by leveraging prior probability knowledge of labels, achieves better label perturbation, striking a balance between privacy and data utility.

In detail, the authors consider the global model $M_t$ as a probabilistic classifier to each client. Given an unlabeled sample x, this classifier enables the assignment of a probability $p_y$ for each class $y\in \left[K\right]$, and each client partitions its distilled data into T non-overlapping subsets. The probabilistic classifier initially outputs an identical probability distribution across all classes in the data.

Algorithm 2 RRTop-k

Require:  A label $y\in \left[K\right]$$k\in \left[K\right]$, prior $p=(p_1,\dots ,p_K)$         Let $Y_k$ be the set of k labels with maximum prior probability (with ties broken arbitrarily) 2: if  $y\in Y_k$  then         Then output y with probability $\frac{ϵ}{ϵ+k-1}$ and output $y^{\prime }\in Y_k\setminus \left\{y\right\}$ with probability $\frac{1}{ϵ+k-1}$ 4: else         Output an element from $Y_k$ uniformly at random 6: end if

Algorithm 3 RRWithPrior

Require:  A label $y\in \left[K\right]$ prior $p=(p_1,\dots ,p_K)$       for  $k\in \left[K\right]$  do       Compute $w_k:=\frac{ϵ}{ϵ+k-1}·\left({\sum }_{j\notin Y_k}{p}_j\right)$, where $Y_k$ is the set of k labels with maximum prior probability (with ties broken arbitrarily) 3: end for     Let $k^{*}=arg{max}_{k\in \left[K\right]}{w}_k$     Return an output of RRTop-k$\left(y\right)$ with $k=k^{*}$

Using the probabilistic classifier obtained from the previous round of training, the authors assign corresponding probabilities $p=(p_1,\dots ,p_K)$ for each sample $x_i$ in the subset. Utilizing these probabilities, p helps to refine the randomized response (RR): specifically, through the RRWithPriorp algorithm. The perturbed labels obtained through RRWithPriorp contribute to the formation of the perturbed dataset. All the data subjected to RRWithPriorp are combined into a larger noise set and sent to the central server for training. Then, the server learns the prior knowledge of the dataset and sends the global model $M_t$ to clients as the probabilistic classifier. The authors explain the entire process in Algorithm 4.

Algorithm 4 One-Shot Federated Learning with Label Differential Privacy

Input: Client synthetic dataset $S_k$, deep neural network parameterized with w, global learning rate learning rate ${\eta }_g$, global model M   Output: Noise set for central server training   for each client k do 4:       Partition the distilled data into T non-overlapping subsets           for $t=1,\dots ,T$ do            Use the classifier $M_t$ from the previous round to assign probabilities $p=(p_1,\dots ,p_K)$            Refine randomized response using the RRWithPriorp algorithm 8:        Form the perturbed dataset with the obtained labels        end for        Combine all perturbed data into a noise set        Send the noise set to the central server 12: end for      Server executes:      for  $k=1,\cdots ,K$  do          Transmit set of synthetic samples from client $S_k$ to the server 16: end for      Aggregate synthesized dataset from each client and update weights to $w_{r+1}$ on client noise set by SGD with the learning rate ${\eta }_g$ and send the global model $M_t$ as classifier to clients

3.3.3. Proof of Differential Privacy

To demonstrate that this algorithm conforms to the definition of differential privacy, it is imperative to demonstrate that the RRWithPrior algorithm satisfies differential privacy. The authors begin by scrutinizing the differential privacy properties of the RRTop-k mechanism shown in Algorithm 2. Consider two arbitrary inputs y and $y^{\prime }$ from the set $\left[K\right]$ and an arbitrary possible outcome $\widehat{y}$ from the set $Y_k$. The authors observe that the probability $Pr[\mathrm{RRTop}\text{-}k\left(y\right)=\widehat{y}]$ is at its maximum when y is the same as $\widehat{y}$, while $Pr[\mathrm{RRTop}\text{-}k\left(y^{\prime }\right)=\widehat{y}]$ is minimized when $y^{\prime }$ is any element of $Y_k\setminus \left\{\widehat{y}\right\}$. This leads to the following relationship:

$$\frac{Pr[\mathrm{RRTop}\text{-}k\left(y\right)=\widehat{y}]}{Pr[\mathrm{RRTop}\text{-}k\left(y^{\prime }\right)=\widehat{y}]}=\frac{\frac{ϵ}{ϵ+k-1}}{\frac{1}{ϵ+k-1}}=ϵ.$$

(10)

Therefore, RRTop-k satisfies $ϵ$-differential privacy.

Moreover, since RRWithPrior can be conceptualized as the RRTop-k algorithm, which calculates the threshold k maximizing $Pr\left[\mathrm{RRTop}\text{-}k\right(y)=y]$, it follows that RRWithPrior also satisfies $ϵ$-differential privacy.

To further substantiate that this algorithm is compliant with differential privacy, the authors consider any $ϵ>0$ and posit that if RRWithPrior is $ϵ$-DP, then this one-shot federated learning with differential privacy (OSFLDP) is $ϵ$-LabelDP. The algorithm maintains $ϵ$-DP even when disclosing all T models $M^{\left(1\right)},\dots ,M^{\left(T\right)}$ and all randomized labels $y_1,\dots ,y_n$. The joint probability for any conceivable set of output models $m^{\left(1\right)},\dots ,m^{\left(T\right)}$ and corresponding output labels $z_1,\dots ,z_n$ can be partitioned into a product of individual probabilities for each model and a product of conditional probabilities for each label given by the trained global model.

When examining two datasets D and $D^{\prime }$ that are identical except for the label of a single user, the probability terms for D and $D^{\prime }$ are congruent for all but one term related to the differing user’s label. This particular term is indicative of the probability that RRTop-k assigns a specific label. As RRWithPrior guarantees $ϵ$-DP, it ensures that the discrepancy in probabilities caused by changing one user’s label is bounded by $e^{ϵ}$. Hence, this algorithm satisfies $ϵ$-differential privacy, thereby fulfilling $ϵ$-LabelDP as desired.

4. Experiments

4.1. Experimental Settings

In this work, the authors conduct experiments on MNIST [46], FashionMNIST [47], CIFAR10 [48], and SVHN [49]. In the experiments, the authors use a three-layer convolutional network (ConvNet-3) [50] with 128 filters and instance normalization for each client to employ local gradient matching and to serve as a probabilistic classifier in each client. The authors select the batch size as 128 for representative real images, and the authors conduct 300 matching iterations in each client, inside each of which 100 inner loops are conducted. SGD [51] is set as the optimizer for each clients’ distilled image generator with a learning rate of 0.01. The clustering process utilizes a matching model for extracting features. The interval between each clustering iteration, referred to as $D_{int}$, is established at 10 iterations. The training efficiency driven by this clustering process is examined in Section 4.5 of the analysis. The authors tune the number of images per class (ipc) within [3, 5, 10] for each client under the IID setting, and the number of clients is 10, but when it comes to the non-IID setting, the authors consider a thornier scene, which is called a label distribution skew case. A previous study [52] showed that among different non-IID data settings, all studied FL algorithms performed worse on the label distribution skew case, and for the label distribution skew setting, the algorithms have the worst accuracy when each party only has data from a single label, which is also called pathological non-IID; thus, the authors focus on this scene to show the performance of the method. The authors design a pathological non-IID scene wherein each client only has one or two classes of data. In this non-IID setting, the authors tune the ipc within [30, 50, 100], and each client’s synthetic images are initialized as representative sampled real images. For the baseline method, the authors choose a batch size of 128 for local training. The authors tune the learning rate at the client from [0.001, 0.01, 0.1] and at the server from [0.01, 0.1, 1], and the local epoch is from [1, 2, 5, 10, 15, 20]. In particular, the authors tune $\mu$ for FedProx in [0.001, 0.01, 0.1, 1].

The authors compare FedGM with four representative model-averaging-based methods. The federated methods are: FedAvg [1], FedProx [9], FedNova [19], and SCAFFOLD [20]; the four federated learning methods and their comparative results are shown in

Table 1. Comparison of results of various methods. For non-IID settings, ipc = 50; for IID settings, ipc = 10. C represents the number of data classes a client has. The authors also include a Dirichlet distribution scenario with $\alpha$ = 0.5. The number of clients is 10.

Dataset	Partitioning	FedAvg	FedProx	SCAFFOLD	FedNova	FedGM
MNIST	$p_k\sim Dir\left(0.5\right)$	98.9% ± 0.1%	98.9% ± 0.1%	99.0% ± 0.1%	98.9% ± 0.1%	96.1% ± 0.1%
	#C = 1	29.8% ± 7.9%	40.9% ± 23.1%	39.9% ± 0.2%	39.2% ± 22.1%	99.3% ± 0.1%
	#C = 2	97.0% ± 0.4%	96.4% ± 0.3%	95.9% ± 0.2%	94.5% ± 1.5%	99.2% ± 0.1%
	IID	99.1% ± 0.1%	99.1% ± 0.1%	99.1% ± 0.2%	99.1% ± 0.2%	99.2% ± 0.1%
FMNIST	$p_k\sim Dir\left(0.5\right)$	88.1% ± 0.1%	88.1% ± 0.4%	88.1% ± 0.5%	88.6% ± 0.3%	89.6% ± 0.3%
	#C = 1	10.2% ± 0.1	28.8% ± 0.3%	11.8% ± 0.2%	15.8% ± 0.2%	85.7% ± 0.1%
	#C = 2	74.3% ± 0.1%	71.3% ± 0.1%	42.8% ± 28.7%	69.5% ± 5.7%	85.3% ± 0.1%
	IID	89.3% ± 0.1%	89.1% ± 0.2%	89.8% ± 0.1%	89.4% ± 0.3%	87% ± 0.0%
CIFAR-10	$p_k\sim Dir\left(0.5\right)$	68.4% ± 0.3%	65.9% ± 0.5%	65.8% ± 0.9%	65.3% ± 1.5%	61.4% ± 0.3%
	#C = 1	10.3% ± 0.5%	12.3% ± 2.0%	10.0% ± 0.0%	10.0% ± 0.0%	74.5% ± 0.1%
	#C = 2	49.8% ± 3.3%	50.7% ± 1.7%	49.1% ± 1.7%	46.5% ± 3.5%	75.5% ± 0.2%
	IID	72.4% ± 0.2%	70.2% ± 0.1%	71.5% ± 0.3%	69.5% ± 1.0%	62.4% ± 0.4%
SVHN	$p_k\sim Dir\left(0.5\right)$	86.1% ± 0.7%	86.6% ± 0.9%	86.8% ± 0.3%	86.4% ± 0.6%	86.5% ± 0.3%
	#C = 1	11.1% ± 0.1%	18.6% ± 0.2%	6.8% ± 0.0%	10.6% ± 0.2%	85.3% ± 0.2%
	#C = 2	79.2% ± 0.5%	80.3% ± 0.5%	64% ± 10.6%	72.4% ± 3.8%	85.7% ± 0.4%
	IID	87.5% ± 0.3%	85.5% ± 0.7%	88.5% ± 0.2%	87.4% ± 0.4%	87.9% ± 0.4%

.

4.2. Comparison with Other One-Shot FL Methods

In this section, the authors compare this new approach with the relatively novel one-shot FL methods. The novel one-shot FL methods are FedDM [39], DENSE [53], FedD3 [23], and FedDF [54]. In comparison with the latest one-shot FL methods, FedDM [39] is essentially a multi-round FL approach. However, due to its methodological approximations, it has been adapted for evaluation in a one-shot environment. The authors configured the system with 10 clients and compared the performance of this new method against these baseline methods on the MNIST [46] and CIFAR-10 datasets [48]. The comparisons were conducted under both IID settings and a scenario with a Dirichlet distribution with $\alpha$ = 0.5. Through this comparison, the authors found that FedGM can achieve outstanding results, as shown in

Table 2. Results compared with other one-shot FL methods. For non-IID settings, ipc = 50; for IID settings, ipc = 10. The authors set ipc = 10 for each method.

Dataset	Partitioning	FedGM (Ours)	FedDM	FedDF	FedD3	DENSE
MNIST	IID	99.21%	98.01%	96.13%	94.71%	99.13%
	$p_k\sim Dir\left(0.5\right)$	96.13%	99.01%	92.23%	94.1%	95.82%
CIFAR10	IID	62.41%	58.12%	54.12%	49.31%	64.1%
	$p_k\sim Dir\left(0.5\right)$	62.93%	61.83%	53.56%	40.15%	62.19%

.

4.3. Efficient and Robust Training on Heterogeneous Data

The evaluation of FedGM focuses on two key aspects: the communication volume required to achieve optimal accuracy and the performance under data heterogeneity; the authors use one-shot FedAvg as the baseline for comparison.

In FedGM, communication overhead is generated by transmitting condensed images (using 1 × 8 or 3 × 8 bits per grayscale or RGB pixel, respectively), whereas in traditional federated learning approaches, this overhead is due to the transmission of model parameters and requires 32 bits per parameter.

In Figure 6 and Figure 7, the authors find that with unlimited communication rounds, FedAvg outperforms the new method under the IID scenario, but the best performance of one-shot FedAvg (OS-FedAvg) can be reached by using FedGM with only around one-third of the communication volume on average. However, when the data heterogeneity across clients increases, the accuracy of the new method remains unchanged, while the prediction accuracy of FedAvg decreases notably, which demonstrates that the new method is more efficient under conditions of limited communication resources and significant data heterogeneity.

4.4. Performance with DP Guarantee

As demonstrated in Section 3.3, the labels of the client’s distilled dataset are protected with label differential privacy when being transmitted to server, and the label differential privacy in FedGM can satisfy $ϵ$-differential privacy. In this approach, each client is preconfigured to divide its distilled dataset into two non-overlapping datasets. One half is used directly in the RRwithprior process with a probability classifier that operates without learned prior knowledge, and this portion sends noise data to the server for learning prior knowledge. The other half utilizes the existing prior knowledge to obtain improved perturbed labels. These labels are then sent to the server for second rounds of training. The authors compare FedGM with traditional LDP-FedAvg and find FedGM can achieve higher accuracy compared with the baseline, as shown in Figure 8, Figure 9 and Figure 10. The traditional LDP-FedAvg undergoes multiple rounds of communication with the central party for training. In contrast, the one-shot method requires only two rounds of communication: the first round is to acquire prior knowledge, and the second round is for further optimization training with label differential privacy utilizing the acquired prior knowledge from the global model.

4.5. Analyzing the Efficiency Brought by Matching Representative Images

The authors show the training efficiency curve of matching the synthetic images with only the random or representative samples in Figure 11; different numbers of images per class are also considered in Figure 11. The authors have observed that by matching synthetic images with representative samples, the gradient loss between the original and synthetic datasets decreases more rapidly and significantly. This suggests that the synthetic dataset exhibits similar training performance to the original dataset. Additionally, it implies reduced training time and enhanced performance for client-side local private data. Moreover, the authors also note that a larger ipc correlates with fewer required training rounds and smaller gradient loss.

Example clustering and sub-cluster center results of matching representative images are also shown in Figure 12.

In conclusion, the authors find that with matching representative images, each client needs fewer epochs to train synthetic data, and the training accuracy with synthetic data is obviously improved.

4.6. Comparison with Transmitting Real Images

The authors’ approach is pitted against REAL, which transmits authentic images with dimensions identical to those of FedGM (ipc = 10) under the IID setting. To elaborate, REAL attains a test accuracy of on CIFAR10 using the default configuration. However, it falls short of surpassing FedGM, which achieves a test accuracy of 67.5% ± 0.2%. This suggests that the acquired synthetic dataset is capable of encapsulating a more extensive range of information from the entire dataset as opposed to merely using a handful of images.

5. Conclusions and Limitation

In the endeavor to advance the domain of federated learning, FedGM is introduced; it is a pioneering one-shot federated learning framework engineered to diminish the necessity for recurrent communication rounds, which frequently encumber traditional federated learning systems. This innovative approach not only optimizes communication efficiency but also significantly enhances the training effectiveness of the distilled dataset. By capitalizing on the unique strategy of leveraging synthetic datasets that mimic the original data’s distribution, FedGM achieves remarkable training accuracy and privacy protection while demonstrating an exceptional balance between performance and efficiency.

FedGM’s methodology of employing label differential privacy further exemplifies its novel contribution to ensuring data privacy while maintaining high utility in training models. This strategy, which is meticulously designed to protect the labels of the pseudo datasets, showcases a sophisticated understanding of the challenges inherent in federated learning environments. The empirical results, which are corroborated by extensive experiments, underscore the superior performance of FedGM over traditional methods, especially in the context of non-IID data scenarios and under stringent privacy constraints.

Moreover, the research illuminates the significance of visual privacy in federated learning, which is a novel area that has received limited attention in prior studies. The findings reveal that, while the distilled images themselves may not contain sensitive information, the labels attached to them possess a higher degree of sensitivity, necessitating the innovative approach to label privacy adopted by FedGM.

However, the study also acknowledges its limitations: particularly concerning the scalability of the synthetic dataset with an increasing number of clients. This recognition of potential constraints not only underscores the authenticity and originality of the work but also paves the way for future research to explore innovative solutions to these challenges.