Mutual Knowledge Distillation-Based Communication Optimization Method for Cross-Organizational Federated Learning

Abstract

With the increasing severity of data privacy and security issues, cross-organizational federated learning is facing challenges in communication efficiency and cost. Knowledge distillation, as an effective model compression technique, can reduce model size without significantly compromising accuracy, thereby lowering communication overhead. However, existing knowledge distillation methods either employ static distillation loss weights, ignoring bandwidth variations in communication networks, or fail to effectively account for bandwidth heterogeneity among different nodes, leading to communication bottlenecks. To enhance the overall system efficiency, there is an urgent need to find new methods that enable models to achieve optimal performance in resource-constrained environments. This paper proposes a communication optimization method based on mutual knowledge distillation (Fed-MKD) to address the bottleneck issues caused by high communication costs in cross-organizational federated learning. By leveraging a mutual distillation mechanism, Fed-MKD enables collaborative training of teacher and student models locally while reducing the frequency and size of global model transmissions to optimize communication. Our experimental results demonstrate that, compared to classical knowledge distillation methods, Fed-MKD significantly improves communication efficiency, with compression ratios ranging from 4.89× to 28.45×. Furthermore, Fed-MKD achieves up to 4.34× acceleration in convergence time across multiple datasets. These findings highlight the significant practical value of Fed-MKD in environments with heterogeneous data distributions and limited communication resources.

1. Introduction

Federated learning (FL) is an emerging distributed machine learning paradigm that enables multiple clients to collaboratively train a global model without sharing raw data. As model size increases, particularly with deep learning models like BERT [1] and CLIP [2], communication overhead becomes a major bottleneck. These large models demand significant computational resources and produce large gradient updates, resulting in high communication costs, which are especially challenging for resource-constrained devices [3,4,5].

Although communication optimization in federated learning has made some progress, existing methods often fail to effectively address the communication bottleneck posed by large models, especially in scenarios with heterogeneous client resources. Current mainstream solutions, such as reducing communication frequency [6,7] or model sparsification [8,9,10], do not fundamentally alleviate the communication burden.

Reducing communication frequency can lower per-round communication costs by decreasing the number of communications. However, this may hinder model convergence and degrade the final performance of the global model in some cases. On the other hand, model sparsification techniques reduce the amount of data transmission by discarding certain parameters or gradients, but they typically lead to a degradation in the model’s representational capacity.

To address these issues, we leverage knowledge distillation (KD) as a model compression technique for cross-organizational federated learning, which not only reduces the model size but also lowers communication costs while maintaining performance. This paper presents a federated learning communication optimization method based on a mutual knowledge distillation framework (Fed-MKD), specifically designed for cross-organizational federated learning scenarios. By transferring the knowledge of large teacher models to smaller student models, our approach effectively alleviates the communication burden in federated learning systems.

The main contributions of this work are summarized as follows:

(1)

We propose a new cross-organizational federated learning framework that allows multiple organizations to collaborate in training models without sharing sensitive data.

(2)

To optimize the federated learning process, we implement dynamic decision-making for local training rounds, eliminating the need for frequent model updates or transmissions. This approach reduces communication frequency, thereby lowering communication costs while preserving the effectiveness of model training.

(3)

Fed-MKD dynamically adjusts the distillation loss weights during local training, allowing each client to flexibly adjust the distillation strategy based on the current training progress and network conditions. This adaptive adjustment enhances the efficiency of the distillation process and ensures that model accuracy is maintained even when communication frequency is reduced.

2. Related Work

Research on using knowledge distillation to optimize communication can be broadly categorized into two types: joint distillation and distillation compression.

2.1. Joint Distillation

The core idea of joint distillation methods is to avoid directly transmitting model updates. Instead, local model predictions on a shared public dataset (i.e., “soft labels” or “output probability distributions”) are exchanged between clients. Since clients are required only to exchange prediction results rather than model weights or gradients, the volume of data transmitted is significantly reduced, thereby effectively lowering communication costs.

For example, Sui et al. [11] proposed FedED, a method that does not transmit local parameters but instead uploads the aggregated prediction results of local models to train a global model. This approach reduces communication costs but relies on sharing a public dataset, which may pose significant privacy risks in practical applications. Lin et al. [12] proposed an ensemble distillation model aggregation approach, wherein the global model is trained using unlabeled data outputs from client models. This partially mitigates privacy exposure risks and allows the aggregation of models with varying sizes and structures. However, it still depends on local data outputs from clients and does not effectively reduce the communication overhead introduced by large-scale models. Meanwhile, FedGKT [13] employs client–server collaborative distillation to address communication and computation bottlenecks in large-scale model training on edge devices. Nonetheless, it remains limited in terms of privacy protection, fairness assurance, and adaptability to dynamic environments.

2.2. Distillation Compression

The second category focuses on distillation compression to optimize federated learning communication costs. This approach involves having the student model replicate the outputs of the teacher model(e.g., prediction results, hidden layer representations). This enables the student model to approximate the performance of the teacher model while utilizing fewer parameters. During the model update process, clients only need to transmit the compressed student model parameters, which improves communication efficiency and reduces computational overhead.

For instance, FedDF [12] implements federated learning through soft label distillation, where clients only transmit the probability distributions of the model outputs. This approach improves privacy protection and communication efficiency, but the knowledge transfer efficiency may be limited in scenarios with large architectural differences (e.g., CNNs vs. Transformers). FedMD [14] allows clients and servers to exchange prediction distributions on a public dataset without repeatedly uploading and downloading large-scale model parameters. However, it faces the issue of frequent prediction distribution transmission. FedKD [15] combines adaptive mutual knowledge distillation and gradient compression techniques, effectively enhancing communication efficiency and reducing training communication costs. But this method relies on multiple rounds of local training on the client side, which may prevent it from fully adapting to changes in dynamic environments during the training process.

In summary, while significant progress has been made in optimizing communication overhead in federated learning through knowledge distillation, several limitations remain. Joint distillation methods rely on the sharing of public datasets, which introduces privacy leakage risks. Distillation compression techniques reduce communication overhead but do not effectively address adaptability challenges in dynamic environments.

This motivates our work to account for both communication efficiency and privacy protection by proposing a communication optimization method for federated learning based on mutual knowledge distillation (Fed-MKD). Our approach not only reduces communication frequency but also improves model compression by adaptively adjusting the distillation loss function weights, all while avoiding the direct exchange of sensitive data and enhancing privacy protection.

3. Preliminary

3.1. Problem Formulation

In this section, we provide a detailed definition of the problem to be addressed in the context of cross-organizational federated learning and present the corresponding system model. To facilitate understanding, key notations used throughout this study are summarized in

Table 1. Notions and descriptions.

Notation	Definition
N	Number of DCs participating in the training process
w	Model parameters, $w\in {\mathbb{R}}^d$, where d is the dimensionality of the parameters
$D_i$	The sample set (local dataset) independently owned and used for local model training by the i-th DC
D	Entire dataset, encompassing all data utilized by the DCs, $D_i\subseteq D$
$l(w,x)$	The loss function computed under model parameters w for an input data x
$F\left(w\right)$	Objective function for global model optimization
$L\left(w\right)$	Loss function of the global model at the r-th round
$w^i$	Local model parameters used by $D{C}_i$
$w_r$	Global model parameters aggregated at the r-th round
$w_t^i/w_s^i$	The teacher/student model parameters in $D{C}_i$
$g^i$	Gradient at DCi
$g_t^i/g_s^i$	The gradients of the teacher/student model in $D{C}_i$
${\eta }^i$	Learning rate of $D{C}_i$, controlling the step size of each gradient update
R	Upper bound of the training frequency for the global model
$thr$	Target threshold for the expected
${\tau }_r^i$	Communication time between $D{C}_i$ and the PS during the r-th round of training
$Pre{d}_t\left(Z^t\right)/Pre{d}_s\left(Z^s\right)$	Teacher/student model predictions
$L_t^d/L_s^d$	Teacher/student model distillation loss
$L_t^h/L_s^h$	Teacher/student model original loss
$L_t/L_s$	Teacher/student model final loss combining distillation loss with original loss
$Y^t/Y^s$	Teacher/student model hard decision (actual prediction result of the model)
$U^t/U^s$	Teacher/student model soft decision (probability distribution output)
$A,\beta ,\alpha$	Hyperparameters
E	Number of global training rounds between all participating DCs

.

In a cross-organizational federated learning system, all participating DCs use the same machine learning model for training [16,17]. These models share the same loss function $\ell (w,z)$, where $w\in {\mathbb{R}}^d$ represents the model parameters [18].

Assume a cross-organizational federated learning system involving N DCs participating in joint model training, where each DC relies on a parameter server (PS) for model parameter aggregation. Each DC maintains its own independent dataset $D_i$, and these datasets are mutually disjoint, i.e., $D={\bigcup }_i{D}_i$ (with $D_i\ne D_j$ when $i\ne j$). During the training process, all DCs optimize the same objective function according to Reference [19] which is expressed by:

$$\mathrm{minimize}:F\left(w\right)={\displaystyle \frac{1}{\left|D\right|}}\sum _{x\in D}l(w,x)$$

(1)

where $\left|D\right|$ represents the number of samples in the dataset D.

Each $D{C}_i$ updates the local model parameters $w_i$ using the stochastic gradient descent (SGD) algorithm [20] to obtain an optimal global model $w^{*}$. During each training iteration, DC computes the gradient $g_i$ of the local model using the following formula:

$$g^i={\displaystyle \frac{1}{|D_i|}}\sum _{x\in D_i}{\nabla }_{w_i}l(w_i,x)$$

(2)

where $|D_i|$ denotes the number of samples in $D{C}_i$, and ${\nabla }_{w_i}l(w_i,x)$ represents the gradient of the loss function with respect to the model parameters $w_i$, indicating the direction of model updates computed by $D{C}_i$ on its local dataset $D_i$.

The update process for the model parameters $w_i$ is as follows [21]:

$$w^i=w^i-{\eta }^i·g^i$$

(3)

where ${\eta }^i$ is the learning rate of $D{C}_i$, controlling the step size of each parameter update.

Through this process, $D{C}_i$ updates its model parameters based on the corresponding local data. Once all DCs complete their local updates, they upload their model parameters to the PS. The global model w is updated through the following weighted averaging approach [22]:

$$w=\sum _i{\displaystyle \frac{|D_i|}{\left|D\right|}}{w}^i$$

(4)

Here, ${\displaystyle \frac{|D_i|}{\left|D\right|}}$ represents the weight proportion of $D{C}_i$ in the global model. Through this aggregation approach, the local knowledge of each DC is effectively integrated, ultimately yielding a global model w.

In cross-organizational federated learning, due to factors such as non-IID data, communication constraints, and local updates, it is often challenging to find a globally optimal solution. Training is typically terminated early when the objective function value approaches a predefined threshold. Therefore, the core objective is to enhance the communication efficiency of cross-organizational federated learning by reducing the communication time required for each round of global aggregation, thereby achieving a more efficient training process.

Let ${\tau }_r^i$ denote the communication time of $D{C}_i$ during the r-th round of global aggregation, $F\left(w_R\right)$ represent the objective function value after R rounds of global aggregation, and $thr$ be the predefined threshold mentioned earlier. The problem addressed in our paper is formally defined as

$$\begin{array}{cc}\hfill \mathrm{minimize}:& \sum _{r=1}^R\underset{i\in [1,N]}{max}{\tau }_r^i\hfill \\ \hfill \mathrm{subject}\mathrm{to}:& F\left(w_R\right)\le thr\hfill \end{array}$$

(5)

The optimization objective in Equation (5) aims to minimize the maximum communication time among all participating DCs during each global aggregation round. This objective is fundamentally linked to the core performance metrics in federated learning: the minimization of maximum communication time is achieved through student model compression and a dynamic adjustment strategy, while the constraint $F\left(w_R\right)\le thr$ ensures that this optimization is not achieved at the expense of model accuracy.

Hence, the optimization goal of minimizing communication time complements the typical objective of maximizing model accuracy. By enabling more local training rounds before each global aggregation, it simultaneously optimizes both communication time and model accuracy. This balance enhances the overall efficiency of the federated learning system, ensuring accelerated convergence and reduced communication overhead in scenarios with heterogeneous data and resources.

It is noteworthy that Fed-MKD compresses the original large-scale model using knowledge distillation techniques. This approach requires each DC to simultaneously maintain and train two models of different sizes during the training process. In the Fed-MKD framework, the global model used is the smaller student model. The specific implementation of this method and its optimization strategies will be discussed in detail in the subsequent sections.

3.2. System Model Architecture

Some researchers have proposed transferring the knowledge of two models through knowledge distillation, where only the student model’s parameters are transmitted during global model aggregation, thereby significantly reducing the communication data volume. However, in cross-organizational federated learning environments, the datasets owned by different clients are often non-independent and identically distributed. As a result, the teacher model locally trained at each data center must simultaneously learn the features and knowledge from other client models.

To tackle this issue, we introduce a mutual distillation framework [15], in which both two models are trained together and learn from each other. The student model learns more precise discriminative methods from the teacher model, while the teacher model aggregates student models from different clients and learns the features and information from each client model.

This mutual distillation framework promotes knowledge exchange between two models through a bidirectional learning mechanism, thereby enhancing the efficiency and accuracy of federated learning systems. The cross-organizational federated model system based on the mutual distillation architecture is shown in Figure 1.

In cross-organizational federated learning, multi-modal data (MRI images and EEG time-series signals in medical scenarios) are often distributed across heterogeneous clients, leading to significant differences in the data modalities held by different organizations [23]. Additionally, traditional federated learning faces difficulties in establishing a unified feature representation space, and the semantic gap between modalities can result in inefficient knowledge transfer. The Fed-MKD framework, as shown in Figure 1, addresses these challenges through an innovative mutual distillation mechanism and is compatible with multi-modal data scenarios.

At the feature extraction level, Fed-MKD does not mandate a unified network architecture. Each client can autonomously select the optimal feature extractor based on local data characteristics (CNN for image processing, LSTM for time-series signal processing), with the only requirement being that the final prediction layer dimensions are consistent. This flexibility is achieved through a dual-model design: the teacher model retains a complete modality-specific feature extraction pipeline, while the student model aligns the logit distributions across modalities via the KL divergence in the logit space to enable cross-modal knowledge transfer. This architecture inherently possesses modality independence, allowing clients with different data modalities to maintain their native feature extraction processes, while reaching consensus through matching the probability distributions in the prediction space.

For each client participating in the training, it is assumed that the training is conducted within a DC. Each DC simultaneously trains both two models, using the same local data for training. The cross-organizational federated learning training process based on the mutual distillation architecture involves the following steps:

• Each DC trains two models using local data (the local training process includes mutual distillation between two models);
• DC pushes the updated student model parameters to PS;
• The PS aggregates the parameter updates from each DC;
• The PS generates the global student model using the aggregated results;
• DC pulls the updated global student model from the PS;
• DC updates the local student model using the global model.

This process optimizes the student model through shared model parameters. The mutual distillation framework enables the teacher model and student model to learn from each other, promoting knowledge sharing and integration across organizations.

4. Methodology

4.1. Mutual Knowledge Distillation Strategy

In the Fed-MKD framework, within each DC, both two models undergo mutual distillation. These two models have different loss functions when using the SGD strategy. Therefore, in the mutual distillation system, the loss calculation for two models must be performed separately, as shown in Figure 2.

The following provides a detailed explanation of each step in the implementation of Fed-MKD.

Let $Z^t=\{z_1^t,z_2^t,\dots ,z_n^t\}$ be the logits (i.e., the raw scores) output by the teacher model for a classification problem on a training sample x, and $Z^s=\{z_1^s,z_2^s,\dots ,z_n^s\}$ be the initial class probabilities output by the student model for the same sample, where n is the number of classes in the classification task. The softmax function with distillation temperature T (a hyperparameter that controls the smoothness of the soft labels generated by the teacher model) is defined as follows:

$$y_i={\displaystyle \frac{exp(z_i/T)}{{\sum }_{j=1}^{n}exp(z_j/T)}}$$

(6)

In particular, $P\left(y_i\right)$ is the predicted probability for class i, $z_i$ is the logit for class i. ${\sum }_{j=1}^{n}exp\left(z_j\right)$ is the sum of the exponentiated logits for all classes.

To more intuitively illustrate the physical significance of the distillation temperature, we provide a schematic of the output probability calculation process and present examples of the results obtained under different distillation temperatures T in Figure 3. The teacher network (upper part) and the student network (lower part) both receive the same training data. Next, the teacher network outputs intermediate features and prediction results, and calculates the soft targets using the temperature $T^t$, which are then used to align knowledge with the student network.

For $T<1$, the probability distribution sharpens, reflecting increased model confidence in the highest logits and greater concentration of probabilities on a single class. At $T=1$, Equation (6) corresponds to the standard softmax function. When $T>1$, tthe softmax-generated probability distribution becomes smoother, and as T approaches infinity, the output converges to a uniform distribution, with class probabilities becoming nearly equal.

Let $Z^t$ and $Z^s$ go through the traditional softmax layer with $T=1$ to obtain the standard decision rule (i.e., hard target). Let $Y^t=\{y_1^t,y_2^t,\dots ,y_n^t\}$ and $Y^s=\{y_1^s,y_2^s,\dots ,y_n^s\}$ represent the raw logits output by two models, respectively. Let the true label of sample x be $Y=\{y_1,y_2,\dots ,y_n\}$, where for any $1\le i\le n$, $y_i\in \{0,1\}$ and ${\sum }_{i=1}^n{y}_i=1$, i.e., only the correct class has a label of 1, and all other classes have a label of 0.

Based on the work presented in reference [24,25,26], for two probability distributions $P=\{p_1,p_2,\dots ,p_n\}$ and $Q=\{q_1,q_2,\dots ,q_n\}$, the cross-entropy between Q and P is defined as

$$H(P,Q)=-\sum _{i=1}^n{p}_{i}log\left(q_i\right)$$

(7)

$p_i$ is the probability of class i in the true distribution P (typically obtained through hard labels). $q_i$ is the probability of class i in the model’s predicted distribution Q. The difference between the two probability distributions P and Q can be measured using their Kullback–Leibler (KL) divergence [27], which is defined as

$$D_{\mathrm{KL}}(P\Vert Q)=\sum _{i=1}^n{p}_{i}log\left({\displaystyle \frac{p_i}{q_i}}\right)$$

(8)

According to Reference [28], for the teacher model and the student model, the original loss (i.e., hard loss) of the sample x is commonly defined as the cross-entropy between the predicted decision rule $l_t^h\left(x\right)=H(Y,Y^t)$ and the true label $l_s^h\left(x\right)=H(Y,Y^s)$. For simplicity, we omit model parameters and only use the indices t and s to represent the teacher and student models, respectively.

Let $Z^t$ and $Z^s$ go through the softmax layers with temperatures $T=T^t$ and $T=T^s$, respectively, to obtain the soft decision rules, i.e., $U^t=\{u_1^t,u_2^t,\dots ,u_n^t\}$ and $U^s=\{u_1^s,u_2^s,\dots ,u_n^s\}$.

Define the distillation loss as the KL divergence between the two models’ probability distributions. The distillation loss for teacher model with sample x is $l_t^d\left(x\right)=D_{\mathrm{KL}}(U^s\Vert U^t)$. The distillation loss for the student model is then defined as $l_s^d\left(x\right)=D_{\mathrm{KL}}(U^t,U^s).$

Based on the previous calculations, for a training dataset D, the original loss of the teacher model is given by

$$l_t^h={\displaystyle \frac{1}{\left|D\right|}}\sum _{x\in D}{l}_t^h\left(x\right)$$

(9)

where $\left|D\right|$ denotes the total number of samples in the dataset D, and $l_t^h\left(x\right)$ is the loss function for the teacher model on a given sample x.

Similarly, the original loss for the student model is given by

$$l_s^h={\displaystyle \frac{1}{\left|D\right|}}\sum _{x\in D}{l}_s^h\left(x\right)$$

(10)

The teacher model’s distillation loss is defined as the average KL divergence between the soft targets produced by the teacher and the student model:

$$l_t^d={\displaystyle \frac{1}{\left|D\right|}}\sum _{x\in D}{l}_t^d\left(x\right)$$

(11)

Similarly, the distillation loss for the student model, based on the output of the teacher model, is given by

$$l_s^d={\displaystyle \frac{1}{\left|D\right|}}\sum _{x\in D}{l}_s^d\left(x\right)$$

(12)

The final loss function of the teacher model is computed as a weighted sum of the hard loss and the distillation loss:

$$L_t=\alpha ·L_t^h+(1-\alpha )·L_t^d$$

(13)

where $\alpha$ is a predefined constant that controls the trade-off between the hard loss $L_t^h$ and the distillation loss $L_t^d$.

Similarly, the final loss function for the student model is

$$L_s=\beta ·L_s^h+(1-\beta )·L_s^d$$

(14)

4.2. Implementation Process of Fed-MKD

4.2.1. Local Training Rounds Adjustment

To further reduce the communication cost under resource-constrained environments in cross-organizational federated learning, we propose a strategy that decreases the communication frequency between clients and the PS. The decision mechanism of this strategy is based on the global model’s performance improvement to determine the number of local training rounds before each aggregation. Specifically, the parameter server monitors the performance change of the global model between the last two aggregations.

If the performance improvement between the two aggregations is large, clients will perform more local training rounds to improve the global model. In contrast, if the performance improvement is small, fewer local training rounds will be performed to avoid over-training.

We assume that $L_r$ represents the loss of the global model after the r-th aggregation on the test dataset. The performance improvement between the $r-1$-th and r-th aggregations can be calculated as

$$\Delta L_r=L_{r-1}-L_r$$

(15)

where $L_{r-1}$ corresponds to the performance of the global model after $r-1$-th aggregation.

Then, we use a constant A to set the upper limit for the number of local training rounds, thereby controlling the proportional relationship between the number of local training rounds and the performance improvement. By multiplying the relative performance improvement by the constant A, we can obtain an initial target for the number of local training rounds:

$$E_r^{*}=A·{\displaystyle \frac{L_{r-1}-L_r}{L_{r-1}}}$$

(16)

Considering that at least one round of local training is required in practical applications, we need to set a maximum function to ensure that the number of local training rounds is not less than 1. The final decision-making mechanism is typically represented by the following formula:

$$E_r=max\left(A·{\displaystyle \frac{L_{r-1}-L_r}{L_{r-1}}},1\right)$$

(17)

We can observe that $E_r\in [1,A]$ from this formula,. Based on this strategy, if the performance of the aggregated global model improves significantly compared to the last aggregation, the client will perform more rounds of local training. Otherwise, fewer local training rounds will be conducted to avoiding overtraining.

4.2.2. Teacher Model Training with Adaptive Loss Weights

To enhance the distillation performance of the teacher model, this work introduces an adaptive mechanism for adjusting the weights of the teacher model’s loss function. This mechanism regulates the information transfer during the distillation process by controlling the weight ratio between the hard label loss ($L_h$) and the distillation loss ($L_d$) in the teacher model’s loss function.

We use the constant $\delta$ to influence the initial weight of the teacher model’s original loss. The value of $\delta$ should be chosen based on the dataset and the desired balance between the teacher model’s fit to local data and its ability to perform distillation. Another hyper-parameter $ϵ$ determines the rate of change in the weight adjustment during training. It should be set based on the total number of local training rounds E before the next global aggregation. The weight of the teacher model’s original loss needs to be constrained within the range $[\delta ,\delta +ϵ]$, where $0\le \delta ,\delta +ϵ\le 1$.

Let $\alpha =\delta \left(1+{\displaystyle \frac{ϵlne}{\delta lnE}}\right)$, where E denotes the total number of local training rounds to be performed before the next aggregation, and e represents the number of local training rounds already completed in the current global cycle. Substituting $\alpha$ into Equation (13), the final loss function of the teacher model can be expressed as follows:

$$L_t=\delta \left(1+{\displaystyle \frac{ϵlne}{\delta lnE}}\right)·L_h+\left(1-\delta \left(1+{\displaystyle \frac{ϵlne}{\delta lnE}}\right)\right)·L_d,E>1$$

(18)

Notice that, when $E=1$, $\alpha$ reverts to its original constant. The teacher model loss function can then be described as

$$\begin{array}{c}\hfill L_t=\left\{\begin{array}{cc}\delta \left(1+{\displaystyle \frac{ϵlne}{\delta lnE}}\right)·L_t^h+\left(1-\delta \left(1+{\displaystyle \frac{ϵlne}{\delta lnE}}\right)\right)·L_t^d\hfill & E>1\hfill \\ \alpha ·L_t^h+\left(1-\alpha \right)·L_t^d\hfill & E=1\hfill \end{array}\right.\end{array}$$

(19)

It can be observed that, during the initial phase of each local training round, the distillation loss of the teacher model dominates the overall loss. This is attributed to the student model having just undergone aggregation, integrating information from other DCs. To enhance the teacher model’s ability to capture features from other DCs, the weight of the distillation loss is increased early in local training. As training progresses, this weight gradually decreases, allowing the teacher model to focus more on optimizing performance for the local dataset.

Since the dynamic weight adjustment mechanism in the Fed-MKD framework introduces a modality-aware adaptive strategy, it significantly enhances knowledge transfer efficiency in multi-modal federated learning scenarios. For high-dimensional image data, during the initial training phase, the hard label loss weight $\alpha$ is gradually adjusted towards the upper limit $\delta$ using a monotonically increasing function (as shown in Equation (18)). This design effectively stabilizes the gradient update process of the convolutional kernels, avoiding the common feature extractor oscillation problem in traditional fixed-weight schemes. For low signal-to-noise ratio (SNR) time-series data such as EEG signals, Fed-MKD automatically enhances the distillation loss weight $(1-\alpha \to ϵ)$. The adjustment magnitude is negatively correlated with the SNR: $ϵ={ϵ}_0+\gamma \left(1-{\displaystyle \frac{SNR}{SN{R}_0}}\right),$ where $\gamma$ is the modality-sensitive coefficient. This nonlinear mapping function ensures that, when the SNR is low, the model automatically allocates higher weights to the distillation loss, thereby significantly improving the model’s robustness in noisy environments.

4.3. The Specific Algorithm Design

This section elaborates on the algorithmic workflows designed for the participating client DCs and the PS in Algorithm 1 and Algorithm 2, respectively.

Algorithm 1 delineates the internal training process of the DCs within the Fed-MKD system. The input consists of the total number of global training rounds R and the local dataset $D_i$. During each global training round, DC i pulls the current global student model from the PS to update the local student model and retrieves the number of local training epochs E required for the current global round (Lines 3–4). Subsequently, DC i performs E rounds of local training (Lines 5–13). For each mini-batch, DC i inputs the data into both two models to obtain their respective predictions (Line 7). Based on these predictions, the original loss $L_t^h$, distillation loss $L_t^d$ for the teacher model, and the original loss $L_s^h$, distillation loss $L_s^d$ for the student model are computed as described in Section 4.1 (Line 8). The final losses $L_t$ and $L_s$ for the teacher and student models are calculated by weighted summation of the original and distillation losses (Lines 9–10). Notably, to enable the teacher model to better learn features from other participating DCs, the weights of $L_t^h$ and $L_t^d$ in $L_t$ are adaptively adjusted during the local training process. Following this, the gradients ${\mathbf{g}}_t^e$ and ${\mathbf{g}}_s^e$ for the two models are computed based on $L_t$ and $L_s$, respectively (Lines 11–12). These gradients are then used to update the local parameters $w_t^i$ and $w_s^i$ of the teacher and student models (Lines 14–15). Finally, DC i uploads the local student model parameters $w_s^i$ to the PS (Line 17).

Algorithm 2 describes the operational process of the PS in the Fed-MKD system. Initially, the PS randomly initializes the parameters of the global student model (Line 1), followed by iterative global training rounds (Lines 2–10). In each global training round, the PS collects all local student model parameters from the participating DCs (Lines 3–5). Subsequently, the PS computes the aggregated parameters $w_r$ and update the global parameters (Line 6). The performance of the updated global model is evaluated using a test dataset, and the loss function value for the current round is obtained (Line 7). Next, the number of local training epochs $E_r$ for the DCs before the next global aggregation is calculated based on the relative reduction in the loss function between the current and previous rounds (Line 8). Finally, the PS broadcasts the current global student model parameters $w_r$ and the number of local training epochs $E_r$ to all DCs (Line 9).

Algorithm 1 Fed-MKD (Participating DC $i:i=1,2,\dots ,N$)

Require:  $R,D_i$   1: $D\leftarrow$ Split $D_i$ into mini-batches of size k;   2: for  $r\in R$  do   3:    Pull the global student model from PS and update the local student model $w_s^j\leftarrow w_r$;   4:    Pull the number of local training epochs $E\leftarrow E_r$ for the r-th cycle from PS;   5:    for $e\in E$ do   6:      for mini-batch $d\in D$ do   7:         Obtain predictions from the teacher model and student model $Pre{d}_t$, $Pre{d}_s$;   8:         Compute losses $L_t^h$, $L_t^d$, $L_s^h$, $L_s^d$;   9:         Compute the teacher model loss $L_t$ with Equation (19); 10:         Compute the student model loss $L_s=\beta ·L_s^h+(1-\beta )·L_s^d$; 11:         Compute the teacher model gradient ${\mathbf{g}}_t^e\leftarrow {\mathbf{g}}_t^e+\nabla L_t$; 12:         Compute the student model gradient ${\mathbf{g}}_s^e\leftarrow {\mathbf{g}}_s^e+\nabla L_s$; 13:      end for 14:      Update the teacher model $w_t^j\leftarrow w_t^j-{\eta }_1·{\mathbf{g}}_t^e$; 15:      Update the student model $w_s^j\leftarrow w_s^j-{\eta }_s·{\mathbf{g}}_s^e$; 16:    end for 17:    Upload $w_s^j$ to PS. 18: end for

Algorithm 2 Fed-MKD (Parameter Server)

Require:  $R,N$   1: Initialize global model parameters $w_0\leftarrow w_{init}$;   2: for  $r\in R$  do   3:    for $i\in N_i$ do   4:      Receive $w_s^i$ from client DC i;   5:    end for   6:    Aggregate and update global model parameters $w_r={\sum }_i{w}_s^i/N_i$;   7:    Compute loss $L_r\leftarrow \mathrm{test}\left(w_r\right)$ using the test dataset;   8:    Compute the number of local training epochs $E_r=max\left(\left|{\displaystyle \frac{L_{r-1}-L_r}{L_{r-1}}}\right|,1\right)$;   9:    Broadcast $w_r,E_r$ to all DCs. 10: end for

In cross-organizational federated learning frameworks, computational complexity directly impacts the scalability and deployment feasibility of a given method. We quantify the computational complexity of the proposed Fed-MKD approach. The analysis results can be used to assess the scalability of different knowledge distillation methods when the number of clients increases or when the data scale expands.

Based on a typical knowledge distillation pipeline, we define four key computational components: forward propagation, backward propagation, loss computation, and parameter aggregation, among which forward and backward propagation usually constitute the primary computational bottleneck.

Let $d_1$ denote the number of parameters in the teacher model and $d_2$ that of the student model ($d_2\ll d_1$). Let B represent the local batch size, C the total number of classes in the classification task (i.e., the output dimension of the model), E the number of local training epochs before each aggregation, and e the total number of global communication rounds. Then, the total computational complexity of Fed-MKD can be expressed as

$${\mathcal{C}}_{\mathrm{Fed}\text{-}\mathrm{MKD}}=\mathcal{O}\left(e·\left(E·B·(2d_1^2+3d_2^2+C^2)+K·(d_1+d_2\right)\right)$$

(20)

The computational complexity of Fed-MKD per iteration can be decomposed as

(1)

Teacher model forward pass: $\mathcal{O}(B·d_1^2)$;

(2)

Student model forward pass: $\mathcal{O}(B·d_2^2)$;

(3)

Mutual distillation loss calculation (KL divergence): $\mathcal{O}(B·C^2)$;

(4)

Student model backward propagation: $\mathcal{O}(B·d_2^2)$;

(5)

Dynamic weight update: $\mathcal{O}(d_1+d_2)$.

Although maintaining both the teacher model (with parameter size $d_1$) and the student model (with parameter size $d_2$) may seem to increase the computational burden, the actual per-round complexity is $\mathcal{O}\left(B\left(2d_1^2+3d_2^2+C^2\right)\right)+\mathcal{O}\left(K\left(d_1+d_2\right)\right)$.

Compared to the baseline methods used in the experimental section, the differences in computational efficiency mainly stem from the architectural design. FedMD [14] only utilizes the teacher model, results in a per-round complexity of $\mathcal{O}\left(B\left(2d_1^2+C\right)\right)$, with the measured FLOPs being $81.5\pm 2.3\times {10}^9$, and its communication cost is 4.1 times that of Fed-MKD. FedGKT [13], although having the lowest computational cost ($\mathcal{O}\left(B\left(2d_2^2+C^2\right)\right)$), lacks the guidance of the teacher model and requires an additional 15% training epochs to achieve comparable accuracy (p < 0.01). Notably, while the theoretical complexity of Fed-MKD is similar to that of the fixed-weight variant Fed-MKDFW [15], the former achieves a 19% practical speedup due to its dynamic adjustment mechanism, which reduces unnecessary computations by adapting to non-IID data.

5. Simulation Experiment and Results Analysis

5.1. Experimental Setup

5.1.1. Simulation Platform Setup

The platform used for experimental evaluation is a server workstation running the Windows 11 operating system, equipped with an Intel Core i9 14900K processor and an NVIDIA^® GeForce^® GTX 4060 Ti graphics card, with 128 GB of memory. The operating system is the Windows 11 Workstation version, and the simulation code is implemented using the Pytorch 2.3.0 framework.

The simulated cross-organizational federated learning system in the experiment consists of 16 client DCs and 1 PS. To control variables in the experiment, this chapter ignores the bandwidth differences between wide-area network links and sets the bidirectional bandwidth between DCs and PS to 100 Mbps.

5.1.2. Dataset and Model Parameters

This study conducted experiments using three commonly used datasets: Fashion-MNIST [29], CIFAR-100 [30], and Mini-ImageNet [31]. The Mini-ImageNet dataset was preprocessed following the method described in [32] to adapt it for classification tasks, dividing the data into training, validation, and test sets.

For Fashion-MNIST, both the teacher model and the student model are custom CNNs. The teacher model consists of two convolutional layers, two pooling layers, and one fully connected layer, with the number of convolutional kernels ranging from 16 to 256. The student model has the same hierarchical structure as the teacher model, but the number of convolutional kernels is reduced to a range of 8–64. For the CIFAR-100 dataset, we conducted two sets of experiments. In the first set, the teacher model was implemented using ResNet50, while the student model was a customized residual network, ResNet-Mini, designed with fewer residual blocks. In the second set, ResNet101 and ResNet18 were employed as the teacher model and the student model. For Mini-ImageNet, ResNet101 and ResNet18 were used as the teacher model and the student model, respectively.

In terms of hyperparameters, we utilized the SGD optimizer with momentum for gradient descent, with a mini-batch size set to 128. The weights and biases of the models were initialized using a Gaussian distribution [33] with a mean of 0 and a variance of 0.1. For Fashion-MNIST, the number of training epochs was set to 100, while for CIFAR-100 and Mini-ImageNet, it was set to 500. The learning rate was set to $1\times {10}^{-3}$. To enhance training accuracy, a decaying learning rate was employed for training the CIFAR-100 and Mini-ImageNet dataset.

5.1.3. Comparative Schemes and Performance Metrics

The comparative methods used in the experiments are described as follows:

• FedMD [14]: Locally trains only a larger teacher model to accelerate the learning process during training. This method serves as a baseline for comparing the effectiveness of training acceleration.
• FedGKT [13]: Locally trains only the student model, aiming to achieve performance comparable to the teacher model with fewer parameters.
• FedKD [15]: Employs the original mutual distillation framework for training.
• Fed-MKDFW: Utilizes fixed loss weights during training with Fed-MKD, designed to evaluate the advantages of Fed-MKD’s adaptive adjustment of distillation weights under non-IID data distribution scenarios.

Moreover, we use four performance metrics with practical research significance to evaluate the communication optimization performance of Fed-MKD in cross-organizational federated learning: (1) model test accuracy; (2) convergence time; (3) total compression ratio; and (4) model training performance under non-independent and identically distributed (non-IID) settings.

Model accuracy refers to the prediction accuracy or correctness of the model on a given task, typically calculated on a test set. It is used to measure the model’s ability to correctly classify unseen data. In the knowledge distillation system used in this paper, only the student model participates in aggregation, and the global model is derived from it. Generally, the faster the global model accuracy improves, the faster the convergence speed.

For the training time required for the model to achieve convergence, if the change in the target value is less than 0.5% over ten consecutive epochs, the model is declared to have converged. Convergence detection begins only after the target value reaches a relatively optimal value, to avoid premature convergence judgments in cases where the loss curve exhibits step-like behavior.

The third metric is the ratio of the total data size transmitted by the baseline method (FedMD) at convergence to the total data size transmitted by other methods. The total compression ratio of the baseline is set to 1 in this experiment. Let $S_b=N{E}_b{M}_b$ denote the total data size transmitted by the baseline model, where N is the number of DCs, $E_b$ is the number of epochs required for convergence, and $M_b$ is the number of parameters in the baseline model. Similarly, let $S_j=N{E}_j{M}_j$ represent the total data size transmitted by method j. The total compression ratio for method j is defined as ${\displaystyle \frac{S_b}{S_j}}={\displaystyle \frac{E_b{M}_b}{E_j{M}_j}}.$

The fourth metric measures the final accuracy of the global model under the scenario where training data across different DCs are not independent and identically distributed (non-IID). The non-IID setup follows the methodology described in [34]. This non-IID setting method first sorts the training data based on class labels, and then randomly assigns an exclusive subset of classes for each data center, with the size of the subset denoted as $\xi$. Here, $\xi$ represents the number of classes allocated to each data center. A smaller value of $\xi$ implies that the data diversity of an individual client is lower and that the differences in data distribution among different clients are greater.

The varying class assignments demonstrate the inherent data diversity between different DCs, which simulates real-world scenarios where clients may have different data distributions. This non-IID setup helps in testing the robustness of the federated learning method in conditions wherein the data are not uniformly distributed across all clients. For Fashion-MNIST, $\xi$ is set to 8 and 4, while for CIFAR-100 and Mini-ImageNet, $\xi$ is set to 80 and 40.

5.2. Experimental Results and Analysis

5.2.1. Relationship Between Model Accuracy and Training Time

Figure 4 depicts the curves of model accuracy over time in four sets of experiments. Note that, in order to make the relationship and trend of the curve clearer, Figure 4 only captures part of the training process. The accuracy of the curve in the graph is not necessarily the accuracy of the model in the training. Final specific accuracy data will be presented in

Table 2. Experimental numerical results.

Dataset	Training System	Network Architecture	Communication Model Parameters	Top-1 Accuracy	Inference Speedup	Compression Ratio
Fashion-MNIST	FedMD (Teacher)	Custom CNN	0.32 M	91.42%	1.00	1.00
	FedGKT (Student)	Custom CNN	0.038 M	89.81% (−1.61%)	0.98×	7.21×
	FedKD	-	0.038 M	90.41% (−1.28%)	1.26×	9.43×
	Fed-MKD	-	0.038 M	90.95% (−0.47%)	2.79×	28.45×
CIFAR-100	FedMD (Teacher)	ResNet50	24.47 M	74.12%	1.00	1.00
	FedGKT (Student)	ResNet-Mini	5.37 M	62.55% (−11.57%)	1.90×	2.72×
	FedKD	-	5.37 M	64.59% (−9.53%)	2.20×	2.45×
	Fed-MKD	-	5.37 M	69.48% (−4.64%)	3.72×	4.89×
CIFAR-100	FedMD (Teacher)	ResNet101	43.06 M	76.15%	1.00	1.00
	FedGKT (Student)	ResNet18	10.58 M	71.19% (−4.96%)	0.98×	1.84×
	FedKD	-	10.58 M	72.68% (−3.47%)	1.13×	2.30×
	Fed-MKD	-	10.58 M	75.13% (−1.02%)	3.44×	9.16×
Mini-ImageNet	FedMD (Teacher)	ResNet101	43.06 M	69.34%	1.00	1.00
	FedGKT (Student)	ResNet18	10.58 M	64.89% (−4.45%)	2.73×	4.25×
	FedKD	-	10.58 M	66.73% (2.61%)	2.08×	3.47×
	Fed-MKD	-	10.58 M	67.96% (1.38%)	4.34×	8.68×

.

Overall, Fed-MKD demonstrated significant acceleration effects across all training scenarios. The transmission of the smaller student model undoubtedly contributed to communication speedup, as shown in Figure 4a. In experiments with relatively large model sizes (Figure 4b–d), the curves for Fed-MKD exhibited a faster initial rise compared to the FedMD method, which transmits the teacher model. However, due to the inherent limitations in the performance of the student model, its accuracy was eventually surpassed by FedMD. The larger the teacher model, the more pronounced the acceleration effect. Furthermore, reducing communication frequency further enhanced the speedup, as reflected in the figure, where the Fed-MKD curve consistently showed the largest gap compared to the baseline method using teacher model transmission (black curve).

5.2.2. Model Top-1 Accuracy and Convergence Time Speedup Ratio

Table 2 summarizes the numerical results from four different experimental scenarios. The datasets used in these experiments include Fashion-MNIST (with teacher and student models being custom CNNs of different sizes), CIFAR-100 (with a ResNet50 teacher model and a ResNet-Mini student model), CIFAR-100 (with a ResNet101 teacher model and a ResNet18 student model), and Mini-ImageNet (with a ResNet101 teacher model and a ResNet18 student model). For each experimental setup, the different training methods used the same learning rate, mini-batch size, and other initial configurations. The table primarily compares the differences in communication model parameters, model accuracy, convergence speedup, and overall compression ratio between the various training methods.

From the Top-1 accuracy shown in Table 2, it can be observed that, during Fashion-MNIST training, the optimization of the student model’s performance using Fed-MKD is not significant, with only a 1.14% improvement. This is because the dataset is relatively simple, and the accuracy gap between the student and teacher models in independent training is small, leaving little room for improvement. In contrast, on the other two datasets, Fed-MKD significantly improved the accuracy of ResNet-Mini and ResNet18 by 6.93% and 3.94%, respectively, on CIFAR-100; and by 3.07% on Mini-ImageNet.

Comparing to the baseline FedMD (Teacher), although the reduction in model size using Fed-MKD leads to some loss in accuracy, it significantly reduces communication costs and accelerates the training process. Furthermore, Fed-MKD consistently outperforms the standard mutual knowledge distillation method, FedKD. This is because Fed-MKD reduces communication frequency, enabling more local training rounds within the same time frame compared to FedKD.

Figure 5 visually demonstrates the comparison of convergence speeds between Fed-MKD and other training methods, with specific numerical results also shown in Table 2. The baseline for convergence acceleration is FedMD (Teacher), so its bar height in Figure 5 is always set to 1. In all experiments, Fed-MKD achieves the best convergence speedup, with the best performance observed during Mini-ImageNet training, where the convergence speed is accelerated by a factor of 4.34 compared to the baseline.

5.2.3. Total Compression Ratio

The overall data compression ratio for the total amount of data transmitted before convergence for each training method across the four experimental setups is summarized in Table 2. Clearly, transmitting smaller models consistently achieves a compression ratio greater than 1 in the experiments. This indicates that, although transmitting smaller models may result in more training rounds, it generally reduces the amount of data transmitted, thereby alleviating the communication burden in cross-organizational federated learning.

Since Fed-MKD reduces both the size of the transmitted model and the communication frequency, it achieves the smallest data transmission volume and consistently provides the highest compression ratio among all training methods in each experiment. The compression ratio for Fed-MKD ranges from 4.89× to 28.45×. Notably, the compression ratio for FedKD does not necessarily exceed that of the Student model. This is because both methods transmit models of the same size, and the compression ratio is determined by the number of training rounds required for model convergence. In some cases, the student model can converge quickly on its own, while in FedKD, the global model’s accuracy continues to improve under the effect of knowledge distillation, which is consistent with the analysis of convergence speedup discussed earlier.

5.2.4. Global Model Accuracy Under the Non-IID Setting

We evaluated the performance of Fed-MKD under non-IID data settings, as shown in Figure 6. Overall, knowledge distillation still improves the accuracy of the student model under non-IID data, and Fed-MKD consistently minimizes the accuracy loss compared to the Teacher model across all scenarios. The performance of the complete Fed-MKD slightly outperforms LFCKD-FW, which uses a fixed distillation loss function weight, with the difference being more apparent when $\xi$ = 4, as shown in Figure 6a. This is because, under non-IID conditions, the features from models in other DCs are more significant, and the weight adjustment mechanism in Fed-MKD allows the teacher model to better learn the features from other models after each aggregation.

Figure 6b,c present the results from the same dataset, with Figure 6d showing a larger gap between FedMD (Teacher) and the other training methods. This is because the Student model, ResNet-Mini, used in this experiment is a custom neural network with relatively poor performance when facing complex datasets. As a result, there is less room for performance improvement through knowledge distillation.

Fed-MKD decouples local feature learning from global knowledge distillation, achieving a controlled balance of data dependency. The teacher model maintains a strong fitting capability on local data through the hard label loss (Equation (8)), thereby preserving the necessary sensitivity to local features, which is proportional to $\propto {\displaystyle \frac{1}{\left|D_i\right|}}.$ In contrast, the student model absorbs knowledge from other clients via the KL divergence (Equation (9)) and maintains cross-scenario consistency through the dynamic distillation weight (Equation (19)). Therefore, through its architecture-independent design and dynamic generalization mechanism, Fed-MKD effectively supports efficient migration to heterogeneous data scenarios.

However, the adjustment range for distillation loss weight is a tunable parameter in practice. It is not easy to find an optimal setting for each scenario during the experiment. How to select or generate a suitable student model and efficiently determine the appropriate parameters for adjusting distillation loss weights remains a topic for future work.

6. Conclusions

This study investigates the communication bottleneck issues in cross-organizational federated learning systems and WAN environments and proposes a low-communication-frequency knowledge distillation method (Fed-MKD) to reduce communication costs. For the proposed Fed-MKD method, we provide a detailed description of its implementation process and elaborate on the specific execution steps of DCs and the PS within this framework. Through mathematical analysis, we also explore the impact of the reduced communication frequency strategy on model convergence.

Extensive experimental results on public datasets validate the effectiveness of the proposed Fed-MKD in optimizing communication. Notably, under non-IID data settings, the integration of an adaptive distillation loss weight adjustment strategy significantly improves both communication efficiency and model performance. These findings underscore the practical advantages of Fed-MKD in cross-organizational federated learning, highlighting its robustness and adaptability in real-world applications. The ability of Fed-MKD to optimize communication while maintaining high model performance has important implications for deploying federated learning in resource-constrained environments, where communication costs are a major concern.

Additionally, the model design under the Fed-MKD framework demonstrates strong generalizability. Thanks to the mutual distillation mechanism, the model is not only capable of adapting to the specific data characteristics of the local environment but is also applicable to different data scenarios. This means that, once trained in a particular data environment, the model can quickly adapt to other diverse data environments through fine-tuning or minimal additional training. This cross-scenario adaptability enables Fed-MKD to perform exceptionally well in scenarios involving multi-modal data and non-IID data.