Evaluating the Impact of Mobility on Differentially Private Federated Learning ^†

Abstract

This paper investigates differential privacy in federated learning. This topic has been actively examined in conventional network environments, but few studies have investigated it in the Internet of Vehicles, especially considering various mobility patterns. In particular, this work aims to measure and enumerate the trade-off between accuracy of performance and the level of data protection and evaluate how mobility patterns affect it. To this end, this paper proposes a method considering three factors: learning models, vehicle mobility, and a privacy algorithm. By taking into account mobility patterns, local differential privacy is enhanced with an adaptive clipping method and applied to a mobility-based federated learning model. Experiments run the model on vehicular networks with two different mobility scenarios representing a non-accident traffic situation and traffic events, respectively. Results show that our privacy-enhanced federated learning models degrade accuracy performance by 2.96–3.26% on average, which is compared to the performance drop (42.97% on average) in conventional federated learning models.

1. Introduction

The future smart transportation system is expected to see collaboration among intelligent autonomous driving systems, named the Internet of Vehicles, which will ultimately improve road safety [1]. While recent research has applied machine learning techniques to realize this vision, security issues still remain in the process of collecting data from each vehicle and transferring them to a central server. Federated learning has been proposed to address these issues by allowing each vehicle to perform model training locally and only sending updated model parameters to the server [2]. As an emerging technology, however, it still has various technical challenges that have not yet been solved [3].

For instance, inference attacks in joint learning on autonomous vehicles, where an attacker exploits a training model to infer information that should be unknown, may cause privacy problems [4,5,6]. Recent differential privacy may alleviate this problem; it is possible to protect personal information while maintaining data distribution characteristics by adding noise to the data [7]. However, when differential privacy is applied to model training, there is a trade-off between privacy preservation and model accuracy, showing lower accuracy than non-privacy models. The primary goal of this research is to measure and enumerate this trade-off in the Internet of Vehicles. While recent studies are actively exploring the performance issue of differentially private federated learning models in conventional network environments, there are few studies that address this topic in the Internet of Vehicles, especially taking into account mobility patterns.

To clarify the accuracy of performance and protection level in the Internet of Vehicles, this paper proposes a method considering three factors: learning models, vehicle mobility, and a privacy algorithm. First, the method considers two learning models: a conventional learning model that is widely used in federated learning settings and a mobility-based federated learning model that uses vehicular knowledge generated according to vehicles’ movement patterns. Next, it imports two mobility models: reference point group mobility and random waypoint. These models represent normal situations and road events like traffic accidents, respectively. Regarding the third factor, this paper proposes to adopt a local differential privacy with adaptive clipping method after considering the nature of the Internet of Vehicles. After applying our privacy algorithm to the learning models, experiments are run with various noise values, and the accuracy performance of the models is measured under different mobility scenarios. In non-accident traffic situations, the experimental results show that privacy-enhanced federated learning models degrade performance by 3.26% on average, which is acceptable; elevated levels of data protection could compensate for the loss. In the case of traffic events, two learning models show very different outputs. With the mobility-based model, the privacy enhancement does not harm performance much, with a 2.96% drop. With the conventional model, however, the degradation rate reaches 42.97% (from 73.17% to 41.73%).

The main contributions of this paper are summarized as follows:

• This work designs a method considering three factors—learning models, vehicle mobility, and a privacy algorithm—in order to evaluate the impact of mobility on differentially private federated learning.
• This work modifies a differential privacy algorithm that takes the features of IoV into account. As a result, it works seamlessly on a mobility-based federated learning model.
• The experimental results discover that a conventional learning model that is widely used in federated learning does not work well in a mobile, vehicular environment, especially when enhanced with a privacy method, and that an adaptive clipping method is more useful than prior methods because it guarantees model performance and the target degree of privacy preservation.

The rest of the paper is organized as follows: Section 2 reviews two underlying technologies: federated learning and differential privacy. Section 3 briefly introduces the intrinsic properties of the Internet of Vehicles and shows how the underlying technologies have been applied to them. Section 4 proposes a new method for evaluating the impact of mobility, including a new differential privacy algorithm running on mobility-based federated learning for the Internet of Vehicles. Section 5 sets up our experiments, which is followed by analyses of the experimental results in Section 6. Section 7 analyzes the communication cost and error convergence of the proposed learning model. Finally, Section 8 concludes this work.

2. Technology Reviews

2.1. Federated Learning

In a conventional deep learning setup, a central server collects data from geographically distributed devices and consumes all the collected data for training, testing, and updating models. This can cause communication efficiency problems and various security problems [8]. An emerging federated learning (FL) method is expected to resolve parts of the problems by not requiring raw data to be transmitted to the central server. Instead, it allows distributed clients (participants) to train a model at their side without sending data to the server. As the model is trained locally, participants only send the updated parameters to the server. Figure 1 shows the action processes of federated learning [9].

• A server coordinator randomly selects clients to participate in federated learning. The selected participants download a current global model from the coordinator. Figure 1 assumes all four clients are selected as participants.
• Each participant collects data locally and uses them to train the received models.
• After training, each participant transmits the parameters of the locally updated global model to the coordinator. Then, the coordinator integrates the values received from all of the participants and learns (updates) the global model again.
• The process is repeated from step 1 unless a predefined condition on the model is satisfied. Otherwise, the global model is finalized and distributed to all the participants.

The most notable feature of federated learning is the distribution of data and workload among training participants. Thus, local data are not easily exposed to large amounts of data breaches or sharing with third parties. This allows various parties to share learning models while protecting sensitive information. Distributing the workload of learning algorithms across participants can substantially improve performance. Due to these benefits, federated learning has been applied to a variety of scenarios.

2.1.1. Federated Learning on Mobility Applications

This work is interested in federated learning models in mobility applications. Federated learning fits latency-sensitive mobility applications well in the sense that it improves network efficiency by communicating only model parameters instead of large amounts of raw data and that a local model can quickly adapt to changes in the environment of a device. Most of the previous studies have tried to solve research problems related to the mobility of humans and moving objects such as vehicles. Human activity recognition aims to detect the movements of a human, such as sitting, running, jumping, lying down, or going upstairs, using cameras and sensors [10,11,12,13,14,15]. Location prediction focuses on finding the next locations of users [16,17,18]. Point-of-interest recommendation suggests a location to users who might be interested [19]. The authors in [20,21] use federated learning for the indoor localization of humans, i.e., finding their positions accurately. Similar to this, Brik et al. [22] propose a method that tracks an object in an indoor environment. Autonomous control for autonomous vehicles aims to track vehicles so as to reduce traffic accidents [23]. In references [24,25], the authors try to solve collision avoidance problems by sensing the movement of many vehicles. Traffic sign recognition is another interesting topic that classifies traffic signs by using cameras and on-vehicle sensors [26,27]. Energy demand prediction aims to predict when an electrical vehicle needs to be charged by using travel information about the vehicle, such as velocity, distance, and the number of drives taken [28,29].

2.1.2. Inference Attack on Federated Learning

Although federated learning provides a privacy solution by keeping local data on users’ sides, learning parameters can still be subject to various attacks. A malicious entity can see the parameters sent by users, and an honest but curious aggregator can identify a user’s contribution when computing the change in the global model. Among the various threat models in federated learning, this work is interested in inference attacks. An attacker infers users’ sensitive information as the parameters can leak unintended features of training data [4,5,30,31]. The simplest way is to infer the model based on continuous queries and responses. Figure 2 captures a sample scenario of an inference attack on federated learning. Recent research has reported various forms of inference attacks [32,33]. In order to defend from the attack, many previous works have adopted well-known privacy-preserving techniques: homomorphic encryption, secure multiparty computation, and differential privacy. Among them, this work explores the last one.

2.2. Differential Privacy

Differential privacy (DP) ensures a similar distribution of responses to queries, regardless of whether an individual’s record exists in a database [7]. Because an attacker cannot distinguish responses, DP makes it impossible to infer more information about a particular individual. In this way, it protects personal data while ensuring the usefulness of data utilization. DP is defined below [34].

Definition 1.

(ε, δ): differential privacy

If databases D and $D^{\prime }$ with a record difference for any algorithm M are established as follows, then (ε, δ)—differential privacy—is satisfied.

$$\epsilon >0,\delta \ge 0,Pr[M\left(D\right)=t]\le e^{\epsilon }Pr[M\left(D^{\prime }\right)=t]+\delta .$$

(1)

If δ is 0, then δ-difference privacy is satisfied. The smaller the δ value, the smaller the distribution difference between the two databases, thus ensuring greater privacy. In general, the value of δ is less than $1/\left|D\right|$.

In the literature, differential privacy has been applied to federated learning scenarios. The authors in [35,36] investigate a federated learning setup where a coordinator is assumed to be trusted and is responsible for adding noise to aggregate local parameters in order to ensure the privacy of all of the data from users. In another setting, the assumption of a trusted coordinator is eliminated, and data owners are responsible for perturbing private information locally before uploading it to the untrusted coordinator [37,38,39]. With an untrusted coordinator, the research works in [40,41] examine an additive noise mechanism where local parameters are perturbed with less noise but then encrypted to ensure privacy before forwarding to the coordinator.

3. Internet of Vehicles

This section begins with a brief introduction to the terminologies related to the Internet of Vehicles. Connected vehicles refer to vehicles that can communicate with their external environments via wireless communication technologies [42]. With the connectivity property, they are also represented as vehicle-to-everything (V2X); here, “everything” often includes neighboring vehicles, road infrastructure, pedestrians, satellites, and so on. Vehicular networking, as evolved from vehicular ad hoc networking, represents networking perspectives among connected objects on top of the connectivity. Connected vehicles and vehicular networking can be considered as building blocks of the Internet of Vehicles (IoV), which represents a mobile and distributed computing environment. It is characterized by collecting, sharing, and processing information in a collaborative manner between connected vehicles [1]. These technologies together are expected to enable the evolution to the next-generation road infrastructure: cooperative intelligent transportation systems (C-ITSs). While the terms are somewhat different, this work uses them interchangeably as it is more concerned with how to apply federated learning and differential privacy to them.

Two of the most distinguishing features of IoV are the mobility and time–space validity of data. First, vehicles move at a very high speed along roads. In addition, the speed can change in a wide range, for example, when a vehicle traveling at 60 km/h stops in front of a crosswalk. The high speed and varying mobility make the connectivity unstable, which eventually adversely affects the operations of federated learning. Next, data generated in IoV tend to have locality in terms of space and time. Vehicle-generated content has its own spatial scope of utility and temporal scope of validity. For instance, road safety messages are only interesting to vehicles in the vicinity and may only be valid for 30 min, even in the same location. The time–space validity of the data implies the scalability of data processing applications like federated learning, since old data are obsolete.

3.1. Federated Learning on Internet of Vehicles

While the previous section lists studies on federated learning in vehicle scenarios, most of them simply utilize the scenarios as applications. Few address systematic perspectives of IoV. There remain many issues that need to be addressed before applying federated learning to real-world vehicles [9,43]. We note that many of these issues are attributed to the mobility of vehicles, and the following are representative examples: (1) Since vehicles tend to only move on roads, training data between participating vehicles may be biased if model training is not considered. This can cause overfitting problems. (2) Due to the high mobility of vehicles, the characteristics of training data are extremely dynamic, and the amount of data also changes quickly. (3) When a coordinator selects participating vehicles randomly, it is possible that vehicles with very important information (e.g., those witnessing an accident) may not be selected as training nodes. (4) Because each vehicle has different motion characteristics and data distributions, the input data during training have non-independent and identically distributed (non-IID) characteristics. In order to deal with the problems, Deveaux et al. [43,44] investigate an interesting area: vehicular knowledge networking. They first examine the meaning and scope of knowledge on vehicular networks and propose a networking framework needed for describing, generating, storing, and sharing knowledge among vehicles. Using the framework, they show federated learning orchestration for a vehicular environment; a coordinator selects vehicles carefully by making use of vehicular knowledge to improve learning performance.

3.2. Differential Privacy on Internet of Vehicles

In the literature on IoV, differential privacy has often been applied for the protection of trajectory data and location privacy. Zhang et al. [45] develop a geographically indistinguishable method that leverages differential privacy to obscure a vehicle’s location within a specific range. Qian et al. [46] propose a location privacy-preserving method in vehicle-based crowdsensing networks. Each vehicle obfuscates its own location data using differential privacy and then submits them to an application platform. Ma et al. [47] adopt differential privacy when releasing a vehicle’s trajectory data. It uses a dynamic sampling method to process the data and a privacy budget allocation method for location protection. Cai et al. [48] propose a method of releasing a trajectory using a differential privacy dataset in IoV. The method constructs a noisy prefix tree based on the spatiotemporal characteristics of the trajectory data and utilizes a Markov chain to minimize the cost of adding noise. Li et al. [49] present a quadtree-based spatial decomposition algorithm to protect the location privacy of electric vehicles in a vehicle-to-grid network. The algorithm leverages a quadtree and Bernoulli random sampling to enhance relative error with respect to the privacy budget. Sun et al. [50] propose a method of synthesizing private and realistic trajectory data by using a geography-aware grid that integrates the public geography structures of a target area.

In another side of the literature, differential privacy has been used in IoV for general purposes. Zhao et al. [38] investigate the integration of federated learning with differential privacy to enable crowdsourcing applications for both privacy and network efficiency. Batool et al. [51] propose a two-layer framework for private data sharing in vehicular networks. Each vehicle adds noise to local raw data and then forwards them to nearby a roadside unit that is responsible for training a local model with the perturbed data. The unit then uploads the trained parameters to a coordinating server. Olowononi et al. [52] apply both federated learning and differential privacy to connected vehicles to make vehicular cyber–physical systems resilient to adversarial attacks. To enhance resiliency, they use a layer-wise relevance propagation method that can regulate the selection of perturbation values. Khaliq et al. [53] investigate a privacy-preserved parking recommender system. They propose a mutual authentication mechanism using elliptic curve cryptography and make use of differential privacy to perturb data when sharing users’ historical parking data with third-party recommender systems. Sani et al. [54] summarize privacy-preserving techniques including differential privacy applied to electric vehicles.

There is a very short list of research that applied both federated learning and differential privacy to the Internet of Vehicles. To the best of the authors’ knowledge, few prior studies have applied them in an environment considering mobility patterns and evaluated the impact of mobility on their performance, which are the final goals of this work.

4. Proposed Method

4.1. Mobility-Based Federated Learning

With respect to federated learning, this work considers two models: a conventional model and a mobility-based model. In particular, the vehicular knowledge networking introduced in the previous section efficiently identifies and resolves federating issues in the processes of client selection and communication, such as bias in training data, fairness, and non-IID arising from two features of IoV (vehicle mobility and the time–space validity of data) [3]. In this sense, this work takes it as our mobility-based federated learning model. Figure 3 illustrates the model’s architecture and operations. Each vehicle periodically collects its own semantic information (i.e., knowledge) that includes its location, training environment, and kinematic information (step 1 in the figure). In the process of participant selection, vehicles share their own knowledge with a coordinator, who then identifies which vehicles have the necessary data for model training and invites appropriate ones to join.

4.2. Mobility-Enhanced Differential Privacy

This section presents the design of a novel differential privacy algorithm that takes the features of IoV into account, and as a result, it works seamlessly on the mobility-based federated learning model. Technically, the algorithm uses local differential privacy to handle mobility and an adaptive clipping method to deal with the locality of IoV data that are related to fairness and non-IID. Local differential privacy is a method of applying differential privacy when one does not trust a server. Differential privacy is executed at the user level, and obfuscated data are then submitted to the server. In terms of federated learning, noise is added to the updated parameter values of each vehicle, and the values are transmitted to a coordinator. In this research, user-level (local) differential privacy is used to ensure stronger privacy. To this end, a proposed algorithm updates a combined averaging algorithm running on the coordinator. A general method of ensuring local differential privacy in federated learning is to limit the contribution of each training participant’s model update to a certain value and then add noise. In this case, adaptive clipping uses a quartile value of the updated norm distribution instead of a fixed norm value.

By applying these and referencing previous works [35,55,56,57,58,59,60,61], our algorithm for differentially private federated learning on the Internet of Vehicles is shown in Algorithm 1. We note that symbols are borrowed from conventional local differential privacy equations. In the algorithm, let m be the number of participating vehicles and $\gamma \in [0,1]$ be the target quantile of the Gaussian distribution at which we want to clip. For each round t, ${\mathcal{Q}}^t$ denotes a set of vehicles sampled, $C^t$ is a clipping threshold, and ${\eta }_C$ is a learning rate. Each vehicle i sends a usual delta update ${\Delta }_i^t$ and additionally a bit $b_i^t$ that is necessary for adjusting the clipping threshold, where $b_i^t={\mathbb{I}}_{\left|\right|{\Delta }_i^t{\left|\right|}_2\le C^t}$. When denoting an average value, ${\overline{b}}^t=\frac{1}{m}{\sum }_{i\in {\mathcal{Q}}^t}{b}_i^t$, our algorithm follows a geometric update rule: $C\leftarrow C·exp(-{\eta }_C(\overline{b}-\gamma ))$ [57]. To perturb sensitive information ${\overline{b}}^t$, Gaussian noise is added to the sum, obtaining ${\tilde{b}}^t=\frac{1}{m}({\sum }_{i\in {\mathcal{Q}}^t}{b}_i^t+\mathcal{N}(O,{\sigma }_b^2))$. The training procedure initializes two parameters: it sets a model to ${\theta }^0$ and a clipping threshold to $C^0$. It also takes two noise parameters: a noise standard deviation on clipped counts $\sum b_i^t$ (${\sigma }_b$) and a noise multiplier on the vector sum $\sum {\Delta }_i^t$ ($z_{\Delta }$), where $z_{\Delta }={(z^{-2}-{\left(2{\sigma }_b\right)}^{-2})}^{-\frac{1}{2}}$. The combined averaging algorithm on the coordinator first limits the updated parameter values of each vehicle to not exceed the clipping critical value before integrating the values. Then, it adds Gaussian noise to the sum, which is used to update the global parameters and clipping values. The major updates are summarized below:

Algorithm 1 Mobility-based differentially private federated learning with adaptive clipping for Internet of Vehicles

1: procedure Training(m, $\gamma$, ${\eta }_c$, ${\eta }_s$, ${\eta }_C$, $z_{\Delta }$, ${\sigma }_b$, $\beta$)  2:     for each round t = 0, 1, 2,... do  3:         ${\mathcal{Q}}^t\leftarrow$ a set of vehicles sampled in round t  4:         for each vehicle $i\in {\mathcal{Q}}^t$ do  5:            (${\Delta }_i^t$, $b_i^t$) ← FedAvg(i, ${\theta }^t$, ${\eta }_c$, $C^t$)  6:         end for  7:         ${\sigma }_{\Delta }\leftarrow z_{\Delta }{C}^t$  8:         ${\tilde{\Delta }}^t=\frac{1}{m}({\sum }_{i\in {\mathcal{Q}}^t}{\Delta }_i^t+\mathcal{N}(0,I{\sigma }_{\Delta }^2))$  9:         ${\overline{\Delta }}^t=\beta {\overline{\Delta }}^{t-1}+(1-\beta ){\tilde{\Delta }}^t$ 10:         ${\theta }^{t+1}\leftarrow {\theta }^t+{\eta }_s{\overline{\Delta }}^t$ 11:         ${\tilde{b}}^t=\frac{1}{m}({\sum }_{i\in {\mathcal{Q}}^t}{b}_i^t+\mathcal{N}(O,{\sigma }_b^2))$ 12:         $C^{t+1}\leftarrow C^t·exp(-{\eta }_C({\tilde{b}}^t-\gamma ))$ 13:     end for 14: end procedure   15: procedure FedAvg(i, ${\theta }^0$, $\eta$, C) 16:     $\theta \leftarrow {\theta }^0$ 17:     $\mathcal{S}\leftarrow$ vehicle i’s local data split into batches 18:     for batch $s\in \mathcal{S}$ do 19:         $\theta \leftarrow \theta -\eta \nabla \mathcal{L}(\theta ;s)$ 20:     end for 21:     $\Delta \leftarrow \theta -{\theta }^0$ 22:     $b\leftarrow {\mathbb{I}}_{\left|\right|\Delta \left|\right|\le C}$ 23:     ${\Delta }^{\prime }\leftarrow \Delta ·min(1,\frac{C}{\left|\right|\Delta \left|\right|})$ 24:     return $({\Delta }^{\prime },b)$ 25: end procedure

• The model update values of vehicles participating in the training are clipped before transmission to the federated learning coordinator to limit the individual effects of each vehicle.
• To obscure the effects of individual vehicles, it adds sufficient noise values to the total values updated before using the combined averaging algorithm.
• To determine the value to be clipped, adaptive clipping is applied and Gaussian is used for noise.

4.3. Mobility Pattern

This work considers two mobility patterns: a random waypoint (RWP) model [62] and a reference point group mobility (RPGM) model [63]. In RWP, mobile nodes (vehicles) move freely and randomly without restrictions. That is, the direction, speed, and destination are all chosen randomly, independent of other nodes. Its simplicity and wide availability make it one of the most popular mobility models. Its movement includes pause times between changes in destination and speed. A mobile node begins by staying in a location for a fixed number of seconds (pause time). Upon timeout, the node chooses a random destination in a simulation area and a random speed between min_speed and max_speed, where min_speed is not zero. It then travels to the destination at this speed. Upon arrival, the node stays again for a fixed period before choosing another random destination and speed. It repeats this process until the end of the simulation. This work adopts the RWP model to simulate events that do not occur frequently but are important for training, such as traffic accidents [64]. In the model, a rare-event generation mechanism is used for sensing training data input. With respect to differential privacy, a noise multiplier is applied when updating model parameter values for adaptive clipping.

In RPGM, mobile nodes are divided into groups, and group members move together. Each group has a logical center that travels through a random path following the RWP model. The motion of the center characterizes the movement of member nodes, including direction and speed. Each member node randomly moves around its own predefined reference point that is in the vicinity of the center and whose location is continuously updated according to the center’s mobility. The node’s random motion within the group also follows the RWP model. This way, the node follows the center’s mobility closely, with some deviation. Figure 4 illustrates the movement of three mobile nodes with the RPGM model. At time t, three black dots represent reference points, $RP\left(t\right)$, for the three mobile nodes (denoted as big white circles). The motion of the logical center is represented by a group motion vector, $\overline{GM}$, that is used to calculate new locations of the reference points, $RP(t+1)$, at time $t+1$. Then, the new position of a node is calculated by summing the new reference point with a random motion vector, $\overline{RM}$. The direction of $\overline{RM}$ is uniformly distributed between 0 and 2$\pi$, and its length is also uniformly distributed within a predefined radius centered at $RP(t+1)$. The RPGM model is one of the most popular group mobility models to represent vehicle platooning, and thus it is adopted in this work to represent vehicles’ mobility in non-accident situations [65]. The model uses a grid-based observation mechanism as a training input data sensing method in a federated learning setting and an adaptive clipping method for differential privacy.

5. Experimental Setup

The goal of the experiments and evaluation is to assess the accuracy and privacy performance of the proposed, differentially private, mobility-based federated learning model. For the experiments, we use digit images in the Federated Extended MNIST (FEMNIST) dataset that partitions the EMNIST dataset based on writers [66]. As such, the data replicate non-IID data distribution for federated learning. The mobility-based federated learning model (named mobile FL in the resulting graphs) is implemented by using the tensorflow-federated and tensorflow-privacy libraries. For performance comparison, we also implement a conventional federated learning model (named vanilla FL), where a coordinator selects participants randomly.

Experiments are run with 750 vehicles in total deployed in a one-square-kilometer area for one hour. Each vehicle is a candidate for a participant who executes a training process and is assumed to be connected to the coordinator at all times. The coordinator selects participating vehicles for each learning; it takes different selection strategies in different learning models and mobility patterns. Experiments are run in two different scenarios that differ in mobility models, mechanisms for data sensing, and differential privacy algorithms. In the first scenario with the RPGM model, each vehicle periodically detects its location and training data every 10 s for the mobile FL model. In the vanilla FL model, a coordinator clusters vehicles into groups based on their location and selects vehicles (as training nodes) from each group that detect the most recent training data. The second scenario with the RWP model sets a speed range of 5–20 m/s, a pause probability of 0.5, and a maximum pause duration of 5 min. Events are randomly generated following a Poisson distribution. An event can occur anywhere inside the simulation area, and once it occurs, it stays active for a limited amount of time. An active event is represented as a small area. Once a vehicle moves into the event area, it detects and records data samples related to the event. To account for the space–time locality, stored data are removed after a certain period of time after recording or when the vehicle drives away from the event area.

Privacy performance is evaluated by a noise value in differential privacy. For experiments, the noise sensitivity of the training models (i.e., the mobile FL model and the vanilla FL model) must be determined by adaptive clipping. To this end, we conduct preliminary experiments after each model is multiplied by noise multiplier (z) values of 0, 0.5, and 0.75. The results identify a noise tolerance level that does not harm the accuracy performance critically at each model with a relatively small number of vehicles (10 vehicles). They can also provide the number of vehicles and noise multiplier values, which is necessary to ensure the accuracy of a model that an application wants, given target $\delta$ and $\epsilon$ values. Next, the experiments run final trainings with these numbers. We note that a noise multiplier value of 0 represents that differential privacy is not applied to a training model, whereas values of 0.5 or 0.75 indicate privacy-enhanced training models in this work.

6. Performance Evaluation

This section presents the experimental results in terms of accuracy and loss performance (our preliminary data were presented as a poster in [67]). We show results in the scenario with the RPGM model first, as it represents vehicles mobility in non-accident situations. Then, the experimental results in the case of events (e.g., car accidents) are given.

6.1. Scenario 1. Reference Point Group Mobility

Figure 5 and Figure 6 illustrate performance in terms of model accuracy and training loss in vanilla FL and mobile FL, respectively, in the RPGM mobility scenario. The figures also show the impacts of different noise levels applied to them for privacy. The graphs clearly demonstrate the trade-off between accuracy and privacy. We initially set the performance of models with the noise multiplier of 0.5 as the target performance. The ($\delta$, $\epsilon$) values are set to (2, 1 × 10⁵), which are widely accepted. Then, each model adjusts both the number of vehicles participating in learning per round and the noise value until it achieves the target performance for privacy. Our experiments find that 36 participating vehicles and a noise multiplier value of 1.6 (while keeping the training steps at 200) ensure the required level of privacy and accuracy performance for the models.

Table 1. Enumerated model accuracy and loss in the RPGM model.

Model	Accuracy (%)	Loss
Vanilla FL, z = 0	80.07	0.6301
Vanilla FL, z = 0.5 (privacy-enhanced)	77.35	0.6970
Mobile FL, z = 0	82.86	0.5542
Mobile FL, z = 0.5 (privacy-enhanced)	80.28	0.6327

enumerates and compares the accuracy and loss values of the four training models (vanilla FL, mobile FL, and their two privacy-enhanced models). Mobile FL shows the best performance. The privacy-enhanced mobile FL model performs with 80.28% accuracy, which is degraded by 3.11% when compared to mobile FL. It is interesting to observe that it performs better than vanilla FL; that is, it is able to protect private data and predict results more correctly than conventional FL models. The accuracy performance of the two privacy-enhanced FL models degrades by 3.26% on average compared to non-privacy FL models, but this loss is fully compensated for by their capabilities in protecting vehicles against inference attacks.

6.2. Scenario 2. Random Waypoint

Figure 7 and Figure 8 illustrate performance in terms of model accuracy and training loss in vanilla FL and mobile FL, respectively, in the RWP mobility scenario. The figures also show the impacts of different noise levels applied to them for privacy. In the graphs, a noticeable observation is that performance in vanilla FL drops more significantly than that in mobile FL as the noise multiplier value increases. In the latter, a coordinator makes use of its knowledge base maintaining vehicles’ semantic information and selects vehicles that have more data needed for learning as participants before training. This method is particularly important in this scenario because a small number of vehicles may store data associated with accidents, and most of them are highly likely to be selected as participants, which thus prevents dramatic performance degradation.

Table 2. Enumerated model accuracy and loss in the RWP model.

Model	Accuracy (%)	Loss
Vanilla FL, z = 0	73.17	0.8369
Vanilla FL, z = 0.5 (privacy-enhanced)	41.73	1.7769
Mobile FL, z = 0	82.26	0.5691
Mobile FL, z = 0.5 (privacy-enhanced)	79.87	0.6388

summarizes the experimental results. Just like in scenario 1, mobile FL performs best, and the privacy-enhanced mobile FL model shows a 2.96% lower accuracy performance than the former. The latter also performs better than vanilla FL by 9.16%, which is a much larger gap than that in scenario 1. The worst performance is from the privacy-enhanced vanilla FL model. Vehicles having no knowledge about accidents are equally likely to be selected as participants, and their update values are noised, which results in a sharp drop in performance, from 73.17% to 41.73%.

6.3. Membership Inference Attack

The last experiment sets up a membership inference attack [6] on the first mobility scenario that can ensure the desired level of privacy preservation by applying the adaptive clipping method.

Table 3. Success rate of membership inference attack in four training models in the RPGM scenario.

Model	Attack Success Rate
Vanilla FL, z = 0	0.77
Vanilla FL, z = 0.5 (privacy-enhanced)	0.68
Mobile FL, z = 0	0.75
Mobile FL, z = 0.5 (privacy-enhanced)	0.64

shows the attack success rates (ASRs) when the attack is launched on the four training models. The closer the value is to 0.5, the less able the attacker is to distinguish between training data. As shown, the two privacy-enhanced models degrade the rate by around 10%, meaning that information is more protected from the attack. However, since the value itself is greater than 0.5, applying differential privacy alone may not be enough to resolve the attack completely. Using additional technologies such as weight pruning [68] could mitigate its effect, which remains an area that will be addressed in one of our future works.

7. Discussion

Communication costs can be a very important constraint in federated learning, as servers and participants need to communicate model parameters frequently. Existing research has taken three strategies to reduce communication overhead: an optimization algorithm, model compression, and client selection. A rational strategy is to reduce communication frequency and increase the time interval of model aggregation, which can be achieved by improving the performance of local training [9]. The authors also found that increasing local iteration time and reducing global communication rounds can increase a global model’s convergence efficiency while ensuring the performance of federated learning. Compressing network traffic by reducing the overall size of the uploaded model parameters is another useful strategy. Recent works [69,70] investigated Top-k sparsification-based compression techniques in which parameters with the largest variations (i.e., k “important” parameters) are selected and transmitted to a server, instead of uploading an entire local model. The authors in [71,72] employed quantization and encoding schemes to further reduce the sizes of the “value” and “index” parts in the k parameters. Furthermore, client selection methods are logical strategies for improving communication efficiency in federated learning. Liu et al. [73] studied a method that detects whether local model updates from clients match a global model’s update trend to avoid sending irrelevant parameters. Deng et al. [74] investigated a method to evaluate clients’ model quality to select high-quality clients for federated learning. Lai et al. [75] proposed a framework to prioritize client participation using data that provide the greatest utility in improving model accuracy and training speed. Nishio and Yonetani [76] designed a new protocol that predefines deadlines for updating and uploading model parameters so that a server can aggregate them in a given time. The mobility-based federated learning in this work requires vehicles to share their own knowledge with a coordinating server, which adds additional communication cost. However, the server preferentially selects vehicles that have important data for model training, which enables satisfactory accuracy to be achieved in fewer communication rounds. In particular, the selection mechanism in this work is similar to that in [74], demonstrating that their framework can improve accuracy performance by 32–113% after 30 learning rounds and can converge to the global optimum with fewer participants.

The convergence of federated learning has been analyzed in many studies. Several works examined the effects of data heterogeneity on convergence by assuming full client participation [55,77,78]. However, not all clients actually participate in each training round, and thus the authors in [79,80,81] analyzed federated learning with unbiased partial client participation where clients are selected uniformly at random. These works show that the federated averaging (FedAvg) algorithm on non-IID datasets achieves a convergence rate of $\mathcal{O}$(1/T), where T is the number of communication rounds. Since the mobility-based federated learning in this work, especially in non-accident situations, takes unbiased partial client participation, its error convergence is expected to follow the same rate as shown in our experimental results in Section 6.1. In the case of car accident events, on the other hand, our learning model takes a biased client selection strategy, and recent empirical studies have shown that biased selection affects convergence speed. Nishio and Yonetani [76] proposed a client selection method based on the client’s radio resources and hardware to reduce communication costs. The authors in [82,83] discovered that selecting clients with higher local losses can sharply increase the rate of convergence. Our experimental results in Section 6.2 do not deviate from the observation that biased selection affects convergence. An error in vanilla FL almost fails to converge, whereas the curves in mobile FL look stabilized and show a similar convergence rate to that in Section 6.1.

8. Conclusions

This work has presented performance issues in differentially private federated learning models running in different mobility scenarios. The accuracy performance and level of protection have been evaluated by considering three factors: learning models, vehicle mobility, and a privacy algorithm. The accuracy of the privacy-enhanced learning models was mostly affected by vehicles’ mobility and the traffic situations on roads. It was also observed that the conventional learning model widely used in federated learning did not work well in the mobile, vehicular environment, especially when enhanced with a privacy method. This is mainly attributed to the performance of the participant selection mechanism, which is highly affected by vehicles’ mobility. An additional observation is that the adaptive clipping method applied in this work is more useful than prior methods because it guarantees model performance and the target degree of privacy preservation and thus can proceed with model learning.

This study focused on privacy and the associated trade-offs in mobility-based federated learning. We plan to consider a weight pruning method, which is a recently proposed alternative to membership inference attacks, to enable more efficient and accurate learning while also showing improved security results. Future works will also examine how well privacy-enhanced learning models protect data against variants of inference attacks and expand on experiments by considering more mobility scenarios.