A Traffic-Aware Federated Imitation Learning Framework for Motion Control at Unsignalized Intersections with Internet of Vehicles

Abstract

The wealth of data and the enhanced computation capabilities of Internet of Vehicles (IoV) enable the optimized motion control of vehicles passing through an intersection without traffic lights. However, more intersections and demands for privacy protection pose new challenges to motion control optimization. Federated Learning (FL) can protect privacy via model interaction in IoV, but traditional FL methods hardly deal with the transportation issue. To address the aforementioned issue, this study proposes a Traffic-Aware Federated Imitation learning framework for Motion Control (TAFI-MC), consisting of Vehicle Interactors (VIs), Edge Trainers (ETs), and a Cloud Aggregator (CA). An Imitation Learning (IL) algorithm is integrated into TAFI-MC to improve motion control. Furthermore, a loss-aware experience selection strategy is explored to reduce communication overhead between ETs and VIs. The experimental results show that the proposed TAFI-MC outperforms imitated rules in the respect of collision avoidance and driving comfort, and the experience selection strategy can reduce communication overheads while ensuring convergence.

1. Introduction

The Internet of Vehicles (IoV) provides ubiquitous connectivity in transportation scenarios, allowing massive data interaction among smart vehicles, road infrastructures, and remote computing facilities. Empowered by IoV, the issues of safety [1] and efficiency [2] can be addressed in a cooperative manner. This study focuses on the optimization of unsignalized intersection management [3], where motions of each vehicle are precisely controlled to pass through an intersection without traffic lights.

Compared to traffic light controls, unsignalized intersection management promises higher efficiency while ensuring driving safety. The process of deciding the vehicle passing sequence is known as scheduling. Two main categories of existing scheduling policies are negotiation-based [4] and reservation-based [5]. Because the actions taken by vehicles depend on real-time driving conditions, which is a typical Markov Decision Process (MDP), Reinforcement Learning (RL) is suitable to address this issue. Guan et al. [6] proposed an RL-based method for guiding a fixed number of vehicles through an intersection. Wu et al. [7] proposed a cooperative RL method to improve traffic efficiency while ensuring safety. This method decoupled the relationship between identity and driving information on vehicles. Jiang et al. [8] proposed a two-stage RL incorporating end-edge-cloud architecture to achieve global optimization among multiple homogeneous intersections. However, low sample efficiency and limited safety performance make practical applications difficult.

Compared with RL’s low sample inefficiency, Imitation Learning (IL) can accelerate the training process and does not need to specify how the task should be performed. This is because the expected behaviors are embedded in expert demonstrations. IL has been one of the popular approaches used to train control policy for many fields for efficiency and simplicity, such as resource scheduling [9], signalized intersection control [10], and smart manufacturing [11]. Nowadays, rule-based strategies are still the mainstream of self-driving development in the industry because of their interpretability. To facilitate self-driving development, many companies established huge scenario libraries [12] and simulator [13]. Due to the existence of thresholds in various rules, there is a bottleneck in improving driving comfort. IL can mitigate the thresholds’ impact during translating rules into neural network knowledge.

Although some progress has been made in learning-based motion control at intersections, three problems remain:

(1)

Isolation: To balance motion control performance and privacy preservation, setting a local center to assist the motion control optimization is essential, which means that some privacy-sensitive data are delivered to the local center. However, for privacy requirements, data exposure to cloud nodes or other peer nodes is prohibited. This constructs data isolation among intersections;

(2)

Heterogeneity: Due to vehicles’ non-uniform spatial distribution, intersections in different areas carry different traffic flows. One of the traffic flow characteristics is the flow rate difference. Due to different traffic flows at different intersections, generated experience data drives obtained the RL model to demonstrate different capabilities for motion control optimization. Therefore, conventional model parameters averaging cannot meet the performance requirements at different intersections;

(3)

Scalability: As the number of IoV-enabled unsignalized intersections grows, data generated by the vehicles increase. Because of the incurred high computation and communication budget, any learning-based algorithm with a centralized property may find it challenging to handle such data.

Communication overhead is an unavoidable topic in distributed deep learning. Many researchers try to reduce the communication overhead while ensuring model convergence. Luo et al. [14] proposed a gradient compression method, which reduced the communication overhead between the master node and the multiple compute nodes. Shi et al. [15] proposed the optimal merged gradient sparsification algorithm based on SGD to solve the high communication overhead caused by gradient sparsity in deep learning. Sattler et al. [16] proposed a robust compression framework, sparse ternary compression, aiming at the limited scope of existing schemes to reduce communication overhead between the server and clients in federated learning. Many existing studies assume that data stay with a deep learning algorithm. This paper considers a scenario in which data and algorithms are separated. Trainers need to collect data from vehicles to generate a neural network model. Inspired by [17], not all experience is equally helpful for model training. A loss-aware experience selection strategy is proposed to reduce the communication overhead by discarding low-value experiences.

In this study, Federated Learning (FL) is chosen to address the challenges mentioned above. FL is a distributed deep learning paradigm, which enables multiple clients to learn a shared model while storing all the training data on clients. Many researchers transfer FL into many areas, such as communication network slicing [18], traffic classification [19], and crowd computing [20] This paradigm strengthens the client’s data independence and raises researchers’ extensive interest. Yu et al. [21] proposed an FL-based cooperative hierarchical caching scheme to address content popularity prediction’s privacy issues in IoV-enabled fog networks. Chen et al. [22] focused on client selection to improve training efficiency in asynchronous FL with unstable connection in IoV. Zhao et al. [23] model the training and transmission latency of a novel FL paradigm combined with blockchain and adopt a duel deep Q-learning algorithm to minimize the system latency. To enhance the perception of IoV applications, Lim et al. [24] activated unmanned aerial vehicles and enabled an FL-based approach for their privacy-preserving collaboration. However, the existing researchers hardly pay attention to transportation issues with FL-empowered IoV.

When privacy problems are involved in vehicle motion control optimization, all data should be kept in a vehicle. To obtain better motion control, this study relaxes the privacy restriction to a local area, that is, a single-intersection area. Different intersections construct independent scopes. To achieve cooperative optimization among intersections, a Traffic-Aware Federated Imitation Learning framework for Motion Control (TAFI-MC) is proposed for acquiring vehicle motion control policies across different intersections. The main contributions of this study are summarized as follows.

• TAFI-MC framework is proposed to optimize motion control across multiple isolated unsignalized intersections cooperatively. This framework contains three parts: vehicle interactors, edge trainers, and one cloud aggregator;
• TAFI-MC integrates an IL algorithm to obtain a safety-oriented motion control policy, which trains the model with the experience from a set of collision avoidance rules;
• A loss-aware experience selection strategy is designed, which can reduce the communication overhead by extra computation. Depending on the reference loss, each interactor generates new experiences and decides whether to upload them.

The rest of this paper is organized as follows: The system architecture is presented in Section 2. The proposed TAFI-MC for vehicle motion control is detailed in Section 3. In Section 4, a loss-aware experience selection strategy is presented. The experimental results and discussions are provided in Section 5. Section 6 concludes this paper.

2. System Architecture

This section focus on a hierarchical network in an urban scenario, which includes one Cloud Aggregator (CA), tens of Edge Trainers (ETs), and hundreds of Vehicle Interactors (VIs), as shown in Figure 1. The CA is deployed in the remote cloud and connects to a set of ETs via a reliable backhaul link. These ETs are denoted by $\mathbb{ET}=\{E{T}_1,E{T}_2,\dots ,E{T}_n,\dots ,E{T}_N\}$, where N is the number of ETs. Each ET i serves its wirelessly connected VIs, denoted by $\mathbb{VI}=\{V{I}_1,V{I}_2,\dots ,V{I}_i,\dots ,V{I}_{m(t)}\}$. Note that, under ET, the number of VI, $m(t)$, varies over time t. ETs and VIs are equipped with adequate computing power. The local model at each ET is trained by combining uploaded information from various VIs. CA is used to create a global model by combining local information of different ETs. There are three assumptions to support our work, which is similar to related work [25], on the aspect of unsignalized intersection control. For simplification, this study focuses on longitudinal control. All vehicles proceed straight through the intersection area. All vehicles can measure kinetic information, adhere to the set acceleration, and communicate with adjacent nodes, i.e., ETs and adjacent VIs.

The longitudinal motion of vehicles is given by

$$\begin{array}{cc}\hfill x^{long}(t+1)& =x^{long}(t)-v(t)T-\frac{1}{2}a(t)T^2\hfill \\ \hfill v(t+1)& =v(t)+a(t)T\hfill \end{array},$$

(1)

where $x^{long}$ is the displacement, v and a are the velocity and acceleration, respectively, and T is the discrete-time step. The change of vehicle motion state depends on the input, i.e., acceleration, at the previous time step. Note that $x^{long}(t)$ represents the distance to the conflict point in intersection, so subtraction is used in the equation instead of addition.

This paper uses distributed decision-making for motion control. That is, each vehicle constructs its cyberspace and maps adjacent vehicles to cyber objects. As a result, vehicle i decides its action $a_i$ based on its surroundings.

$$a_i=P(\overrightarrow{s_i}|\theta )$$

(2)

where $P(·|\theta )$ is a $\theta$-parameterized policy for decision-making. $\overrightarrow{s_i}$ is the state vector of vehicle i and $\overrightarrow{s_i}=\{s_i,\overrightarrow{s_{-i}}\}$. $s_i$ is a state of the ego-vehicle i, including position, velocity, and acceleration. Moreover, $\overrightarrow{s_{-i}}$ is a vehicle set other than vehicle i. $|\overrightarrow{s_{-i}}|$, the number of vehicle set $\overrightarrow{s_{-i}}$ is defined by a selection scheme. To simulate traffic flow, the number of arrived vehicles for entering each intersection during the period t is defined as $V_q(t)$. It follows a Poisson process with the parameter $\lambda$:

$$P\left(V_q(t)=g\right)=\frac{{(\lambda t)}^g}{g!}{e}^{-\lambda t}$$

(3)

where g equals the number of vehicles generated in a period t. The introduction of the Poisson process means that vehicles are dynamically created, which is similar to real traffic.

Figure 1 presents a three-layered federated deep learning architecture. The bottom layer contains VIs, a collection surrounding the information via vehicular communication. The middle layer includes several ETs equipped with computing servers and experience pools. The top layer has an aggregator, which forms global models with ETs’ local models. The multiple connected VIs interact with the environment and individually upload experience data to the local experience pool on the corresponding ETs. Each ET first generates a local model and acquires the experience from connected VIs. Then, each ET uses the local computing ability to compute a local model based on the received data. Next, the ET sends the local model to the VIs for motion control and the CA for global model aggregation. Finally, the CA aggregates the models and distributes the global model back to each ET. The above steps are repeated until a convergent global model is achieved. The trained model in this work is specially developed to output the vehicles’ accelerations in response to the contextual information surrounding the VI.

3. Federated Imitation Learning Framework for Motion Control

The proposed TAFI-MC framework is elaborated in this section. First, Traffic-Aware Federated Learning is described. Then, we introduce a set of collision avoidance rules as a basis for further optimization. Finally, an IL algorithm for motion control is investigated using rules.

3.1. Traffic-Aware Federated Learning

FL enables collaborative training of a deep neural network model among ETs under CA’s orchestration by storing the training data on each ET at intersections. With FL, privacy-related data can be kept within the scope of ET. After ET training, data is turned into a neural network model, which makes it hard to extract raw information. These privacy-related data include the vehicles’ identity and position, inferring their destination, and drivers’ identity. FL not only significantly reduces the vehicle’s privacy risk but also significantly reduces the communication overhead caused by centralized machine learning [26]. FL is enabled by multiple communication rounds (computing iteration). To conduct model training, N intersections with known traffic flows and the corresponding ETs are selected. The N ETs are indexed by n. Then, each ETs retrieves a global model from the CA and trains it with data collection from vehicles in the intersection area using its local model. Following the ET’s local training, the updated weights and gradients are returned to the CA. Finally, the CA aggregates the models collected from vehicles to construct an updated global model. Furthermore, the final-trained models are distributed to the vehicles to ensure that they pass through the intersection.

The details of the designed FL iteration consists of the following steps:

(1)

Model Distribution: A set of ETs at intersections participate in FL training. The CA distributed the global model ${\omega }_r$ to ETs. The ET n trains the global model with local data for a new model ${\omega }_r^n$. The index of communication rounds is represented by r.

(2)

Experience Upload: To improve motion control performance, each vehicle should consider other vehicles’ states for inference. However, the efficiency of vehicular distributed training is significantly low because of the insufficient number of collected samples and non-uniform distribution. Under the above setting, it is essential to use centralized training and distributed execution [26]. Then, each vehicle interacts with the environment, i.e., other vehicles, and generates enormous experience data to upload. The corresponding ET uses this data to train the local model.

(3)

FL Model Training: The proposed FL’s third step is to train the model by using local data uploaded by vehicles. Let $Exp=\{Ex{p}_1,Ex{p}_2,\dots ,Ex{p}_n,\dots ,Ex{p}_N\}$ represent the experience data stored in selected ETs. $Ex{p}_n$ denotes a local experience of the $n\mathrm{th}$ ET with a length $d_n$, $d_n=\left|Ex{p}_n\right|$. d is the size of the entire data among the selected ETs. The goal of the FL is to minimize the loss function $L(\omega )$:

$$\begin{array}{cc}\hfill \underset{\omega }{min}L(\omega )& =\sum _{n=1}^N\frac{d_n}{d}{L}_n(\omega )\mathrm{where}\hfill \\ \hfill L_n(\omega )& =\frac{1}{d_j}\sum _{j\in H_k}{l}_j(\omega ),\hfill \end{array}$$

(4)

where $l_j(\omega )$ is a loss of the motion control on the $j\mathrm{th}$ experience batch in $Ex{p}_n$ with the parameters of model $\omega$, and $d_j$ is the experience number in the $j\mathrm{th}$ batch. n denotes the index of total selected ETs N. $L_n(\omega )$ represents the local loss function of ET n. Then, minimizing the weighted average of the local loss function $L_n(\omega )$ is equal to optimizing the loss function $l(\omega )$ of FL. With this step, the experience with sensitive information is transformed into a neural network model, which is hard to extract sensitive information from.

(4)

Upload Updated Model: The fourth step is to upload the local model ${\omega }_{t+1}^n$ from ETs to the CA. The communication overhead exceeds the computing overhead [27]. The model can be compressed before being uploaded to the CA to reduce communication overhead.

(5)

Weighted Aggregation: After ETs upload their models, the fifth step is to produce a new global model ${\omega }_{r+1}$ by computing a weighted sum of all received models ${\omega }_r^n$. For the next training iteration, the newly generated global model is used. Federated Averaging (FedAVG), which is commonly used in FL, increases the proportion of local computing and decreases mini-batch sizes. In FedAVG, each ET adds computation by iterating the local updates $w_r^n\leftarrow w_r^n-\eta \nabla L_n\left(w_r^n\right)$ multiple times before the aggregation step in the CA. To aggregate the model, the weighted averaging algorithm is implemented. The weights for parameter aggregation are determined by the traffic flow of each intersection, which is ${\gamma }_n=F_n/F$. $F_n$ and F denote the traffic flow on $n\mathrm{th}$ intersections and the flow sum of all intersections, respectively. The aggregate method can then be re-written as

$$w_{r+1}\leftarrow w_r-\eta \sum _{n=1}^N{\gamma }_n\frac{d_n}{d}{w}_{r+1}^n.$$

(5)

Selected intersections with a higher traffic flow contribute more and are given greater weight in model aggregation.

3.2. Imitation Learning for Motion Control

IL model for motion control is trained in the above FL framework. It is used to obtain the motion control ability from collision avoidance rules. IL and RL both depend on environment interaction. Unlike RL, which obtains the desired behaviors based on the hidden objectives, IL directly clones the desired behaviors. IL can overcome RL’s highly uncertain initial state and the sparse reward, which can lead to an exploration trap. Therefore, this section explains how to imitate the end-to-end motion control policy using existing rules.

As shown in Figure 2, two modules are presented to support motion control policy acquisition in ET. The upper module is a set of collision avoidance rules to output expert experience, a deterministic action with fixed experience input. As the final motion control policy carrier, a deep neural network is continuously updated in the lower module. The network is updated with loss, which comes from the difference between the output action from the two modules. The proposed IL is shown in Algorithm 1.

Algorithm 1: Imitation learning for motion control.

1: Collision avoidance rules guide vehicles to make actions a 2: The state s and action a are recorded as expert experience $(s,a)$ to upload to the corresponding ET’s experience buffer 3: ET samples a batch of experience $\overrightarrow{Exp}$ from the pool 4: $\overrightarrow{Exp}$ is simultaneously forwarded to two modules, collision avoidance rules, and deep neural network 5: The loss is calculated with the output of the two modules, ${\pi }_{\theta }(\overrightarrow{s_i})$ and $a_i$ 6: The deep neural network is updated by minimizing the loss $$L(\overrightarrow{Exp})=\frac{1}{B}\sum _{i=1}^B{\left|{\pi }_{\theta }\left(\overrightarrow{s_i}\right)-a_i\right|}^2.$$ (6)

In Equation (6), $\overrightarrow{Exp}$ is a batch of experience, and the batch size is B, given in the experiment part. ${\pi }_{\theta }(·)$ is a $\theta$-parameterized policy, and each iteration updates parameter $\theta$. The state vector $\overrightarrow{s_i}$ including the state of current vehicle and surrounding vehicles, i.e., $\overrightarrow{s_i}=\{s_i,\overrightarrow{s_{-i}}\}=\{s_i,s_{i,1},\dots ,s_{i,n}\}$. n is the number of the closest vehicles to be considered into a state vector. Each state $s_{*}$ includes position, velocity, and acceleration, i.e., $s_{*}=\{x^{long},v,a\}$. The definition of closest vehicles relies on the cyber-lane, which is described in Section 3.3. The loss function can drive the deep neural network to produce safety-oriented strategies.

3.3. Collision Avoidance Rules

In this section, the concept of Cyber-Lane (CL) is introduced to reconstruct vehicles’ relationships. Vehicles in different trajectories travel in different Physical-Lanes (PLs), and these trajectories intersect at conflict points. As presented in Figure 3, lane-A is taken as an example. The generation of CL-A is based on PL-A, and vehicles on the conflict lane are projected to CL-A centered on the conflict point. In other words, the PL number is equal to the CL number. After being projected, $D1\to A$ and $B1\to A$ appear on CL-A. From the standpoint of CL, the position relationship of all vehicles is reorganized. Vehicle $A2$ appears between vehicles $B1$ and $D1$ on CL-A, and vehicle $A2$ treats vehicle $B1$ or $D1$, instead of vehicle $A1$, as the closest vehicle. The action of vehicle $B1$ and $D1$ will naturally be considered by vehicle $A1$.

The set of rules considers three factors, including space, time, and acceleration. Space is the first factor, which can directly determine whether the collision has occurred. As a second-order factor, time considers whether the vehicle will collide in the future. The acceleration indicates whether the collision will be avoided. In this paper, the above factors are quantified as Safety Value (SV).

The first factor is space. The safety value for space $S{V}_{j,s}$ is calculated as below,

$$S{V}_{j,s}=log({(\frac{d_{j,nearest}}{{\alpha }_s})}^{{\beta }_s}),$$

(7)

where $d_{j,nearest}$ denotes the distance between vehicle j and its nearest vehicle on the virtual lane. ${\alpha }_s$ normalizes $d_{j,nearest}$, and it can be treated as the expected headway distance. ${\beta }_s$ increases the offset to improve the $log(·)$ effect. The SV and nearest inter-vehicle space distance have a positive correlation.

The second factor is time. The SV for time $S{V}_{j,t}$ is calculated as below,

$$S{V}_{j,t}=\left\{\begin{array}{cc}-{\left[\frac{{\alpha }_t}{tanh(-t_{j,nearest})}\right]}^{{\beta }_t}& 0<t_{j,nearest}<1\\ 2& otherwise\end{array},\right.$$

(8)

where $t_{i,nearest}$ denotes Time To Collision (TTC) between vehicle j and its nearest vehicle. The function $tanh(·)$ is used to mark the nearby collision risk in the sensitive range, where $t_{i,nearest}$ is less than 1. TTC and SV have a negative correlation.

The third factor is acceleration. The SV for acceleration $S{V}_{j,acc}$ is calculated as below,

$$S{V}_{j,acc}={\lambda }_{acc}\times ac{c}_{j,front}\times log\left(min{\left(\frac{d_{j,front}}{d_{threshold}},{\alpha }_{acc}\right)}^{{\beta }_{acc}}\right),$$

(9)

where $d_{j,front}$ is the distance from vehicle j to its front vehicle, $ac{c}_{j,f}$ is the acceleration of the vehicle in front of vehicle j, and $d_{threshold}$ is the space distance safety threshold. $min(·)$ is used to control the range of $\frac{d_{j,front}}{d_{threshold}}$ within $[0,{\alpha }_{acc}]$. Discount factor ${\lambda }_{acc}$ is introduced to limit the influence of acceleration in the calculation of SV.

The combination of SV is calculated as follows,

$$\begin{array}{cc}\hfill S{V}_j& =Comb\left(S{V}_{j,s},S{V}_{j,t},S{V}_{j,acc}|S{V}_{max},S{V}_{min}\right)\hfill \\ & =clip\left(\left(S{V}_{j,d}+S{V}_{j,t}+S{V}_{j,acc}\right),S{V}_{max},S{V}_{min}\right),\hfill \end{array}$$

(10)

where $S{V}_{j,d}$, $S{V}_{j,t}$, and $S{V}_{j,acc}$ are defined above. To obtain a proper acceleration value in Equation (11), $clip(·)$ is used to limit the maximum and minimum values. A larger $S{V}_j$ indicates that the vehicle j is driving in a safer environment. Based on the above SV, the ego vehicle’s action can be calculated as follows,

$$a_{exe}=\left\{\begin{array}{cc}\left|\frac{SV}{\eta }\right|& d_f\le d_b\\ \frac{SV}{\eta }& d_f>d_b\end{array}\right.,$$

(11)

where $d_f$ is the distance to the vehicle in front, $d_b$ is the distance to the vehicle behind, and $\eta$ is used to convert safety value to action, i.e. ego vehicle’s acceleration. The experimental results shown in Section 5 demonstrate that rules can achieve collision avoidance under different traffic flows.

4. Loss-Aware Experience Selection Strategy

In the setting of the proposed IL in Section 3, the experience, i.e., $(s,a)$ tuple, is generated by each vehicle and uploaded to ETs for training at 10 Hz. This will consume many communication resources when the number of vehicles near an intersection is enormous. This section introduces computing for communication, where additional computation is performed to reduce communication overhead. In this section, the extra computation is placed on vehicles and edge nodes. Vehicles calculate the loss and compare it to thresholds given by edge nodes. Edge nodes produce a threshold for loss comparison. Therefore, combined with the concept of computing for communication, a loss-aware experience selection strategy is proposed to discard experience that helps in model training.

As displayed in Figure 4, the proposed strategy is applied between vehicle interactors and edge trainers. When a vehicle enters an intersection area, it requests edge trainers for the most recent model and threshold. Then, using vehicular collision avoidance rules, all vehicles interact with the environment and generate experience $(s,a)$. The vehicle acts with other vehicles and outputs action loss between the rules and the model. The loss is compared with loss threshold $Th.$ If the action loss is larger than $Th.$, the corresponding experience $(s,a)$ can be uploaded to the edge trainers, and vice versa. Finally, the acquired experiences enable the edge trainers to perform IL training and output a new threshold. The threshold is calculated on ETs as follows,

$$Th.=sort(\overrightarrow{Exp},“loss”)[p\times B][“loss”]$$

(12)

where $\overrightarrow{Exp}$ is a batch of experience for training, and its size is B. The function $sort(·,“loss”)$ sorts experiences in ascending order of loss value. p is the discard rate. According to Equation (12), the threshold is the $(p\times B)\mathrm{th}$ smallest loss in the experience batch. Because the experience will be partly discarded, the communication overhead is reduced.

5. Results and Discussion

5.1. Simulation Settings

The proposed algorithm is trained and evaluated in a self-designed intersection motion control platform, developed with Python 3.5. The platform contains four of the same intersections, but they differ in traffic flow rates. Considering that different intersections have different traffic flow characteristics, isolated intersections with different traffic flows are set up to verify the motion control brought by TAFI-MC. In this experiment, the traffic flows are 300/900/1500/2100 veh/lane/hour, where veh means the number of vehicles. According to different traffic flow rates, vehicles are generated using the Poisson process and allowed to go straight without steering. The related motion control parameters are listed in

Table 1. Experimental Parameters.

Parameter	Value
Simulator
Lane length (m)	150
Vehicle size (m)	2
Velocity (m/s)	$[6,13]$
Initial velocity (m/s)	10
Acceleration (m/s2)	$[-3,3]$
Discrete-time step T (s)	$0.1$
Safety Value
Space normalization factor ${\alpha }_s$	10
Space exponetial factor ${\beta }_s$	10
Time linear factor ${\alpha }_t$	$1.5$
Time exponetial factor ${\beta }_t$	2
Acceleration exponetial factor ${\alpha }_{acc}$	$1.5$
Acceleration exponetial factor ${\beta }_{acc}$	12
Acceleration linear factor ${\lambda }_{acc}$	$0.2$
Safety value upper bound $S{V}_{max}$	20
Safety value lower bound $S{V}_{min}$	$-20$
Conversion factor $\eta$	3
Fusion factor $\omega$	$0.2$
Weighting factor $\epsilon$	$0.5$
Vehicle Selection
Number of the closest vehicle n	5

. All source codes about the proposed TAFI-MC framework are provided on GitHub (https://github.com/shogun2015/TAFI-MC (accessed on 3 December 2021)).

The starting point of this study is to balance model performance and privacy protection. Therefore, this study allows ETs to collect vehicle information in their corresponding intersections to complete IL training. In other words, each IL training only handles a single flow-rate traffic. In the proposed schemes, the neural network (NN) is used to imitate the collision-free rules by minimizing the action loss between the NN and the rules. There is only one NN, containing three dense layers and two normalization layers, and ReLU is used as the activation function in the hidden layers. The output layer is activated by $tanh(·)$. To fit the acceleration range, the NN output is multiplied by 3. The complete hyper-parameters are listed in

Table 2. Parameters for Neural Networks.

Parameter	Value
Discounted factor $\gamma$	0.8
Batch Size B	48
Soft update factor $\tau$	0.99
Episode	50
Learning rate	0.001 → 0
Optimizer	Adam
Network Architecture
Dense layer 1#	64
Dense layer 2#	64
Dense layer 3#	1

. However, because of poor interpretability and limited safety performance of end-to-end NN inference, a weighted operation is added below,

$$a_{exe}=\epsilon \times a_{NN}+(1-\epsilon )\times a_{rule},$$

(13)

where $\epsilon$ is a factor for smoothing the NN output $a_{NN}$ with a rule output $a_{rule}$ to ensure safe driving. In the following experiment results, Model denotes action output using NN only, Model+Rule represents mixed output using NN and rules.

5.2. Metric

Three indicators are chosen to comprehensively evaluate the performance of the proposed motion control methods at intersections, i.e., collision rate, average jerk, and average velocity.

Collision rate is the first metric to evaluate the motion control safety, which is shown below,

$$r_{collision}=\frac{n_{collision}}{N_{veh}},$$

(14)

where $n_{collision}$ is the number of collisions, and $N_{veh}$ is the total number of vehicles. A larger $r_{collision}$ means the motion control algorithm is incapable of achieving collision avoidance. The algorithm in this study is designed to reduce the metric value to 0.

Average jerk is an important metric to evaluate motion control comfort. As presented in [28,29], the average jerk can be defined as below,

$$J_{avg}=\frac{1}{N_{veh}}\sum _{i=1}^{N_{veh}}\sum _t{j}_{i,t}^2,$$

(15)

where $j_{i,t}$ is the $i\mathrm{th}$ vehicle’s jerk. The jerk is defined by $j_{i,t}={\dot{a}}_{i,t}$, and $a_{i,t}$ is the $i\mathrm{th}$ vehicle’s acceleration at the time step t. A larger $J_{avg}$ indicates more frequent or sharp acceleration and deceleration, resulting in more severe driving discomfort.

While ensuring a low collision rate, average velocity is a metric that must be improved. It can be defined as below,

$$v_{avg}=\frac{1}{N_{veh}}\sum _{i=1}^{N_{veh}}\frac{l_{lane}}{t_i},$$

(16)

where $t_i$ is the $i\mathrm{th}$ vehicle’s travel time and $l_{lane}$ is the lane length, which is given in Table 1. A larger $v_{avg}$ represents vehicles with the proposed algorithm that will drive faster, resulting in high throughput for the transportation system.

5.3. Discussion

The entire experiment is divided into two parts. In the first part, the benchmark Reinforcement Learning (RL), the proposed IL, and TAFI-MC are evaluated from the three metrics: collision rate, average jerk, average velocity, as described in Section 3.3. The IL and RL are trained in four traffic flows (i.e., 300/900/1500/2100 veh/lane/hour), and verified in seven traffic flows (i.e., 300/600/900/1200/1500/1800/2100 veh/lane/hour). The set of collision avoidance rules described in Section 3.3 is represented by a black line in Figure 5a, Figure 6a, and Figure 7a. TAFI-MC is evaluated with two aggregation methods (same-proportion and traffic-aware). Same-proportion means the four models have the same weight, whereas traffic-aware means the weight is $1:3:5:7$. The second part verifies the performance of the proposed loss-aware experience selection strategy with different discard factors p.

Figure 5 depicts the comparison of collision rate with the three algorithms, IL, RL, and TAFI-MC. In Figure 5a, the black line remains at zero collision, which means that the rules have good collision avoidance ability at all experimental flow rates. This is because the rules are safety oriented and avoid all collisions from the perspective of position, speed, and acceleration. In addition, the results demonstrate that the proposed IL scheme can help the model learn collision avoidance from the rules. Only in high flow evaluation environment will the model trained in low flow produce some collisions. This is because, in the low traffic flow, the experience samples come from large inter-vehicle distances. The model trained by this experience has difficulties guiding a vehicle to handle a small inter-vehicle distance. As shown in Figure 5b, the single traffic flow training makes RL show unacceptable collision rates, because RL is more dependent on interaction samples, which is low efficiency. RL’s low safety performance exceeds the correction ability of rules. In Figure 5c, traffic-aware model aggregation method outperforms the same-proportion method. The traffic-aware method does not need rules correction. To sum up, IL can obtain stronger collision avoidance ability than RL to greatly reduce the collision rate, and TAFI-MC with the proposed traffic-aware model aggregation can enhance collision avoidance.

Figure 6 shows the comparison of average jerks with the three algorithms. In Figure 6a, collision avoidance rules show the highest average jerk, because rule-making inevitably adopts the threshold trigger mechanism. Only considered states can be mapped to actions. Compared with rules, IL significantly reduces jerks than rules. The model training process helps IL complete the state-action mapping by gradually approaching the rules with a low learning rate. Note that, although rules can reduce collision rates, their threshold trigger mechanism will increase jerks inevitably. As discussed in the collision rate, the model trained in high traffic is also slightly insufficient when it comes to dealing with low traffic. This is mainly reflected in that the high-traffic model has higher average jerk than the low-traffic model. The performance of RL, in Figure 6b, is similar to IL. This demonstrates that the control policies based on interaction experiences have similar effects on driving comfort. As shown in Figure 6c, TAFI-MC with traffic-aware model aggregation can further reduce average jerks by up to 41.37% than IL trained models at any single traffic flow.

Taking a comprehensive view at the three figures in Figure 7, we can see that the three algorithms achieve high average velocity, which also means high traffic efficiency and throughput. With the increase in traffic flow, the average velocity has decreased slightly. This is because more vehicles rise the collision probability of vehicles, and the collision avoidance motions of vehicles reduce the average velocity. In Figure 7a,b, models trained under sparse traffic flows show relatively high velocity, because a small number of vehicles makes it hard to cause the collision avoidance motion of vehicles. In Figure 7c, FL aggregates the gap of different models’ motion control. Furthermore, traffic-awareness brings slightly lower velocity, but ensures zero collision rate. In combination with Figure 5, Figure 6 and Figure 7, the reduction of collision rates inevitably leads to the increase of average jerks. The most obvious ones are the models trained in the most sparse traffic flow, i.e., 300 veh/lane/hour. Fortunately, FL introduced by TAFI-MC makes up for this deficiency and keeps different metrics at better values.

In Figure 8, the convergent loss curves are represented by solid lines and the non-convergent curves are represented by dotted lines. The results demonstrate that, with the number of discarded experiences increasing, the IL model is more difficult to converge. It shows the loss curves with different discard factors p. When p is no more than 10% and the training step exceeds 4000, the trained models can be convergent and stable. It can be easily found that the models converge and stabilize from 6000 training steps, thus only the communication overheads before convergence is counted. In addition, the loss curves peak at about 2000 training steps. Since that, losses and models have been gradually stable. In the following analysis, only the convergent model (i.e., models with $p=1\%,2\%,5\%,10\%$) is considered.

In Figure 9, the cumulative communication reduction rate is presented. A higher p brings less communication overhead. At the beginning of model training, the VIs’ model parameters are random. As a threshold, the reference loss discards more experience, which leads to a greater reduction in communication overhead. After the loss curves peak at 2000 training steps, the communication reduction rates gradually decrease with the training progress. When the model converges, the cumulative communication overhead can be reduced by 0.44%, 1.65%, 5.6%, and 12.80% with different discard factors.

According to the performance in

Table 3. Performance with discard factors.

Discard Factor		1%	2%	5%	10%
Collision Rate	Model	36%	39%	44%	42%
$r_{collision}$	Model+Rule	0%	0%	0%	0%
Average Jerk	Model	21.55	13.08	11.11	23.85
$J_{avg}$	Model+Rule	137.15	113.23	108.83	130.83
Average Velocity	Model	12.22	12.06	12.06	12.24
$v_{avg}$	Model+Rule	12.21	12.21	12.19	12.26

, the reduction of experience did not lead to performance degradation. In short, the proposed loss-aware experience selection strategy reduces the communication overhead while ensuring model performance.

6. Conclusions

A Traffic-Aware Federated Imitation Learning Framework for Motion Control was proposed to optimize motion control across multiple isolated unsignalized intersections. The framework contains vehicle interactors, edge trainers, and a cloud aggregator. Data privacy restriction is relaxed from vehicle interactors to edge trainers at intersections to balance privacy-preserving and motion control. The framework integrated a traffic-aware model aggregation for intersections with different traffic flows. Then, an IL algorithm was proposed for action cloning from a set of collision avoidance rules to improve the safety capability of end-to-end learning. Furthermore, this paper explored a loss-aware experience selection strategy to reduce the communication overhead through additional computation on interactors and trainers. The extensive experiment revealed that the proposed IL algorithm could achieve collision avoidance and improve the driving comfort. The TAFI-MC framework meets the privacy demand and further improves driving comfort. The proposed experience selection strategy can reduce the communication overhead while ensuring convergence.

Our future work will focus on the modeling and theoretical analysis of the relationship between the interactors and trainers in terms of communication and model training. Based on this analysis, we believe it will help accelerate model training while significantly reducing the communication overhead.