A Resilient Intelligent Traffic Signal Control Scheme for Accident Scenario at Intersections via Deep Reinforcement Learning

Abstract

Deep reinforcement learning methods have shown promising results in the development of adaptive traffic signal controllers. Accidents, weather conditions, or special events all have the potential to abruptly alter the traffic flow in real life. The traffic light must take immediate and appropriate action based on a reasonable understanding of the environment. In this way, traffic congestion would be prevented. In this paper, we develop a reliable controller for such a highly dynamic environment and investigate the resilience of these controllers to a variety of environmental disruptions, such as accidents. In this method, the agent is provided with a complete understanding of the environment by discretizing the intersection and modifying the state space. The proposed algorithm is independent of the location and time of accidents. If the location of the accident changes, the agent does not need to be retrained. The agent is trained using deep Q-learning and experience replay. The model is evaluated in the traffic microsimulator SUMO. The simulation results demonstrate that the proposed method is effective at shortening queues when there is disruption.

1. Introduction

Traffic congestion has numerous negative effects, including air pollution, extended travel times, and energy loss. Despite the fact that expanding roads may appear to be an easy way to reduce traffic congestion, the investment would be costly and ineffective in the long run. Therefore, optimizing traffic signal control would be the best option. This would make existing infrastructures work better [1].

1.1. Literature Review

Fixed timing and adaptive traffic light control are the two main approaches to controlling traffic lights. Instead of taking into account real-time traffic data, fixed timing control uses historical data to determine the timing of traffic lights [2]. However, weather conditions, accidents, and special events can have an impact on traffic demand, which is not always constant. Consequently, the productivity of traffic light control at fixed times is compromised when the traffic request is dynamic. Based on real-time traffic demand, adaptive traffic signal control (ATSC) can optimize the duration of green and red traffic signals using inductive loops to detect vehicles [3,4,5]. The well-known adaptive signal control systems SCOOT [6], SCATS [7], OPAC [8], and PRODYN [9] regulate signal timings by resolving optimization issues.

A model-based predictive controller was presented in [10], which aims to reduce traffic congestion by reducing the number of cars waiting at red lights. In addition to analytical models, simulation models can also be used to analyze traffic system responses under various conditions. Through similar models of traffic flow, these models can mimic real-world processes and obtain information. There are two types of traffic simulation models: microscopic and macroscopic. A microscopic simulation model takes into account the individual behavior of the driver as well as interactions with other vehicles and pedestrians, while a macroscopic simulation considers the entire traffic flow. There are many microsimulation packages used for studying urban traffic dynamics, including AIMSUN, MATSim, Paramics, SUMO, and VISSIM. The simulation models AIMSUN, VISSIM, and SUMO allow the user to create and control models with an external programming language via APIs. Traffic signal control uses a microsimulation model of the network to evaluate the current solution, and the results of each simulation are fed back into the proposed algorithm until a stopping criterion is reached [11]. There has been some research in the literature that analyzes the traffic signal control problem from a multiobjective perspective [12,13]. Ref. [12] used a fuzzy programming approach in which different weight coefficients were assigned to the different optimization objectives. After assigning the weights, the multiobjective function was transformed into a single objective. Ref. [14] formulated optimization problems for traffic signals as a mixed-integer linear programming problem and a nonlinear programming model, respectively. Using convex (quadratic) programming, [15] optimized traffic signal timings at an isolated intersection. In [16], a new algorithm for traffic control optimization was proposed using the number of vehicles in the queue. A comparison of this algorithm with GA and PSA algorithms has shown a 10 percent and 16 percent improvement in queue length reduction, respectively.

Artificial intelligence techniques have recently been applied to traffic signal control [17]. In this area, the reinforcement learning (RL) approach enables the adaptive control of the traffic signals, where the control problem is formulated as a Markov decision process (MDP) [18]. Reinforcement learning can model complex environments to derive optimal control actions by interacting with dynamic systems and observing changes in their behavior. In this approach, when an agent interacts with the environment, it tries to maximize the specified cumulative reward to determine the optimal state-action policy [19]. In reinforcement learning, function approximations are used to compute the value of each action in applications where the state space becomes larger [20].

Numerous studies have investigated the use of deep reinforcement learning techniques for traffic light control [21,22,23,24,25,26,27,28]. These techniques incorporate reinforcement learning and deep learning [29]. Deep learning can help in making optimal decisions in complex traffic environments where decision-making is challenging. This is because multiple layers in deep networks enable choosing the best approximations for decisions.

An agent formulation in RL that deals with the definition of state, action, and reward undoubtedly has implications for the performance of the algorithm. The reward is a numerical value sent by the environment when the agent performs an action. This value is interpreted as an objective function in optimization problems; thus, the total reward should be maximized throughout the entire optimization process. In traffic signal control, various factors are considered as reward functions, such as total queue length, waiting time, number of stops, and average speed, to name a few. In some studies, the weighted sum of multiple factors is considered as the reward [22,30,31,32]. It is possible to perform different types of actions, such as setting the duration of the current phase [24], maintaining the current phase or moving to the next phase, or selecting the next phase from a predetermined set [20,21]. A state is defined as the agent’s knowledge about its environment. Consequently, positions, average speed, queue length, waiting time, phase, and duration of phases at traffic lights are some of the elements of states considered in the literature [22,24,33,34,35]. Researchers proposed using image representations of intersections as states to better perceive the environment [24,36]. However, there are cases where complex states such as images do not lead to excellent performance and simple states are preferable [30]. Ref. [1] presented discrete traffic state encoding (DTSE), where each lane is divided into cells. Based on [1], ref. [35] proposed a deep reinforcement learning method for controlling traffic lights at a single intersection by discretizing the lanes. Each cell has a different length. In the state vector, the presence of a vehicle within the cell is indicated as 1 and the absence of a vehicle is indicated as 0. This state definition has resulted in the desired performance in low traffic volume scenarios. A reinforcement learning method was proposed in [37] for achieving maximum intersection throughput. Based on the total pressure and the total queue length at an intersection, [37] defines an adaptive reward function that uses an exponential approach. Recently, safety and accident analysis have become increasingly popular in traffic signal control [38,39,40,41]. An adaptive traffic signal control algorithm was proposed in [38] to maximize traffic efficiency and safety simultaneously. In [38], high-resolution real-time traffic data are used as the input to the agent, which selects the appropriate signal phase every second to reduce the vehicle delay and crash risk at the intersection. There is a discussion in [41] about the robustness of deep reinforcement learning-based controllers with respect to a variety of uncertainty scenarios.

1.2. Contribution

According to the literature review, since the trained agent (traffic light) in the traffic light control understands the environment through its state representation, environmental disruptions can have a negative impact on the training process. Some of the things that make the environment unpredictable and cause disruption include bad weather, accidents, the stopping of one or more vehicles, changes in the traffic density during special events, and other factors. As a result, the control algorithm is unquestionably more resilient when it has a better understanding of its surroundings. The following is a summary of the contributions made by our study:

• In this paper, we explore the resilience of a deep reinforcement learning algorithm on a single four-way intersection under accident conditions. In [35], a deep reinforcement learning approach to adaptive traffic light control is presented in which a vector of a vehicle’s presence or absence is considered as an observation of the environment. However, the proposed algorithm in [35] is not resilient under accident conditions, especially in a high traffic scenario. Hence, we modified the state space, so that three types of information will be used for the intersection instead of relying only on the presence of the vehicle as in [35]. The reason for this is that providing inputs that indicate a change in the environmental conditions associated with an accident gives the agent a more realistic perception of the environment. Therefore, we need input variables that accurately reflect changes in the environment. Since an accident causes cars to stop, the agent can better understand the current state of the intersection by adding the number of queued cars to that state. Consequently, this paper considers the presence or absence of cars and the number of queued cars as well as the current traffic light phase as the state. We discuss its utility for a resilient adaptive traffic management system.
• The proposed algorithm is independent of the location and time of the accident. To demonstrate this, several accidents are simulated at different locations and times in the environment to assess the effectiveness of the method. In this paper, we simulate an accident by halting a vehicle for a period of time on the lane. This can lead to traffic congestion.
• To the best of our knowledge, there are no studies on the occurrence of multiple accidents whose duration overlaps. In this paper, two different cars are halted at two different places, so their stopping times overlap. It can be observed that the algorithm converges and the corresponding control signal avoids a long queue of cars on the intersection.
• The proposed resilient control algorithm will be able to handle accidents regardless of where they occur. In other words, if the location of the accident in the training phase differs from the location of the accident during the testing phase, the agent does not need to be retrained to become resilient.
• The proposed method increases the convergence rate compared to the existing deep reinforcement learning algorithm in [35] as well as to the prior artwork. This means that the algorithm converges to the optimal control signal with fewer episodes during training.

1.3. Organization

Since it is very difficult to obtain a model of the environment with its associated uncertainties for the traffic control problem, a model-free learning algorithm is required. On the other hand, the problem defined in this study has a discrete action space, so the agent can determine the value of each action and choose the one with the highest value. Thus, a value-based algorithm can be used to learn. A simple learning algorithm with these properties that is capable of addressing the aforementioned problem is the Q-learning algorithm, a model-free reinforcement learning technique that is widely used in this field. Therefore, the control problem is modeled as deep reinforcement learning and the Q-learning algorithm is employed as the learning algorithm. In our model, a fully connected network approximates the Q-values for the actions. In addition, experience replay and target networks are used to avoid the divergence of the algorithm [33].

To evaluate the performance of the algorithm, a simulation of urban mobility (SUMO) is used [42]. This simulator can receive observations of intersections and apply the action to the agent through an application programming interface (API). Thus, a real-time traffic simulation is enabled. Even in high traffic situations, the evaluation results prove that our approach can effectively deal with the disruption of the environment.

The framework of the proposed methodology is summarized in Figure 1. The rest of this article is structured as follows: Section 2 presents the problem formulation. The details of the proposed model including the definition of the state, action, reward, and deep network are described in Section 3. In Section 4, the algorithm is evaluated by simulation and comparative results are presented to prove the effectiveness of the proposed method. Finally, in Section 5, the paper is concluded.

2. Problem Statement

2.1. Problem Assumption

In this paper, a single four-way intersection is considered as the environment simulated in the SUMO. The traffic light is utilized as an agent to control the traffic flow. The environment is observed by agents, who use this information to make decisions. The reward for the previously chosen action is also calculated using a measure of the current traffic situation. For the agent training, a sample of data is stored in the memory that contains information about the last simulation steps. Based on the reward received and the state of the environment, the agent now chooses a new action, and the simulation continues. The environment is shown in Figure 2. There are a total of four lanes on each road and each lane is 750 m long. In the right lane, drivers can turn right or go straight, in the two middle lanes, drivers can only go straight, whereas in the left lane, only a left turn is allowed. As shown in

Table 1. Signal phases.

	Link	N→S  S→N	N→E	E→W  W→E	E→S
Phase		N→W  S→E	S→W	E→N  W→S	W→N
Phase 1
Phase 2
Phase 3
Phase 4
Phase 5
Phase 6
Phase 7
Phase 8

, the traffic signal in this problem is configured to have four main phases, called green phases, and four auxiliary phases, called yellow phases. E, W, N, and S stand for the east, west, north, and south arms, respectively. The times for the green phase and the yellow phase are fixed at 10 and 4 s, respectively. When there are vehicles coming from multiple directions at an intersection, a single traffic signal may not be sufficient to control all the traffic. Therefore, multiple traffic lights must work together at an intersection. Traffic lights direct vehicles at such an intersection from nonconflicting directions simultaneously by changing the traffic light statuses [34].

In this paper, each arm is divided into four lanes, with the left lane controlled by its own traffic light, while the other three are controlled by a common traffic light. Consequently, signal control refers to the process of selecting an appropriate phase from predefined phases. Moreover, the disruption of the environment is considered as an accident. Despite the occurrence of accidents in the environment, our goal is to train the agent to maximize the traffic flow through the intersection so that the length of the queue appears shorter even in high traffic volume scenarios.

2.2. Optimization Goal

It is essential to keep in mind that the lane in which a car has crashed will be blocked in the event of an accident. As a result, the issue at hand is how to select the most effective phase from the set of predetermined phases for the traffic light in order to ensure that other vehicles can cross the intersection without waiting in line and that the intersection operates as efficiently as possible.

The principal objective of this work is to expand the traffic stream at the intersection and lessen the length of the queue that is shaped when a mishap happens. Let ${\mathrm{q}}_{\mathrm{l}}$ be the number of vehicles in the queue on lane $\mathrm{l}$ in a time step, and let ${\mathrm{Q}}_{\mathrm{m}}$ be the maximum length of the queue formed on all lanes of the intersection during the entire time period t. ${\mathrm{Q}}_{\mathrm{m}}$ has the following definition, where $\mathrm{L}$ is the set of all the lanes at the intersection:

$${\mathrm{Q}}_{\mathrm{m}}=\max {\displaystyle \sum }_{\mathrm{l}=1}^{\mathrm{L}}{\mathrm{q}}_{\mathrm{l}}$$

(1)

Our goal is to shorten the maximum length of a traffic accident queue in all lanes, especially during rush hour. As a result, this paper’s optimization problem is to reduce the maximum length of queues on all lanes formed during the time period t.

2.3. Methodology: Background on Deep Reinforcement Learning

Figure 3 depicts the standard cycle of reinforcement learning.

At the beginning of time step t, the agent observes the intersection state ${\mathrm{s}}_{\mathrm{t}}$ based on its interaction with the environment. After observing this state, an action ${\mathrm{a}}_{\mathrm{t}}$ is taken and the traffic signal is actuated by the agent. Due to the movement of the vehicles, the state of the environment changes to ${\mathrm{s}}_{\mathrm{t}+\mathrm{1}}$ and the agent receives a reward signal ${\mathrm{r}}_{\mathrm{t}+\mathrm{1}}$ at the end of the time step. Based on a performance measure, a reward signal informs the agent about the appropriateness of the action performed [33].

A reinforcement learning strategy is about learning an optimal policy that maximizes the cumulative expected rewards based on the current state. If the agent knows the optimal Q-values of successive states, the optimal action selection policy is simply to choose the action that yields the highest cumulative reward. The optimal Q-values $\mathrm{Q}*\left(\mathrm{s}, \mathrm{a}\right)$ are obtained by the recursive equation known as the Bellman optimality equation [20]:

$$\mathrm{Q}*\left(\mathrm{s}, \mathrm{a}\right){=\mathrm{E}\{\mathrm{R}}_{\mathrm{t}}{+\mathsf{\gamma }\max }_{\acute{\mathrm{a}}}\mathrm{Q}*\left({\mathrm{S}}_{\mathrm{t}+1}, \acute{\mathrm{a}}\right)|{\mathrm{S}}_{\mathrm{t}}{=\mathrm{s}, \mathrm{A}}_{\mathrm{t} }=\mathrm{a}\} \mathrm{for} \mathrm{all} \mathrm{s} \in \mathrm{S}, \mathrm{a} \in \mathrm{A}$$

(2)

Based on (3), the agent’s optimal cumulative future reward consists of the immediate reward it receives after choosing action $\mathrm{a}$ in state $\mathrm{s}$ plus the optimal future reward thereafter. The expression $\mathrm{m}\mathrm{a}{\mathrm{x}}_{\acute{\mathrm{a}}}$ means that the most valuable of all possible actions is chosen in state ${\mathrm{S}}_{\mathrm{t}+1}$.

It is worth noting that the solution of (3) requires that the states are finite and the transition probabilities are known. However, in complex traffic environments, there are numerous states, so it is extremely difficult to calculate the Q-value for each state-action pair. In this paper, traffic signal control is formulated as a deep reinforcement learning problem, as illustrated in Figure 4.

To find the optimal signal control policy, a mechanism called deep Q-learning is used. In this approach, the Equation (3) is not solved directly, but a parameterized deep neural network (DNN) is used to approximate the optimal Q-values $\mathrm{Q}*\left(\mathrm{s},\mathrm{a}\right)$ by the Q-learning function.

Through the following function, the Q-value is updated.

$$\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}}{, \mathrm{a}}_{\mathrm{t}}\right)=\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}}{, \mathrm{a}}_{\mathrm{t}}\right){ + \mathsf{\alpha }\left(\right(\mathrm{r}}_{\mathrm{t}+1}{+\mathsf{\gamma }\max }_{\mathrm{a}}\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}+1}{, \mathrm{a}}_{\mathrm{t}+1}\right) - \mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}}{, \mathrm{a}}_{\mathrm{t}}\right))$$

(3)

Here, $\left({\mathrm{s}}_{\mathrm{t}}{, \mathrm{a}}_{\mathrm{t}}\right)$ and $\left({\mathrm{s}}_{\mathrm{t}+1}{, \mathrm{a}}_{\mathrm{t}+1}\right)$ represent the current and next state-action pairs, respectively.$\mathsf{\alpha }$ is the learning rate and the reward ${\mathrm{r}}_{\mathrm{t}+1}$ is the reward that results from performing action ${\mathrm{a}}_{\mathrm{t}}$ in state ${\mathrm{s}}_{\mathrm{t}}$. $\gamma$ is the discount factor $0<\mathsf{\gamma }<1,$ which indicates how much importance the agent assigns to future rewards. When $\mathsf{\gamma }$ = 0, the agent considers only immediate rewards, while $\mathsf{\gamma }$ = 1 means that the agent considers future rewards more. After training the agent, the optimal action ${\mathrm{a}}_{\mathrm{t}}$ to take in state ${\mathrm{s}}_{\mathrm{t}}$ is the one that maximizes $\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)$.

3. Proposed Model Construction

Using the reinforcement learning approach, states, actions, and rewards should be defined when developing a traffic light control system. The following is a description of our model’s three components.

3.1. State Definition

The state must provide sufficient information about the intersection, particularly the distribution of vehicles on each arm, for the agent to effectively optimize the traffic flow. Figure 5 provides an illustration of the road’s proposed discretization in [35]. In this work, we use additional information to define the state in the same way.

As shown in Figure 5, a cell that is closer to a traffic light is smaller than one that is farther away, since an accurate perception of the distribution of vehicles near traffic lights is more important.

It is worth noting that the trained agents should be equipped with sufficient knowledge of the environment. On the one hand, the presence or absence of a vehicle may not provide sufficient information about the environment. On the other hand, accidents affect the number of vehicles in the queue. To make the algorithm more resilient, this information should be given to the agent as an observation of the environment. It should be noted that, in our method, each intersection arm consists of 20 cells with different sizes (Figure 5), so the intersection contains 80 cells in total. Each cell represents one element in the state vector. This means that there are 80 elements in the vectors E and Q.

Adding the 80 numbers indicating the presence or absence of vehicles in each cell with the number of vehicles in the queue in each cell gives 160 as the vector dimension. On the other hand, the traffic light phase is also a 4-dimensional vector added to the state vector. As a result, the state vector used for training and testing contains 164 elements ((20 × 4 × 2) + 4 = 164). This means that the dimension of the state vector is 160 × 1.

As a result, a vector $\mathrm{E}={\left[1,1,0,1,0,\dots ,1\right]}^{\mathrm{T}}$ is used to represent the presence or absence of vehicles in each cell, such that this vector indicates the presence or absence of vehicles as 1 and 0, respectively. T denotes the transpose operator. The vector $\mathrm{Q}={\left[5,3,0,2,2,\dots ,1\right]}^{\mathrm{T}}$ (the numbers are given as examples) is also used to indicate the vehicles in the queue in each cell. Moreover, the current green phase of the traffic light is encoded as a four-dimensional one-hot vector $\mathrm{P}={\left[1,0,0,0\right]}^{\mathrm{T}}$ since there are four possible green phases for the agent. Once the green light is turned on for a phase, the corresponding entry in the vector is set to 1.

Consequently, in this paper, the vector of the presence or absence of vehicles along with the vector of the number of queued cars in each cell, and the current traffic light phase, are concatenated as the state vector $\mathrm{S}=\left[{\mathrm{E}}^{\mathrm{T}},{\mathrm{Q}}^{\mathrm{T}},{\mathrm{P}}^{\mathrm{T}}\right]{ }^{\mathrm{T}}.$

3.2. Action

Depending on the current state of the intersection, the agent must perform the appropriate action. The action space of this paper is defined by choosing the green phase from a predefined set of green phases such as: {NSA, NSLA, EWA, EWLA}. NSA or north-south advance means that the green phase applies to vehicles in the northern and southern arms that want to go straight or turn right. Similarly, EWA means that the green phase applies to vehicles going straight or wanting to turn right in the westbound or eastbound arms. NSLA means that the green phase is active for vehicles turning left in the north and south arms. The same is true for vehicles on the right and left arms wishing to turn left in EWLA. As mentioned earlier, the green phase time and the yellow phase time correspond to 10 s and 4 s, respectively. If the action in step t is different from the action in step t−1, a yellow phase of 4 s is activated between the two actions. Otherwise, there is no yellow phase, i.e., the current green phase continues [35].

3.3. Reward

The reward should be defined based on a measure of traffic efficiency. Since the main objective of this work is to reduce the length of the queue caused by an accident, the waiting time, which is equal to the length of the queue, is considered as the performance measure. Therefore, the reward is defined as the reduction in the cumulative waiting time between two consecutive actions. The reward function is represented as follows:

$${\mathrm{r}}_{\mathrm{t}}{=\mathrm{wt}}_{\mathrm{t}- 1}{ – \mathrm{wt}}_{\mathrm{t}}$$

(4)

in which:

$${\mathrm{wt}}_{\mathrm{t}}={\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{w}}_{\mathrm{I},\mathrm{t}}$$

(5)

The waiting time of vehicle $\mathrm{i}$ at time step t is denoted by ${\mathrm{w}}_{\mathrm{i},\mathrm{t}}$. Here, the waiting time refers to the time in seconds that a vehicle has a speed of less than 0.1 m/s. $\mathrm{N}$ is the total number of vehicles in time step t. The reward function is such that its positive value represents a successful action by the agent. Choosing the right action results in fewer vehicles in the queue at the current time step $\mathrm{t}$ than at the previous time step $\mathrm{t}-1$, and the waiting time for the vehicles is also reduced. In this way, the reward grows over time.

3.4. Deep Neural Network Structure

In this paper, a fully connected network is used to estimate the Q-values $\mathrm{Q}\left(\mathrm{s},\mathrm{a},\mathsf{\theta }\right)$ for all actions $\mathrm{a}\in \mathrm{A}$ under the observed state $\mathrm{S}$. A visual representation of the deep neural network is shown in Figure 6. Intersection state ${\mathrm{s}}_{\mathrm{t}}$ is the input to the network and a vector of the estimated value $\mathrm{Q}\left(\mathrm{s},\mathrm{a},\mathsf{\theta }\right)$ is generated as the output of the network. $\mathsf{\theta }$ is the parameter of the network. The parameters should be learned based on traffic data so that $\mathrm{Q}\left(\mathrm{s},\mathrm{a},\mathsf{\theta }\right)$ approaches the optimal value $\mathrm{Q}*\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)$.

Obviously, the state vector observed in the environment is the input of the deep neural network. Therefore, the number of neurons in the input vector must equal the number of elements in the state vector. As explained in Section 3.1, the dimensions of the state vector are 164 × 1, which corresponds to the discretization of the intersection arms. As a result, an input layer with 164 neurons is required for this kind of deep neural network. Accordingly, the input layer consists of 164 neurons. In addition, this network contains 5 hidden layers with 400 neurons with a rectifier nonlinearity activation function (ReLU) and an output layer consisting of 4 neurons with a linear activation function. This activation function is most commonly used in hidden layers. By using this function, the problem of vanishing gradients is prevented. Due to its fixed derivative (slope) for one linear component and zero derivative for the other, the learning process using the ReLU function is much faster. Moreover, as the ReLU does not contain exponential terms, the calculations are much faster [43]. The neurons of the output layer indicate the Q-value of an action with respect to a state.

3.5. Model Training

To train the deep network, the techniques of experience replay and target Q-network configuration are used. In these techniques, observations are grouped into batches that are used to train the agent. A batch consists of experiences or samples, where $\mathrm{e}=\left\{{\mathrm{s}}_{\mathrm{t}}, {\mathrm{a}}_{\mathrm{t}}, {\mathrm{r}}_{\mathrm{t}+1},{\mathrm{s}}_{\mathrm{t}+1}\right\}$. After performing an action in state ${\mathrm{s}}_{\mathrm{t}}$, ${\mathrm{r}}_{\mathrm{t}+1}$ is the reward received when the environment evolves to state ${\mathrm{s}}_{\mathrm{t}+1}$. Through experience replay, correlations between samples are eliminated and current Q-network parameters will be copied to the target network using target Q-network configuration. In each training instance, to learn the deep neural network parameters, the agent requires training data including the inputs $\mathrm{e}=\left\{{\mathrm{s}}_{\mathrm{t}}, {\mathrm{a}}_{\mathrm{t}}, {\mathrm{r}}_{\mathrm{t}+1},{\mathrm{s}}_{\mathrm{t}+1}\right\}$ and outputs of the target Q-values $\mathrm{Q}*\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right).$ The input data for the training are randomly retrieved from the replay memory (see Figure 7) and each sample in each batch is used for the training. The target Q-values $\mathrm{Q}*\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)$ are estimated as:

$$\mathrm{Q}*\left({\mathrm{s}}_{\mathrm{t}}{, \mathrm{a}}_{\mathrm{t}}\right){=\mathrm{r}}_{\mathrm{t}+1}{+\mathsf{\gamma }\max }_{{\mathrm{a}}_{\mathrm{t}+1}}\acute{\mathrm{Q}}\left({\mathrm{s}}_{\mathrm{t}+1}{, \mathrm{a}}_{\mathrm{t}+1}, \mathsf{\theta }\right)$$

(6)

where $\acute{\mathrm{Q}}\left({\mathrm{s}}_{\mathrm{t}+1},{\mathrm{a}}_{\mathrm{t}\mathrm{+}1},\mathsf{\theta }\right)$ is the output of a separate target network with a similar structure to the main deep neural network and denotes the Q-value associated with a subsequent action after the action ${\mathrm{a}}_{\mathrm{t}}$ is performed in state ${\mathrm{s}}_{\mathrm{t}}$.

To update the estimated Q-values, (7) uses the information available in each sample. For each training instance, the number of samples retrieved from the memory forms the batch size, which in this case is 100. The replay memory is used to store these batches and then apply them to the agent at different time steps. The memory has a finite capacity, indicating the number of samples it can store, which in this work is set to 5000. Old samples are discarded when the memory is full.

The network parameters should be learned in a way that minimizes the following mean square error (MSE) function of the Q-values as loss function. A stochastic gradient descent method, which performs a parameter update for each training example, is used to minimize the mean square error.

$$\mathrm{loss} \mathrm{function}=\frac{1}{\mathrm{m}}{\displaystyle \sum }_{\mathrm{t}=1}^{\mathrm{m}}\{\left({\mathrm{r}}_{\mathrm{t}+1}{+\mathsf{\gamma }\max }_{{\mathrm{a}}_{\mathrm{t}+1}}\acute{\mathrm{Q}}\left({\mathrm{s}}_{\mathrm{t}+1}{, \mathrm{a}}_{\mathrm{t}+1}, \mathsf{\theta }\right)\right) - \mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}}{, \mathrm{a}}_{\mathrm{t}}, \mathsf{\theta }\right)\}$$

(7)

in which the input data size is m.

After training, the agent is expected to be able to estimate the optimal Q-values and learn the optimal action selection policy. Since the agent has only experienced a limited number of states and not the entire state space, the Q-values for the states not experienced may not be as accurate as they could be. In addition, the state space itself may be constantly changing, leading to inefficiencies in the previously estimated Q-values.

Therefore, the agent should decide whether to use the Q-values it has already learned and select the action with the highest Q-value (exploitation) or explore other possible actions to improve the estimation of the Q-values and eventually optimize the action policy. In this paper, the $\mathsf{\epsilon }-\mathrm{greedy}$ method is used as the action policy. In this method, a probability $\mathsf{\epsilon }$ is defined such that the agent takes an exploratory action with probability $\mathsf{\epsilon }$ and chooses the exploitative action with the highest estimated Q-values with probability 1 −$\mathsf{\epsilon }$. Note that, at the beginning of the training, $\mathsf{\epsilon }=1,$ which means that the agent performs only exploratory actions. After a certain period of training, the exploratory operation starts. The formulation of the $\mathsf{\epsilon }-\mathrm{greedy}$ is presented below:

$$\mathsf{\epsilon }=1 - \frac{\mathrm{current} \mathrm{episode}}{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{episodes}}$$

(8)

4. Evaluation Results

In this section, we describe the simulation results to evaluate the performance of the proposed approach. We perform the evaluations in the SUMO simulator where real-time traffic is simulated. Figure 8 shows the studied environment for the evaluation along with the simulated data on the SUMO platform. Our analysis focuses on a single intersection to understand the resilience of deep RLs. Different network configurations and other input distributions will be explored in a future work. Traffic congestion can be the result of accidents that disrupt the traffic system. Different conditions and events such as accidents should be considered when training the agent so that the controllers learn to be resilient in uncertain environments. To create a realistic situation, in this paper, we simulate a lane closure caused by an unpredictable accident by stopping a southbound vehicle. During each episode, the vehicle stops until the simulation time expires. Thus, the control signal should be adjusted so that a long queue does not form on the road where the accident occurred.

During the simulation, we try to maximize the defined reward, i.e., minimize the cumulative delay of all vehicles, and minimize the length of the vehicle queue in case of an accident.

4.1. Traffic Generation Process

Several probability distributions have been proposed in the literature that can be used to model a real traffic flow [44]. To move closer to reality, we refer to [35], where the travel demand is based on the Weibull distribution. Figure 9 shows the traffic generated in an episode with the number of vehicles at each step of the simulation. This distribution clearly represents the traffic flow during a whole day. Since the number of vehicles increases in the initial phase, it is a peak hour. Over time, the number of arriving cars decreases, describing the gradual weakening of the traffic congestion.

Inspired by the shape of the Weibull distribution (Figure 9), the way to apply this distribution to the traffic flow is that a small number of vehicles initially enter the environment and this number gradually increases, indicating the peak hours of the traffic. Then, the number of vehicles entering the environment gradually decreases, indicating the end hours of the day. It should be noted that, in each episode, a random generator function is used to randomly determine the vehicle’s origin and destination.

It should be noted that the traffic flows at different intersections may vary significantly in reality. Therefore, the results presented here represent what the performance would be for a typical all-day traffic volume. The authors intend to evaluate the proposed algorithm under different distributions for generating traffic with short-term fluctuations in a future work.

4.2. Setup for Microscopic Traffic Simulator

The impact of the accident is studied in two different demand scenarios with different traffic volumes: high traffic volume scenario and low traffic volume scenario. In the low traffic volume scenario, 600 vehicles approach the intersection, while in the high traffic volume scenario, 3000 vehicles approach the intersection. In both scenarios, 75% of the vehicles travel straight through the intersection and 25% turn left or right [35].

The agent’s training lasts 50 episodes. Each episode consists of 5400 steps and the time frequency provided by the SUMO is 1 s per step. The parameters of the algorithm are listed in

Table 2. Parameters for reinforcement learning method.

Parameter	Value
Discount factor $\mathsf{\gamma }$	0.75
Learning rate	0.001
Replay memory size	5000
Batch size	100
Starting $\mathsf{\epsilon }$	1

.

4.3. Model Validation Using Different Travel Demand Scenarios

The resilience of the RL approach is analyzed in both the high and low traffic scenarios. The performance of the trained agent is studied from two perspectives: the reward curve during the training and the queue formed during an accident, according to the waiting time or the dwell time of the vehicles. Moreover, the proposed RL approach is compared with the performance of [35] in terms of the resilience to environmental uncertainties, such as an accident.

Accident simulation: To study the effects of an accident, we try to simulate a route closure caused by an accident in the simulator SUMO. To accomplish this, one of the cars approaching the traffic light stops before the intersection in one of the lanes. The accident can happen at any place in the area and at any time. In such a scenario, the agent gains access to the accident during the training.

First, we stop one of the cars in one of the southbound lanes of the intersection 50 m before the intersection. Figure 10 shows the reward curve during the training based on the cumulative negative reward in a low traffic scenario, and Figure 11 shows this curve in a high traffic scenario.

The performance of the agent is indicated by these curves. It can be observed that the agent is well trained in both scenarios and the cumulative reward increases as the training progresses. Moreover, the reward curve shows that a stable policy has been learned. This means that there is no oscillation between choosing bad and good actions and the algorithm does not deviate to bad decisions. Moreover, the presented approach achieves a fast convergence to an appropriate policy. These results show the resilience of the proposed algorithm to possible accidents and, thus, reduce traffic congestion.

Other metrics to evaluate an agent’s performance are the average queue length and the cumulative delay during learning, as shown in Figure 12 and Figure 13, respectively, for the high traffic scenario. The vehicle delay is defined as the time the vehicle is steady between two consecutive actions. As the training progresses, the queue length and the delay decrease.

At the beginning of the learning process, the agent only explores the environment and performs its actions randomly. Due to these random actions, many vehicles are queued, resulting in high delays in the early stages of learning. The learning episodes progress and the agent starts to choose exploitative actions rather than exploratory actions due to a better understanding of the action-value function. In this way, the agent makes its decisions in a more optimal way, reducing the cumulative delay and average queue length.

4.4. A Comparison Based on Performance Measure

To show the efficiency of the proposed model in reducing the queue length in the presence of accidents, a comparison is made between the proposed approach [35,41]. In [35], the traffic control problem is handled without disruption in the environment, and the agent is trained without an accident occurring. Furthermore, the agent perceives the environment only by a vector, including whether a vehicle is present or not, and it takes actions depending on this. To evaluate the algorithm presented in [35], stopping a vehicle causes an accident in the environment in a high traffic scenario, and then the controller is trained on the new conditions.

There have been few studies that examine the impact of an accident in a single intersection, but the closest study to our paper is [41], which investigates the robustness of a deep RL-based controller in the presence of disruption, such as accidents. Similarly, the learning algorithm in this article is also DQN, and uses a fully connected network as the approximation fraction. We use the state vector from [41] as the input to our algorithm and train the agent with these new inputs to compare the performances of our algorithm and the algorithm presented in [41].

Figure 14 illustrates the maximum queue length during the test result. The longest queue is formed during the peak traffic hours. As can be seen, the presented approach reduces the maximum queue length by 15% compared to Conventional Deep RL method in [35] and by about 9% compared to Deep RL method in [41]. Based on this performance metric, we can conclude that the proposed algorithm has developed a superior control policy.

Moreover, the cumulative reward of [35] during the training episodes in the presence of an accident is presented in Figure 15. The observed fluctuations show the tendency of instability of the learned policy and, hence, it takes much more time to reach convergence compared to the proposed method. This is because a lower understanding of the environment degrades the performance of the algorithm when there is disruption in the environment.

Table 3. Overview of average value of metrics for agents during training for accident in southern arm.

	Proposed Method	Conventional Deep RL Method in [35]
Average cumulative negative reward	−152,050	−190,530
Average cumulative delay (seconds)	431,000	476,070
Average queue length (vehicles)	79	88

shows the numerical comparison between the two agents in [35] and the proposed method during the training in the presence of an accident. The average values of the cumulative negative rewards, cumulative delays, and queue length were used as the performance measures. These values are summed over 50 episodes and then averaged. The proposed resilient traffic signal control method outperforms the conventional deep RL method in [35] in all performance metrics. In comparison with [35], the average time delay and average queue length have both decreased by 10%, and the average cumulative reward has increased by 20%. The authors will test the algorithm on a real dataset and evaluate the error rate of the actual queue length compared to the queue length determined in the simulation.

4.5. Changing the Accident Location

Since accidents can happen at any place, in this section we change the location of the accident and stop a car at three different locations in the northern, western, and eastern arms of the system to prove the effectiveness of the proposed controller. Figure 16 and Figure 17 show the cumulative reward curve and the average queue length curve, respectively, for a high traffic scenario when the accident occurs in the western arm 50 m from the intersection. The reward curve converges after a few episodes, demonstrating the stability of the proposed policy. During the learning process, the agent also reduced the length of the vehicle queues, indicating that there is no traffic congestion.

In order to change the location of the accident, we stopped a car approaching a traffic light 30 m before the intersection in the eastern arm. According to Figure 18 and Figure 19, the algorithm is able to reduce the queue length and increase the cumulative rewards during the training. It seems that the closer the accident is to the intersection, the more episodes the agent needs to learn for the proposed strategy to converge.

After that, we moved the accident location 30 m from the intersection in the northern arm and verified that the proposed method of traffic control was effective in this case. Figure 20 and Figure 21 show the training results in this case, again showing a decreasing queue length and increasing cumulative reward.

The observed trend confirms that the proposed traffic control method is efficient no matter when or where the accident occurs. The control signal prevents a long queue of vehicles, regardless of the location of the accident. The numerical results of the performance evaluation of the two agents in the proposed method and [35] when training in the presence of an accident at different locations are presented in

Table 4. Overview of average value of metrics for agents during training by changing accident location.

	Accident Location	Proposed Method	Conventional Deep RL Method in [35]
Average cumulative negative reward	W	−176,950	−213,280
	E	−200,550	−279,030
	N	−174,660	−216,240
Average cumulative delay (seconds)	W	506,200	506,800
	E	538,780	629,350
	N	492,340	530,110
Average queue length (vehicles)	W	92	95
	E	99	117
	N	91	99

. N, E, and W refer to the north, east, and west arms of the intersection, respectively. By comparing the performance metrics in the table, we can conclude that the proposed method is generally more effective. In comparison with the conventional method in [35], it has increased the rewards by an average of 22% while reducing the queue length and delays by about 10%.

In another experiment, we will investigate the performance of the proposed algorithm in a situation where an accident occurred in the intersection or just after it and very close to it. In order to accomplish this, we stop a vehicle moving away from the intersection in the northern arm for a period of time. The location of the stop is just 5 m after the intersection. In fact, it can be said that this accident happened almost inside the intersection. The training results are presented in Figure 22 and Figure 23 and the queue length curve during the test is shown in Figure 24. The convergence of the reward curve with little fluctuation confirms the stability of the learned policy, and the reduction in the queue length curve during the training shows the desired performance of the proposed method in reducing the traffic.

The queue length curve in the test phase (Figure 24) shows that the highest queue length occurred during the peak hours, but then, the queue gradually reduced. Therefore, the designed controller is reasonably capable of handling the accidents within the intersection.

4.6. Changing the Accident Time

In this experiment, we change the time of the occurring accident and stop a vehicle at a later time than in the previous experiments. The location of the stop is 50 m from the intersection in the western arm approaching the intersection. We have implemented this experiment in a high traffic scenario. Moreover, in this case, the vehicle is stopped only for a period of time. This means that the accident is temporary and the accident does not occur during the entire training period.

Figure 25 shows the reward curve and Figure 26 shows the average queue length during the training. As you can see, the cumulative reward increases with the training time and then converges. Therefore, the proposed algorithm can achieve stability regardless of the time of the accident. During the learning process, the length of the vehicle queues also decreases, indicating the evacuation of the traffic.

There is also an experiment where the time and location of the accident are changed, and the accident occurs 350 m from the intersection. The vehicle is stopped only for a certain time. In other words, the accident is temporary. This case evaluates the algorithm’s performance when the accident occurs far from the intersection. The cumulative reward and average queue length are illustrated in Figure 27 and Figure 28, respectively. These curves show that the performance of the algorithm is independent of the time and location of the accident. Over time, the reward curve has converged well, and the queue length has decreased.

4.7. Multiple Simultaneous Accidents

As a test of the proposed method in a more complex environment, we simulated two accidents in this section by stopping two cars at two different locations so that the duration of their stops overlapped. One car stopped 700 m before the intersection in the southern arm and another stopped 730 m before the intersection in the northern arm. It should be noted that the starting time of the stopping cars is not the same, but their stopping periods overlap.

As shown in Figure 29 and Figure 30, the accumulative reward and average queue length curves illustrate the stability and success of the proposed strategy even in the presence of two accidents at different locations.

Lastly, we compare the maximum queue length of the proposed method with that of the conventional Deep RL method in [35] to further evaluate the proposed method. The proposed method reduces the maximum queue length at the intersection by about 24% as shown in Figure 31.

4.8. Model Validation with Different Accidents in Training and Testing

In this section, we perform some experiments where the accident location in the training phase differs from the accident location in the testing phase. In this way, the agent is not retrained under new conditions and, thus, the resilience of the algorithm will be investigated. We consider three scenarios and apply our proposed algorithm to them. These scenarios were tested in high traffic volumes. To verify the effectiveness of our approach, the defined scenarios are also applied to [35,41], and our proposed algorithm− is compared to these works in terms of the maximum queue length formed during the testing phase.

• First scenario

In the first scenario, we train the agent for the case where an accident occurs in the southbound arm at a distance of 350 m from the intersection for a period of time. A trained agent is then tested in an environment where an accident has occurred at a distance of 550 m from the intersection in the eastbound arm which is almost far from the traffic light. The queue length curve during the test result is shown in Figure 32. In this high-volume traffic flow, it is observed that the maximum queue length is about 122 vehicles during the peak hours and then, it is well emptied. During this evaluation, the proposed algorithm proved resilient, and the agent optimized the control signal without having to be retrained for the new accident conditions.

• Second scenario

In the second scenario, the agent is trained with an accident occurring in the westbound arm at a distance of 30 m from the intersection, throughout the whole training period. We then tested this agent at a 50-m distance from the southern arm of the intersection where an accident occurred.

Table 5. Maximum queue length during the test result without retraining in second scenario.

	Proposed Method	Conventional Deep RL Method in [35]	[41]
Maximum queue length during test phase (vehicles)	140	153	158

shows the maximum queue length for our method compared with [35,41]. It is demonstrated that the proposed control algorithm is resilient to accidents in the environment and can eliminate their effects without retraining. There is a gradual dissipation of the queue formed behind the light and there is a maximum of 140 cars in the peak hours. Based on the results of the presented approach, the maximum queue length can be reduced by 8% compared to [35] and by about 11% compared to [41].

• Third scenario

An accident occurred 50 m from the intersection in the third scenario, during the training. The agent is then put to the test, assuming the accident occurred on the eastbound arm and 30 m from the intersection. A numerical comparison of the agents in [35,41] and the proposed method is shown in

Table 6. Maximum queue length during the test result without retraining in third scenario.

	Proposed Method	Conventional Deep RL Method in [35]	[41]
Maximum queue length during test phase (vehicles)	186	261	243

. Comparing the presented approach to [35,41], we observe that the maximum queue length is reduced by 23% and 28%, respectively.

5. Discussion

In order to implement such control algorithms, adaptive traffic signal controllers can be used, which can learn the control signals from the data from a variety of sources, such as traffic detection sensors. A traffic signal that is adaptive adjusts according to the actual traffic demand. With the development of embedded systems technology, electronic boards, software packages, and closed-loop control systems, the implementation of such algorithms is now possible. Therefore, these adaptive control algorithms can be implemented with a combination of hardware and software [45].

Our proposed model works both in computer control systems and as an additional application. Considering that the testing phase of the algorithm does not require a high volume of processing, the proposed algorithm can be implemented both as an application in computer control systems and as a processor as part of an isolated signaling controller. The distinctive feature of the proposed system is that it can be fully implemented on an embedded platform by a single-board computer and a core as a computing unit.

Considering the concerns, the proposed algorithm can be extended to multiple intersections. However, currently, this study is limited to one intersection and the proposed model has been trained for one intersection. To achieve an optimal policy for all intersections, we need to use cooperative learning methods. The development of the proposed algorithm for application in environments with multiple intersections is one of the future works of the authors. Another limitation is that, in the action, the duration of the green light is not dynamically adjusted. Rather, the action selects the green phase from a predefined set. It is also planned to dynamically adjust the green time of the traffic light in the future. Moreover, considering a specific distribution for traffic flow generation can be one of the limitations of the proposed algorithm. Of course, it should be mentioned that the choice of the Weibull distribution for generating the traffic is based on the traffic of a typical day, which includes rush hours.

The data gathered for this study cover a wide range of traffic flow patterns, so it is as close as possible to actual conditions. The vehicle’s origin and destination are selected at random in each episode using a random generator function. High and low traffic scenarios are taken into consideration, with 25% of vehicles turning left or right at the intersection and 75% going straight. A traffic detector is a basic piece of any intelligent transportation framework. They can collect information about the traffic conditions, such as volumes, speeds, occupancy, and travel times, for better traffic management. There are two basic kinds of traffic detectors.

One type is called detectors on the roadway, such as induction loops, magnetometers, and built-in temperature sensors. The other type of traffic detectors is over-roadway detectors, such as video cameras, radar, Bluetooth/Wi-Fi, and weather sensors. Due to the fact that we have generated the required data through simulation, any sensor that can process and provide this information can be used. For example, this data can be obtained by sensing devices such as high-resolution cameras and image processing systems. The captured images should be processed in real time with an image processing toolkit. The image processing software can capture the traffic images from a camera, detect and count the moving vehicles, estimate the traffic density, and control the traffic signals according to the processed results [46]. On the other hand, traffic data collection often requires an effective method of calibrating surveillance cameras efficiently and accurately. Camera calibration is a necessary step to determine the real positions of the vehicles that appear in the video. For the calibration of the detection devices, many methods have been proposed in numerous articles to achieve the desired result [47,48].

6. Conclusions

A deep reinforcement learning framework is proposed in this paper to improve the efficiency of the traffic flow in the presence of accidents and fluctuating traffic demand. The accident is simulated by halting a vehicle approaching a traffic light in one lane. To provide the agent with enough information about the environment, each road is discretized into cells of different sizes based on [35]. To make the deep RL controller more resilient to possible uncertainties, the extended state representation was used while training the agent in the presence of an accident. The state contains three types of information: a vector indicating the presence or absence of vehicles, the number of vehicles in the queue in each cell, and the traffic phase. The work uses the SUMO, a realistic traffic simulator, to provide an environment for training and evaluating the RL agents. The DQN is used as a learning algorithm and a fully connected network approximates the Q-values for the actions. The simulation results demonstrate that, regardless of the time, location, or number of accidents, the algorithm learned a stable strategy. In addition, when the accident location changes, the proposed resilient control algorithm does not need to be retrained. Thus, the control calculation can deal with mishaps that happen at any place without retraining. In order to verify the effectiveness of our main contributions, we compare the proposed method to [35,41]. There was a significant reduction in the queue length compared to [35,41] when the accidents occur, so that no vehicles had to wait for long periods of time to cross the intersection. The convergence speed is also better with the proposed model than with [35].

In complex networks, the concepts of optimizing hyperparameters and locating critical nodes based on centrality will be pursued in the future. Our next phase of research could look into how the proposed algorithm for autonomous vehicles evolved given the significance of the accident occurrence and the resilience of adaptive traffic control algorithms in the discussion of automated vehicles.