Decision Making for Self-Driving Vehicles in Unexpected Environments Using Efficient Reinforcement Learning Methods

Abstract

Deep reinforcement learning (DRL) enables autonomous vehicles to perform complex decision making using neural networks. However, previous DRL networks only output decisions, so there is no way to determine whether the decision is proper. Reinforcement learning agents may continue to produce wrong decisions in unexpected environments not encountered during the learning process. In particular, one wrong decision can lead to an accident in autonomous driving. Therefore, it is necessary to indicate whether the action is a reasonable decision. As one such method, uncertainty can inform whether the agent’s decision is appropriate for practical application where safety must be guaranteed. Therefore, this paper provides uncertainty in the decision by proposing DeepSet-Q with Gaussian mixture (DwGM-Q), which converges the existing DeepSet-Q and mixture density network (MDN). Calculating uncertainty with the Gaussian mixture model (GMM) produced from MDN made it possible to calculate faster than the existing ensemble method. Moreover, it verified how the agent responds to the unlearned situation through the Simulation of Urban MObility (SUMO) simulator and compared the uncertainty of the decision between the learned and nontrained situation.

1. Introduction

Autonomous driving on highways is divided mainly into longitudinal and lateral decisions [1]. The longitudinal decision sets the vehicle’s longitudinal target speed, while the lateral decision determines whether to change the lane. Furthermore, the appropriate target speed is necessary to contribute to the driver’s safety by maintaining a certain distance from the vehicle in front. The lane-changing decision is a crucial action that allows autonomous vehicles to reach their destination quickly and is, also, an action for overtaking, obstacle avoidance, cut-in/cut-out, etc. Therefore, the autonomous vehicle can be efficiently operated by safely driving and arriving at the destination in the shortest time when the longitudinal and lateral decisions are harmonized.

Decision-making problems in autonomous driving applied rule-based methods such as Intelligent Driver Model (IDM) [2] and Minimizing Overall Braking Induced by Lane Changes (MOBIL) [3]. In addition, these methods have been successfully used in the Defense Advanced Research Projects Agency (DARPA) Urban Challenge (DUC) [4,5,6]. However, rule-based methods have disadvantages in that they are designed for a specific environment and are difficult to extend to actual autonomous driving, where various situations can occur.

Conversely, deep reinforcement learning (DRL) has the potential for better scalability and generalization than non-learning methods for autonomous driving decision-making problems. For this reason, DRLs are widely studied in the fields of alarm systems [7], education [8], medicine [9], and sensors [10]. Huegle et al. [11,12] proposed DeepSet-Q and DeepScene-Q, which considered multiple vehicles in a region of interest (ROI) and achieved state-of-the-art quality in 2019 and 2020. However, most existing DRLs are black-box models, so the output process is unknown, and confidence in a decision is unevaluated. In particular, the agent’s decision may cause an accident in an unexpected environment that the agent has never experienced during the learning process. McAllister et al. [13] mentioned that quantifying confidence is necessary to improve stability.

A common way to evaluate confidence in decisions is to quantify uncertainty, divided into epistemic and aleatoric uncertainty [14]. Epistemic uncertainty refers to the uncertainty of the model and can be reduced as it learns from additional data, while aleatoric uncertainty indicates randomness that arises from the nature of data and cannot be reduced through further learning. Both uncertainties are necessary to make risk-aware decisions, specifically epistemic uncertainty that can distinguish between learned and unlearned situations.

The problem of reaching the destination safely and quickly can be replaced by the difficulty of maintaining the target speed without accident. This paper used the Simulation of Urban MObility (SUMO) simulator to verify whether an autonomous vehicle could safely and continuously maintain its target speed in a highway environment. In addition, it was checked whether the agent could detect an inappropriate decision from the estimated uncertainty for an unexpected environment that was not encountered during the learning process. This study’s contributions are as follows:

(1)

This paper proposed DeepSet-Q with Gaussian mixture (DwGM-Q), which considered the unexpected environments that the DeepSet-Q agent cannot.

(2)

A method different from the existing process was presented, and the estimation speed was increased compared to the uncertainty inference method using the current ensemble.

(3)

The uncertainty estimation result was shown through the SUMO simulator.

2. Related Work

The classic DRL algorithm has recently shown successful results in lane-change decisions [15,16,17]. However, the classical DRL outputs the expected action value as a scalar, so reliability is not guaranteed. Bayesian approaches were first used to represent reliability in the field of deep learning [18]. It represents the produced value as the probability distribution rather than a scalar. The uncertainty quantification can be derived from variance in probability distributions, and dropout was used to estimate uncertainty [19]. This method is costly because it requires several samplings to obtain one uncertainty. However, the expensive uncertainty estimation through the ensemble method can be alleviated through parallelization [20]. After producing a Gaussian mixture model (GMM) using a mixture density network (MDN), a study was published quantifying the uncertainty through GMM [21].

The Bayesian approach has also been extended and used in DRL. Bayesian reinforcement learning (BRL) represents the expected action value as a probability distribution rather than a scalar. An estimated uncertainty through MC (Monte-Carlo) dropout [22] and quantified uncertainty from multiple networks through the ensemble technique have been reported in some studies [23,24]. Nevertheless, all of these methods require calculating the network of the same structure multiple times to quantify the uncertainty once.

This paper provides confidence by estimating uncertainty in the existing DeepSet-Q and calculating uncertainty as one network through a mixture density model (MDM). Quantifying uncertainty using MDN eliminated the need for sampling in the existing Bayesian method and excluded the need to calculate the same network multiple times as in the ensemble method. Consequently, it is possible to reduce the computing power required for calculation, which can be used in an embedded environment at a lower price. Research on computing offload has recently been conducted in various fields, which is an essential procedure for practical application [25,26].

3. Method

3.1. Reinforcement Learning

Reinforcement learning (RL) is a subfield of machine learning in which agents interact with some environment to learn policies that maximize expected future returns. This policy determines what action to take on an arbitrary state quantity. The environment transitions to the next state when the work is done, and the agent is rewarded. Reinforcement learning problems can be modeled as Markov decision-making processes (MDPs). Expressed as a tuple, it is equivalent to $\left(S,A,P,R,\mathsf{\gamma }\right),$ where S is the state space, A is the action space, P is the state transition probability, R is the immediate reward, and $\mathsf{\gamma }$ is the discount factor.

In Bayesian reinforcement learning, the reward can be obtained through a probabilistic basis approach. Therefore, the return $Z^{\pi }=\sum _{k=t}^{\infty }{\gamma }^t{R}_t$ is also a random variable for the fixed policy $\pi$. Q-learning aims to learn the optimal action-value function $Q^{\ast }\left(s_t,a_t\right)$, defined in Equation (1) [27].

$$Q^{\pi }\left(s_t,a_t\right)=E(Z^{\pi }\left(s,a\right)=\underset{\pi }{\max }E[({{\displaystyle \sum }}_{t=0}^{\infty }{\gamma }^{t}R\left(s_t,a_t\right)|s_t=s, a_t=a, \pi )$$

(1)

This value function is called the expectation of the value distribution. Through a neural network, the optimal behavioral value function is approximated as $Q\left(s,a; \theta \right)\approx Q^{\ast }\left(s, a\right)$ [28].

$${\mathrm{T}}^{\pi }\mathrm{Z}\left(\mathrm{x}, \mathrm{a}\right)=\mathrm{R}\left(\mathrm{x}, \mathrm{a}\right)+{ \mathsf{\gamma }\mathrm{P}}^{\pi }\mathrm{Z}\left({\mathrm{X}}^{\prime }, {\mathrm{A}}^{\prime }\right)$$

(2)

In Equation (2), $X^{\prime } ~ P\left(·|x,a\right), A^{\prime } ~ \pi \left(·|X^{\prime }\right)$, the target value is estimated through three random variables. The value distribution in RL is expressed, as follows, through the distributional Bellman operator ${\mathrm{T}}^{\pi }$ for a fixed policy $\pi$ and shown to be a contraction mapping for the maximal form of the Wasserstein metric [29].

3.2. Mixture Density Network (MDN)

MDN was first proposed in the research conducted by Bishop [30], focusing on mixture density networks and outputs ${\pi }_j, {\mu }_j, {\sigma }_j$ constituting the GMM. The mixture model refers to a distribution linearly combined with basic distributions, and j.jGMM pertains to several Gaussian distributions that are linearly combined.

$$p\left(x\right)={\displaystyle \sum }_{k=1}^K{\pi }_j\left(x\right)N(x | {\mu }_j,{\sigma }_j)$$

(3)

In the above equation, ${\pi }_j$ is the weight of the j-th Gaussian distribution, ${\mu }_j$ is the average of the Gaussian distribution, and ${\sigma }_j$ is the standard deviation of the Gaussian distribution. Moreover, ${\mathsf{\pi }}_{\mathrm{j}}$∈[0, 1] and ${\displaystyle \sum }_{j=1}^K{\pi }_j=1$ are satisfied for K Gaussian distributions.

3.3. Uncertainty Estimation

The equations for estimating aleatoric uncertainty caused by inherent noise contained from GMM data, and epistemic uncertainty caused by the untrained model were proven in the study conducted by Choi et al. [21]. When K is defined as the number of Gaussian distributions, the total uncertainty for the input vector x follows the following equation:

$${\sigma }_t={\displaystyle \sum }_{j=1}^K{\pi }_j{\sigma }_j\left(s\right)+{\displaystyle \sum }_{j=1}^K{\pi }_j\left(x\right)\Vert {\mu }_j\left(x\right)-{\displaystyle \sum }_{k=1}^K{\pi }_k\left(x\right){\mu }_k\left(x\right){\Vert }^2$$

(4)

In Equation (4), ${\sigma }_t$ denotes the sum of the uncertainties, ${\pi }_j$ denotes the weight of the i-th Gaussian model, and ${\mu }_i\left(x\right)$denotes the average of the i-th Gaussian model.

$${\sigma }_e={\displaystyle \sum }_{k=1}^K{\pi }_k\left(x\right)\Vert {\mu }_k\left(x\right)-{\displaystyle \sum }_{j=1}^K{\pi }_j\left(x\right){\mu }_j\left(x\right)\Vert$$

(5)

$${\sigma }_a={\displaystyle \sum }_{k=1}^K{\pi }_k\left(x\right){\sigma }_k\left(x\right)$$

(6)

In Equations (5) and (6), ${\sigma }_e$ corresponds to epistemic uncertainty, and ${\sigma }_a$ corresponds to aleatoric uncertainty. ${\sigma }_e$ for each decision can be calculated using Equation (5) from the variables obtained using MDN. It can be confirmed that Equations (5) and (6) are decoupled from Equation (4). Uncertainty is a value that is inversely proportional to the reliability and, as the value increases, it can be considered that the corresponding decision causes a malfunction. Equation (6) refers to uncertainty caused by probabilistic variations in a random event.

4. DeepSet with Gaussian Mixture Q Network (DwGM-Q)

4.1. Network Architecture

Figure 1 compares the existing DeepSet network structure with the DwGM-Q network. Feature points for surrounding vehicles are extracted from the $\varnothing$ network. Then, through pooling, permutation invariant features are selected. After that, the feature vector extracted from the $\rho$ network and the state of the ego vehicle are concatenated. The Q network then shows the expected action value for each decision. It can be seen that the existing DeepSet-Q produced a value that corresponds to a scalar. In the case of an ensemble, the final Q value was obtained by summing all the output vectors of DeepSet-Q and then taking the average value. The average (μ), standard deviation (σ), and weight (π) of several Gaussian distributions were the proposed network results. In DwGM-Q with K Gaussian distribution, the Q value is defined as follows

$$Q\left(s_t,a_t\right)={\displaystyle \sum }_{j=1}^K{\pi }_j\left(s_t, a_t\right){\mu }_j\left(s_t,a_t\right)$$

(7)

The Q value occupied the probability distribution form. If the Q value comes out as a probability distribution, the uncertainty can be obtained.

4.2. Weight Update

Jensen–Shannon divergence (JSD) was used to find the difference in probability distribution between the target Gaussian mixture model and the current Gaussian mixture model [31]. To learn GMM in reinforcement learning, the distance of two Gaussian distributions $Z_1$ and $Z_2$ is explained in Equation (8).

$$\begin{array}{cc}\hfill JT{D}^2\left(Z_1, Z_2\right)=& {\displaystyle {\displaystyle \sum }_{i, j}{\pi }_i{\pi }_{j}N\left({\pi }_i;{\pi }_j, {\sigma }_i^2+{\sigma }_j^2\right)+{\displaystyle \sum }_{i, j}{w}_i{w}_{j}N\left(m_i;m_j, s_i^2+s_j^2\right)}\hfill \\ & -2 {\displaystyle {\displaystyle \sum }_{i, j}}{\pi }_i{w}_{j}N\left({\mu }_i;m_j, {\sigma }_i^2+s_j^2\right)\hfill \end{array}$$

(8)

JSD has excellent properties free from the disjoint support issue, which arises when using the KL-divergence loss, and exhibits unbiased sample gradients, contrary to the Wasserstein distance.

A training procedure of a DwGM-Q is shown in Algorithm 1.

Algorithm 1: Training DwGM-Q

Initialize $Q=\left( \varnothing , \rho , Q\right)and Q^t=\left( \varnothing {,}^t {\rho }^t, Q^t\right)$

Set replay buffer $\Re$

for episode = 1, M do

initialize random state $s_i$

for step = 1, T do

if  $e~ u\left(0,1\right)<ϵ$

${\sigma }_e \leftarrow \sigma \left(\pi , \mu \right)$ (Equation (5))

$select a_t as the probability$

$proportional to the value of {\sigma }_e$

else

$Q\left(s_t,a_t\right)$ = ${\displaystyle \sum }_{j=1}^K{\pi }_j\left(s_t, a_t\right){\mu }_j\left(s_t,a_t\right)$

select $a_t=argma{x}_{a}Q\left(s_t,a_t\right)$

execute $a_t$ and observe reward $r_t$ and $s_{t+1}$

Store transition ($s_t,a_t, r_t, s_{t+1}$) in $\Re$

Sample random minibatch of transitions ($s_t,a_t, r_t, s_{t+1}$) from $\Re$

$Q\left(s_{t+1},a_{t+1}\right)= {\displaystyle \sum }_j{\pi }_j\left(s_{t+1},a_{t+1}\right), {\mu }_j\left(s_{t+1},a_{t+1}\right)$

$a^{\ast }\leftarrow \arg \underset{a}{\max }Q\left(s_{t+1},a_{t+1}\right)$

${\mathrm{T}}^{\pi }Z\left(s_t,a_t\right) \leftarrow {\displaystyle \sum }_j{\pi }_j\left(s_t^{\prime }, a_t\right)N(\gamma \mu \left(s_{t+1} ,a^{\ast }\right)+r, {\gamma }^2{\sigma }_j{\left(s_{t+1}, a^{\ast }\right)}^2)$

Compute JTD loss (Equation (8))

Update weights

4.3. Decision Procedure

In Figure 2, $c_v^{safe}$ is a threshold for uncertainty and determines whether to perform $a_{safe}$. $c_v$ is the uncertainty estimated by DwGM-Q and calculated as follows:

$$c_v=\frac{\alpha }{M}{\displaystyle \sum }_{i=1}^M{\sigma }_t^{\left(i\right)}$$

(9)

where M is the number of actions, and $\alpha$is the normalization factor. As the learning of DwGM-Q progresses, the GMM value approaches a single distribution. Figure 2 describes the procedure for making a decision. In input data preprocessing, the state is created with vehicle information. The details of the state are covered in Section 4. The DwGM-Q network provides the expected action value of each action in the form of GMM. From the $\pi , \mu , \sigma$ of the GMM mixture model, uncertainty can be estimated through Equations (5) and (9). The Q value is also estimated by Equation (7). If the obtained $c_v$ is smaller than $c_v^{safe}$, the agent’s decision is performed, assuming that the decision is reasonable. On the other hand, if it is greater than the threshold, the agent’s decision is considered unreasonable, and $a_{safe}$ is performed.

5. Environment Setup

Input, reward, decision type, and experimental environment were the identical neural networks stated in the conducted research by Hoel et al. [19].

5.1. State Input

As shown in

Table 1. Neural network input.

Parameter	Value	Comments
${\phi }_1$	$\frac{2y_0}{y_{\max }}-1$	Ego lane
${\phi }_2$	$\frac{2v_{x,0}}{x_{max}}-1$	Ego vehicle speed
${\phi }_3$	$sgn\left(v_{y, 0}\right)$	Lane-change info
${\phi }_{4i}$	$\frac{x_i-x_0}{x_{sensor}}$	Relative long. Position of i-th vehicle
${\phi }_{4i+1}$	$\frac{y_i-y_0}{y_{max}}$	Relative latitude. Position of i-th vehicle
${\phi }_{4i+2}$	$\frac{y_i-y_0}{y_{max}}$	Relative speed of i-th vehicle
${\phi }_{4i+3}$	$sgn\left(v_{y, i}\right)$	Lane-change state of i-th vehicle

, ($x_0, y_0$) means the location information of the ego vehicle and ($x_i, y_i$) denotes the location information of the i-th surrounding vehicle. The $v_{x,i}, v_{y,i}$ is defined as the absolute speed of the i-th surrounding vehicle in the x and y directions. In addition, $x_{sensor}$ refers to the longitudinal ROI that the sensor of the autonomous vehicle can detect. When the vector representing the information of the own vehicle at time t is $x_t^{static}$ and the vector representing the information of the i-th own vehicle is $x_t^i$, $x_t^{static}=\left\{{\phi }_1, {\phi }_2, {\phi }_3\right\}$, therefore, $x_t^i=\left\{{\phi }_{4i}, {\phi }_{4i+1}, {\phi }_{4i+2}, {\phi }_{4i+3}\right\}$. The input vector is $s_t=\{x_t^{static},x_t^1, x_t^2, \dots , x_t^n$} if there are n number of surrounding vehicles in the ROI.

5.2. Action

The reinforcement learning agent was designed to have lane keeping, right and left lane change in the lateral direction, and acceleration of {−1, 0, 1} m/s² in the longitudinal direction. Moreover, there was an emergency stop decision with −4 m/s² for lane keeping. This decision was expressed as $a_{safe},$ the most conservative decision. When the uncertainty exceeds the threshold, $a_{safe}$ is executed. Therefore, the agent had ten decisions: three each in the horizontal and vertical directions, and one additional emergency stop decision.

5.3. Reward

The agent obtained a reward in the amount of $1-\frac{v_{target}-v}{v_{target}}$ for every step. This agent induced the vehicle to overtake the slow vehicle while maintaining the target speed. Penalties of −1 and −10 were given to prevent reckless lane change if the time to collision (TTC) is less than 2.5 s with the vehicle in front. In addition, a penalty of −10 was given even when an emergency brake decision was performed.

6. Experiment

A SUMO simulator was applied for this experiment. This simulator can obtain the state quantity by inputting longitudinal acceleration and lane-change decisions. The surrounding vehicle’s speed was set between 15 and 35 m/s ($3.6 \mathrm{km}/\mathrm{h}=1 \mathrm{m}/\mathrm{s}$), and the targeted speed of the ego vehicle was set to 25 m/s. After making 100 decisions during one episode, the agent and surrounding vehicles were randomly initialized again. The ego vehicle corresponds to the green truck, as illustrated in Figure 3. Furthermore, the vehicle is displayed in yellow and red as it approaches 15 m/s and 35 m/s. Uncertainties were observed for three scenarios.

All environments were limited to three-lane highways. In scenario A (normal scenario), yellow and red represented vehicles at 15 to 35 m/s. Meanwhile, a white vehicle existed in scenario B (stop scenario), whereas a purple vehicle with a 55 m/s speed appeared in scenario C (speeding scenario). These white and purple vehicles presented in scenarios 2 and 3 correspond to surrounding vehicles missed during the learning process. This experiment presented whether the agent collides and observes the uncertainty values in unexpected situations. Parameters for training the agent are shown in Appendix A (see

Table A1. DeepSet-Q hyperparameters.

Learning rate	1 × 104
$\rho$ node	80
$\varnothing$ node	20
${ϵ}_{min}$	0.1
Batch size	64
Buffer size	100,000

and

Table A2. DwGM-Q hyperparameters.

Learning rate	1 × 104
# of Gaussian	5
$\rho$ node	80
$\varnothing$ node	20
${ϵ}_{min}$	0.1
Batch size	64
Buffer size	100,000

). Adam optimizer was also used to optimize the reinforcement learning network.

The agent’s output and uncertainty estimation results for the normal scenario are shown in Figure 4. In Figure 4B, for the situation of Figure 4C, the GMM output result for each action can be checked. The values 0–9 of Figure 4B represent ten decisions: 0, 1, 2, 3 represent 0, 1, −1, −4 $\mathrm{m}/{\mathrm{s}}^2$ as longitudinal direction and lane-keeping decisions as lateral direction, respectively; 4, 5, 6 represent 0, 1, −1 $\mathrm{m}/{\mathrm{s}}^2$, left-lane-change decision, and 7, 8, 9 represent 0, 1, −1 $\mathrm{m}/{\mathrm{s}}^2$, right-lane-change decision. The uncertainty values were all lower than the threshold $c_v^{safe}$ during an episode in the normal scenario. In Figure 4B, the probability of the expected action value of the lane-change decision has many peaks. This indicates that the epistemic uncertainty increases from Equation (5). However, as the learning progressed, the value gradually decreased. In Figure 4B, the probability of the expected action value of the lane-change decision has many peaks. This indicates that the epistemic uncertainty increases from Equation (5). However, as the learning progressed, the value gradually decreased.

In Figure 5A, the DwGM-Q agent observes the uncertainty value in the stopping scenario. The $c_v$ exceeds the $c_v^{safe}$ after the stop vehicle is detected. Figure 5B shows the output of the expected action value after the stop vehicle is detected. The GMM of most decisions has many peaks, which causes the uncertainty $c_v$ to be increased in Figure 5A. Despite the unexpected environment that was not experienced in the learning process, the uncertainty value exceeded the threshold $c_v^{safe}$, causing $a_{safe}$ to be performed, and the episode was completed without a collision.

Figure 6C shows the situation in which the speeding vehicle passes through the ego vehicle. When the speeding vehicle passes through the agent, the $c_v$ exceeds $c_v^{safe}$, and $a_{safe}$ is performed. Figure 6B shows the GMM values at that time. In Figure 6B, most GMMs are not single distributed, but have high deviation. This causes the $c_v$ to be increased.

Table 2. Experiment Results.

	Normal Scenario			Unexpected Scenario
				Stop Scenario		Speeding Scenario
	Mean Velocity (m/s)	Collision Rate (%)	Steps Per s (SPS)	Collision Rate (%)	Steps per s (SPS)	Collision Rate (%)	Steps Per s (SPS)
DeepSet-Q	21.9	2	213.5	100	213.6	100	213.6
DeepSet-Q (15–55 m/s)	21.8	3	213.6	100	213.5	0	213.5
RPF- Ensemble	23.4	2	165.5	0	165.3	0	165.6
DwGM-Q	22.1	2	184.4	0	184.5	0	184.5

shows the performance of DeepSet-Q, RPF-Ensemble [23], and DwGM for the three types of scenarios. The value for the performance was quantified through 100 episodes.

There is no significant difference between the average speed and the collision rate for all methods in a standard scenario. The DeepSet-Q for SPS was the highest of all agents, and the proposed method was the second highest. However, the existing DeepSet-Q agent caused a crash in unexpected scenarios. On the other hand, the RPF-ensemble method or the proposed method, does not cause crashes even in unexpected scenarios. Therefore, this result shows that the proposed method can also cope with unexpected scenarios, and the computational speed is also faster than the existing method. The DwGM-Q network is faster than the ensemble method because it can obtain the uncertainty directly through a single calculation without calculating the network several times.

In addition, we have trained DeepSet-Q networks in an environment where sur-rounding vehicles have speeds of 15–55 m/s. This means that the DeepSet-Q agent has learned the speeding scenario. The DeepSet-Q agent performed well without colliding in the speeding scenario but collided in the stop scenario. This shows that if the DeepSet-Q agent learns an unexpected scenario, it also performs well. However, there are numerous unexpected situations, such as sensor malfunctions, various behaviors of aggressive vehicles, and complex situations that occur in real life. It is challenging to learn them all. Therefore, it is meaningful to deal with unexpected scenarios using uncertainty.

7. Conclusions

The existing DeepSet-Q algorithm failed to detect uncertainty and avoided collisions for unexpected scenarios. Existing ensemble methods avoid collisions in unexpected scenarios. However, they have the disadvantage of taking a lot of computation time due to multi-network tasks. This paper quantified the uncertainty by adding MDN to the end of the existing DeepSet-Q network to solve these problems. The experiment showed that uncertainty was well detected, and the calculation speed was faster than the existing ensemble method. The most conservative decision was selected as the break action, but it can also be replaced by something like a rule-based decision controller with higher usability.

The proposed method is proper when safety and real time are guaranteed in embedded environments such as autonomous driving. Although this paper deals with decision making in autonomous driving, it can be used in various fields, such as vehicle control and aircraft control.

Furthermore, The DwGM-Q agent could distinguish between aleatoric and epistemic uncertainty contrary to the ensemble technique. However, for DwGM-Q networks, the learning time took approximately four times longer than for DeepSet-Q. Reducing learning time with various learning techniques is also a task to be solved [32,33,34]. Since the uncertainty threshold is set through a heuristic, it is also necessary to establish a standard for automatically quantifying the threshold depending on the situation.