An Interacting Multiple Model Approach for Target Intent Estimation at Urban Intersection for Application to Automated Driving Vehicle

Featured Application

Motion Prediction, Target Intent Inference, Urban Intersection, Automated Driving.

Abstract

Research shows that urban intersections are a hotspot for traffic accidents which cause major human injuries. Predicting turning, passing, and stop maneuvers against surrounding vehicles is considered to be fundamental for advanced driver assistance systems (ADAS), or automated driving systems in urban intersections. In order to estimate the target intent in such situations, an interacting multiple model (IMM)-based intersection-target-intent estimation algorithm is proposed. A driver model is developed to represent the driver’s maneuvering on the intersection using an IMM-based target intent classification algorithm. The performance of the intersection-target-intent estimation algorithm is examined through simulation studies. It is demonstrated that the intention of a target vehicle is successfully predicted based on observations at an individual intersection by proposed algorithms.

1. Introduction

Advanced driving assistance system (ADAS) and autonomous driving system (AD) are expected to be at the center of future transportation systems as well as highly enhance traffic safety. To successfully bring autonomous vehicles (AVs) into our lives, they must be capable of managing complex urban environments, including various participants and complex traffic infrastructure. Among them, intersections are one of the most dangerous urban traffic infrastructures. It is reported that 21.5% of fatalities and even 40% of all accidents in the United States occur at intersections [1]. Although recent commercialized vehicles are equipped with ADAS functions, human drivers still need to focus on complex driving situations such as urban intersections. To deal with this, the predict-and-plan approach is typically applied; during the ‘predict’ step, the ego vehicle integrates numerous signals from sources such as sensors or traffic infrastructure and predicts future actions of several agents in the vicinity. In the planning step, the ego vehicle generates maneuvers based on the trajectory from the predict step. Therefore, the predict step is essential in the entire decision-making process, so several researchers have tried to develop methods to predict the intention of other traffic agents which not only have behavior patterns but also inherent uncertainty.

Recently, most of cars that have been newly released have a certain level of ADAS built into the vehicle. For instance, in the most recently released GV80, Hyundai Motors group provides general ADAS functions such as forward collision-avoidance assist (FCA) and lane keeping assist (LKA) by default. The FCA offers a more diverse range of recognition options, including when a vehicle passes through an intersection or is laterally approaching. It also provides Highway Driving Assist II (HDAII), which includes machine-learning-based adaptive cruise control (ACC), automatic lane change assist when operating direction indicator lamps, and near-field vehicle recognition technology [2]. Hyundai Motors Group also announced that they have a plan to mass-produce highway driving pilots (HDPs) in the near future, which enables the vehicle to drive on its own, even if the driver leaves the steering wheel when driving on highways [3]. Practically, it is equivalent to SAE (Society of Automotive Engineer) Level 3, which is often compared to Tesla’s Navigate on Autopilot (NOA). Tesla’s autopilot is capable of steering, acceleration, and deceleration of the vehicle. In addition, the most recently updated feature, NOA, based on autopilot, enables the vehicle to change lanes and overtake other cars using map data [4]. As such, the vehicles on the market with most advanced ADAS are assessed to be within the SAE Level 2–3. Now, most of the conventional carmakers and vehicle technology providers are testing AD vehicles that are equivalent to SAE Level 4 within a certain area, and they have claimed that these verified AD vehicles can be used as shuttles and taxis, for example, ‘Waymo One’ of Waymo [5], Google’s subsidiary, ‘Robotaxi’ of Tesla [6], and ‘Pilot Project’ by Mercedes Benz, and Bosch [7]. This trend of technological advancement once again confirms that the vehicle’s responsibilities are increasing, and the perception and control technology must ensure a certain level.

As stated above, since obtaining effective vehicle behavior analysis is critical for urban autonomous driving, tracking and calculating of the expected trajectory must be conducted. To better estimate the trajectory of vehicle movement, the performance of the state of estimation algorithms is one of the most significant factors. Thus, various approaches that have been theorized in literature have been tested, such as dynamic Bayesian networks (DBN), hidden Markov models (HMM), support vector machines (SVM), interacting multiple model (IMM), and vehicle-to-vehicle (V2V) communication [8,9,10,11,12]. Traditional learning methods like DBN and HMM are frequently used since they are simple, fast, and do not require lots of data to become trained [13]. For instance, S. Lefèvre et al. uses DBN, combining probabilistically uncertain observations on the vehicle’s behavior and local characteristic information to estimate driver maneuvers [14]. In addition, Streubel et al. proposed a prediction framework based on HMM using a database of real driving data such as speed, acceleration, and yaw rate [15]. As a discriminative approach, SVM is used as a learning framework for binary classification. Aoude et al. present a comparison and validation of performances of SVM, HMM, and other traditional approaches consisting of TTI-, RDP-, and SDR-based algorithms [16]. In recent years, along with the advancement of information and communication technologies, V2V communication is often applied to the prediction protocol. Aoki et al. introduce a decentralized intersection protocol for mixed traffic environments where all automated vehicles use V2V communications [17]. Since this information-driven control research area is active, not only V2V but also vehicle-to-infrastructure (V2I), vehicle-to-pedestrian (V2P), technologies are arousing interest.

As stated above, the IMM algorithm is a method used to determine the expected target trajectory in selecting among filter models derived from the behavior of targets. To select the best hypothesis, the IMM consists of four main steps: interaction, filtering, update, and combination. At the interaction step, the initial values of each model for the filtering step are calculated by receiving signals from sensors. These initial values are transferred to the filtering step. In the filtering step, each model independently conducts the calculation and deduces the state prediction. Typically, a Kalman filter (KF) is applied at this step. Then, the probabilities of models are updated at the update step, and by combining the probabilities and sending them to the interaction step again at the combination step, one cycle of the IMM is finished.

Since the IMM was introduced in the late 1980s, it is generally known and has been implemented on a wide range of conditions. Tsunashima et al. used IMM to estimate vehicle state differing road friction [18]. Rubin et al. illustrate that the noble IMM estimator works accurately with the external disturbances and inputs of steering [19]. As the IMM is verified in plenty of conditions, the IMM algorithms are also often modified. Wang et al. presents an adaptive IMM which utilizes the models for an adaptive turn rate in order to track a target for maneuvering [20]. In addition, the IMMs are classified depending on how they use multi-model fusion criteria as scalar weight IMM (SIMM), diagonal-matrix-weight interacting multiple model, and matrix weight interacting multiple model [21,22]. These new approaches weigh differently when mixing beginning model values with states and corresponding covariances at each step of the IMM algorithm. Fu et al. also introduced H∞ filtering into the distributed interacting multiple model (DIMM) algorithm instead of KF, for target tracking of maneuvering during which the measurement noise is statistically unknown [23]. Park et al. propose a new algorithm suitable for multi-object tracking using multi-data fusion by applying centralized KF to a typical IMM. Since it is reasonable and easily adaptable, the IMM is applied in a variety of fields which require tracking of the maneuvering target. The IMM filter is often applied in aircraft tracking problems, such as in the study by Li et al. [24]. Tong et al. used IMM for 3D human motion tracking [25]. For application to autonomous vehicles, an IMM filter that integrates market commercial GPS and vehicle local sensors is implemented to develop a vehicle localization algorithm [26]. Since the IMM algorithm is incorporated in both the kinematic and dynamic model of the vehicle, the positioning performance is improved demonstrating high accuracy under the various driving conditions.

Thanks to the advantages described above, we propose an intersection-target-intent estimation algorithm based on IMM [27].

In this research, we focus on the target intent estimation algorithm at urban intersections, and the paper is structured as follows. The next section demonstrates use of the intersection driver behavior model to represent the intent of the driver’s maneuvering at the intersection. In Section 3, an IMM-based target intent classification algorithm is demonstrated by reflecting continuity of driver behavior while improving the accuracy of state prediction. The target intent estimation algorithm for urban intersections is verified via simulation studies in Section 4. Finally, conclusions are provided in Section 5.

2. Intersection Driver Behavior Model

The target road is set to be an intersection of two-way streets. Under these circumstances, if the target approaches the intersection using the leftmost lane from the south, there are eight possible maneuvers recognized, as shown in Figure 1. The possible trajectory is depicted as a red line, the entrance as a red cross, and the exit as a blue circle. Note that there are 64 possible maneuvers if we consider each case of possible entrance. To describe all of these behaviors, the behavior of the target vehicle is represented by path-tracking, with slowing down before turning left, right, or making a U-turn. In order to illustrate these motions, two vectors (state and input vector) are established as shown below:

$$x={\left[\begin{array}{cccc}{p}_x& p_y& \psi & v_x\end{array}\right]}^T$$

(1)

$$u={\left[\begin{array}{cc}{a}_x& \gamma \end{array}\right]}^T$$

(2)

where $x$ and $y$ are the x- and y-axis of each frame, respectively; $p$ illustrates the relative position; $\psi$ depicts the relative yaw angle; $v$ is the velocity; $\gamma$ is the yaw rate; $a$ is the notation of the acceleration. The input vector is calculated by the driver model as follows:

$$u[\mathrm{t}]=f_{D.M}(x[\mathrm{t}],\mathrm{traj}(m))$$

(3)

where $f_{D.M}$ is the driver model and $\mathrm{traj}(m)$ is a reference path of the target vehicle when it intends to go the m-th exit.

3. IMM/EKF-Based Intersection-Target-Intent Estimation

By receiving the validated measurement for every single target, extended Kalman filters (EKFs) are implemented as the local filter which matches every single mode. This is incorporated using the interacting multiple model (IMM) approach to retain the best estimate of the state vectors and the best matching modes. The algorithm for estimation of intersection target intent is described below. It is noted that the time index is to be omitted unless otherwise needed for clear explanations.

(1). Interaction ($\forall i,j\in M/\forall n\in T$):

With the $N_{mode}$ weights of ${\mu }_n^i\left[k-1\right]$, the $N_{mode}$ specifically depicts ${\widehat{x}}_n^i\left[k-1\right]$, and the $N_{mode}$ related covariance ${\widehat{P}}_n^i\left[k-1\right]$, which are used to calculate the mixed condition for an initial condition which is used for a filter matched to the state of mode $j$. It should be noted that if the implemented modes contain different dimensions of state vectors, the lower dimension state of an augmentation with zeros observed may cause a bias toward zero for components of the larger state vector. According to [21], to avoid the issue of ‘biasing’, a simple procedure is suggested, combined with a proper augmentation of the smaller state’s covariance, and it yields unbiased and consistent mixing.

Mode probability with Predictions:

$$\begin{array}{ll}{\overline{\mu }}_n^j& \triangleq \mathrm{Pr}\left\{m_n\left[k\right]=j|z_n\left[k-1\right]\right\}\\ & ={\displaystyle \sum _i\left\{{\Phi }_{ij}\left[k-1\right]{\mu }_n^i\left[k-1\right]\right\}}\end{array}$$

(4)

Mixing probability:

$$\begin{array}{ll}{\mu }_n^{i|j}& \triangleq \mathrm{Pr}\left\{m_n\left[k-1\right]=i|m_n\left[k\right]=j,z_n\left[k-1\right]\right\}\\ & =\frac{{\Phi }_{ij}\left[k-1\right]{\mu }_n^i\left[k-1\right]}{{\overline{\mu }}_n^j}\end{array}$$

(5)

Mixed condition:

$${\widehat{x}}_n^{0j}\left[k-1\right]={\displaystyle \sum _i{\widehat{x}}_n^i\left[k-1\right]{\mu }_n^{i|j}}$$

(6)

$${\widehat{P}}_n^{0j}\left[k-1\right]={\displaystyle \sum _i\left[{\widehat{P}}_n^i\left[k-1\right]+\left\{\begin{array}{l}\left({\widehat{x}}_n^i\left[k-1\right]-{\widehat{x}}_n^{0j}\left[k-1\right]\right)\\ \times {\left({\widehat{x}}_n^i\left[k-1\right]-{\widehat{x}}_n^{0j}\left[k-1\right]\right)}^T\end{array}\right\}\right]{\mu }_n^{i|j}}$$

(7)

(2). EKF Approach:

Every single pair of the $N_{mode}$ values has weights of ${\widehat{x}}_n^{0j}\left[k-1\right]$, ${\widehat{P}}_n^{0j}\left[k-1\right]$, and it is used to input to an EKF approach which is matched to the state of mode $j$.

Time update:

$$F_n^j\left[k-1\right]={\frac{\partial f_n^j}{\partial x}|}_{{\widehat{x}}_n^{0j}\left[k-1\right],\widehat{u}\left[k-1\right]}$$

(8)

$${\overline{x}}_n^j\left[k\right]=f_n^j\left({\widehat{x}}_n^{0j}\left[k-1\right],\widehat{u}\left[k-1\right]\right)$$

(9)

$${\overline{P}}_n^j\left[k\right]=F_n^j\left[k-1\right]\cdot {\widehat{P}}_n^{0j}\cdot \left[k-1\right]F_n^j{\left[k-1\right]}^T+W_n^j\left[k-1\right]$$

(10)

Filter gain:

$$H_n^j\left[k\right]={\frac{\partial h_n^j}{\partial x}|}_{{\overline{x}}_n^j\left[k\right],\widehat{u}\left[k\right]}$$

(11)

$$S_n^j\left[k\right]=H_n^j\left[k\right]\cdot {\overline{P}}_n^j\left[k\right]\cdot H_n^j{\left[k\right]}^T+V_n^j\left[k\right]$$

(12)

$$K_n^j\left[k\right]={\overline{P}}_n^j\left[k\right]\cdot H_n^j{\left[k\right]}^T\cdot S_n^j{\left[k\right]}^{-1}$$

(13)

Innovation:

$$r_n^j\left[k\right]=z_n^{}\left[k\right]-h_n^j\left({\overline{x}}_n^j\left[k\right],\widehat{u}\left[k\right]\right)$$

(14)

Measurement update:

$${\widehat{x}}_n^j\left[k\right]={\overline{x}}_n^j\left[k\right]+K_n^j\left[k\right]\left\{r_n^j\left[k\right]\right\}$$

(15)

$${\widehat{P}}_n^j\left[k\right]=\left(I-K_n^j\left[k\right]\cdot H_n^j\left[k\right]\right){\overline{P}}_n^j\left[k\right]$$

(16)

(3). Mode Probability Update:

From the innovations of $N_{mode}$, the EKFs are updated.

Likelihood function:

$${\mathsf{\Lambda }}_n^j={\left|2\pi S_n^j\right|}^{-1/2}\cdot \exp \left[-\frac{1}{2}\cdot {\left(r_n^j\left[k\right]\right)}^T\cdot {\left(S_n^j\right)}^{-1}\cdot \left(r_n^j\left[k\right]\right)\right]$$

(17)

Mode probability:

$${\mu }_n^j=\frac{{\overline{\mu }}_n^j\cdot {\mathsf{\Lambda }}_n^j}{{\displaystyle \sum _i{\overline{\mu }}_n^i\cdot {\mathsf{\Lambda }}_n^i}}$$

(18)

(4). Combination Step:

This is just for output purposes, and ${\widehat{x}}_n^{}\left[k\right]$ and ${\widehat{P}}_n^{}\left[k\right]$ are calculated with the

Combined condition:

$${\widehat{x}}_n^{}\left[k\right]={\displaystyle \sum _j{\widehat{x}}_n^j\left[k\right]{\mu }_n^j}$$

(19)

$${\widehat{P}}_n^{}\left[k\right]={\displaystyle \sum _j\left\{{\widehat{P}}_n^j\left[k\right]+\left({\widehat{x}}_n^j\left[k\right]-{\widehat{x}}_n^{}\left[k\right]\right){\left({\widehat{x}}_n^j{}_j\left[k\right]-{\widehat{x}}_n^{}\left[k\right]\right)}^T\right\}{\mu }_n^j}.$$

(20)

4. Simulation Results

The developed IMM-based intersection-target-intent estimation algorithm is implemented and verified via simulation studies. The test scenario is a left turn situation of the target vehicle. To check the generality of the algorithm, the target is set to approach the intersection from the east using the most left lane. The initial speed and the set speed are 50 kph which is too high for a stable and safe turn. The velocity profile in the case of the left turn is illustrated in Figure 2. As briefly shown in the figure, a proposed driver model slows down the car approaching the intersection and performs a stable left turn at low speed even though the set speed is 50 kph.

Figure 3 demonstrates the mode probability adaptation results along the axis of travel length. We can see that the updated result is very similar to the real mode. From 20–40 m, which describes a period of the approaching target vehicle at the intersection with deceleration, it can be observed that the probability of mode 5 has the highest rank, which means the left turn to exit #5. The observed result is that we cannot tell which exit the target vehicle will head to during that period if we only measure pose and heading. Since two vectors (state and input vector) were established in Section 2, this allows prediction of the motion of the target with accuracy and reliability. From this result, the proposed approach delivers good results from the viewpoint of classification performance and the target vehicle’s behavior predictions.

5. Discussion & Conclusions

We propose a simple and real-time approach to estimate a target vehicle’s intent at urban intersections. It is based on a driver model for representing the real driver’s intersection maneuvering and use of an IMM to classify the target intent. It is demonstrated that the proposed method allows us to achieve good classification performance via simulation study.

This paper focused on the implementation of the proposed algorithm and a real road test. By applying the proposed algorithm to an emergency driver support system, or an automated driving technology, it is expected that intersection safety can be significantly improved.

Since the IMM approach using extended Kalman filters (EKFs) for multi-target state estimation of intelligent vehicle has already been verified by Figure 4, and in the work of Suh et al. [28], the proposed algorithm will be improved in the urban intersection, which is a more complex driving situation using the perception of a multi-target vehicle. Furthermore, it is expected to be equipped as a generalized perception module for various applications of advanced driver assistance systems (ADAS) which use the lane change assistance system, intersection driving assistance system, steering assist system, side-crash prevention system, and autonomous emergency braking system. Using our proposed algorithm, it is possible to make generic assessments as well as an overall assessment of collision risks with multi-target vehicles in complex urban intersections.

Considering a mixed traffic situation, which would include many other cars and more lanes with different types of intersection shapes in urban roads, a deep learning approach should be taken into account in future research.

The deep-learning approach is also useful to streamline the method stated in this paper to some degree of complex urban road. Since urban traffic is quite complicated and hard to predict, the algorithm needs to be improved using both a rule-based and deep-learning approach. The rule-based EKF approach, as previously demonstrated in this paper, is still beneficial to guarantee the safety of a vehicle in any driving situation.