Real-Time Inter-Vehicle Data Fusion Based on a New Metric for Evidence Distance in Autonomous Vehicle Systems

Abstract

Safety is a major concern for autonomous vehicle driving. The autonomous vehicles relying solely on ego-vehicle sensors have limitations in dealing with collisions. The risk can be reduced by communicating with other vehicles sharing sensed information. In this paper, we study a real-time multisource data fusion scheme based on Dempster–Shafer theory of evidence (DS) through cooperative vehicle-to-vehicle (V2V) communications. The classical DS can produce erroneous outputs when confidences are highly conflicting. In particular, a false negative error on object detection may lead to serious accidents. We propose a novel metric to measure distance between confidences on the collected data taking the asymmetry in the importance of sensed events into account. Based on the proposed distance, we consider the weighted DS combination in order to fuse multiple confidences. The proposed scheme is simple, and enables the vehicles to combine confidences in real-time. From real-world experiments, we show that it is feasible to lower the false negative rate by the proposed data fusion scheme, and to avoid accidents using the proposed collision warning system.

1. Introduction

Autonomous vehicles have been considered as a key technology for intelligent transportation systems. According to the research by National Highway Traffic Safety Administration (NHTSA) in 2015, human error causes around 94% traffic crashes [1], and autonomous vehicles are expected to reduce traffic crashes drastically by minimizing drivers’ mistakes [2]. The design of operating software as well as diagnostics tool for autonomous driving have been actively investigated [3]. In particular, a proper anti-collision system is one of the most crucial components of autonomous vehicle systems.

The information including location, distance between adjacent vehicles, and the vehicle velocity can be obtained by the equipped sensors such as radar, Global Positioning System (GPS), and cameras. Collision avoidance systems based on sensing data are actively studied [4,5], including systems to fuse the sensing data obtained from multiple sensors, e.g., combining data from radar and cameras [6]. However, autonomous vehicle systems relying solely on ego-vehicle sensors have limitations in dealing with the collisions due to reasons such as bad weather, sensor malfunctioning, or blind spots, etc. [7,8,9]. The risk can be reduced by utilizing sensed information from the neighboring vehicles through Vehicle-to-Vehicle (V2V) communication [10,11] such as Dedicated Short Range Communications (DSRC) [12] which can support safety applications in high data rates [13,14].

In order to leverage sensed data from multiple vehicles in real-time, we require a data fusion system with low complexity and high reliability. In multisensor systems, there is uncertainty caused by inaccuracy of sensors or complicated sensing environment [15]. In this respect, the “Dempster–Shafer theory of evidence (DS)” [16,17] is a widely used evidence theory to comprehensively fuse multisource information including uncertainty. It does not require any priori information and requires a simple calculation where the outcome can be reinforced as more data become available [18,19,20].

Although DS has many advantages, the classical DS combination may yield unreasonable outputs when evidences are highly conflicting [18,19,20]. This is critical for autonomous vehicle safety, i.e., vehicles may have erroneous sensing results. Data fusion can cause either false positive or false negative errors. The former is a false alarm, i.e., the vehicle falsely identifies a nonexisting object as an existing one. The latter represents misdetection of an existing object or obstacles. In the context of autonomous vehicle safety, false negatives are significantly dangerous [21] according to the case studies on autonomous vehicle accidents [22]. Therefore, we would like to put more emphasis on the confidence in existence rather than nonexistence of objects, and this necessity results in an asymmetry in the importance of sensed events. However, most of existing works do not consider this aspect. Although there are many works which incorporate the degree of conflict into a fusion method [23,24,25,26,27], the studies on evidence distance taking the aforementioned issue into account are rare. In this paper, we develop a novel metric for evidence distance to measure conflict by assigning asymmetric weights to sensed results and propose a real-time data fusion scheme based on the new metric. In addition, we outline a V2V communication method for anti-collision systems in which the proposed data fusion scheme can be effectively applied.

We summarize the main contributions of this paper as follows.

(1)

A new metric for evidence distance for real-time data fusion. We propose a novel metric for autonomous vehicle to measure distance between confidences on detected events of asymmetric importance. Next, we propose real-time data fusion scheme based on the new distance, which combines multiple sensing data from adjacent vehicles. We show that the proposed scheme has low complexity, and one can obtain consistent output irrespective of the order of the data to be combined, which is beneficial in the autonomous vehicle environment. In other words, the method is insensitive to external factors that may affect data arrival time such as channel states or vehicle speed.

(2)

An integrated collision warning protocol for autonomous vehicles. We propose a cooperative collision warning protocol integrating the proposed data fusion scheme. For real-time operation, the system is composed of technologies in various fields that can guarantee low latency and high reliability. By experiments, we show that the method takes only a few milliseconds to return credible detection of nearby objects.

(3)

Experiments with real-world data. By simulation, we show that the proposed scheme reduces false negative errors as compared to the existing methods. In addition, we conduct real-world experiments and show that the proposed system not only prevents collision from blind spots, but also compensates for inaccurate sensing information.

A comprehensive evaluation of the proposed data fusion shows that our scheme reduces fault diagnosis compared to the baseline methods. In addition, the proposed scheme has low complexity, which enables combining gathered data in real-time. Next, experiment results reveal that the proposed collision warning system with data fusion can successfully detect and avoid imminent collisions.

The rest of this paper is organized as follows. In Section 2, we present related works. In Section 3, we introduce background evidence theory and propose the data fusion schemes based on a new metric for evidence distance. The collision warning system that involves the proposed data fusion processes is proposed in Section 4. Section 5 shows the performance evaluation and experiment results with real-world data, and Section 6 concludes this paper.

2. Related Work

There have been several works which implement DS-based multisource data fusion for autonomous vehicle safety. In [28], a method for calculating the probabilities of existence of nearby objects was proposed. From the local sensing results at ego-vehicle, the existence probability is obtained by the DS fusion. A trajectory selection problem was considered in [29], which proposed a decision-making algorithm based on multicriteria such as safety, comfort, and energy consumption. The DS is used for risk assessment, which combines multiple local risk levels obtained from the predefined multiple criteria. However, as their schemes rely only on ego-vehicle’s information, it has limitations on collisions predictions. In [30,31], collision avoidance schemes based on occupancy grid for autonomous vehicle were considered. The state of cells in occupancy grid is determined by the DS that combines the Light Detection And Ranging (LiDAR) data. In [32], the authors proposed a real-time object detection algorithm. The DS is used to fuse the features extracted from sparse point clouds, and experimental results showed good efficiency in the urban traffic environments. However, most of these works focus on the classical DS combination scheme, which may result in erroneous output when the input confidences are highly conflicting.

To overcome the shortcoming of classical DS dealing with conflicting confidences, several distance metrics for evidence and improved DS fusion schemes were proposed. The metric called the Jaccard index, which is based on the cardinalities of the union and intersection of two subsets, is widely used [24]. In [27], a new distance measure which is more sensitive to change of evidence was presented. A pignistic probability transformation based distance measures were studied in [33,34]. Despite many attempts, distance metrics accounting for the asymmetry in the importance of sensed events were not fully studied. To incorporate new distance metrics in DS, modified DS fusion methods were proposed. In [25], a new weighted DS combination scheme was proposed where weights are calculated by distance measure. Moreover, in order to manage confidence conflicting, weighted combination schemes based on conventional distance with uncertainty [26] or ambiguity measures [35,36] were also proposed. However, the above state-of-the-art DS fusion schemes use a distance metric which assumes the evidences are of equal importance. Therefore, it is difficult to apply those fusion methods directly to the autonomous vehicle environment in which the evidences and related events potentially differ in their importance.

In [37], collision avoidance at intersections was studied with real-world experiments. V2V communication is used to share each vehicle’s information such as velocity, and then vehicle trajectories are estimated by Kalman filter. Based on trajectory estimation, each vehicle controls its velocity to avoid collision. In [38], a system for cooperative collision avoidance for overtaking scenario was proposed. For forward (rear-end) collision avoidance, the work in [5] uses deep learning, and the authors of [39] presented a collision avoidance scheme considering the measurement uncertainty in sensors. Although there exist many studies on collision prediction and avoidance, in-depth studies on data fusion for cooperative sensing in autonomous vehicles are necessary.

3. Proposed Real-Time Data Fusion Method Based on a New Distance Metric

Autonomous vehicles can exchange sensing data acquired from various sensors through V2V communication. In order to leverage sensing data from multiple sources in real-time, we require a data fusion system with low complexity and high reliability. In this section, we introduce an inter-vehicle data fusion method for identifying nearby objects.

We classify the type of sensing data into two categories: “object” and “trajectory”. The former is associated with the existence of an object and, given that the object exists, the class of the object. The latter is related with trajectory of an object such as position and velocity. According to these data types, we will adopt different data fusion schemes. We first discuss data fusion associated with “object” type data.

3.1. Data Fusion for Object Information

3.1.1. Dempster–Shafer Theory of Evidence

In general, with more data available, higher credibility in the prediction can be achieved. At the same time, under the autonomous vehicle environment, the decision should be made in real-time. In a multisensor environment, there exists uncertainty due to limited accuracy of sensor measurements or complicated sensing environment [15]. To address these issues, we adopt Dempster–Shafer theory of evidence, in short DS [17], as a tool for combining gathered data in order to improve detection accuracy by reinforcing the output based on the sensed data from peer vehicles. Misdetection of objects due to erroneous sensing can be avoided by fusing the data obtained from peers. Moreover, DS is effective in managing uncertainty in multisensor environments [15]. We begin by summarizing the mathematical framework of DS [16,19,20,40,41].

In DS, the frame of discernment [17] is the set of all possible mutually exclusive and complete states of system denoted by $\mathsf{\Theta }$. A Basic Probability Assignment (BPA) function $m(·)$ is a mass function which assigns confidence in each subset of $\mathsf{\Theta }$, i.e., $m:2^{\mathsf{\Theta }}\to [0,1]$. The BPA function satisfies the following conditions,

$$\begin{array}{cc} \hfill & m(\varnothing )=0, m\left(X\right)\ge 0, \forall X\subseteq \mathsf{\Theta },\hfill \end{array}$$

(1)

$$\begin{array}{cc} \hfill & \sum _{X\subseteq \mathsf{\Theta }}m\left(X\right)=1.\hfill \end{array}$$

(2)

In this paper, we will use term of confidences to indicate BPAs. The elements of the power set $2^{\mathsf{\Theta }}$ having a non-zero mass are called focal elements.

We combine multiple confidences into a single output using the DS rule of combination. Let us consider two confidences, $m_1\left(X\right)$ and $m_2\left(X\right)$, which believe X is true. The DS fuses these confidences [17], denoted by $m_1\left(X\right)\oplus m_2\left(X\right)$, as follows,

$$\begin{array}{c}{m}_1\left(X\right)\oplus m_2\left(X\right)=\frac{{\sum }_{Y\cap Z=X}{m}_1\left(Y\right)m_2\left(Z\right)}{1-{\sum }_{Y\cap Z=\varnothing }{m}_1\left(Y\right)m_2\left(Z\right)}.\hfill \end{array}$$

(3)

We determine the existence and class of an object by two consecutive stages of DS. Prior to classifying objects, we identify whether a detected object indeed exists or not, i.e., an inaccurate sensor can report that an object is detected even if there is nothing, and vice versa. Thus, we first identify the existence, and, if the existence of an object positively identified, second determine the class of the object. The process of the object information data fusion is depicted in Figure 1.

3.1.2. Data Fusion for the Existence of Objects

For the problem of determining existence, the frame of discernment is given by ${\mathsf{\Theta }}_E=\left\{\mathrm{existence},\mathrm{non}-\mathrm{existence}\right\}$. In this paper, we use notation $2^{{\mathsf{\Theta }}_E}$ to represent the powerset of ${\mathsf{\Theta }}_E$, i.e., $2^{{\mathsf{\Theta }}_E}=\{${existence}, {non-existence}, {uncertainty (=existence or non-existence)}} where the elements are denoted by E, N, and U, respectively. Let us define $m_k^i\left(X\right)$ as the confidence at vehicle i about object k that is committed to $X\in 2^{{\mathsf{\Theta }}_E}$, and $M_k^i=\{m_k^i\left(\mathrm{E}\right),m_k^i\left(\mathrm{N}\right),m_k^i\left(\mathrm{U}\right)\}$ is defined to be the set of confidences. For two vehicles i and j, we can calculate $m_k^i\left(X\right)\oplus m_k^j\left(X\right)$ for each element $X\in \{\mathrm{E},\mathrm{N},\mathrm{U}\}$ from (3) as follows,

$$\begin{array}{cc}{m}_k^i\left(\mathrm{E}\right)\oplus m_k^j\left(\mathrm{E}\right)=\hfill & \frac{m_k^i\left(\mathrm{E}\right)m_k^j\left(\mathrm{E}\right)+m_k^i\left(\mathrm{E}\right)m_k^j\left(\mathrm{U}\right)+m_k^i\left(\mathrm{U}\right)m_k^j\left(\mathrm{E}\right)}{1-\kappa },\hfill \\ m_k^i\left(\mathrm{N}\right)\oplus m_k^j\left(\mathrm{N}\right)=\hfill & \frac{m_k^i\left(\mathrm{N}\right)m_k^j\left(\mathrm{N}\right)+m_k^i\left(\mathrm{N}\right)m_k^j\left(\mathrm{U}\right)+m_k^i\left(\mathrm{U}\right)m_k^j\left(\mathrm{N}\right)}{1-\kappa },\hfill \\ m_k^i\left(\mathrm{U}\right)\oplus m_k^j\left(\mathrm{U}\right)=\hfill & \frac{m_k^i\left(\mathrm{U}\right)m_k^j\left(\mathrm{U}\right)}{1-\kappa },\hfill \end{array}$$

(4)

where $\kappa =m_k^i\left(\mathrm{E}\right)m_k^j\left(\mathrm{N}\right)+m_k^i\left(\mathrm{N}\right)m_k^j\left(\mathrm{E}\right)$. For instance, given $M_k^i=\{0.88,0,0.12\}$ and $M_k^j=\{0,0.7,0.3\}$, $m_k^i\left(\mathrm{E}\right)\oplus m_k^j\left(\mathrm{E}\right)$ is given by $\frac{0.88\times 0.3}{1-0.88\times 0.7}\approx 0.68$.

Below we define U or the “uncertainty” element, in a rigorous manner. There are two types of uncertainty. The first type is the uncertainty due to inherent limitation of a sensor. For example, a vision sensor cannot detect objects behind obstacles. As shown in Figure 2, the vehicle cannot recognize the existence of the completely occluded object unless a peer vehicle reports the object to that vehicle even if the object is within the sensing area. In this case, the confidence assignments by vehicle i about k is given by

$$\begin{array}{c}{m}_k^i\left(\mathrm{E}\right)=m_k^i\left(\mathrm{N}\right)=0, m_k^i\left(\mathrm{U}\right)=1.\hfill \end{array}$$

(5)

The second type of uncertainty is from the limited accuracy of sensors. The accuracy of a sensor may be known in advance from its manufacturer [42]. In addition, the confidence can be adjusted for the environment such as weather condition or the distance to the object [16]. For example, we can set the BPAs for clear and rainy days to $\{0.9,0,0.1\}$ and $\{0.9-\delta ,0,0.1+\delta \}$, respectively, where $\delta$ is a system parameter.

However, the classical DS combination may yield unreasonable outputs when evidences are highly conflicting. This problem is due to combining in (3) which ignores the conflicts in evidence. In order to illustrate this problem, let us consider the following example [19,40,43]. Given 3 events $Z=\{A,B,C\}$ and 2 Observers, Observer 1 and 2 assign $m_1\left(A\right)=0.9, m_1\left(B\right)=0.1, m_1\left(C\right)=0.0$ and $m_2\left(A\right)=0.0, m_2\left(B\right)=0.1, m_2\left(C\right)=0.9$ as confidences in Z, respectively. Although Observers are quite certain that either event A or C has occurred, the combination assigns the highest confidence to event B as $m_1\left(B\right)\oplus m_2\left(B\right)=1$. Therefore, in order to deal with highly conflicting evidences, we consider the evidence of distance in order to measure the degree of conflict among evidences [24,33].

3.1.3. Proposed Metric for Evidence Distance

Based on the confidence assignment, in Jousseleme’s approach [24], the similarity between two subsets is measured by their cardinality using the Jaccard index, $\frac{|A\cap B|}{|A\cup B|}$, and the distance increases with the degree of conflict between two sets of confidences. Specifically, given two confidences $m_1$ and $m_2$ on the same frame of discernment $\mathsf{\Theta }$, the Jousselme distance between them is defined as follows [24],

$$\begin{array}{c}{\overline{d}}_{1,2}=\sqrt{\frac{1}{2}{({\mathit{m}}_1-{\mathit{m}}_2)}^{\mathrm{T}}P({\mathit{m}}_1-{\mathit{m}}_2)},\hfill \end{array}$$

(6)

where ${\mathit{m}}_1$ and ${\mathit{m}}_2$ are $|2^{\mathsf{\Theta }}|\times 1$ vectors associated with $m_1$ and $m_2$, respectively, and P is $|2^{\mathsf{\Theta }}|\times |2^{\mathsf{\Theta }}|$ matrix whose elements are defined by

$$\begin{array}{c}P(A,B)=\frac{|A\cap B|}{|A\cup B|}, \forall A,B\in 2^{\mathsf{\Theta }}.\hfill \end{array}$$

(7)

The underlying premise of this approach is that elements of the frame of discernment are equally important. However, in navigational safety, false negative is significantly dangerous [44,45]. That is, the sensor did not report the existence of an object although the object existed. Therefore, we would like to put more emphasis on the confidence of E than the other elements. This results in an asymmetrical importance among the elements in the frame of discernment. In this paper, we propose a novel distance metric to measure conflict by applying asymmetric weights to each element in the frame of discernment, and the weighted DS combination [15,25,26,35] is considered to fuse the confidences according to the proposed conflict distance.

Definition 1.

For the proposed distance metric, we define the weighted cardinality for $A\in 2^{\mathsf{\Theta }}$ as follows,

$$\begin{array}{c}{\left|A\right|}_w:=\sum _{x\in A}w\left(x\right),\hfill \end{array}$$

(8)

where $w(·)$ is a weight function $w:\mathsf{\Theta }\to {\mathbb{R}}_{++}$.

Through $w(·)$, we can flexibly adjust the weights depending on the importance of the associated elements, i.e., a larger weight indicates a higher importance. By doing so, we can put asymmetric emphasis on the elements in the frame of discernment. Our approach generalizes the similarity measure used by the existing methods in which the volume of similarity is determined by the number of elements in subsets having identical weight [24,27]. We compute the weighted similarity matrix Q as follows,

$$\begin{array}{c}Q(A,B):=\frac{{|A\cap B|}_w}{{|A\cup B|}_w}=\frac{{\sum }_{x\in A\cap B}w\left(x\right)}{{\sum }_{x\in A\cup B}w\left(x\right)}, \forall A,B\in 2^{\mathsf{\Theta }}.\hfill \end{array}$$

(9)

For ${\mathsf{\Theta }}_E$, given the confidences of vehicle i and j about object k, the distance between them is defined by

$$\begin{array}{c}{d}_k^{ij}=\sqrt{\frac{1}{2}{({\mathit{m}}_k^i-{\mathit{m}}_k^j)}^{\mathrm{T}}Q({\mathit{m}}_k^i-{\mathit{m}}_k^j)},\hfill \end{array}$$

(10)

where ${\mathit{m}}_k^i$ is $3\times 1$ vector of ${\left(m_k^i\left(\mathrm{E}\right) m_k^i\left(\mathrm{N}\right) m_k^i\left(\mathrm{U}\right)\right)}^{\mathrm{T}}$. We show that the proposed distance in (10) satisfies all four conditions on distance metrics, which are non-negativity, nondegeneracy, symmetry, and triangle inequality [24].

Proposition 1.

The proposed distance $d_k^{ij}$ is a distance metric on $2^{{\mathsf{\Theta }}_E}$.

Proof of Proposition 1.

For $2^{{\mathsf{\Theta }}_E}$, we have 3 elements given by $\{\mathrm{E},\mathrm{N},\mathrm{U}\}$. Let us denote the weighted cardinalities corresponding to each element as $w\left(\mathrm{e}\right)$, $w\left(\mathrm{n}\right)$, and $w\left(\mathrm{u}\right)$, respectively, i.e., $w\left(\mathrm{e}\right):={\sum }_{x\in \mathrm{E}}w\left(x\right)$. We can compute the weighted similarity matrix Q by (9), and suppose that the matrix Q is partitioned into the form such that

$$\begin{array}{c}Q=\left[\begin{array}{ccc}1& 0& Q(\mathrm{E},\mathrm{U})\\ 0& 1& Q(\mathrm{N},\mathrm{U})\\ Q(\mathrm{E},\mathrm{U})& Q(\mathrm{N},\mathrm{U})& 1\end{array}\right]=\left[\begin{array}{cc}A& B\\ B^{\mathrm{T}}& C\end{array}\right], \mathrm{where} A=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right],B=\left[\begin{array}{c}Q(\mathrm{E},\mathrm{U})\\ Q(\mathrm{N},\mathrm{U})\end{array}\right].\hfill \end{array}$$

(11)

Let $Q/A$ be the Schur complement of A in Q, which is defined by $C-B^{\mathrm{T}}{A}^{-1}B$. Then, Q is positive definite if and only if the following properties hold,

$$\begin{array}{c}C1:A\succ 0, C2:Q/A=C-B^{\mathrm{T}}{A}^{-1}B\succ 0.\hfill \end{array}$$

(12)

$C1$ is satisfied by property of identity matrix A. $C2$ can be expressed as

$$\begin{array}{cc} \hfill & 1>Q{(\mathrm{E},\mathrm{U})}^2+Q{(\mathrm{N},\mathrm{U})}^2\hfill \end{array}$$

$$\begin{array}{ll}\Rightarrow \hfill & 1>{\left(\frac{{\sum }_{x\in \mathrm{E}\cap \mathrm{U}}w\left(x\right)}{{\sum }_{x\in \mathrm{E}\cup \mathrm{U}}w\left(x\right)}\right)}^2+{\left(\frac{{\sum }_{x\in \mathrm{N}\cap \mathrm{U}}w\left(x\right)}{{\sum }_{x\in \mathrm{N}\cup \mathrm{U}}w\left(x\right)}\right)}^2={\left(\frac{w\left(\mathrm{e}\right)}{w\left(\mathrm{u}\right)}\right)}^2+{\left(\frac{w\left(\mathrm{n}\right)}{w\left(\mathrm{u}\right)}\right)}^2\hfill \end{array}$$

(13)

$$\begin{array}{cc}\Rightarrow \hfill & {\left(w\left(\mathrm{e}\right)+w\left(\mathrm{n}\right)\right)}^2>w^2\left(\mathrm{e}\right)+w^2\left(\mathrm{n}\right).\hfill \end{array}$$

(14)

The relation (13) can be derived by Definition 1 in (8) and $\mathrm{E}\cup \mathrm{N}=\mathrm{U}$. Inequality (14) holds due to $w\left(\mathrm{u}\right)=w\left(\mathrm{e}\right)+w\left(\mathrm{n}\right)$, and $w(·)>0$. Thus, (10) is an induced norm [24,46], which implies that it satisfies all of four conditions on the metric distance. □

We illustrate an example of the proposed distance metric in Figure 3. In both cases, Observers 1 and 2 have the same Jousselme distance given by 0.7649 due to the symmetry of confidences of E and N. In our approach, given the weights as $w\left(\mathrm{e}\right)=2$, $w\left(\mathrm{n}\right)=1$, and $w\left(\mathrm{u}\right)=3$, we obtain new distance $d_k^{12}$ for each case as 0.7147 and 0.8124, respectively. The result can be explained as follows. We assign high importance to “existence” by $w\left(\mathrm{e}\right)>w\left(\mathrm{n}\right)$. For the case of $m_k^1\left(\mathrm{E}\right)=0.9$ and $m_k^2\left(\mathrm{N}\right)=0.6$, the level of conflict between confidences is high, but Observer 1 is highly confident of existence, i.e., $m_k^1\left(\mathrm{E}\right)=0.9$. In this case, it is regarded that two confidence vectors have a shorter distance than that by the conventional metric. Contrary to this, Observer 1 in case 2 has relatively low confidence in existence than case 1, i.e., $m_k^1\left(\mathrm{E}\right)=0.6$. In spite of the same degree of conflict, by putting more emphasis on E, the dissimilarity between the confidences become greater. Therefore, we obtain the distance in case 2 given by $0.8124>0.7649$, which will affect the result of data fusion. Next, we explain how the existence of the object is determined based on the proposed distance.

3.1.4. Proposed Data Fusion for Determining Existence

Based on the proposed distance, we leverage technique of the weighted DS combination rule for data fusion [15,23,26,35]. Let us define ${\mathcal{M}}_i$ as the set of vehicles who can communicate with vehicle i and i itself, and ${\mathcal{I}}_i$ as the set of detected objects which are located in vehicle i’s sensing area. Denote the set of vehicles associated with vehicle i who can detect object k by ${\mathcal{M}}_{i\left(k\right)}=\{j\in {\mathcal{M}}_i|k\in {\mathcal{I}}_j\}$. Suppose the vehicle i currently receives n-th data point. Define the set of vehicles from which vehicle i has received sensing data on object k including i as ${\mathcal{M}}_{i\left(k\right)}\left(n\right)$. We have that $|{\mathcal{M}}_{i\left(k\right)}\left(n\right)|=n+1$. Without loss of generality (WLOG), we assume that j-th data is transmitted from vehicle $j\in {\mathcal{M}}_{i\left(k\right)}\setminus \left\{i\right\}$. Vehicle i calculates distances by (10) between $m_k^n$ and $m_k^l,\forall l\in {\mathcal{M}}_{i\left(k\right)}(n-1)$, and we define $n\times 1$ similarity matrix [23,25,26], which is given by

$$\begin{array}{c}S{M}_k^{ni}=\mathbf{1}-{[d_k^{ni} d_k^{n1} d_k^{n2} \cdots d_k^{nn-1}]}^{\mathrm{T}}.\hfill \end{array}$$

(15)

where $\mathbf{1}$ is all-ones matrix. Due to symmetry property from Proposition 1, we have $d_k^{ij}=d_k^{ji},\forall j=1,\cdots ,n$, and thus $S^i\left(n\right)$ is a symmetric matrix. Based on (15), we can construct n-th block matrix $S^i\left(n\right)$ such that

$$\begin{array}{c}{S}^i\left(n\right)=\left[\begin{array}{cc}{S}^i(n-1)& S{M}_k^{ni}\\ {\left(S{M}_k^{ni}\right)}^{\mathrm{T}}& 1\end{array}\right].\hfill \end{array}$$

(16)

Let us denote $(p,q)$ element of $S^i\left(n\right)$ as $S_{(p,q)}^i\left(n\right)$. We define $(n+1)\times 1$ supplementary matrix $SU$ and credibility matrix $CR$ [23,26], respectively, where p-th elements are defined by

$$\begin{array}{c}S{U}_p=\sum _{q=1,q\ne p}^{n+1}{S}_{(p,q)}^i\left(n\right), C{R}_p=\frac{S{U}_p}{{\sum }_{p=1}^{n+1}S{U}_p},\forall p=1,\cdots ,n+1.\hfill \end{array}$$

(17)

Note that $p=1$ represents the values from vehicle i, and the derived $C{R}_p$ can be regarded as the weights on $(p-1)$-th received confidence for all $p\in [2,n+1]$. Let us define $3\times 1$ weighted confidence vector about object k at vehicle i as ${\tilde{\mathit{m}}}_k^i$. We calculate ${\tilde{\mathit{m}}}_k^i$ based on (17) such that

$$\begin{array}{cc}{\tilde{\mathit{m}}}_k^i\hfill & =(C{R}_1\times {\mathit{m}}_k^i)+\sum _{l=1}^n(C{R}_{l+1}\times {\mathit{m}}_k^l).\hfill \end{array}$$

(18)

The elements in (18) are the weighted confidences of E, N, and U. We define the set of elements of ${\tilde{\mathit{m}}}_k^i$ as ${\tilde{M}}_k^i=\{{\tilde{m}}_k^i\left(\mathrm{E}\right),{\tilde{m}}_k^i\left(\mathrm{N}\right),{\tilde{m}}_k^i\left(\mathrm{U}\right)\}$. Next, we can apply the classical DS procedure in (3) to fuse ${\tilde{M}}_k^i$ for $n-1$ times [25,26,35,47]. The algorithm for existence calculation is presented in Algorithm 1.

Algorithm 1 Existence calculation for i about object k

1: function$[\mathcal{E},\mathcal{N},\mathcal{U}]=$ Existence $(S^i(n-1),M_k^j,\forall j\in {\mathcal{M}}_{i\left(k\right)}\left(n\right))$ 2: Initialize: $\mathcal{E},\mathcal{N},\mathcal{U}\leftarrow 0$ 3: $d_k^{nj}\leftarrow$ Calculate conflict distance between ${\mathit{m}}_k^n$ and ${\mathit{m}}_k^j$, $\forall j\in {\mathcal{M}}_{i\left(k\right)}\left(n\right)\setminus \left\{n\right\}$ from (10) 4: $S{M}_k^{ni}\leftarrow \mathbf{1}-{[d_k^{ni} d_k^{n1} d_k^{n2} \cdots d_k^{nn-1}]}^{\mathrm{T}}$   5: $S^i\left(n\right)\leftarrow \left[\begin{array}{cc}{S}^i(n-1)& S{M}_k^{ni}\\ {\left(S{M}_k^{ni}\right)}^{\mathrm{T}}& 1\end{array}\right]$   6: Calculate ${\tilde{\mathit{m}}}_k^i$ based on $S^i\left(n\right)$ by (17) and (18) 7: $\{\mathcal{E},\mathcal{N},\mathcal{U}\}\leftarrow \{{\tilde{m}}_k^i\left(\mathrm{E}\right),{\tilde{m}}_k^i\left(\mathrm{N}\right),{\tilde{m}}_k^i\left(\mathrm{U}\right)\}$ 8: for$t=1,\cdots ,n$do 9:  $\mathcal{E}\leftarrow \mathcal{E}\oplus {\tilde{m}}_k^j\left(\mathrm{E}\right)$, $\mathcal{N}\leftarrow \mathcal{N}\oplus {\tilde{m}}_k^j\left(\mathrm{N}\right)$, $\mathcal{U}\leftarrow \mathcal{U}\oplus {\tilde{m}}_k^j\left(\mathrm{U}\right)$ 10: end for 11: return$\{\mathcal{E},\mathcal{N},\mathcal{U}\}$ 12: end function

Note that the proposed scheme has polynomial time complexity, which enables its real-time data fusion. The DS rule in general has high computation complexity for data fusion, $O\left(2^{2\left|\mathsf{\Theta }\right|}\right)$, which is a main drawback. However, we have 3 elements in $2^{{\mathsf{\Theta }}_E}$, and thus the complexity of calculating the DS rule is $O\left(1\right)$. Specifically, the vehicles which detect object k participate in the data fusion, thus vehicle i can obtain the fused output with $O\left(\right|{\mathcal{M}}_{i\left(k\right)}\left|\right)$ complexity for each time instant, which in turn we have $O\left(N\right)$ complexity in the worst case due to $|{\mathcal{M}}_{i\left(k\right)}|\le N$. Furthermore, the considered data fusion scheme is valid irrespective of the order of data arrivals as well.

Proposition 2.

Given a set of data points, let ${\mathsf{\Pi }}_u$ and ${\mathsf{\Pi }}_v$ be the arbitrary order of data points where ${\mathsf{\Pi }}_u\ne {\mathsf{\Pi }}_u$ for $u\ne v$, and we denote corresponding outputs as ${\tilde{M}}_k^i\left({\mathsf{\Pi }}_u\right)$ and ${\tilde{M}}_k^i\left({\mathsf{\Pi }}_v\right)$, respectively. Then, we have ${\tilde{M}}_k^i\left({\mathsf{\Pi }}_u\right)={\tilde{M}}_k^i\left({\mathsf{\Pi }}_v\right)$.

Proof of Proposition 2.

Suppose that vehicle i receives n-th data. We suppose that there are different data arrival orders which are denoted by ${\mathsf{\Pi }}_u$ and ${\mathsf{\Pi }}_v$, $|{\mathsf{\Pi }}_u|=|{\mathsf{\Pi }}_v|=n$. The hat and check symbols denote the notations related to ${\mathsf{\Pi }}_u$ and ${\mathsf{\Pi }}_v$, respectively, e.g., ${\widehat{S}}^i\left(n\right)$ and ${\check{S}}^i\left(n\right)$ are similarity block matrices in (16) corresponding to ${\mathsf{\Pi }}_u$ and ${\mathsf{\Pi }}_v$. WLOG, we focus on the data comes from vehicle j. Let us define the order of this data in ${\mathsf{\Pi }}_u$ and ${\mathsf{\Pi }}_v$ as ${\mathsf{\Pi }}_u^j$ and ${\mathsf{\Pi }}_u^j$, respectively. For ${\mathsf{\Pi }}_u^j$-th data in ${\mathsf{\Pi }}_u$, we can express the associated element in supplementary matrix such that

$$\begin{array}{cc}{\widehat{SU}}_{{\mathsf{\Pi }}_u^j+1}\hfill & =\sum _{q=1,q\ne {\mathsf{\Pi }}_u^j+1}^{n+1}{\widehat{S}}_{({\mathsf{\Pi }}_u^j+1,q)}^i\left(n\right)\hfill \end{array}$$

(19)

$$\begin{array}{cc} \hfill & =(1-{\widehat{d}}_k^{{\mathsf{\Pi }}_u^{j}i})+\sum _{q<{\mathsf{\Pi }}_u^j}(1-{\widehat{d}}_k^{{\mathsf{\Pi }}_u^{j}q})+\sum _{q>{\mathsf{\Pi }}_u^j}(1-{\widehat{d}}_k^{q{\mathsf{\Pi }}_u^j})\hfill \end{array}$$

(20)

$$\begin{array}{cc} \hfill & =(1-{\widehat{d}}_k^{{\mathsf{\Pi }}_u^{j}i})+\sum _{q=1,q\ne {\mathsf{\Pi }}_u^j}^n(1-{\widehat{d}}^{{\mathsf{\Pi }}_u^{j}q})\hfill \end{array}$$

(21)

$$\begin{array}{cc} \hfill & =(1-{\check{d}}_k^{{\mathsf{\Pi }}_v^{j}i})+\sum _{q=1,q\ne {\mathsf{\Pi }}_v^j}^n(1-{\check{d}}^{{\mathsf{\Pi }}_v^{j}q})={\check{SU}}_{{\mathsf{\Pi }}_v^j+1}\hfill \end{array}$$

(22)

The equalities in (19) and (20) come from the definitions in (15) and (17), respectively. The third equality in (21) holds due to the symmetry of the proposed distance. Finally, we can derive the last equality, as the both ${\mathsf{\Pi }}_u^j$-th and ${\mathsf{\Pi }}_v^j$-th data refer to the same information from vehicle j. Therefore, we have that

$$\begin{array}{c}{\widehat{CR}}_{{\mathsf{\Pi }}_u^j+1}\times {\widehat{m}}_k^{{\mathsf{\Pi }}_u^j}={\check{CR}}_{{\mathsf{\Pi }}_v^j+1}\times {\check{m}}_k^{{\mathsf{\Pi }}_v^j}.\hfill \end{array}$$

(23)

We can apply this process to all the data points, which results in the same weighted output irrespective of any arrival order. □

We can extend Proposition 2 for any order of data points, which results in that given a set of data points, we have the same final output irrespective of the order of fusion. This property guarantees an independence of the data fusion scheme. In other words, as it is a data-driven fusion method, we can focus on the data synthesis rather than external influence such as channel condition, etc. Finally, we determine the existence of object if the calculated result from Algorithm 1 is greater than the predefined threshold $h_e$, $\mathcal{E}\ge h_e$. Next, we proceed to the decision-making for object classification.

3.1.5. Proposed Data Fusion for Object Classification

Next, we consider the object classification process. The frame of discernment for classification is given by the set of objects in the roads, e.g., ${\mathsf{\Theta }}_C$={Sedan, SUV, Pedestrian, Bicycle,$\cdots \}$. Let $c_i^k\left(X\right)$ be the output from object detectors of vehicle i using about class X for object k. We define the set of outputs as $C_k^i=\{c_k^i\left(X\right),\forall X\subseteq {\mathsf{\Theta }}_C\}$. Any object detector providing confidence value for each element in $C_k^i$ can be applied, e.g., deep neural networks [48], classified vector quantization [49], and k-nearest neighbor [50]. In general, the outputs from the object detection system are not suitable for direct use as confidences [51]. Instead, we apply temperature scaling technique [51] to classifier output, and the calibrated result ${\overline{c}}_k^i(·)$ is defined by

$$\begin{array}{c}{\overline{c}}_k^i\left(X\right)=\frac{\exp (c_k^i\left(X\right)/T)}{{\sum }_{Y\in 2^{{\mathsf{\Theta }}_C}}\exp (c_k^i\left(Y\right)/T)}, \forall X\in 2^{{\mathsf{\Theta }}_C},\hfill \end{array}$$

(24)

where temperature T is a control parameter. We can tune the deviation among confidences by adjusting the parameter T. This adjustment allows the result to be “soft” while preserving the existing confidence [52]. The classifier assigns confidence (BPA) to each element of ${\mathsf{\Theta }}_C$, and it satisfies the conditions in (1) and (2). It is reasonable for a classifier to produce distinct probabilities for each candidate class instead of a single output across multiple classes. Therefore, we have the same number of focal elements to $|{\mathsf{\Theta }}_C|$.

Next, we will calculate the distance of confidences as in (10) using calibrated confidences. Assume that vehicle i receives n-th data point. Note that only the vehicles who identify “existence” with respect to object k can have the corresponding class confidences, which is defined by $E_k^i=\left\{j\right|m_k^j\left(\mathrm{E}\right)>0,\forall j\in {\mathcal{M}}_{i\left(k\right)}\left(n\right)\}$. The same procedure in (15)–(18) is applied to obtain the weighted confidences, and the DS combination is performed with respect to the weighted confidences for $|E_k^i|-1$ times. Let us denote the obtained DS combination results for classification as $W_k^i\left(X\right),\forall X\in 2^{{\mathsf{\Theta }}_C}$. Finally, we choose the class with the highest confidence in the estimated class given by

$$\begin{array}{cc}C{L}_k^i\hfill & =\underset{X\in 2^{{\mathsf{\Theta }}_C}}{\arg \max }{W}_k^i\left(X\right)\hfill \end{array}$$

(25)

where $C{L}_k^i$ is the class decided at vehicle i about object k. The classification process is presented in Algorithm 2.

Algorithm 2 Classification fusion for vehicle i about object k

1: function$\left[\mathcal{C}\right]=$ Class $(C_k^i,\forall i\in E_k^i)$ 2: Initialize: $\mathcal{C},\mathcal{W}=\varnothing$, 3: for$i\in E_k^i$do 4:  ${\overline{c}}_k^i\left(X\right)\leftarrow \frac{exp(c_k^i\left(X\right)/T)}{{\sum }_{j=1}^{|{\mathsf{\Theta }}_C|}exp(c_k^j\left(X\right)/T)}, \forall X\in 2^{{\mathsf{\Theta }}_C}$ 5: end for 6: ${\tilde{c}}_k^i\left(X\right)$← Calculate the weighted confidences according to (10) and (15)–(18) based on ${\overline{c}}_k^i\left(X\right)$, $\forall X\in 2^{{\mathsf{\Theta }}_C}$ 7: for$j=1,\cdots ,|E_k^i|-1$do 8:  $W_k^i\left(X\right)\leftarrow W_k^i\left(X\right)\oplus {\tilde{c}}_k^j\left(X\right)$, $\forall X\in 2^{{\mathsf{\Theta }}_C}$ 9: end for 10: $\mathcal{C}\leftarrow {\arg \max }_{X\in 2^{{\mathsf{\Theta }}_C}}{W}_k^i\left(X\right)$ 11: return$\mathcal{C}$ 12: end function

We can show that the data fusion for classification also has polynomial time complexity as follows. Under the classification fusion, as explained previously, we have independent confidences for each class. Consequently, the valid power set of ${\mathsf{\Theta }}_C$ is equal to ${\mathsf{\Theta }}_C$, which results in $O\left(\right|{\mathsf{\Theta }}_C{|}^2)$ complexity to calculate the distance between confidences. Therefore, in total, the complexity of the class fusion process is $O\left(\right|E_k^i\left|\right|{\mathsf{\Theta }}_C{|}^2)$.

The integrated decision-making is outlined in Algorithm 3. The algorithm uses functions of Existence and Class from Algorithms 1 and 2. Given the collected data, vehicle i first validates the existence of the reported object (step 3). If the existence is confirmed (step 4), the algorithm determines the class of the detected object by fusing the shared confidences (step 5). An example is illustrated in Figure 4. There are 3 vehicles, V1–V3, and an object k which is within the sensing ranges of them, e.g., ${\mathcal{M}}_{\mathrm{V}1\left(k\right)}=\left\{\mathrm{V}1,\mathrm{V}2,\mathrm{V}3\right\}$. Unlike V2 and V3, it is impossible for V1 to detect k, as the object is located in the uncertainty area of V1. Moreover, we assume that V3 fails to detect the object due to, e.g., bad weather or sensor malfunctioning. We set the confidences for each vehicle as in the tables in Figure 4. As only V2 recognizes the object, it has class confidences, i.e., $E_k^1=\left\{\mathrm{V}2\right\}$. In this case, through inter-vehicle communications, V1 receives the detection results, then verify the existence of the object. From Algorithm 3, V1 can derive the final outputs for existence and class as $\{0.58,0.34,0.08\}$ and $\{0.5,0.31,0.19\}$, respectively. We set $h_e=0.5$, thus V1 is able to not only indirectly detect the presence of the object, but also find the class of the object.

From the experiments in Section 5, we will show that the proposed method operates in real-time, and thus is suitable for the autonomous vehicle environment. Next, we discuss data fusion algorithm for “trajectory” type data.

Algorithm 3 Decision-making for vehicle i about object k

1: Given $M_k^j$, $C_k^j$, $\forall j\in {\mathcal{M}}_{i\left(k\right)}\left(n\right)$, $S^i(n-1)$, $h_e$ 2: Derive $E_k^i$ based on $M_k^j$ and ${\mathcal{M}}_{i\left(k\right)}\left(n\right)$ 3: $[\mathcal{E},\mathcal{N},\mathcal{U}]=$ EXISTENCE $(S^i(n-1),M_k^j,\forall j\in {\mathcal{M}}_{i\left(k\right)}\left(n\right))$ 4: if$\mathcal{E}\ge h_e$then 5:  $\left[\mathcal{C}\right]=$ CLASS $(C_k^i,\forall i\in E_k^i)$ 6: end if

3.2. Data Fusion for Trajectory Information

The fusion of trajectory information of the object requires a different approach from object type information. The trajectory information for the detected object consists of the measured position, velocity, and the measurement accuracy of the sensor. As explained previously, sensors have an inherent limitation in accuracy. We will use a “multisensor weighted data fusion” technique [53,54,55] so as to combine the reported sensing results and accuracy. The technique provides a fast and robust scheme to overcome the dynamic changes in the vehicular environment.

Let us define $v_k^j$ and ${\gamma }_k^j$ as the velocity of object k measured by vehicle j and the associated accuracy, respectively. These data are transmitted to nearby vehicles through V2V communications. We denote $f_k^i$ as the fused output based on the data gathered at vehicle i about object k. As in the case of classification, only the vehicles with $m_k^j\left(\mathrm{E}\right)>0,\forall j\in {\mathcal{M}}_{i\left(k\right)}\left(n\right)$ have the data on object k. Thus, we have that

$$\begin{array}{c}{f}_k^i=\sum _{j\in E_k^i}{w}_k^j{v}_k^j\hfill \end{array}$$

(26)

where $w_k^j$ is the weight of $v_k^j$, and ${\sum }_{j\in E_k^i}{w}_k^j=1$. We assume that the sensor measurements are independent and follow a normal distribution with standard deviation ${\gamma }_k^j$ (measurement accuracy) [56,57]. According to the extreme value theory of multivariate function [54,55,58], we can obtain the weights corresponding to the minimum mean square error as below.

$$\begin{array}{c}{w}_k^j=\frac{1}{{\left({\gamma }_k^j\right)}^2{\sum }_{l\in E_k^i}\frac{1}{{\left({\gamma }_k^l\right)}^2}}.\hfill \end{array}$$

(27)

The weights in (27) can be interpreted that large variance (low accuracy) results in low weight on measurements. Note that this scheme does not require any prior information, and computing (26) has polynomial time complexity, $O\left(\right|E_k^i\left|\right)$. In our experiment, on Intel(R) Core(TM) Xeon 2.4 GHz, it takes only $7\times {10}^{-6}$ s on average in combining 1000 points of velocity data, which is a negligible delay. Likewise, we can combine the location data as well, which are used to estimate the object trajectories.

4. Proposed Collision Warning System and V2V Communications

In this section, we propose a collision warning system and V2V communications integrated with the proposed multisource inter-vehicle data fusion methods.

4.1. Overall System Model

For autonomous vehicles, it is necessary for the embedded collision warning system to quickly perceive the surrounding environment and make decisions. In Figure 5, we illustrate an overall architecture of the proposed collision warning system. We summarize the components of the system as follows. Each vehicle obtains raw data from the equipped sensors such as cameras and radars. Through the object detection process, the object class and its confidences are extracted from the raw data. The trajectory information such as location and velocity are tagged to the object information in real-time. Next, we propose a compact message format for communications to achieve low latency. The encapsulated messages is shared via V2V communication, which provides high data rates and reliability such as DSRC. Next, the aforementioned data fusion algorithms are applied to determine the states of the detected objects. Based on them, the future trajectories of the detected objects are estimated by Kalman filter with cosine effect compensation [59,60]. From the estimated trajectories, the proposed system warns an imminent collision. In this paper, we mainly consider collision warning protocol of the system in detail.

4.2. Object Sensing

Let us consider methods to obtain object type data. Recently, LiDAR-based autonomous vehicle systems have been investigated [31,32,61]. However, due to its high cost [62], most of commercial vehicles such as Tesla and Mercedes-Benz adopt the computer vision as primary object detection rather than LiDAR [45]. Therefore, in this paper, we mainly consider the vision-based object detection system [63,64].

In addition to object type data, trajectory data including location and velocity can be obtained through the radar [42]. However, to obtain precise measurements, the system requires a supplemental calibration process to compensate measurement errors caused by “cosine effect” [59,60,65]. The radar can measure relative speed of the object precisely if the target moves directly in a direction of radar signal. Otherwise, the measured value does not coincide with the actual velocity. Let ${\varphi }_k^i$ be the angle between the direction of vehicle i’s radar signal and moving direction of object k. Note that information about object k can be obtained by utilizing historic trajectories. We denote the the measured and compensated velocities about object k for i as $V_M^{i,k}$ and $V_C^{i,k}$, respectively. Then, we can calculate the compensated value by $V_C^{i,k}=\frac{V_M^{i,k}}{cos{\varphi }_k^i}$. In the following, we describe how to transmit these data.

4.3. Message Format

The proposed data format is shown in Figure 6. It is composed of the number of detected objects (N), sensor accuracies (A), confidence (L), and trajectory data. The sensor accuracy is in 4-byte floating point format which contains vision sensor accuracy. The rest are used to set uncertainty of confidences and calculate (26) and (27). Next, the size of confidence part varies depending on the predetermined list of classes, ${\mathsf{\Theta }}_C$, and we allocate 2 bytes of short data type for each class. The position of the detected object is 24 bytes long consisting of longitude $\left(x\right)$, latitude $\left(y\right)$, and height $\left(z\right)$, which follows the GPS coordination. We use the float data type of 4 bytes for conveying velocity information. The aforementioned field is repeated for N times, thus the size of the entire packet is given by $13+N(2\times L+28)$ bytes.

Next, we consider the latency of transmitting vehicle information in the proposed message format. Suppose we use a standard SAE 2735 Basic Safety Message (BSM) format [66] which contains ID, position, speed, heading, etc. We will set the basic BSM to be 500 bytes long, which consists of 300 bytes for the default size of BSM [67,68] plus reserved space for other application such as high-end security. If the system classifies 16 categories and detects 20 objects $(L=16,N=20)$, the proposed entire safety message is about 1.7 Kbytes, which takes only a few milliseconds in existing V2V communications such as DSRC [11].

4.4. V2V Communication Systems

In this section, we investigate V2V communication system to share the data. Specifically, we define a V2V communication process to maintain the information up to date as possible, and take full advantage of the previously presented data fusion scheme.

Each vehicle is equipped with cameras that record surroundings in F frame-per-second (FPS). We denote k-th video frame as $F_k$. For a general video, F is fixed, e.g., $F=30$. In each frame, the vehicle performs object detection, and the result is broadcast periodically every 100 ms according to the standard [69]. Likewise, the detection results of other vehicles arrive periodically. For vehicle i, the data fusion output with respect to k-th frame and set of vehicles from which the data is successfully received, X, is denoted by ${\mathsf{\Gamma }}_X^{i_k}$. We describe the proposed data communication process with an example presented in Figure 7 as follows.

(i)

Data transmission: at each round, the vehicle transmits the most recently obtained result rather than the combined one, i.e., for the second transmission, V1 sends ${\mathsf{\Gamma }}_{\left\{1\right\}}^{1_4}$ instead of ${\mathsf{\Gamma }}_{\{1,2,3\}}^{1_4}$. This is because of protecting originality and avoiding data contamination.

(ii)

Data fusion and update: for every frame, as soon as the vehicle acquires the sensing result, the data combination between the acquired and received data is performed using the data fusion method proposed in Section 3. In this process, note that every received data has an expiration time of 100 ms, i.e., if a new data is not received within 100 ms from the same vehicle, the data is deleted. This case implies that either the transmitted message is obstructed or the vehicle has left the area of interest. In both cases, it can be considered that the data is invalid. For instance, between $F_6$ and $F_7$, V1 updates its data fusion output as ${\mathsf{\Gamma }}_{\{1,3\}}^{1_6}$ rather than ${\mathsf{\Gamma }}_{\{1,2,3\}}^{1_6}$ as there is no new data from V2 and the stored data is expired.

In order to implement this procedure in practice, data fusion should be completed within a short time. Nevertheless, the proposed data fusion method is feasible due to low complexity. Furthermore, from Proposition 2, the vehicle no longer has to consider the order of fusion.

4.5. Collision Determination

The fused output of the trajectory information can be applied to estimate the future trajectory of the detected objects. However, in practice, as the measurement has certain level of noise, it is hard to predict trajectories of vehicles correctly. Therefore, we employ a Kalman filter for noise mitigation and precise estimation [70]. We assume that the sensors monitor nearby objects with $t_V$ time interval which is identical to regular video frame. For simplicity, we consider 2-dimensional case mainly in this section, and it can be extended to 3-dimension simply with the same manner. The state and control input of Kalman filter at time t are defined by $x\left[t\right]$ and $u\left[t\right]$, respectively, which consists of x-y coordinates of positions $\left(s\right)$, velocities $\left(v\right)$, and accelerations $\left(a\right)$ such that

$$\begin{array}{cc}x\left[t\right]\hfill & ={(s_x\left[t\right] s_y\left[t\right] v_x\left[t\right] v_y\left[t\right])}^{\mathrm{T}}, u\left[t\right]={(a_x\left[t\right] a_x\left[t\right])}^{\mathrm{T}}.\hfill \end{array}$$

(28)

Let us denote the measurements of position and velocity at time t as $y\left[t\right]$. Then, the process dynamics can be expressed as

$$\begin{array}{c}x[t+1]=\left[\begin{array}{cccc}1& 0& t_V& 0\\ 0& 1& 0& t_V\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]x\left[t\right]+\left[\begin{array}{cc}\frac{{\left(t_V\right)}^2}{2}& 0\\ 0& \frac{{\left(t_V\right)}^2}{2}\\ 1& 0\\ 0& 1\end{array}\right]u\left[t\right]+N_p\left[t\right], y\left[t\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]x\left[t\right]+N_m\left[t\right],\hfill \end{array}$$

(29)

where $N_p\left[t\right]$ and $N_m\left[t\right]$ represent process and measurement noises which are white and independent normal distributions with zero mean and noise covariance matrices Q and R, respectively, i.e., $N_p\sim \mathcal{N}(0,Q)$ and $N_m\sim \mathcal{N}(0,R)$. From (28) and (29), we can estimate trajectory of objects precisely [71].

Next, in order to determine collision, we define the “safety area” of the objects [31,72,73], which is a circle of radius $R_s$ as depicted in Figure 8. Note that under the proposed system, we can determine collisions precisely by utilizing class information. We decide an accident when safety areas of objects and ego-vehicle are overlapped. In other words, along with the estimated trajectories, vehicles examine whether the safety areas overlap, and the anti-collision system would be triggered if an collision is predicted.

5. Simulation and Experimental Results

In this section, we evaluate the performance of the proposed systems. First, we investigate the latency and improvement of the proposed data fusion schemes. Second, we test for accident scenarios using real-world data. The first scenario concerns predicting accidents caused by blind spots, which can be avoided using V2V communications. The second scenario shows how an erroneous detection can be corrected.

5.1. Simulation Results: Performance Evaluation of Data Fusion Schemes

We first evaluate the delays for the proposed data fusion schemes in Section 3. The simulations are performed on Intel(R) Core(TM) Xeon 2.4 GHz using MATLAB R2014a. We calculate average latency of performing Algorithms 1 and 2, and trajectory data fusion in (26) and (27) to synthesize randomly generated 10 data points for 1000 iterations. Specifically, to satisfy conditions in (1) and (2), the confidence for existence, $m\left(\mathrm{E}\right)$, is modeled by normal distribution $\mathcal{N}(0.5,0.03)$, and the rest are divided into equally for “nonexistence” and “uncertainty”, i.e., $m\left(\mathrm{N}\right)=m\left(\mathrm{U}\right)=\frac{1-m\left(\mathrm{E}\right)}{2}$. We consider 16 classes, and the corresponding confidences are also generated according to $\mathcal{N}(0.5,0.03)$. As a representative, velocity is mainly considered for trajectory type data, which follows $\mathcal{N}(70,0.01)$ [74]. The results are presented in

Table 1. Time consumptions of the proposed data fusion schemes for 10 data points.

Data Type	Object		Trajectory
	Existence	Classifiction
Value [s]	$4.8\times {10}^{-4}$	$5.0\times {10}^{-4}$	$2.8\times {10}^{-6}$

.

Obviously, the data fusion for object type data takes larger calculation time compared to that of trajectory type. Nevertheless, the latency is only about 1 ms for entire data fusion processes, which demonstrates that the proposed data fusion scheme can operate in real-time.

Next, we compute the performance of the proposed scheme in terms of the false negative rate (FNR) defined by

$$\begin{array}{c}\mathrm{FNR}=\frac{\mathrm{Number} \mathrm{of} \mathrm{false} \mathrm{negatives}}{\mathrm{Number} \mathrm{of} \mathrm{false} \mathrm{negatives} + \mathrm{Number} \mathrm{of} \mathrm{true} \mathrm{positives}}\hfill \end{array}$$

where “true positive” and “false negative” are existence and nonexistence decisions conditional on the existence of object, respectively. We assume that there is an object and 10 vehicles. Each vehicle has one sensor which is in either normal or defective state. Depending on the state of the sensor, vehicles allocate the confidence in existence or nonexistence according to normal distribution with $\mathcal{N}(0.7,0.09)$. The rest are divided into other elements uniformly to satisfy the condition in (2), e.g., a vehicle i with normal sensor can have $M_k^i=\{0.8,0.1,0.1\}$.

In Figure 9, we plot the average FNR values for 10,000 iterations by changing the number of normal sensors. We compare the proposed scheme with classic DS, the weighted DS based on Jousselme distance [25], and the modified DS scheme presented in [26]. As shown in Figure 9, in all cases, the proposed schemes outperform the other data fusion methods in terms of reducing FNR. Specifically, for five normal sensors case, our method can reduce FNR about 18% compared to all the baselines. In addition, the proposed scheme achieves FNR reductions over classic DS (64.8%), Ref. [25] (50.9%), and [26] (65.5%) for seven normal sensors case. This result shows that we can improve the driver’s safety, even if there are high conflicts among confidences. In the following, we experiment how the proposed collision warning scheme can prevent from predicted accidents in real-world.

5.2. Experimental Parameters

The two different image sensors (camera) supporting HD (1280 × 720) and full HD (1920 × 1080) resolutions videos with 30 FPS are used to capture real-time vehicular environment. A YOLOv3 [48] is exploited to object detection and classification. We consider 5.9 GHz DSRC network for V2V communication. For compatibility, we consider the proposed packet header contains detection results encapsulated with the basic payload of 500 bytes [75]. The packet is transmitted in 10 Hz period, which is applicable for safety message formats such as Cooperative Awareness Message (CAM), Decentralized Environmental Notification Message (DENM), and BSM used in DSRC. We consider the weighted similarity matrix which has elements such that $Q(\mathrm{E},\mathrm{U})=100/101,Q(\mathrm{N},\mathrm{U})=1/101$. The collision occurrence scenarios are modeled by MATLAB on Intel(R) Core(TM) Xeon 2.4 GHz with NVIDIA GeForce 1080Ti GPU. The detailed experimental parameters are presented in

Table 2. Experimental parameters.

Parameter	Value
Video(Image) size [pixels]	1280 × 720, 1920 × 1080
Frame per second	30 FPS
Object detection (ML)	YOLOv3
Number of classifications	16
Basic message size	500 bytes/msg
Message transmission period	10 Hz
Frequency	5.9 GHz
Bandwidth	10 MHz
Modulation and coding scheme	16 QAM
Data rate	18 Mbps
Safety area (Car, Human)	4.8 m, 0.5 m
Deceleration rate	5 m/${\mathrm{s}}^2$
Distance accuracy	0.1 m
Velocity accuracy	0.1 m/s

.

5.3. Experimental Result 1: Anti-Collision via V2V

In this scenario, we perform experiments of the proposed anti-collision system with real-world data. As shown in Figure 10a–c, V1 cannot recognize the pedestrian who suddenly appears from an uncertainty area caused by parking cars in a narrow street. However, V2, which comes from the opposite site, is able to detect the pedestrian. The actual images of vehicles’ front view are presented in Figure 10a,b including detection results. A top-view illustration is depicted in Figure 10c. V1 with 36 km/h and V2 with 47 km/h speed drive and a distance between them is about 20 m. A pedestrian walks westwards as 5.5 km/h in 15 m away from V1.

In Figure 11, we plot the measured and estimated values of position, velocity, and calculated acceleration of the pedestrian. In Figure 11b, it is shown that the cosine effect compensation can have a high impact on measurements. As pedestrian and V2 come closer, the angle between them increases, which causes the measured velocities to be smaller than actual values. It can be seen that the estimated values are quite close to ground truth values. The precise estimations in Figure 11a,b result in correct acceleration values (direction) as in Figure 11c.

Next, we explore how confidences of V1 change over time. We assume that the current frame of V1 and V2 starts at $t=0$. In this frame, V2 detects seven objects including the pedestrian, which takes 22.46 ms. The detection outcome is encapsulated as the proposed format with 1053 bytes. The delay is only 4.68 ms. As soon as V1 receives data from V2 at $t=27.1$ ms, the data fusion is performed. As the pedestrian is located in the uncertainty area of V1, we set confidences to $\{0.0,0.0,1.0\}$. V1 takes 0.88 ms of processing time in the decision-making, prediction, and the decision to issue warning. In addition, the data fusion system returns the highest confidence on the detection of a person. The whole decision process takes only about 27.9 ms. During the process, the confidence on existence changes from 0.0 to 0.66. We impose 0.5 threshold to determine existence of object. The detailed changes in confidences are illustrated in Figure 12; it shows that V1 can start to decelerate after moving by the distance of only 0.27 m even if V1 cannot directly recognize the pedestrian. The detailed results are addressed in

Table 3. Experimental results for the anti-collision case.

		V1	V2	Output
Existence	Existence	0.0	0.85	0.66
	Non-existence	0.0	0.05	0.03
	Uncertainty	1.0	0.1	0.31
Class	Person	-	0.153	0.153
	Other 15 classes (car, bus, ⋯)	-	5.64 $\times {10}^{-2}$	5.64 $\times {10}^{-2}$
Latency	Detection time [ms]	24.09	22.46
	Communication delay [ms]	-	4.68	28.02
	(message size [bytes])	(-)	(1053)	(totally)
	Estimation+Decision [ms]	0.88	-

.

Finally, we investigate a collision scenario through the environment modeling as shown in Figure 13. Each object has own safety area, e.g., car has 4.8 m and human is set to 0.5 m. Prediction line indicates future trace until 7 s later, and star dots imply that the accident is expected to happen when moving objects reach those locations. Vehicles are assumed to decelerate with 5 m/${\mathrm{s}}^2$ [76]. Figure 13b,c shows the locations of objects without and with the proposed system only after 1.2 s, respectively. In Figure 13b, we can find that an accident occurs between V1 and pedestrian. On the other hand, V1 can slow down its speed by receiving data from V2, and avoids the predicted collision as shown in Figure 13c. From experimental results, we show that the proposed inter-vehicle data fusion takes less than 30 ms from object detection to decision-making. In addition, the existence of the pedestrian in blind spot can be identified by the proposed data fusion scheme. These results demonstrate that we can prevent from the predicted accident in real-time.

5.4. Experimental Result 2: Sensing Failover

The next scenario concerns a detection failover through V2V cooperation. As shown in Figure 14, V1 with constant speed of 50 km/h is moving in a narrow space. However, V2 driving in 5 m/${\mathrm{s}}^2$ cannot obtain a clear video of V1 due to broken sensor. Therefore, in this case, V1 is regarded as “nonexistence” rather than “uncertainty” by V2, which represents a false negative error [44]. In addition, we assume that V1 cannot communicate with other vehicles due to a malfunction of the communication tools or lack of communication capability. We assume that there are similar problem at V3, i.e., it is hard to obtain clear image on V1. Under this set-up, V2 cannot perceive the movement of V1 at all, and the crash of V1 and V2 is imminent. However, the other cars (V4, V5) located at nearby V1 can recognize the movement of V1 as presented in Figure 14e,f. They transmit this information to the V2 from which the accident can be avoided. The cameras equipped at V2–V5 record V1’s movement simultaneously. We manipulated cameras at V2 and V3 so that they produce blurred videos. Figure 14a,c shows the original clear front views, and Figure 14b,d shows the blurred images at the same time.

We compare the measurements and estimation of V1 in Figure 15. Due to the dynamics of V1, the angles between nearby vehicles are changed, which results in inaccurate measurements as shown in Figure 15b. However, Figure 15a–c demonstrates that we obtain precise estimation values by the proposed scheme, and the predicted collisions are avoided.

We demonstrate how the confidences change as data sources are added in Figure 16. Especially, we compare the performance of the proposed scheme with the conventional weighted data fusion method exploiting Jousselme’s distance. For abnormal states of V2 and V3, we assume that they have high confidence in “nonexistence” about V1, and the other vehicles are set to have confidence in existence of V1 as 0.7 and 0.9, respectively. In Figure 16a,b, we plot changes of confidences for the proposed and existing weighted DS schemes, respectively. The detailed parameters and experimental results are presented in

Table 4. Experimental results for the sensing failover case.

		V2	V3	V4	V5	Output(V2)
Existence	Existence	0.1	0.1	0.7	0.9	0.58
	Non-existence	0.8	0.75	0.1	0.05	0.42
	Uncertainty	0.1	0.15	0.2	0.05	0.0
Class	Car	-	-	0.153	0.153	0.329
	Other 15 classes (person, bus, ⋯)	-	-	5.64 $\times {10}^{-2}$	5.64 $\times {10}^{-2}$	4.47 $\times {10}^{-2}$
Latency	Detection time [ms]	21.53	23.44	22.41	24.36
	Communication delay [ms]	-	2.28	3.88	4.15	29.39
	(message size [bytes])	(-)	(513)	(873)	(933)	(totally)
	Estimation+Decision [ms]	0.85	-	-

. As the data are received sequentially, we can find that the confidence in existence is changed in both figures. However, under the baseline scheme, the final confidence on existence is 0.48, which in turn makes V2 decide “nonexistence”. Contrary to this, the proposed scheme shows 0.58 confidence in existence, thus V2 can recognize V1 successfully. From this result, it is shown that the proposed scheme can reduce the false negative error as compared to the conventional data fusion method.

Next, we consider latency. The maximum packet length is 993 bytes, which requires only 4.4 ms to receive the packet, and we have 21 ms processing delay for detection. It takes 0.85 ms on average for V2 to perform decision-making including the estimation of future trajectory of V1. Moreover, V2 decides that the target object is a car. In total, only 29.36 ms is required until V2 begins decelerating, which is much shorter than human reaction time.

An environment modeling for this scenario is shown in Figure 17. As shown in Figure 17b, it is predicted that the accident will happen after 0.7 s. However, under the proposed system, V2 can stop as soon as recognizing movement of the V1 as in Figure 17c.

6. Conclusions

In this paper, we studied a real-time inter-vehicle data fusion scheme for autonomous vehicle safety which combines multiple sensing results transmitted by adjacent vehicles through V2V communication. First, we proposed a novel metric to measure distance between conflicting confidences on the detected event. We considered a new similarity matrix which takes asymmetric weights of the existence of an object which can be flexibly adjusted depending on its importance. Our approach is applicable to autonomous vehicle by reducing fault diagnosis in vehicular environment, especially false negative errors. According to the proposed distance measure, the weighted DS combination was used to fuse multiple confidences. It was shown that the proposed data fusion scheme has polynomial time complexity, which enables real-time combination of confidences. From simulation, we also show that the proposed scheme can work in real-time and achieve a lower false negative rate compared to both the classical and existing weighted DS fusion schemes. By conducting real-world experiments, we show that the accidents can be prevented by our method. Moreover, even when the ego-vehicle measurements are incorrect, the proposed system can overcome sensing failures through the real-time data fusion via inter-vehicle cooperation.

For future work, we plan to investigate collision avoidance maneuver based on vehicle kinematic model which can be utilized in real-world scenarios such as overtaking and lane changing. As an application of this technique, we will consider a compensation method for trajectory estimation errors by comparing possible maneuvers and estimated movement. Moreover, precise mapping of the surrounding environment is required for reliable maneuver. We also plan to use the data obtained from the vision sensors with the deep learning-based Simultaneous Localization and Mapping (SLAM) method for future research.