**4. Results**

In this chapter, the results of the statistical analysis are presented, showing the behaviour and the differences of the sensor model compared to the Ground Truth and the real sensor. In order to implement the DGT-SMV procedure and to illustrate the potential of the method, the driving scenario defined in Section 3.2.1 was used. As already described in Section 3.3.2, the modified data set with the super-positioned deviation was used for the evaluation of the RSI sensor model.

The data of both, the real and the simulated radar sensor, was accordingly to the radar properties split in a near range (0 to 60 m) and a far range (60 to 200 m) section. The used radar sensor can detect objects in the near range between 0 and 60 m in near and far range mode, which results in an overlap of the two sensor modes. For this reason, the far range was further subdivided into those data from the 0 to 60 m and 60 to 200 m for detailed analysis.

#### *4.1. Comparison of Simulated and Measured Radar Signals*

In Figures 7–9, visualizations of various statistical analysis of the near range radar and radar model are shown. Figure 7a,b presents the visualization of the radar detection points, which are associated to the corresponding dynamic target. The grey scale of each target point indicates its relative velocity, and the stroked line indicates the bounding box of the target vehicle. The contour lines in this plots are representing the multi-variant distribution of the reflection points. In Figure 8, the PDF's of the deviation to the reference point <sup>P</sup>*ref*(*<sup>x</sup>*, *y*) in *x*- and *y*-direction of the realizations are shown.

The probability distribution can not only be used for the qualitative assessment of the distribution of the reflections but also serves as a basic prerequisite for the calculation of the Jensen–Shannon divergence. In Figure 8a, the deviation in the longitudinal direction, and in Figure 8b, the deviation in the lateral direction is shown. Figure 9 shows a PDF of

the deviation of the relative velocity of each radar detection point in comparison to the Ground Truth relative velocity of the target vehicle.

(**a**)

 (**b**)

**Figure 7.** Evaluation of the detections in the range of 0 to 60 m of the near-range radar sensor and sensor model. (**a**) Scatterplot of detections for the real sensor. (**b**) Scatterplot of detections for the sensor model.

**Figure 8.** Evaluation of the detections in the range of 0 to 60 m of the near-range radar sensor and sensor model. (**a**) PDF of the deviation in the *x*-direction from <sup>P</sup>*ref*(*<sup>x</sup>*, *y*) of the real sensor and sensor model. (**b**) PDF of the deviation in the *y*-direction from <sup>P</sup>*ref*(*<sup>x</sup>*, *y*) of the real sensor and sensor model.

**Figure 9.** Evaluation of the detections in the range from 0 to 60 m of the near-range radar sensor and sensor model: PDF of the relative velocity in the *x*-direction from the reference velocity of the real sensor and sensor model.

In Figures 10–12, the visualization of the statistical analysis of the far range sensor in the near range section are shown. The results of the far range sensor for the far range section (60 to 200 m) can be found in Figures A1–A3.

(**a**)

**Figure 10.** Evaluation of the detections in the range of 0 to 60 m of the far-range radar sensor and sensor model. (**a**) Scatterplot of detections of the real sensor. (**b**) Scatterplot of detections of the sensor model.

 (**b**)

**Figure 11.** Evaluation of the detections in the range of 0 to 60 m of the far-range radar sensor and sensor model. (**a**) PDF of the deviation in the *x*-direction from <sup>P</sup>*ref*(*<sup>x</sup>*, *y*) of the real sensor and sensor model. (**b**) PDF of the deviation in the *y*-direction from <sup>P</sup>*ref*(*<sup>x</sup>*, *y*) of the real sensor and sensor model.

#### *4.2. Performance Metrics*

When evaluating the performance of a virtual sensor for accuracy or fidelity, the correct performance metric should be selected to meet the requirements of the application. As described in [36], the data sets under comparison can be treated with or without uncertainty. Since, in our application, both experimental and predicted values are treated with uncertainty, comparison in the shape of a non-parametric discrete distribution is one promising solution.

The Jensen–Shannon Divergence (JSD) measures the distance between two discrete distributions by comparing the shape of two PDFs, one of which is the accuracy reference (real sensor data) and the other the output of a virtual model. JSD has two important features: first, JSD includes all the statistical information known about each distribution in the comparison. This means that the comparison is not limited to the average behaviour of the distributions. Second, it provides a real mathematical metric.

Since the Jensen–Shannon distance is a real mathematical metric, using the property that the value of *DistJS*(P||Q) is always a real number in the closed interval between 0 and 1, and if the value is 0, then the two distributions, P and Q, are the same; otherwise they differ as much as possible, a quantitative comparison can be made between the sets of simulation.

In Tables 2–4, the JSD is expressed as a percentage for the near range as well as for the far range. The evaluated variables are *ζ<sup>s</sup>*,*<sup>r</sup>*<sup>Δ</sup>(*x*), defining the JSD metric for the relative distance in *x*, *ζ<sup>s</sup>*,*r*,<sup>Δ</sup>(*y*), for the relative distance in *y* and *ζ<sup>s</sup>*,*r*,<sup>Δ</sup>(*v*) for the relative velocity *v*.


**Table 2.** The results for the near-range radar sensor, detection range 0 < x < 60 [m].

**Table 3.** The results for the far-range radar sensor, detection range 0 < x < 60 [m].


**Table 4.** The results for the far-range radar sensor, detection range 60 < x < 200 [m].


## **5. Discussion**

Inspecting the results, a quick overview on the performance of the virtual sensor can immediately be achieved by the JSD metrics, where, for the deviations *ζ<sup>s</sup>*,*<sup>r</sup>*Δ, 0% is perfect performance and 100% is the worst performance. In our example, it can be seen that, in the far range, the virtual sensor is more accurate in reproducing the relative velocity than in the relative distance. For the relative velocity, the JSD of *ζ<sup>s</sup>*,*<sup>r</sup>*<sup>Δ</sup>(*v*) is 34.2% up to 60 m and 25.6% up to 200 m. In the near range, the performance is worse at 51%.

For the relative distance, the better performance is seen in the *x* direction. The JSD of *ζ<sup>s</sup>*,*<sup>r</sup>*<sup>Δ</sup>(*x*) is 46.5% in the far range up to 60 m and 44.1% up to 200 m, for the near range 54.8% was observed. In the *y* direction, the related JSD values of *ζ<sup>s</sup>*,*<sup>r</sup>*<sup>Δ</sup>(*y*) are 65.1%, 52.8% and 53.1%, respectively.

This result is confirmed by visual inspection of the PDF illustrated in Figures 8, 9, 11 and 12 as well as the scatter plots in Figures 7 and 10. Comparing the shape of the PDFs, the strengths and shortcomings of the sensor model can be assessed, providing recommendations for parameter tuning and drawing conclusions on the validity of the results. The explanation of the results may be found in the specific modelling approach of the commercial radar-sensor model and is not part of this paper.
