**3. Results**

In total, 1071 recognitions were possible (7 signs per scenario × 9 scenarios × 17 vehicles). In 21.20% of the cases, signs were recognized correctly. In 72.46%, signs were not recognized at all, while in 6.35%, signs were wrongly recognized by vehicles. Of course, these percentages vary between each scenario. As expected, in the control scenario (scenario 1), the highest level of correctly recognized signs was recorded (73.11%). On the other hand, in the same scenario, 21.58% of the signs were not recognized at all (mainly prohibition of overtaking, but also in some cases speed limit signs). This is due to the fact that the type of signs which TSRS recognize and display to the driver varies to some extent between brands. Moreover, some of the signs (5.04%) were wrongly recognized even in the control scenario (for example, 90 km/h was recognized as 30 km/h). In the second scenario (black outline of the signs), the number of unrecognized signs increased to 36.13%, while the percentage of wrongly recognized signs stayed the same as in scenario 1 (5.04%).

A major decrease in sign recognition occurred in scenarios 3 and 4 in which minor changes were made to the signs. Namely, the percentage of correctly recognized signs in scenarios 3 and 4 fell to 20.17% and 4.20%, respectively. In addition to the increase in unrecognized signs, there was also an increase in wrongly recognized signs—26.89% in scenario 3 and 14.29% in scenario 4.

When signs were half covered by black paper (scenarios 4, 5 and 6) as well as by graffiti (scenario 9), the percentage of unrecognized signs was between 98% and 100%. In scenario 8, 33.61% of the signs were correctly recognized (63.03% of signs unrecognized), while 3.36% were wrongly recognized (mainly speed limit signs).

A graphical presentation of the aforementioned results is shown in Figure 3 while the overall results for each vehicle per each scenario are presented in Appendix A.

**Figure 3.** Percentages of "correctly recognized", "unrecognized" and "wrongly recognized" signs per scenario.

As mentioned in the Data Analysis section, the scenarios were grouped into four categories: control (scenario 1), minor changes (scenarios 2, 3 and 4), medium changes (scenarios 8 and 9) and major changes (scenarios 5, 6 and 7). This was done in order to analyze how different levels of changes on signs affect recognition accuracy. For each vehicle and group, mean value was computed. Overall, the decrease in correctly recognized signs between controlled conditions and minor, medium and major changes is 62%, 77% and 99%, respectively (Figure 4).

**Figure 4.** Percentages of recognized signs per each category.

Although differences between each category are evident, a repeated measure ANOVA with Bonferroni adjustment was used to test the differences between the control condition and each group. The results of ANOVA analysis shown in Table 1 confirms that a statistical difference (*p* < 0.05) between the control condition and each group representing different levels of graphical changes of signs exists.


**Table 1.** Results of ANOVA.

Furthermore, we tested the difference between traffic sign recognition of each vehicle in each scenario with Cochran's Q test. Since scenarios 6, 7 and 9 after grouping did not have variations in values (no values for group "correctly recognized signs"), Cochran's Q test was not performed on them.

Overall, a statistical difference (*p* < 0.05) in traffic sign recognition between tested vehicles was recorded for scenarios 1, 2, 3 and 8, while for scenarios 4 and 5, no statistical difference was found (*p* > 0.05), as shown in Table 2.

**Table 2.** Results of Cochran's Q test—significant differences in traffic sign recognition between tested vehicles per analyzed scenarios.


## **4. Discussion**

Due to their grea<sup>t</sup> potential in reducing road accidents, ADAS technologies have become one of the fastest-growing safety application areas. In the European Union, ADAS systems are mandatory for all new and certified vehicles starting from 2022 and all new registered vehicles by 2024 [15]. One of those systems is traffic sign recognition, which provides drivers with information about traffic signs.

However, the analysis of market-ready TSRS shows that the functionalities of systems differ to some extent between car brands, mainly in the type and the number of signs that can be recognized and displayed to the driver. Some of the cars display only speed limit signs, some besides speed limit display prohibition of overtaking, while some are, in addition to the above, displaying main warning signs such as "dangerous curve", "pedestrian crossing", etc. Furthermore, while the majority of TSRS signs are displayed in color, there are a few brands that display signs in black and white.

The results of this study show that TSRS accuracy differs between car brands and graphical clarity of the sign. In other words, each scenario with graphical changes on signs had significantly lower recognition level compared to the control condition (ranging from 62% to 99%). It is important to note that even the control condition did not have a 100% recognition and, contrary to expectations, had 5% of wrongly recognized signs. This is precisely due to the differences in TSRS between car brands. Moreover, between each category of graphical change and the control condition, there has been a statistically significant difference in the number of correctly recognized signs. The results sugges<sup>t</sup>

that even small changes in the design of a sign, such as the change of the outline color or a minor change in the symbol, can drastically affect TSRS accuracy. This result is in accordance with previous studies [4,12].

Although graphical changes on the signs were created artificially in this study, similar situations exist in real road conditions [16,17] and may result in TSRS errors. Such errors may affect the driver in different ways. Studies have shown that drivers in general perceive a relatively low amount of traffic signs [18–20], and thus an additional display of signs in the vehicle (especially speed limits) should increase overall compliance with traffic regulations. On the other hand, if the sign is not recognized by the TSRS, the driver will not have additional information, which may result in inappropriate and/or improper driving behavior. Furthermore, if TSRS recognizes the signs but wrongly (for example 60 km/h as 80 km/h), such error may confuse drivers and thus distract them, which again may lead to inappropriate and/or improper driving behavior. This result confirms the importance of proper maintenance of signs and their surroundings as highlighted in several recommendations and publications [4,21,22].

Since the general functioning and accuracy of TSRS significantly differ between car brands, a "standardization" of such systems is needed. Standardization, in this sense, implies defining the minimal number and types of signs which every TSRS should recognize, the way signs are displayed to the drivers and minimal levels of recognition accuracy at least for properly placed and maintained signs during daytime and low visibility conditions (night-time, rain, fog, etc.). Although the last requirement is difficult to define since recognition accuracy depends on the number of factors [10–13], the standardization of TSRS could potentially accelerate their development and possibly eliminate some of the current problems. In addition, recent studies emphasize that, although ADAS systems provide a significant progress in safety, they also may distract drivers [23,24]. Thus, the education of drivers regarding the functionality and limitations of TSRS is needed as well in order to avoid or at least decrease potential confusion of drivers. The education altogether with the standardization could decrease the driver's distraction caused by TSRS. Finally, in order to increase the accuracy of TSRS, a database of traffic signs for each country should be established. The database in this sense should have all the variations of signs and their meaning for each country. In the primal stage, it could be developed at least for speed limits so the algorithms used for sign classification and recognition can be "taught" and thus the accuracy of the whole system may be increased.

Although this study provides valuable results, there are some limitations. The study was conducted in controlled conditions (practically a closed road section, dry daytime weather, lack of other traffic, all new traffic signs, relatively low driving speed, etc.) that are not generally present in the real world. However, such controlled conditions represent the best-case scenario, meaning that TSRS accuracy would decrease even more in real world conditions. Moreover, the study included only 17 cars from 14 brands. Even though the used sample is relatively small, most of the major car brands were included and all cars had the best equipment provided by the manufacturer. Finally, due to unavailability of the technical data, proper comparison of the TSRS was not possible.
