Evaluating the Associations between Forward Collision Warning Severity and Driving Context

Round 1

Reviewer 1 Report

Comments for the authors of the article titled “Evaluating the Associations between Forward Collision Warning Severity and Driving Context”:

First, I would like to congratulate the authors on their work, the presented article is informative, enjoyable and very relevant to the track. It is noticeable that a lot to effort went into this study. I like the way FCW solutions were analyzed, which of course is not easy. There are things that in my humble opinion can be better taken care of:

1. Line 194, “…annotators reviewed each clip for validity…” a bit of information regarding how many people were analyzing each clip would be nice considering that categorizing of the clips may be subjective in nature. Thus, if only one analyst was assigned a clip, that could have a negative impact on the conclusions drawn based on this study.

2. What could be the reason behind line 246, “this has not changed the overall count of the ratings of interest: imminent crashes and nuisances.”?

3. This research analyses two consumer available FCW systems. I would like to know which ones. Depending on which solutions were analyzed, the results as well as the inferences could change, thus some information on these systems is necessary.

Finally, I would like to end with a suggestion. The novelty of this work seems to me is the methodology of analyzing the effectiveness of FCW systems. Thus, I would restructure the article a bit, focus on presenting more information on the methodology and later use the two FCW systems as use cases to look at how well the methodology was able to evaluate these systems, since the current results have only relevance when talking about these two systems that were used in the evaluations. The results cannot be used to make a global generic inference on the state of the art of the FCW related systems, it should be kept in mind FCW generation is also a research area that is making great advances every day.

Author Response

Reviewer 1

Thank you for your thoughtful review of our manuscript. We’ve tried to address all of your comments, which we list below (in red), followed by our response.

First, I would like to congratulate the authors on their work, the presented article is informative, enjoyable and very relevant to the track. It is noticeable that a lot to effort went into this study. I like the way FCW solutions were analyzed, which of course is not easy. There are things that in my humble opinion can be better taken care of:

1. Line 194, “…annotators reviewed each clip for validity…” a bit of information regarding how many people were analyzing each clip would be nice considering that categorizing of the clips may be subjective in nature. Thus, if only one analyst was assigned a clip, that could have a negative impact on the conclusions drawn based on this study.

We’ve clarified that most of the clips were coded by a single analyst:

“While the team of analysts viewed each of the 9 “crash possible and imminent” FCW epochs and agreed on the categorization, the majority of the other epochs were coded by a single analyst.”

This was admittedly a weakness in the study, and we’ve added more references to this in the Discussion.

Also, while 15% of the contextual annotations were reviewed by other analysts to examine agreement (which informal appraisal deemed high), and aggregate accelerations of vehicles differed substantially across subjective assessments of severity (suggesting subjective assessments were associated with objective performance differences), a formal interrater reliability analysis would quantify the ability to which coders can identify both the severity of FCW alerts and the attributes of context in which they occur, and the overall use of multiple coders to assess each FCW epoch would likely improve the accuracy of the results.

2. What could be the reason behind line 246, “this has not changed the overall count of the ratings of interest: imminent crashes and nuisances.”?

We clarified the language here to indicate that the overall differences in rates of FCW issuances between the two vehicles under study was not apparently the case for nuisance alerts and crash possible and imminent alerts:

Notably, there were not major differences in counts between vehicles in the categories of prime interest: crash possible and imminent and nuisance alert.

3. This research analyses two consumer available FCW systems. I would like to know which ones. Depending on which solutions were analyzed, the results as well as the inferences could change, thus some information on these systems is necessary.

We’ve highlighted the two vehicles under study earlier in the paper now, referring to them directly in the study goals in the last paragraph of the Introduction:

In light of the strong potential benefits of FCW systems and evidence for reduced crash risk associated with FCW systems, both experimentally and on the road, but sub-optimal levels of FCW system acceptance and trust, the current study leveraged data from production vehicles in a FOT study in order to identify real world FCW alerts, the contexts in which they occurred, and the apparent necessity of their issuances, in two production vehicles with FCW systems: the 2016 Range Rover Evoque and the 2017 Volvo S90.

4. Finally, I would like to end with a suggestion. The novelty of this work seems to me is the methodology of analyzing the effectiveness of FCW systems. Thus, I would restructure the article a bit, focus on presenting more information on the methodology and later use the two FCW systems as use cases to look at how well the methodology was able to evaluate these systems, since the current results have only relevance when talking about these two systems that were used in the evaluations. The results cannot be used to make a global generic inference on the state of the art of the FCW related systems, it should be kept in mind FCW generation is also a research area that is making great advances every day.

We like this suggestion, and have adjusted our paper in several places to emphasize that methodology development was a goal of the study.

(Abstract, last sentence) Modeling outcomes provide clues for reducing nuisance and false alerts, and overall the method of using subjective video annotation combined with vehicle kinematics shows promise for investigating FCW alerts in the real world.

(Introduction, last paragraph) The goals of this study were fourfold: 1. develop a method using subjective video annotation of video of FCW issuances in naturalistic driving to estimate alert severity, and use driving kinematics to validate those estimates; : 2. estimate the probability of an FCW in response to imminent collision (i.e., a situation in which a rapid evasive maneuver is required, such as steering or hard braking) versus a collision impossible (i.e., a situation in which a forward collision could not happen)—what Parasuraman et al. [16] referred to as a “false alarm” in the alerting scenario in which a broad range of crash possibility was considered; 3. demonstrate an association between these probabilities and vehicle dynamic data; 4. model the context under which imminent and collision impossible FCW events are likely to occur to better estimate the posterior probability of collision likelihood (and, conversely, false alarm likelihood) informing the calibration of FCW systems.

(3.2, last paragraph) While the number of critical alerts (especially alerts to imminent hazards) is too small for a by-subject analysis of average minimum deceleration by severity, the non-overlapping nature of the aggregate plots at critical time-intervals is suggestive that subjective ratings of FCW severity reflect significant differences in deceleration profiles, and is supportive of the use of subjective ratings from video FCW epochs to investigate the patterns of FCW issuances in naturalistic driving.

(Discussion, first paragraph) The goals of this study were to develop a method for using subjective video annotation to estimate the probability of an FCW in response to an imminent collision event versus a collision highly unlikely situation, as well as to characterize the vehicle dynamics and contexts associated with each class of event.

(Discussion, last paragraph) Overall, the use of video for the annotation of FCW severity in naturalistic driving shows promise, especially when paired with kinematic data for some level of validation. This can be utilized by researchers who do not have access to CAN radar, lidar, or other diagnostic information about FCW system performance, or who haven’t instrumented test vehicles with other sensors. While the combining of subjective assessments with objective multi-sensor time-series data would provide better accuracy and insight, front-facing with some way to identify FCW issuances (such as instrument cluster video) offer investigators a tool for assessing the severity of alerts drivers are receiving in naturalistic studies.

Author Response File: Author Response.pdf

Reviewer 2 Report

1. Author should give more description of the process of obtaining experimental data，for example,  the video and vehicle dynamics data before and after FCW issued for analysis, but the article does not explain how to ensure the synchronization of these extracted data.
2. Author seems to assume the reader's knowledge of some of the relevant content,for example, logistic mixed effects regressions were used in "3.3 Severity by Context" to obtain the Table 3, but there was no corresponding calculation principle and process description. Some key calculation processes or descriptions should be supplemented.
3. When analyzing the interaction effect of multiple factors, the author made analysis based on the data given in FIG. 3 and FIG. 4. However, it is obvious that the conclusion based on the above two figures is not enough to explain the interaction effect of each factor,  significance experiment should be designed for quantitative calculation and analysis.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

1. Authors should emphasize contribution and novelty in the abstract. Also, numerical results should be presented in the abstract and conclusion.
2. The problem definition, motivation and more recent related work must be added in the introduction section. My suggestion is to divide the introduction into three subsections: motivation and incitement, literature review and contribution and paper organization.
3. It helps to appreciate the paper by having a related work section. The authors should consider more recent research done in the field of their study. The reader may want to see how this work differs from other previous works. Related work section can be further strengthened by discussing various recent related article.
4. The paper should especially include similar studies conducted in 2020 and 2021 in the introduction section.In addition, authors should do more detailed research and add more references.
5. The article missed presenting the research novelty. Moreover, there is no discussion of user requirements, technological options and support for the decisions made at the design.
6. The comparison to other improved schemes is required. The further technical details should be included. This paper should summarize those results and give a comprehensive performance comparison with previous works.
7. The images used in the article are of low quality. Moreover, formats of reference list lack consistency.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

This paper analyzes the associations between Forward Collision Warning (FCW) severity and driving context based on the natural FCW data over 2,000. Data was collected by two modern vehicles. To analyze the driving context and the likelihood of a crash during each alert, cabin and forward camera logs were used. The analysis results were used to model the likelihood of an alert representing a possible collision Before publication, please review the following matters.

1. Section 2.1 explained the criteria for selecting 37 participants. Those with traffic accidents or traffic violations are excluded, so why? Please explain in more detail about the criteria for selecting participants.
2. Is there a reason for using Range Rover Evoke and Volvo S90 as target vehicles? Sensitivity will be different for each vehicle because the sensor type and algorithm implementing FCW are different. According to sections 2.2 and 2.3, FCW seems to have used vehicle mass production specifications. We need a supplementary explanation for selection of target vehicles and used FCW algorithm.
3. The analysis of the driving situation presented in Section 2.3 is too subjective. It is not appropriate for an analyst to judge only by looking at additional recorded videos. I think the possibility of an accident can be evaluated accurately with information that can recognize surrounding objects and analyze the relative position and speed by using LiDAR and RADAR.
4. In general, FCW provides a sensitivity control function to the driver, which is expected to change the overall results of Section 3. Considering these points, can it be suggested in more detail how to utilize the results presented in this paper?
5. The false alarm of FCW implemented old target vehicles is thought to be a limitation in the accuracy of information obtained through sensors and a lack of interpretation of driving situations. Therefore, it is expected that the accuracy of FCW will increase when sensors are reinforced with various functions. In particular, RADAR, LiDAR, and precision maps are also used in mass-produced vehicles in addition to cameras. So, what is the significance of this study from this point of view?

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

 I would like to thank the authors for the revised version they have prepared taking into account the previous comments. Although I find the current version satisfactory, I would have liked a more in depth description of the methodology, the reason being, another researcher in the future should be able to reproduce the setup in question and gather similar results as presented in this study. This one of the reasons we publish and share our findings with the scientific community. That being said, congratulations and happy new year.

Reviewer 2 Report

All questions have been addressed satisfactorily

Reviewer 3 Report

This paper has edited and revised according to the reviewer's suggestions.

Author Response

"This paper has edited and revised according to the reviewer's suggestions."

Reviewer 4 Report

All previous comments have been reflected in the revised manuscript. However, I don't think it's fully explained that this study interpreted the situation only through camera information. Please explain a little more whether the method proposed in this study is better than interpreting the situation through measured data using other sensors.

Author Response

 Please see the attachment.

Author Response File: Author Response.pdf

Round 3

Reviewer 4 Report

All comments have been reflected in the manuscript.