Game-Based Simulation and Study of Pedestrian-Automated Vehicle Interactions

Round 1

Reviewer 1 Report

The results in the paper are interesting, but will be more convingcing if more participants are involved. Here I recommend a conditionally accept. But the results and the discussion part should be refined to give more insightful descriptions. 

The authors should emphasize more on how to use the results they obtained in practice. But this doesn't affect the paper to be excellent.

Author Response

We thank the reviewer for their comments and acknowledgement that the paper is interesting and high quality. While a larger sample would be preferrable, and may potentially yield additional insights of interest, we prefer to keep this slated as a future extension of the presented work and instead focus on the application design, development, and preliminary work.

The rationale for publishing this design-heavy article prior to conducting a larger study is that the timely sharing of preliminary survey results alongside a release of the game executable will help us to recruit additional participants, including those of diverse backgrounds and technological familiarity, for future studies. Having completed peer review will build a stronger and more compelling case in “selling” test subjects on participating as they will understand better the contribution their efforts make towards advancing knowledge.

We also plan to share the paper and its preliminary results within popular media as one technique for gaining additional viewership and participation; Ewad, et al. found great success soliciting participation in “The Moral Machine Experiment” in this manner – most of their participation came after initial publication. Early publication will also help to raise awareness of some of the less-visible sociotechnical concerns related to automated vehicles with a wider audience, providing a catalyst for industry and academia to develop socially-conscious automated vehicles and associated software that strike an appropriate balance between safety and broader social optimality.

To address this specifically in the manuscript text, we have added additional statements clarifying that the sample size could be increased to yield more significant and robust results, and also by noting that the game and associated survey forms will remain active for readers to participate in the game and share their results for future extensions of this submission. Additionally, we have made more explicit some of the key findings, e.g. that AV designers must be cognizant that familiarity breeds a disregard for safety around automated vehicles, and that there may be a "socially optimal" level of discomfort that strikes a healthy balance among safety, efficiency, and mutual respect of on- and near-road agents.

Reviewer 2 Report

The paper presents a new simulator that facilitates exploration of interactions among human players, acting as pedestrians, and virtual human driven and automated vehicles. Results obtained with 89 participants (players) in different urban scenarios are given and discussed. Based on this experience the Authors include suggestions for developing of the simulator.

The proposed simulator is one of two known that implement the pedestrian point of view. The described research is very preliminary but the paper presents the research direction and this is its important advantage. Therefore the work is worth publishing.

Minor remark: line 247 [?].

Author Response

We thank the reviewer for their thoughtful summary of our work and their acknowledgement that the work is worthy of publication. We appreciate that they note that the purpose of our work is to showcase the need for, and development of, a pedestrian point-of-view simulator to study automated vehicle interactions, as well as to share the preliminary results from an 89-person survey to inform future industrial and academic studies.

We have addressed the missing reference in line 247 (currently line 322).

Reviewer 3 Report

The paper designed and evaluated a method for investigating the interaction between pedestrians and interactions. The method designed could be useful, however, the manuscript is poorly written. It reads more like a technical report rather than a scientific journal publication. Major revision is needed before being considered for publication.

The first person pronouns were used all over the paper. It should be avoided in academic writing.

Section 2-background and motivation should be restructured and rewrite. Move the background information into the Section 1, add more literature review. More in-depth review of previous research regarding evaluating pedestrian’s interaction with AV should be included in this section.  

The entire Section 2.2 reads more like methods rather than an introduction/literature review, it should be rewritten. Move the relevant information to the method section. Page 162-171, these five categories should be rewritten with explanations of why each category is necessary and provide references, and this should be in the method section.

What’s research gaps identified in reviewing the literature? There should be a section, ideally at the end of the literature review, to discuss about research gaps identified.

What are the research questions the research is trying to solve here? What about the overall research aim? All these should be clearly presented before talking about methods, otherwise, it confuses readers.

Line 237 authors stated ‘we aimed to create a meaningful interactive user experience’ this statement is confusing and makes little sense without more a clear and detailed explanation. More explanation must be given.

What about the sample/participants recruitment? There should be a section discussing this in the method section. What are the sample recruitment criteria? What about the ethical considerations?

How the data was analysed? What statistical tests were used?

The result section is only half a page. More details, and figures should be added to present the results.

The discussion is also very poor. Bullet points should not be used in there. This section should have been focused on interpreting the results in light of the existing literature. It should be rewritten.

Future work section, the download link should not have existed among the main text of a paper. Move it to the Appendix. 

Author Response

• The paper designed and evaluated a method for investigating the interaction between pedestrians and interactions. The method designed could be useful, however, the manuscript is poorly written. It reads more like a technical report rather than a scientific journal publication. Major revision is needed before being considered for publication.

We thank the reviewer for their acknowledgement that the designed method may be useful in studying pedestrian/vehicle interactions.

We note that the submitted manuscript presents the motivation for the study, the design of the study and, explicitly, the creation of enabling tools. We include preliminary survey results to demonstrate the efficacy of the created novel game-based survey tool. In the revised text, we note explicitly that the sample size is small, suggesting broader participation may be necessary for robust results. Early publication will help us to attract the necessary additional participants.

In our view, the creation of enabling tools is at least as important to the study narrative as the results. We note that Reviewer #2 supported the paper’s publication while simultaneously acknowledging that the focus of the manuscript was in part centered on the creation of a tool novel within the self-driving domain. This focus on building a tool integral to future research may explain why, stylistically, the manuscript feels akin to a technical report.

We also note from the MDPI Automation “Aims & Scope” that the journal covers both “robotics and applications,” as well as “human-machine systems and interfaces.” It is common in journals covering such topics to publish articles describing tool design, development, and deployment in addition to those focused solely on experimental design and data analysis. We feel as though the contribution, scope, and presentation of the submitted manuscript is appropriate given the aims and scope of the journal. We note also that Reviewer #1 and Reviewer #2 were satisfied with the content, clarity, and significance of the work.

• The first person pronouns were used all over the paper. It should be avoided in academic writing.

We thank the reviewer for their suggestion to transition away from first person pronouns; we note, however, that the notion of avoiding first person pronouns in academic writing is a stylistic preference and is not as widely held as once was true.

First person writing is often seen as “humanizing” scientific work, which would seem appropriate for work relating to human-vehicle interaction – it makes the work more familiar and accessible. More broadly, scientific writing has moved largely towards first person, active voice, and present tense. Examples of such writing are found across journals, including within MDPI’s Automation. See “Inspection Application in an Industrial Environment with Collaborative Robots," as one recently-published example.

• Section 2-background and motivation should be restructured and rewrite. Move the background information into the Section 1, add more literature review. More in-depth review of previous research regarding evaluating pedestrian’s interaction with AV should be included in this section.  

We thank the reviewer for their suggestions on reorganization, and have re-read the manuscript as well as had additional colleagues read our article. While the organization might be improved with the recommended changes, none of the non-author readers had difficulty understanding the content of the manuscript. We believe that with the reviewer’s additional suggested text motivating the study and making explicit the studied research questions that the manuscript is already more readable than the previous submission. Further, we note that Reviewer #1 and #2 did not have issues with the organization of the manuscript. Neither recommended any organizational or stylistic changes.

Regarding references and literature review, we have added additional references to prior studies in this area that are timely and relevant to MDPI Automation’s readership. We would welcome the reviewer’s suggestions for additional specific, essential prior works to review and include that were not in the initial submission.

• The entire Section 2.2 reads more like methods rather than an introduction/literature review, it should be rewritten. Move the relevant information to the method section. Page 162-171, these five categories should be rewritten with explanations of why each category is necessary and provide references, and this should be in the method section.

Please see our reply to #3. Regarding each category’s inclusion, we thank the reviewer for suggesting this important clarifying text and have added a short explanation to each category indicating why it is necessary to this study.

• What’s research gaps identified in reviewing the literature? There should be a section, ideally at the end of the literature review, to discuss about research gaps identified.

We have added a short section at the end of the literature review to make explicit the identified research gaps and the unmet need for the simulator and survey.

• What are the research questions the research is trying to solve here? What about the overall research aim? All these should be clearly presented before talking about methods, otherwise, it confuses readers.

We have added a section in early in the text to make more explicit the questions we seek to answer:

• • What are typical pedestrian / vehicle interaction patterns for varying degrees of autonomy?
  • Can goal-driven humans effectively “coexist” with goal-driven automated vehicles, e.g. in a social context?
  • Does the degree to which a pedestrian is familiar with autonomy change these interactions? How?
  • Does the knowledge that a vehicle is automated change a pedestrian’s interaction patterns? How?
  • Does exposure to automated vehicles increase or decrease pedestrian trust in those vehicles?

We have also made more explicit the answers to these questions in the Conclusions section.

• Line 237 authors stated ‘we aimed to create a meaningful interactive user experience’ this statement is confusing and makes little sense without more a clear and detailed explanation. More explanation must be given.

We aimed to create a meaningful experience to enhance embodiment and thereby increase the likelihood of accurate, useful, and repeatable results.

• What about the sample/participants recruitment? There should be a section discussing this in the method section. What are the sample recruitment criteria? What about the ethical considerations?

Participants were recruited by word of mouth. There were no exclusion criteria, except for eliminating data captured by study designers who may be biased due to overfamiliarity. We have clarified this in the text. We note also in text that due to the nature of the study and data collected, the study was deemed to have no significant ethical concerns and was IRB exempted.

• How the data was analysed? What statistical tests were used?

Given the preliminary nature of the study, simple metrics such as the average score and standard deviation were computed for the response to each question. Results were presented to the reader directly, and raw data are available on the linked Harvard Dataverse. Deeper analysis was not conducted due to the relatively smaller sample size and the preliminary nature of the study. Tests for statistical significance will be part of the analysis of the broader results captured upon the widespread use of the designed tool.

• The result section is only half a page. More details, and figures should be added to present the results.

The results section has been slightly expanded to reflect the answers to the specific questions asked at the outset of the study. We believe the length to be appropriate given the split focus of the paper on tool development and preliminary result presentation.

• The discussion is also very poor. Bullet points should not be used in there. This section should have been focused on interpreting the results considering the existing literature. It should be rewritten.

We thank the reviewer for the suggestion to expand the discussion to remove bullet points. We have edited the bullet points and surrounding text somewhat for clarity.

We note that bullet points are increasingly commonly used to convey rich information in a compact form, including in previous MDPI Automation Submissions ( https://www.mdpi.com/2673-4052/3/1/5/htm ). Some journals, particularly those under the Elsevier umbrella, specifically invite the submissions of research highlights in the form of bullet points (see: https://www.elsevier.com/authors/tools-and-resources/highlights).

We have expanded the discussion somewhat to reflect the apparent answers to our research questions, as well as the implications of these answers.

• Future work section, the download link should not have existed among the main text of a paper. Move it to the Appendix. 

The link to the download for the application and data has been moved to the Appendix, alongside added text reflecting the content of the survey consent form that is integrated within the application.

Round 2

Reviewer 3 Report

Thanks for addressing my comments. The quality of the paper has been improved. There are no further comments from me.