User Experience in Neurofeedback Applications Using AR as Feedback Modality

Round 1

Reviewer 1 Report

In this manuscript, the authors report the user experience results of a neurofeedback experiment on ninety-nine neurotypical participants. The experiment had a two by two experimental design comparing (1) sham versus (2) real feedback and (3) 2D traditional versus (4) augmented reality (AR) feedback. The manuscript is globally well written though I have the following important critics that lead me to recommend a reject and resubmit.

Most of all, this article lacks the analyses on neurophysiological data and on all the different experimental conditions. I believe, the presentation of the user experience results is too light for a dedicated article. The user experience should be put into perspective with to the neurofeedback performances of their participants. For instance, the participants using AR stimuli reported that they felt more in control that the participants using 2D stimuli, though did they really have better control? Are the participants aware of their control ability? Are these performances specific to the band power that the participants were asked to regulated? Could the participants learn to regulate their brain activity? An analysis of the neurofeedback performances would answer this question.

The user experience should also be assessed taking into account all the different conditions that the participants experienced. For the time being, grouping the real and sham feedback conditions, the authors cannot really talk about feedback as some of the stimuli were not providing any feedback to the participants. Also, if the 2D or the AR stimuli did not represent the neurophysiological activity of the participants, then they might have been more discouraged and this discouragement might not have been the same in all the conditions, thus the need to assess independently the conditions.

I am not sure for the rationale behind the experimental protocol. First, if the goal was to provide augmented reality feedback, why didn’t the authors use an AR headset? This would have greatly limited the lag when moving one’s head and resulting discomfort of using AR. Also, could the fact that the participants seeing the 2D feedback still wore the VR headset without benefiting from it have unblinded some on the group that they were in, decreased their motivation and greatly impacted their user experience?

Some references are missing. For instance, there are other relevant references to argue for the use of a virtual feedback such as Lécuyer et al., 2016, Marshall et al., 2013 or Ron-Angevin et al. 2009. Also, the authors do not discuss any previous studies using augmented reality for neurofeedback, are they not aware of any study that used it in such context? As for the user experience measures, the authors do not provide their hypothesis and related work on the matter. Which were the previous articles that measured PANAS, SSQ, FSK and TUI in neurofeedback/BCI experiments? Finally, there is a lot of self-citation in this article, as 9 cited articles over 32 articles are from the authors (28%). This is a lot.

Minor:

·         There are different numbers of participants in the abstract (2x50) and the content of the manuscript (50 + 49).

·         There are a few spaces at the end of sentences before the end point that should be removed and also some that should be added after it.

·         References are missing for the following sentences:

·         (page 1) “If the users are successful in up-regulating their SMR rhythm, cognitive improvements may follow.”

·         (page 9) “VR-based NF studies report such increased negative reactions especially in patient populations of higher age having limited prior experience with VR-technology.”

·         Why do the authors make this hypothesis: “The participants probably found the development and implementation of AR-based feedback more complex and time-consuming than 2D feedback.”? I believe that if the participants felt that AR feedback would be better, then they should not care about the development and implementation time, as they would not be involved in these stages.

References:

·         Lécuyer, A. (2016). BCIs and video games: State of the art with the openvibe2 project. Brain–Computer Interfaces 2: Technology and Applications, pages 85–99.

·         Marshall, D., Coyle, D., Wilson, S., and Callaghan, M. (2013). Games, gameplay, and BCI: the state of the art. IEEE Transactions on Computational Intelligence and AI in Games, 5(2):82–99.

·         Ron-Angevin, R. and Díaz-Estrella, A. (2009). Brain–computer interface: Changes in performance using virtual reality techniques. Neur let, 449(2):123–127.

Author Response

Dear Reviewer,

we would like to thank you for your elaborated suggestions/comments, they contributed to increasing the quality of our research. We addressed all points, please see the attachment.

Best regards

Lisa Berger

Author Response File: Author Response.pdf

Reviewer 2 Report

1. Paper summary

In the presented study, authors focus on the effects of Augmented Reality (AR) based visual feedback on subjective user experience including positive/negative affect, cybersickness, flow experience, and experience with the use of this technology and compare it with a traditional 2D feedback modality. The results show the potential advantage of the use of AR-based feedback in NF applications over traditional feedback modalities.

2. Strengths

• The paper proposes a human study to understand the important problem of assessing user experience of AR.

• The writing of the paper is generally good and easy to follow.

• The paper provides additional materials for verification.

3. Weaknesses

• The paper doesn’t compare their work with existing research in the related work section.

• The paper doesn’t discuss the limitations in a Threats To Validity section.

4. Comments for authors

1. Significance

1. This paper includes much about the statistical significance of the experiments, but given that some p values are high, authors may want to discuss more about the reason.

2. There can be some more discussions about the criterion and process of selecting participants

2. Soundness

1. The paper might also need to reflect the content of the questionnaire for better understanding of the experiement.

2. It is unclear that if participants had experiences with AR before.

3. Novelty

1. The paper is the first work to conduct a human study to compare the user experience of participants undergoing one session of SMR-based NF training which promotes the novelty of the paper. 

4. Presentation

1. Authors may want to add a section of Threats To Validity to discussion the limitations of the paper and how authors managed to overcome the restrictions.

2. Authors may want to discuss some existing research work in a new section.

5. Verifiability

1. The paper provides additional materials for verification.

6. Some minor comments

1. It will be better if authors can share the feedback from the participants in their paper.

Line 43: signal .

Line 44: follow .

Author Response

Dear Reviewer,

we would like to thank you for your elaborated suggestions/comments, they contributed to increasing the quality of our research. We addressed all points, please see the attachment.

Best regards

Lisa Berger

Author Response File: Author Response.pdf

Reviewer 3 Report

The article compares 2D feedback and feedback via AR as methods of neurofeedback on brain activity. It examines the influence of the two methods on perceived mood, CS, flow, the subjects' attitudes, perceived success, enjoyment, control level, concentration, and physical conditions when wearing the glasses.

The introduction leads to the research question in a well-structured manner. However, it remains unclear what the authors consider the main difference when they place the "plant" feedback on a table instead of in a purely virtual environment. This difference must be explained in more detail, as the added value of using AR in this context is otherwise not apparent to the reader. Without an explanation, the study runs the risk of appearing to be an artificial finger exercise. This would diminish its originality and contribution to the research field. 

The material and methods are well and comprehensibly described, the instruments are well chosen with regard to the research question. The results are reproducible based on the description of the methods. The results are comprehensively presented. 

The discussion addresses relevant aspects in terms of the research field. The authors state, that the low level of CS is a main difference compared to VR. It remains unclear why the treatmant groups should differ in their CS at all, as the setting is the same in both cases. With regard to the comparison with VR, the interpretation seems somewhat strained, as the perception of CS depends crucially on the system used (and, for example, the frame rate of the system). It would be helpful if the authors were able to take up this aspect again and go into more detail, as they have already done to explain CS in AR. Specifically: Would different results have been expected with VR if, as in the study, no head movement had been necessary?

The Conclusions summarize the main results of the study in an easily understandable way. However, as already indicated above, it would be necessary to work out more precisely what the special feature of AR should be in this study. Without this clarification, the conclusion that AR is a promising tool appears to be insufficiently substantiated.

Overall, the manuscript is clearly written, relevant to the  field, and presents methods and results in a well-structured manner. The proportion of self-citations is 28%, which seems relatively high to me.

Author Response

Dear Reviewer,

we would like to thank you for your elaborated suggestions/comments, they contributed to increasing the quality of our research. We addressed all points, please see the attachment.

Best regards

Lisa Berger

Author Response File: Author Response.pdf