A Stray Light Detection Model for VR Head-Mounted Display Based on Visual Perception

Round 1

Reviewer 1 Report

In the paper, a stray light qualitative evaluation method is presented. Comments:
1.    Please explicitly enlist contributions of this work at the end of section 1.
2.    The analysis of interobserver variability should be supported by the calculation of the Fleiss’ kappa. It is typically used for this purpose.
3.    Why those features are used? Are they enough to capture interesting information?  They are described only in two lines (284-285, page 9).
4.    The paper lacks an analysis of the influence of image processing steps, feature extraction procedures, or features themselves on the performance. The selection of color space or contour detection algorithm is not investigated nor justified experimentally. Are all those steps needed? The size of the dilation operator is arbitrary. Many questions can be asked here. The entire image processing part of the paper is quite superficial.
5.    It seems that there is no place for Precision in Table 3.  It should be reported along with the standard deviation of the results for all performance criteria.
6.    Classification problems should be analyzed to shed more light on the difficulty of the problem.
7.    Obtained features should be visualized. Please use t-sne, PCA, or MDS. 
8.    The results cannot be replicated by a reader. A link to a database and a code of one of the classifiers allowing replication of its results shown in Table 3 should be provided in the paper.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Please see the attached pdf file.

 

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper is focused on finding an automatic procedure to detect the presence of the stray light phenomenon in VR head-mounted displays, starting from the human perception of it.
A two-step procedure, consisting of (i) finding a threshold of stray light humans' perception, through a custom-made experiment, and (ii) design a model to detect possible stray light, is implemented to achieve the goal.

The work is original and significant and, thus, it is likely to be interesting for readers that operates in the field of hardware (but also software) for VR.

The overall quality of the presentation is quite good, but the paper lacks in introducing properly and clearly the steps that led to the final results (both in the Abstract and in the Introduction section).
Moreover, the used methodology remains obscure in some points concerning the perception experiment and the link between it and the automatic detection model.

A list of needed modifications follows:

- In the Abstract, at least a sentence more about the background within which the study is framed should be added. In addition, the aim of the work is just sketched at lines 12-13 and, whereas it should be presented more extensively and clearly. The consequentiality between each step of the used procedure should also be shown clearly.

- In the Introduction, stray light phenomenon should be presented more in detail.

- Some citations about the numerous recent applications of VR in different fields can add value to the background presentation, like for example:

1) in medicine: Boros, M.; Sventekova, E.; Cidlinova, A.; Bardy, M.; Batrlova, K. Application of VR Technology to the Training of Paramedics. Appl. Sci. 2022, 12, 1172. https://doi.org/10.3390/app12031172;

2) in education: Conesa-Pastor, J.; Contero, M. EVM: An Educational Virtual Reality Modeling Tool; Evaluation Study with Freshman Engineering Students. Appl. Sci. 2022, 12, 390. https://doi.org/10.3390/app12010390;

3) in inclusivity: Zingoni, A.; Taborri, J.; Panetti, V.; Bonechi, S.; Aparicio-Martínez, P.; Pinzi, S.; Calabrò, G. Investigating Issues and Needs of Dyslexic Students at University: Proof of Concept of an Artificial Intelligence and Virtual Reality-Based Supporting Platform and Preliminary Results. Appl. Sci. 2021, 11, 4624. https://doi.org/10.3390/app11104624;

4) in cultural heritage: Popovici, D.-M.; Iordache, D.; Comes, R.; Neamțu, C.G.D.; Băutu, E. Interactive Exploration of Virtual Heritage by Means of Natural Gestures. Appl. Sci. 2022, 12, 4452. https://doi.org/10.3390/app12094452;

5) in human-computer interaction: Adnan, M.; Sardaraz, M.; Tahir, M.; Dar, M.N.; Alduailij, M.; Alduailij, M. A Robust Framework for Real-Time Iris Landmarks Detection Using Deep Learning. Appl. Sci. 2022, 12, 5700. https://doi.org/10.3390/app12115700.

- Lines 102-104: Citation needed.

- Lines 130-135: As for the Abstract, also here the aim and the procedural steps of the work are just sketched briefly. They need to be introduced more extensively and clearly.

- Lines 166-167: Not clear, it should be explained better.

- Lines 198-199: Not clear. This is a crucial point, but it is just sketched. It should be explained more extensively.

- Lines 205-206: It is worthy to spend a few words more about the calculation of CV.

- In the Methods section, the used procedure is very well presented, but it is difficult to understand the final aim of the psychophysical experiment, what is the threshold to be calculated, how it is calculated and how it will be used for the stray light automatic detection procedure. An in-depth explanation about these points should be provided.

- Line 271: Typo: useless closed bracket

- Lines 283-284: This step is fundamental and deserve to appear in the block diagram in Fig. 8.

- The utility of the psychophysical experiment for the design of the automatic detection model is not stated properly. The connection between the two parts of the work should be more evident, in order to give a logical sense to the whole procedure.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The article is devoted to the study of such an applied problem as identifying stray light in VR display modules. To solve this problem, the authors propose combines psychophysical experiments and image-based measurement from three VR devices. Known methods K-Nearest Neighbors (KNN), Logistic Regression (LR), Support Vector Machine (SVM), and Random Forest (RF) are used for testing, and the results are evaluated according to known criteria (see formulas (1) - (4), line 306).

 Remarks on the content and design of the work.

 1. It is necessary to write more clearly the statement of a problem (lines 130-135) and to specify its novelty and the offered advantages

2. There are repetitions of reductions (see, for example, HMD line 10, line 36). There are other repetitions. This needs to be corrected throughout the text.

3. Exist abbreviations that are not decrypted (see, for example, CCD / CMOS line 75). This can be confusing for the untrained reader. This needs to be corrected throughout the text.

4. On the other hand, there are the same abbreviations denoting different values. For example, coefficient of variation (CV) line 206 and cross-validation (CV) line 296. This needs to be corrected throughout the text.

5. Table 3 does not have a column with the results calculated by formula (4) line 306. This is important to do.

6. References cited by the authors of the article are very old (as early as 1969, 1972) and do not fully reflect the current state of developments in this area. It is necessary to make a more detailed review of the literature on this topic over the past 3-5 years.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 5 Report

A Stray Light Detection Model for VR Head-Mounted Display Based on Visual Perception

The considered manuscript is dedicated to stray light perception in 3 VR devices. The authors perform an experimental study with 40 participants and devise a machine learning model predicting whether the stray light is perceived by a VR user or not. Overall, the topic is practical for VR assessment/development and relevant for Applied Sciences. However, I would recommend that the authors improve several important aspects before the paper is accepted.

First of all, they need to better position their study with respect to the current VR technologies. Currently, only 6 references out of 33 are within the last 5 years, while all the others are older.
Also, they must include the discussion section, in which they compare their results with the state-of-the-art and list the take-aways and the new knowledge (if any) with respect to the human visual perception.
They also need to be more specific about the use of the results to improve UX in VR. Their machine learning model is rather basic, with the yes/no classification, so does it have real applicability?

Next, they should better structure and somehow extend the methodology section.
For instance, how many data records were used to train and test the model?
Why did the authors decide to mix all the three VR sets in the data for the model (if I got this right)?
209 "the number of valid data is 37, 38, and 34 sets" - I would recommend using relative values when referring to valid data items.

The grammar and style needs some fixing. Particularly, the use of present tense when referring to past events (e.g., designing and running the experiment) is sometimes confusing. I would recommend using past tense instead.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The paper can be accepted as is.

Author Response

Thanks for reviewing the paper and giving comments.

Reviewer 2 Report

The reviewer recommends the revised manuscript for publication in Applied Sciences Journal.

Author Response

Thanks for reviewing the paper and giving comments.

Reviewer 4 Report

The authors, in general, correctly understood my remarks and made the necessary corrections regarding abbreviations, tables, more modern literature sources.

The quality of the article has improved.

Author Response

Thanks for reviewing the paper and giving comments.

Reviewer 5 Report

I have read the authors' replies to my comments and the updated version of the manuscript. Although I see some improvements and useful clarifications, I'm sorry to say that the authors did not address the comments in their essence, so I have to recommend another round of modifications.

The review of the field has not been improved. The authors have inserted just one sentence to superficially reply to the comment: "Many 56 recent studies have shown that VR can be applied in various fields such as medicine [10], 57 education [11], inclusivity [12], cultural heritage [13], and human-computer interaction 58 [14-16] to improve the quality of human life." I cannot agree that this single sentence results in a significant improvement.

The Discussion section is still missing. I cannot agree with the authors' reply that there no works of their peers to compare with or to use as the perspective. The findings related to human perception are not limited to VR/AR, and there are relevant publications in e.g. astronomy:

Pfisterer, R. N., Pompea, S. M., & Ellis, S. (2018). Recent advances in stray light modeling for large telescope/observatory systems. Modeling, Systems Engineering, and Project Management for Astronomy VIII, 10705, 705-714.

The use of the past tense is still recommended, as per the scientific writing style. But I suppose it is up to the editors to make the final decision.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 3

Reviewer 5 Report

I have read the authors' reply and the updated version of the manuscript. The review in the Introduction has been reasonably improved. Although I still believe that the Discussion section would benefit the paper, I do not mind it being published in the current form. So, I recommend acceptance and wish the authors best luck in their future research work.