Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation^†

Round 1

Reviewer 1 Report

Manuskrypt ID applsci-1014107 „Real-time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation”

The manuscript describes how to find and improve methods of non-invasive assessment of heart rate variability. The problem is presented in a very detailed way. All descriptions are clearly formulated and based on the available literature. The objectives are clearly stated as is the description of the methods used. The manuscript will certainly be an interesting aspect of the new methods of assessing heart rate variability and can be accepted for publication in its current form.

Author Response

The authors would like to thank the reviewer for his/her valuable effort and comments.

Reviewer 2 Report

Overall:
The article has three main goals: 1) Present a novel processing pipeline for rPPG and compare it in different datasets 2) Introduce new rPPG dataset, VicarPPG2; and 3) provide manual and clean annotations for existing rPPG datasets.

Major comments:
1. Captions in Tables 3 and 4 are over the over the tables. Improve the presentation of such tables.

Minor comments:
1. Is there are reason to not make public the StableSet and EatingSet rPPG datasets?
2. In section 3.3.1, the range 0.7 - 4 Hz corresponds to 42 - 240 bpm (not 200 bpm).
3. Always add and space between quantity and units, i.e., 60 Hz instead of 60Hz.
4. For sake of comparison use the same y axis for all the subplots in Figure 6. Same comment for Figure 7 and 8.
5. Figure 9, put caption at the bottom, and use the same y axis when required.
6. Figure 10, caption location. For Figure 10b, use different shapes for the databases.
7. Figure 11, caption location. For VIPL-HR and Moli-PPG use same y axes.

Author Response

First of all, we thank the reviewer for their effort and the very valuable feedback.

In this note, we address each of the reviewer's comments:

1. "Captions in Tables 3 and 4 are over the … tables. Improve the presentation of such tables."
   
   
   • The captions of Tables 3 and 4 are intentionally placed over areas of the table that would have otherwise contained only empty cells. This placement was intentional because we wanted to place both HR and HRV tables together on the same ‘horizontalized’ page. This way, the results of both HR and HRV can be seen at a glance on this easy-to-find page, thereby turning it into a sort of go-to reference page for results and making comparison of methods easier for the reader.
     
   • However, as the reviewer noticed, it can appear as if the captions are blocking actual content of the table. To clear this out, we have now added double-lined bottom borders to the table to emphasize that the table limits end to the top and left of the caption area. In addition, we have also added text that explicitly states this. We hope that this makes the presentation of the results much clearer. We thank the reviewer for pointing out this flaw, and hope that they see the merit of saving space to place both tables together on this page.
     
     
2. "Is there (any) reason to not make public the StableSet and EatingSet rPPG datasets?"
   
   
   • In the StableSet and EatingSet datasets, a majority of the participants did not consent to the sharing of their video material outside of the project group. Therefore, we are only able/allowed to share the results of our analysis on these datasets, but not the videos themselves.
   • We have added clarifying text to the dataset descriptions which briefly explain that these datasets are not publicly available.
     
     
3. "In section 3.3.1, the range 0.7 - 4 Hz corresponds to 42 - 240 bpm (not 200 bpm)."
   
   
   • Indeed, the Hz to bpm conversion was incorrect. We have now corrected the text to say “42 - 240 bpm”.
     
4. "Always add and space between quantity and units, i.e., 60 Hz instead of 60Hz."
   
   
   • We agree. We have reviewed the whole text and now ensured this consistent spacing between quantity and units.
     
5. "For sake of comparison use the same y axis for all the subplots in Figure 6. Same comment for Figure 7 and 8."
   "Figure 9 … use the same y axis when required."
   "(Figure 12) … For VIPL-HR and Moli-PPG use same y axes."
   
   
   • Figures 6, 7, 8, 9, and 12, all contain groups of bar charts, typically having separate charts for each dataset (with the bars representing different conditions within the dataset). Because the datasets vary vastly in terms of their content, quality, and difficulty, the errors measured on them also like in vastly different ranges (e.g., errors on PURE lie in the range of 0.2 - 0.9 bpm, while errors in VIPL are between 3 - 20 bpm). If all these error results from different datasets were plotted on the same y-axis, many variations in results would not be noticeable at all. Therefore, the y-axes have been scaled differently for different datasets.
     
   • Besides, the results only focus on intra-dataset conditional comparisons (for which the bars already use a common y-axis within the graph), and not on inter/cross-dataset comparisons. Thus, the value added by having the same y-axis across different graphs is minimal.
     
     
6. "Figure 9, put caption at the bottom. …"
   "Figure 10, caption location. …"
   "Figure 11, caption location. …"
   
   
   • We have now adjusted the location of the captions of these figures such that they appear consistently at the bottom part of the figures.
     
     
7. "… For Figure 10b, use different shapes for the databases."
   
   
   • We thank the reviewer for this good suggestion! We have now introduced different unique shapes to denote different datasets (in addition to colours, which was already the case).