iBVP Dataset: RGB-Thermal rPPG Dataset with High Resolution Signal Quality Labels

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This work constructed a new imaging blood volume pulse dataset of synchronized RGB and thermal infrared videos with PPG ground-truth signals. This work performed dense signal quality assessment to discard noisy segments of ground-truth and corresponding video frames. An end-to-end machine learning framework was further presented for reliable estimation of BVP signals.

 

Some major concerns need to be addressed in the revision.

(1) Some approaches using more recent deep learning techniques like [1-3] could be discussed in Section Introduction to enrich the related works.

(2) The examples with RGB, thermal and PPG signals should be jointly illustrated in the manuscript, while now only RGB and PPG samples are shown separately.

(3) The proposed method seems to only use RGB modality to estimate the BVP signal, while the influences of thermal modality and the combination of RGB and thermal are not shown. The authors are suggested to compare the two modalities and their combination in one deep model on the self-collected data.

(4) If possible, newly published methods in the year of 2022 or 2023 are suggested to use in the comparison experiments.

[1] Yu, Z., Shen, Y., Shi, J., Zhao, H., Cui, Y., Zhang, J., ... & Zhao, G. (2023). Physformer++: Facial video-based physiological measurement with slowfast temporal difference transformer. International Journal of Computer Vision, 131(6), 1307-1330.

[2] Liu, L., Xia, Z., Zhang, X., Peng, J., Feng, X., & Zhao, G. (2023). Information-enhanced network for noncontact heart rate estimation from facial videos. IEEE Transactions on Circuits and Systems for Video Technology.

[3] Zhang, X., Xia, Z., Dai, J., Liu, L., Peng, J., & Feng, X. (2023). MSDN: A multistage deep network for heart-rate estimation from facial videos. IEEE Transactions on Instrumentation and Measurement.

Comments on the Quality of English Language

None.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Non-invasive methodologies like rPPG have an undeniable and easily justifiable value in the domain of Biomedical Engineering, and the creation and sharing of related datasets and model architectures/implementations enable collaboration and facilitate the development of new diagnostic tools, procedures, etc.

This is a well-written and methodologically sound paper that presents a dataset (iBVP: a combination of facial video [RGB and thermal] with quality-labeled PPG signals) and an ML framework (iBVPNet) and explores the effectiveness of an evaluation metric (MACC) for rPPG method assessment.

There are some issues that should be addressed, as follows:

-In lines 291-292, the authors state that IR thermal image frames are "uniquely" provided by the iBVP dataset. However, in Table 1, there is a reference to another dataset, MMSE-HR, that also provides this kind of data. If this is a matter of free access, this should be clarified.

- In the study, the iBVP dataset is tested with different rPPG models (including iBVPNet). However, there are no comparisons regarding the performance of iBVPNet with the use of other existing datasets. Throughout the study, iBVPNet is assessed utilizing only the iBVP dataset, providing no indication of its performance using diverse datasets. If there is a reason for that (technical difficulties, reserved for future research, or others), I believe that the authors should discuss about it in the respective section(s).

- As far as the metric MACC (Maximum Amplitude of Cross Correlation) is concerned, a) there is an ambiguity with the existing literature, in which "MACC" is referred also as Maximum Averaged Cross-Correlation. The authors should address this ambiguity and maybe provide a short reference to how exactly this metric is derived; b) there is no justification regarding its effectiveness and why this metric should be used instead of the widely used in the literature (SNR, RMSE, etc.)

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Overall, the paper presents a valuable contribution to the field of rPPG research, but further clarifications, evaluations, and discussions are needed to enhance its impact and relevance. Addressing the below points would improve the overall quality and comprehensiveness of the paper.

 

1.      The conversation regarding the constraints and difficulties of utilizing thermal infrared imaging for rPPG extraction is enlightening. The paper should explore the specific technical challenges and possible solutions more thoroughly. Offering practical suggestions or strategies to address these challenges would enhance the conversation and better direct future research efforts.

2.      The study may have had a restricted participant pool in terms of demographics (e.g., age, gender, ethnicity), potentially impacting the generalizability of the results. Increasing the diversity of participants would improve the dataset's strength and the model's performance among various population groups.

3.      The study primarily examines signal quality assessment and BVP signal extraction, without thorough annotation or analysis of other pertinent physiological signals or contextual information such as respiration rate or skin perfusion. Adding more annotations could enhance the dataset and enable more thorough analyses.

4.      The criteria for assessing signal quality can be subjective or biased, depending on the expertise of the annotators or the algorithms used. Improving inter-rater reliability or investigating other methods for assessing signal quality would strengthen the accuracy and consistency of the results.

5.      Ground-truth PPG signals are utilized as references for training and evaluation, but their accuracy and reliability may not be assured, particularly in difficult recording conditions or for specific participants. Evaluating the accuracy of the ground-truth signals and investigating possible sources of error or uncertainty would enhance the reliability of the dataset and model assessments.

 

6.      The paper emphasizes the significance of signal quality evaluation in rPPG research and presents a new method (SQA-PhysMD) for deducing detailed signal quality metrics. Exploring the effectiveness and limitations of SQA-PhysMD, along with comparing it to other methods, would enhance our understanding of its utility and implications for rPPG applications.

 

Comments on the Quality of English Language

Overall, the paper presents a valuable contribution to the field of rPPG research, but further clarifications, evaluations, and discussions are needed to enhance its impact and relevance. Addressing the below points would improve the overall quality and comprehensiveness of the paper.

 

1.      The conversation regarding the constraints and difficulties of utilizing thermal infrared imaging for rPPG extraction is enlightening. The paper should explore the specific technical challenges and possible solutions more thoroughly. Offering practical suggestions or strategies to address these challenges would enhance the conversation and better direct future research efforts.

2.      The study may have had a restricted participant pool in terms of demographics (e.g., age, gender, ethnicity), potentially impacting the generalizability of the results. Increasing the diversity of participants would improve the dataset's strength and the model's performance among various population groups.

3.      The study primarily examines signal quality assessment and BVP signal extraction, without thorough annotation or analysis of other pertinent physiological signals or contextual information such as respiration rate or skin perfusion. Adding more annotations could enhance the dataset and enable more thorough analyses.

4.      The criteria for assessing signal quality can be subjective or biased, depending on the expertise of the annotators or the algorithms used. Improving inter-rater reliability or investigating other methods for assessing signal quality would strengthen the accuracy and consistency of the results.

5.      Ground-truth PPG signals are utilized as references for training and evaluation, but their accuracy and reliability may not be assured, particularly in difficult recording conditions or for specific participants. Evaluating the accuracy of the ground-truth signals and investigating possible sources of error or uncertainty would enhance the reliability of the dataset and model assessments.

6.      The paper emphasizes the significance of signal quality evaluation in rPPG research and presents a new method (SQA-PhysMD) for deducing detailed signal quality metrics. Exploring the effectiveness and limitations of SQA-PhysMD, along with comparing it to other methods, would enhance our understanding of its utility and implications for rPPG applications.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

The authors presented a new rPPG dataset named iBVP in this manuscript. They also introduced the SQA-PhysMD method for signal quality assessment. In addition, the iBVPNet network was proposed for BVP signal estimation and experiments were carried out on the iBVP dataset, compared with several SOTA methods. The results demonstrated the efficacy of the iBVPNet model on the RGB channel. Overall, the methodology and results are presented concisely and clearly. The manuscript is well-organized and easy to follow. Here are some comments and suggestions:

1.      Only the gender information of the participants was included. Suggest providing additional demographic information such as age range and race. It might be beneficial to discuss if race or other factors may impact accuracy.

2.      Suggest expanding the discussion on qualitative comparison results, and perhaps highlighting several key differences in Fig. 5.

3.      May include the information about hardware configs. and software environment information for better repeatability.

4.      May directly include the hardware specs in Sec. 2.3. It may not be necessary to list them as references.

5.      A non-deidentified face appeared in Fig. 4, confirming the compliance with privacy policies.

6.      Provide a link to the directory of the dataset for accessibility after the acceptance of this manuscript.

 

7.      Correct some minor issues, such as defining “HCI” in Line 2, Page 1.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf