Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging
Abstract
:1. Introduction
- None/Minimal: AHI < 5 per hour
- Mild: AHI ≥ 5, but < 15 per hour
- Moderate: AHI ≥ 15, but < 30 per hour
- Severe: AHI ≥ 30 per hour
2. Proposed Method
2.1. Respiration Monitoring
2.1.1. Automatic ROI Detection
- (1)
- The highest temperature point is detected by using minMaxLoc. The minMaxLoc function is one of the OpenCV [38] libraries that returns minimum and maximum intensities found in an image with their (x,y) coordinates. It is assumed that the maximum pixel intensities of the thermal image refer to a human’s heat signature that is not covered by a blanket. The maximum pixel intensities found in the image correspond to the highest temperature of the body. We set the pixel to the center of the observation area. Then we draw a rectangle around the pixel, with the size of the square pixels depending on original frame resolution. In [39], the authors compared the ROI size of , , , , and pixels. They found that the size of the ROIs for respiratory rate estimation is usually smaller than that for heart rate estimation. Therefore, in this study, we consider the three different sizes as , , and , as shown in Figure 2. The result in empirical research has shown that the pixels provided the highest accuracy in accordance with the original frame resolution of 640 × 480.
- (2)
2.1.2. Breathing Motion Detection
2.1.3. Respiration Signal Analysis
- (1)
- The respiration can be extracted by detecting the chest movements, the breaths airflow, and the temperature change around the nostrils. However, the detection in a specific method cannot be guaranteed in the sleep monitoring because of the fixed camera position, and an independent subject posture may make the region out from the camera view. In such a case, an alternative method for respiration detection is required. It is reasonable to assume that the respiration can be detected by blending the temperature change of ROIs and breathing motion. Therefore, we combine three signals by employing the root mean square (RMS) to calculate the average of the respiration signals as (4).
- (2)
- The 3rd order of Butterworth bandpass filter [41] with a lower cutoff frequency of 0.05 Hz and a higher cutoff frequency of 1.5 Hz was applied. The frequency bound is equivalent to 3–90 bpm, based on the typical RR for an adult person (12–20 bpm) and monitoring the abnormal RR that is less than 12 bpm and higher than 20 bpm.
- (3)
- The Savitzky–Golay (SG) filter is a least-square polynomial filter that reduces noises while retaining the shape and height of waveform peaks [42]. Here, the SG filter was used to smooth the signal after the bandpass filter. The SG filter’s output increased the precision of the data without distorting the signal tendency. There are two parameters of the SG filter, including window length and the filter order, which closely relates to the performance of the filter. In this study, we tested the parameters and selected the optimal values to get the best-filtered signal, i.e., the window length of 51 and the polynomial order at 3rd were used. The result of SG filter still includes the small peaks, and thus a moving average is calculated to detect only the desired peaks and ignore small ones.
- (4)
- The fusion signal in Figure 4a was smoothed by the SG filter and moving average (see Figure 4b), and then the number of peaks is counted. Figure 4c depicted the peaks detection of the experiment signal, followed by the peaks detection of the reference signal in Figure 4d, which are assumed to correspond to the number of breaths. The findpeaks function is used with adjusting the width as 10 based on empirical research.
- (5)
- The number of peaks is calculated as breaths per minute (bpm) for each 60 s slice of input video (1020 samples at 17 fps) and was compared with the reference RR. For performance comparison, the accuracy of the RR estimation was tested using the RMSE defined as (5)
2.2. Body Movements Detection
3. Experimental Results
3.1. Experimental Setup
3.2. Respiratory Rate Estimation and Body Movements Detection
4. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Wheatley, I. Respiratory rate 3: How to take an accurate measurement. Available online: https://www.nursingtimes.net/clinical-archive/respiratory-clinical-archive/respiratory-rate-3-how-to-take-an-accurate-measurement-25-06-2018/ (accessed on 30 January 2020).
- Sheppard, M.; Wright, M.W. Principles and Practice of High Dependency Nursing; Elsevier Health Sciences: Amsterdam, The Netherlands, 2006. [Google Scholar]
- Cretikos, M.; Chen, J.; Hillman, K.; Bellomo, R.; Finfer, S.; Flabouris, A.; Investigators, M.S. The objective medical emergency team activation criteria: A case–control study. Resuscitation 2007, 73, 62–72. [Google Scholar] [CrossRef] [PubMed]
- Somers, V. American Heart Association Council for High Blood Pressure Research Professional Education Committee, Council on Clinical Cardiology. American Heart Association Stroke Council. American Heart Association Council on Cardiovascular Nursing. American College of Cardiology Foundation. Sleep apnea and cardiovascular disease: An American Heart Association/American College Of Cardiology Foundation Scientific Statement from the American Heart Association Council for High Blood Pressure Research Professional Education Committee, Council on Clinical Cardiology, Stroke Council, and Council On Cardiovascular Nursing. In collaboration with the National Heart, Lung, and Blood Institute National Center on Sleep Disorders Research (National Institutes of Health). Circulation 2008, 118, 1080–1111. [Google Scholar] [PubMed]
- Meoli, A.L.; Casey, K.R.; Clark, R.W.; Coleman, J.A.; Fayle, R.W.; Troell, R.J.; Iber, C.; Clinical Practice Review Committee. Hypopnea in sleep-disordered breathing in adults. Sleep 2001, 24, 469–470. [Google Scholar] [PubMed]
- Harvard Medical School, H.U. Understanding the Results: Sleep Apnea. Available online: http://healthysleep.med.harvard.edu/sleep-apnea/diagnosing-osa/understanding-results (accessed on 20 February 2020).
- Zagaria, M.A.E. Periodic Limb Movement Disorder, Restless Legs Syndrome, and Pain. US Pharm. 2015, 40, 19–21. [Google Scholar]
- Madhushri, P.; Ahmed, B.; Penzel, T.; Jovanov, E. Periodic leg movement (PLM) monitoring using a distributed body sensor network. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 1837–1840. [Google Scholar]
- Koolen, N.; Decroupet, O.; Dereymaeker, A.; Jansen, K.; Vervisch, J.; Matic, V.; Vanrumste, B.; Naulaers, G.; Van Huffel, S.; De Vos, M. Automated Respiration Detection from Neonatal Video Data. In Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM-2015), Lisbon, Portugal, 10–12 January 2015; pp. 164–169. [Google Scholar]
- AL-Khalidi, F.Q.; Saatchi, R.; Burke, D.; Elphick, H. Tracking human face features in thermal images for respiration monitoring. In Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications-AICCSA 2010, Hammamet, Tunisia, 16–19 May 2010; pp. 1–6. [Google Scholar] [CrossRef] [Green Version]
- Lee, Y.S.; Pathirana, P.N.; Steinfort, C.L.; Caelli, T. Monitoring and analysis of respiratory patterns using microwave doppler radar. IEEE J. Transl. Eng. Health Med. 2014, 2, 1–12. [Google Scholar] [CrossRef] [PubMed]
- Al-Naji, A.; Chahl, J. Remote respiratory monitoring system based on developing motion magnification technique. Biomed. Signal Process. Control 2016, 29, 1–10. [Google Scholar] [CrossRef]
- Nakajima, K.; Matsumoto, Y.; Tamura, T. A monitor for posture changes and respiration in bed using real time image sequence analysis. In Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No. 00CH37143), Chicago, IL, USA, 23–28 July 2000; Volume 1, pp. 51–54. [Google Scholar]
- Frigola, M.; Amat, J.; Pagès, J. Vision based respiratory monitoring system. In Proceedings of the 10th Mediterranean Conference on Control and Automation (MED 2002), Lisbon, Portugal, 9–12 July 2002; pp. 9–13. [Google Scholar]
- Wiesner, S.; Yaniv, Z. Monitoring patient respiration using a single optical camera. In Proceedings of the 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, France, 22–26 August 2007; pp. 2740–2743. [Google Scholar]
- Abbas, A.K.; Heimann, K.; Jergus, K.; Orlikowsky, T.; Leonhardt, S. Neonatal non-contact respiratory monitoring based on real-time infrared thermography. Biomed. Eng. Online 2011, 10, 93. [Google Scholar] [CrossRef] [Green Version]
- Fei, J.; Pavlidis, I. Thermistor at a distance: Unobtrusive measurement of breathing. IEEE Trans. Biomed. Eng. 2009, 57, 988–998. [Google Scholar]
- Lewis, G.F.; Gatto, R.G.; Porges, S.W. A novel method for extracting respiration rate and relative tidal volume from infrared thermography. Psychophysiology 2011, 48, 877–887. [Google Scholar] [CrossRef] [Green Version]
- Murthy, R.; Pavlidis, I. Noncontact measurement of breathing function. IEEE Eng. Med. Biol. Mag. 2006, 25, 57–67. [Google Scholar] [CrossRef] [PubMed]
- Murthy, R.; Pavlidis, I.; Tsiamyrtzis, P. Touchless monitoring of breathing function. In Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, CA, USA, 1–5 September 2004; Volume 1, pp. 1196–1199. [Google Scholar]
- Pereira, C.B.; Yu, X.; Czaplik, M.; Rossaint, R.; Blazek, V.; Leonhardt, S. Remote monitoring of breathing dynamics using infrared thermography. Biomed. Opt. Express 2015, 6, 4378–4394. [Google Scholar] [CrossRef] [PubMed]
- Alkali, A.H.; Saatchi, R.; Elphick, H.; Burke, D. Facial tracking in thermal images for real-time noncontact respiration rate monitoring. In Proceedings of the 2013 European Modelling Symposium, Manchester, UK, 20–22 November 2013; pp. 265–270. [Google Scholar]
- Alkali, A.H.; Saatchi, R.; Elphick, H.; Burke, D. Thermal image processing for real-time non-contact respiration rate monitoring. IET Circuits Devices Syst. 2017, 11, 142–148. [Google Scholar] [CrossRef]
- Bennett, S.L.; Goubran, R.; Knoefel, F. The detection of breathing behavior using Eulerian-enhanced thermal video. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 7474–7477. [Google Scholar] [CrossRef]
- Pereira, C.B.; Yu, X.; Czaplik, M.; Blazek, V.; Venema, B.; Leonhardt, S. Estimation of breathing rate in thermal imaging videos: A pilot study on healthy human subjects. J. Clin. Monit. Comput. 2017, 31, 1241–1254. [Google Scholar] [CrossRef]
- Kwasniewska, A.; Szankin, M.; Ruminski, J.; Kaczmarek, M. Evaluating Accuracy of Respiratory Rate Estimation from Super Resolved Thermal Imagery. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 2744–2747. [Google Scholar]
- Jagadev, P.; Giri, L.I. Non-contact monitoring of human respiration using infrared thermography and machine learning. Infrared Phys. Technol. 2020, 104, 103117. [Google Scholar] [CrossRef]
- Al-khalidi, D.F.; Saatchi, R.; Elphick, H.; Burke, D. Tracing the Region of Interest in Thermal Human Face for Respiration Monitoring. Int. J. Comput. Appl. 2015, 119, 42–46. [Google Scholar] [CrossRef]
- Fei, J.; Pavlidis, I. Analysis of breathing air flow patterns in thermal imaging. In Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, New York, NY, USA, 30 August–3 September 2006; pp. 946–952. [Google Scholar]
- Usman, M.; Evans, R.; Saatchi, R.; Kingshott, R.; Elphick, H. Non-invasive respiration monitoring by thermal imaging to detect sleep apnoea. submitted.
- Fei, J.; Pavlidis, I.; Murthy, J. Thermal vision for sleep apnea monitoring. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, London, UK, 20–24 September 2009; Springer: Berlin/Heidelberg, Germany, 2009; pp. 1084–1091. [Google Scholar]
- Seba, A.; Istrate, D.; Guettari, T.; Ugon, A.; Pinna, A.; Garda, P. Thermal-Signature-Based Sleep Analysis Sensor. Informatics 2017, 4, 37. [Google Scholar] [CrossRef] [Green Version]
- Chen, Z.; Wang, Y. Sleep monitoring using an infrared thermal array sensor. In Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems; International Society for Optics and Photonics: Bellingham, WA, USA, 2019; Volume 10970, p. 109701D. [Google Scholar]
- Al-Kalidi, F.; Elphick, H.; Saatchi, R.; Burke, D. Respiratory rate measurement in children using a thermal camera. Int. J. Sci. Eng. Res. 2015, 6, 1748–1756. [Google Scholar]
- Hu, M.; Zhai, G.; Li, D.; Fan, Y.; Duan, H.; Zhu, W.; Yang, X. Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation. PLoS ONE 2018, 13, e0190466. [Google Scholar] [CrossRef] [Green Version]
- Pereira, C.B.; Yu, X.; Goos, T.; Reiss, I.; Orlikowsky, T.; Heimann, K.; Venema, B.; Blazek, V.; Leonhardt, S.; Teichmann, D. Noncontact monitoring of respiratory rate in newborn infants using thermal imaging. IEEE Trans. Biomed. Eng. 2018, 66, 1105–1114. [Google Scholar] [CrossRef]
- Lorato, I.; Bakkes, T.; Stuijk, S.; Meftah, M.; De Haan, G. Unobtrusive respiratory flow monitoring using a thermopile array: A feasibility study. Appl. Sci. 2019, 9, 2449. [Google Scholar] [CrossRef] [Green Version]
- Opencv Dev Team. Operations on Arrays. 2019. Available online: https://docs.opencv.org/3.4/d2/de8/group__core__array.html (accessed on 20 February 2020).
- Tarassenko, L.; Villarroel, M.; Guazzi, A.; Jorge, J.; Clifton, D.A.; Pugh, C. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol. Meas. 2014. [Google Scholar] [CrossRef]
- Kumbhar, P.G.; Holambe, S.N. A Review of Image Thresholding Techniques. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2015, 5, 160–163. [Google Scholar]
- Virtanen, P.; Gommers, R.; Oliphant, T.E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; et al. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nat. Methods 2020, 17, 261–272. [Google Scholar] [CrossRef] [Green Version]
- Schafer, R.W. What is a Savitzky-Golay filter? [lecture notes]. IEEE Signal Process. Mag. 2011, 28, 111–117. [Google Scholar] [CrossRef]
- Sanchez-Marin, F.J.; Calixto-Carrera, S.; Villaseñor-Mora, C. Novel approach to assess the emissivity of the human skin. J. Biomed. Opt. 2009, 14, 024006. [Google Scholar] [CrossRef]
- Vernier. Go Direct® Respiration Belt. Available online: https://www.vernier.com/manuals/gdx-rb/ (accessed on 20 March 2020).
ROI Localization | |||||||
---|---|---|---|---|---|---|---|
Authors | Subjects | Exp Duration | Controlled Env | Simulated Breathing | Selection/ Detection | Area | Tracking |
Usman et al. [30] | Adult | 5 min | Yes | Yes | M | Nostrils | Yes |
Fei et al. [31] | Adult | 60 min | Yes | No | A-S | Nostrils | Yes |
Al-Khalidi et al. [34] | Children | 2 min | Yes | No | A-S | Tip of the nose | Yes |
Hu et al. [35] | Adult | 10 min | Yes | Yes | A-S | Nose, mouth | Yes |
Abbas et al. [16] | Infant | 2 min | Yes | No | M | Nostrils | No |
Pereira et al. [36] | Infant | 5 min | No | No | A-D | N/A | No |
Lorato et al. [37] | Adult | 2 min | Yes | Yes | A-D | N/A | No |
Our proposed | Adult | 60–90 min | No | No | A-D | N/A | No |
Subjects | Gender | Age (years) | Height (cm) | Weight (kg) | BMI (kg/m) |
---|---|---|---|---|---|
S01 | F | 28 | 162 | 56 | 21.34 |
S02 | F | 36 | 167 | 52 | 18.65 |
S03 | F | 31 | 162 | 50 | 19.05 |
S04 | F | 29 | 163 | 53 | 19.95 |
S05 | F | 32 | 158 | 54 | 21.63 |
S06 | M | 25 | 161 | 70 | 27.01 |
S07 | F | 31 | 151 | 47 | 20.61 |
S08 | F | 29 | 160 | 50 | 19.53 |
S09 | M | 30 | 168 | 55 | 19.49 |
S10 | F | 28 | 159 | 58 | 22.94 |
S11 | M | 28 | 180 | 75 | 23.15 |
S12 | M | 26 | 169 | 58 | 20.31 |
S13 | M | 27 | 168 | 59 | 20.90 |
S14 | M | 29 | 168 | 78 | 27.64 |
S15 | F | 32 | 153 | 56 | 23.92 |
S16 | F | 37 | 153 | 47 | 20.08 |
Subjects | Respiratory Rate (bpm) | Body Movements | ||||||
---|---|---|---|---|---|---|---|---|
Duration (s) | Reference | Experiment | RMSE | #Movements | #Frames | Duration (s) | Degree | |
S01 | 5371.05 | 12.71 | 14.05 | 1.56 | 14 | 269 | 15.69 | 1.12 |
S02 | 5397.54 | 13.69 | 14.22 | 1.11 | 15 | 199 | 11.65 | 0.78 |
S03 | 5379.37 | 16.75 | 14.78 | 2.20 | 7 | 63 | 3.69 | 0.53 |
S04 | 5192.31 | 12.23 | 13.37 | 2.00 | 35 | 642 | 37.86 | 1.08 |
S05 | 5212.78 | 17.62 | 14.39 | 3.32 | 9 | 200 | 11.72 | 1.30 |
S06 | 5200.51 | 16.45 | 14.48 | 2.23 | 9 | 214 | 12.53 | 1.39 |
S07 | 5332.39 | 14.38 | 14.36 | 1.47 | 16 | 218 | 12.80 | 0.80 |
S08 | 3495.39 | 14.65 | 14.29 | 1.18 | 15 | 749 | 43.48 | 2.90 |
S09 | 5407.22 | 12.17 | 14.60 | 2.68 | 0 | 3 | 0.17 | 0.00 |
S10 | 4520.26 | 14.91 | 14.79 | 0.75 | 5 | 91 | 5.33 | 1.07 |
S11 | 5346.70 | 13.12 | 13.76 | 1.25 | 16 | 417 | 24.42 | 1.53 |
S12 | 5361.45 | 13.25 | 15.37 | 2.35 | 7 | 140 | 8.20 | 1.17 |
S13 | 5399.74 | 18.61 | 15.99 | 2.79 | 5 | 20 | 1.17 | 0.23 |
S14 | 5380.52 | 15.15 | 14.32 | 1.49 | 16 | 535 | 31.14 | 1.95 |
S15 | 5315.23 | 16.32 | 14.40 | 1.99 | 12 | 225 | 13.19 | 1.10 |
S16 | 4287.12 | 14.43 | 14.41 | 0.72 | 6 | 250 | 14.51 | 2.42 |
Mean | 5100.00 | 14.78 | 14.47 | 1.82 | 1.21 | |||
STD | 537.63 | 1.93 | 0.60 | 0.75 | 0.74 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jakkaew, P.; Onoye, T. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors 2020, 20, 6307. https://doi.org/10.3390/s20216307
Jakkaew P, Onoye T. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors. 2020; 20(21):6307. https://doi.org/10.3390/s20216307
Chicago/Turabian StyleJakkaew, Prasara, and Takao Onoye. 2020. "Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging" Sensors 20, no. 21: 6307. https://doi.org/10.3390/s20216307
APA StyleJakkaew, P., & Onoye, T. (2020). Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors, 20(21), 6307. https://doi.org/10.3390/s20216307