Prediction of Health Problems Using Deep Learning Images and Bio-Signals

Abstract

We propose a study to predict health abnormalities by analyzing body temperature and the heart rate variability parameters of pulse waves. The research method firstly selects fever by applying a deep learning model to thermal images, and secondly, extracts heart rate variability from pulse waves through a photo-plethysmograph sensor. It analyzes the relevance of body temperature and health status by dividing the presence or absence of fever cases and comparing parameters related to autonomic nerves and stress cases. As a result of the experiment, the control group with normal body temperature had a mean pulse variability of 37.65, SDNN of 64.83, and RMSSD of 42.24. In contrast, in the experimental group, which consisted of individuals with fever, the average pulse variability was 31.91, the SDNN was 42.34, and the RMSSD was 26.80. Our research expects to be applicable to a thermal imaging system that can measure body temperature and bio-signals together and show the predicted results of health status for fever cases.

1. Introduction

When it comes to infection rates and number of deaths, COVID-19 has shown unprecedented records around the globe, and it is causing tremendous social and economic losses. At present, major countries around the world are establishing response systems in various areas in order to prevent this infectious disease from spreading further. Major characteristics of highly contagious viruses (e.g., influenza, MERS, COVID-19, etc.) are fever, coughing, and helplessness. When the condition is fever, the virus causes headache, muscular pain, vomiting, diarrhea, etc. [1].

One basic method to identify individuals suspected of infection and to prevent a secondary infection is to check fever by means of non-contact measuring methods such as thermal imaging camera and infrared thermometer. Non-contact thermometers recognize the heat generated from a human body by means of a sensor and convert it into numerical values. This type of means are widely used since they are less expensive. However, it is difficult to measure many individuals at once in a highly condensed space, since such methods should be manipulated manually in a close distance from the subject. Recently, as demands for thermal imaging cameras increased due to COVID-19, systems have been developed to detect multiple individuals in one screen frame, measure individuals’ body temperature, and issue alarms according to the designated critical values. When body temperature is detected by means of a thermal imaging camera, one way to increase the accuracy is to detect the human face in the image, map that area in the thermal image, and derive valid numerical values in the corresponding area [2,3,4].

Face recognition technology has been a major research subject in the area of image processing for the past decades, and it is widely used to identify and verify humans. The typical method of face detection is to find important features of a human face (e.g., eyes, nose, mouth, etc.) from input images and to normalize face images of a certain size in reference to such features. Based on Viola and Jones’ algorithm released in the early 2000s, the Boosted Haar Feature and Cascade algorithm was suggested in a relevant major study to extract features and detect a human face [5]. Thereafter, a convolutional neural network (CNN) and regions with a convolutional neural network (R-CNN) were proposed, which contributed significantly to the area of object detection and recognition in the form of deep learning [6,7,8]. Major examples of deep learning models are faster R-CNNs [9], upgrading the previous version of R-CNNs in terms of speed and performance, which extract a target area and classify objects from that area and you only look once (YOLO) [10], which distinguishes objects from the entire image at once. There has been continued research on face detection by means of such deep learning models for security and authentication [11,12], and recently, as the number of people wearing a mask to prevent droplet infection increases, some studies suggest methods to detect faces wearing a mask [13].

Measuring the body temperature to diagnose fever is the most basic and fundamental practice in the medical field to check the human body’s health condition quickly. Fever is a normal reaction to a human body’s immuno-stimulant, and it is closely related to the autonomous nervous system. Heart rate variability (HRV) is a parameter to digitize and analyze such autonomous nerve reactions. It is referred to as an indicator to analyze fine changes in heart beats in a scientific and quantitative manner and to determine conditions of the autonomous nervous system such as antagonism, stress, blood vessel condition, etc. HRV can be measured based on pulse waves or electrocardiogram measurements by means of photo-plethysmograph (PPG), which is a non-invasive sensing method [14,15,16]. Various methods are currently employed to determine the health status of various organisms. Examples include research on the causes of adverse health effects on organisms such as humans, plants, animals, bio-signal detection research using biosensors and machine learning, and research to identify abnormal signs using thermal image data [17,18,19,20,21,22,23,24,25]. There are, however, insufficient studies on the relationship between HRV and stress analysis for fever symptoms and on the detection of specific objects using deep learning for thermal imaging.

This study suggests a method to predict and identify individuals with fever by applying a convolutional neural network model specially designed for image detection to thermal images. Bio-signals of selected subjects are measured by means of a PPG sensor, and the correlation between the derived HRV parameter and body temperature is examined to predict any possible abnormal symptoms. Section 2 introduces prior studies related to our research topic. Section 3 discusses, first of all, the way to identify individuals with fever by applying deep learning to thermal imaging images, and then, secondly, discusses the types of parameters that can be derived to measure bio-signals and analyze health conditions by means of a PPG sensor. Section 4 presents the findings of the analysis on the correlation based on the body temperature and HRV experiment results. Finally, Section 5 presents the conclusions and future issues.

2. Related Work

In this section, we introduce research on face detection using deep learning and similar works on health status prediction using biometric signals. The method for detecting faces has continued to evolve from traditional object detection techniques (non-neural-based methods) to neural-based methods research. Face recognition is an access control system for security and is mainly used for recognizing and contrasting human faces in general images. However, in recent years, with the advent of a virus with a high mortality rate requires the preliminary control of building visitors, the frequency of facial recognition studies in thermal imaging cameras has increased.

2.1. Deep-Learning-Based Object Detection

Object detection models based on neural networks are largely divided into two-stage detector and one-stage detector. A two-stage detector is a structure in which the region proposal and classification stage are separated and operated sequentially, and R-CNN-based deep learning models are applicable. The R-CNN model is a model that separates region proposal, CNN, classifier, and regression and has a disadvantage in that the object detection performance is excellent, but the learning speed is slow. To solve this problem, fast R-CNN, which improves learning speed and performance by learning CNN, classification, and regression in a network by applying an end-to-end technique to a model, was proposed. However, for R-CNN and fast R-CNN, the process of generating a candidate region was performed independently of CNN, leaving inefficient problems in terms of learning speed. In order to improve this, the faster R-CNN to which the region proposal network (RPN) was applied was proposed, and the computational performance of the R-CNN model was enhanced. A one-stage detector is a structure in which regional proposals and classifications are made simultaneously, and it can solve both classification and localization problems at the same time. Representative deep learning models using one-stage detectors include the single-shot multibox detector (SSD) [26] and YOLO. YOLO is a model that divides the input image into a grid and predicts the bounding box and class probability for each cell. It is superior to R-CNN in terms of learning speed. However, if an object smaller than the grid is detected, there is a problem that the accuracy is lowered, so the version has been continuously upgraded. SSD is a model proposed to solve using multiscale feature maps and anchors [27]. In summary, the two-stage detectors are suitable for tasks that require a slow but high detection accuracy because the region suggestion and classification proceed sequentially. The one-stage detectors perform two steps at a time, so they learn faster but are used for tasks that require real-time detection due to their low accuracy.

2.2. Face Detection

The basic method for detecting a human face is to vectorize key features such as the eyes, nose, and mouth and detect or recognize an object by discriminating the distance between the feature points [28]. Recently, a recognition system for the purpose of controlling the access of non-mask users was required, so it was necessary to be able to inform whether or not a mask was worn. In addition, even when wearing a mask, it should be possible to expand to a system that can measure the temperature of an object using a thermal imaging camera. For this reason, it is necessary to accurately recognize the face area of the mask wearer. For example, there is a study that predicts the face area based on the human eye and extracts the mask wearing area [29] or predicts whether the mask is worn by applying various deep learning algorithms [30,31]. A representative study has recognized a face with an RGB camera, extracted the region of interest from the thermal imaging camera image, and then combined it to detect whether there is heat in the face region [32,33,34].

2.3. Prediction of Health Status Using Bio-Signals

Research on health condition prediction using biometric signals has been studied for a long time as precedent works for wearable devices and telemedicine services. Body temperature and pulse waves are bio-signals that can be used to quickly and easily check health status in a non-invasive measurement method. Body temperature is the easiest way to check your health because the immune system inside the body works to generate heat when you catch a cold or respiratory disease. Pulse wave is a waveform that records the beats of the chest wall and large blood vessels according to the beat of the heart, and heart rate (HR) and HRV can be obtained by analyzing the components of the waveform. HR and HRV are closely related to stress and affect health status [35,36]. However, since it is difficult to determine the state of health only by body temperature, it is determined by measuring other bio-signals together. In the past, research on systems that measure and integrate pulse waves, electrocardiograms, and blood pressure was the main focus. However, recently, as body temperature measurement has become the main focus, methods to measure body temperature, heart rate, and respiration using RGB-thermal image sensors are also being studied [37,38].

Our research combines individually separated methods into one to measure multiple bio-signals and predict health abnormalities through comparison.

3. Materials and Methods

The method proposed in this study consists of 3 parts, as shown in Figure 1: (1) detection and alarming of individuals with fever cases by means of deep learning from thermal images; (2) bio-signal measurement by means of a PPG sensor; and (3) prediction of abnormal symptoms of the health condition by means of deep learning and bio-signals.

In order to predict abnormal symptoms of the subject’s health condition by means of deep learning and bio-signals, first of all, the faster R-CNN model was applied to thermal imaging images, and individuals with fever were detected. The faster R-CNN model was optimized for learning and fever case detection in reference to datasets collected through a thermal camera. For fever cases selected through the deep learning model, bio-signals were measured by means of a PPG sensor. Based on the bio-signal measurement data, the correlation parameters of the HRV, autonomous nervous system, and stress were derived. Finally, the correlation between HRV parameters and the subjects was analyzed to determine any symptoms of health problems.

3.1. Object Detection in Thermal Image

3.1.1. Network Architecture

Faster R-CNN is an upgrade of the previous version of CNN. After the process shown in Figure 2, not only the object class but also the object location, area, and width in the given image are represented with a bounding box. The convolutional network part of faster R-CNN consists of the convolution layer and pooling layer, just like CNN. The feature map derived from this process is connected to the RPN and fully connected layer. Convolutional operation blurs images or highlights a certain corner or line, while pooling operation secures consistency of the specific location of a certain feature in the image. The feature map is generated by extracting image features after the alternate pass of these two layers. The RPN applies the anchor box to the feature map in a sliding window form. Anchor boxes may be varied in size and proportion. They suggest various areas where subjects to be detected may be located. In the suggested areas, the intersection over union (IOU) is calculated and a certain level of measurement is set as the condition for the final detection area. The classification and bounding box are then presented [9].

3.1.2. Learning

In order to detect individuals with fever from thermal images, it is required to secure a deep learning model that was trained in advance by means of datasets of individuals with fever. First of all, in order to acquire datasets, 500 jpg files of 768 × 576 pixel (BlueRed color mode) were collected by means of IMI TECH Co., Ltd’s (Anyang, Republic of Korea), the intellectual thermal image-based temperature sensing system (IMT-831GD). Collected thermal images were classified to individuals with fever and in normal conditions as shown in Figure 3. Figure 3a shows an image sample of the fever group, while Figure 3b shows a sample of the normal group. Collected thermal images went through the steps of annotation and labeling to be used as datasets for training of the deep learning model. Thermal images indicated the skin temperature in color. Acquired images indicated fine heat distribution on the face with color gradation. Accordingly, images were classified in consideration of heat distribution to high temperature and normal temperature. Datasets generated from this annotation process were used to train the faster R-CNN model. In this way, a new weight model for fever detection was acquired.

3.2. Prediction System of Health Status

3.2.1. Pulse Wave and HRV

To determine health conditions, methods to measure and analyze bio-signals are commonly used. Bio-signals are signals of biological phenomenon in a human body measured in an invasive or non-invasive way by means of various types of sensors. Since biological tissues generating such signals are combined organically, they are quite complicated and sensitive to external environments. Thus, they may reflect abnormal conditions when measured. Among biological signals, signals that best reflect the biological state of a body are temperature, pulse waves, and electrocardiogram. Among these, pulse waves and electrocardiograms are used to measure HRV, which is information relevant to heart beats. HRV-based methods are designed to measure the state of a body simply in a non-invasive manner and provide scientific and reliable biological information [15].

An electrocardiogram is used to measure the electric activity of a heart in a non-invasive manner by attaching 2 or more electrodes on the arms, legs, and chest. While it is possible to obtain various types of cardiac biological information on heart beats, irregular pulses, myocardial infarction, etc., it is difficult to measure in a crowded public place with many people [16]. Thus, pulse wave measuring methods to measure HRV on the basis of optical characteristics of skin are commonly used instead. PPG methods to measure pulse waves by means of a light source are used to extract information on heart beats by using an optimal detector and infrared ray or red/green light-emitting diodes. PPG-based heart beat measurement requires no separate electrodes attached to the skin and needs only a compact sensor on fingers or wrists to measure bio-signals. Thus, this method is convenient to be used in a public place. As shown in Figure 4, when light is emitted with finger surfaces facing each other, and with light-emitting diodes and light-receiving diodes attached on both upper and lower sides, it is possible to detect changes in the light permeability at the time (QRS complex) when blood passes through the peripheral blood vessel depending on the heart beats. Since the acquired pulse waves are small, signals obtained through an amplifying circuit are analyzed. As the obtained pulse waves are observed, it is possible to judge vascular diseases such as HRV, irregular pulse, and contraction.

3.2.2. Health Status and HRV

In order to examine health conditions, recently, not only physical indicators, but mental and stress-related indicators are also considered. The HRV is useful in analyzing such physical and mental health conditions in an integrated manner. Methods to analyze the HRV include ways to examine the frequency range and time domain in examination of autonomous nerve states. In reference to parameters collected from such methods as variables, stress indicators are derived and used in such methods.

Heart beat changes reflect complex biological activities related to autonomous nerves and various biological elements such as respiration, blood pressure, temperature, blood vessel, exercise, etc. Biological mechanisms have different frequency elements, respectively, and heart beat changes reflect the sum of such frequency elements. When the HRV is examined, a certain frequency range is analyzed to measure the influence of each mechanism. For frequency range analysis, parameters of very low frequency (VLF; 0.0033–0.04 Hz), low frequency (LF; 0.04–0.15 Hz), high frequency (HF; 0.15–0.4 Hz), and LF/HF are used. LF is a parameter reflecting activities of both the sympathetic system and parasympathetic system. When there is a disease causing an excessive reaction of the sympathetic system, LF increases. HF elements are closely related to the electric stability of a hart. When there are reactions such as stress, panic, and anxiety, HF decreases [2].

The time-domain analysis method is to calculate and analyze the time interval of normal heart beats. Parameters related to time domain analysis include SDNN (standard deviation of normal to normal interval) and RMSSD (root mean square of successive differences). As shown in Equation (1), SDNN is a parameter reflecting the general state of heart beat variance in reference to the standard deviation of heart beats. It indicates the ability to resist stress. As shown in Equation (2), RMSSD reflects short-term variance of heart beats. It is highly related to the parasympathetic system and decreases when the object is stressed [39].

$$SDNN=\sqrt{\frac{1}{N}{\sum }_{i=1}^{N}R{R}_i-{\overline{RR}}^2}$$

(1)

$$RMSSD=\sqrt{\frac{1}{N-1}{\sum }_{i=1}^{N-1}{(R{R}_{i+1}-R{R}_i)}^2}$$

(2)

4. Experimentation and Discussion

By means of the model with a weight adjusted through dataset learning, as stated in 2.1 above, images were acquired in real time from a thermal image camera and classified to identify individuals with fever. In order to analyze the HRV, UBioMacpa developed byBiosense Creative Co., Ltd. (Seoul in Republic of Korea) were utilized. Ten individuals were divided into groups with or without fever symptoms for the experiment. In

Table 1. Experimental Results of Control group and Experimental group.

(a) Control group (normal body temperature)
N	Pulse Variability	Mean BPM	SDNN	RMSSD
1	44.68	77.90	58.65	38.00
2	39.78	74.50	55.90	42.10
3	35.23	77.00	73.80	44.10
4	34.25	76.00	66.00	40.20
5	35.62	75.12	63.20	43.05
6	38.57	74.14	67.35	44.26
7	37.25	72.54	65.00	41.00
8	35.42	76.00	71.00	44.25
9	37.14	74.12	69.54	46.00
10	38.52	77.00	57.85	39.45
Average	37.65	75.26	64.83	42.24
(b) Experimental group (fever body temperature)
N	Pulse Variability	Mean BPM	SDNN	RMSSD
1	24.79	93.40	45.20	28.10
2	30.14	86.30	44.80	27.41
3	31.84	82.40	40.90	25.10
4	33.53	83.00	43.20	26.15
5	35.00	85.45	41.54	26.25
6	29.12	80.56	40.25	24.26
7	34.51	91.00	41.33	25.32
8	33.34	87.42	43.11	27.56
9	34.16	80.34	40.21	28.16
10	32.71	83.49	42.89	27.47
Average	31.91	85.34	42.34	26.58

shown below, Table 1(a) represents the control group, which is the normal body temperature group; Table 1(b) represents the experimental group identified as having fever symptoms; and Figure 5 displays the results. HRV parameters—pulse variability, mean beat per min (meanBPM), SDNN, and RMSSD—were compared to analyze correlations. Table 1 displays the measured values of ten control subjects and ten experimental subjects, for a total of twenty subjects. On average, the pulse variability of the control group was 37.65, mean beat per min was 75.26, SDNN was 64.83, and RMSS was 42.24. The pulse variability value of the experimental group was 31.91, mean beat per min was 91.00, SDNN was 42.34, and RMSSD was 25.32. As shown in Table 1 and Figure 5, the experimental group showed a lower level of pulse variability, SDNN, and RMSSD than the control group. Only the mean beat per min was higher than that of the latter.

Figure 6 shows the heart beat distributions and health conditions of the control group and experimental group. Figure 6a shows the control group, and Figure 6b shows the experimental group. As shown in these figures, the control group showed a distinctive triangular form of heart beat distribution, which indicates a good health condition. In contrast, the experimental group’s heart beat distribution is not of a clear triangular form, which indicates that stress was accumulated and the objects were not in a good health condition. The left side of the graph indicates the state of stress and irregular pulses in reference to the balance of autonomous nerves. The control group is in the range of normal health conditions, while the experimental group shows symptoms of chronic stress and the initial stage of irregular pulses.

The green table on the right of Figure 6 is a table showing the autonomic nerve balance. The image was extracted using the left heart rate distribution graph in Figure 6 and the mean BPM, SDNN, and RMSSD in Table 1. According to the positions of the green square pointer in Figure 6a,b, the stress and health status of the subject can be checked. A small green square pointer indicates stability when it is centered on the red bounding box. The green square pointer is based on the red bounding box, and the closer it is to the upper left, the more the parasympathetic nerve is activated, and the lower the right, the sympathetic nerve is activated. Hyperactivity of the parasympathetic nerve can be predicted to show signs of health abnormalities such as arrhythmia, as well as psychologically complex states such as anxiety and fear. Conversely, the activation of the sympathetic nerve can be predicted as a cause of chronic stress and disease, accompanied by a state of physical complexity as the state of tension and excitement of the body continues to be maintained. It was confirmed that the experimental group in Figure 6b was generally located in arrhythmia and acute stress rather than chronic stress. On the heart rate distribution graph, it was confirmed that the control group had a long bar graph and was centralized, while the experimental group had a short bar graph and was dispersed.

Table 2. Comparative result of detection model performance of the proposed system.

Detection Model	mAP (%)	Inference Speed (FPS)
Faster R-CNN (proposed system)	76.4	5
YOLO	73.7	81
SSD300	74.3	46

compares the detection performance (mean average precision) and frames per second (FPS) of faster R-CNN, a deep learning model used in the proposed system, with YOLO and SSD, which are similar models, in order to compare the performance of the detection models.

Table 3. Comparison of the characteristics of the existing system and the proposed system.

	Thermal Imaging System	Proposed System
Face Detection (mask)	Yes	Yes
Body Temperature	Yes	Yes
HRV and Stress Prediction	No	Yes

shows the functions of the existing thermal imaging system and our proposed system. In most of the thermal image access management systems, a face recognition algorithm is applied to the general image to find a face, and the body temperature is derived by mapping it with the thermal image in the corresponding area. The face recognition algorithm of the proposed system and the existing thermal imaging system both detect the face area; however, the thermal imaging system detects faces in the input real color image, whereas the proposed system is trained to detect faces directly in the thermal image. Deep learning is applied to the thermal image to determine if a person is recognized and if the individual has a fever. Existing thermal imaging systems can recognize faces and measure body temperature, but it is difficult to accurately predict the health status of a measurer because it does not have a separate function for measuring other bio-signals. Our proposed study uses the PPG sensor to additionally acquire the person’s heart rate and HRV data and compare and analyze the correlation with body temperature, so it is possible to predict the health status of the subject and make more accurate discrimination. Due to the fact that the proposed system detects faces from thermal images with different textures, the detection accuracy may be lower than that of existing thermal imaging systems; however, it has been confirmed that detecting the shape of the face and whether a mask is worn is not a problem.

5. Conclusions

This study presents a method to identify individuals with fever based on thermal images when deep learning is applied. In order to investigate the relationship between body temperature and health status, a study was proposed to extract HRV with a PPG sensor and predict the symptoms of abnormal health of the subject. As a result of the experiment, it turned out that the group with fever showed a low level of pulse variability, SDNN, and RMSSD and a high level of mean beat per min. In addition, it was possible to determine based on the heart beat distribution that individuals of high body temperature would not be in a good health condition due to accumulated stress. Rather, HRV parameters indicated that such individuals suffered acute stress and were in the initial stage of irregular pulses. In order to prevent the spread of the virus, it is necessary to make more detailed judgment in the field. Since the existing access control system only measures body temperature, it was difficult to determine the health status or condition of the subject. In order to accurately predict the abnormal symptoms of the body, it is necessary to analyze the correlation with other bio-signals that can predict health conditions. Therefore, we measured two bio-signals that can be easily checked in the field and confirmed that there is a correlation between body temperature and HRV.

Our study is expected to increase reliability with more detailed bio-signal information and health condition prediction results in specific situations in which abnormal symptoms need to be screened. If the research results are automated with an algorithm for analyzing the correlation between body temperature and HRV parameters, it is expected that it can be applied to a health condition monitoring system. Furthermore, it is expected that it will be used in a wider range of fields if the system is designed to detect regions in each part of the body and analyze the correlation between body heat information and the bio-signals of a specific part.